Rhizomatic Education

31 October 2008

In this article Cormier discusses the ways in which the curriculum may need to change to reflect the new ways in which knowledge is constructed – through networks and communities of learners, as opposed to the traditional model of content transmitted by teachers. There is one reference to assessment where Cormier looks at how the traditional assessment is against that content.

To quote the article

“Information is the foundation of knowledge. The information in any given field consists of facts and figures, such as may be found in the technical reference manuals of learning; in a nonrhizomatic model, individual experts translate information into knowledge through the application of checks and balances involving peer review and rigorous assessment against a preexisting body of knowledge. The peers and experts are themselves vetted through a similar sanctioning process that is the purview, largely, of degree-granting institutions. This process carries the prestige of a thousand-year history, and the canon of what has traditionally been considered knowledge is grounded in this historicity as a self-referential set of comparative valuations that ensure the growth of knowledge by incremental, verified, and institutionally authorized steps. In this model, the experts are the arbiters of the canon. The expert translation of data into verified knowledge is the central process guiding traditional curriculum development.”

The other side of the coin is not discussed but I suspect that in the new ‘rhizomatic’ (and I’m no biologist so the metaphor is lost on me) model has peer and expert assessment. It’s the old thing about how do we know something is good – often because it is valued by those who need to use it (predictive validity as per the discussion at Cambridge – value <=> validity).

Linked from ERN (George Siemens)

Cormier, D (2008) Rhizomatic Education: Community as Curriculum in Innovate, 4, 5, July 2008 [online] available at http://innovateonline.info/index.php?view=article&id=550&action=article accessed 31/10/08


Sir Mike Tomlinson lecture

22 October 2008

Sir Mike Tomlinson, chair of the working group on 14-19 reform that led to the 2003 Tomlinson Report, came to NTU today to give the RSA Shipley Lecture. This year the lecture was also a memorial to former NTU and RSA stalwart Anne Bloomfield. The subject, dear to her heart and to Sir Mike’s, was “Vocational education should be a rounded education“.

With the backdrop of the history of attempted introductions of vocational education (the 1944 Butler Education Act with its tripartite system, TVEI, GNVQs, Curriculum 2000 and Diplomas), Tomlinson argued for the move away from debates about ‘parity of esteem’ towards a view of the ‘credibility and value’ of qualifications. Echoes here of the value and validity arguments of Monday’s seminar at Cambridge.

It was also notable that the lecture included issues of how ‘true’ vocational education must have

  • relevance to 16-year olds (face validity),
  • a knowledge base that is used in, and applied to, occupational areas – however broadly drawn (validty determined by use, not by the test itself)
  • a theoretical content balanced with sector-based skills (content validity)

Again this echoes with Monday. Another thread running through was the role of assessment (systems) in undermining the vocational educational initiatives – TVEI assessment becoming ‘traditional’, GNVQ assessment being changed to equate to GCSE/A level, key skills being decoupled from GNVQs, Curriculum 2000’s insistence on equal numbers of units per qualification with a convergence of assessment types.

Also mentioned, although not in the same sense of ‘undermining’ was the persistence of the BTEC model and the way that NVQs were never envisaged to be other than accreditation mechanisms for on the job training.

The BTEC model of occupational credibility and general education was the model that was paramount in the description of vocational education with the caveat ‘what is general education’?

Throughout I was wondering where ICT fits into all this. Never mentioned as a ‘subject’ nor even as a ‘skill’ it was conspicuous by its absence. It is, of course, present in the specialised diplomas and as a functional skill although the former may be bedevilled by the wide diversity of the sector it is serving, I fear.

Tomlinson was upbeat about the Diplomas but focused especially on the need to get a true progression from level 1 through to 3. The custom of level 1 being what you get if you fail level 2 (GCSE grades D-G rather than A*-C) must not be repeated he urged. Also the need to get level 2 threshold systems so that learners who do not reach that threshold at , I would say the magical (and arbitrary, age of 16 could do so by subsequent credit points accumulation – rather than ‘repeating GCSEs’, a model that doesn’t serve well, he argued.

Another hour of useful insights.


Cambridge Assessment seminar

21 October 2008

I attended  a seminar, on the subject of validity, one of a series of events run by Cambridge Assessment (CA). It was led by Andrew Watts from CA.

This was extremely informative and useful, challenging my notions of assessment. As the basis for his theoretical standpoint Andrew used these  texts 

  • Brennan, R (2004), Educational Measurement (4th edition). Westport, CT: Greenwood
  • Downing, S (2006) Twelve Steps for Effective Test Development in Downing, S and Haldyna, T (2006) Handbook of TEst Development. NY: Routledge
  • Gronlund, N (2005), Assessment of Student Achievement (8th edition). NY: Allyn and Bacon [NB 9th edition (2008) now available by Gronlund and Waugh]

He also referred to articles published in CA’s Research Matters and used some of the IELTS materials as examplars. 

The main premise, after Gronlund, is that there is no such thing as a valid test/assessment per se. The validity is driven by the purposes of the test. Thus a test that may well be valid in one context may not be in another. The validity, he argued, is driven by the uses to which the assessment is put. In this respect, he gave an analagy with money. Money only has value when it is put to some use. The ntoes themselves are fairly worthless (except in the esoteric world of the numismatist). Assessments, analogously, have no validity until they are put to use.

Thus a test of English for entrance to a UK university (IELTS) is valid if, the UK university system validates it. Here then is the concept of consequential validity.  It is also only valid if it fits the context of those taking it. Here is the concept of face validity – the assessment must be ‘appealing’ to those taking it.

Despite these different facets of validity (and others were covered – predictive validity, concurrent validity, construct validity, content validity), Gronlund argues that validity is a unitary concept. This echoes Cronbach and Messick as discussed earlier. There is no validity without all of these facets I suppose would be one way of looking at this.

Gronlund also argues that validity cannot itself be determined – it can only be inferred. In particular, inferred from statements that are made about, and uses that are made of, the assessment.

The full list of chacteristics that were cited from Gronlund are that validity

  • is inferred from available evidence and not measured itself
  • depends on many different types of evidence
  • is expressed by degree (high, moderate, low)
  • is specific to a particular use
  • refers to the inferences drawn, not the instrument
  • is a unitary concept
  • is concerned with the consequences of using an assessment

Some issues arising for me here are that the purposes of ICT assessment at 16 are sometimes, perhaps, far from clear. Is it to certificate someone’s capability in ICT so that they may do a particular type of job, or have a level of skills for employment generally, or have an underpinning for further study or have general life skills, or something else, or all of these? Is ‘success’ in assessment of ICT at 16 a necessary pre requisite for A level study? For entrance to college? For employment? 

In particular I think the issue that hit me hardest was – is there face validity: do the students perceive it as a valid assessment (whatever ‘it’ is).

One final point – reliability was considered to be an aspect of validity (scoring validity in the ESOL framework of CA).


KS3 SATS scrapped in England

16 October 2008

This somewhat unexpected announcement was made this week. Tests for 14 year olds in maths, English and science have been scrapped. Given that many schools start their GCSE/level 2 courses at 13 now, especially in ICT, this might change radically the ways in which the middle years of secondary are organised. It may also affect students’ perceptions of assessment as they will not have had those high stakes external tests at 14.


Tutorial part 2

16 October 2008

I had a tutorial (by telephone) today with the other part of my supervisory team. An interesting model emerges that develops the earlier one:

What emerged was a clarity of vision: I am looking at

A how year 11 students perceive ICT capability and
B how the assessment system (at 16) perceives it.

My project is to define the difference between A and B and to suggest ways in which the two may be aligned.

What now emerges is the more sophisticated notion of a number of views of what ICT capability is, with some sort of Heideggeran absolute at the intersection. Thus there may be four views of what ICT capability is:

  • the view of the awarding bodies
  • the view of the students
  • the view of the education system (policy)
  • the observed view from research

Is there also then a Heideggeran absolute, autonomous view somewhere in the intersection of all these?

We also talked about the notions of perception and interpretation of the students view and came down to the question: How authentic and relevant does assessment feel to students? This, of course, has limitations as due to precisely because of the hermeneutical considerations of it the students’ view.

Building on the notion of the abstract view that would define assessment of ICT in absolute terms (and my stance which rejects this in favour of the diversity of views listed above), we then talked about the importance of the social cultural view in which students’ interpretations are coloured by their class, peer groups, families etc.

One final concept is the emergence of literature on assessment as learning and how the ‘teaching to the tests’ means that students are spoon fed and do not learn beyond the framework of assessment.


History of assessment

14 October 2008

Blindingly obvious I guess, but nevertheless not a field I have mined much. “History of assessment” needto be a significant contextual filter for my research.

I am attending some of the seminars of the Cambridge Assessment Network as and when I can (Kathryn Ecclestone on Ofqual, Harry Torrance on Policy and Practice). There are others that I have been unable to attend but would like to have done. Fortunately useful overviews such as this one from Helen Patrick are often put online.


Taxonomy of difficulties in the assessment of ICT

2 October 2008

This paper, from the Assessment in Education Unit at Leeds, is again bang on the line of what I am doing. Albeit that they are looking at the KS3 onscreen ICT tests (the AEU were commissioned as part of the evaluation of that project). Nevertheless there are some very pertinent analyses of the didfficulties students, and the system, encounter in assessment of ICT. For example

… sources of difficulty that relate to the subject being assessed. The assessment of ICT brings particular risks. As McFarlane (2001) points out, assessment of ICT capability need not in itself be computer-based, but as in this case it was, the sources of difficulty in our set that are associated with this aspect all relate also to on-screen assessment, e.g.

Pupils know enough to succeed in the tasks without using ICT for all the steps.

The demands in the interaction between tasks and software on short-term memory and organisational skills are greater than the level of ICT capability that is being assessed.


Activity theory and ICT

2 October 2008

Now that the new academic year is under way, and I am in a new job (again) I hope to be able to crack on with this project. If not.. well I need to decide one way or the other.

Anyway, I am in the middle of three days study leave and ma busily writing up what I have so far. In doing so I have also come across some useful things which I will include here.

Firstly a paper from the School of Education conference at Leeds University by Aisha Walker. This makes some interesting links between activity theory and attainment in ICT and provides a model that may be useful when I come to look at the data colleection and analysis. 

The title “What does it mean to be good at ICT” really caught my eye. That’s what it’s all about isn’t it?


A three axis model

1 May 2008

Taking the ideas from the previous post and putting them into a diagram I get this

Some assessment uses ICT (or technology) – this is e-assessment (x axis).

Some assessment is designed to assess ICT capability (y axis).

Elliott’s Assessment 2.0 seems to be using ICT, not as e-assessment, but as a medium for allowing judgement to be made about the ICT capability (z axis).

Now of course, analysing any one particular assessment methodology one could locate it in this three-dimensional space. for example:

A traditional written paper would be on the y-axis. The NAA online assessment activities designed for KS3 would be in the space between all three axes (with perhaps a lower y- and z-values than x-value. Coursework would have an x-value of 0 but would have some components of y and z. Online assessments such as the driving test would be on the x-axis.

My questions here are “Where is the highest validity”? and “Where is the highest reliability?”. How does one use Elliott’s Assessment 2.0 to determine success in a certificated qualification?


Elliott (2008) Assessment 2.0

1 May 2008

Much as I dislike the nomenclature (Assessment 2.0), I found this paper by Bobby Elliott (and thanks to my colleague Bruce Nightingale and the ALT newsletter for bring the name to my attention) illuminating on many levels. Firstly here was someone making the links between theways in which technology is reportedly used by young people and the ways it could be used for technology. Secondly the author works for a government agency – the Scottish Qualifications Authority (SQA). Is this evidence of policymakers thoughts are changing to embrace the vicarious ways in which evidence of learning can be presented by technological opportunity?

My thoughts return though to the Macfarlane distinction between assessment of technology (eg the ICT curriculum) and assessment through technology (ie the methodology). This paper by Elliott seems to be moving a little away from the latter and perhaps towards the former. But perhaps, also, it is defining a third axis – assessment of technological capability through evidence presented through that technology. Maybe it is asking the question ‘What should be assessing?’ (ie the curriculum) rather than ‘How should we assess it’ (the methodology). But more than that it is saying can we assess the ‘what’ through the ‘how’.

The impressive list of tools that may be used for evidence presenting (and assessment) in Elliott’s paper also underlines my sceptcism of a one size fits all technological solution to assessment. And when I look down that list I am reminded of surveys presented by Terry Freedman (at a TDA conference in Nov 07) and others that show that young people’s use of tools is very diverse and very thinly spread. It is also very transient – MySpace here today gone tomorrow.

The very tool that Elliott uses to present his iPaper may well be a case in point. What if an awarding body decided Scribd was the thing to use. How long before it becomes the sliced bread superceded by the next best thing? How do we build in agility for assessment so that it does not become an exercise in rewarding the fashionable? (as opposed to the current system which rewards the old-fashionable).

PS Yes it’s been a long time… Higher Education management – ie my temporary role for 07/08 – and PhDs don’t easily mix… but I know that is also my excuse… and I’m sticking to it…


Futurelab (2006) Social software and learning

20 July 2007

I hadn’t come across this paper until today…

To quote from the executive summary…

“This paper is focused on exploring the inter-relationship between two key trends in the field of educational technologies. In the educational arena, we are increasingly witnessing a change in the view of what education is for, with a growing emphasis on the need to support young people not only to acquire knowledge and information, but to develop the resources and skills necessary to engage with social and technical change, and to continue learning throughout the rest of their lives. In the technological arena, we are witnessing the rapid proliferation of technologies which are less about ‘narrowcasting’ to individuals, than the creation of communities and resources in which individuals come together to learn, collaborate and build knowledge (social software). It is the intersection of these two trends which, we believe, offers significant potential for the development of new approaches to education. ”

These new approaches include those encapsulated in these quotes

“Today, the use of social software in education is still in its infancy and many actions will be required across policy, practice and developer communities before it becomes widespread and effective. From a policy perspective, we need to encourage the evolution of the National Curriculum to one which takes account of new relationships with knowledge, and we need to develop assessment practices which respond to new approaches to learning and new competencies we expect learners to develop.

and

“A rigid curriculum inhibits the development of the knowledge and skills that may be useful in the 21st century. If we are to promote the benefits of problem solving and collaboration then they need to be validated and legitimated by the assessment system. This is the greatest challenge for education policy.

There is some sort of mapping in my mind between the two trends identified in the first paragraph and the two aspects of my research.

The report has a change in the purpose of education and its interface with the change in technologies. I have the two views of ICT/assessment of ICT, (and maybe also views of the purpose of ICT in education). Putting these four onto a diagram I see that two are to do with system, theories etc and two are to do with learners, users, practice.

futurelabandme1.gif


General Teaching Council report calls for no school tests for under-16s

11 June 2007

Call to ban all school tests for under-16s | UK News | The Observer (10 June 2007)


“All national exams should be abolished for children under 16 because the stress caused by over-testing is poisoning attitudes towards education, according to an influential teaching body.

In a remarkable attack on the government’s policy of rolling national testing of children from the age of seven, the General Teaching Council is calling for a ‘fundamental and urgent review of the testing regime’. In a report it says exams are failing to improve standards, leaving pupils demotivated and stressed and encouraging bored teenagers to drop out of school.”

Demotivation, stress and, crucially for my work , poisoned attitudes. Will year 11s thoughts on the validity of assessment at 16 be coloured by their experience of testing (and other assessment) pre-16. Fairly inevitable I should think…


QCA annual report on ICT for 2005/06

25 March 2007

QCA  have published their annual report on ICT (and other subjects) as part of their “monitoring the curriculum” exercise. The outputs from this report will (or at least should) influence their review of the secondary curriculum. The report formed part of the basis of this BBC article

Some key points in my reading of the report:

The aims of the national curriculum:

  • There is a clear recognition (by schools) of the potential of ICT to help develop pupils’ enjoyment of, and commitment to, learning.
  • Almost half of year 8 said that they enjoyed ICT compared with only 14 per cent who said that they disliked it.
  • More than a quarter of teacher disagree that the PoS ‘helps give pupils the opportunity to be creative, innovative and enterprising’ supporting previous findings that a significant amount of ICT learning and teaching continues to focus on elementary application of basic skills.
  • QCA believes that there are enormous possibilities in ICT for creativity, enquiry and innovation and the secondary curriculum review has enabled us to bring this to the forefront in the ICT programme of study (PoS). However, there may be additional barriers to using ICT in this way in schools and this needs further investigation.

Assessment

  • There is work still to be done to assist teachers with assessing ICT. Schools  say they need teacher assessment guidelines/materials for assessing pupils’ progress and continued professional development.
  • QCA has recommended that the on-screen key stage 3 test should be rolled out on a non-statutory basis, but will work to develop the test as a formative assessment tool to support teaching and learning.
  • In the survey of year 8 pupils, nearly a third of pupils felt that the level of ICT work they were being given was too easy.

Questions particular to ICT

  • There remains a lack of consistency and coherence in the ICT qualifications currently on offer, which is unhelpful for users and employers.
  • Although uptake of ICT qualifications continues to rise, enquiries to QCA indicate that for some schools the choice of qualification is made on the basis of points for league tables rather than on the appropriateness of the qualification to the learner.
  • QCA has commissioned an in-depth research probe into the qualifications offered in schools and the progression routes offered post-16. For example, if, at A level, more than 45 per cent of pupils are deciding not to continue with ICT at A2 because of their poor results at AS, it would be useful to find out the prior qualifications of those pupils who decide not to continue or whether there are other factors involved.

DfES “Making Good Progress” report

25 March 2007

This is the report that the previous post refers to… and here is an apposite quote…

A key part of the pilot is to try new approaches to
assessing when pupils are ready for moving ahead.
That judgement will of course come initially from the
teacher, but pilot schools will use external tests to
validate teacher assessments, and provide a clearer
benchmark by which parents and pupils can
measure progress. The pilot schools will identify
pupils in the relevant key stage whom they believe
are at or close to achieving a full level of progress,
who would be likely to pass a “test for progress” in
December 2007 (and at six-monthly intervals
thereafter). That decision would be discussed and
agreed with pupils, and involving parents to the
fullest possible extent.

If this is the policy for pre-14 what is the concomitant for 14-19? How does this square with the threat to the Diplomas (which seem to provide much of this flexibility – albeit with ’employers’ added into the last sentence).


Are school tests on their way out?

25 March 2007

BBC NEWS | Education | Are school tests on their way out?
Something extraordinary seems to be happening to school tests in England.

It could be the most radical change since the tests began in the early 1990s, when they were still called Sats.

Yet this is a “softly, softly” revolution. The tests appear to be on their way out, at least in the form we know them today. But ministers do not want to give the impression they are easing up on accountability.

Meanwhile others keep letting the cat out of the bag. This week it was the turn of the head of the examinations watchdog, Ken Boston.

He suggested that national tests for all pupils could be phased out within three years and replaced by a test taken by just a small sample of pupils, sufficient to give a national picture of education standards.

Also reported in the Guardian and elsewhere … including this with some “firefighting” comment by the DfES


Diplomas may go horribly wrong

20 March 2007

The BBC report Alan Johnson’s message to the Association of School and College Leaders with the headline  “Diplomas may go horribly wrong“. If horribly wrong means that  they won’t bring parity of opportunity and esteem then I guess he’s right. The Telegraph report has it that they are being scrapped post-16. So just as we prepare for another attempt to make the curriculum relevant then the goalposts are moved…


Some things to add to the pile of reading…

7 March 2007

Dereks Blog: Taking formal education beyond exams
An article from Malaysia about a change of approach to assessment

DfES: research into ICT
A whole pile of research into ICT in schools and homes

EU: Impact of ICT
Advertised on the elearningeuropa.info newsletter, an impact study in 17 EU nations


QCA review of the National Curriculum

6 February 2007

The Qualifications and Curriculum Authority (QCA) have published the draft of the new National Curriculum (NC) for secondary schools in England. This contains slimmed down programmes of study for all subjects (although ICT was always fairly slim). It also contains a section on the way in which ICT as a functional skill across the curriculum might be specified – and maps this to the subject ICT. This seems an odd mapping in many ways, as both sides of the reationship could be interchanged.

Also of note is the section on assessment strategies. These include some clear guidance on not just relying  the test at the end. I wonder how this will actually pan out at Key Stage 4 (14-16 year olds). Will schools stop entering all and sundry for a full blown level 2 qualification in ICT? Will the functional skills and some assesement of ICT use against KS4 criteria be enough? Or is this too obscure to be bothered with. What will the reporting requirements be for ICT at KS4? It is one of few subjects left in the NC at this level.

There is abother aspect of assesment that caught my eye on first read (and how much harder is a website than a set of printed documents to read!?!) . This was to do with taking a range of evidence

It does mean that more of what learners ordinarily do and know in the classroom is taken into account when teachers come to make a periodic assessment of learners’ progress at the end of term or half-year. For example, all teachers are continually making small-scale judgements about learners’ progress, achievements or the support they require when, over a number of lessons, they are reading or writing a lengthy text, planning and revising a design brief, or researching a historical figure in books or online. Such knowledge tends to be overlooked when only the final outcome, artefact or test is assessed, but it can make a vital contribution to periodic assessment.

What about what learners “ordinarily do” outside the classroom?


EPPI review (2005) Motivation and assessment

27 January 2007

Smith C, Dakers J, Dow W, Head G, Sutherland M, Irwin R (2005) A systematic review of what pupils, aged 11–16, believe impacts on their motivation to learn in the classroom. In: Research Evidence in Education Library. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London.

This EPPI review, cited by Gilbert, is focusing on motivation of 11-16 year olds. Its main findings identify six themes in the key to motivation. Each theme may have some relevance here. Italics represent direct quotes from the summary of the review.

  • The role of self : how is the learner’s own constructs represented in their view of learning? How does the role of the ‘group’ affect this?
  • Utility: Students are more motivated by activities they perceive to be useful or relevant.
  • Pedagogical issues: Pupils prefer activities that are fun, collaborative, informal and active.
  • Influence of peers: Linked to role of self
  • Learning. Pupils believe that effort is important and can make a difference; they are influenced by the expectations of teachers and the wider community.
  • Curriculum. A curriculum can isolate pupils from their peers and from the subject matter. Some pupils believe it is restricted in what it recognises as achievement; assessment influences how pupils see themselves as learners and social beings. The way that the curriculum is mediated can send messages that it is not accessible at all.

In this last point, the role of assessment is raised. So what does the review have to say about assessment in general?

The way that assessment of the curriculum is constructed and practised in school appears to influence how pupils see themselves as learners and social beings. (Summary, page 4)

… assessment [has a role] in nurturing or negatively influencing motivation (page 6 and page 63)

…the recent systematic review of the impact of summative assessment and tests on student’s motivation for learning acknowledges that ‘motivation is a complex concept’ that ‘embraces… self efficacy, self regulation, interest, locus of control, self esteem, goal orientation and learning disposition’ (Harlen and Deakin Crick, 2002:1) (page 8 of the EPPI review)

Students’ motivation is influenced by their ‘affective assessment’ (Rychlak, 1988) of events, premises and actions which are perceived as meaningful to their existence. (page 35, and linked to ‘logical learning theory’ (uncited))

Student satisfaction with their ‘academic performance tended to be influenced both by grouping, curricular and assessment practices and by its relationship to perceived vocational opportunities’ (Hufton et al., 2002:282). (page 45)

…learning situations that were authentic – in other words, appeared real and relevant to the pupils – could positively influence pupil motivation… ‘Sharing the assessment process with students is another way to capture students’ motivation…When students and teachers analyse pieces of writing together in an exchange of views, students can retain a sense of individual authority as authors and teachers convey standards of writing in an authentic context’ (Potter et al. 2001:53) (page 47 of EPPI)

Harlen W, Deakin Crick R (2002) A systematic review of the impact of summative assessment and tests on students motivation for learning. Version 1.1. In: Research Evidence in Education Library. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London.

Hufton NR, Elliott JG, Illushin L (2002) Educational motivation and engagement: qualitative accounts from three countries. British Educational Research Journal 28: 265–289.

Potter EF, McCormick CB, Busching BA (2001) Academic and life goals: insights from adolescent writers. High School Journal 85: 45–55.


Education 2020, the Gilbert Report (2006)

26 January 2007

The Gilbert report on Education 2020 contains a wealth of findings (or sentiments anyway) that have relevance to my research.

Personalisation, it begins, means assessment-centred, learner-centred and knowledge-centred… “Close attention is paid to learners’ knowledge, skills, understanding and attitudes. Learning is connected to what they already know (including from outside the classroom).”… “Sufficient time is always given for learners’ reflection.” (page 8 and citing Branford et al, 2000) – this ties in well with the meta-learning findings of Demos (2007).

“…schools therefore need increasingly to respond to: [..] far greater access to, and reliance on, technology as a means of conducting daily interactions and transactions ” (page 10, with references in Annex B). “The pace of technological change will continue to increase exponentially. Increases in ‘bandwidth’ will lead to arise in internet-based services, particularly access to video and television. Costs associated with hardware, software and data storage will decrease further. This is likely to result in near-universal access to personal, multi-functional devices, smarter software integrated with global standards and increasing amounts of information being available to search on line (with faster search engines). Using ICT will be natural for most pupils and for an increasing majority of teachers. ” (page 11)

“strengthening the relationship between learning and teaching through: … dialogue between teachers and pupils, encouraging pupils to explore their ideas through talk, to ask and answer questions, to listen to their teachers and peers, to build on the ideas of others and to reflect on what they have learnt” (page 15)

“Pupils are more likely to be engaged with the curriculum they are offered if they believe it is relevant and if they are given opportunities to take ownership of their learning. Learning, clearly, is not confined to the time they spend in school” (page 22, citing EPPI, 2005)

gilbertfig4.gif

Figure 4 – Ways in which technology might contribute to personalising learning (page 29)

The recommendations on page 30 stop someway short of recognising the relationship between technology inside and outside of formal classroom use however. There is a nod towards it in this extract: “We recommend that…all local authorities should develop plans for engaging all schools in their area on how personalising learning could and should influence the way they approach capital projects… Alongside the design of school buildings, schools will need to consider: – what kind of ICT investment and infrastructure will support desired new ways of working – how the school site and environment beyond the buildings can promote learning and pupils’ engagement… goverment should set standards for software, tools and services commonly used by schools to facilitate exchange and collaboration within and between schools software packages from home.”

Bransford J.D., Brown A. L. and Cocking R. (eds.), How people learn: brain, mind, experience and school, National Academy Press, Washington DC, 2000. teaching principles and the design of quality tools for educators. Technical report

Eppi Centre Review: Asystematic review of what pupils, aged 11-16, believe impacts on their motivation to learn in the classroom, 2005. Available at: http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=304


Wiliam (2000) on reliability and validity

25 January 2007

Wiliam’s paper, referenced by Mike Baker in his BBC summary, is not actually about the validity of National Curriculum (or any other) formal tests per se. It is about the inherent issues of validity and reliability of testing. The reduction of reliability comes from the inability of students to perform exactly the same way in tests. If they were to take the same test several times then they would expect to get different scores, argues Wiliam. This seems intuitively sensible, if impossible to prove as you can’t ever take a test again without it either being a different test or without you learning from your first attempt. The position is a theoretical one. Wiliam uses a simple statistical model to come up with the figures that are used in the BBC report. It is not that a test is 32% inaccurate, but that 32% is the number of misclassifications that might be expected given the nature of testing and quantitative scoring. The stats used by Baker are, themselves, theoretical, and should not be used as ‘headline figures’.

Wiliam then goes on to look at reliability of grades. He points out that we might intuitively know that it would be unreliable to say a student who scores 75% must be ‘better’ than one who scores 74%. But if the results are reported as grades we are more likely to confer reliability to the statement ‘the student achieving the higher level is better ‘.

On validity Wiliam says little in this paper but does point out the tension between validity and reliability. Sometimes making a test reliable means it becomes less valid. He cites the example of the divergent thinker who comes up with an alternative good answer that is not on the markscheme and who therefore receives no credit. this is a standard response by examining teams designed to eliminate differences between markers. While contingencies are always in place to consider exceptional answers, if they are not spotted until the end of the marking period then they cannot be accommodated. If several thousand scripts/tests have already been marked, they cannot be gone back over because one examiner feels that one alternative answer discovered late on should be rewarded. You either reward all those who came up with it or none. Usually it is none for pragmatic reasons, not for reasons of validity.

Wiliam (2000) Reliability, validity, and all that jazz in Education 3-13 vol 29(3) pp 9-13 available online at http://www.aaia.org.uk/pdf/2001DYLANPAPER3.PDF

and citing

Wiliam, D. (1992). Some technical issues in assessment: a user’s guide. British Journal for Curriculum and Assessment, 2(3), 11-20.

Wiliam, D. (1996). National curriculum assessments and programmes of study: validity and impact. British Educational Research Journal, 22(1), 129-141.


Openquals is now NDAQ

24 January 2007

QCA’s website of accredited qualifications, Openquals, is now known as the National Database of Approved Qualifications (NDAQ). It carries the logos of three of the UK’s qualifications’ authorities – QCA (England), CEA (Northern Ireland) and ACAC (Wales/Cymru). The SQA in Scotland is notable by its absence.

NDAQ is easier to ‘pronounce’, harder to find on Google and is easier on the eye – slightly. The myriad options available at school levels in ICT * are still bewildering. Maybe they will help with ‘personalisation’ but will they help to more validly represent learner’s abilities, achievements, capabilities?

* NDAQ has ICT, Openquals had IT… the nomenclature confusion continues…


BBC: Testing times for school assessment

24 January 2007

The BBC’s Education correspondent Mike Baker gives a very readable account of the changes ahead in the assessment system in his report of 6 January 2007 – Testing times for school assessment.

His main thrust is that changes to the system are coming in. Some of these are reflected in subsequent events that I have written about like the revamping of league tables and possible scrapping of the online ICT test… although the latter of these presumably would have helped personalisation if it was an on-demand test.

The changes, concludes Baker, are due to the growing clamour for that most voguish of educational shibboleths – personalisation.

In the article, he reflects on the Gilbert report from the HM Chief Inspector into personalisation and on how the recommendations of the report might necessarily lead to a greater role for teacher assessment. He ties this in with an IPPR study into the tensions between the dimensions of validity and accountability of assessment. Again teacher assessment is recommended by the authors as a way of enhancing both dimensions. Finally he cites Dylan Wiliam’s research into the ‘shockingly’ (Baker’s word) inaccurate methods of formal assessment.

A very useful summary.

Miles Berry also summarises the Gilbert Report in his blog, again very useful.


Assessment: ‘for learning’, formative and summative

19 January 2007

One of the features of WordPress (and many other blogs) is the reporting of search terms that have been used, which then result in the blog being found.

Yesterday, the search terms reported included assessment for learning and inclusion.

This  got me thinking that I hadn’t really made any use of the simple taxonomy of assessment. Assessment for learning is formative as it informs further learning (Black and Wiliam, 1998). My focus is really on summative learning.

A study of pupil perceptions of assessment for learning  (years 7 to 10) was carried out by Cowie (1995). I guess part of my research will be looking at pupil (or student) perceptions of summative learning. It will be interesting to compare results to those found by Cowie.

The Cowie paper was cited on the DfES Standards Site, in a section called the Research Informed Practice Site. I’m not sure about the initials this provides, but the site may well be a useful one both for this research and for my teaching. I hadn’t come across it before. It is useful, not just for its own sake, but because it provides digests of articles…

Black, P and Wiliam, D (1998) Assessment and Classroom Learning in Assessment in Education, Vol. 5, No. 1, pp 7-74

Cowie, B (2005) Pupil commentary on assessment for learning in The Curriculum Journal, Vol. 16, No. 2, June 2005, pp. 137 – 151

DfES (2007), TRIPS – the Research Informed Practice Site, London: DfES [online] available at http://www.standards.dfes.gov.uk/research/ accessed 19/01/07


Revamped school league tables published

13 January 2007

The DfES have published the 2005/06 school league tables.

Making is debut is the new measure of ‘threshold performance’ – out with 5 A*-C GCSEs or equivalent, in with a level 2 threshold that must include English and mathematics. The threshold is reached when a student has sufficient level 2 passes. This sufficiency is still achieved by 5 GCSEs, and level 2 when the GCSEs are A*-C. In this respect the only change is the mandatory need to pass Enlish and mathematics. It had been feared that making these two subjects a necessary factor in including student ‘success’ in a school’s figures would mean a drastic reduction in average levels of success. The percentage of students reaching threshold fell to 46% – down from 59% under the old measure of 5 A*-C GCSEs. Whilst a significant drop it may not be as bad as some feared.

For my own part, and for ICT, I was interested to see how the change had affected the pioneer of GNVQ IT for all (or at least the majority). Thomas Telford School was lauded a few years ago because of its dramatic increase in 5 A*-C percentage. I had always suspected that this was because of the GNVQ counting for four of them. It is interesting to note therefore that the school registered 95% in the new measure where English and mattics have become compulsory. So maybe 80% (or 4 GCSE-equivalent) qualifications didn’t distort the tables as much as seemed likely. This despite the headlines and horror stories as in this from the BBC:

A few canny pioneers realised that there was a vocational qualification, the GNVQ, which was worth four GCSEs at grade C or above. Add one more GCSE, in any subject, and, hey presto, you meet the target. To be fair, many of the original followers of the GNVQ route felt it offered the best educational option for their pupils. Others, though, realised it was an effective way of avoiding the consequences of falling below government targets. Eventually the government realised that large numbers of schools were achieving the threshold without their pupils achieving GCSE passes in maths and English.

Hence this year’s new requirement that the five GCSE passes must include maths and English. The effects on some schools have been dramatic. One school went from a score of 82% passing the equivalent of five A*-Cs to just 16% when maths and English were included. Many other schools, which had been climbing up the tables in recent years, found themselves slithering back down again. Presumably they will now find new ways of targeting performance in maths and English, no doubt at the cost of something else.

The more significant change though may be use of the phrase ‘ level 2 threshold’. GCSEs are no longer explicitly mentioned in the language of the threshold (even thought the DfES’s own link still says GCSE tables). A GCSE at A*-C is now just a 20% contribution to threshold. Many other qualifications can also contribute, as they could before. But now DIDA, for example, is not described as equivalent to 4 GCSEs but a level 2 DIDA pass is described instead as 80% of threshold.

The list of accredited qualifications, their level, and contribution to threshold is maintained by QCA at the Openquals website (soon to be renamed NDAQ: National Database of Accredited Qualifications). For level 2 ICT/IT, Openquals lists 71 qualifications across the awarding bodies. The market is wide open. The gold standard at 16 has changed.

Also this year saw a change in the value added measure include other factors other than just improvement on performance compared to level 3. Now the profile of the school’s students is taken into account and a figure based on a notional national baseline of 1000 is reported in the Contextual Value Added measure (CVA). Schools such as Greenwood Dale in Nottingham are lauded, quite rightly, for reasons of value-added and not just level 2 performance. This contextual value added doesn’t tell the whole story though, and Leicestershire’s relatively poor performance might have something to do structure in the county. The 14-19 colleges certainly seem poor relations to Nottinghamshire’s comprehensives in respect of CVA even though,  in my opinion, 14-19 is a much more coherent age range. Maybe it is not measuring like with like when there is a change at 14? A classic case of measuring soemthing other than what is puported to be measured?


Moss (1992): validity and assessment of performance

11 January 2007

Pamela Moss’s 1992 paper “Shifting conceptions of validity in educational measurement: implications for performance assessment” (1) is cited by Wiliam in his modification of Messick’s four-facet model. It would seem from the figure I extracted from Wiliam’s paper that he is suggesting that Moss is providing an extra dimension to the evidential paradigm. That was what I saw on first reading. On turning to Moss’s paper and re-reading Wiliam I am now not so sure of where he is placing Moss vis a vis Messick.

Moss’s paper is an overview of the landscape of construct validity from the inception by Cronbach and Meehl in 1955 (2) to its publication in 1992. In doing so she looks at evidential and interpretive aspects of the models of Cronbach (1980) (3) and Messick. The latter is not seen as being purely evidential as Wiliam’s paper might suggest.

The thrust of Moss is that a review was needed of the “Standards” of what might be called the Establishment of (American) assesment and measurement (AERA, APA, NCME). This review, she argues, is because of the emergence of performance assessment as a science (and as commonly used tool) to complement test/item-based assessment. She compares this to the contemporaneous diminuition of the dominance of positivism.

In performance assessment there is a strong interpretive base. The learner will interpret the task and manifest skills, knowledge and understanding through their performance. the assessor will re-interpret this performance to provide evidence to which rules of validity must be applied.

(1) Moss, P (1992) Shifting Conceptions of Validity in Educational Measurement: Implications for Performance Assessment in Review of Educational Research, Vol. 62, No. 3. (Autumn, 1992), pp. 229-258.

(2) Cronbach, L.J. and Meehl, P.E. (1955) Construct validity in psychological tests in Psychological Bulletin, 52, 281-302 also available online at http://psychclassics.yorku.ca/Cronbach/construct.htm

(3) Cronbach, L.J. (1980). Validity on parole: How can we go straight? in New directions for Testing and Measurement, 5, 99-108.


John Naughton: Welcome to IT class, children; log on and be bored stiff

10 January 2007

Writing in The Observer, John Naughton caricatures another vignette….

There’s a surreal quality to it, conjuring up images of kids trudging into ICT classes and being taught how to use a mouse and click on hyperlinks; receiving instructions in the creation of documents using Microsoft Word and of spreadsheets using Excel; being taught how to create a toy database using Access and a cod PowerPoint presentation; and generally being bored out of their minds.

Then the kids go home and log on to Bebo or MySpace to update their profiles, run half a dozen simultaneous instant messaging conversations, use Skype to make free phone calls, rip music from CDs they’ve borrowed from friends, twiddle their thumbs to send incomprehensible text messages, view silly videos on YouTube and use BitTorrent to download episodes of Lost. When you ask them what they did at school, they grimace and say: ‘We made a PowerPoint presentation, dad. Yuck!’


Embretson (1983): construct representation

10 January 2007

The “top left” quadrant of Wiliam’s enhancement of Messick’s four-facet model of validity deals with within-domain evidential/interpretive validity. How is the assessment designed so as to provide constructs that evidence that which is to be assessed within the domain. He cites Embretson (1983) (1) as providing part of the conceptual model for this quadrant.

Embretson’s model distinguishes between construct representation and nomothetic span. In the former, assessment designed so that it is situated in tasks that represent that which is to be assessed. In the latter it is designed to correlate with other tasks deemed valid.

Mislevy et al (2002) (2) discuss the model in the context of the “psychometric principles” of validity, reliability and comparability. They relate the task model to three other models – the student’s learning, the assessment (or measurement) of this learning and the scoring. Their argument appears to be that the construct representation resonates more with the psychometric principles than does nomothetic span, but that both may be needed.

In the context of my research it would seem that I am doing some sort of comparison between the two parts of Embretson’s dichotomy. Construct representation – using what the students have learnt by way of ICT capability to provide an assessment. Nomothetic span – using some assessment that correlates to this as measured by other assessments.

Is use of the former inherently more engaging than the latter? Does it fit with student’s own constructs of what they have learnt?

(1) Embretson, S. E. (1983), Construct validity: Construct representation versus nomothetic span in Psychological Bulletin, 93, 179-197.

(2) Mislevy R, Wilson M, Ercikan K, Chudowsky N (2002), Psychometric Principles in Student Assessment CSE Technical Report 583, Centre for Studies in Evaluation, LA also available online at http://www.cse.ucla.edu/reports/TR583.pdf


Wiliam’s model of construct validity (1996)

9 January 2007

Wiliam (1996) offers a model that starts from Messick’s four-facet model (1) of validity (subsequently, (1996), enhanced to six facets) and applies it the National Curriculum. Wiliam’s analysis has much to offer when looking at assessment at 16. He takes Messick’s distinction of the evidential and consequential in assessment and adds Moss’s (1992) interpretative basis to the former. Assessment validity needs to be looked at through the evidence, the interpretation and the impact (consequence). For each of these two bases – evidential/interpretive and consequential – Wiliam then builds on Messick’s other dimension of within- and beyond-domain.

Wiliam (1994)
Wiliam then examines each of the four zones in turn.

In regard of within-domain inferences Wiliam explains the work of Popham and others in trying to establish valid tests that test all, and only, the domain that is intended to be tested. The concluding criticism of the validity NC tests may well apply to any external traditional examination – they are unrepresentative of the domain because of their length compared to the length/volume of learning.

For beyond-domain inferences Wiliam cites the predictive nature of the use of test results. High performance in X predicts high performance in Y. He cites Guilford in saying that it doesn’t matter how this correlation is arrived at, merely that it is reliable. The test might not be valid though as it may not be in the same domain. For ICT at 16 there may be aspects of the achievement that is given far greater importance than maybe it should. A learner gets Key Skills level 2 in ICT (2) therefore s/he is functionally literate in ICT. It doesn’t matter how the level 2 was achieved.

Within-domain impact is of particular importance to the design of ICT assessments, I believe. Hence the move towards onscreen testing – it’s ICT so the the technology must be used to assess the capability. In Wiliam’s words, it “must look right” (p132).

Finally, Wiliam considers beyond-domain impact or consequence. In looking at National Curriculum testing, Wiliam argues, some of the validity is driven (or driven away) by beyond-domain impacts such as league tables – these are much higher stakes for schools than learners and so the validity of the assessment is corrupted.

(1) Messick, “Validity,” 20; Lorrie A. Shepard, “Evaluating Test Validity,” in Review of Educational Research, ed. Linda Darling-Hammond (Washington, DC: AERA, 1993), 405-50. cited in Orton (1994)

(2) The functional/key skill component of ICT learning is referred to as IT

Updates:

10/01/07 Post on Embretson (1983)

11/01/07 Post on Moss (1992) 


The Futurelab model and ICT at 16

9 January 2007

So, taking the model from the Futurelab literature review, how might the dimensions of construct validity manifest themselves in assessment of ICT at 16 – the domain of my study.

Content validity: are items fully representative of the topic being measured?

Here might be included a study of what is included in assessments and an analysis of those against the stated assessment objectives, the content of specifications and, coming back to my specific focus, the topic (ICT learning) as constructed by the learners. What do 16-year olds identify as ICT?

Convergent validity: given the domain definition, are constructs which should be related to each other actually observed to be related to each other?

Here there is something about the relationship between the things above I think. Is there convergence between the assessment objectives, between learners’ constructs and between the two sets? I think there is more to explore here but haven’t quite got my head around it yet…

Discriminant validity: given the domain definition, are constructs which should not be related to each other actually observed to be unrelated?

This is more tricky. Why would there be “constructs which should not be related to each other”? Is this to do with identifying things that are mutually exclusive? Is formal and informal learning ever like this?

Concurrent validity: does the test correlate highly with other tests which supposedly measure the same things?

This too is tricky, but there is something here for me about the relationship between teacher assessment and test results I think


An overview of the Higher Education ICT Literacy Assessment

7 January 2007

David M. Williamson, Irvin R. Katz, and Irwin Kirsch (2005) [online PDF] available at http://www7.nationalacademies.org/bose/
ICT%20Fluency_Assessment_Overview_Article.pdf

This paper, originally presented to the 2005 AERA conference, contains a wealth of argument about the validity of ICT assessment. Here the topic is presented as ICT literacy. This word ‘literacy’ is laden with other connotations for me – to do with natural use of (in this case ICT). If someone is ICT literate that means so much more than saying they are ICT competent. One is about understandings, internalisations the other about surface skills I believe. The paper’s context is in HE, specifically an assessment that measures a HE student’s abilities to use technology to research, organize and communicate information. What is says though goes much beyond this context and speaks to my interest in assessment at 16.

The authors start from the findings of a 2001 panel looking at ICT assessment. The panel identified a number key issues of concern to policy makers and practitioners in the education community:

  • ICT is changing the very nature and relevance of knowledge and information.
  • ICT literacy, in its highest form, has the potential to change the way we live, learn and work.
  • ICT literacy cannot be defined primarily as the mastery of technical skills.
  • There is a lack of information about the current levels of ICT literacy both within and among countries.

In the amplification of the second bullet point they state “The transformative nature of information and communication technologies might similarly influence and change not only the kinds of activities we perform at school, at home and in our communities but also how we engage in those activities.” (ibid, p5)

They then go on to distinguish between issues of access and of proficiency – stating that research into the Digital Divide is insufficient in addressing issues of measuring ICT literacy. Providing the access is not enough – many schools found this with the introduction of Regional Broadband Consortia (or maybe the RBCs found this – I suspect schools knew already!).

The paper then discusses evidence-centred design of assessments. Again, the context for this paper is different to mine as they are trying to design an Internet-delievered test, an approach which may be running into difficulty in English schools. Nevertheless they provide a concise overview of this field and validity theory (Messick, 1989), psychometrics (Mislevy, 1994), philosophy (Toulmin, 1958), and jurisprudence (Wigmore, 1937). The process of assessment design they identify consists of four key questions:

  • Purpose: Who is being measured and why are we measuring them? What types of decisions will we be making about people on the basis of this assessment?
  • Proficiencies: What proficiencies of people do we want to measure to make appropriate claims from the assessment?
  • Evidence: How will we recognize and interpret observable evidence of these proficiencies so that we can make these claims?
  • Tasks: Given limitations on test design, how can we design situations that will elicit the observable evidence needed?

These issues again seem central to my thinking at this stage.

Later they break down ICT literacy into seven key evidences – things that are to be measured (or assessed).

  • Define: The ability to use ICT tools to identify and appropriately represent information need.
  • Access: The ability to collect and/or retrieve information in digital environments.
  • Manage: The ability to apply an existing organizational or classification scheme for digital information.
  • Integrate: The ability to interpret and represent digital information.
  • Evaluate: The ability to determine the degree to which digital information satisfies the needs of the task in ICT environments.
  • Create: The ability to generate information by adapting, applying, designing, or inventing information in ICT environments.
  • Communicate: The ability to communicate information properly in context in ICT environments.

This model seems rather to close to a skills taxonomy for my liking but it may be useful as one model among many for trying to look at how learner’s construct their knowledge.


Futurelab lit review on e-assessment (2004)

7 January 2007

Futurelab, a UK technology and learning ‘thinktank’, commissioned this 2004 literature review (1) on e-assessment. In compiling the report, the authors (Jim Ridgway and Sean McCusker, School of Education, University of Durham and Daniel Pead, School of Education, University of Nottingham) have, not surprisingly, covered a lot of ground to do with assessment per se and not just its technologically-enabled version.

In talking about the use of e-portfolios, the report concludes that “Reliable teacher assessment is enabled. There is likely to be extensive use of teacher assessment of those aspects of performance best judged by humans (including extended pieces of work assembled into portfolios)” (ibid, page 2)… for me, hidden in this is the validity argument. It comes through reliability of teacher assessment, and extended pieces of work. Both of these should help validity I believe.

Section 1 of the report then talks of the nature of assessment – formative and summative. Throughout this the authors continually refer back to the purpose and validity of assessment. Also, the learner is placed at the centre of the the described processes. Of particular note for me is the mendacity quotient, whereby summative assessment often encourages students to actively hide what they don’t know.

Section 2 discusses how and where assessment should be driven, with the focus also on technology as the report is on e-assessment. There are some more generally-applicable points covered here though. “Metcalfe’s Law” of increasing value through networks is used to underpin the need to tie assessment into rapidly increasing technologically-enabled social networks. More simply, perhaps, the use of peer networks for assessment might also be part of this… My work aims to look at the validity of external assessment by using self and peer viewpoints as comparators. In addition to social changes, the report identifies other drivers on assessment change as globalisation, mass education, defending democracy and government-led policies. Here there is a disappointing (for me) relative sparsity of focus on the needs of the learner, although demands of lifelong learning are brought out in the section summary (ibid, page 9).

Section 3 discusses the developments in e-assessment. Or so its section heading states. Actually there is much in here about assessment in general and the need to make it relevant to learner needs and valid. “… some [developments] reflect a desire to improve the technical quality of assessment (such as increased scoring reliability), and to make the assessment process more convenient and more useful to users (by the introduction of on-demand testing, and fast reporting of results, for example).” (ibid, p15).

Under the heading “Opportunities and challenges for e-assessment”, section 4 is a rich vein of resources and opinion on the use of assessment to assess deeper level skills, understandings etc. While the summary of the section appears very parsimonious about what has been written, sub-section 4.1 is full of how assessment should be enabling learning.

Finally, the appendix on page 24 is a good overview of, to use its title, “The fundamentals of assessment” .

(1) Ridgway J, McCusker S and Pead D (2004) Literature Review of E-assessment: A Report for Futurelab [online PDF] available at http://www.futurelab.org.uk/download/pdfs/research/lit_reviews/
futurelab_review_10.pdf


Another policy change?

5 January 2007

One chapter of my thesis will almost certainly deal with the policy on assessment of ICT in England… (it would make a boring Mastermind subject though). Today the BBC report yet another possible change. Although it is to do with the possible scrapping of proposed introduction of compulsory on-screen testing at KS3, some of the sentiments expressed might be equally applicable to other assessment regimes. One ICT subject leader Roger Distill is quoted by the BBC:

“One does not need to be a computer geek to realise that the technologies in the real world will have moved on amazingly in that time, while education, as usual, gets left behind as we continue to train our students in the old and limited techniques required to succeed in the test.”

The question here is what is “the real world” of the 14- or 16-year old?


Technology, learning and inclusion

4 January 2007

I have just returned from a very useful conference-cum-seminar-cum- network meeting on inclusion and special educational needs. This was held at my own Nottingham Trent University (NTU). The themes were research and inter-agency working in the areas of special educational needs and inclusion. Hopefully the embryonic network that came together to plan the day will be a basis for future sharing and collaboration.

There was a useful (for me) distinction made by paediatrician Linda Marden between inter-agency (agencies working together) and multi-agency (many agencies not necessarily working together). The background for this, and my reason for being there, is of course the all-pervasive Every Child Matters agenda where multi-agency is the preferred term… but I think Linda makes a very valid an pertinent point.

It was good to hear of research in this field, to hear to phrase co-researcher again when applied to research group including the young people at The Shepherd School Nottingham who have been round the world helping disseminate findings of research projects. These projects include those of the Virtual Reality Applications Research Team at the University of Nottingham and the Interactive Systems Research Group at NTU.

The work of these projects in engaging young people in learning with and through technology reminded me of the excellent work done by my former colleagues at the now-vastly-downsized Ultralab. Established and led by Stephen Heppell and the Richard Millwood some of the lab’s legacy into digital creativity is being carried on by Matt Eaves, Hal Maclean and others under the Cleveratom banner.

Continuing the ex-Ultralab theme and maybe more directly aligned to the theme of inclusion (at least if one defines inclusion as trying to include those who are excluded (!)) is the ongoing work of NotSchool and of Jonathan Furness at the Stepping Stones School for children with hemiplegia.

What’s this to do with my PhD? Well for me it resonates well with the themes of authentic learning and ICT. If ICT becomes so enmeshed in the learning, as all of these projects demonstrate, how is it possible to assess it?


Authentic learning and assessment

3 January 2007

The term authentic learning has been around some time, apparently, although I had not come across it before today. This paper cites Archbald and Newman (1998) as the first to apply authentic to learning and assessment, although I’m not sure that this is truly the first use of the term.

It does seem to encapsulate what I am trying to investigate though as this list of points from Maureen O’Rourke (2001) suggests:

Use of ICT to provide students with greater opportunities for communication, collaboration, thinking and creativity also provides us with challenges in terms of authentic assessment. The Australian National Schools Network has recently launched a national Authentic Learning and Digital Portfolios project. Beginning with a focus on the whole person, school communities are clarifying what young people should know, understand and be able to do at particular stages of their education.

[..] The project aims to bring learning and assessment together with:

• students having significant control in the construction of their portfolios
• the portfolio structure providing opportunities for feedback, questioning and reflection
• assessment moving to a more central part of the learning process, conducted with students rather than on them
• rich, authentic tasks providing evidence of learning in multiple domains

Guess I need to follow up the Authentic Learning and Digital Portfolios project, although my experience of DIDA suggests that simply using e-portfolios is no guarantee of authentic assessment or learning.

Some texts on the subject: Tombari and Borich, Guba & Lincoln


Gap in Knowledge: Assessment of ICT v Assessment with ICT

2 January 2007

I have it ingrained in my psyche that one of the key things about doctoral work is the need to prove that one is inquiring into a ‘gap in the knowledge’. This has always been problematic for me. What is knowledge? How do I prove that the gap exists. Simply because I don’t know of something, doesn’t mean it doesn’t exist (three negatives there…). I might think there is a gap, only to be blissfully unaware that someone else has filled it (or, worse, is filling it as I speak/procrastinate).

Notwithstanding this, the gap in the knowledge that I identify is located somewhere in the aridity that is the apparent dearth of writing on the assessment of ICT. Put assessment and ICT into Google and you get 1.42m results. Most of these appear to be about using ICT in the assessment process. Assessment with ICT.

McCormick (2004), writing for the ERNIST project and elsewhere, cites Macfarlane (2001) and Thelwall (2000) in defining a taxonomy for the relationships between ICT and assessment. While his first category is ‘Assessing ICT skills and understanding’, it would seem that this is ignored in the rest of his paper. There is, instead, focus on use of ICT for assessment and affordances provided by use of ICT in other subjects for the assessment of those subjects. Indeed, Thelwall’s work is solely of computer-aided assessment.

Similarly the EPPI studies on ICT and assessment deal with how it is used in assessment or how it can help assess creative and thinking skills in different ways to other media.

So is there a gap in knowledge? Like Popper, I cannot prove that there is but if there is it is somewhere in all of this mist. How do you know when you’ve found a gap anyway? What does the edge of a gap look like?


“So what’s your PhD about?”

30 December 2006

One of my supervisors, Karen, made the very helpful suggestion that I need to be able to sum up my PhD research to someone I might meet in a ‘lay’ setting – down the pub or on the bus if you like. This situation raised itself over Christmas during family gatherings. “So what’s you PhD about, Pete?”, I was asked. This led to some conversation about ICT and assessment and qualifications. Some from the point of view of professionals, some from parents (of school students), some from ‘lay’ observers of education.

It led onto a discussion at about 3 am as stuff poured from my head. I grabbed a piece of paper and jotted it down. A sort of concept map if you like.

What should students learn? What should be on the curriculum? What is 'should' anyway?

What engages students, learners? Can it be generalised?

What about validity of assessment per se? If teachers teach to tests and learners learn to pass them, does it invalidate the assessment? Is the learning of skills and facts held in to high regard (learning those facts and skills needed to pass)?

Are the previous questions concerns for all subjects not just ICT? What is the specialness about ICT?


How does ICT relate to other subjects? Is the D and T needs/solution/evaluation design cycle a key feature of all learning? Is problem solving at the heart of sustained learning?


What is the relationship of 'assessment objectives' to assessments? Do awarding bodies really reflect the former in the latter?


Whither creativity?

Just a bunch of ideas from the middle of the night but worth noting down maybe…


Assessment methodologies at 16

22 December 2006

There has been much in the news about changing methodologies for assessment at 16. Most of this has been about the unsuitability of coursework as a valid tool. The solution, to reduce the amount of assessment outside of ‘exams’, seems to be a bit of a knee jerk. But then education is always at the forefront of politicians’ ‘need to do somthing’.

One thing that is also changing though is the introduction of e-portfolios at 16. The DIDA qualification requires this and maybe this helps to deepen the understanding of the application of ICT. Or maybe the e-portfolio becomes a barrier? Certainly it would help assess students’ ability to use an e-portfolio – but we need to be careful that this is the central thing being assessed. How might it be applied to the creative use of ICT demonstrated by students at BAFTA for example?