Sir Mike Tomlinson lecture

22 October 2008

Sir Mike Tomlinson, chair of the working group on 14-19 reform that led to the 2003 Tomlinson Report, came to NTU today to give the RSA Shipley Lecture. This year the lecture was also a memorial to former NTU and RSA stalwart Anne Bloomfield. The subject, dear to her heart and to Sir Mike’s, was “Vocational education should be a rounded education“.

With the backdrop of the history of attempted introductions of vocational education (the 1944 Butler Education Act with its tripartite system, TVEI, GNVQs, Curriculum 2000 and Diplomas), Tomlinson argued for the move away from debates about ‘parity of esteem’ towards a view of the ‘credibility and value’ of qualifications. Echoes here of the value and validity arguments of Monday’s seminar at Cambridge.

It was also notable that the lecture included issues of how ‘true’ vocational education must have

  • relevance to 16-year olds (face validity),
  • a knowledge base that is used in, and applied to, occupational areas – however broadly drawn (validty determined by use, not by the test itself)
  • a theoretical content balanced with sector-based skills (content validity)

Again this echoes with Monday. Another thread running through was the role of assessment (systems) in undermining the vocational educational initiatives – TVEI assessment becoming ‘traditional’, GNVQ assessment being changed to equate to GCSE/A level, key skills being decoupled from GNVQs, Curriculum 2000’s insistence on equal numbers of units per qualification with a convergence of assessment types.

Also mentioned, although not in the same sense of ‘undermining’ was the persistence of the BTEC model and the way that NVQs were never envisaged to be other than accreditation mechanisms for on the job training.

The BTEC model of occupational credibility and general education was the model that was paramount in the description of vocational education with the caveat ‘what is general education’?

Throughout I was wondering where ICT fits into all this. Never mentioned as a ‘subject’ nor even as a ‘skill’ it was conspicuous by its absence. It is, of course, present in the specialised diplomas and as a functional skill although the former may be bedevilled by the wide diversity of the sector it is serving, I fear.

Tomlinson was upbeat about the Diplomas but focused especially on the need to get a true progression from level 1 through to 3. The custom of level 1 being what you get if you fail level 2 (GCSE grades D-G rather than A*-C) must not be repeated he urged. Also the need to get level 2 threshold systems so that learners who do not reach that threshold at , I would say the magical (and arbitrary, age of 16 could do so by subsequent credit points accumulation – rather than ‘repeating GCSEs’, a model that doesn’t serve well, he argued.

Another hour of useful insights.


Cambridge Assessment seminar

21 October 2008

I attended  a seminar, on the subject of validity, one of a series of events run by Cambridge Assessment (CA). It was led by Andrew Watts from CA.

This was extremely informative and useful, challenging my notions of assessment. As the basis for his theoretical standpoint Andrew used these  texts 

  • Brennan, R (2004), Educational Measurement (4th edition). Westport, CT: Greenwood
  • Downing, S (2006) Twelve Steps for Effective Test Development in Downing, S and Haldyna, T (2006) Handbook of TEst Development. NY: Routledge
  • Gronlund, N (2005), Assessment of Student Achievement (8th edition). NY: Allyn and Bacon [NB 9th edition (2008) now available by Gronlund and Waugh]

He also referred to articles published in CA’s Research Matters and used some of the IELTS materials as examplars. 

The main premise, after Gronlund, is that there is no such thing as a valid test/assessment per se. The validity is driven by the purposes of the test. Thus a test that may well be valid in one context may not be in another. The validity, he argued, is driven by the uses to which the assessment is put. In this respect, he gave an analagy with money. Money only has value when it is put to some use. The ntoes themselves are fairly worthless (except in the esoteric world of the numismatist). Assessments, analogously, have no validity until they are put to use.

Thus a test of English for entrance to a UK university (IELTS) is valid if, the UK university system validates it. Here then is the concept of consequential validity.  It is also only valid if it fits the context of those taking it. Here is the concept of face validity – the assessment must be ‘appealing’ to those taking it.

Despite these different facets of validity (and others were covered – predictive validity, concurrent validity, construct validity, content validity), Gronlund argues that validity is a unitary concept. This echoes Cronbach and Messick as discussed earlier. There is no validity without all of these facets I suppose would be one way of looking at this.

Gronlund also argues that validity cannot itself be determined – it can only be inferred. In particular, inferred from statements that are made about, and uses that are made of, the assessment.

The full list of chacteristics that were cited from Gronlund are that validity

  • is inferred from available evidence and not measured itself
  • depends on many different types of evidence
  • is expressed by degree (high, moderate, low)
  • is specific to a particular use
  • refers to the inferences drawn, not the instrument
  • is a unitary concept
  • is concerned with the consequences of using an assessment

Some issues arising for me here are that the purposes of ICT assessment at 16 are sometimes, perhaps, far from clear. Is it to certificate someone’s capability in ICT so that they may do a particular type of job, or have a level of skills for employment generally, or have an underpinning for further study or have general life skills, or something else, or all of these? Is ‘success’ in assessment of ICT at 16 a necessary pre requisite for A level study? For entrance to college? For employment? 

In particular I think the issue that hit me hardest was – is there face validity: do the students perceive it as a valid assessment (whatever ‘it’ is).

One final point – reliability was considered to be an aspect of validity (scoring validity in the ESOL framework of CA).

KS3 SATS scrapped in England

16 October 2008

This somewhat unexpected announcement was made this week. Tests for 14 year olds in maths, English and science have been scrapped. Given that many schools start their GCSE/level 2 courses at 13 now, especially in ICT, this might change radically the ways in which the middle years of secondary are organised. It may also affect students’ perceptions of assessment as they will not have had those high stakes external tests at 14.

History of assessment

14 October 2008

Blindingly obvious I guess, but nevertheless not a field I have mined much. “History of assessment” needto be a significant contextual filter for my research.

I am attending some of the seminars of the Cambridge Assessment Network as and when I can (Kathryn Ecclestone on Ofqual, Harry Torrance on Policy and Practice). There are others that I have been unable to attend but would like to have done. Fortunately useful overviews such as this one from Helen Patrick are often put online.

Becta report into Web 2.0

6 October 2008

Becta have published a research report (Crook & Harrison, 2008) on use of Web 2.0 by learners, teachers, at home, in school etc. This statement in the summary caught my eye:

Pupils feel a sense of ownership and engagement when they publish their work online and this can encourage attention to detail and an overall improved quality of work. Some teachers reported using publication of work to encourage peer assessment.

Where is the use of these tools in extrenal assessment? Come to that, where is the use of peer assessment in external assessment.

Also noticeable is the emergence of yet another Rogers’ adoption curve – with the earley adopters being the young ones etc… is this true? DO teachers really not use Web 2.0 tools? How does that square with the quote above? It is borne out in the research of Solheim (2007,  which cites OfCOM’s 2006 statistic of 40% of ‘adults’ having used social networking sites*, compared to 70% of 16-24 year olds and Comscore’s 2007 finding that over half (1.3 million) of Facebook’s new users in the previous year were 25 or older.

* OfCOM’2 2008 report tates that ‘only’ 21% have ‘set up a profile’ on such sites.

Comscore (2007), Facebook sees flood of new traffic [online] available at accessed 06/10/08

Crook, C and Harrison, C (2008)  Web 2.0 Technologies for Learning at Key Stages 3 and 4: Summary Report, Becta [online] available at accessed 06/10/08

Ofcom (2006), The communications market. [online] available at accessed 06/10/08

Ofcom (2008),The communications market report, [online] available at accessed 06/10/08

Solheim, H (2007) Digital Natives versus Higher Education: who is ready for whom, MSc dissertation, University of Southampton.

Taxonomy of difficulties in the assessment of ICT

2 October 2008

This paper, from the Assessment in Education Unit at Leeds, is again bang on the line of what I am doing. Albeit that they are looking at the KS3 onscreen ICT tests (the AEU were commissioned as part of the evaluation of that project). Nevertheless there are some very pertinent analyses of the didfficulties students, and the system, encounter in assessment of ICT. For example

… sources of difficulty that relate to the subject being assessed. The assessment of ICT brings particular risks. As McFarlane (2001) points out, assessment of ICT capability need not in itself be computer-based, but as in this case it was, the sources of difficulty in our set that are associated with this aspect all relate also to on-screen assessment, e.g.

Pupils know enough to succeed in the tasks without using ICT for all the steps.

The demands in the interaction between tasks and software on short-term memory and organisational skills are greater than the level of ICT capability that is being assessed.

Activity theory and ICT

2 October 2008

Now that the new academic year is under way, and I am in a new job (again) I hope to be able to crack on with this project. If not.. well I need to decide one way or the other.

Anyway, I am in the middle of three days study leave and ma busily writing up what I have so far. In doing so I have also come across some useful things which I will include here.

Firstly a paper from the School of Education conference at Leeds University by Aisha Walker. This makes some interesting links between activity theory and attainment in ICT and provides a model that may be useful when I come to look at the data colleection and analysis. 

The title “What does it mean to be good at ICT” really caught my eye. That’s what it’s all about isn’t it?