A three axis model

1 May 2008

Taking the ideas from the previous post and putting them into a diagram I get this

Some assessment uses ICT (or technology) - this is e-assessment (x axis).

Some assessment is designed to assess ICT capability (y axis).

Elliott’s Assessment 2.0 seems to be using ICT, not as e-assessment, but as a medium for allowing judgement to be made about the ICT capability (z axis).

Now of course, analysing any one particular assessment methodology one could locate it in this three-dimensional space. for example:

A traditional written paper would be on the y-axis. The NAA online assessment activities designed for KS3 would be in the space between all three axes (with perhaps a lower y- and z-values than x-value. Coursework would have an x-value of 0 but would have some components of y and z. Online assessments such as the driving test would be on the x-axis.

My questions here are “Where is the highest validity”? and “Where is the highest reliability?”. How does one use Elliott’s Assessment 2.0 to determine success in a certificated qualification?


Elliott (2008) Assessment 2.0

1 May 2008

Much as I dislike the nomenclature (Assessment 2.0), I found this paper by Bobby Elliott (and thanks to my colleague Bruce Nightingale and the ALT newsletter for bring the name to my attention) illuminating on many levels. Firstly here was someone making the links between theways in which technology is reportedly used by young people and the ways it could be used for technology. Secondly the author works for a government agency - the Scottish Qualifications Authority (SQA). Is this evidence of policymakers thoughts are changing to embrace the vicarious ways in which evidence of learning can be presented by technological opportunity?

My thoughts return though to the Macfarlane distinction between assessment of technology (eg the ICT curriculum) and assessment through technology (ie the methodology). This paper by Elliott seems to be moving a little away from the latter and perhaps towards the former. But perhaps, also, it is defining a third axis - assessment of technological capability through evidence presented through that technology. Maybe it is asking the question ‘What should be assessing?’ (ie the curriculum) rather than ‘How should we assess it’ (the methodology). But more than that it is saying can we assess the ‘what’ through the ‘how’.

The impressive list of tools that may be used for evidence presenting (and assessment) in Elliott’s paper also underlines my sceptcism of a one size fits all technological solution to assessment. And when I look down that list I am reminded of surveys presented by Terry Freedman (at a TDA conference in Nov 07) and others that show that young people’s use of tools is very diverse and very thinly spread. It is also very transient - MySpace here today gone tomorrow.

The very tool that Elliott uses to present his iPaper may well be a case in point. What if an awarding body decided Scribd was the thing to use. How long before it becomes the sliced bread superceded by the next best thing? How do we build in agility for assessment so that it does not become an exercise in rewarding the fashionable? (as opposed to the current system which rewards the old-fashionable).

PS Yes it’s been a long time… Higher Education management - ie my temporary role for 07/08 - and PhDs don’t easily mix… but I know that is also my excuse… and I’m sticking to it…


Two reports on usage of media/online tools

13 September 2007

This week’s eLearning Resources and News from George Siemens draws attention to two recent reports.

Firstly from Deloitte a report on ‘media democracy in the US‘. (PDF)

71% of 13-24s read content created by others, with 56% claiming to be creators of online content.

Secondly, and more enlightening, a report from the US National School Boards’  Association (CREATING & CONNECTING//Research and Guidelines on Online Social — and Educational — Networking ) looks at the use of online tools by teens. It has a section on the educational uses of social netowrking - 59% report they use such channels for learning purposes.

It identifies 31% of teens as being non-conformists. Those who break rules (or maybe conventions) on access and safety. These non-conformists lead the way in creation, sharing and networking online.

Also interesting were the stats that show almost universal expectations across board districts that students should use online sources for learning, while at the same banning networking and communication tools in school. And also an very low percentage of teens reporting having met someone face-to-face having first met them through the Internet (but maybe they just aren’t saying). Still a very interesting picture of the educational dimensions of social networking.


The context of use

25 July 2007

Reading the Futurelab report Beyond the Digital Divide: Rethinking digital inclusion for the 21st century (Selwyn and Facer, 2007) (1) I am struck by their exposition of a number of digital divides - beynd the ‘traditional’ one of access. In particular, there is the divide caused by a misalignment of the culture of the context in which ICT is being used.

This leads me to address the question: How can we assess the ICT capability of a person without assessing the mediating effect of the context and culture in which they are operating? In particular the context of the school and the education system.

To quote the report (p16) “If the wider cultural context of use (such as the workplace,  school or home) does not fi t well with the culture of the ICT application, then  use will not easily follow. As such ICT use is not just based on the individual  being able to ‘understand’ the potential benefi ts of ICT use, but how well  ICT-based activity ‘fi ts’ with the wider contexts within which they are  operating.”

(1) Selwyn, N and Facer, K (2007)  Beyond the Digital Divide: Rethinking digital inclusion for the 21st century, Bristol: Futurelab available online at http://www.futurelab.org.uk/resources/documents/opening_education/Digital_Divide.pdf accessed 25/07/07


Heppell on ICT, creativity and the need for systemic change

22 July 2007

In a discussion in the Naace online community (1), Stephen Heppell comments on the ever-changing nature of tools and the permanence of ICT as a force for creativity. His comments resonate with the ‘practice’ side of my model in the previous post and how it can be counter to the theory/system side.

Heppell reflects on young people’s changing usage of tools - MySpace is so yesterday, Facebook today, something else tomorrow. Further, he points out that young people never e-mail now. My step daughter added a different lens to this - they do e-mail but only when communicating with adults or when they are passing on links, files etc.

Some selected quotes from Heppell’s comments are posted here (reproduced from the posting to the Naace mailing list by permission of Stephen Heppell, they should not be reproduced without citing this context)

“Most efforts that I can remember to establish standards in educational ICT have failed. And that is no loss. They hold everything back … The whole world of ICT is so organic and changes so rapidly - one minute MySpace is cool, the next it’s where your grandad goes. Just as adults get their blackberries finally emailing to each other, so children have stopped emailing altogether (”it’s what your dad does..”). And so on.”

And then, on the need for systems and policy to match students’ ambitions:

“Systems are never ambitious for children. Children are, so are their teachers, parents and others are too, but without a shared vision of just how good all this can be, it all founders into a generation of coasting kids delivering on dull targets. If you word search the “Higher Standards, Better Schools For All” white paper for example you will find the word “creativity” is entirely absent, as indeed is the word “ingenuity”.

“[I , and others,] constantly see, and are delighted by, just how ambitious children can be for their learning - especially where it is mixed age, project based, over a decent length of time, shared and not capped in any way. We need to lock that ambition into policy.”

On students a leaders of learning

“Last week I was in a school working with a group of young secondary children who were busy designing a CPD workshop to bring their teachers up to speed with the cool things they might do on Facebook, with why poking isn’t rude any more, with Bebo and myArtSpace and YouTube Comments and so on. They were very sanguine about what their teachers needed to know and were in turn interested as to the ideas their teachers might have about using these new places and spaces in learning. There is a rich irony in imagining that down the corridor their teachers might have been busy parsing a policy document to plan the ICT curriculum for those same children!”

On the need for assessment ot be relevant to this debate:

“I think if Hollywood can measure satisfaction as people leave the rough cut of a movie (and then fund the re-shoot of an ending as with Pride and Prejudice in the US version) then we can measure creative esteem, ingenuity, delight, satisfaction and so on. All or any would be more helpful measures than cohort aggregate exam passes.”

On his related projects:

“Have a look at www.learnometer.net or indeed at www.heppell.net/doctoral if you want to see where I imagine all this will be going. The learnometer project is already under way.”

(1) Naace is the association for the advancement of ICT in education. Its community has a private mailing list from which these comments are drawn (they should not be reproduced without citing this context)


Futurelab (2006) Social software and learning

20 July 2007

I hadn’t come across this paper until today…

To quote from the executive summary…

“This paper is focused on exploring the inter-relationship between two key trends in the field of educational technologies. In the educational arena, we are increasingly witnessing a change in the view of what education is for, with a growing emphasis on the need to support young people not only to acquire knowledge and information, but to develop the resources and skills necessary to engage with social and technical change, and to continue learning throughout the rest of their lives. In the technological arena, we are witnessing the rapid proliferation of technologies which are less about ‘narrowcasting’ to individuals, than the creation of communities and resources in which individuals come together to learn, collaborate and build knowledge (social software). It is the intersection of these two trends which, we believe, offers significant potential for the development of new approaches to education. ”

These new approaches include those encapsulated in these quotes

“Today, the use of social software in education is still in its infancy and many actions will be required across policy, practice and developer communities before it becomes widespread and effective. From a policy perspective, we need to encourage the evolution of the National Curriculum to one which takes account of new relationships with knowledge, and we need to develop assessment practices which respond to new approaches to learning and new competencies we expect learners to develop.

and

“A rigid curriculum inhibits the development of the knowledge and skills that may be useful in the 21st century. If we are to promote the benefits of problem solving and collaboration then they need to be validated and legitimated by the assessment system. This is the greatest challenge for education policy.

There is some sort of mapping in my mind between the two trends identified in the first paragraph and the two aspects of my research.

The report has a change in the purpose of education and its interface with the change in technologies. I have the two views of ICT/assessment of ICT, (and maybe also views of the purpose of ICT in education). Putting these four onto a diagram I see that two are to do with system, theories etc and two are to do with learners, users, practice.

futurelabandme1.gif


Future Forces and New School

2 July 2007

Derek Wenmoth has  recentlyposted two articles on his blog that have relevance to the area of my study.

Firstly, the reinvention of schooling in Knowsley - schools being replaced by learning centres with ‘rich tasks’ and enhanced possibilities of personalised learning. Does the very concept of school mitigate against the alignment of assessment with capability?

Secondly, the seemingly avant garde yet, in some way, strongly pragmatic map of future forces affecting education. Here the landscape of current trends and dilemmas is mapped out. Crucial for me are the aspects of schooling and the characteristics of “Generation Y” or “Millennials” (Howe and Strauss).


Reframing and a timeline!

26 June 2007

Following on from my tutorial at NTU I took the landscape to my ‘external advisor’, Peter Twining (of Schome fame). We spent an hour and a half in heated discussion. Heated to the extent that my brain fried but all very amicable! The outcomes were firstly a re-framing of my thoughts - and probably of my aims although that can wait for a while, and secondly a timeline for the project.

What emerged was a clarity of vision: I am looking at

A how year 11 students perceive ICT capability and
B how the assessment system (at 16) perceives it.

My project is to define the difference between A and B and to suggest ways in which the two may be aligned. This latter point, of a PhD thesis making recommendations, is one of the doctoral level learning outcomes that I hadn’t really paid attention to. Actually I hadn’t come across any of these outcomes before this month… I’ll post something about them if I can find an electronic copy or time to type them up!

I also came away with a timeline. The literature review that have embarked on will need to give way to a finding my way to a suitable methodology. This will require a change of focus of reading to look more at the methodology and methods I wish to adopt so that I may collect data in the coming academic year. Part of this discussion will be to look at the literature around ascertaining student’s perceptions and gathering the student voice.

I will also need to consider the impact of eliciting views from students in school situations as opposed to outside school. The choice of data collection instruments will also be subject to discussion - will interviews suffice, or will observation of their capability be necessary. It is likely that a piloting of a range of tools will be needed with a fuller data collection in 2008/09.

This data collection, together with the literature review, will yield information about A above. Further review of the literature, this time on policy, together with examination of assessment materials (exams, coursework assignments), will yield information on B and reveal the differences between them. This will then lead to the recommendation phase.

A rough timeline has been developed (click image to see it full size):

timelineatjune07.gif

Whither my landscape in this simplified model? The landscape had four features - assessment, learning, policy and technology. These may be seen in the model, I believe:

  • assessment is in A and B
  • learning is in A
  • policy is in B
  • technology is in A and B

PDPs, training and support for research degree students - the Anglia model

19 June 2007

My previous post was with Anglia Polytechnic (now Anglia Ruskin) University. While there I served on the Education Faculty’s research degrees committee (RDC) and also attended the University RDC. One of the things that I was involved in was early steps to develop the use of personal development planning (PDP) tools.  It is interesting to revisit this two years later on their website “Planning your research training”

I was reminded of this by discussions at the PhD supervisors’ course at my current employer (Nottingham Trent University). There is a need for us to look at this aspect of PhD support and guidance we felt.

The APU (ARU) materials came were stimulated by papers from UKGrad. Their website contains a PDP database that lists many other case studies on the development of such support and training.


General Teaching Council report calls for no school tests for under-16s

11 June 2007

Call to ban all school tests for under-16s | UK News | The Observer (10 June 2007)


“All national exams should be abolished for children under 16 because the stress caused by over-testing is poisoning attitudes towards education, according to an influential teaching body.

In a remarkable attack on the government’s policy of rolling national testing of children from the age of seven, the General Teaching Council is calling for a ‘fundamental and urgent review of the testing regime’. In a report it says exams are failing to improve standards, leaving pupils demotivated and stressed and encouraging bored teenagers to drop out of school.”

Demotivation, stress and, crucially for my work , poisoned attitudes. Will year 11s thoughts on the validity of assessment at 16 be coloured by their experience of testing (and other assessment) pre-16. Fairly inevitable I should think…


Innovate - NetGeneration

6 June 2007

Innovate - June/July 2007 Volume 3, Issue 5

The latest issue of the online journal has more articles on the “Net generation”.  Of particular interest, perhaps is the small scale ethnographic research of Lohnes and Kinzer who  found that college students still see education as being about face to face contact with teachers even thought they see ICT-mediated commuication tools as essential parts of their everyday lives.


Tutorial

31 May 2007

Had the tutorial this morning, and very helpful it was too. What emerges is a landscape.

It would seem that I have four key concepts - assessment, learning, policy and technology. Each of these informs the landscape. We talked about the need to paint this landscape and then draw out the salient features of it that inform my research questions. In the foreground of all of this is the learner perception/construct of their learning in ICT and the way in which it is assessed. Lurking over the landscape like some cloud is the thorny question - what is ICT anyway. This provides another theme which filters the light and colours the landscape.

Maybe I need to paint a picture.

We also talked about what the research is not about, and how that needs to be explained in my writing. In particular e-assessment - while a hot topic, it is not something that is especially relevant to my aims and less relevant still to the students I’ll be researching into as they won’t have had any e-assessment (probably).

Then there is the nature of ICT (the cloud above) and of assessment itself. We talked a lot about the so-called problem of ICT assessment at 16 being too easy in that it just assesses what people know rather than what was learnt in school. Actually I don’t see this as a problem. I think we need to look at our assessment and accreditation system to ensure it is fit for purpose (and valid). Why shouldn’t we give accreditation students who can demonstrate the four pillars of knowledge, understanding, attributes and skills at the appropriate level. Does it have to be only accreditation of the value added by schools.

This then led to another picture - a continuum going from the individual at one end, through family, friends, peers, teachers, schools to the education system itself. Each of my four features might have dimensions in each of these.

And the landscape metaphor has broken down… no picture needed perhaps!


14000 words

29 May 2007

I’m  minded of the thermometers you get outside churches, promoting their tower repair funds. You know the ones that show how much has been raised by a red blob creeping up a scale in the style of old style temperature measuring devices.

I’ve a tutorial this week and so felt obliged to get my literature review into some semblance of order. Currently there are 14 000 words. Unlike the fund raising gauge, though, I suspect this will fall before long as I cut out the bits that are not contributing to the thesis. 

Still it is good to look at that little word count at the bottom of Word and see six digits.

One thing that is taking shape is the concept map - now turned into chapter titles.

1.    Personal reflections
2.    Students’ construction of their learning
3.    Policy
4.    Technologies for learning
5.    Assessment
6.    Learning

These are how they emerged. I guess a better sequence might be

1.   Learning
2.    Assessment
3.    Policy
4.    Technologies for learning
5.    Students’ construction of their learning
6.    Personal reflections

This way the story builds up to chapter 5. After this will come the definitive statement of research questions and then the primary research with the students themselves. So far three schools have come on board and would be willing to have me interview students in the next academic year.

Once this is all done (!) I see the thesis structure to be developing like this

1.     Introduction, rationale etc
2.    Learning
3.    Assessment
4.    Policy
5.    Technologies for learning
6.    Students’ construction of their learning
7.    Statment of research questions
8.    Methodology
9.    Methods
10.  Analysis of data
11.   Findings
12.  Conclusion: the thesis
13.   Personal reflections

This will give me something to talk about on Thursday at the tutorial!


Article for ITTE Newsletter

11 April 2007

I wrote an article for the latest edition of the ITTE newsletter. Entitled Weblogs, PhDs and Google-generated concept lists it reflects on the process of doing a PhD. In particular issues of using a weblog and the way in which links to the log might help generate a concept map of my reading.


Innovate - The Net Generation issue

5 April 2007

Innovate - April/May 2007 Volume 3, Issue 4

Welcome to the April/May issue of Innovate. Over the last twenty years, a new generation of students has started to appear, first in our K-12 schools and more recently in our colleges and universities. Known as the Net Generation, this is a generation that has grown up with video games, computers, and the Internet. The expectations, attitudes, and technological fluency of this new generation present both a challenge and an opportunity for educators. In this special issue of Innovate, we examine how educators and educational systems can respond to the challenge and leverage the opportunity.


Becta publish Emerging technologies for learning (vol 2)

31 March 2007

Becta - Emerging technologies for learning

These publications consider how emerging technologies may impact on education in the medium term.

They are not intended to be a comprehensive review of educational technologies, but offer some highlights across the broad spectrum of developments and trends. They highlight some of the possibilities that are developing and the potential for technology to transform our ways of working, learning and interacting over the next three to five years.
Emerging technologies for learning

This publication includes the chapters:

* Emerging trends in social software for education (PDF 470KB, Lee Bryant, Headshift)
* Learning networks in practice (PDF 508KB, Stephen Downes, NRC)
* The challenge of new digital literacies and the ‘hidden curriculum’ (PDF 385KB, Jo Twist, ippr)
* How to teach with technology: keeping both teachers and students comfortable in an era of exponential change (PDF 311KB, Marc Prensky)
* Games in education (PDF 596KB, Keri Facer, Futurelab; Tim Dumbleton, Becta)
* Ubiquitous computing (PDF 866KB, David Ley, Becta)


Pupil voice

29 March 2007

So what of my aim about personal constructs of learning?

I had an interesting meeting about some research into undertaken by Barry Mearns, ICT consultant with Leicestershire’s School Improvement Service. The stimulus for his work was the latest round of subject leader development meetings and his belief (with which I wholeheartedly concur) that pupils (or students) have much to tell us about the way in which learning and teaching is organised. The students that Barry interviewed are currently in year 10 ICT groups in a county 14-19 college and there is an opportunity here for me to build on this in my future research by talking to the same students next year. Thanks to Roy Roberts at the college for making the connection for me.

This notion of “pupil voice” is becoming increasingly visible in teacher development activities that I come across. Initially with the Specialist Schools and Academies Trust and now with the DfES.

Looking on the DfES’ research informed practice (TRIPS) website I see that Pupil Voice is an emerging strand in the literature also. Some findings from the papers ther are given below. The first with its comment about constructivism is interesting itself. Maybe there is something here about the climate in the particular classes? Was uncertainty not a feature students expected n the mathematics classes investigated? How generalisable is there claim, given its surprising nature.

Dorman and Adams (2004) concluded “that teachers need to ensure their classrooms are high quality teaching and learning environments. Student confidence was likely to be raised in classrooms characterised by high levels of co-operation, harmony, genuine teacher support, student cohesiveness, task orientation and equity. Teachers who dwelt on student failures rather than helping them to build progression and who created an environment characterised by competition and conflict did not improve levels of student confidence.”

They went on to say that “Two particular concepts have been found in other research to have a positive effect on student learning. The first is constructivism, in which students make sense of the world by linking new ideas to understandings they have already built up. The second is pupil dialogue, which proposes that pupils develop their understanding through discussion. There is a broadly accepted view that constructivist learning environments support academic efficacy. So the researchers were surprised that their results showed little variation in student academic efficacy that could be accounted for by the … scales specifically intended to look at constructivist classrooms.

One possible explanation, proposed by the researchers, is “that constructivism is concerned with critical thinking and higher order learning, whereas academic efficacy concerns one’s ability to perform specific academic tasks. They suggest that it is feasible that constructivism engenders a degree of uncertainty which, in a traditional school system which values certainty, could create a loss of confidence in students. “

Whitehead and Clough (2004) “set out to explore the views of students on the factors which they thought helped or hindered their learning. The 139 Year 8 pupils came from two schools in an EAZ. In this first stage report, the researchers found that students preferred learning activities which enabled them to work in friendship groups through practical work and discussion with peers. Many students disliked whole class work. They were motivated by a variety of factors, including future prospects. 80% of pupils identified poor behaviour by others as the main factor hindering their learning. The research found the majority of pupils responded positively to being consulted about their learning environment.”

McIntyre et al (2005) “found considerable agreement between pupils in their views of teaching and learning. They preferred lessons that were less teacher-led and appreciated interactive teaching that gave them ownership of their learning. They also wanted more opportunities to collaborate with their peers.”

Elsewhere, a study by Cresswell et al (2006) for the NCSL looks at the links between personalisation (or flexibility as they conclude it to be) of the 14-19 curriculum. Here they cite the importance of pupil voice and the key role played by applied qualifications, including ICT, in leveraging personalisation.

Cresswell, L et al (2006) Personalising the curriculum at 14–19, NCSL Research Associate report available online at http://www.ncsl.org.uk/media/DC1/F7/personalising-the-curriculum-14-19.pdf accessed March 2007

Dorman, J. and Adams, J., (2004) Associations between students’ perceptions of classroom environment and academic efficacy in Australian and British secondary schools in Westminster Studies in Education, Vol. 27, No. 1, April 2004

McIntyre D et al (2005), What can teachers learn from listening to their pupils? in Research Papers in Education, 20 (2) pp. 149-168

Whitehead, J. and Clough, N. (2004), Pupils, the forgotten partners in Education Action Zones in Journal of Education Policy Vol. 19, No. 2, March 2004


QCA annual report on ICT for 2005/06

25 March 2007

QCA  have published their annual report on ICT (and other subjects) as part of their “monitoring the curriculum” exercise. The outputs from this report will (or at least should) influence their review of the secondary curriculum. The report formed part of the basis of this BBC article

Some key points in my reading of the report:

The aims of the national curriculum:

  • There is a clear recognition (by schools) of the potential of ICT to help develop pupils’ enjoyment of, and commitment to, learning.
  • Almost half of year 8 said that they enjoyed ICT compared with only 14 per cent who said that they disliked it.
  • More than a quarter of teacher disagree that the PoS ‘helps give pupils the opportunity to be creative, innovative and enterprising’ supporting previous findings that a significant amount of ICT learning and teaching continues to focus on elementary application of basic skills.
  • QCA believes that there are enormous possibilities in ICT for creativity, enquiry and innovation and the secondary curriculum review has enabled us to bring this to the forefront in the ICT programme of study (PoS). However, there may be additional barriers to using ICT in this way in schools and this needs further investigation.

Assessment

  • There is work still to be done to assist teachers with assessing ICT. Schools  say they need teacher assessment guidelines/materials for assessing pupils’ progress and continued professional development.
  • QCA has recommended that the on-screen key stage 3 test should be rolled out on a non-statutory basis, but will work to develop the test as a formative assessment tool to support teaching and learning.
  • In the survey of year 8 pupils, nearly a third of pupils felt that the level of ICT work they were being given was too easy.

Questions particular to ICT

  • There remains a lack of consistency and coherence in the ICT qualifications currently on offer, which is unhelpful for users and employers.
  • Although uptake of ICT qualifications continues to rise, enquiries to QCA indicate that for some schools the choice of qualification is made on the basis of points for league tables rather than on the appropriateness of the qualification to the learner.
  • QCA has commissioned an in-depth research probe into the qualifications offered in schools and the progression routes offered post-16. For example, if, at A level, more than 45 per cent of pupils are deciding not to continue with ICT at A2 because of their poor results at AS, it would be useful to find out the prior qualifications of those pupils who decide not to continue or whether there are other factors involved.

DfES “Making Good Progress” report

25 March 2007

This is the report that the previous post refers to… and here is an apposite quote…

A key part of the pilot is to try new approaches to
assessing when pupils are ready for moving ahead.
That judgement will of course come initially from the
teacher, but pilot schools will use external tests to
validate teacher assessments, and provide a clearer
benchmark by which parents and pupils can
measure progress. The pilot schools will identify
pupils in the relevant key stage whom they believe
are at or close to achieving a full level of progress,
who would be likely to pass a “test for progress” in
December 2007 (and at six-monthly intervals
thereafter). That decision would be discussed and
agreed with pupils, and involving parents to the
fullest possible extent.

If this is the policy for pre-14 what is the concomitant for 14-19? How does this square with the threat to the Diplomas (which seem to provide much of this flexibility - albeit with ‘employers’ added into the last sentence).


Are school tests on their way out?

25 March 2007

BBC NEWS | Education | Are school tests on their way out?
Something extraordinary seems to be happening to school tests in England.

It could be the most radical change since the tests began in the early 1990s, when they were still called Sats.

Yet this is a “softly, softly” revolution. The tests appear to be on their way out, at least in the form we know them today. But ministers do not want to give the impression they are easing up on accountability.

Meanwhile others keep letting the cat out of the bag. This week it was the turn of the head of the examinations watchdog, Ken Boston.

He suggested that national tests for all pupils could be phased out within three years and replaced by a test taken by just a small sample of pupils, sufficient to give a national picture of education standards.

Also reported in the Guardian and elsewhere … including this with some “firefighting” comment by the DfES


TALL blog Some real data on Web 2.0 use

24 March 2007

TALL blog » Blog Archive » Some real data on Web 2.0 use

Results of a survey into the use of Web 2.0 tools in FE and HE (via Derek Wenmoth)

The relatviely low use of these tools (except for Wikipedia), despite their hype and evangelistic use by a few, seems to concur with my infromal findings at HE level. It will be interesting to see what it’s like for 16-year olds…


Diplomas may go horribly wrong

20 March 2007

The BBC report Alan Johnson’s message to the Association of School and College Leaders with the headline  “Diplomas may go horribly wrong“. If horribly wrong means that  they won’t bring parity of opportunity and esteem then I guess he’s right. The Telegraph report has it that they are being scrapped post-16. So just as we prepare for another attempt to make the curriculum relevant then the goalposts are moved…


Pew Internet: Social Networking and Teens

16 March 2007

Pew Internet: Social Networking and Teens

Report published 7 January 2007

Abstract

A social networking site is an online place where a user can create a profile and build a personal network that connects him or her to other users. In the past five years, such sites have rocketed from a niche activity into a phenomenon that engages tens of millions of internet users. More than half (55%) of all online American youths ages 12-17 use online social networking sites, according to a new national survey of teenagers conducted by the Pew Internet & American Life Project.

The survey also finds that older teens, particularly girls, are more likely to use these sites. For girls, social networking sites are primarily places to reinforce pre-existing friendships; for boys, the networks also provide opportunities for flirting and making new friends.


Dereks Blog: Under 18 Blogs, Wikis & Social Networks

16 March 2007

Under 18 Blogs, Wikis & Social Networks

More from Derek Wenmoth… here he reports on a panel discussion that addressed these questions:

  1. What are young people getting out of their online lives?
  2. What is reality when it comes to dangers for young people online?
  3. What kinds of social, technological/design solutions are there once we identify experiences we want to facilitate and/or prevent?

Some things to add to the pile of reading…

7 March 2007

Dereks Blog: Taking formal education beyond exams
An article from Malaysia about a change of approach to assessment

DfES: research into ICT
A whole pile of research into ICT in schools and homes

EU: Impact of ICT
Advertised on the elearningeuropa.info newsletter, an impact study in 17 EU nations


Video Games in Education

5 March 2007

Dereks Blog: Video Games in Education

Derek Wenmoth points us in the direction of this video from Orange County USA. It talks about the use of video games for learning. The other question then is, as always. how do you assess it?


Project Approval

6 February 2007

I found out today that the university has accepted my proposal. This means that I am now officially started! Good news!

I am minded of the informal findings of Vernon Trafford, who ran the Anglia PhD training programme. There seemed to be little correlation between titles of approved projects and the final theses resulting from them. Oh well… we’ll see!


QCA review of the National Curriculum

6 February 2007

The Qualifications and Curriculum Authority (QCA) have published the draft of the new National Curriculum (NC) for secondary schools in England. This contains slimmed down programmes of study for all subjects (although ICT was always fairly slim). It also contains a section on the way in which ICT as a functional skill across the curriculum might be specified - and maps this to the subject ICT. This seems an odd mapping in many ways, as both sides of the reationship could be interchanged.

Also of note is the section on assessment strategies. These include some clear guidance on not just relying  the test at the end. I wonder how this will actually pan out at Key Stage 4 (14-16 year olds). Will schools stop entering all and sundry for a full blown level 2 qualification in ICT? Will the functional skills and some assesement of ICT use against KS4 criteria be enough? Or is this too obscure to be bothered with. What will the reporting requirements be for ICT at KS4? It is one of few subjects left in the NC at this level.

There is abother aspect of assesment that caught my eye on first read (and how much harder is a website than a set of printed documents to read!?!) . This was to do with taking a range of evidence

It does mean that more of what learners ordinarily do and know in the classroom is taken into account when teachers come to make a periodic assessment of learners’ progress at the end of term or half-year. For example, all teachers are continually making small-scale judgements about learners’ progress, achievements or the support they require when, over a number of lessons, they are reading or writing a lengthy text, planning and revising a design brief, or researching a historical figure in books or online. Such knowledge tends to be overlooked when only the final outcome, artefact or test is assessed, but it can make a vital contribution to periodic assessment.

What about what learners “ordinarily do” outside the classroom?


Developing a concept map

1 February 2007

Why haven’t I posted so much recently. There are two answers to this question. One is that my teaching load is at its peak and time has got squeezed for a few weeks. The other is that perhaps things are starting to clarify and maybe beg for some more extended writing. Is it time to start thinking about structure of the thesis - or at least the preliminary chapters?

So after a couple of months of reading where am I? Have I been able to refine my concepts and approach any? Does my research still make the same sense to the person the pub? I acknowledge the lack of reading on personal constructs but, that aside, what I have read seems to be ready for some meta reflection and organisation. Mind you that might just be procrastination. But i think I need to dig out my ‘How to write your thesis’ books.

One thing that has happened is that this blog has been hit by people alighting from searh engines. The software (WordPress) keeps a tally of what they typed in to the search to reach the blog. This list makes for itneresting reading and may help infrom the structure. Is this some sort of concept list collection tool?

Terms typed into search engines to reach this blog (Dec and Jan, most frequent at the top - although most frequent is very small, the maximum is seven occurrences for any term):

informal learning bebo
\”roger distill\” school
Demos 2007
validation ict
assessment validity Ict level 2
construct validity messick
ICT and authentic assessment
non-formal learning
Tombari & Borich portfolio
2006 5 GCSE passes school league table
assessing ict capability
assessment for learning and inclusion
assessment in ICT lesson
Assessment of informal learning
authentic assessment validity reliability
characteristics of a valid website
construct representation
construct validity model
contextual value added
define assessment validity
define formal informal assessment
demos ict report
dfes 2007 gcse league tables
Empirical Research Report on Assessment
evidence of validation in a test of creativity
formal and informal learning opportunities
formal learning
formal, informal, and non-formal
How does ICT affects Young people
ICT ASSESSMENT mouse
ict school league tables
Literature review of E-assessment Ridgway
meaning of valid in assessments
messick’s model of validity
nomothetic messick
pamela moss shifting conceptions of validity
peer assessment ict
recording and retrieving information and
Roger Distill ICT
school league tables 2007
school league tables telford
validity in ict
validity knowledge transfer vignette
validity on parole and cronbach
validity on parole: how can we go straight
What Does \”Bias\” mean and ICT
what does formal and informal mean ict


Too much technology in the classroom?

28 January 2007

This BBC report asks if there is “Too much technology in the classroom”?. A fairly light piece, it does make reference to the interface between students’ use of technology outside of school and in it.


EPPI review (2005) Motivation and assessment

27 January 2007

Smith C, Dakers J, Dow W, Head G, Sutherland M, Irwin R (2005) A systematic review of what pupils, aged 11–16, believe impacts on their motivation to learn in the classroom. In: Research Evidence in Education Library. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London.

This EPPI review, cited by Gilbert, is focusing on motivation of 11-16 year olds. Its main findings identify six themes in the key to motivation. Each theme may have some relevance here. Italics represent direct quotes from the summary of the review.

  • The role of self : how is the learner’s own constructs represented in their view of learning? How does the role of the ‘group’ affect this?
  • Utility: Students are more motivated by activities they perceive to be useful or relevant.
  • Pedagogical issues: Pupils prefer activities that are fun, collaborative, informal and active.
  • Influence of peers: Linked to role of self
  • Learning. Pupils believe that effort is important and can make a difference; they are influenced by the expectations of teachers and the wider community.
  • Curriculum. A curriculum can isolate pupils from their peers and from the subject matter. Some pupils believe it is restricted in what it recognises as achievement; assessment influences how pupils see themselves as learners and social beings. The way that the curriculum is mediated can send messages that it is not accessible at all.

In this last point, the role of assessment is raised. So what does the review have to say about assessment in general?

The way that assessment of the curriculum is constructed and practised in school appears to influence how pupils see themselves as learners and social beings. (Summary, page 4)

… assessment [has a role] in nurturing or negatively influencing motivation (page 6 and page 63)

…the recent systematic review of the impact of summative assessment and tests on student’s motivation for learning acknowledges that ‘motivation is a complex concept’ that ‘embraces… self efficacy, self regulation, interest, locus of control, self esteem, goal orientation and learning disposition’ (Harlen and Deakin Crick, 2002:1) (page 8 of the EPPI review)

Students’ motivation is influenced by their ‘affective assessment’ (Rychlak, 198 8) of events, premises and actions which are perceived as meaningful to their existence. (page 35, and linked to ‘logical learning theory’ (uncited))

Student satisfaction with their ‘academic performance tended to be influenced both by grouping, curricular and assessment practices and by its relationship to perceived vocational opportunities’ (Hufton et al., 2002:282). (page 45)

…learning situations that were authentic – in other words, appeared real and relevant to the pupils – could positively influence pupil motivation… ‘Sharing the assessment process with students is another way to capture students’ motivation…When students and teachers analyse pieces of writing together in an exchange of views, students can retain a sense of individual authority as authors and teachers convey standards of writing in an authentic context’ (Potter et al. 2001:53) (page 47 of EPPI)

Harlen W, Deakin Crick R (2002) A systematic review of the impact of summative assessment and tests on students motivation for learning. Version 1.1. In: Research Evidence in Education Library. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London.

Hufton NR, Elliott JG, Illushin L (2002) Educational motivation and engagement: qualitative accounts from three countries. British Educational Research Journal 28: 265–289.

Potter EF, McCormick CB, Busching BA (2001) Academic and life goals: insights from adolescent writers. High School Journal 85: 45–55.


Education 2020, the Gilbert Report (2006)

26 January 2007

The Gilbert report on Education 2020 contains a wealth of findings (or sentiments anyway) that have relevance to my research.

Personalisation, it begins, means assessment-centred, learner-centred and knowledge-centred… “Close attention is paid to learners’ knowledge, skills, understanding and attitudes. Learning is connected to what they already know (including from outside the classroom).”… “Sufficient time is always given for learners’ reflection.” (page 8 and citing Branford et al, 2000) - this ties in well with the meta-learning findings of Demos (2007).

“…schools therefore need increasingly to respond to: [..] far greater access to, and reliance on, technology as a means of conducting daily interactions and transactions ” (page 10, with references in Annex B). “The pace of technological change will continue to increase exponentially. Increases in ‘bandwidth’ will lead to arise in internet-based services, particularly access to video and television. Costs associated with hardware, software and data storage will decrease further. This is likely to result in near-universal access to personal, multi-functional devices, smarter software integrated with global standards and increasing amounts of information being available to search on line (with faster search engines). Using ICT will be natural for most pupils and for an increasing majority of teachers. ” (page 11)

“strengthening the relationship between learning and teaching through: … dialogue between teachers and pupils, encouraging pupils to explore their ideas through talk, to ask and answer questions, to listen to their teachers and peers, to build on the ideas of others and to reflect on what they have learnt” (page 15)

“Pupils are more likely to be engaged with the curriculum they are offered if they believe it is relevant and if they are given opportunities to take ownership of their learning. Learning, clearly, is not confined to the time they spend in school” (page 22, citing EPPI, 2005)

gilbertfig4.gif

Figure 4 – Ways in which technology might contribute to personalising learning (page 29)

The recommendations on page 30 stop someway short of recognising the relationship between technology inside and outside of formal classroom use however. There is a nod towards it in this extract: “We recommend that…all local authorities should develop plans for engaging all schools in their area on how personalising learning could and should influence the way they approach capital projects… Alongside the design of school buildings, schools will need to consider: – what kind of ICT investment and infrastructure will support desired new ways of working – how the school site and environment beyond the buildings can promote learning and pupils’ engagement… goverment should set standards for software, tools and services commonly used by schools to facilitate exchange and collaboration within and between schools software packages from home.”

Bransford J.D., Brown A. L. and Cocking R. (eds.), How people learn: brain, mind, experience and school, National Academy Press, Washington DC, 2000. teaching principles and the design of quality tools for educators. Technical report

Eppi Centre Review: Asystematic review of what pupils, aged 11-16, believe impacts on their motivation to learn in the classroom, 2005. Available at: http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=304


Wiliam (2000) on reliability and validity

25 January 2007

Wiliam’s paper, referenced by Mike Baker in his BBC summary, is not actually about the validity of National Curriculum (or any other) formal tests per se. It is about the inherent issues of validity and reliability of testing. The reduction of reliability comes from the inability of students to perform exactly the same way in tests. If they were to take the same test several times then they would expect to get different scores, argues Wiliam. This seems intuitively sensible, if impossible to prove as you can’t ever take a test again without it either being a different test or without you learning from your first attempt. The position is a theoretical one. Wiliam uses a simple statistical model to come up with the figures that are used in the BBC report. It is not that a test is 32% inaccurate, but that 32% is the number of misclassifications that might be expected given the nature of testing and quantitative scoring. The stats used by Baker are, themselves, theoretical, and should not be used as ‘headline figures’.

Wiliam then goes on to look at reliability of grades. He points out that we might intuitively know that it would be unreliable to say a student who scores 75% must be ‘better’ than one who scores 74%. But if the results are reported as grades we are more likely to confer reliability to the statement ‘the student achieving the higher level is better ‘.

On validity Wiliam says little in this paper but does point out the tension between validity and reliability. Sometimes making a test reliable means it becomes less valid. He cites the example of the divergent thinker who comes up with an alternative good answer that is not on the markscheme and who therefore receives no credit. this is a standard response by examining teams designed to eliminate differences between markers. While contingencies are always in place to consider exceptional answers, if they are not spotted until the end of the marking period then they cannot be accommodated. If several thousand scripts/tests have already been marked, they cannot be gone back over because one examiner feels that one alternative answer discovered late on should be rewarded. You either reward all those who came up with it or none. Usually it is none for pragmatic reasons, not for reasons of validity.

Wiliam (2000) Reliability, validity, and all that jazz in Education 3-13 vol 29(3) pp 9-13 available online at http://www.aaia.org.uk/pdf/2001DYLANPAPER3.PDF

and citing

Wiliam, D. (1992). Some technical issues in assessment: a user’s guide. British Journal for Curriculum and Assessment, 2(3), 11-20.

Wiliam, D. (1996). National curriculum assessments and programmes of study: validity and impact. British Educational Research Journal, 22(1), 129-141.


Openquals is now NDAQ

24 January 2007

QCA’s website of accredited qualifications, Openquals, is now known as the National Database of Approved Qualifications (NDAQ). It carries the logos of three of the UK’s qualifications’ authorities - QCA (England), CEA (Northern Ireland) and ACAC (Wales/Cymru). The SQA in Scotland is notable by its absence.

NDAQ is easier to ‘pronounce’, harder to find on Google and is easier on the eye - slightly. The myriad options available at school levels in ICT * are still bewildering. Maybe they will help with ‘personalisation’ but will they help to more validly represent learner’s abilities, achievements, capabilities?

* NDAQ has ICT, Openquals had IT… the nomenclature confusion continues…


BBC: Testing times for school assessment

24 January 2007

The BBC’s Education correspondent Mike Baker gives a very readable account of the changes ahead in the assessment system in his report of 6 January 2007 - Testing times for school assessment.

His main thrust is that changes to the system are coming in. Some of these are reflected in subsequent events that I have written about like the revamping of league tables and possible scrapping of the online ICT test… although the latter of these presumably would have helped personalisation if it was an on-demand test.

The changes, concludes Baker, are due to the growing clamour for that most voguish of educational shibboleths - personalisation.

In the article, he reflects on the Gilbert report from the HM Chief Inspector into personalisation and on how the recommendations of the report might necessarily lead to a greater role for teacher assessment. He ties this in with an IPPR study into the tensions between the dimensions of validity and accountability of assessment. Again teacher assessment is recommended by the authors as a way of enhancing both dimensions. Finally he cites Dylan Wiliam’s research into the ’shockingly’ (Baker’s word) inaccurate methods of formal assessment.

A very useful summary.

Miles Berry also summarises the Gilbert Report in his blog, again very useful.


Assessment: ‘for learning’, formative and summative

19 January 2007

One of the features of WordPress (and many other blogs) is the reporting of search terms that have been used, which then result in the blog being found.

Yesterday, the search terms reported included assessment for learning and inclusion.

This  got me thinking that I hadn’t really made any use of the simple taxonomy of assessment. Assessment for learning is formative as it informs further learning (Black and Wiliam, 1998). My focus is really on summative learning.

A study of pupil perceptions of assessment for learning  (years 7 to 10) was carried out by Cowie (1995). I guess part of my research will be looking at pupil (or student) perceptions of summative learning. It will be interesting to compare results to those found by Cowie.

The Cowie paper was cited on the DfES Standards Site, in a section called the Research Informed Practice Site. I’m not sure about the initials this provides, but the site may well be a useful one both for this research and for my teaching. I hadn’t come across it before. It is useful, not just for its own sake, but because it provides digests of articles…

Black, P and Wiliam, D (199 8) Assessment and Classroom Learning in Assessment in Education, Vol. 5, No. 1, pp 7-74

Cowie, B (2005) Pupil commentary on assessment for learning in The Curriculum Journal, Vol. 16, No. 2, June 2005, pp. 137 – 151

DfES (2007), TRIPS - the Research Informed Practice Site, London: DfES [online] available at http://www.standards.dfes.gov.uk/research/ accessed 19/01/07


Demos (2007): four characteristics of informal learning

18 January 2007

The Demos report  identify groupings of young people. Those that regularly use instant messaging, text messaging and online spaces to interact with peers are classified as ‘everyday communicators’. Those that adopt new technologies and are comfortable with a wide range of technologies are known as ‘digital pioneers’. I have no problem with the latter although I wonder whether the first is a causal or effectual label. Are they ‘everyday communicators’ because the technology enables them to communicate everyday? Or are they using the technology for communication because communicate is what they do - with or without technology?

In other words  what is the driver - their need for communication or their ‘digital native’ ability to use technology?

Taking the digital pioneers, the report then identifies four characteristics of their informal learning: self-motivation ownership, purpose and peer-to-peer communication. The last  being common to the other group, who are not identified as pioneers.

Taking out of context of the report these are fairly unremarkable. We learn best when we are self motivated, take ownership and have a purpose. Maybe the difference here is in the ownership. Digital pioneers take ownership of the technology perhaps. They go beyond the everyday use, exploiting new techniques and resources. These are the ones who are comfortable in trying out new technological tools to develop their learning - often manifested through creative products such as multimedia. That, at least, would make a far more distinctive definition for me.

And how does this relate to assessment? Is there something in the notion of self-motivation and ownership that distinguishes the higher levels? In another guise this week I have been looking at final year undergraduate and first year postgraduate assessment criteria. Higher levels of achievement are marked by ‘autonomy’. What is this if it is not creativity borne out of self-motivation and ownership?


Demos (2007) on creativity

17 January 2007

Returning to the Demos report I see a weak if discernible thread running through it - the relationship between creativity and technology. Or, put another way, the ways in which technology supports or enhances creative skills.

The report cites earlier Demos research in which young people are asked to rank ‘life skills’ in order of importance. Creativity comes a ‘only the eighth most important’ (page 27). In considering the life skills a dichotomy is established between traditional skills for the knowledge economy and the newer skills developed through growing up with technology. These are equated, in some ways, to creativity - or at least to those needed for the creative industries (p 24). Further, when surveying parents, 47% of men and 40% of women believed that their children’s use of technology helpe