How much of what kind of assessment?

Chris Rust and Mandy Jack initiated an exchange about assessment volume, kind and equivalence on the SEDA maillist.
Hours are most easily countable (and even that is not easy). There are also the number of assessment points and the “weight” given to each assessed point. Chris asserted that:
the only meaningful comparison is student hours and providing the total identified learning hours for the module (including the assessment activities) aren’t over 10 per unit credit there’s no problem.
I asked for clarification on the “total identified learning hours” per unit credit.
This is a slight diversion, but the question of equivalence is becoming increasingly important. So, too, is the question of “challenge”.
Toetenel and Rienties (2016) analysed 157 Open University course designs by all levels and activity types. Assessment (as designed or intended by those course teams) hovered around 25% of the time for all 3 UG “levels” and about 21% for PG. I do not know how much time the students actually did put in to each activity. Galvez-Bravo (2016), “… found that modules with more assessments had higher feedback (module appraisal) marks.” Simpson et al (2019) in a UK/US comparison found more assessment (both in number of assessment points and total volume) led to higher grades.
So a should a 15 credit course (150 hours) be expected (or designed) to have about 30-40 hours of assessment arranged over between one and as many as eight points in a semester? Two to four assessed tasks of 10-20 hours each? Where “exam revision” counts towards the total assessment load (two or three hours of exam and 10 hours of cramming the day before)?

Badges: Learning Gain or Just a Game, and what’s wrong with that?

Badges are Digital image files with text metadata stating criteria for which the badge has been earned. Badges are (presently) self-certified by Learner or Earner and  Self-certified by Provider or Issuer. Below are resources for a short session I ran for the Technology Experimentation Group (TEG). Continue reading “Badges: Learning Gain or Just a Game, and what’s wrong with that?”

A note on content, courses/curricula, and credentials

This note recounts a potted recent history of developments to do with online content and courses and speculates about the future of credentials in respect of the purpose of a university.

When learning management systems (LMS) or virtual learning environments (VLEs) were in their infancy around the turn of the century, faculty opposition to their introduction was sometimes expressed. Two reasons were often given (among others): if I put my lecture notes on the web no one will come to my class; and variations on a personal IPR theme: the students, the university or third party institutions will steal my content. Content was considered “king” and as long as universities and academics “owned” the content, their position was secure

MIT, with the Open Courseware initiative shattered the content-is-king myth. All the content from a leading university was made freely available: curricula, syllabus, reading lists, slide sets and exam papers.

The defensive focus shifted to courses. It wasn’t the content per se that was important. What the faculty and the institution did was select, organise, interpret, analyse and re-present content through curricula presented in courses (or modules, blocks, units, etc): sequenced events of limited duration (often a semester) presented in various modes (face to face and at distance)..

In 2007 groups of academics started to offer open online courses, hosted at universities but not requiring enrolment or a fee. This open online course movement became truly massive (MOOC) in 2011 when Stanford and MIT began to offer open online courses. From these beginnings spin out ventures (Coursera, Udacity, EdX, FutureLearn and others) started offering open online courses to the higher education “market”.

Now academic defensiveness has shifted. It is not the content and it is not the course or the curriculum that are offered uniquely by the university and faculty. It is the credential. Universities can award degrees and the degree, backed up by quality assurance processes is the guarantor of learning quality and the unique proposition which protects the value of the university.

Is it?

There is a movement towards micro-credentialling, or “badges” which I suggest is more important that many faculty and academics allow. I wrote briefly about badges here. I suggest this movement will continue the trend of opening up the university proposition and further challenge the role of the university in society.

A note on badges

On FSLT13 Badges were  awarded for completion of each of the four activities. Participants who wanted to collect the FSLT13 badgesl needed to register and enrol on the Moodle – AND needed to sign up for a Mozilla Backpack. Badges do not carry any academic credit but are a fun way to signal engagement with the course. Badges were be awarded using the WP Badger plug-in for WordPress, which implements the Mozilla open badges framework, Mozilla Backpack and Persona.

Why badges? We are doing this course to explore some of the developments on the cutting edge of contemporary learning and teaching practice. Badges for lifelong learning are on this rapidly approaching horizon: see Mozilla Open Badges Blog, HASTAC What’s on your badge list, and James Michie’s excellent and balanced presentation on badges on his Open Online Course #crit101.

First thoughts on the final (?) draft Strategy for Enhancing the Student Experience

Oxford Brookes Academic Enhancement and Standards Committee has made its final (?) modifications to the draft Strategy for Enhancing the Student Experience (SESE). The two objectives of this strategy are to:

  • [implement] approaches centred on critical reflection, impact evaluation and continuous enhancement of the student experience.
  • [maximise] student involvement in the development of policies and practices for teaching and learning and in extra-curricular, student led initiatives.

The strategy has principles and guidelines and (eventually) will have maps. Overall the tone is a bit directive: there is a lot of insistance and requirement, but the intentions are (I believe) worthy ones.

From a PCTHE perspective some of the key messages of this document are:

  • All staff who support learning will participate annually in high quality professional development (3.5.2)
  • All academic staff who support learning will engage with processes of evaluation, reflection and research into pedagogic practice (3.5.4)
  • The fundamental purpose of assessment will be to help students learn by providing formative feedback. (3.6.2) (so obvious but so often ignored in practice)
  • Students will be expected to take responsibility for their own learning, to actively engage with feedback and assessment (3.3.1) (this is perhaps one of the key messages of the Assessment Compact)
  • Students [will have the opportunity] to provide input and play a role from the outset in the development of new programmes. (4.3.4)

From a wider educational development perspective:

  • [With our] internationally recognised in-house expertise in educational development, we commit to routinely carrying out impact assessment, review and revision of all significant academic development initiatives and of measures taken forward in the SESE and the consequent strategy maps. (4.4.3.1)

Learning technology messages

  • [People will be] able to use technology to shape their own learning environment and interactions. (3.2.2)
  • [The] curriculum will be enriched by technologies that empower students’ development as self-regulating, digitally literate learners, able to shape their own learning interactions and author their own digital artefacts. (3.4.5)
  • The physical environment will be augmented by digital environments and technologies in ways which support a distinctive Brookes learning experience (3.7.2)
  • The functional access, skills and practices necessary to become a confident, agile adopter of a range of technologies for personal, academic and professional use. To be able to use appropriate technology to search for high-quality information; critically to evaluate and engage with the information obtained; reflect on and record learning, and professional and personal development; and engage productively in relevant online communities. (4.1. d)

Writing a Course Review

I recently met with a student who had been unsuccessful in achieving the criteria for one of the assignments on the PCTHE. Most of my comments are of a general nature regarding the writing of a course review, so I thought I would post them.

  1. Back up your assertions with evidence. As teachers and students in higher education we must be familiar with the many forms that evidence can take. Avoid unsubstantiated statements and in every case where an assertion is made, think of how it might be evidenced.
  2. In general we are interested in your experience, ideas and theories, not your ability to reproduce the theories of others. Theory is simply an answer to the question “why”? That is, one of the principal purposes of theory is explanatory. It does help to recognise the theories of others that came before and have also experienced similar questions and come up with their own explanations. One of the principal theories that underlies this course is Constructive Alignment. Frequently problems in course reviews arise at least in part due to a lack of such alignment. And, a solution might, in part, lie in taking an aligned approach to course and assessment design and implementation. Remember there should be alignment between objectives – or “intended learning outcomes” (what the learners should be able to DO), learning activities in the classroom (or wider teaching environment), which allow the students to practice the objectives, and assessments which allow the students to demonstrate that they can do what ever the objectives suggest. Assessments should be backed up with criteria by which that doing can be evaluated.
  3. Tell the story. Start at the begining:
  • In the first paragraph state the key point of the paper (write this after you have written the rest)
  • I first taught this course …
  • It attracts such and such kind of students. Be as precise as you can with the observable demographics: age range, gender mix, full-time/part-time, in employment or not, etc. Avoid simplistic psychometrics (learning styles, etc) unless you have a particular reason to use them.
  • The course as inherited had these characteristics:stage, CATS points, programme context; external professional bodies (referees); aims (what you and the referees want to achieve); learning objectives (what the students must DO); teaching activities; assessments
  • The course as inherited had these problems (whatever they were). Provide evidence for how you know these were problems: i.e. attendance, participation, engagement, outcome (exam) results, informal feedback, your own observation, discussion with students, discussion with colleagues, etc. Draw on the writings and theories of others to show that these problems are not unique to your situation. That is to say, use outside authorities to show that your problems may be generalisable to a wider context.
  • In light of the students and the problems observed, you have made the following changes: changes implemented during operation in the first year (i.e. through reflection IN practice); changes implemented in a revision (i.e. through reflection ON practice)
  • Evaluation: how will you know the changes have been effective?
  • Next steps