Dictation mark scheme - what on earth is going on?

I wrote the week before last about my experiences with gap-fill dictation in class, and where I felt it took my pedagogy. That also inspired me to look really hard at the new ‘section B’ of the new GCSE exam. Here I’m going to look at the two specimen papers for AQA French, German and Spanish and see whether we can infer from them exactly what the exam board really is trying to assess here. I’m not going to go into debates about how dictation in French is by definition harder than it might be in German or Spanish, but try and dig into what’s properly going on here.

(A tangent. I think the exam boards do a lot of great things and it’s a bit of a thankless task, but I would LOVE it if the commentary provided a justification as to how they created the questions in the paper and what research or evidence this was based on. It seemss a pity that we have to second-guess like this, and the commentary that they do offer is piecemeal and cursory.)

Anyway, here goes. 

Exam structure

Within the listening exam, the dictation task counts for 20%. That is, 8 marks out of 40 at Foundation, and 10 marks out of 50 at Higher. That means the dictation counts for 5% of the student’s final grade. 

Of these scores, it’s even between AO1, which is ‘communication of meaning’ and AO3, which is ‘transcription and grammatical accuracy’. For the communication of meaning mark, we are to ‘consider the spoken extracts as a whole’ and we are to look at this before we look at the ‘grammatical accuracy’ score. 

However, dichotomising these two is false. Communication of meaning by definition depends on some level of grammatical accuracy; perhaps we don’t need to worry about that too much, though, because of course a student is very unlikely to get 5/5 on communication of meaning if the transcription is ‘very rarely correct’ and the level of grammatical accuracy is ‘very limited’.

Mark scheme confusion: French in detail

Nonetheless, going over the mark scheme carefully leaves me confused right at the start, looking at all three languages.

In specimen 1 higher, the sentences dictated are as follows: 

For French:

1. Notre chien / est noir

2. Un ami écoute / avec attention.

3. Il faut dire / aux enfants / de manger sain.

4. J’ai mis / du jambon / dans ce plat.

5. Je serai / pilote d’avion.

The commentary tells us that the first sentence is to build confidence - in other words, that it should be easy. Sentence 2 is testing silent final e, tion and n-liaison. I feel that the ideal ‘un ami’ is contrived - surely ‘mon ami’ would have also tested the n-liaison in a way that would be much more familiar to students. Yes it would have led to ambiguity - is it ‘mon ami’ or ‘mon amie’ - but given that the sample mark scheme allows ‘Un amie’, why not go with it? 

Sentence 3 claims to be testing (among other things) the x-liaison, but phonologically a z-liaison would be identical: will they allow that? Because if they don’t, they’re testing grammar knowledge. A similar issue arises with ‘j’ai mis’ in sentence 4: they claim to be assessing silent final consonants, but there are many ways one could spell ‘mis’ correctly in French. And why ‘dans ce plat’, which sounds just like the much more frequent ‘danse’. ‘Dans le’ would be less of a trap.  

There is as yet no commentary document for sample 2.

The foundation sentences are:

1. J’aime bien / le cinéma.

2. Ma copine / est jolie.

3. Le lundi / on mange / du poulet.

4. Vous portez / une belle / chemise.

Both foundation and higher include two words that are outside the prescribed vocabulary list. In the higher paper these are ‘jambon’ and ‘avion’, and in the foundation, ‘poulet’ and ‘chemise’. I think ‘poulet’ is mean, given that it could be spelled poulé, poulez, even poulaient, and this is likely to compromise a foundation student’s concentration as they puzzle out a possibly unknown word. Shouldn’t there be a guidance that non-vocabulary list words do not contain sounds which could be formed in multiple ways in French? Fromage, pomme, even pamplemousse would have been less of a stretch. The mark scheme does  say that perfection isn’t required, but ‘numerous minor errors’ are likely to have an impact on the mark for AO3, and the choice of ‘poulet’ could cause an additional error.

For German:

1. Wir haben / keine Hausaufgaben / bekommen.

2. Zu viel Schokolade / ist nicht gesund.

3. Ich werde / heute Abend / Gitarre spielen.

4. Man sollte / mehr für / die Umwelt tun.

5. Unsere Nachbarn / fahren / in den Urlaub.

And for foundation:

1. Er mag / sein neues Haus.

2. Die Kantine / ist prima.

3. Du musst / jede Woche / spazieren gehen.

4. Ich esse / einen grünen Apfel.

Thinking about the non-list words, here we have Schokolade and Gitarre at higher, and ‘prima’ and ‘Apfel’ at foundation; much more orthographically transparent than the French words. 

For Spanish:

1. Su cumpleaños / es el quince / de junio.

2. La plaza / estaba llena / de turistas.

3. Las celebridades / ganaron / mucho dinero.

4. Hay descuentos / para estudiantes / los viernes.

5. No debes compartir / tus datos personales.

And for foundation: 

1. Mi hermano / juega / al baloncesto.

2. Muchos turistas / admiran / la plaza.

3. Es una ciudad / pequeña.

4. Hay cuatro torres / en el castillo.

Again, as far as non-list words are concerned, at foundation we have ‘admiran’ and ‘torre’, which seem tough to me. Admiran patently demonstrates an overlap between sound-symbol-correspondence and grammatical knowledge, and ‘torre’, which claims to be testing the ‘rr’, is therefore honing in on a sound which is going to be inherently difficult for English speakers. Yet at higher level, on of the non-list words is ‘personal’ ! 

What’s driving this mark scheme?

I have other things to say about the German and the Spanish, but will weave these in through a lens of attempting to pick apart what the underpinning foundations are for these tasks.

The commentary on grammatical accuracy does not even address the following, in the ‘top marks’ section:

In the German, the lack of capital letters on two nouns.

In the French, ‘de mangé’ for ‘de manger’

In the Spanish, ‘Hay discuentos’ written as ‘hay des cuentos’ 

For AO1 - communication of meaning - it seems that a key criteria is whether ‘the reader has to pause to think about what has been written’, but this is a very woolly descriptor. Who is the reader? This is as bad as the previous terminology of the ‘sympathetic native speaker’ in the last GCSE but one. So we’re a bit stuck here. I would have to stop and pause with ‘de mangé’ and ‘des cuentos’.

Going back to the grammar criteria, AO3 - in the French they tell us a misspelling of ‘je serait’ for ‘je serai’ means ambiguity and dropping a mark, in communication of meaning - and ‘je seré’ would also not be permitted. Still they don’t seem to be concerned about ‘il faus’ for ‘il faut’, ‘es’ for ‘est’.

‘Il écout avec’ is deemed not OK, even though they have been transparent about testing other liaisons and theoretically, this would work with a t-liaison (think ‘il est avec’). Honestly, I can’t work out what the underpinning rules might be here at all. The go-to rule might be ‘is there an alternative spelling that actually exists? (eg in the German disallowing the confusion between ‘Mann’ and ‘mann’) , and if so, they must provide the correct spelling’ - but that would mean that ‘j’ai manger’ wouldn’t be allowed. In the Spanish, a confusion between the dictated ‘no debes’ and the student’s offering ‘no debeis’ is apparently fine.

It gets worse when you look at the German. Because even though ‘unser’ for ‘unsere’ seemed to be fine to be overlooked in ‘student 1’, in ‘student 2’ it counts (rightfully) as ‘incorrect case ending’ and contributes to a lower mark for AO3. To penalise a student for the difference in spelling between Mann and man, though, seems overly mean. And to me ‘wir habe kein Hausaufgaben bekomme’ just sounds like dialect; I wouldn’t have to pause to make sense of that!

A good idea gone bad?

So now we are in a position where a good idea has gone bad. I still applaud the washback which should contribute to more care and thought given to actually teaching listening, not just testing it in the classroom (see this blog). But the commentary on the markschemes is so opaque as to be utterly unhelpful. At one point it even talks about the number of words in all five sentences, and how many of them have been accurately transcribed. If that’s how you want to do it, tell us explicitly, AQA! Tell us ‘count the correctly transcribed words’ and give us one of those little grids telling us how many ticks corresponds to how many points. This would at least give this element of the mark scheme some level of reliability. Because at the moment it is neither valid nor reliable.

Student examples

And one more thing. Where DO they get these student examples? They seem contrived to me and very much at odds with what I would see in the French or German classroom (I don’t have enough experience with teaching any kind of transcription in Spanish to comment on those ones). And I’m afraid that this infers to me something that I have long since worried about: that AQA simply do not pilot any of their questions or future papers: it’s just a best guess. The drift into something that looks like Dutch in the German student 3 (for ‘Man sollte mehr für die Umwelt tun’, this hypothetical offering is ‘mann sollte meer vor de oomvelt tu’) is nothing short of contrived.

Where does that leave the teachers?

Throughout this post I have suggested ways for AQA to improve their dictation offering. I don't want to end this post without some attempt at usefulness or constructiveness for the teachers (including me) that now find themselves teaching dictation with a mark scheme that is as almost as opaque as English sound-symbol correspondences. So what's to be done?

Firstly - don't worry too much. It's only 5% of the final exam.

Secondly, do teach your kids how to spell in the language you're teaching - there is enormous overlap with the demands of dictation and those of the new read-aloud task. Do your best but don't panic about it.

Thirdly, teaching sound-symbol correspondence should be seen as a golden-thread that runs through very many of your langugaes lessons from the start of year 7. My students all have a little grid at the back of their books with three columns: 'letter / sound / example'. And nearly every time a student mispronounces a word, they hop back to that part of their book to check and 'sound out' the word. By the end of year 8 they are good spellers and pronouncers.

A call for transparency

So do you know what? Perhaps my tangent at the start of this post wasn’t much of a tangent after all. Perhaps AQA (other exam boards are available!) really would benefit from being very transparent about how they have come to these commentaries. Because I think this might actually force a rigour that doesn’t seem to be there at the moment; or force a conversation at exam board level about quite what they are trying to achieve with this dictation lark.

PS: people at AQA, I’d LOVE to help you get this right. I think there are ways it could be fixed fairly easily! You know how to get in touch with me; respond here, or, I’ve already written to you asking if I can help out with the listening exam setting!)