Research directions
Four Research Directions
These four directions were chosen because each names something that is broken in an ordinary Indian classroom and that we do not know how to fix. This is not a survey. There are questions in education research that are more general, better funded and much further along, and we have left those to the people already working on them.
- Directions
- 4
- Studies begun
- None
- Results published
- None
- Findings that end a direction
- Published
Scope
Why these four
What is left are the four we cannot get around: what can honestly be said about what one student knows, what would make different work for different students better rather than merely different, what a school can observe that moves when learning moves, and which parts of teaching should be left alone. Each is stated as a question about a particular room over a particular stretch of time, with the difficulty that makes it hard, what would count as progress to someone outside this company, and the finding that would tell us we are wrong. The methods are not here and neither are results.
A finding that one of these directions is wrong will be published; a finding that it is right will not be claimed. When a falsifier below resolves, what appears beside that direction is one word, abandoned or revised or sustained, and the date it changed, and nothing else. The asymmetry is deliberate: an outcome reported in a single word cannot be used to sell anything, and it is the only form in which a reader can watch a direction on this page die.
We have a commercial interest in all four of these questions having encouraging answers, which is why the falsifying findings are written down before the work rather than after it. Where a reader expects a number there is none, because a number produced by the party who benefits from it is not evidence.
Index
The four, and how each is set out
Every section below runs in the same order: the question, why it is asked, the specific difficulty, what would count as progress, the finding that would end it, and what is not known. The order does not change between directions, so the four can be read against one another.
Direction 01
What Can Be Said About One Child
For a named student in a named section, can a claim about what that student knows be made small enough for a teacher to act on inside one lesson, and specific enough that a second teacher, reading the same student work, would name the same thing? That is a demand a school can put to any supplier and to any teacher: tell me what this child needs next, in words I can teach on Tuesday, and put it in a form I can find wrong by Friday. Whether a claim of that kind can be made at all is the question.
Why this direction
A mark arrives late and says nothing about what to do next. A teacher with sixty students and a syllabus to finish cannot afford a diagnosis that takes longer than the remedy. To intervene at all she needs a claim about one particular child that fits the ten minutes she has, and that is specific enough for her to tell afterwards whether it was wrong. A score cannot carry a claim of that shape. What could is the question, and it may have no answer.
The specific difficulty
The failure mode
A student gets a question wrong. She may have misread it in a language not spoken at home. She may have slipped in arithmetic. She may be missing a prerequisite from two years ago. She may have run out of time. A score attributes all four to one dimension. The obvious repair, a topic label such as weak in trigonometry, is a coarser score wearing a diagnosis’s clothes, because trigonometry is not a thing a teacher can teach on Tuesday. A claim has to be fine enough to name something teachable, and honest about the very large part of a student that no classroom evidence has ever touched: in a register of marks, absence of evidence and evidence of absence are the same entry. A confidently wrong claim delivered to a busy teacher costs more than no claim at all, because she will act on it once and never again. And it is not known whether what a student knows comes apart into nameable pieces at all. This direction assumes it does. The assumption is load-bearing and it has not been earned.
What would count as progress
- A published vocabulary for stating a claim about a student that is precise enough for a teacher to contradict, together with a record of the claims teachers did contradict.
- Evidence that a claim about a student means the same thing to two teachers who have not spoken to each other. What two teachers reading the same work agree on today is itself unknown, and establishing it is part of the work rather than an assumption behind it.
- A list, fixed in advance, of the questions such a claim must answer and the questions it refuses to answer, not revised after the fact.
- Evidence about whether such a claim survives the week, in a room that went on teaching.
Falsification
The finding that would end this direction
The direction is abandoned if a claim of this kind, put in front of the teacher who could act on it, changes nothing about what she does that week. A claim that alters no lesson is a description, and she has descriptions already. It is abandoned equally if two teachers reading the same student work cannot be brought to name the same thing, because something that means different things to different readers is not a claim about the child. And it is abandoned if what it names is never anything a teacher can do something about in the time she has.
What we do not know
- Whether the order in which a subject has to be learnt is a property of the subject or an artefact of the order in which it happened to be taught, and whether it therefore differs between two schools following one syllabus.
- How to speak honestly about knowing that is real but fragile: the student who can do it on Tuesday, in her first language, sitting beside a friend, and cannot do it on Friday.
- How to distinguish an idea a student lacks from an idea a student was never asked about, when the only evidence is whatever the classroom happened to produce.
- Whether a claim about a student has a useful lifetime at all, or whether the classroom moves faster than any account of it.
- What a more particular account of a child does under the same pressure that deformed the mark. A measurement that compresses produced a curriculum that compresses; an account that names more things offers more surfaces to perform on, and it would be taught to, and curated, the moment a school is held to it or a teacher is appraised on how many of those things she closes. Whether that deformation is worse for being the finer one, and what would tell us it had begun, are open questions, and the argument this site makes against the mark applies here with the same force.
Direction 02
Student Personalisation
Take two students in the same section who are given different work. What must differ between their work, beyond its order and its difficulty, for each of them to be better off after a fortnight than they would have been on the common sequence? And what could the teacher observe within that fortnight that separates work which was better from work which was merely different? The question is asked of two students and one fortnight of work, and of nothing larger.
Why this direction
Every Indian section holds an attainment spread that would be split into three classes elsewhere, and it is taught as one. Most of what is sold as personalisation reorders a fixed sequence or moves difficulty by one step, which leaves the weakest student doing the same thing more slowly and the strongest doing it sooner. The teacher is then asked to defend a difference whose reason she cannot see, to a parent who wants to know why her child got something else.
The specific difficulty
The failure mode
The characteristic failure is a system that gives a struggling student easier and easier work. Its own numbers improve, because she now answers correctly, while she practises around the missing idea instead of through it and drifts further from the paper the board will actually set. The second failure is invisibility. Work that differs for a reason the teacher cannot see is work she cannot defend to a parent or fold into tomorrow’s lesson, so she stops using it. The third failure is arithmetic. One teacher, fifty students, forty minutes: any differentiation that requires her to run fifty plans is not a plan, and any differentiation that cannot be said aloud in the room has quietly left the classroom. Underneath all three sits a problem with no clean escape: the same student cannot be given both the different work and the common sequence, and the students in one section are not independent of one another.
What would count as progress
- Work given to two students in one section that differs in what it asks them to do, not only in how many steps or how hard, with the reason stated to the teacher before the lesson rather than after it.
- Documented decisions not to personalise: cases where the honest answer was the common sequence, and that is what was given.
- Comparisons designed and read by people who gain nothing from the result, including the teacher who has to live with it.
- A form of divergence one teacher can still run as one lesson in one room, described plainly enough that a school could copy it without us.
Falsification
The finding that would end this direction
The direction is abandoned if the only difference that survives scrutiny is quantity: work that differs in how much and how hard and in nothing else is more practice, and should be sold under that name. The other condition cannot be put at risk yet, and that is worth saying plainly rather than leaving a reader to find it. Whether a student is left better off, rather than merely busier in a different way, waits on an observation of the kind Outcome Orientation is looking for and has not found. This direction is therefore sequenced behind that one and cannot be falsified before it. The outcome we consider most likely is that the lever was the teacher who knew the child and not the difference in the work, which runs against our commercial interest.
What we do not know
- The counterfactual. The same student cannot take both paths, and the students in one room affect each other.
- Whether differentiation accumulates across a term or is erased by the next lesson the whole class sits through together.
- What to do about the student for whom the right response is not different work at all, but ten minutes with an adult who has time.
- Whether personalisation widens the spread inside a room whose students will all be graded, in the end, on one paper and one scale.
Direction 03
Outcome Orientation
What can a school observe about one section over one term that moves when learning moves and stays still when only activity moves, at what interval, and at what cost in teacher time? The question is asked of one section over one term, and the cost is counted in minutes taken from a teacher who has none to spare.
Why this direction
Schools already have a measure. It is the mark, and it arrives too late to change anything. Nearly everything offered in its place is a count of activity, and activity can be manufactured in a fortnight by any school under pressure to show a result. A school deciding whether to continue with anything needs something it can observe during the term, that cannot be inflated by instructing teachers to inflate it, and that does not cost time the timetable does not contain.
The specific difficulty
The failure mode
Consumption measures have one fatal property: they rise when someone is told to raise them. Counts of activity can all be produced on demand, and under pressure they will be. The mark at the other end is honest but late, and it carries exam technique, question paper variance and outside coaching inside it, so it cannot be read as learning either. The gap between the two is where cheap measures fall out. Any observation that costs a teacher real time is performed once for the report and then quietly abandoned, which leaves in circulation only the measures nobody trusts; the honest observation has to be a by-product of teaching that was going to happen anyway. Attribution is not solvable by patience either, because a school changes several things in one term and the term ends only once. And we would be the party reporting any observation we propose, which is the reason it has to be one a school can make without us, and read against us.
What would count as progress
- A candidate observation stated in advance, with what it cannot see written down beside it before anyone looks at the data.
- An observation a school is still making in the third term without being asked, because it tells them something the marks did not.
- A published case in which the observation reported no change while activity rose, and the school accepted the report.
- Instruments described well enough that a school could run them without us, and a competitor could adopt them.
Falsification
The finding that would end this direction
The direction is finished if every observation we propose can be raised by telling teachers to raise it, with nothing changing in what students know: all of them would then be consumption in new clothing. It is finished equally if a school that makes the observation decides nothing differently for having made it. An observation that changes no decision is a cost with a report attached, and the right conclusion is to stop.
What we do not know
- There is no agreed definition of learning gain within a syllabus term that is not itself another examination.
- Performance and learning may come apart. What a student can do at the end of a chapter need not be a guide to what will still be there at the board exam, and we know of no cheap way to tell the two apart. We have not conducted a systematic review of what is known about this and do not offer one here.
- Attribution across simultaneous changes, in an institution that cannot be asked to hold everything else still.
- The cost side has no accounting at all. Nobody can say what an hour of a teacher’s attention is worth against an hour of her students’.
Direction 04
Pedagogy and Skill Development
Within a single lesson, which teaching tasks can be taken off the teacher without taking away the knowledge of her class that performing them gave her, and which must never leave her at all? And for a skill that no board paper examines, such as reasoning aloud, noticing that an answer is unreasonable, or doing one’s own work, what performance in an ordinary lesson would let a school say that the skill developed? Two questions, and each is asked at its own scale: the first one teaching task at a time, inside the lesson that task belongs to; the second by what a student does in front of the class, over a term.
Why this direction
The tasks easiest to hand to a machine are marking, correcting and tracking, and those are exactly the tasks through which a teacher comes to know her class. Removing them can leave her with more time and less understanding, a bad trade that is slow to show itself. Meanwhile the abilities that matter most outside the examination hall are the ones no board paper asks for, and in an Indian school what is not examined is not taught for long. Marking and tracking are among the things schools ask to have taken over, and they are the things this direction suspects should stay with the teacher.
The specific difficulty
The failure mode
Marking is not only marking. It is how a teacher finds out that many children made the same mistake, which is how she plans the next lesson. Automate it and she receives a report while the knowledge the report was meant to encode is destroyed at its source: the time is saved and the class becomes less known. We would expect that failure to be quiet and slow, which is most of what makes it dangerous. The second half is harder still. The moment an unexamined skill becomes assessable it becomes performable. Describe honest reasoning precisely enough to recognise it and we would expect a section of fifty to learn the shape of the description long before anyone thinks to revise it, so what develops is compliance with a rubric. And in a room that size, any recognition that requires watching one student closely is paid for by the other forty-nine not being watched.
What would count as progress
- A published list of teaching tasks we decline to automate, with the reasons, that does not shrink when it becomes commercially inconvenient.
- Something still in use after an evaluation has ended and nobody is watching whether it is.
- A description of an unexamined skill written in terms of what a student does in front of the class, not in terms of what a system infers about her.
- Teachers reporting that they know their class better than before, and being right when what they say is set against their students’ own work.
Falsification
The finding that would end this direction
The direction is abandoned if teachers whose marking is automated become less able to say who in their class is lost, and the understanding that used to arrive as a by-product of marking does not return by some other route. The design would then be wrong however much time it saves. And if describing an unexamined skill precisely enough to recognise it is the very thing that makes students perform the description instead of the skill, then the recognition is doing harm and should be withdrawn.
What we do not know
- Whether professional judgement decays with disuse across a teaching career, and if so how fast. The timescale is decades and the evidence would take that long to gather.
- Whether recognition of an unexamined skill means anything to a family choosing a school in a market that reads marks.
- Whether any skill the board does not examine can survive the two months before a board examination.
- How reasoning aloud can be developed, and fairly recognised, when the reasoning must happen in a language that is not the one spoken at home.
Publication policy
How a direction on this page ends
One class of finding is not withheld, and the asymmetry is deliberate. Every direction on this site names an observation that would show it to be wrong. When such an observation arrives, the outcome is published here as a bare direction, abandoned or revised or sustained, with the date it changed, whether or not a study is ever written about it. A null finding is published on the same terms. A finding that a direction is right is not claimed at all until it can be published as a study with its method attached. The half of the record that could be used to sell something is the half that has to wait; the half that costs us is published as it arrives, and a reader deciding whether any of this is load-bearing should watch the status of the directions rather than this paragraph.
| Direction | Status | Last changed |
|---|---|---|
| What Can Be Said About One Child | No study begun | — |
| Student Personalisation | No study begun | — |
| Outcome Orientation | No study begun | — |
| Pedagogy and Skill Development | No study begun | — |