The mark is not the measurement

A mark is a scalar, produced at the end of a unit, a term or a board year, by a person with a stack of papers and a syllabus still to finish. It stands in for everything a student knows about a subject.

This is not an argument against examinations. A common paper, externally set and externally marked, is among the fairer instruments Indian schooling has: comparable across schools that share nothing else, portable, auditable, and indifferent to whom a family knows. It is also close to incorruptible, and that is the property any richer account of a student will find hardest to match. A detailed claim about a child, produced by a vendor, inside a system that allocates scarce seats, invites precisely the pressure the examination was built to resist.

The argument here is narrower. The mark is a serviceable instrument for ranking and a poor instrument for teaching, and Indian schooling uses it for both, because at the point where a decision about a child is made it has nothing else it trusts.

Consider two students who score sixty-two on the same mathematics paper. The first can carry out every procedure the paper asks for and loses marks in every long question to one habit: she drops the sign when she distributes a subtraction across a bracket. That is a single motor error, ten minutes of attention, correctable on a Tuesday. The second answered the quadratics section without a mistake and could not start the word problems, because he cannot turn four lines of English into an equation. His difficulty is not in mathematics. Both are sixty-two, both are entered in the register as average, and if the school runs remedial classes at all, both are sent to the same one and given more of the same paper.

What the compression discards is never general. It is always something specific, and always the thing the teacher would have needed: which topic, which misconception, which prerequisite, which skill. There is a fifth loss, different in kind from those four, and it is the one that should trouble people most: which moment.

Marks are produced after. After the unit is closed, after the class has moved on, after the misconception has been rehearsed for three weeks and become the student’s settled method. The window in which that sign error was cheap to fix opened in the second week and shut long before the report card was printed. A number that arrives after the moment in which it could have been used is not a measurement of learning. It is a record of an outcome.

Because the mark is the measurement the system trusts most, teaching bends towards it: previous years’ papers, the expected question, the marking scheme, the answer that scores. That is not a failure of teachers, who are behaving rationally under the only instrument they are held to. It is a property of the instrument. A measurement that compresses will, given enough pressure and enough years, produce a curriculum that compresses.

That argument does not stop at the mark. Anything proposed in its place inherits it, and a finer account of a child offers more surfaces to optimise than a single number does, not fewer. If a school is ranked on how many gaps it closes, gaps will close on paper. If a teacher is appraised on the same account, it becomes a document written for the appraiser rather than a description of a child. Our view is that this deformation would be worse than the mark’s, because a description is easier to curate than a common paper marked by a stranger, and the sign that it has begun is a school asking for the record to be improved rather than for the child to be taught. Before it buys anything, a school should ask any supplier proposing to stand in for the mark who sees the account, who is ranked on it, and what happens to it at an inspection.

No remedy is proposed here, and the reason is not modesty. Diagnostic alternatives to the mark have been put to Indian schools for a long time, some of them by the state. We have not surveyed that record and do not offer a survey of it here, so we name no programme and give no count. Our impression, which is an impression and not a finding, is that they failed on the teacher’s week more often than on their theory, by asking for time that does not exist. Any account of this problem that does not begin from the teacher’s week is describing a different country.

Academic intelligence is not a product and not a polite synonym for artificial intelligence. Intelligence here carries the older sense it has in military intelligence or market intelligence: knowledge about a situation, assembled deliberately, in order to act well inside it. The adjective does the limiting work. Knowledge about learning, assembled in order to teach.

The distinction that matters is between a record and a model. A record states what a student produced: this answer, that score, this many minutes, on that date. Indian schools already hold far more records than they can use, and digitising a record does not change its kind. A model states what a record cannot: what a student knows, what she can do, and what she is ready for next.

Those are three different kinds of claim about a person, not one claim with three names. Knows is a claim about a present state that is not directly observable. Can do is a claim about a capability under conditions, and the conditions are doing more work than anyone admits. Is ready for is a claim about a future, inferred from a past that was thinly observed. A system that treats the three as interchangeable will be confidently wrong in a way that is hard to detect from outside it, and harder from inside.

This is why the difficulty is representational before it is technological. The questions below have to be answered before any question of method arises, and a claim that passes over them has not avoided answering them. It has answered them by accident.

We will not describe the representations we work with, here or anywhere on this site. Nor is that silence a settled answer being withheld: whether the useful unit is the student, a topic, or something narrower again is an open question here, and this site writes it as one. The point survives whatever anyone chooses to build: if the representation is wrong, nothing downstream repairs it, and a great deal downstream will make a wrong representation persuasive, which is to say fluent, confident, rendered in a dashboard and unfalsifiable. The field’s characteristic failure is not weak technology. It is strong technology resting on an unexamined account of what a learner is.

Academic intelligence, used as this page uses it, states an intent and not a capability. Nothing on this site should be read as a report that anything answering to it has been built, and anyone using the term, ourselves included, can fairly be asked which of the three claims above they mean and what observation would show them to be wrong about it.

A model of a learner is easy to make plausible and hard to make survive.

Plausibility is cheap because the case everyone reasons from is clean: one learner, dense and continuous evidence, an attentive tutor who can ask a follow-up question the moment something looks wrong, a single language, unlimited time, no syllabus deadline. Much of what we have read about adaptive learning describes, structurally, that situation. The Indian classroom inverts nearly every one of those conditions at once, and the inversions interact.

What is genuinely unknown is whether an account of a learner that is adequate under those conditions degrades gracefully or collapses. Our working belief is that it collapses, and that the collapse is not accuracy falling by some margin but the model’s claims quietly changing meaning. A claim earned in a tutoring setting becomes, in a room of fifty-five with shared devices and circulated notebooks, a claim about something else entirely, while continuing to be worded in the same way.

There is also an evidence problem that no care on our part resolves. We have not conducted a systematic review of the literature on learning technology and do not offer one here, so this site states no proportion and makes no claim about that literature as a whole. The premise we work from is narrower: the studies we have actually read were run in other class sizes, other languages, other assessment cultures and other incentive structures, and transfer to a Class X section in a Tier-2 Indian school is something a supplier should have to show rather than assume. That is a question a school can put to anyone, us included. Where was your evidence gathered, in what class size, in which language of instruction, and who paid for it.

One blunt test, which we would want applied to any claim of this kind and have applied to none of ours. Show an experienced teacher several accounts a system has produced of students in her own section, with the names removed, and mix in among them an account of a child she has never taught. Ask her to place them. If she cannot do better than she would by guessing, or cannot reject the stranger, she has been shown fluent prose and not an account of anyone. If she can, two answers are worth having, and the more useful of the two is that is wrong about Divya, because a falsification is something a person can act on the same afternoon. A test with no foil in it measures plausibility, which is the property such prose has whether or not there is anything behind it.

What we do not know, stated plainly. Whether a representation adequate for procedural mathematics carries over to a subject in which the answer is an argument rather than a result. How much of what presents as a knowledge gap in an English-medium classroom is a language gap wearing its clothes; we have not measured that fraction, and if it is large, a great deal of what is read as evidence about what a student knows is evidence about how fast she reads a second language. Whether the evidence a real school can afford to collect, in the time it actually has, is sufficient in principle to support the claims people want to make from it. And the uncomfortable one: whether a teacher told something true and specific about a student on Tuesday can act on it at all, or whether time binds so tightly that better information changes nothing. If that is the answer, most of this work is beside the point, and we have no basis yet for assuming it is not.

The standard is narrow, it is inconvenient, and it is the one we would want applied to anybody else making claims about Indian classrooms.

The work is worth doing if a teacher of fifty-five, given no additional hours in her week, ends a Tuesday knowing one true and specific thing about one named student that she would otherwise have learnt in March, and can do something about it before Thursday. That is the whole of the ambition. It is a smaller claim than the field usually makes and considerably harder to satisfy, because every constraint in it is real: no extra time, a named student, an actionable window, and true.

We have not met that standard, and nothing on this page should be read as a report that we have. The conditions below are what we would accept as failure, stated in advance rather than discovered later in a testimonial.