Measuring Learning

The count that decides what to improve is a closed-book reconstruction done about a week after study, scored against the highest and the lowest level of performance the assessment asks for. Pages covered, lectures sat, and hours logged are not part of that count.

The count has one more item: the name of the part of the learning process that is currently capping every other part. A skill is good enough when it is no longer that part.

Core takeaways

• The count that decides what to improve is a closed-book reconstruction done about a week later, scored at the highest and the lowest level the assessment asks for.
• Hours logged, pages covered, and lectures sat measure activity. They do not show what is still there a week later, and they hide the time later spent relearning.
• Learned material sits at one of three levels: isolated recall, use of one idea, and concepts held as a network with judgment about which relations matter. The third level is the target.
• Efficient learning means reaching the mastery the purpose requires, holding it at each required level, and doing both in less time. It is a direction; there is no formula to compute it.
• A rate limiter is the part of the process that caps every other part. It is found by reflection every week or two, and it is the next thing to work on.
• A skill is good enough when it is no longer the rate limiter. No absolute standard exists; enough is decided by comparison with the rest of the system.

What a count of hours leaves out

A count of hours spent covering material measures activity. It does not measure learning. It leaves out whether anything is still there a week later. It leaves out the quality of what was built and whether it can be used in more than one way. It also hides the time spent later relearning what was forgotten. That hidden relearning time makes the count expensive as well as incomplete.

Two feelings inflate the count. Rereading feels easy, and a reader has a sense of having covered a stretch of material. People judge what they know by those two feelings, and both are poorly calibrated against what can be produced later. Material reread until it is fluent feels known, and that feeling of fluency is what the count of coverage records. Rereading is also the method learners prefer, and that preference inflates the sense of coverage further.

Three levels of knowing

Pages covered do not say how much has been learned. The useful count is a position on a scale of levels. The scale is this system’s own. It is built from published taxonomies of learning outcomes and does not copy either taxonomy word for word.

Level	What it lets you do	How it shows up on a check
Isolated recall	Recite facts and recognise terminology	Flashcards on leftover facts, cover-copy-check
One-idea use	Explain a single concept and solve a simple problem with it	Extended or atypical applications still stall
Relational and evaluative	Hold concepts as a network and handle novel, interrelated problems	The check asks which relations matter; it does not count how many facts remain

The first level is built by repetition: flashcards on leftover facts, and cover-copy-check. At the second level a learner can explain a single concept and solve simple problems with it, but extended or atypical applications still stall. That stall separates the second level from the third. At the third level a learner holds concepts as a network and handles novel, interrelated problems. The third level is the target.

The levels are ranked for a reason. Lower-order work produces only lower-order outcomes. Higher-order work produces both lower-order and higher-order outcomes. Both kinds of work are needed. The lower-order repetitions belong as a supplement, and they belong closer to the assessment. A higher level does not mean more relations. Identifying how concepts relate helps moderately. Weighing which of those relations matter is where the encoding deepens.

Connected knowledge is more usable and usually more durable. A well-drilled isolated fact can still outlast a sloppy comparison. So a first-level limiter is still worked on, even though the third level is the long-term target.

The third row of the table holds several levels. Higher Order Learning describes them one by one. The full scale from recognition to usable knowledge, level by level, is on Knowledge Mastery From Recognition to Usable Knowledge.

Two questions for one week of work

If the measure stops at coverage, the work keeps producing coverage rather than the long-term target. A system measure has to answer two questions, and both are asked of the same week of work.

1. What can still be produced later, at the level the purpose asks for?
2. What is currently capping everything else?

The second question exists because a week of technique can be improved endlessly, so no absolute standard says when to stop. Enough is always comparative. The comparison is with the other parts of the same system. It is not a comparison with a target level.

The week-later check

The learner names the highest and the lowest level of performance the assessment actually asks for. About a week later, the learner tries to produce the material at each of those levels and compares the result with what will be demanded. The week is this system’s own interval: its spacing cadence already puts a retrieval session about a week out. The check does not produce a percentage, a ratio, or an average.

The check uses up one study session. It also produces a list of what could not be produced at the higher levels. It is slower and less pleasant than rereading. That is its cost.

What efficient learning means

Efficient learning, as this system defines it, is three conditions met together: reaching the mastery the purpose requires, holding it at each level required, and getting both in less time. That is a direction: retained usable mastery per hour. Pages per hour is not the measure. There is no formula to compute. A home-made quotient is the coverage count with arithmetic added. Efficiency is also relative to the requirement. The same week of work is efficient against one assessment and not efficient against another.

Better encoding causes less forgetting, so less relearning is needed, and total time falls a great deal. Better organisation makes the material mean more, so it needs repeating less often. Once the method caps the level reachable, doubling the hours moves the result very little. More hours help until the work needs something the current method cannot do. At large volumes the forgetting rate is high enough that full retention is never reached, and the shortfall is worst at the higher levels.

The arithmetic behind the efficiency idea has no published validation.

The part that caps the rest

A rate limiter is the part of the learning process that caps every other part. However good the other parts are, the result cannot rise past the limiter. A learner with strong technique who procrastinates is limited by procrastination. Technique is not the limit in that case. Effort spent on the other parts does not raise the result past the cap.

Work on the limiter next, even when you are not yet sure which part it is. Trying to find it and address it improves the whole process more smoothly than working on other parts first.

The limiting part changes over time. A month from now the cap may be time management instead of mindset, either because time management got worse or because mindset got better. A plateau is the signal that a new limiter has appeared. A plateau can also be a measurement ceiling or a drop in motivation.

Limiters are found by reflection every week or two. That interval is a rule of thumb; it has not been measured as the best interval. Two methods are used in the reflection. Marginal Gains improves the current limiter by a small named amount and stacks those improvements. Kolbs Experiential Cycle is the pass done after an attempt: what happened, how it felt and why, what rule that suggests, and what changes next time.

A method is learned.
It is used on real work.
The attempt is reflected on.
The barriers that showed up are named and ranked by how much each caps the rest.
The top one is the next piece of work.

A self-check in a plateau week that only asks the easy question is the coverage count again in a different form. Searching for a limiter can also become a way to put off the repetitions when those repetitions are the limiter. When the limiter has been named correctly, improvement shows up in parts of the process that were not touched. If two rounds of searching for the limiter change nothing about the next session, the limiter is the one already named, and the next move is the work already named. No trial shows that a reflection loop run every two weeks finds the true constraint.

When a skill is good enough

A skill is good enough when it is no longer the rate limiter. Techniques can be improved endlessly, so there is no absolute standard to stop at. The count that decides what to improve is the week-later reconstruction together with the name of what is currently capping the rest. Enough is decided by comparison. No standard decides it.

How to practice this

1. Name the highest and the lowest level of performance the assessment asks for. About a week after studying, close the book and reconstruct the material at each of those levels. Notice what you can still produce at each level against what will be demanded.
2. Write down what you could not produce at the higher level. Do not turn the result into a percentage, a ratio, or an average. Notice that the check is slower and less pleasant than rereading; that is its cost.
3. Next time you reread something until it feels fluent, test it closed-book a week later. Notice the gap between how known it felt and what you can produce.
4. Every week or two, review the last attempt. Ask what happened, how it felt and why, what rule that suggests, and what changes next time. Name the barriers that showed up and rank them by how much each caps the rest.
5. Improve the top barrier by a small named amount, even if unsure it is the real limiter. Notice whether parts of the process you did not touch also improve. That improvement is the sign the limiter was named correctly.
6. If two rounds of this search change nothing about the next session, stop searching. The limiter is the one already named, and the next move is the work already named. Treat a skill as good enough once it is no longer the limiter.

Related pages

• Marginal Gains: the practice that the week-later check and the limiter name are the measurement for.
• Metacognition The Control Layer: what to notice while measuring. The week-later check and the limiter name are what to measure.

Sources

• Goodhart 1975; Campbell 1979: metrics that become targets.
• Bjork, Dunlosky & Kornell 2013; Dunlosky & Lipko 2007; Koriat 1997: fluency and coverage as poorly calibrated cues to what has been learned.
• Karpicke, Butler & Roediger 2009: the preference for rereading.
• Anderson & Krathwohl 2001; Biggs & Collis 1982: the two published taxonomies the levels are built from.
• Craik & Lockhart 1972; Craik & Tulving 1975: depth of processing and retention.
• Cepeda et al. 2006; Rawson & Dunlosky 2011: spacing, retrieval practice, and the relearning cost.
• Goldratt 1984: throughput set by the constraint.