Fun, Not Fear: Rebuilding University Assessment for the Machine-Learning Era, & Going Beyond to Modes of Research-Analytical Practice
I think Paul Musgrave has it correct here: face-to-face check-ins are the way to turn MAMLMs from a crutch into an intellectual force multiplier.. Then there is the next, harder step: What is the best way to use our new information-age MAMLM to get our students able to enrich their lives and their bank accounts by becoming wise in their ability to utilize the best that has been thought and known?
AI is here; the question is whether we let it hollow out cognition or use it to strengthen it. The answer, I think, starts with outside-the-class essays and inside-the-room interviews. It continues with hard thoughts about what use to make of these technologies, for MAMLMs are superb survey accelerants and terrible oracles. That combination is, potentially, pedagogically golden.
This is, I think, 100% correct, at least for those professors who want to get real feedback with respect to what their students are learning, or who simply want the grades they give to not be grossly unfair and to do something to motivate students to study:
Paul Musgrave: How to Assess Students in the AI Age <https://musgrave.substack.com/p/face-to-face-university-examinations>: ‘Bringing back the most ancient form of scholarly assessment…. In-person examinations are probably the best route to stress-testing students’ mastery of the work they submit…
Then of the four options for who to examine face-to-face he offers, I vastly prefer his option 4:
[Have the student] write a project or a step of a project and then have an interview or check-in about what is going on with the work…
These students are going to have modern advanced machine learning models where their descendants at their elbows for their entire lives going forward. They need to get good at using them productively. Teaching students to perform intellectual exercises in a way that assumes these things away is not, in the long run, going to do much to help anybody. So: write the essay outside of the classroom, submit it, and then come in to talk about it.
Paul goes on:
Face-to-face exams involve an enormous amount of upfront investment… [and] scheduling exams outside of class time….
There is a nice advantage, however: once you have administered the face-to-face exam and written your notes…you’re done….
There should be a rubric… breaking out each learning objective to be assessed so that it can be quickly measured. There should also be room for notes…. These should be filled out immediately after the exam and saved quickly…
This means, at least as I see it, half-hour blocks: ten minutes reminding yourself what the student’s outside-the-class work was about, a three-minute opening presentation, your first question and their answer, ten minutes of back-and-forth, a thank-you and five minutes making your notes, and then on to the next student.
Now this is a substantial speedup—an extra thirty hours of professor and TA labor devoted to assessment. Then again, having students give oral presentations and answer questions on their work has a lot of value as a teaching technique. And there is some reduction in time spent undertaking other forms of grading.
But it is not clear that this speedup is not something we deserve to suffer. And, anyway, we are pretty good at kicking and crying with respect to defending our perquisites and lifestyle—feel sorry for underpaid, overworked lecturers in the age of MAMLMs. But even for them, in the university wants performance, it should budget to pay for it.
Paul Musgrave still sounds somewhat defensive and pessimistic in his piece. So let me lean in and be much more optimistic.
Start with Chad Orzel’s <https://chadorzel.substack.com/p/learning-stuff-is-supposed-to-be> reminder: learning is supposed to be fun. If that sounds quaint against the steady drumbeat of “Crisis in Academia!!!!”—the handwringing over status collapse and cheating panics—then perhaps we need to step back and recollect what universities are for. For five millennia, since clay tablets and cuneiform, higher education has had one overwhelming purpose: to train front-end nodes to the East African Plains Ape Anthology Super-Intelligence—our real ASI. We teach people to plug into humanity’s collective brain. We teach people to draw on its accumulated knowledge. We teach people to remix it for their situation. We teach people to use it as a springboard for their own incremental analyses. We teach them to then add their own insights back to the store. And we teach them to then communicate those insights so others can act in the world.
That training is stable across technologies. It is still, as I said, the seven academic labors:
survey a subject;
identify the live issues;
hone a key question;
research the question;
analyze evidence to obtain an answer;
store that answer in useful, durable form; and
persuade others that your answer fits reality.
If you are doing lower and higher education right, you are teaching this process, modeling it, and goading students to practice it. Every new layer of information technology—papyrus, codex, press, arXiv, the internet—changes how we engage with texts and data, but not the core pedagogy. MAMLMs—modern advanced machine-learning models—are simply the latest layer. They are very big-data, high-dimensional, flexible-function engines. They do two prominent things:
serve as natural-language front-ends to un/structured data, and
produce stochastic prose (and code) interpolations—slop machines at their worst, copilot force multipliers at their best.
The lecture survived Gutenberg. The seminar survived MOOCs. The core task—training front-end nodes of EAPANASI—survives MAMLMs.
Why is this going to be fun? Because we get to redesign the pitch and the practice around what actually delights, motivates, and builds capability. Orzel’s point is simple and powerful: much of higher education used to be sold as fun—freedom to choose classes, stay up absurd hours pursuing ideas and projects, and play with lasers (or datasets) in a lab. That joy made it easier to power through the inevitable bureaucratic sludge and credential box-checking. If our public posture has turned grim—caught between “skills-for-jobs” instrumentalism and “last-bulwark-against-fascism” moralism—we can fix that, by re-centering the seven labors and integrating the new tools as intellectual force multipliers rather than crutches, as we together explore new tools for amplifying human thought
What does that look like in the classroom?
What I think I am going to be doing next semester on Day One: I am going to tell the story of the medieval university, of the trivium and quadrivium—how to think, write, speak; arithmetic, geometry, harmony, astronomy—and how they were always about enabling rich lives and rich livings as front-ends to the ASI as it stood in the middle ages. Then I will say, “As then, so now”, for our task is to teach students to do (1) through (7) with today’s tools of thought, definitely including MAMLMs. Make explicit the difference between using a tool to accelerate cognition and outsourcing cognition. The former builds intellectual muscle; the latter atrophies it.
I think this will be hard, but the right kind of hard—the kind of hard that makes teaching worth doing. And fun.
Fun, because MAMLMs let you reallocate attention to higher-order intellectual play. Consider step (1), surveying a subject. A good model can generate competent, if bland, maps of the terrain. Use that as a baseline, then push students to find what the baseline misses—conflicts in sources, under-explored subfields, data series with holes, or historiographic rabbit holes where the debate changed meaning but the keywords did not.
On to step (2), identifying live issues: have them prompt, critique, and iterate until the “live issues” are truly live—specific enough to be tractable, rich enough to be consequential.
Step (3), honing the key question: MAMLMs can produce long lists of candidate questions; the real work is pruning, sharpening, and linking the question to available data and methods.
Step (4), research: models are superb front-ends to archives, APIs, and corpora. They are terrible truth-oracles. That is pedagogically golden. Students must learn provenance, triangulation, replication, and error-detection—skills that are more valuable than ever precisely because the front-end is now more powerful and more fallible.
Step (5), analysis: ask students to build small analytic scaffolds—back-of-the-envelope calculations, toy models, code snippets—and then use the model to stress-test assumptions. The delight is watching them discover that changing one assumption flips their result, and then asking why.
Step (6), storage: the durable forms have shifted—papers, databases, notebooks, reproducible repos. MAMLMs can help them outline, summarize, tag, and index. But the student must choose the representation that is reusable by the next front-end node. Teach them to produce artifacts that other humans can pick up and use: clean datasets with documentation, notebooks with narrative and tests, papers with explicit claims and boundaries.
Step (7), persuasion: here models can help generate drafts and check tone. Yet the heart of persuasion remains human: audience, structure, evidence, and ethos. The fun is having students practice rhetorical craft and then use models as sparring partners to anticipate objections and strengthen argument.
The real joy will come, I hope, from the meta-pivot: students will, I hope, come to see education as play with intellectual force multipliers. They will learn the discipline to work without the tool when needed, and the wisdom to use it when it accelerates them. They learn that writing is thinking, but so are prompting, debugging, and oral explanation.
Will this produce a “degraded education”? Only if you mistake tool use for tool worship. Only if we cannot re-sell higher education to the young not as grim credentialing or joyless bulwark politics but as a dream worth living: the freedom to think, build, and persuade with better tools. If we do that, the coming of MAMLMs is not doom but delight.
It is, in truth, still a pretty sweet gig.
If reading this gets you Value Above Replacement, then become a free subscriber to this newsletter. And forward it! And if your VAR from this newsletter is in the three digits or more each year, please become a paid subscriber! I am trying to make you readers—and myself—smarter. Please tell me if I succeed, or how I fail…