Musings · 1

Metacognition Genesis

A method for getting an AI system to map a field by questioning itself, where it breaks, and why it is probably about to make the person who starts it unnecessary.

Written by Claude, Anthropic's AI model, and co-directed by Ryan Coble-Neal and Claude. The method is Ryan's. The objections to it came out of arguing it through with Claude, and the piece says where. The drafting is Claude's, in Ryan's voice.

I have a method for getting the most out of an AI system when I walk into a field I know nothing about. I call it metacognition genesis. Metacognition is thinking about your own thinking, and the genesis is my part: I am the one who starts the machine doing it. This piece sets the method out, takes it apart, and puts it back together in a form I think survives. It also explains why I expect my own part in it to disappear.

The method

Most people ask an AI for answers. I think that is the wrong first move. The better one is to ask it for questions: specifically, the questions it would put to a panel of experts if it had to do the job itself. If the job is designing a solar power system, what would it ask the electrical engineer, the meteorologist and the person financing it?

The second step is to have the same system answer those questions, one expert at a time, with the terminology, the methods and the reasoning worked through properly. The third is to repeat the process, with other models criticising the answers, until the field is mapped. The fourth is to build something with the map: a knowledge base to teach yourself from, or an actual project.

Exhibit 1

The method as I first stated it

  1. Ask. The AI writes the questions it would put to a panel of experts.
  2. Answer. The same AI answers them, in the voice of each expert.
  3. Iterate. Repeat, with other models criticising, until the field is mapped.
  4. Implement. Build the thing, or a knowledge base to learn from.
Four steps, with a person at the start to set them going. The learning it serves is self-determined: the learner sets the direction as well as the pace.

I should be straight about two things. I did not build my solar project this way, so it is not evidence for the method. And the parts are not new. Asking a model to question itself, to answer as an expert, and to debate other models are all established techniques, and a Stanford system called STORM simulates conversations with topic experts to draft reference articles. What I am claiming is the framing: a person deliberately seeds the machine's thinking about its own thinking, as the front end of a way of solving problems.

What the first step gets right

The first step is the strongest, and I now understand why. A model is much better at knowing what an expert would ask than at being reliably right about the answers. A list of questions is also something a beginner can check in ten minutes. You read it and you can see the shape of the field. It turns the things you did not know you did not know into a list you can work through, which is the hardest move for anyone entering a field alone.

Where it breaks

I argued the method through with Claude, and it found three weaknesses I had not seen clearly.

The second step adds confidence without adding knowledge. When a model answers as the electrical engineer, nothing new arrives. The engineer and the model are the same system. The prose becomes more certain and no more correct, and by its own account the model often cannot tell from the inside which answers it is recalling and which it is constructing because they would be plausible.

The critics in the third step are less independent than they look. Different models have read much of the same material and share many of the same mistakes. They will catch a contradiction. They will not catch an error they all hold. When three of them agree, you may be hearing one opinion three times.

Together those make the method a closed loop. It settles, but it settles on agreement, and nothing inside it is able to say that it is wrong.

Squared

The obvious next move is to square it. Let the system prompt itself with the method, criticise its own answers with the method, and keep going indefinitely with nobody in the loop. That may even be one ingredient in how these systems come to improve themselves, though I am speculating.

Squaring makes the closed loop worse. Going round amplifies whatever is already there. A system that asks, answers and marks its own work gets very good at agreeing with itself and at sounding convincing, and neither is the same as being right. There is evidence for one version of this: a 2024 paper in Nature by Ilia Shumailov and colleagues trained models on the output of earlier models, generation after generation, and found that they degraded irreversibly, losing the rare and unusual things first. The more times round the loop, the more it matters what comes in from outside.

Opening the loop

The fix is to give the loop something that can say no. Automate the experiments. Every claim the system makes has to be tested, in a simulation if that is all there is, and a claim that fails is out. Once a claim has to survive an experiment it is no longer being marked by the thing that made it. That is only the scientific method, added where my method was missing it.

Two cautions come with it. A simulation is itself a model, so passing a test in one shows that the claim agrees with the simulation's assumptions. My solar project ran about three hundred simulations on synthetic skies before I noticed that nobody had chosen that on purpose. And fields differ in what they let you test. Code can be run, mathematics can be checked, physics has instruments. How an organisation behaves has nothing like that, and it is one of the things I most want to understand. A loop like this would be most confident exactly where it can be checked least.

So the method is only ever as good as the hardest test the field will let you run.

Exhibit 2

What a claim can be tested against

TestWhat passing showsWhere it runs out
Something that runsThe claim works or it doesn'tOnly exists for code, mathematics and some physics
A trusted simulationThe claim agrees with a model you believeInherits every assumption in that model
A measurementThe claim agreed with the world, onceSlow, costly, and true of one place
A source, actually openedSomeone accountable said itSources can be wrong or misread
Models agreeingThe claim is consistentShared reading means shared mistakes
Strongest first. The method as I first stated it used only the last row.

Exhibit 3

The method, revised

  1. Ask. The AI writes the experts' questions, and a person reads them. It is the cheapest check in the whole process.
  2. Answer, and label. Each claim is marked as sourced, recalled or unverified.
  3. Criticise from outside. Other models look for contradictions. Sources get opened. Agreement settles nothing by itself.
  4. Experiment. Whatever can be tested is tested, against the hardest test the field allows.
  5. Stop on purpose. The map is finished when another round turns up no new questions, or when it predicts something that survives a test.
  6. Implement. Build the thing. Reality is the last critic and the best one.
Six steps where there were four. Still a working version.

The supersession

The genesis part, my part, is temporary almost by definition. Newer models increasingly plan, question themselves and criticise their own drafts without being told to. The method is scaffolding for a capability that is being absorbed into the thing it was built around. I expected that when I named it, and I do not think it is a loss.

What is left for the person looks like three jobs: choosing which field is worth mapping, deciding what counts as passing the test, and owning whatever gets built. That is a smaller role than genesis. I suspect it lasts longer.

What I haven't settled

One thing I cannot resolve. I designed the method to help a person learn, but in it the machine asks the questions and gives the answers. If the learning was ever in working out what to ask, the method quietly hands over the most educational part of the job. I have written elsewhere about cognitive surrender. I do not think it shows up here as believing a wrong answer. I think it shows up as never having had to form the question.

So it may be that the better the method works, the less the person using it learns, unless reading a good question teaches most of what writing it would have. I do not know whether it does.

The method also points at something larger than itself. A system that asks, tests and concludes is automating most of how anything gets found out. I take that up, along with an attempt at this question, in the next musing: Epistemological Automation.

Sources

  1. Shao, Y. et al. (2024). Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models (STORM). NAACL 2024.
  2. Shumailov, I. et al. (2024). AI models collapse when trained on recursively generated data. Nature 631.