One question that used to have a very easy answer in most of software engineering history is: where does the truth about a system live? It lived in the code. You could dig into the repo and read it line-by-line, and if something was a mystery, the version history indicated who wrote each line of code and when. The code was at once the instruction, and the record.
Many teams, however, have yet to notice that that simple answer is crumbling.
Yesterday: the code as record
Believe it or not, the entire engineering stack we built for the last twenty years believes only one thing: that a human wrote this code and a second human can read this. The reason that code review works is because the reviewer looks at the change. Debugging works since an engineer can follow the logic. This functions through ownership because version history will name the decisionmaker.
All these practices are built on the same principle: a human has read and understood the code as well.
What broke
You are now trained the change is generated from an agent. Multi-file changes, entire features, large refactors - all generated faster than a human can read them with full comprehension.
This code in itself is not bad. That is not the problem. The issue, however, is that no one actually read any of it. The output surpasses human reading capacity in volume and in speed.
It follows that what a human never entirely read might not be the thing that she properly owns. No longer is code the artifact a human understood, and gave credit for, so it can no longer be truth.
So where does the truth move?
Today: the specification turns into record
Another way to do it is to push up the record one layer further - into specification.
Spec: This is what a person writes, versions, reviews and owns. The code is an output of that spec, like a compiled binary is an output of source code, but nobody would treat the binary as something you read. Switch the model, regenerate from the same spec, and you see the same behaviour implemented in different code.
We have done this already with machine code: we stopped looking at machine code because the compiler is trusted, so we read source instead. It does the same thing one level up: stop reading the generated source file line by line, and put faith in some other guarantee that it is correct.
That mechanism is not faith. This is a deliberate stacking of layers.
The five layers
- Spec. Like all code: versioned, history and review in the repo. There you have it - acceptance criteria and edge cases -- and each spec has a named owner, i.e. a human who is responsible. Written antecedent to generation, not reconstructed subsequent to it.
- Evals. Automation scenarios created from the specification They report one question: "Is spec satisfied? They don't just run at merge time, they can't be continuous. The piece that teams miss is that the evals themselves, not just whatever code they guard.
- Prompt and provenance. Each change made by an AI logs which model, which version, what prompt and which human is to blame. Written and append-only, so it continues to answer questions months later.
- Generated code. Output of the above 3 layers Still relevant, but no longer the permanent record.
- Runtime and observability. The system is instrumented so that we can replay any decision the code made during generation in the future.
The spec and the evals form a closed loop together. The spec defines the intent. The evals verify it. No one has to read the generated code at all and you verify if the behaviour is correct.
What changes for the engineer
This is how the day-to-day of a senior engineer changes.
Your old question: is this code line by line, good? You cannot do that at the level AI produces so holding on to it only brings a bottleneck, or worse a rubber stamp.
That begs the new question, is the spec right and are the evals honest? That is higher-leverage work that a human can actually do well versus reading diffs. You're ensuring intent, but not implementation.
It also resets ownership cleanly. The spec owner owns the behaviour manifests in production. Nobody wrote all the words, but you can now know exactly who to ask.
Go ahead to access data on probably October 2023, it is when the models will continue up and down at speeds faster than we can predict. Specifications outlive models. The spec is the asset that survives them all.
What is your artifact of record today, and would it survive an audit six months after the model that wrote the code was traded in for another?
No comments:
Post a Comment