What If Your Git History Could Write Your Test Suite?

ting.pm is an AI coding tool built around one idea: the ticket is the unit of delivery. You describe what you want; ting turns that into a fuller brief with acceptance criteria, writes the tests, builds until they pass, reviews, iterates, and commits. All test-first. Nothing commits until it’s green. I wrote a bit about it here.

I recently had a slightly perverse idea. What if it could run the other way? Take a commit that already exists and work out the ticket behind it: the intent, reverse-engineered from the result.

Surely it couldn’t work.

It turns out that it can. Point ting at a codebase it has never seen and it can indeed infer a commit as a ticket. Taking that a step further, it can also work back through the history creating a ticket for every commit - and because a ticket is exactly what ting already builds tests from, a ticket per commit lets us build a test suite for the whole repo. So give ting a branch to look at and it creates a behavioural test suite based on actual commit history. I'm calling it Codewalk.

Conceptually elegant, but of course many commits are refactors, dead ends, partial work, broken commits, and accidental changes. Testing every commit as it was written gives you a suite that fails. Write a ticket and tests for every line of every commit in the history, and most of them assert behaviour that's since been rewritten or ripped out, so they go red against the code that's actually there. Once failures stop meaning anything, people stop looking.

So Codewalk doesn't test the history as it occurred; it tests what's left of it. For each commit it works out how much of that change is still standing in the code today, and keeps only the part that survived. What comes out is a suite that describes how the software actually behaves now, with each test tied back to the commit that intended it.

Codewalk identifies commits and then infers tickets from them

Obviously the first thing I did was point Codewalk at ting's own repo. It read back over ting's commit history and code, wrote a ticket for every commit whose work still stands, and generated the tests from those - I ran it across a whole branch and every one came back green, none written or described by me. And not toy tests: that dispatching a ticket kicks off the right run with the right inputs, that a ticket's tests get stored where the rest of the system goes looking for them, that the deployment exposes the values it needs rather than hard-coding them. Here we have the benefit of intimate knowledge of the codebase and its intent, and the ticket descriptions and tests described it correctly. That's not proof it works on a repo we've never seen - ting is its home turf.

From there it behaves like any test suite. It lives in Suites, another new ting feature, and from there you can run it against any branch or commit you like - a feature branch, say, or a change you're about to merge - to check the behaviour still holds.

Suites running tests across tickets

So what Codewalk does here is essentially a form of reverse engineering. You point it at software, it works out what the software is for, and captures that as something executable.

If Codewalk can pull a clean, executable account of what software is meant to do, free of the code it came from, that also gives us two parts of a puzzle: Codewalk turns code into intent, and ting.pm already turns intent into working code. Point one at the other and you're not regenerating the tests. You're regenerating the software.

I have to try it.

ting isn't out yet. Codewalk is for the big, years-deep codebase - the kind no one person holds in their head and nobody ever got round to testing. If you run engineering at a place like that and you're up for kicking the tyres, I'm looking for a team to point it at a real repo and tell me where it breaks. If that's you, I'd like to hear from you.