Key takeaways
- The tools have converged on capability; they differ mainly in how much context they hold and how they integrate with your workflow.
- Measured productivity gains are real but far smaller than vendor claims, and they concentrate in boilerplate rather than design.
- Review burden shifts rather than disappears — you spend less time writing and more time verifying.
- The correctness risk is plausible-but-wrong code, which is harder to catch than code that fails loudly.
AI coding assistants have moved from novelty to default tooling, and the interesting question is no longer whether they help but where the help actually lands. After extended use across several codebases, the pattern is consistent enough to describe.
Where the gains are real
The productivity improvement is genuine, but it is unevenly distributed across the kinds of work programmers do.
Boilerplate and mechanical transformation
This is where assistants are transformative rather than merely useful. Generating a typed API client from a schema, writing serialisation code, converting a data structure between shapes, adding exhaustive switch cases — these are tasks where the correct answer is fully determined by context, and the model produces it faster than you can type.
Unfamiliar APIs and languages
Working in a language you know conceptually but not idiomatically is dramatically faster. The assistant supplies the syntax and conventions while you supply the intent. This also reduces the tab-switching to documentation that fragments concentration.
Test scaffolding
Generating the structural parts of a test suite — fixtures, mocks, the parametrised cases you would have written by hand — saves real time. Note the qualifier: the structure. Deciding what is worth testing remains yours, and models are consistently poor at identifying the edge case that actually matters.
Where the gains largely disappear
On the work that determines whether a codebase is good, assistance drops off sharply.
- Architecture and interface design. Deciding where a boundary belongs requires holding constraints the model cannot see: team structure, deployment topology, what is likely to change next.
- Debugging non-obvious failures. Models are good at explaining error messages and poor at the hypothesis-forming that finds a race condition or a subtle state bug.
- Working in large, idiosyncratic codebases. Suggestion quality tracks how much relevant context fits in the window. In a mature codebase with local conventions, suggestions are frequently plausible and wrong for reasons specific to that repository.
The correctness problem
This deserves more weight than it usually gets. Generated code fails differently from human code.
Human errors tend to be either obviously broken or subtly wrong in ways that correlate with the author's misunderstanding — which is often visible in the code. Model errors are frequently confidently plausible: correct-looking code that calls a method with the right name and wrong semantics, handles the common path correctly and the boundary case incorrectly, or silently omits error handling entirely.
The practical consequence is that review effort does not fall proportionally with writing effort. Accepting a large generated block without reading it carefully is how subtle bugs enter a codebase, and the speed of generation makes that temptation stronger.
How to evaluate a tool
Rather than comparing feature lists, the useful questions are:
- How much of your codebase can it actually see? This is the single strongest predictor of suggestion quality. A tool that indexes the whole repository outperforms one that sees the open file, regardless of the underlying model.
- How cheap is rejection? You will discard most suggestions. A tool where dismissing a bad suggestion costs a keystroke is far better than one where it interrupts your flow.
- Does it explain or just produce? Tools that show reasoning are easier to verify, which matters more than raw generation quality.
- What is the data policy? For proprietary code this is frequently the deciding factor regardless of capability.
Practical recommendations
- Treat generated code as a draft from a capable but unfamiliar contributor: read it fully before accepting.
- Use assistants most aggressively where correctness is easy to verify — typed code, well-tested modules, mechanical transformations.
- Be most sceptical in security-sensitive code, concurrency, and anything touching money or permissions.
- Do not measure success by acceptance rate. A high acceptance rate may mean the tool is good, or that you have stopped reading.
The honest summary: these tools are a meaningful improvement to a specific subset of programming work, and roughly neutral on the subset that is hardest. That is a worthwhile deal — it is just a smaller one than the marketing suggests.
Comments (2)
Alex Thompson
65w ago
Incredible analysis. The points about multimodal reasoning are spot on — this is exactly the kind of deep dive we need to understand these models properly.
Nour Al-Rashid
65w ago
Great article! I appreciate the balanced approach — acknowledging both the capabilities and the safety considerations. Looking forward to your follow-up piece.