Reading Room

You lean on Claude Code for real engineering work, so this tells you exactly where to keep a human checking the AI's patches instead of trusting them.

Researchers at Off-by-1 Labs tested how well today's best AI models fix known security bugs in code, with the paper published through 1Password. The models produced a genuinely correct fix only about one time in four. The rest of the time they did one of three things: left the bug half fixed, made unrelated changes to code nobody asked them to touch, or introduced a brand new security hole while claiming to fix the original one.

The researchers also tested the obvious fallback: have a human review the AI's proposed patch before it ships. Their finding was that this does not really solve the problem either. Properly checking whether an AI-written patch is safe takes so much careful thought that it comes close to being as much work as writing the patch yourself. Their conclusion: for security fixes specifically, write the patch by hand rather than asking an AI to draft one and checking its work afterward.

The full paper is said to include practical advice on what kind of context helps a model do better at this particular task, though those specifics were not part of what got saved here.

This maps directly onto the loop patterns you already use and write about, so it's worth sharpening your thinking against someone else's real production experience.

The framing from this interview: 2023 was the year of writing good prompts, 2024 was about giving the AI the right background material, 2025 was about the "harness" (the surrounding software that lets an AI actually take actions), and the claim is that 2026 is becoming the year of "loop engineering," building small automated cycles that run themselves. The guest is Tyler Folkman, Chief AI and Product Officer at JobNimbus, a company that just closed Utah's largest-ever Series B (a later-stage startup funding round).

Folkman's definition of a real loop, as opposed to just telling an agent to run forever: it fetches its own input, does the work, has a gate (a checkpoint that decides if the result is good enough to continue), writes out a named result, then goes back and fetches its next input on its own.

His most useful, and most counterintuitive, advice: write your first loop by hand, not with AI. His reasoning: he compares an AI-written loop to an e-bike, once the motor does the pedaling, you stop pedaling yourself, meaning you stop doing the thinking that used to sharpen your own judgment. His method: open an empty file, write a short plain-language instruction by hand, test it, improve it, and only then hand the AI a one-line command to wrap it into a loop, then close the loop by hand once to see what it's actually doing and ask it how to improve itself for next time.

Folkman recommends product managers build at least five loops. Customer outreach is the one he personally gets the most value from, since the real cost was never writing the message, it was finding time to figure out who to contact. A thinking-partner loop does the opposite of most AI tools: instead of producing work, it pushes back on the person's own thinking rather than doing it for them. He is blunt that skipping this and forwarding AI output straight up the chain "is not doing product work."

A divergent-prototypes loop builds several product variants from one thin prompt, in his example a crew-scheduling tool for a contractor, then having a simulated customer review the options before any real person sees them. Tyler runs several of these loops side by side using a terminal tool called herdr, which launches multiple AI coding assistants at once whenever a prompt includes the word "agent."

Compressing an AI model down to a smaller size, called quantization, is not one standard thing: three different quantization methods on the same model weights can swing code-writing accuracy by close to 10 points and speed by 20 to 30 percent on identical hardware. The better methods, like AWQ, protect the small set of values that carry the most meaning; plain rounding does not, so it's worth checking the method, not just the file size, before running or evaluating a local model.