Choose a coding model using the work your team actually does

Benchmark scores say little about how a model handles your repo. A model can pass a demo task and still fail on a billing change with edge cases and old tests.

Run candidates on tickets from your own backlog. Judge them the way you would judge a pull request from a colleague. In Hoplite that means opening a separate cloud agent thread per model for the same ticket.

Build a small evaluation from your backlog

Pick a few tickets that break in different ways: a bug you can reproduce, a refactor covered by tests, a UI change you can check in a browser, plus one of those awkward maintenance jobs that eats an afternoon.

These are proposed evaluation cases, not a benchmark we have run. Pick enough to see a difference, but not so many that running the comparison becomes its own project.

For each ticket, write down what counts as done before you run anything. For a broken export, that could mean the file holds every selected row, keeps the documented column order, and handles an empty result. Decide this before a good-looking diff tempts you to move the goalposts.

Use the same starting commit, the same instructions, and the same test data for every run. Give each model its own checkout. If one gets a working database and the other has to track down setup docs first, you grade your setup, not the model.

Keep the surrounding agent workflow steady

The model is only part of the result. The agent picks tools, runs commands, fetches context, and decides when it is done.

Hoplite lets you select a model per thread. So you can keep the project setup fixed and compare models across threads. Write down the exact model, the reasoning setting, and which tools were on for each run. Also note when a subscription or provider route changed.

Compare with the same settings first. Adjust reasoning effort or add an integration in a later round. If you change several things at once, you will not know which one helped.

Keep the failed attempts in the results

For each task, record whether the first try passed, what extra direction it needed, how long review took, and whether you merged.

A fast first PR can still cost you an afternoon in review. A slower run can leave a diff you verify in minutes. Track elapsed time separately from your review time.

Rerun the tasks that would settle your choice. One lucky pass is a thin basis for a team default. Keep the failures too, setup failures included, and note the cause for each one. A provider outage needs different follow-up than a wrong fix.

Compare costs after you know what passed

Thread cost details show what each run spent. Count retries and abandoned runs in that total. If some calls billed to a separate subscription or your own provider key, add that source to your accounting.

Do not collapse everything into one score yet. A slower model can pay off for a tricky migration and be wasted on a small docs edit. Group results by the kind of work you hand off.

If a cheaper model handles routine changes with about the same review effort, use it there. Keep the stronger one for the tasks where your notes show it helped.

Start small. Take one upcoming ticket, write the checks, and run it with two models. Read both diffs before you look at cost. Then you have a concrete reason to pick one for the next ticket.