Cost per successful task: how to price and monitor AI agents

Cost per run answers one question: what did this attempt cost? It says nothing about whether the attempt worked.

Agents retry tools and call the model again with the results. Often a person checks the output. And sometimes, after all of that, the task still fails and somebody does it by hand. All of it was paid for. If you only compare the price of a run, the option that fails most often can look like the bargain.

Cost per successful task = cost per run / success rate.

If a run costs EUR 0.40 and 80% of runs succeed, each success costs EUR 0.50. You paid for the failures too.

There's a second way to get the same number, and it's the one I'd put in a query: add up everything you spent on a task type in a period, then divide by the number of successful tasks in that period. Same result, and nobody has to agree on what an average run is.

Three terms, so we mean the same thing. A task is one piece of work with a clear end, such as "draft a reminder for this overdue invoice". A run is one attempt at it, including every model call, tool call, retry and review inside that attempt. The success rate is successful runs divided by all runs, for one task type over one period.

Keep task types apart. A payment reminder and a reply to a disputed invoice are different jobs, with different costs and different odds. Average them together and the number describes neither.

Everything one attempt uses. If you leave a part out, the cheap option wins for the wrong reason.

Every model call. A single run is rarely a single call. There's a planning call, then a call after each tool result, sometimes a final check. Log each one and sum them per run.

Retries. A retry inside a run is more model calls and more tool calls. Count them where they happen. Folded into an average, they hide the reason a cheap model turns out expensive.

Tool costs. Paid search APIs, document parsing, anything billed per call. If a tool is free to call, record its duration anyway. Slow tools are usually the first sign of a loop.

Human review minutes. If a person reads the draft, checks the amount and presses send, that time is part of the cost. Pick one internal rate per hour and use it everywhere. It doesn't need to be exact. It needs to be the same every time.

Model prices change, so read them from your provider's pricing page and keep them in one settings file. Then a price change is a one-line edit.

The formula has a blind spot. It counts what a failed run cost you to produce. It doesn't count the damage when a failure goes unnoticed, like a reminder with the wrong amount reaching a customer. That's a job for evals and human checkpoints, and it's the reason I'd never lower review time just to improve this number.

The success rate is only as good as the definition behind it. "The output looked fine" isn't a definition. You want something a script can verify.

Name what must be true when the work is done: which rows exist, which fields hold which values, which message is waiting in the queue. Then name what must not have happened. No duplicate. No change to another customer's data.

For a reminder task, success could read: one draft exists for the right invoice, the amount and due date match the invoice, there's no second draft, and the user sent it without changing any facts.

Write the definition down before you start measuring. If you change it later, start a new measurement period, because the old numbers and the new ones no longer mean the same thing. I go through how to test this properly in evals that check the end state.

Everything in this section is hypothetical. The prices, the rates and the review times are made up to show the arithmetic. They aren't measurements from any product or client.

Picture an invoicing tool in the style of QuickBilling with one agent task: draft a payment reminder that the user sends without edits.

Model A, the careful one. Model calls per run, retries included: EUR 0.12. Tool calls: EUR 0.03. Review: 30 seconds at an internal rate of EUR 30 an hour, so EUR 0.25. Cost per run: EUR 0.40. Say 80 of 100 runs meet the definition of success. Cost per successful task: 0.40 / 0.80 = EUR 0.50.

Model B, the cheap one. Model calls per run: EUR 0.02. Tools: EUR 0.03. Review is still 30 seconds, EUR 0.25, because a person has to read the draft whichever model wrote it. Cost per run: EUR 0.30. But its drafts need fixing more often, and only 50 of 100 runs succeed. Cost per successful task: 0.30 / 0.50 = EUR 0.60.

So the model bill fell by 83% and the cost of a finished reminder went up by 20%.

Two things to take from this. First, review time is the largest line for both models. While someone still reads every draft, shaving tokens does little. Second, the comparison only works because both models were judged against the same definition of success. Change the definition between the two and the numbers mean nothing.

None of this can be calculated unless the data exists. The rule I follow: every model call gets a price and a task id at the moment it happens.

A task id is one identifier shared by everything that belongs to one attempt, so the model calls, tool calls, retries and the review all carry it. Without it, costs are scattered across logs and nobody can say what one reminder cost.

Put it all in one table. Then cost per run is a sum grouped by task id, success rate is a count grouped by task type, and dividing one by the other gives you the number.

You don't have to build the logging yourself. Tracing tools do most of it. Langfuse, for example, tracks usage and cost on each model call, either from values you send or calculated from the model's price definition. What you still have to add is the task id, the review minutes and the success flag, because only your system knows those.

Here's what I'd record:

• Per model call: task id, task type, model name, input and output tokens, price in EUR.
• Per tool call: task id, tool name, success or error, duration, and cost if the tool is billed.
• Per retry: task id and the reason for the retry.
• Per review: task id, minutes spent, and whether the reviewer accepted, edited or rejected.
• Per task: the final verdict against your definition of success.

A budget cap stops a run when its spending passes a limit. Without one, an agent stuck in a tool loop keeps paying for calls until someone notices.

I'd set two per task type.

The soft cap flags a run as unusual. Set it a bit above the normal cost of a run. When a run crosses it, log a warning with the task id and let the run continue. Read the warnings once a week. They tend to point at a confusing input, a slow tool or a prompt that got longer.

The hard cap ends the run. Set it at a multiple of the normal cost. When a run crosses it, stop the agent, save what it has, and mark the task failed. Don't retry automatically. A run that needed several times the usual budget will probably need it again, and a person should look first.

Start with generous caps and tighten them once you've seen a few weeks of real runs. If you want this built into the platform, it's part of what I do under cost and safety rails.

Costs creep. Documents get longer, a model version changes, a new kind of input causes more retries. Each change is small and no single run looks wrong.

So alert on the trend. Compare cost per successful task for the last seven days with the seven days before, per task type, using the same definition of success. Decide on a threshold, write it down, and leave it alone for a month or two so you learn what ordinary variation looks like.

Put the five most expensive task ids of the week into the alert itself, so whoever picks it up can open those traces straight away. And if the drift started right after a prompt change, roll the prompt back first and investigate second.

When the alert fires, check four things in this order:

• Did the success rate fall?
• Did retries per run go up?
• Did review minutes go up?
• Did a unit price change?

Sooner or later a cheaper model appears and someone asks whether you should move. This number gives you a way to answer without arguing.

Run both models side by side on the same task type. Send part of the real traffic to each, with the same success checks and the same review rules. Let it run long enough to cover your normal variety of inputs. For a low-volume task that might be a couple of weeks.

Then compare cost per run, success rate and cost per successful task. If the new model is cheaper per run but succeeds less often, do the division before anyone celebrates.

Look at how each model fails as well. One that fails by stopping and saying it couldn't finish is easier to live with than one that fails by writing a duplicate or touching the wrong row. At a similar cost per success, I'd take the one that fails cleanly.

Move one task type at a time and keep the caps tight while you do. If cost per success rises, switch back. If you'd like help setting this up, get in touch.

If you work with a coding agent, paste this into it from the repository that holds your agent. It adds logging and a report for one task type and leaves the agent's behaviour alone. Replace the parts in square brackets first.

Do one task type and read the first weekly report before you add a second. If it shows a cost you weren't counting, fix that first.

Goal: Add cost per successful task tracking for one task type, without changing what the agent does.

Inputs: Task type: [for example, first payment reminder draft]. Success means: [for example, one draft linked to the right invoice, amount and due date match the invoice, no duplicate draft, sent by the user without edits to facts]. Current model: [model name]. Model prices: I will paste them from the provider's pricing page; ask me for them and do not guess. Internal review rate per hour: [amount in EUR].

Constraints: Add logging and one report only. Do not change prompts, tools or agent behaviour. Read prices from a single settings file. Do not write secrets, customer names or message bodies to the logs. Show me the plan and the files you will touch before you edit anything.

Definition of done: Every model call logs task id, task type, model name, input and output tokens, and price in EUR. Every tool call and retry logs task id, tool name, result and duration. Review logs task id, minutes, and accept, edit or reject. Each task stores a final success flag against the definition above. One weekly query returns, per task type: number of runs, cost per run, success rate, and cost per successful task. A soft and a hard budget cap exist per task type as settings, with the hard cap stopping the run. A short note explains how to change prices and caps.