Steer · Chapter 02

Measuring autonomy

Share of transitions automated, human hours per unit of revenue, escalation rate.

What did it cost to get a result you were willing to use? A provider’s token bill answers only part of that question. Your time checking, correcting and restarting the work belongs in the calculation, along with failed attempts and the services the workflow depends on.

Measure cost per accepted outcome. If ten runs produce five usable results, the costs of all ten runs belong to those five results. Record review minutes separately from cash spending, and label any value you assign to human time as an assumption. This makes the remaining work visible.

Compare models on the same work

Give each candidate the same representative cases and acceptance criteria. Count corrections as well as successful first attempts. A cheaper model can be the more expensive choice if someone must supervise every answer; an expensive model can also be unnecessary for a predictable task.

Begin with a simple ledger of run, model, attempts, charges, review time and accepted result. Add tracing, such as Langfuse, when it helps explain costs you cannot otherwise see. A trace records what happened; spending controls must be enforced by the provider or execution system and checked with a harmless low limit.

Decide what the saving makes possible

Compare the new process with the observed baseline, including the effort of maintaining it. Ask whether the business gained contribution, capacity or room for work that needs human attention. Revenue and the number of automated steps are insufficient on their own.

Choose a model when you can explain its cost at the quality the job requires. Add model routing only when its measured benefit covers the extra testing and operational complexity. Revisit the choice when the task, prices or evidence change.

Tools and setup

Existing implementation: Langfuse's tracing quickstart documents project credentials, supported instrumentation and inspecting a first trace. A trace supports measurement; it does not by itself enforce a spending limit.

  1. Begin with a ledger: run ID, model/version, attempts, provider charge, other variable cost, review minutes, outcome accepted and resulting value. Separate estimates, cash costs and time valuations.
  2. If automated tracing is useful, create a Langfuse project in an appropriate hosting region. Create credentials in project settings and store them in a secret manager/environment outside source control. Choose the documented integration matching the runtime, instrument one run, flush/export as required and inspect its trace. Redact restricted payloads before transmission.
  3. Compare permitted models on the same cases and rubric. Count failed attempts and human corrections. Set enforceable budget controls in the provider or execution system; verify stop behavior using a harmless low limit. If a limit is only advisory, say so.

Setup paths checked against linked documentation on 15 November 2026. Verify current account support during setup; reader integrations have not been tested here.

Work through this with your agent

Paste the request into your assistant. It will interview you and help produce the working artifact.

Interview me to measure the economics of one workflow. Ask at most three questions per turn and wait. Obtain accepted outcome volume, value or revenue, baseline human hours, current tool/model costs, review, rework, failure costs and one-time integration effort. Distinguish estimates from measured amounts and avoid counting saved time as realized cash automatically. Set a per-run and period budget, owner and stop/escalation rule before paid tests. Using authorized tools and spend, compare available model options on the same small representative cases against one acceptance rubric. Record quality, latency, total cost per accepted outcome and human effort. Test shorter instructions/results before assuming cheaper outputs preserve quality. Choose the simplest passing option; add routing only if its benefit exceeds coordination costs. Save COST-LEDGER.csv and MODEL-DECISION.md. Explain how spend limits are enforced; label advisory limits honestly. Do not switch production models without the agreed regression and approval process.

Use these documented setup paths as references; check current account support before acting:
https://langfuse.com/docs/observability/get-started

The Hypothesis Diff · weekly

Know when a hypothesis is stressed or changes.

One email a week: which hypotheses hold, which are stressed, which changed, and the tools worth trying. Nobody else publishes what stopped being true.