Provider selection for batch agent jobs comes down to one design choice most pipelines never make on purpose: how often to decide. A job that enriches ten thousand records can pick one provider at the start and use it to the end, or it can re-pick for every call. Both are defensible in some cases, and both are expensive when applied to the wrong job.
The decision has a granularity
There are three practical granularities.
Deciding once per job runs one comparison and one decision before the first call. Every record goes to the winner. Deciding per call runs the selection again for every record, so the provider can change from one call to the next. Deciding per chunk sits between them: the job is split into ranges of records or windows of time, and the decision is made once per range.
The right granularity depends less on the size of the job than on two questions. How different are the records from each other? And how fast do the candidates' price and behaviour change while the job runs?
What deciding once gets right
A single decision has advantages that are easy to underrate.
The output is consistent. Every record is processed by the same provider, so field names, formats and coverage gaps are the same across the whole result set. A job that switches providers mid-way can return a dataset where the first half and the second half disagree on how a field is formatted, which someone then has to reconcile.
The cost of deciding is paid once. The comparison, the decision and the record of it happen before any paid call, and the job then runs without further selection overhead.
The audit is simple. One decision explains every call in the job.
Deciding once breaks down in three ways: the winner's rate limit stretches the job past its deadline, the winner starts failing halfway through, or the winner's price changes after the decision was made. A job with no plan for any of these either stalls or keeps paying for calls that no longer meet the buyer's requirements.
When per-call selection pays for itself
Re-deciding on every call earns its cost in a narrower set of cases than it is usually applied to.
It makes sense when records are genuinely different. A batch that mixes company domains, registration numbers and free-text names may need different providers for different inputs, because not every candidate accepts every input. That is an input compatibility problem, and it is better solved by grouping records by input type and deciding once per group than by re-running the full selection for each record.
It also makes sense when the candidates' terms move faster than the job. If prices or availability change within minutes, a decision made at the start can be stale before the job ends.
Outside those cases, per-call selection adds a decision to every record, multiplies the records to keep, and invites the inconsistent output described above.
The middle path: decide per chunk, re-decide on triggers
For most batch jobs, the useful default is to decide once per chunk and to define in advance the events that force a new decision. The triggers belong in the job definition, not in an operator's head.
- Failures over the most recent calls rise above a set share, not a single error.
- The quoted price for the next call exceeds the price the decision was made on.
- The winner's rate limit means the remaining records will not finish before the deadline.
- The input type changes, for example when the job moves from one group of records to the next.
- The chunk's time window ends, so the decision is refreshed even if nothing went wrong.
When a trigger fires, run the decision again with the same requirements rather than switching to a hardcoded backup. The runner-up from the original decision is the obvious candidate, but its price or measurements may have changed as well, and a fresh decision checks that.
Switching safely needs two pieces of bookkeeping. Checkpoint which records each provider completed, so the job resumes from the first record without a confirmed result. And give every paid call an idempotency key tied to the record, so a retry after a timeout cannot charge twice for the same record.
Constraints and preferences belong to the job, not the call
A batch job has constraints that a single call does not. A deadline turns a rate limit into a hard limit: a provider that accepts a fixed number of calls per minute sets a minimum duration for the job, and if that duration exceeds the deadline the provider should be removed before ranking, not discovered as a problem halfway through.
The budget behaves the same way. A per-call price ceiling is not enough when the job has a total spend limit, which is the case budget-bound API selection addresses. The total is the per-call price multiplied by the number of records plus the retries, and it should be checked before the first call.
The preference also tends to differ from interactive work. A batch that nobody is waiting on can usually optimize for price rather than latency, while still holding a reliability floor so that cheap failures do not eat the savings.
Record the decision at the granularity you made it
One decision record per call is noise. One record per job hides the re-decisions that actually changed the route. Record one decision per chunk, with the range of records it covered, plus a new record each time a trigger forces a change and the reason it fired.
That record answers the questions that come up afterwards: why the second half of the dataset came from a different provider, what the job actually cost against what was planned, and whether the switch was triggered by the rules or by someone's judgement. It is the batch-shaped version of an audit trail for agent API selection.
Where Apiosk fits
Apiosk's requirement parameters are designed to be passed once and reused down the chain: `max_price`, `max_latency_ms`, `min_reliability`, `settlement=apiosk|direct`, `require_all_inputs` and `optimize_for`. A batch job can state them once in its job definition and run `/v1/decide` at the start of each chunk or whenever a trigger fires, with the same requirements each time.
Each decision returns one provider, the rule that picked it, every rejected candidate with the exact constraint that removed it, and the runners-up in order. Scores behind it are relative to that candidate set, and dimensions Apiosk has not measured are dropped and named rather than scored as zero. `POST /v1/decisions` is free and unauthenticated, and records which provider was chosen and which alternatives were rejected, so each chunk's choice stays traceable.
To wire this into an agent, start from the integration examples for agents.
Frequently asked questions
Should an agent choose a new API provider for every call in a batch?
Usually not. Per-call selection adds a decision to every record and can mix output formats from different providers in one result set. It earns its cost when records differ enough to need different providers, or when terms change quickly during the job.
What should trigger a new provider decision in the middle of a batch?
A rise in failures over the most recent calls, a change in the quoted price, the winner's rate limit putting the deadline at risk, or a change in the type of input being processed. Each trigger should be defined before the job starts, so the switch is a rule rather than an operator's call.
How do you switch providers mid-batch without paying twice?
Checkpoint which records each provider completed and give every paid call an idempotency key tied to the record. When the job resumes on a new provider, it starts from the first record without a confirmed result instead of replaying the chunk.
Does a batch job need one decision record or thousands?
One per chunk is usually enough. Record the chosen provider, the rejected alternatives and the range of records the decision covered, plus a new record whenever a trigger forces a re-decision.