Sourced guide

Batch API: cut LLM costs 50% for work that can wait

Send requests that do not need an instant reply through the Batch API instead of the standard endpoint, and the same call costs half as much. Here is the mechanism, the limits, and where it pays.

The problem

Real-time endpoints price every token the same whether the answer needs to land in the next second or the next hour. Large offline jobs, evaluation runs, content batches, and bulk classification rarely need an instant reply, yet most teams still send them through the standard endpoint at full price.

The mechanism

The Batch API lets you submit many requests together for asynchronous processing instead of one call at a time. The provider queues each request, works through the batch independently, and bills every request in it at half the standard input and output token price. A batch is capped at 100,000 requests or 256 MB, whichever limit is hit first.

Most batches finish inside an hour. You can pull results as soon as every request in the batch completes, or after 24 hours, whichever comes first, since a batch expires at the 24-hour mark if it has not finished. Anything still queued when a batch expires is never billed.

Turn it on

Submit a batch instead of a normal request: one batch endpoint call carrying many individual requests, each with its own custom_id so you can match results back to the request that produced them. Poll the batch's status, then fetch results once processing ends.

Batch requests draw on their own rate limits, separate from the standard endpoint, so a large batch run does not eat into the quota your live, real-time traffic needs. A batch cannot pre-warm a prompt cache mid-run, so plan caching separately if a batch also reuses a large shared prefix.

Confirm it worked

Each result in the batch carries its own status: succeeded, errored, canceled, or expired. Expired requests are not billed, so a spike in expired results is the signal to check demand-based slowdowns or resubmit with a longer runway. Confirm the per-request status, not just that the batch call itself returned success.

Where it pays most

Anything that can wait an hour and does not need a person staring at the screen for the answer. Bulk classification, large-scale evaluation runs, content moderation sweeps, and offline research digests are the clearest cases. The more requests you can queue at once, the more of your bill moves from full price to half price.