If your business uses AI APIs for high-volume processing — classifying thousands of records, generating content for large datasets, analysing collections of documents — you may be paying full real-time API prices for work that does not need immediate results. Both OpenAI and Anthropic offer batch processing APIs that process requests asynchronously at significantly reduced pricing. For the right workflows, this single change can cut processing costs by 50% with no change to output quality.
What Batch Processing Means
Standard API calls are synchronous: you send a request and wait for an immediate response, typically within a few seconds. This real-time capability costs full price because the provider’s infrastructure must handle your request with low latency. Batch processing sends a collection of requests to be processed over a longer window — typically up to 24 hours — with no latency guarantee. Because the provider can schedule batch jobs during off-peak periods and use infrastructure more efficiently, they pass the savings to customers through lower pricing.
The implication for your workflows: any AI processing task where you do not need the result immediately is a candidate for batch pricing. Nightly data processing runs, bulk content generation, large document analysis jobs, weekly reporting workflows — all of these have flexible timing and are ideal batch candidates.
OpenAI Batch API
OpenAI’s Batch API, introduced in 2024, processes requests at 50% of the standard API price with a 24-hour completion window. You submit a JSONL file containing up to 50,000 requests, and OpenAI processes them and returns the results when complete (typically within minutes to a few hours for most batch sizes). The Batch API supports GPT-4o, GPT-4o Mini, and other models, making it applicable to a wide range of workflows.
The implementation requires submitting requests in JSONL format rather than individual API calls — a code change rather than a configuration change. For developers, this is a straightforward modification to an existing workflow.
Batch vs Real-Time: Cost Comparison
| Scenario | Real-Time Cost | Batch Cost | Saving |
|---|---|---|---|
| 1,000 GPT-4o Mini classifications | ~$0.08 | ~$0.04 | 50% |
| 10,000 GPT-4o summaries | ~$25 | ~$12.50 | 50% |
| 100,000 GPT-4o Mini extractions/day | ~$3/day | ~$1.50/day | $547/year |
Anthropic Message Batches API
Anthropic’s Message Batches API offers 50% discounted pricing on Claude Haiku and Claude Sonnet for batch processing. The mechanics are similar to OpenAI’s: submit a collection of requests, receive results when processing is complete. The same workflows that benefit from OpenAI batch processing benefit from Anthropic batch processing for teams using Claude.
Identifying Your Batch Candidates
Review your AI API usage and ask for each workflow: does this need a real-time response, or does it need a response before the next morning? Customer-facing chatbots need real-time responses — they cannot batch. Nightly database enrichment, weekly content generation runs, monthly report processing, and bulk classification jobs all have flexible timing. For each flexible-timing workflow currently using standard API calls, switching to batch pricing immediately captures 50% cost savings with no engineering change beyond the API call format. For most businesses with meaningful API volume, this is a quick engineering task with a clear and ongoing financial return.
How to Structure a Batch Request
Each request in a batch is a JSON object with four fields: a custom_id you define for tracking, the HTTP method (POST), the API endpoint (/v1/chat/completions), and the body containing model, messages, and max_tokens. You write one request per line in a JSONL file and upload it to the batch endpoint. OpenAI processes all requests within 24 hours and returns a results file in the same JSONL format, with each line containing the custom_id and the completion. The custom_id is your key for matching results to inputs — use a meaningful identifier (like a database record ID) rather than sequential numbers, so you can correlate results with their source records even if the results arrive in a different order than submitted.
Error handling in batch jobs differs from synchronous calls. Instead of catching exceptions in real-time, you check the error field in each result line after the job completes. Build a results processor that separates successful completions from errors, logs error details alongside their custom_ids, and either retries failures in a new batch or routes them to a fallback process. A batch where 98% of requests succeed and 2% fail is still a good outcome — the key is not silently discarding the failures.
Monitoring Batch Job Progress
Batch jobs can be polled via the API to check status. The status transitions from validating to in_progress to completed (or failed). For jobs you need to act on promptly after completion, implement a polling loop that checks status every few minutes and processes results as soon as they are available. For overnight jobs where timing is flexible, a simple cron that runs at a fixed time in the morning and processes any completed batch jobs is cleaner than continuous polling. Store batch job IDs in your database alongside the input records so you can always retrieve results for a specific job if your processing step fails after the job completes.
Switch your highest-volume non-time-sensitive AI workflow to batch processing this quarter. At meaningful API volume, the 50% cost reduction is significant enough to justify the engineering investment.
Managing Batch Job Failures Gracefully
Batch jobs fail differently from synchronous API calls. A synchronous failure is caught immediately in your application; a batch failure appears hours later in a results file. Build your batch processing to handle partial failures gracefully: when you download results, separate successful completions from errors, log the errors with their custom_ids and error codes, and decide how to handle each category — retry transient failures in a new batch, route persistent failures to synchronous processing or manual review, and alert if the failure rate exceeds an acceptable threshold. A batch pipeline that handles partial failures gracefully can process millions of records reliably even when some percentage of individual requests fail.
Combining Batch Processing With Other Cost Optimisations
Batch processing’s 50% cost reduction is additive with other optimisations. Prompt compression reduces per-call token costs regardless of whether calls are synchronous or batched — compressing a prompt by 30% and batching the resulting calls captures both savings simultaneously. Model routing directs batch-eligible tasks to the cheapest model that meets quality requirements — combining batch pricing with a cheaper model tier can reduce costs by 60–70% compared to synchronous premium model usage. Build these optimisations in order: first identify which workflows are batch-eligible, then compress prompts, then route to the minimum viable model, then switch to batch processing. Each step compounds the previous one.
The batch API is the highest-leverage single change available for reducing API costs on high-volume, non-time-sensitive workflows. If you have any workflow processing more than 1,000 requests per day that does not require real-time responses, migrating it to batch processing should be among your first cost optimisation priorities.
Combining Batch and Streaming for Hybrid Workflows
Not all steps in a complex AI workflow have the same latency tolerance. Batch processing suits offline, non-interactive steps where 24-hour turnaround is acceptable. Streaming suits interactive steps where users expect a response within seconds. Hybrid architectures use batch for the expensive, non-interactive parts (nightly data enrichment, document indexing, background research) and streaming for the interactive parts (real-time Q&A, live drafting assistance). The cost savings from batch processing the offline parts reduce your overall AI budget, freeing spend for the streaming interactions where low latency is actually worth the premium. Identify which steps in your AI workflows genuinely need to be synchronous and interactive, and batch everything else — the cost efficiency of this separation is typically 30–50% on total AI spend.
Monitoring Batch Job Health
Batch AI jobs that run overnight or on scheduled intervals need monitoring that catches silent failures before they affect dependent processes. Instrument your batch jobs with three health signals: a heartbeat log entry at the start of each run, a completion log entry at the end, and an alert if the completion entry does not appear within a defined window after the expected start time. For batch jobs that process a known number of records, add a completeness check: if the job processed significantly fewer records than expected, flag for investigation even if it completed without error. A batch job that processed 80% of the expected records and exited cleanly produced incomplete results — the clean exit is not sufficient evidence that everything worked. Completeness monitoring catches these partial success failures that standard error monitoring misses.