Batch Request Support for Cost Optimization

[crmne]

TL;DR: Add support for batch requests to optimize costs for non-time-sensitive AI tasks.

Background

When working with AI APIs like OpenAI and Anthropic, batch requests are often substantially cheaper than individual calls. For example, OpenAI offers approximately 50% lower prices when using input caching, and Anthropic offers similar discounts for cached prompts.

Many applications don't need immediate responses. These non-time-sensitive workloads would benefit enormously from batching.

Benefits

• Cost savings: 40-50% discount on most provider APIs
• Simplified workflow: No need to manage complex batching logic in application code
• Resource optimization: Less strain on provider APIs and client applications

Thoughts?

[danielpaul]

Hey @crmne love this project. Thank you so much for your work on this.

Batch requests would be great to have for one of the projects that I'm working on.

[marckohlbrugge]

I could really use a Batch Request feature. I think it can be divided up into roughly these parts:

1. Generate a singular API request and render that as JSON rather than actually making the call to the provider
2. Combine multiple such calls (typically in JSONL format)
3. Make a request to the provider's batch request end-point to start processing
4. Make a request to provider to fetch status / results of said Batch Request
5. Process the response of the finished Batch Request

The current gem functionality is very close to providing step 1. The remaining 4-5 steps can be implemented by the end-user for the foreseeable future. This eventually allows us to extract the best ideas from the community into the docs and eventually into the gem.

To make step 1 complete, I think we need to make the following changes:

1. Some way to set the "output format" to "batch request". This would be fairly straight forward. Something like .with_output_format(:batch_request) could set an @output_format instance variable.

2. Change RubyLLM::Chat.complete to check for that @output_format variable. If its set to :batch_request, we would call @provider.render_payload_for_batch_request and return instead of @provider.complete and parsing the response.

ruby_llm/lib/ruby_llm/chat.rb

Lines 131 to 151 in 5248267

	def complete(&) # rubocop:disable Metrics/PerceivedComplexity
	response = @provider.complete(
	messages,
	tools: @tools,
	temperature: @temperature,
	model: @model.id,
	params: @params,
	schema: @schema,
	&wrap_streaming_block(&)
	)
	@on[:new_message]&.call unless block_given?
	# Parse JSON if schema was set
	if @schema && response.content.is_a?(String)
	begin
	response.content = JSON.parse(response.content)
	rescue JSON::ParserError
	# If parsing fails, keep content as string
	end
	end

3. RubyLLM::Provider would have a new method called render_payload_for_batch_request which basically just wraps render_payload similar to complete, but without making an actual request. It would need to be formatted in the proper way for making a Batch Request. (not sure if it's exactly the same as render_payload, but we should have a wrapper method either way to allow for easy overriding)

ruby_llm/lib/ruby_llm/provider.rb

Lines 43 to 63 in 5248267

	def complete(messages, tools:, temperature:, model:, params: {}, schema: nil, &) # rubocop:disable Metrics/ParameterLists
	normalized_temperature = maybe_normalize_temperature(temperature, model)
	payload = Utils.deep_merge(
	params,
	render_payload(
	messages,
	tools: tools,
	temperature: normalized_temperature,
	model: model,
	stream: block_given?,
	schema: schema
	)
	)
	if block_given?
	stream_response @connection, payload, &
	else
	sync_response @connection, payload
	end
	end

These names are just the first thing that come to mind. I'm sure we can come up with something better. We'd also need checks to see if the model has batch requesting ability and some of the other common RubyLLM logic.

Another use case for rendering the JSON payload without actually making the API call are tests. Being able to construct a request with RubyLLM and verifying the generated JSON payload could be useful for RubyLLM's own tests and end-user tests.

If there are more possible use cases for different outputs formats of the request payload, we might want to opt for dependency injection instead ( .with_output_formatter(BatchRequestOutputFormatter) ). But maybe it's overkill.

What do you think?

[marckohlbrugge]

Took a stab at it: #342

Semi-pseudo code.