Structured outputs `Map` handling has confusing error message and isn't well documented

[fydelity]

I found that when you pass an object with a map to StructuredChatCompletionCreateParams the API returns a 400 response Caused by: com.openai.errors.BadRequestException: 400: Invalid schema for response_format 'json-schema-from-TestMap': In context=(), 'required' is required to be supplied and to be an array including every key in properties. Extra required key 'coolMap' supplied.

Object

  private static class TestMap {
    public Map<String, Integer> coolMap;
  }

Full stack

Caused by: com.openai.errors.BadRequestException: 400: Invalid schema for response_format 'json-schema-from-TestMap': In context=(), 'required' is required to be supplied and to be an array including every key in properties. Extra required key 'coolMap' supplied.
	at com.openai.errors.BadRequestException$Builder.build(BadRequestException.kt:88) ~[openai-java-core-2.0.0.jar:2.0.0]
	at com.openai.core.handlers.ErrorHandler$withErrorHandler$1.handle(ErrorHandler.kt:48) ~[openai-java-core-2.0.0.jar:2.0.0]
	at com.openai.services.blocking.chat.ChatCompletionServiceImpl$WithRawResponseImpl$create$1.invoke(ChatCompletionServiceImpl.kt:122) ~[openai-java-core-2.0.0.jar:2.0.0]
	at com.openai.services.blocking.chat.ChatCompletionServiceImpl$WithRawResponseImpl$create$1.invoke(ChatCompletionServiceImpl.kt:120) ~[openai-java-core-2.0.0.jar:2.0.0]
	at com.openai.core.http.HttpResponseForKt$parseable$1$parsed$2.invoke(HttpResponseFor.kt:14) ~[openai-java-core-2.0.0.jar:2.0.0]
	at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:74) ~[kotlin-stdlib-1.9.24.jar:1.9.24-release-822]
	at com.openai.core.http.HttpResponseForKt$parseable$1.getParsed(HttpResponseFor.kt:14) ~[openai-java-core-2.0.0.jar:2.0.0]
	at com.openai.core.http.HttpResponseForKt$parseable$1.parse(HttpResponseFor.kt:16) ~[openai-java-core-2.0.0.jar:2.0.0]
	at com.openai.services.blocking.chat.ChatCompletionServiceImpl.create(ChatCompletionServiceImpl.kt:56) ~[openai-java-core-2.0.0.jar:2.0.0]
	at com.openai.services.blocking.chat.ChatCompletionService.create(ChatCompletionService.kt:80) ~[openai-java-core-2.0.0.jar:2.0.0]
	at com.openai.services.blocking.chat.ChatCompletionService.create(ChatCompletionService.kt:67) ~[openai-java-core-2.0.0.jar:2.0.0]
	at com.zeroClick.core.chatgpt.ChatGPT.run(ChatGPT.java:108) ~[classes/:na]

[TomerAberbach]

This is actually intended-ish behavior

In JSON schema, the way a map is represented is with additionalProperties: true. However, OpenAI structured outputs requires additionalProperties: false

What you probably want is something like this instead:

class MapLikeThing {
  public List<EntryLikeThing> coolMap;
}

class EntryLikeThing {
  public String key;
  public int value;
}

Uniqueness is not guaranteed, but that's probably fine (you can dedupe, I guess). This also allows you to add descriptions to the key and value (and give them more semantic names)

Anyway, I will repurpose this bug for:

1. Improving this exception message
2. Adding something about this to the README (i.e. how to handle maps)

[bnbarak]

How's the pydantic version handles maps in structured responses?

[TomerAberbach]

I chatted with @RobertCraigie and he doesn't remember it working in the Python SDK either.

Did you observe a different behavior in Python?

[RobertCraigie]

Can confirm I just tried this with the python client and it doesn't work either, this is unfortunately an API limitation.

[bnbarak]

Maybe it's a python nuance, but you can use a dictionary.

from typing import Dict, Optional

from openai import OpenAI
from pydantic import BaseModel

client = OpenAI()

class BookStores(BaseModel):
    listOfBookStores: Optional[Dict[str, int]] = None

completion = client.beta.chat.completions.parse(
    model="gpt-4o-2024-08-06",
    messages=[
        {"role": "system", "content": "You are an expert in NY book stores. The response is the name mapped to the a rank"},
        {"role": "user", "content": "What are the best book stores in NY?"}
    ],
    response_format=BookStores,
)

books = completion.choices[0].message
# '{"listOfBookStores":{"McNally Jackson":1,"The Strand Bookstore":2,"Books Are Magic":3,"Rizzoli Bookstore":4,"Albertine Books":5}}'

I also noticed that this example won't work without the Optional:

class BookStores(BaseModel):
    listOfBookStores: Dict[str, int]

Returns:

BadRequestError: Error code: 400 - {'error': {'message': "Invalid schema for response_format 'BookStores': In context=(), 'required' is required to be supplied and to be an array including every key in properties. Extra required key 'listOfBookStores' supplied.", 'type': 'invalid_request_error', 'param': 'response_format', 'code': None}}