'JsonEOFException: Unexpected end-of-input: expected close marker for Object' when making synchronous openAiChatClient.call

[dolukhanov]

Bug description
When making a basic synchronous openAiChatClient.call( prompt) with a single UserMessage, an Exception is thrown (com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (org.springframework.util.StreamUtils$NonClosingInputStream); line: 1, column: 1]))

I have inspected the traffic using Charles, and the OpenAI API is returning a correct JSON response.

Environment
Please provide as many details as possible: Spring AI version, Java version, which vector store you use if any, etc
OpenJDK 18.0.2
Spring Framework 6.1.4
Spring Boot 3.2.3
Spring Open AI 0.8.0-SNAPSHOT

Steps to reproduce
Basic synchronous call

String message = "Tell me about OpenAI";
UserMessage userMessage = new UserMessage( message );
List<Message> openAiMessages = List.of( userMessage );
Prompt prompt = new Prompt( openAiMessages );
ChatResponse call = openAiChatClient.call( prompt );
Generation generation = call.getResults().get( 0 );
System.out.println( generation.getOutput().getContent() );

Expected behavior
Chat response is printed.

Stack Trace

Caused by: org.springframework.http.converter.HttpMessageNotReadableException: JSON parse error: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (org.springframework.util.StreamUtils$NonClosingInputStream); line: 1, column: 1])
  at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:406) ~[spring-web-6.1.4.jar:6.1.4]
	at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.read(AbstractJackson2HttpMessageConverter.java:354) ~[spring-web-6.1.4.jar:6.1.4]
	at org.springframework.web.client.DefaultRestClient.readWithMessageConverters(DefaultRestClient.java:213) ~[spring-web-6.1.4.jar:6.1.4]
	... 82 common frames omitted
Caused by: com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (org.springframework.util.StreamUtils$NonClosingInputStream); line: 1, column: 1])
 at [Source: (org.springframework.util.StreamUtils$NonClosingInputStream); line: 1, column: 2]
	at com.fasterxml.jackson.core.base.ParserMinimalBase._reportInvalidEOF(ParserMinimalBase.java:697) ~[jackson-core-2.15.4.jar:2.15.4]
	at com.fasterxml.jackson.core.base.ParserBase._handleEOF(ParserBase.java:512) ~[jackson-core-2.15.4.jar:2.15.4]
	at com.fasterxml.jackson.core.base.ParserBase._eofAsNextChar(ParserBase.java:529) ~[jackson-core-2.15.4.jar:2.15.4]
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._skipWSOrEnd(UTF8StreamJsonParser.java:3103) ~[jackson-core-2.15.4.jar:2.15.4]
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:757) ~[jackson-core-2.15.4.jar:2.15.4]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:181) ~[jackson-databind-2.15.4.jar:2.15.4]
	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323) ~[jackson-databind-2.15.4.jar:2.15.4]
	at com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:2105) ~[jackson-databind-2.15.4.jar:2.15.4]
	at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1481) ~[jackson-databind-2.15.4.jar:2.15.4]
	at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:395) ~[spring-web-6.1.4.jar:6.1.4]
	... 84 common frames omitted

[dolukhanov]

I narrowed down the problem to the project having a dependencies on:

 <dependency>
            <groupId>org.apache.httpcomponents.client5</groupId>
            <artifactId>httpclient5</artifactId>
            <version>5.3.1</version>
        </dependency>

        <dependency>
            <groupId>org.apache.httpcomponents.client5</groupId>
            <artifactId>httpclient5-fluent</artifactId>
            <version>5.3.1</version>
        </dependency>

Once these are removed, the code functions as expected.

Leaving this issue open, should it require further investigation.

[markpollack]

I am not able to reproduce. Here is a working sample of your code.
myai.zip

Are you using any ChatOptions configuration in your code? That is an area in the code path where we use Jackson, but if no special options are set, then jackson isn't involved in the code path. Maybe provide a longer stacktrace to localize it better.

[carlspring]

I am experiencing the same issue.

• org.springframework.ai:spring-ai-openai-spring-boot-starter:0.8.0
• OpenJDK: 21.0.2
• Spring Boot: 3,2,3
• Spring Framework: 6.1.4
• Jackson: 2.17.1

package com.carlspring.ai.openai;

import com.carlspring.ai.openai.config.OpenAITestConfiguration;
import org.junit.jupiter.api.Test;
import org.springframework.ai.autoconfigure.openai.OpenAiAutoConfiguration;
import org.springframework.ai.chat.ChatResponse;
import org.springframework.ai.chat.prompt.Prompt;
import org.springframework.ai.openai.OpenAiChatClient;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;

/**
 * @author carlspring
 */
@SpringBootTest(classes = { OpenAITest.class,
                            OpenAiAutoConfiguration.class,
                            OpenAITestConfiguration.class })
public class OpenAITest
{

    @Autowired
    private OpenAiChatClient chatClient;

    @Test
    public void testBasic()
    {
        // Sync request
        ChatResponse response = chatClient.call(new Prompt("Is Linux the same as Unix?"));

        System.out.println(String.join(" ", response.getResults().stream().map((g) -> g.getOutput().toString()).toList()));
    }

}

package com.carlspring.ai.openai.config;

import org.springframework.context.annotation.Bean;
import org.springframework.stereotype.Component;
import org.springframework.web.client.RestClient;

@Component
public class OpenAITestConfiguration
{
    
    @Bean
    public RestClient.Builder restClientBuilder()
    {
        return RestClient.builder();
    }
    
}

04:50:23.924 13-05-2024 | DEBUG | main                 | org.springframework.web.client.DefaultRestClient   | Writing [ChatCompletionRequest[messages=[ChatCompletionMessage[content=Is Linux the same as Unix?, role=USER, name=null, toolCallId=null, toolCalls=null]], model=gpt-3.5-turbo, frequencyPenalty=0.0, logitBias=null, maxTokens=null, n=1, presencePenalty=null, responseFormat=null, seed=null, stop=null, stream=false, temperature=0.7, topP=null, tools=null, toolChoice=null, user=null]] as "application/json" with org.springframework.http.converter.json.MappingJackson2HttpMessageConverter
04:50:26.018 13-05-2024 | DEBUG | main                 | org.springframework.web.client.DefaultRestClient   | Reading to [org.springframework.ai.openai.api.OpenAiApi$ChatCompletion]

org.springframework.web.client.RestClientException: Error while extracting response for type [org.springframework.ai.openai.api.OpenAiApi$ChatCompletion] and content type [application/json]

	at org.springframework.web.client.DefaultRestClient.readWithMessageConverters(DefaultRestClient.java:236)
	at org.springframework.web.client.DefaultRestClient$DefaultResponseSpec.readBody(DefaultRestClient.java:667)
	at org.springframework.web.client.DefaultRestClient$DefaultResponseSpec.toEntityInternal(DefaultRestClient.java:637)
	at org.springframework.web.client.DefaultRestClient$DefaultResponseSpec.toEntity(DefaultRestClient.java:626)
	at org.springframework.ai.openai.api.OpenAiApi.chatCompletionEntity(OpenAiApi.java:656)
	at org.springframework.ai.openai.OpenAiChatClient.chatCompletionWithTools(OpenAiChatClient.java:337)
	at org.springframework.ai.openai.OpenAiChatClient.lambda$call$1(OpenAiChatClient.java:151)
	at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:335)
	at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:211)
	at org.springframework.ai.openai.OpenAiChatClient.call(OpenAiChatClient.java:147)
	at com.carlspring.ai.openai.OpenAITest.testBasic(OpenAITest.java:30)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
Caused by: org.springframework.http.converter.HttpMessageNotReadableException: JSON parse error: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (org.springframework.util.StreamUtils$NonClosingInputStream); line: 1, column: 1])
	at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:406)
	at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.read(AbstractJackson2HttpMessageConverter.java:354)
	at org.springframework.web.client.DefaultRestClient.readWithMessageConverters(DefaultRestClient.java:213)
	... 13 more
Caused by: com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (org.springframework.util.StreamUtils$NonClosingInputStream); line: 1, column: 1])
 at [Source: (org.springframework.util.StreamUtils$NonClosingInputStream); line: 1, column: 2]
	at com.fasterxml.jackson.core.base.ParserMinimalBase._reportInvalidEOF(ParserMinimalBase.java:697)
	at com.fasterxml.jackson.core.base.ParserBase._handleEOF(ParserBase.java:512)
	at com.fasterxml.jackson.core.base.ParserBase._eofAsNextChar(ParserBase.java:529)
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._skipWSOrEnd(UTF8StreamJsonParser.java:3103)
	at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:757)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:181)
	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323)
	at com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:2105)
	at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1481)
	at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:395)
	... 15 more

It seems that the answers here might be related and that the JSON reponse needs to be read fully before trying to read it as an object.

[vwiencek]

I have the same problem and can't figure out how to solve it

[jocax]

I had the same issue and found a workaround by setting the required http header "Accept-Encoding". I applied the header in the Rest.Builder. One the header is set the response can be read without the above error.

@Bean
RestClient.Builder restClientBuilder() {
    return RestClient.builder()
            .defaultHeaders(new Consumer<HttpHeaders>() {
                @Override
                public void accept(HttpHeaders httpHeaders) {
                    // https://github.com/spring-projects/spring-ai/issues/372
                    httpHeaders.set("Accept-Encoding", "gzip, deflate");
                }
            });
}

Hope this helps.

[ThomasVitale]

Spring Boot provides an autoconfigured RestClient.Builder object, including JSON serialization configuration. Have you tried using that one instead of defining one yourself?

When you create a RestClient via RestClient.builder(), you don't get the Spring Boot autoconfiguration, so it's up to you to configure encoding and serialization. If you create one from an autowired RestClient.Builder bean, that is all configured for you and should work out-of-the-box.

[Alien2150]

So my workaround that is not that invasive as overwritting the default RestClient builder:

    @Bean
    fun chatModel(
        commonProperties: OpenAiConnectionProperties,
        chatProperties: OpenAiChatProperties,
        webClientBuilder: WebClient.Builder,
        retryTemplate: RetryTemplate,
        functionCallbackContext: FunctionCallbackContext,
        responseErrorHandler: ResponseErrorHandler,
    ): OpenAiChatModel {
        val restClientBuilder = RestClient.builder().defaultHeaders {
            it.set(HttpHeaders.ACCEPT_ENCODING, "gzip, deflate")
        }

        val openAiApi = OpenAiApi(
            chatProperties.baseUrl ?: commonProperties.baseUrl,
            chatProperties.apiKey ?: commonProperties.apiKey,
            restClientBuilder,
            webClientBuilder,
            responseErrorHandler,
        )

        return OpenAiChatModel(
            openAiApi,
            chatProperties.options,
            functionCallbackContext,
            retryTemplate,
        )
    }

As a suggestion: Could the bean for RestClient builder not use a "Qualifier" annotation so it would not be overwritten by the default one? (OpenAiAutoConfiguration seems to use the default one)

[markpollack]

I was thining the same thing @Alien2150 in relation to setting timeouts per model, using a qualifier annotation that will pick up a specific restClientBuilder. It would seem that this problem would have been solved already with apps calling many microservices, each with different time outs... #512 and related issues discuss it more.

[coderphonui]

So my workaround that is not that invasive as overwritting the default RestClient builder:

    @Bean
    fun chatModel(
        commonProperties: OpenAiConnectionProperties,
        chatProperties: OpenAiChatProperties,
        webClientBuilder: WebClient.Builder,
        retryTemplate: RetryTemplate,
        functionCallbackContext: FunctionCallbackContext,
        responseErrorHandler: ResponseErrorHandler,
    ): OpenAiChatModel {
        val restClientBuilder = RestClient.builder().defaultHeaders {
            it.set(HttpHeaders.ACCEPT_ENCODING, "gzip, deflate")
        }

        val openAiApi = OpenAiApi(
            chatProperties.baseUrl ?: commonProperties.baseUrl,
            chatProperties.apiKey ?: commonProperties.apiKey,
            restClientBuilder,
            webClientBuilder,
            responseErrorHandler,
        )

        return OpenAiChatModel(
            openAiApi,
            chatProperties.options,
            functionCallbackContext,
            retryTemplate,
        )
    }

As a suggestion: Could the bean for RestClient builder not use a "Qualifier" annotation so it would not be overwritten by the default one? (OpenAiAutoConfiguration seems to use the default one)

Adding the httpHeaders.set("Accept-Encoding", "gzip, deflate"); fixed the issue. Thanks!

[asw12]

I encountered and researched this defect a bit further out of curiosity, and found the culprit to be brotli decompression support, hence why using an Accept-Encoding that excludes "br" avoids the problem.

The dependency tree bringing in brotli looks something like:

[INFO] |  +- org.springframework.ai:spring-ai-tika-document-reader:jar:1.0.0-M6:compile
                  ...
[INFO] |  |     +- org.apache.tika:tika-parser-pkg-module:jar:3.0.0:compile
[INFO] |  |     |  +- org.brotli:dec:jar:0.1.2:compile

where that brotli dec-0.1.2.jar was a version that was last published to Maven Central by Google 7 years ago.

There is an issue with BrotliInputStream in that library when they read from an internal buffer without depleting it, that the read method incorrectly reports a -1 value signifying an EOF. There appears to be a fix in later versions, but even though this fix was committed 3 years ago, they never published it.

With the HttpComponents ClientHttpRequestFactory, note that the presence of the BrotliInputStream class is enough to have the underlying HTTP Client 5 library to add the accept header.

So the problem looks like

• Jackson does a preliminary check to see if the body has content by calling read() for one byte.
  • BrotliInputStream populates the buffer with ~8kb worth of data. '{' is read from it.
  • That character is "unread" back to a buffer held by a wrapping PushbackInputStream
• Jackson tries to parse JSON from the InputStream with its parser.
  • The '{' is read back, followed by some hundreds of bytes, but not enough to cause BrotliInputStream's buffer to be depleted.
  • BrotliInputStream reports -1 for the read call, so Jackson only sees a START_OBJECT before an unexpected JsonEOFException is thrown.