NPE Cannot invoke "java.lang.CharSequence.length()" because "csq" is null

[henrikno]

Elasticsearch Version

9.2.0

Installed Plugins

No response

Java Version

bundled

OS Version

cloud.elastic.co

Problem Description

I was uploading symbols to the universal profiling service, and got some errors in the logs.

The profiling service is logging:
(indexReturnPadsFile) [400 Bad Request]: map[error:map[caused_by:map[reason:Cannot invoke "java.lang.CharSequence.length()" because "csq" is null type:null_pointer_exception] reason:[-1:60] failed to parse field [Symbfile.file.id] of type [keyword] in document with id 'Y3wZtJ8gStblRVIE7AOoUgA'. Preview of field's value: '[B@6f9d86ff' root_cause:[map[reason:[-1:60] failed to parse field [Symbfile.file.id] of type [keyword] in document with id 'Y3wZtJ8gStblRVIE7AOoUgA'. Preview of field's value: '[B@6f9d86ff' type:document_parsing_exception]] type:document_parsing_exception] status:400]

Which matches this error in ES logs:

[instance-0000000001][.profiling-returnpads-private-v001] Error while parsing document for index [.profiling-returnpads-private-v001]: [-1:60] failed to parse field [Symbfile.file.id] of type [keyword] in document with id 'Y3wZtJ8gStblRVIE7AOoUgI'. Preview of field's value: '[B@3427843f'
org.elasticsearch.index.mapper.DocumentParsingException: [-1:60] failed to parse field [Symbfile.file.id] of type [keyword] in document with id 'Y3wZtJ8gStblRVIE7AOoUgI'. Preview of field's value: '[B@3427843f'
	at org.elasticsearch.index.mapper.FieldMapper.rethrowAsDocumentParsingException(FieldMapper.java:241) ~[elasticsearch-9.2.0.jar:?]
	at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:194) ~[elasticsearch-9.2.0.jar:?]
	at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:477) ~[elasticsearch-9.2.0.jar:?]
	at org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:829) ~[elasticsearch-9.2.0.jar:?]
	at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:397) ~[elasticsearch-9.2.0.jar:?]
	at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:348) ~[elasticsearch-9.2.0.jar:?]
	at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:457) ~[elasticsearch-9.2.0.jar:?]
	at org.elasticsearch.index.mapper.DocumentParser.doParseObject(DocumentParser.java:545) ~[elasticsearch-9.2.0.jar:?]
	at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:533) ~[elasticsearch-9.2.0.jar:?]
	at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:387) ~[elasticsearch-9.2.0.jar:?]
	at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:348) ~[elasticsearch-9.2.0.jar:?]
	at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:457) ~[elasticsearch-9.2.0.jar:?]
	at org.elasticsearch.index.mapper.DocumentParser.doParseObject(DocumentParser.java:545) ~[elasticsearch-9.2.0.jar:?]
	at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:533) ~[elasticsearch-9.2.0.jar:?]
	at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:387) ~[elasticsearch-9.2.0.jar:?]
	at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:348) ~[elasticsearch-9.2.0.jar:?]
	at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:157) ~[elasticsearch-9.2.0.jar:?]
	at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:103) ~[elasticsearch-9.2.0.jar:?]
	at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:128) ~[elasticsearch-9.2.0.jar:?]
	at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:1092) ~[elasticsearch-9.2.0.jar:?]
	at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:1019) ~[elasticsearch-9.2.0.jar:?]
	at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:963) ~[elasticsearch-9.2.0.jar:?]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:425) ~[elasticsearch-9.2.0.jar:?]
	at org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:276) ~[elasticsearch-9.2.0.jar:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) ~[elasticsearch-9.2.0.jar:?]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:344) ~[elasticsearch-9.2.0.jar:?]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:181) ~[elasticsearch-9.2.0.jar:?]
	at org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:82) ~[elasticsearch-9.2.0.jar:?]
	at org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:245) ~[elasticsearch-9.2.0.jar:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) ~[elasticsearch-9.2.0.jar:?]
	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35) ~[elasticsearch-9.2.0.jar:?]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1076) ~[elasticsearch-9.2.0.jar:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) ~[elasticsearch-9.2.0.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) ~[?:?]
	at java.lang.Thread.run(Thread.java:1474) ~[?:?]
Caused by: java.lang.NullPointerException: Cannot invoke "java.lang.CharSequence.length()" because "csq" is null
	at java.nio.CharBuffer.wrap(CharBuffer.java:573) ~[?:?]
	at java.nio.charset.Charset.encode(Charset.java:915) ~[?:?]
	at org.elasticsearch.xcontent.Text.bytes(Text.java:71) ~[elasticsearch-x-content-9.2.0.jar:?]
	at org.elasticsearch.xcontent.Text.stringLength(Text.java:101) ~[elasticsearch-x-content-9.2.0.jar:?]
	at org.elasticsearch.index.mapper.Mapper$IgnoreAbove.isIgnored(Mapper.java:185) ~[elasticsearch-9.2.0.jar:?]
	at org.elasticsearch.index.mapper.KeywordFieldMapper.indexValue(KeywordFieldMapper.java:1204) ~[elasticsearch-9.2.0.jar:?]
	at org.elasticsearch.index.mapper.KeywordFieldMapper.parseCreateField(KeywordFieldMapper.java:1170) ~[elasticsearch-9.2.0.jar:?]
	at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:192) ~[elasticsearch-9.2.0.jar:?]
	... 34 more

Which at first glance looks like a mapping issue, but the NullPointerException seems like it should not have happened. I think symbolizer service uses SMILE so maybe that's why it's able to make an invalid string?

Steps to Reproduce

Spin up 9.2.0 deployment on cloud.elastic.co
Go to Kibana and enable universal profiling.
Create an API Key

Download symbtool https://artifacts.elastic.co/downloads/prodfiler/symbtool-9.2.0-linux-x86_64.tar.gz

crane export --platform linux/arm64 docker.elastic.co/elastic-agent/elastic-agent-complete:8.19.6@sha256:0053c73331517e194f0c7ba385cd8f5f848bacd0069447404ca69ca05df6248c - | tar -xC foo/
https://github.com/google/go-containerregistry/blob/main/cmd/crane/doc/crane.md

./symbtool-9.2.0-linux-x86_64/symbtool push-symbols executable -u 'https://deployment.symbols.us-central1.gcp.cloud.es.io:443' -t API_KEY -e foo/usr/share/elastic-agent/data/elastic-agent-3778d3/elastic-agent

Check ES and profiler service logs.

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)

[smalyshev]

NPE is not great, but since it ultimately resolves to 400, I wonder how big is the issue here. Is this bad data that we handle a bit sloppily, or is this valid data that we mess up?

Looking into the backtrace, it's in Text.java here:

public UTF8Bytes bytes() {
        if (bytes == null) {
            byte[] byteArray = string.getBytes(StandardCharsets.UTF_8);
            bytes = new UTF8Bytes(byteArray, 0, byteArray.length);
        }
        return bytes;
    }

Note that we check string for being null in other places, but here we just go for it. Which is also suspicious since it sits on the false branch of hasString. Maybe it's not right. @Tim-Brooks what do you think?