Why does the search API sometimes return line as "..."?

[samwarring]

Hello, I have found that for some invocations of the /search API, the search would complete succesfully and return matches to the client, but the matches have line set to "...", and lineNumber set to "" (empty string). I found this usually happens when the search string contains "special characters".

For example, I want to search for std::string in my codebase. I need to escape each : character, and I need to enclose the entire search string in quotes, or else the results include the union of std and string. This gives me a search string of "std\:\:string".

• A full search for that string gives correct results.
• A symbol search for that string on the web GUI gives correct results.
• A symbol search for that string using the REST API gives results where line is "..." and lineNumber is "".

Request is: <server address>/api/v1/search?projects=mainline&symbol=%22std%5C%3A%5C%3Astring%22
Response contains many matching files, but each file has one item {line: "...", lineNumber: "", tag: null}.

Unfortunately, I do not have admin access to the OpenGrok instance I'm using, so I cannot provide server logs, but I do know we're using OpenGrok 1.12.14.

Can someone shed light why I am getting these results? Am I invoking the API incorrectly, or is this a bug? Thank you!

[vladak]

Not really an answer, more likely just a hint for those who would like to debug this: the immediate reason for the elipsis ("...") and lack of line numbers is the missing context:

opengrok/opengrok-indexer/src/main/java/org/opengrok/indexer/search/SearchEngine.java

Lines 518 to 520 in 7b5f14f

	if (!hasContext) {
	ret.add(new Hit(filename, "...", "", false, alt));
	}

Why that appears when searching via the API and not in the UI probably stems from the fact that the search and interpretation of the results is done differently in the UI and API. The API search handling in SearchController uses the SearchEngine class. All the searching and results interpretation is done within that class. For search it uses the IndexSearcher:

opengrok/opengrok-indexer/src/main/java/org/opengrok/indexer/search/SearchEngine.java

Line 210 in 7b5f14f

searcher.search(query, collector);

and then goes through the Documents and uses Context#getContext() to get the context to assemble the list of Hit objects which get passed directly into the API result.

The UI uses the SearchHelper class, namely

opengrok/opengrok-web/src/main/webapp/search.jsp

Line 86 in 7b5f14f

searchHelper.prepareExec(cfg.getRequestedProjects()).executeQuery().prepareSummary();

to perform the search. The results are then interpreted using

opengrok/opengrok-web/src/main/webapp/search.jsp

Line 228 in 7b5f14f

Results.prettyPrint(out, searchHelper, start, start + thispage);

which uses the Summarizer class to interpret the results.

[vladak]

I can replicate this with the OpenSolaris workspace. This happens for all of the std::string symbol matches.

TLDR: due to the differences in search result interpretation between API search and UI search, the API search can match only definitions, not symbols.

The call to getContext() in SearchEngine#results():

opengrok/opengrok-indexer/src/main/java/org/opengrok/indexer/search/SearchEngine.java

Lines 473 to 477 in 7b5f14f

	hasContext = sourceContext.getContext(
	new InputStreamReader(new FileInputStream(
	source + filename), StandardCharsets.UTF_8),
	null, null, null, filename, tags, nhits > 100,
	getDefinition() != null, ret, scopes);

returns false, hence the ... Hit object.

The reason for this is that in Context#getContext() called from results() there is no match in the tags:

opengrok/opengrok-indexer/src/main/java/org/opengrok/indexer/search/context/Context.java

Lines 289 to 291 in 7b5f14f

	for (Definitions.Tag tag : tags.getTags()) {
	for (LineMatcher lineMatcher : m) {
	if (lineMatcher.match(tag.symbol) == LineMatcher.MATCHED) {

The m LineMatcher object contains the std::string broken up into 2 phrase terms which is fine, however the list of strings returned from the getTags() call contains only limited collection of objects. In fact, it contains only definitions (Definition objects), from each of the files and not symbols. This is because the tags are taken from the document like this:

opengrok/opengrok-indexer/src/main/java/org/opengrok/indexer/search/SearchEngine.java

Lines 457 to 460 in 7b5f14f

	IndexableField tagsField = doc.getField(QueryBuilder.TAGS);
	if (tagsField != null) {
	tags = Definitions.deserialize(tagsField.binaryValue().bytes);
	}

The UI uses different code path, namely Results#printPlain() (for this kind of search) which descends into Context#getContext2() (unlike getContext() used by the API search) which uses UnifiedHighlighter that grabs the information about the symbols using the line number matches originate here:

opengrok/opengrok-indexer/src/main/java/org/opengrok/indexer/search/context/Context.java

Lines 214 to 219 in 7b5f14f

	String res = uhi.highlightFieldsUnion(fields, query, docId,
	linelimit);
	if (res != null) {
	dest.append(res);
	return true;
	}

and these already contain HTML markdown.

So, the answer is that completely different way is used to present the results in API and UI search and the API search is able to match only definitions, not symbols. The API search would have switch to some other way, perhaps also use the unified highlighter, albeit with different formatter to avoid the HTML markdown present in the API results. The getContext2() function leverages the ContextFormatter class:

opengrok/opengrok-indexer/src/main/java/org/opengrok/indexer/search/context/Context.java

Lines 194 to 206 in 7b5f14f

	ContextFormatter formatter = new ContextFormatter(args);
	formatter.setUrl(urlPrefixE + pathE);
	formatter.setDefs(tags);
	formatter.setScopes(scopes);
	formatter.setMoreUrl(moreURL);
	formatter.setMoreLimit(linelimit);
	UnifiedHighlighter.Builder uhBuilder = new UnifiedHighlighter.Builder(searcher, anz)
	// .withMaxLength(maxDocCharsToAnalyze)
	// .withHighlightPhrasesStrictly(true)
	// .withHandleMultiTermQuery(true)
	.withBreakIterator(StrictLineBreakIterator::new)
	.withFormatter(formatter);

to add the line number mark down for the line matches via

opengrok/opengrok-indexer/src/main/java/org/opengrok/indexer/search/context/ContextFormatter.java

Line 227 in 7b5f14f

startLine(bld, lineUrl, lhi.getLineno());

[samwarring]

Thank you for your analysis! However, I'm confused when you say "the API can match only definitions, not symbols", are you saying I should only use the web API for defs searches? Because, that does not match what I'm seeing:

• A symbol search for string on the web API gives valid results.
• A full search for string on the web API gives valid results.

[vladak]

Thank you for your analysis! However, I'm confused when you say "the API can match only definitions, not symbols", are you saying I should only use the web API for defs searches? Because, that does not match what I'm seeing:

• A symbol search for string on the web API gives valid results.
• A full search for string on the web API gives valid results.

I should have said: when searching via API, context is provided only for definitions, not symbols.