TIKA-5057: deep dive rewrite of the Onenote parser

.skills/users/file-forensics/SKILL.md

@@ -186,7 +186,7 @@ the defaults are NOT the same everywhere:
   config (it prints a banner saying so) that already enables the marquee
   items — `parseIncrementalUpdates`, `extractMacros`, `extractActions`,
   `extractInlineImages`. On tika-app, `file-forensics-config.json` still adds
-  `extractFontNames`, `extractScripts`, `includeMissingRows`,
+  `extractFontNames`, `extractScripts`, `includeMissingRows`, OneNote page history,
   deleted/moved-content for legacy `.doc` (on `.docx` the convenience
   config already includes it, so testing this switch on a docx shows no
   diff — that's expected), and pins `accessCheckMode` to
@@ -212,6 +212,19 @@ The full switch list:
   embedded entries), `includeDeletedContent` and `includeMoveFromContent`
   (tracked-change deletions and moved-away text in the output),
   `includeMissingRows` (spreadsheet row gaps)
+- `one-note-parser.includePageHistory` — OneNote text and metadata
+  (authors, timestamps, page GUIDs) include every earlier revision and saved
+  page version, not only each page's current content. Off, earlier authors
+  and dates are absent, as for the final state of an incrementally updated
+  PDF. Exception: `.one` (2010–2016) attachments come out either way,
+  including files only an earlier page version used. In `.one` files saved
+  by OneNote 2010–2016, history is not marked as such; it is mixed in with
+  the current text. In newer (FSSHTTPB) sections each saved page version is
+  its own `<div class="page history">`; edits within a version are merged.
+- `one-note-parser.includeDeletedContent` — newer (FSSHTTPB) OneNote
+  sections still store pages moved to the recycle bin; with this set they
+  come out as `<div class="page deleted">`, each older version as
+  `<div class="page deleted history">` (the text may be only in those).
 - `jsoup-parser.extractScripts` — script bodies in HTML (including HTML
   email bodies and HTML attachments) become their own `MACRO`-typed embedded
   entries instead of being silently dropped: JavaScript in a document is
@@ -268,7 +281,9 @@ The file's own story about its origin — read it as testimony, not fact:
   (`Normal.dotm` = stock Word), total edit time, claimed created/modified
   timestamps
 - **`tk:orig-resource-name`** — for xlsx, the absolute path where the file
-  was last saved (Excel records it in the workbook): drive letters, network
+  was last saved (Excel records it in the workbook); on an embedded entry
+  (OLE packages, OneNote attachments and inserted pictures), the path the
+  file had where it was inserted: drive letters, network
   shares, and `C:\Users\<name>\...` usernames leak here. (The `tk:` prefix
   here means Tika surfaced it; the *value* is still the file's own record —
   treat it as a claim like the rest of this section.)

.skills/users/file-forensics/file-forensics-config.json

@@ -28,6 +28,12 @@
         "includeMissingRows": true
       }
     },
+    {
+      "one-note-parser": {
+        "includePageHistory": true,
+        "includeDeletedContent": true
+      }
+    },
     {
       "jsoup-parser": {
         "extractScripts": true

CHANGES.txt

@@ -21,6 +21,15 @@ Release 4.3.0 - ???
      untrained scripts. JunkFilterEncodingDetector takes a HashedJunkModel instead of a
      TextQualityDetector (TIKA-5027).
 
+   * OneNote sections, .one and FSSHTTPB, give each page's current text and metadata
+     only, not every stored revision. Set includePageHistory on one-note-parser, in the
+     config or per request, for earlier revisions and saved page versions, and
+     includeDeletedContent for FSSHTTPB recycle-bin pages. All .one attachments are still
+     extracted. A file over a parse limit throws TikaMemoryLimitException instead of
+     falling back to the legacy text dump. OneNoteTreeWalkerOptions, the public
+     OneNoteLegacyDumpStrings settings and most public classes of the OneNote structure
+     model are removed (TIKA-5057).
+
    * FetcherManager.getFetcher and EmitterManager.getEmitter return a Lease
      (AbstractComponentManager.getComponent is now acquire); close it when done with
      the component and anything it returned. Pipes now closes a
@@ -169,6 +178,57 @@ Release 4.3.0 - ???
 
    * MP4 and QuickTime files with a box over 2 GB, typically an mdat in front of
      moov, keep their metadata: the walk skips such a box instead of stopping (TIKA-5051).
+
+   * OneNote FSSHTTPB sections extract each stored file once, not once per reference
+     to it (TIKA-5057).
+
+   * OneNote attachments and pictures carry their stored file name (resourceName) and,
+     for inserted files, the path they were inserted from (tk:orig-resource-name). .one
+     sections named none before, and an FSSHTTPB file's icon no longer takes the file's
+     name and ATTACHMENT type (TIKA-5057).
+
+   * OneNote note tag labels ("To Do", "Important", custom tags) are written before the
+     paragraphs they tag, as <p class="note-tag"> (TIKA-5057).
+
+   * OneNote pictures that are hyperlinks are written inside <a href>, so their URLs reach
+     link handlers (TIKA-5057).
+
+   * OneNoteParser parses .onetoc2 notebook tables of contents, listing their section
+     names (TIKA-5057).
+
+   * OneNote sections that fell back to the legacy text dump now parse structurally,
+     including OneNote 2007 sections, Aspose.Note files and sections with leftover bytes
+     from earlier saves. OneNote 2003 sections fall back to the dump instead of failing
+     (TIKA-5057).
+
+   * OneNote text that was dropped or garbled is extracted: objects edited in later
+     revisions, text behind element child references, and paragraphs with hyperlinks
+     (now <a href>). Extended ASCII is read as windows-1252, and NULs and U+FFFC
+     placeholders are removed (TIKA-5057).
+
+   * OneNote output adds pictures as <img> with alt text, OneNote's OCR text as
+     <div class="ocr">, and .one handwriting recognition as <span class="handwriting">
+     (TIKA-5057).
+
+   * Password-protected OneNote sections throw EncryptedDocumentException; FSSHTTPB
+     ones were never detected because their GUIDs were MD5-hashed. dc:creator includes
+     original authors. A file shorter than its header throws a TikaException
+     (TIKA-5057).
+
+   * OneNote parsing bounds CPU and memory on crafted files with limits scaled to file
+     size, and rejects more malformed structures up front (TIKA-5057).
+
+   * OneNote .one sections allocate far less and small ones parse much faster, reading
+     through a small page cache instead of one system call per field. Large
+     multi-revision sections can be slower, as every committed revision is now read
+     (TIKA-5057).
+
+   * OneNote legacy text dump: strings crossing a 1 MB read boundary are written once,
+     and strings are judged by the text-quality detector (tika-ml-junkdetect) when it is
+     on the class path (TIKA-5057).
+
+   * tika-ml-junkdetect: JunkDetector has a public no-arg constructor, so ServiceLoader
+     finds it on the class path (TIKA-5057).
 
    * WOFF and WOFF2 fonts (font/woff, font/woff2), SquashFS images
      (application/vnd.squashfs), Adobe Swatch Exchange files, PalmDOC e-books and binary

docs/modules/ROOT/pages/advanced/junk-detection.adoc

@@ -18,12 +18,15 @@
 = Text Quality Scoring (Junk Detection)
 
 The `tika-ml-junkdetect` module scores how much a string reads like real text, in any
-language it was trained on, without being told the language.  Tika uses it in two places:
+language it was trained on, without being told the language.  Tika uses it in three places:
 
 * **charset detection**: of several decodes of the same bytes, which one is the real text
   (see xref:advanced/charset-detection-design.adoc[Charset Detection Pipeline]);
 * **tika-eval**: the `tika-eval:languageness` score of extracted text, plus the worst window
-  and coverage.
+  and coverage;
+* **OneNote legacy text dump**: which scraped strings to keep when a OneNote file cannot be
+  parsed structurally. Without `tika-ml-junkdetect` on the class path the dump falls back
+  to a letter-ratio check.
 
 == What it detects
 

docs/modules/ROOT/pages/formats.adoc

@@ -70,7 +70,11 @@ and
 link:{api}/org/apache/tika/parser/microsoft/ooxml/xwpf/ml2006/Word2006MLParser.html[Word2006MLParser].
 
 link:{api}/org/apache/tika/parser/microsoft/onenote/OneNoteParser.html[OneNoteParser]
-extracts text from OneNote notebooks, and
+extracts text from OneNote sections (`.one`) and notebook tables of contents
+(`.onetoc2`); by default only each page's current version. Set
+`includePageHistory` on `one-note-parser` for earlier versions and
+`includeDeletedContent` for pages in the recycle bin (newer, FSSHTTPB
+sections only), in the config or per request.
 link:{api}/org/apache/tika/parser/microsoft/MSOwnerFileParser.html[MSOwnerFileParser]
 pulls basic metadata from the temporary lock ("owner") files Office leaves
 beside open documents.

docs/modules/ROOT/partials/supported-formats.adoc

@@ -224,6 +224,7 @@
 ** `application/x-chm`
 * `org.apache.tika.parser.microsoft.onenote.` link:{tika-javadoc-url}/org/apache/tika/parser/microsoft/onenote/OneNoteParser.html[OneNoteParser]
 ** `application/onenote; format=one`
+** `application/onenote; format=onetoc2`
 * `org.apache.tika.parser.microsoft.ooxml.` link:{tika-javadoc-url}/org/apache/tika/parser/microsoft/ooxml/OOXMLParser.html[OOXMLParser]
 ** `application/vnd.ms-excel.addin.macroenabled.12`
 ** `application/vnd.ms-excel.sheet.binary.macroenabled.12`

tika-ml/tika-ml-junkdetect/src/main/java/org/apache/tika/ml/junkdetect/JunkDetector.java

@@ -54,23 +54,49 @@ public final class JunkDetector implements TextQualityDetector {
      */
     public static final float MIN_COVERAGE = 0.9f;
 
+    private static volatile HashedJunkModel bundledModel;
+
     private final HashedJunkModel model;
 
+    /**
+     * Loads the bundled model. {@link java.util.ServiceLoader} needs this constructor on the
+     * class path; {@link #provider()} serves only the module path.
+     *
+     * @throws UncheckedIOException if the bundled model cannot be loaded
+     */
+    public JunkDetector() {
+        this(provider().model);
+    }
+
     public JunkDetector(HashedJunkModel model) {
         if (model == null) {
             throw new IllegalArgumentException("model");
         }
         this.model = model;
     }
 
-    /** Loads the bundled model. */
+    /** The bundled model, read once and shared: it is immutable and several megabytes. */
     public static JunkDetector loadFromClasspath() throws IOException {
+        HashedJunkModel model = bundledModel;
+        if (model == null) {
+            synchronized (JunkDetector.class) {
+                model = bundledModel;
+                if (model == null) {
+                    model = readBundledModel();
+                    bundledModel = model;
+                }
+            }
+        }
+        return new JunkDetector(model);
+    }
+
+    private static HashedJunkModel readBundledModel() throws IOException {
         InputStream is = JunkDetector.class.getClassLoader().getResourceAsStream(DEFAULT_MODEL_RESOURCE);
         if (is == null) {
             throw new IOException("Model resource not found on classpath: " + DEFAULT_MODEL_RESOURCE);
         }
         try (InputStream in = is) {
-            return new JunkDetector(HashedJunkModel.load(in));
+            return HashedJunkModel.load(in);
         }
     }
 

tika-ml/tika-ml-junkdetect/src/test/java/org/apache/tika/ml/junkdetect/JunkDetectorSmokeTest.java

@@ -22,11 +22,13 @@
 
 import java.nio.charset.StandardCharsets;
 import java.util.Random;
+import java.util.ServiceLoader;
 
 import org.junit.jupiter.api.BeforeAll;
 import org.junit.jupiter.api.Test;
 
 import org.apache.tika.quality.TextQualityComparison;
+import org.apache.tika.quality.TextQualityDetector;
 import org.apache.tika.quality.TextQualityScore;
 
 /**
@@ -43,6 +45,15 @@ static void loadModel() throws Exception {
         detector = JunkDetector.loadFromClasspath();
     }
 
+    @Test
+    public void testServiceLoaderFindsTheBundledDetector() {
+        // the registration needs a public no-arg constructor on the class path
+        TextQualityDetector detector = ServiceLoader.load(TextQualityDetector.class)
+                .findFirst().orElseThrow();
+        assertTrue(detector instanceof JunkDetector, detector.getClass().getName());
+        assertFalse(detector.score("The quick brown fox jumps over the lazy dog.").isUnknown());
+    }
+
     /**
      * Clean English should score higher than random high-byte garbage interpreted
      * as ISO-8859-1.  Simulates binary data mixed into a text extraction.

tika-parent/pom.xml

@@ -1710,7 +1710,6 @@
               <exclude>**/CharsetDetector.java</exclude>
               <exclude>**/CharsetMatch.java</exclude>
               <exclude>**/CharsetRecog*.java</exclude>
-              <exclude>**/onenote/fsshttpb/unsigned/U*.java</exclude>
               <exclude>**/IPTC.java</exclude>
               <exclude>**/Photoshop.java</exclude>
               <exclude>**/QuattroPro.java</exclude>

tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/CompactID.java

@@ -20,36 +20,4 @@ class CompactID {
     char n;
     long guidIndex; //only occupies 24 bits
     ExtendedGUID guid;
-
-    public char getN() {
-        return n;
-    }
-
-    public CompactID setN(char n) {
-        this.n = n;
-        return this;
-    }
-
-    public long getGuidIndex() {
-        return guidIndex;
-    }
-
-    public CompactID setGuidIndex(long guidIndex) {
-        this.guidIndex = guidIndex;
-        return this;
-    }
-
-    public ExtendedGUID getGuid() {
-        return guid;
-    }
-
-    public CompactID setGuid(ExtendedGUID guid) {
-        this.guid = guid;
-        return this;
-    }
-
-    public String getCompactIDString() {
-        return new StringBuilder().append(guid).append(", index=").append(guidIndex).append(", n=")
-                .append((int) n).toString();
-    }
 }

tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/ExtendedGUID.java

@@ -16,12 +16,9 @@
  */
 package org.apache.tika.parser.microsoft.onenote;
 
-import java.util.ArrayList;
-import java.util.List;
 import java.util.Locale;
 import java.util.Objects;
 
-import org.apache.tika.parser.microsoft.onenote.fsshttpb.util.BitConverter;
 
 public class ExtendedGUID implements Comparable<ExtendedGUID> {
     GUID guid;
@@ -62,46 +59,12 @@ public boolean equals(Object o) {
 
     @Override
     public int hashCode() {
-        return Objects.hash(guid, n);
+        // Objects.hash(guid, n) without its varargs array and boxing
+        return 31 * (31 + Objects.hashCode(guid)) + Long.hashCode(n);
     }
 
     @Override
     public String toString() {
         return String.format(Locale.US, "%s [%d]", guid, n);
     }
-
-    public GUID getGuid() {
-        return guid;
-    }
-
-    public ExtendedGUID setGuid(GUID guid) {
-        this.guid = guid;
-        return this;
-    }
-
-    public String getExtendedGuidString() {
-        return guid.toString() + " [" + n + "]";
-    }
-
-    public long getN() {
-        return n;
-    }
-
-    public ExtendedGUID setN(long n) {
-        this.n = n;
-        return this;
-    }
-
-    /**
-     * This method is used to convert the element of ExtendedGUID object into a byte List.
-     *
-     * @return Return the byte list which store the byte information of ExtendedGUID
-     */
-    public List<Byte> SerializeToByteList() {
-        List<Byte> byteList = new ArrayList<>(guid.toByteArray());
-        for (byte b : BitConverter.getBytes(n)) {
-            byteList.add(b);
-        }
-        return byteList;
-    }
 }