Original character positions incorrect in Spanish annotation

[hans]

@manning reported second-hand that character offset annotations seem to be wrong for Spanish NER output. Investigate.

[hans]

Fixed in 2b0c701.

[manning]

As far as I can see this still isn't correct.

See the test testOffsetsSpacing() that I now have in SpanishTokenizerITest. It's checked in passing, but the answers seem to be wrong for postverbal clitics, no? See the todo items.

[hans]

528215f fixes the TODOs in the integration tests. I'll get a final integration test set up this week that verifies offset / spacing issues once and for all.

[AngledLuffa]

There's still one leftover comment in there, which I've taken no effort to understand:

    // y de el y
    testOffsetsTextOriginalText("y del y", new int[] {0, 2, 3, 6}, new int[] {1, 3, 5, 7},
            new String[] { "y", "de", "el", "y"},
            new String[] { "y", "de", "el", "y"});  // todo [cdm 2017]: it's very unclear if this is what we actually want! Overlaps, concatenation doesn't work.
    // according to offsets, it should be "d" + "el"

What's the plan for this one or the token del in general?