Import unicode-normalization or re-write from scratch?

[sffc]

@markusicu has done a great deal of work on ICU4C's normalizer. It depends on low-level and highly optimized data structures such as UCPTrie.

Writing normalization code from a clean room would allow us to:

1. Use the same core algorithms as ICU4C, allowing better interop of code, data, and clients
2. Build in proper string handling (String encodings (UTF-8/UTF-16) #14)
3. Integrate it with ICU4X's locale data pipeline (including UCD data)

[sffc]

@Manishearth @hsivonen @zbraniecki

[macchiati]

+1

…

On Fri, Apr 17, 2020, 17:26 Shane F. Carr ***@***.***> wrote: @Manishearth <https://github.com/Manishearth> @hsivonen <https://github.com/hsivonen> @zbraniecki <https://github.com/zbraniecki> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#40 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACJLEMC4WFUGTBCIMGIOHLLRNDXTHANCNFSM4MLEESZQ> .

[Manishearth]

I'm okay with cleanroom. We can perhaps use the same API as unicode-normalization.

We could also import it and gradually optimize. I don't have opinions on this.

[hsivonen]

Use the same core algorithms as ICU4C, allowing better interop of code, data, and clients

How do ICU4C and unicode-normalization compare in performance?

We can perhaps use the same API as unicode-normalization.

unicode-normalization uses an iterator over char-based API, which isn't FFI-friendly. I expect we'll end up with that API for Rust callers only and will also a slice-based API for FFI concerns (also for Rust callers that have a slice).

[sffc]

@hsivonen @zbraniecki @Manishearth Have you had an opportunity to do performance testing of ICU4C Normalizer versus the existing Rust Normalizer?

[Manishearth]

Not I, maybe Zibi?

[zbraniecki]

I have not. I can perform if someone more experience with normalization gives me the test samples and calls to test against.

[macchiati]

I think we have some test data. We also test against the Unicode test files, but those are not representative text for performance purposes. Best is a mixture of text from different languages, with the proportion = frequency of text in that language on the web. Mark

…

On Wed, May 13, 2020 at 4:55 PM Zibi Braniecki ***@***.***> wrote: I have not. I can perform if someone more experience with normalization gives me the test samples and calls to test against. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#40 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACJLEMCK6ZQ4PPXYNBZII2DRRMXNVANCNFSM4MLEESZQ> .

[hsivonen]

IIRC, we assigned this to Zibi and not me on last week's call.

[sffc]

See Henri's comment in #66 (comment)

[nciric]

#93 gives us not so clear messaging on what to do in this case (except that current implementation lags behind ICU4C metrics).

If we need to implement data provider/loading in addition to optimization steps, it may be easier to do full rewrite. If anybody needs normalization now they can either use existing crate, or even go with rust_icu that wraps ICU4C.

[sffc]

Re-writing normalizer in ICU4X still has the benefits discussed in the OP. The concern from @hsivonen was that if ICU's normalizer is super slow, then maybe we shouldn't go that route. However, now that we've put that concern to rest, I think we can revisit the advantages of an ICU-based normalizer implementation.

[sffc]

Putting in backlog. Comments from team:

• Manish: The hard part is data loading. It's probably the same amount of work to re-write from scratch than retrofit the existing crate.
• Steven: Having an existing implementation to compare against is good.
• Cira: Having prior art is useful for us to implement.

[filmil]

It's about 2 hours to make unorm.h avaiable in rust. Would this help move things along? For example, it seems like it would enable work on Segmenter (#109) since you won't need to wait until rust native normalization is ready.

[sffc]

It's about 2 hours to make unorm.h avaiable in rust. Would this help move things along? For example, it seems like it would enable work on Segmenter (#109) since you won't need to wait until rust native normalization is ready.

We don't have anyone to work on either Normalizer or Segmenter, so I don't think this is a priority at the current time, although it might be nice to have ready to go.

[hsivonen]

ICU4X has had a normalizer since 1.0.