Skip to content

Tokenizer fails on iOS (RE2 lookahead unsupported) – need regex_lookahead static lib or guidance #16391

@quocanh0712

Description

@quocanh0712

🐛 Describe the bug

Summary
iOS Flutter app using ExecuTorch LLM (Qwen3 0.6B) cannot load the tokenizer because RE2 does not support lookahead (?!\S).
SPM branch: swiftpm-1.1.0.20251223 (no visible regex_lookahead target/lib).
Logs ask to link regex_lookahead, but SPM did not produce the static lib.
Environment
Platform: iOS Simulator (iPhone 16 Pro), macOS, Xcode 15.
ExecuTorch via SwiftPM branch swiftpm-1.1.0.20251223.
App: Flutter, native plugin calling TextRunner.load(modelPath, tokenizerPath).
Model: qwen3_0.6B_model.pte (~518MB).
Tokenizer: tokenizer (1).json (~11MB) containing lookahead.
Logs (Xcode)
E re2.cc:237 Error parsing ... invalid perl operator: (?!E tokenizers:regex.cpp:66 RE2 doesn't support lookahead patterns. Link with regex_lookahead to enable support.I tokenizers:hf_tokenizer.cpp:166 Could not parse pre_tokenizer: Error: 9
What I’ve tried
Patched tokenizer to remove (?!\S) → error disappears, but this is a workaround.
Searched for libregex_lookahead*.a in DerivedData: not found (this SPM branch doesn’t seem to include it).
Backends force-loaded fine; only regex_lookahead is missing.
Questions / help needed

  1. Does the swiftpm-1.1.0.x branch ship a regex_lookahead target/static lib? If yes, how to enable it so SPM produces libregex_lookahead.a?
  2. If not, can you provide guidance or a prebuilt libregex_lookahead.a (simulator/device) for manual linking?
  3. Is there a “clean” tokenizer (no lookahead) recommended for Qwen3 0.6B in the ExecuTorch LLM samples?
    More info
    I can share the 11MB tokenizer via a private link if needed.

Versions

swiftpm-1.1.0.20251223

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions