-
Notifications
You must be signed in to change notification settings - Fork 778
Description
🐛 Describe the bug
Summary
iOS Flutter app using ExecuTorch LLM (Qwen3 0.6B) cannot load the tokenizer because RE2 does not support lookahead (?!\S).
SPM branch: swiftpm-1.1.0.20251223 (no visible regex_lookahead target/lib).
Logs ask to link regex_lookahead, but SPM did not produce the static lib.
Environment
Platform: iOS Simulator (iPhone 16 Pro), macOS, Xcode 15.
ExecuTorch via SwiftPM branch swiftpm-1.1.0.20251223.
App: Flutter, native plugin calling TextRunner.load(modelPath, tokenizerPath).
Model: qwen3_0.6B_model.pte (~518MB).
Tokenizer: tokenizer (1).json (~11MB) containing lookahead.
Logs (Xcode)
E re2.cc:237 Error parsing ... invalid perl operator: (?!E tokenizers:regex.cpp:66 RE2 doesn't support lookahead patterns. Link with regex_lookahead to enable support.I tokenizers:hf_tokenizer.cpp:166 Could not parse pre_tokenizer: Error: 9
What I’ve tried
Patched tokenizer to remove (?!\S) → error disappears, but this is a workaround.
Searched for libregex_lookahead*.a in DerivedData: not found (this SPM branch doesn’t seem to include it).
Backends force-loaded fine; only regex_lookahead is missing.
Questions / help needed
- Does the swiftpm-1.1.0.x branch ship a regex_lookahead target/static lib? If yes, how to enable it so SPM produces libregex_lookahead.a?
- If not, can you provide guidance or a prebuilt libregex_lookahead.a (simulator/device) for manual linking?
- Is there a “clean” tokenizer (no lookahead) recommended for Qwen3 0.6B in the ExecuTorch LLM samples?
More info
I can share the 11MB tokenizer via a private link if needed.
Versions
swiftpm-1.1.0.20251223