-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RAG/Knowledge for No ascii characters #2376
Comments
Hello @jolestar! Welcome to the elizaOS community. Thank you for opening your first issue; we appreciate your contribution. You are now an elizaOS contributor! |
Hello @jolestar I thnk that flter is somehow added to avoid some techniques of prompt injects. |
This issue has been automatically marked as stale due to 30 days of inactivity. |
Is there some conclusion about this? @AIFlowML |
Describe the bug
The knowledge preprocessing function incorrectly removes non-ASCII characters like Chinese text due to an overly restrictive regex pattern that only allows ASCII alphanumeric characters.
To Reproduce
Expected behavior
The preprocess function should preserve non-ASCII characters like Chinese text
The text was updated successfully, but these errors were encountered: