RAG/Knowledge for No ascii characters

[jolestar]

Describe the bug

The knowledge preprocessing function incorrectly removes non-ASCII characters like Chinese text due to an overly restrictive regex pattern that only allows ASCII alphanumeric characters.

To Reproduce

1. Create a knowledge item with non-ASCII characters like Chinese text
2. Call the preprocess function with this content:

const input = "你好 Eliza";
knowledge.preprocess(input)

3. The Chinese characters are removed from the output

Expected behavior

The preprocess function should preserve non-ASCII characters like Chinese text

[github-actions]

Hello @jolestar! Welcome to the elizaOS community. Thank you for opening your first issue; we appreciate your contribution. You are now an elizaOS contributor!

[AIFlowML]

Hello @jolestar I thnk that flter is somehow added to avoid some techniques of prompt injects.
We will address this soon in a more sophisticated way.

[github-actions]

This issue has been automatically marked as stale due to 30 days of inactivity.
If no further activity occurs within 7 days, it will be closed automatically. Please take action if this issue is still relevant.

[jolestar]

Is there some conclusion about this? @AIFlowML