feat(core): add configurable CHUNK_LIMIT environment variable

[RauchenwaldC]

Summary

• Add CHUNK_LIMIT environment variable support for configurable chunk processing limits
• Default value remains 450000 to maintain backward compatibility
• Set minimum value of 1000 to prevent issues with very small limits
• Add configuration documentation to .env.example
• Add console logging to show configured CHUNK_LIMIT value at runtime

Motivation

Currently, the CHUNK_LIMIT is hardcoded to 450000 in the processFileList method. This makes it impossible for users to customize the chunk limit for their specific use cases without modifying the source code. Large codebases may need lower limits, while smaller codebases or users with more processing power may want higher limits.

Changes Made

1. Modified packages/core/src/context.ts:

   • Changed hardcoded CHUNK_LIMIT from 450000 to use envManager.get('CHUNK_LIMIT') || '450000'
   • Added Math.max(1000, ...) to enforce minimum value
   • Added console logging for the configured CHUNK_LIMIT value

2. Updated .env.example:

   • Added CHUNK_LIMIT configuration with documentation
   • Included usage notes and default value information

Test Plan

• Built the project successfully with pnpm build
• Verified backward compatibility (default value 450000 maintained)
• Confirmed minimum value enforcement (1000)
• Added proper documentation in .env.example

Breaking Changes

None - this is a backward compatible enhancement.

Additional Notes

The change follows the same pattern used for EMBEDDING_BATCH_SIZE configuration in the same function, maintaining consistency with existing environment variable handling.