New modifier type `token-optimizer`

[butschster]

Compiled documents generated can become too large and heavy in tokens, which impacts LLM usage costs and performance.

Many times, the content includes unnecessary characters like excessive newlines, whitespace, long comments, and other non-essential parts.

Proposal

Introduce a new modifier called token-optimizer that will help reduce token count in compiled documents without affecting meaning. It should work after all other modifiers and clean up the final content.

Key Features

Option	Type	Default	Description
remove_newlines	bool	true	Remove empty lines and extra newlines.
trim_whitespace	bool	true	Trim leading/trailing spaces from each line.
collapse_spaces	bool	true	Reduce multiple spaces to a single space.
remove_comments	bool	false	Remove inline comments (// or # or /* ... */).
minify_json	bool	false	Minify JSON sources (if used in sources).
max_line_length	int	0	Optional: If >0, collapse lines to fit in max length.

Example Usage

documents:
  - description: Optimized API Docs
    outputPath: docs/optimized-api.md
    sources:
      - type: file
        sourcePaths:
          - src/Api
        modifiers:
          - php-docs
    modifiers:
      - name: token-optimizer
        options:
          remove_newlines: true
          trim_whitespace: true
          collapse_spaces: true
          remove_comments: true