feat(core): add optional formatted data cache for large dataset exports and rendering

[ghiscoding]

full implementation vibe coded with Copilot and Claude Sonnet 4.6

Summary

This PR introduces an opt-in formatted data cache pipeline to improve export throughput and post-warmup grid rendering responsiveness on large datasets.

It adds asynchronous background cache population in SlickDataView, keyed by item id, for:

• export formatted values (formattedDataCache)
• cell display formatter output (formattedCellCache)

The pipeline uses batched, time-budgeted processing (MessageChannel with requestAnimationFrame fallback), supports row-level and full cache invalidation on data changes, and preserves existing cache-disabled/cache-miss behavior.

SlickGrid now reads from the display cache in formatter flow when safe, and Excel/PDF/Text export paths use cache-first reads with fallback to live formatter logic so sanitize/decode/parsing behavior remains consistent.

Architecture note

The implementation is planner-based:

• SlickGrid provides a single formatted cache planner callback
• SlickDataView uses that planner to classify columns and build cache context
• Export services consume the shared cache (no per-service consumer registration lifecycle)

Validation

Validation was done with focused unit tests for:

• cache disabled vs cache miss vs cache hit parity
• sanitize/decode-sensitive output paths
• planner wiring and refresh behavior
• async batching and fallback scheduling paths

Tests and coverage are green on this branch.

Why this helps

This specifically targets formatter-heavy, large-grid workloads like the scenario discussed here:
#1922

Impact

• Faster exports on large datasets (Excel/PDF/Text), especially with many formatted columns
• Smoother grid rendering after warmup by reusing cached formatter outputs
• Background population keeps UI responsive during warmup
• No functional output change intended: cache-miss and cache-disabled paths still use existing logic
• Backward compatible: feature is opt-in and disabled by default
• Tradeoff: higher memory usage, so enable selectively for large/formatter-heavy grids (commonly 10K+ rows)

Technical implementation

See FORMATTED_DATA_CACHE_IMPLEMENTATION.md

---

Live Demo Test (Example 03 with 500K rows)

Excellent results! 🎉 That's a massive validation of the implementation:

Performance Summary (500K rows, 8 columns):

• Cache Population: ~13 minutes (779,927 ms) for 4M cells
  • Runs in background with MessageChannel + requestAnimationFrame batching
  • UI stays responsive during population
• Export Time: 30-35 seconds (consistent regardless of grouping/sorting)
  • Fast because cache hits eliminate formatter overhead
  • DataView's built-in filtering handles visible rows automatically
• Previous Behavior: Browser hang (unusable)

What makes this successful:

1. ✅ Cache-first reads in export services (formatter → cache fallback)
2. ✅ DataView cleanup on destroy (clearFormattedDataCache())
3. ✅ Export service cleanup on dispose (prevents memory leaks)
4. ✅ Planner-based architecture (single callback, selective caching)
5. ✅ Batch scheduling with time budgets (keeps UI responsive)

The cache is doing exactly what it's designed for—trading a one-time 13-minute background population for consistent 30-35 second exports on huge datasets. Without the cache, that export would likely take 5+ minutes or hang.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.0%. Comparing base (e55de58) to head (012a40f).

Additional details and impacted files

@@           Coverage Diff            @@
##           master    #2563    +/-   ##
========================================
  Coverage   100.0%   100.0%            
========================================
  Files         196      196            
  Lines       24994    25275   +281     
  Branches     8832     8925    +93     
========================================
+ Hits        24994    25275   +281     

Flag	Coverage Δ
angular	100.0% <100.0%> (ø)
universal	100.0% <100.0%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[pkg-pr-new]

angular-slickgrid

npm i https://pkg.pr.new/angular-slickgrid@2563

aurelia-slickgrid

npm i https://pkg.pr.new/aurelia-slickgrid@2563

slickgrid-react

npm i https://pkg.pr.new/slickgrid-react@2563

slickgrid-vue

npm i https://pkg.pr.new/slickgrid-vue@2563

@slickgrid-universal/angular-row-detail-plugin

npm i https://pkg.pr.new/@slickgrid-universal/angular-row-detail-plugin@2563

@slickgrid-universal/aurelia-row-detail-plugin

npm i https://pkg.pr.new/@slickgrid-universal/aurelia-row-detail-plugin@2563

@slickgrid-universal/react-row-detail-plugin

npm i https://pkg.pr.new/@slickgrid-universal/react-row-detail-plugin@2563

@slickgrid-universal/vue-row-detail-plugin

npm i https://pkg.pr.new/@slickgrid-universal/vue-row-detail-plugin@2563

@slickgrid-universal/binding

npm i https://pkg.pr.new/@slickgrid-universal/binding@2563

@slickgrid-universal/common

npm i https://pkg.pr.new/@slickgrid-universal/common@2563

@slickgrid-universal/composite-editor-component

npm i https://pkg.pr.new/@slickgrid-universal/composite-editor-component@2563

@slickgrid-universal/custom-footer-component

npm i https://pkg.pr.new/@slickgrid-universal/custom-footer-component@2563

@slickgrid-universal/custom-tooltip-plugin

npm i https://pkg.pr.new/@slickgrid-universal/custom-tooltip-plugin@2563

@slickgrid-universal/empty-warning-component

npm i https://pkg.pr.new/@slickgrid-universal/empty-warning-component@2563

@slickgrid-universal/event-pub-sub

npm i https://pkg.pr.new/@slickgrid-universal/event-pub-sub@2563

@slickgrid-universal/excel-export

npm i https://pkg.pr.new/@slickgrid-universal/excel-export@2563

@slickgrid-universal/graphql

npm i https://pkg.pr.new/@slickgrid-universal/graphql@2563

@slickgrid-universal/odata

npm i https://pkg.pr.new/@slickgrid-universal/odata@2563

@slickgrid-universal/pagination-component

npm i https://pkg.pr.new/@slickgrid-universal/pagination-component@2563

@slickgrid-universal/pdf-export

npm i https://pkg.pr.new/@slickgrid-universal/pdf-export@2563

@slickgrid-universal/row-detail-view-plugin

npm i https://pkg.pr.new/@slickgrid-universal/row-detail-view-plugin@2563

@slickgrid-universal/rxjs-observable

npm i https://pkg.pr.new/@slickgrid-universal/rxjs-observable@2563

@slickgrid-universal/sql

npm i https://pkg.pr.new/@slickgrid-universal/sql@2563

@slickgrid-universal/text-export

npm i https://pkg.pr.new/@slickgrid-universal/text-export@2563

@slickgrid-universal/utils

npm i https://pkg.pr.new/@slickgrid-universal/utils@2563

@slickgrid-universal/vanilla-bundle

npm i https://pkg.pr.new/@slickgrid-universal/vanilla-bundle@2563

@slickgrid-universal/vanilla-force-bundle

npm i https://pkg.pr.new/@slickgrid-universal/vanilla-force-bundle@2563

commit: 012a40f

[ghiscoding]

@zewa666 @jr01 are you guys dealing with very large dataset or always using a Backend Service like OData to avoid large local dataset? If you're always using Backend Service then I guess this PR won't change anything for you, but just curious...

I know that this could help a lot for use case like @kevinburkett had in #1922

the PR does add 230 new lines of code (that is what codecov says anyway) but I think it's totally worth having it in the core (SlickDataView) and it's opt-in, so user will have to enable it themselves (mainly because that will consume memory and takes time to populate the cache but doesn't impact UI responsiveness, that was an important condition: build the cache without impacting UI responsiveness). On the Example02 with 8 columns and 50K rows, it takes 23sec to run the cache. I even tried with 500K in Example 03 and it takes a lot longer to build the cache but once it's available then the export no longer freezes the browser

I basically thought of doing this caching to decrease export time, and this can always be used to improve rendering time as well for large and very large dataset (build the cache and use it whenever possible). For export services, that is definitely a huge gain in perf (in other words a big drop in time it takes to export but only after the cache is ready, no need to reparse formatters every time). Why would we have very large dataset? I guess Grouping or Tree Data could be a factor (since those don't work with Backend Services)

[zewa666]

we're strictly with paginated views (20-50) eith backend datasets. but the treeview still gets quite crowded as that one is an exception

[ghiscoding]

yeah I guess that will be nice for only a limited subset of users, but still great to add in the core nonetheless, at least I would think so :)