sf data bulk export fails, ERR_STRING_TOO_LONG

[tggagne]

Expected result

sf data bulk export should have exported 700K+ records

Actual result

node:buffer:636
    slice: (buf, start, end) => buf.utf8Slice(start, end),
                                    ^

Error: Cannot create a string longer than 0x1fffffe8 characters
    at Object.slice (node:buffer:636:37)
    at Buffer.toString (node:buffer:864:14)
    at MemoryWriteStream.toString (/Users/tgagne/.local/share/sf/client/2.120.3-0e363e5/node_modules/@jsforce/jsforce-node/lib/util/stream.js:43:44)
    at MemoryWriteStream.<anonymous> (/Users/tgagne/.local/share/sf/client/2.120.3-0e363e5/node_modules/@jsforce/jsforce-node/lib/util/stream.js:58:24)
    at MemoryWriteStream.emit (node:events:531:35)
    at finish (node:internal/streams/writable:955:10)
    at node:internal/streams/writable:936:13
    at process.processTicksAndRejections (node:internal/process/task_queues:90:21) {
  code: 'ERR_STRING_TOO_LONG'
}

Node.js v22.21.1

Additional information

System Information

CLI:
@salesforce/cli/2.123.1 darwin-arm64 node-v22.22.0

Plugin Version:
@oclif/plugin-autocomplete 3.2.40 (core)
@oclif/plugin-commands 4.1.40 (core)
@oclif/plugin-help 6.2.37 (core)
@oclif/plugin-not-found 3.2.74 (core)
@oclif/plugin-plugins 5.4.56 (core)
@oclif/plugin-search 1.2.38 (core)
@oclif/plugin-update 4.7.19 (core)
@oclif/plugin-version 2.2.36 (core)
@oclif/plugin-warn-if-update-available 3.1.55 (core)
@oclif/plugin-which 3.2.43 (core)
@salesforce/cli 2.123.1 (core)
agent 1.29.0 (core)
apex 3.9.6 (core)
api 1.3.9 (core)
auth 4.1.5 (core)
code-analyzer 5.10.0 (user)
data 4.0.68 (core)
deploy-retrieve 3.24.8 (core)
info 3.4.101 (core)
limits 3.3.75 (core)
marketplace 1.3.10 (core)
omnistudio-migration-tool 1.7.0 (user)
org 5.9.66 (core)
packaging 2.25.1 (core)
schema 3.3.91 (core)
settings 2.4.55 (core)
signups 2.6.56 (user)
sobject 1.4.82 (core)
telemetry 3.7.2 (core)
templates 56.4.7 (core)
trust 3.7.113 (core)
user 3.6.49 (core)
@salesforce/sfdx-plugin-lwc-test 1.2.1 (user)
@salesforce/sfdx-scanner 4.12.0 (user)
sfdx-git-delta 6.32.2 (user)
vlocityestools 0.25.1 (user)

SF ENV. VARS.
SF_DISABLE_AUTOUPDATE,true
SF_BINPATH,/Users/tgagne/.local/share/sf/client/bin/sf
SF_DISABLE_SOURCE_MEMBER_POLLING,true
SF_SKIP_NEW_VERSION_CHECK,true
SF_UPDATE_INSTRUCTIONS,Use "npm update --global @salesforce/cli" to update npm-based installations.

Windows: false
Shell: zsh
Channel: stable

---

Diagnostics

✅ pass - salesforcedx plugin isn’t installed
✅ pass - you don't have any linked plugins
✅ pass - [@salesforce/plugin-trust] can ping: https://registry.npmjs.org
✅ pass - [@salesforce/plugin-trust] can ping: https://registry.yarnpkg.com
✅ pass - [@salesforce/plugin-trust] can ping: https://registry.npmjs.org/
✅ pass - using latest or latest-rc CLI version
❌ warn - [@salesforce/plugin-deploy-retrieve] sourceApiVersion matches apiVersion
✅ pass - can access: https://test.salesforce.com
✅ pass - can access: https://appexchange.salesforce.com/services/data
✅ pass - can access: https://developer.salesforce.com/media/salesforce-cli/sf/channels/stable/sf-win32-x64-buildmanifest
❌ fail - [@salesforce/plugin-auth] CLI supports v2 crypto

[tggagne]

I believe the problem is with 130K-wide text fields. There are three of them in the original query. When I eliminate them the query completes. Those columns are for json requests and responses.

Is there a way it could work using smaller batch sizes? Is there a way to control the memory node uses?

[git2gus]

This issue has been linked to a new work item: W-21348168

[cristiand391]

Is there a way it could work using smaller batch sizes? Is there a way to control the memory node uses?

Error: Cannot create a string longer than 0x1fffffe8 characters points to a hard limit on V8 (JS engine), don't see this can be overridden by node flags.

This happens because jsforce is buffering the bulk result in memory:
https://github.com/salesforcecli/plugin-data/blob/9bc987ed2621997110cdc947c865abb453c04387/src/bulkUtils.ts#L89

response is written to MemoryWriteStream, then it tries to resolve with its content as string:
https://github.com/jsforce/jsforce/blob/08e19b679c6a2aa1059bf452120594edf2793c43/src/util/stream.ts#L69.

Bulk export results get split across multiple batches (decided by the org), then we fetch each batch here:
https://github.com/salesforcecli/plugin-data/blob/9bc987ed2621997110cdc947c865abb453c04387/src/bulkUtils.ts#L127-L192

You are hitting this error scenario because one of these batches is bigger than the V8 limit.

I see 2 possible fixes:

1. bypass jsforce and any intermediate buffering in the command.
   This is because the bulk v2 api doesn't stream results, so we always get a huge batch to process.
   I'm cooking a patch for you to try this possible fix in a bit.

2. Add support for fetching batches in small chunks via the maxRecords query param:

While we can't control how many batches are created in the org, we could fetch 1 batch in parts by exposing the maxRecords param as a flag (we'll need to implement paging for this):
https://developer.salesforce.com/docs/atlas.en-us.api_asynch.meta/api_asynch/query_get_job_results.htm

[cristiand391]

@tggagne

here's a prerelease of the bulk command with a possible fix if you want to give it a try:

1. install plugin: sf plugins install @salesforce/plugin-data@4.0.74-dev.0
2. do bulk export, validate it worked fine

[tggagne]

@cristiand391 , that worked!

% sf data export bulk --wait 30 -q "SELECT f1__c, f2__c... from BigTable__c" --output-file=output.csv -o orgalias
 ───────────────── Exporting data ─────────────────

 ✔ creating query job 644ms
 ✔ processing the job 7m 23.24s
 ✔ exporting records 9m 4.53s

 Status: JobComplete
 Job Id: 750VQ00000kkhifYAA (​https://secret.my.salesforce.com/lightning/setup/AsyncApiJobStatus/page?address=%2F750VQ00000kkhifYAA​)
 Elapsed Time: 16m 28.43s

777650 records written to output.csv
Thu Feb 26 09:41:00 EST 2026

% ls -l output.csv
-rw-r--r--@ 1 tgagne  staff  13998730299 Feb 26 09:40 output.csv

For the record, this was SOOOOO much faster and easier than DataLoader