Cryptic "JavaScript heap out of memory" error when Transport Stream (.ts) video file is accidentally compiled

[shirakaba]

TypeScript version: 2.8.0-dev.20180127 (also applies to v2.4.2 and v2.6.2).
WebStorm version: 2017.3.3 (also applies to 2017.3.1).
VS Code version: 1.19.3
Search terms: "tsserver heap"

Background:

• Accidentally adding MPEG-TS files (which also use the .ts extension) to the project tree causes the TypeScript service in the WebStorm and VS Code IDEs to hang and provide no useful error message.
• Directly running tsc v2.8.0-dev.20180127 from the command-line works just fine on the .ts files, as does the webpack v3.10.0 build process (via ts-loader v3.2.0), because they only consume whatever files are explicitly imported, however, IDE TypeScript services hang because they analyse every file in the project tree.

MWE

Simpy add a Transport Stream (.ts) video file under your project tree (eg. Big Buck Bunny – 61.7Mb), ie. into a folder at the project root called vids/. Now open a TypeScript source file in the project in an IDE such as VS Code.

const x: number = 0;
x = "hello"; // Should throw error.

Config

Sample tsconfig.json (exact values unimportant; tsc --init likely suffices):

{
 "compilerOptions": {
 "watch": true,
 "sourceMap": true,
 "target": "es5",
 "baseUrl": ".",
 "outDir": "./built",
 "allowJs": true,
 "module": "commonjs",
 "removeComments": true,
 "declaration": false,
 "noImplicitAny": true,
 "noImplicitThis": true,
 "strictNullChecks": true,
 "noLib": false
 }
}

Expected behaviour:

Errors and warnings in TypeScript source should be indicated by syntax highlighting or red underlines in the latest WebStorm IDE and VS Code.

Actual behaviour:

After 1-2 minutes of hanging on startup, each IDE's TypeScript service throws a heap error, and so doesn't get as far as communicating the errors/warnings to the IDE for appropriate syntax highlighting.

WEBSTORM

Log:

Process: TypeScript service version: 2.8.0-dev.20180127
Process: Default service options: {"watch":true,"sourceMap":true,"target":1,"baseUrl":"/Users/me/Sites/proj","outDir":"/Users/me/Sites/proj/built","allowJs":true,"typeRoots":["/Users/me/Sites/proj/node_modules/@types","/Users/me/Documents/git/types"],"module":1,"removeComments":true,"declaration":false,"noImplicitAny":true,"noImplicitThis":true,"strictNullChecks":true,"noLib":false,"configFilePath":"/Users/me/Sites/proj/tsconfig.json"}
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/usr/local/bin/node]
 2: node::FatalException(v8::Isolate*, v8::Local<v8::Value>, v8::Local<v8::Message>) [/usr/local/bin/node]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [/usr/local/bin/node]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/usr/local/bin/node]
 5: v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/usr/local/bin/node]
 6: v8::internal::Runtime_AllocateInTargetSpace(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/local/bin/node]
 7: 0x2f298220463d
 8: 0x2f298237d4ac

VS CODE

Symptoms:

• VS Code gets stuck on [TypeScript Importer]: initializing for about 1-2 minutes.
• Upon failing initialisation, I receive two errors: "Extension host terminated unexpectedly", and "Canceled".
• My 'output', and 'debug console' tabs are all blank, while my 'problems' tab says only "No problems have been detected in the workspace so far."

Log:

console.ts:123 [Extension Host] processWorkspaceFiles Array(13)0: Object$mid: 1external: "file:///Users/me/Sites/proj/lib/css/common.css.d.ts"fsPath: "/Users/me/Sites/proj/lib/css/common.css.d.ts"path: "/Users/me/Sites/proj/lib/css/common.css.d.ts"scheme: "file"__proto__: Object1: Object$mid: 1external: "file:///Users/me/Sites/proj/lib/css/fonts.css.d.ts"fsPath: "/Users/me/Sites/proj/lib/css/fonts.css.d.ts"path: "/Users/me/Sites/proj/lib/css/fonts.css.d.ts"scheme: "file"__proto__: Object2: Object$mid: 1external: "file:///Users/me/Sites/proj/lib/css/Mosaic.css.d.ts"fsPath: "/Users/me/Sites/proj/lib/css/Mosaic.css.d.ts"path: "/Users/me/Sites/proj/lib/css/Mosaic.css.d.ts"scheme: "file"__proto__: Object3: Object$mid: 1external: "file:///Users/me/Sites/proj/lib/css/Prompt.css.d.ts"fsPath: "/Users/me/Sites/proj/lib/css/Prompt.css.d.ts"path: "/Users/me/Sites/proj/lib/css/Prompt.css.d.ts"scheme: "file"__proto__: Object4: Object$mid: 1external: "file:///Users/me/Sites/proj/r/vids/proj-dummy.ts"fsPath: "/Users/me/Sites/proj/r/vids/proj-dummy.ts"path: "/Users/me/Sites/proj/r/vids/proj-dummy.ts"scheme: "file"__proto__: Object5: Object$mid: 1external: "file:///Users/me/Sites/proj/src/startup/Entry.ts"fsPath: "/Users/me/Sites/proj/src/startup/Entry.ts"path: "/Users/me/Sites/proj/src/startup/Entry.ts"scheme: "file"__proto__: Object6: Object7: Object8: Object$mid: 1external: "file:///Users/me/Sites/proj/src/templates/Mosaic.ts"fsPath: "/Users/me/Sites/proj/src/templates/Mosaic.ts"path: "/Users/me/Sites/proj/src/templates/Mosaic.ts"scheme: "file"__proto__: Object9: Object10: Object$mid: 1external: "file:///Users/me/Sites/proj/src/templates/VideoSelect.ts"fsPath: "/Users/me/Sites/proj/src/templates/VideoSelect.ts"path: "/Users/me/Sites/proj/src/templates/VideoSelect.ts"scheme: "file"__proto__: Object11: Object$mid: 1external: "file:///Users/me/Sites/proj/src/ui/MosaicView.ts"fsPath: "/Users/me/Sites/proj/src/ui/MosaicView.ts"path: "/Users/me/Sites/proj/src/ui/MosaicView.ts"scheme: "file"__proto__: Object12: Object$mid: 1external: "file:///Users/me/Sites/proj/src/ui/Splashscreen.ts"fsPath: "/Users/me/Sites/proj/src/ui/Splashscreen.ts"path: "/Users/me/Sites/proj/src/ui/Splashscreen.ts"scheme: "file"__proto__: Objectlength: 13__proto__: Array(0) true false
index.js:38 [uncaught exception]: Error: write EPIPE
onError @ index.js:38
index.js:41 Error: write EPIPE
    at exports._errnoException (util.js:1050)
    at WriteWrap.afterWrite [as oncomplete] (net.js:813)
onError @ index.js:41
events.js:163 Uncaught Error: write EPIPE
    at exports._errnoException (util.js:1050)
    at WriteWrap.afterWrite [as oncomplete] (net.js:813)
extensionService.ts:215 Extension host terminated unexpectedly. Code:  null  Signal:  SIGABRT
t._onExtensionHostCrashed @ extensionService.ts:215
messageService.ts:126 Canceled
e.doShow @ messageService.ts:126
messageService.ts:126 Canceled
e.doShow @ messageService.ts:126
messageService.ts:126 Canceled
e.doShow @ messageService.ts:126
messageService.ts:126 Canceled
e.doShow @ messageService.ts:126
messageService.ts:126 Extension host terminated unexpectedly.
e.doShow @ messageService.ts:126
extensionHost.ts:200 [Extension Host] debugger inspector at %cFATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: %cnode::Abort() [/Applications/Visual Studio Code.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Libraries/libnode.dylib]
 2: node::DLOpen(v8::FunctionCallbackInfo<v8::Value> const&) [/Applications/Visual Studio Code.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Libraries/libnode.dylib]
 3: %cv8::internal::FatalProcessOutOfMemory(char const*) [/Applications/Visual Studio Code.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Libraries/libnode.dylib]
 4: v8::internal::FatalProcessOutOfMemory(char const*) [/Applications/Visual Studio Code.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Libraries/libnode.dylib]
 5: %cv8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/Applications/Visual Studio Code.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Libraries/libnode.dylib]
 6: %cv8::internal::RegisterConfiguration::AreAliases(v8::internal::MachineRepresentation, int, v8::internal::MachineRepresentation, int) const [/Applications/Visual Studio Code.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Libraries/libnode.dylib]
 7: 0x1fec78285bbc
%c
<--- Last few GCs --->

[4387:0x7f94e4013600]   363933 ms: Mark-sweep 2054.8 (2118.3) -> 2054.8 (2114.8) MB, 1969.6 / 0.0 ms  (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 1970 ms) last resort 
[4387:0x7f94e4013600]   365854 ms: Mark-sweep 2054.8 (2114.8) -> 2054.8 (2114.8) MB, 1919.4 / 0.0 ms  last resort 


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x34bdacd2b959 <JS Object>
    1: parseDecorators [/Users/me/.vscode/extensions/rbbit.typescript-hero-0.12.0/node_modules/typescript/lib/typescript.js:~18486] [pc=0x1fec78a9e285](this=0x2d7044613e61 <JS Global Object>)
    2: parseDeclaration [/Users/me/.vscode/extensions/rbbit.typescript-hero-0.12.0/node_modules/typescript/lib/typescript.js:~18167] [pc=0x1fec78a97cd0](this=0x2d7044613e61 <JS Global Object>)
    3:...

Related issues:

I have encountered this error two or three times over the last year (it was present in TypeScript v2.1.5 too), and every time it catches me out, killing a lot of work hours in tracking down the problem again.

#12804 (comment)

Workaround:

MPEG-TS files can alternatively be named .tsv or .tsa, which stops them being consumed as TypeScript files. However, the onus is rather on TypeScript to handle them properly, as MPEG-TS has been using the .ts file extension since at least its release in 1995, well before the TypeScript project.

Proposal:

• Determine whether a .ts file is of MPEG-TS type by examining the first few bytes of it. The Wikipedia page indicates that an MPEG-TS file is merely a "sequence of packets, without any global header". Thus, a valid MPEG-TS file will start with a header packet (the format is again described on the Wikipedia page), beginning with a sync byte, invariably with the value 0x47 (The character 'G'). If there is a chance that this could clash with a valid TypeScript file, then further bytes should be read in.
• Implement a default, yet overridable, file size limit for .ts files to prevent the TypeScript compiler spending too long hanging on them (the existing --disableSizeLimit is for JS only as far as I understand). This would handle the case of invalid MPEG-TS files (not really a big concern, though) or other unforeseen file extension clashes with .ts.
• Provide a better error message.

[j-oliveras]

Duplicated of #21136.

[DanielRosenwasser]

In the meantime you can add the file to your tsconfig.json's top-level exclude list

{
    "compilerOptions": {
        // ...
    },
    "exclude": [
        "your-file-here.ts",
        "your-media-directory-here.ts"
    ]
}

[typescript-bot]

Automatically closing this issue for housekeeping purposes. The issue labels indicate that it is unactionable at the moment or has already been addressed.