Formidable detects MIME-type according to the file extension and not by the real content

[pubmikeb]

Support plan

• which support plan is this issue covered by? (e.g. Community, Sponsor, or
  Enterprise): Community
• is this issue currently blocking your project? (yes/no): no
• is this issue affecting a production system? (yes/no): yes

Context

• node version: 16.4.2
• module (formidable) version: 3.0.0-canary.20210428
• environment (e.g. node, browser, native, OS): Node.js
• used with (i.e. popular names of modules):
• any other relevant information:
  Formidable detects MIME-type according to the file extension and not by the real content. Which means that user can fake the file MIME by changing file's extension and as a result to upload to the server not allowed file types.

BTW, multer detects file's MIME not by the extension only.

What are you trying to achieve or the steps to reproduce?

1. Given JPG file tst.jpg
2. Rename it to tst.pdf
3. Set a break point inside of uploader.parse(req, async (err, fields, files) => {…}
4. Try to upload tst.pdf

uploader.parse(req, async (err, fields, files) => {
	if (err) {
		reject(err);
	} else {…}
});

What was the result you got?

mimetype = application/pdf

What result did you expect?

mimetype = image/jpeg
Since this file is actually JPG file but with a from extension.

[GrosSacASac]

That is good point, and the readme could definitely have a small guide on how to check this.

I don't think we will ever bake in content-based MIME detection since there are thousands of file types (and more created each year) and each one needs special detection. For example to know if it is a jpg is very different algorithm than zip, pdf etc.

[pubmikeb]

since there are thousands of file types (and more created each year) and each one needs special detection

Sure, it's difficult to support all possible types. But a deep check for the very common types (e.g. office, PDF, media) could be a great feature with ability to extend it by custom plugins.

[GrosSacASac]

multer detects file's MIME not by the extension only.

How does multer do it ?

[GrosSacASac]

Sure, it's difficult to support all possible types. But a deep check for the very common types (e.g. office, PDF, media) could be a great feature with ability to extend it by custom plugins.

Common types is very subjective.

And we are trying to go to the opposite direction, make the lib smaller not bigger. For example #718 (comment)

Personally I think we should embrace the npm philosophy of having a lot of modules that do 1 thing and do it well.
1 lib that validates pdf and jpg etc.

Then everyone can decide to import or not, if he needs alongside formidable.

[pubmikeb]

Personally I think we should embrace the npm philosophy of having a lot of modules that do 1 thing and do it well.
1 lib that validates pdf and jpg etc.

Then everyone can decide to import or not, if he needs alongside formidable.

Sure, ideally would be to make formidable-core for the minimalistic and generic as much as possible functionality and a set of plugins e.g. formidable-pdf, formidable-msoffice, etc. So every project could select the required set of supported types and every developer could enrich a set of supported file types by writing a plugin. Something similar to ESLint's set of rules.

[pubmikeb]

And we are trying to go to the opposite direction, make the lib smaller not bigger. For example #718 (comment)

Great approach, zero dependencies and 85 KB package are much better then alternative solutions with 2-5 MB of dependencies.

[pubmikeb]

How does multer do it ?

It looks like the file type is detected by checking the magic number of the buffer.

It worth using the file-type package:

import FileType from "file-type";

(async () => {
	console.log(await FileType.fromFile("Unicorn.png"));
	console.log(await FileType.fromBuffer(fileBuffer));
	console.log(await FileType.fromStream(fileStream));

	//Output: {ext: "png", mime: "image/png'"}
})();

[TheThing]

How does multer do it ?

It looks like the file type is detected by checking the magic number of the buffer.

It worth using the file-type package:

import FileType from "file-type";

(async () => {
	console.log(await FileType.fromFile("Unicorn.png"));
	console.log(await FileType.fromBuffer(fileBuffer));
	console.log(await FileType.fromStream(fileStream));

	//Output: {ext: "png", mime: "image/png'"}
})();

Except file-type adds not a single dependency but twelve. Some of which are redundant at this point:

`-- file-type@16.5.1
  +-- readable-web-to-node-stream@3.0.2
  | `-- readable-stream@3.6.0
  |   +-- inherits@2.0.4
  |   +-- string_decoder@1.3.0
  |   | `-- safe-buffer@5.2.1
  |   `-- util-deprecate@1.0.2
  +-- strtok3@6.1.3
  | +-- @tokenizer/token@0.1.1
  | `-- peek-readable@3.1.4
  `-- token-types@2.1.1
    +-- @tokenizer/token@0.1.1 deduped
    `-- ieee754@1.2.1

And formidable v3 already has plugin support so if you really want file type detection magic, just make a plugin for that.
I just usually don't even bother. If people wanna upload jpeg as pdf, that's their fault.

[pubmikeb]

If people wanna upload jpeg as pdf, that's their fault.

OK, since such functionality is not must have and it takes to implement about 10 minutes, this issue can be closed.

[tunnckoCore]

Leaning more and more towards a separate package like utils or plugins. where we can include such helpers for common things, just like we started with separate examples in the examples dir.

There's never ending discussion between small and big packages. And I'm still bouncing between the two. Okay, small, almost no dependencies package.. but end users anyway will include a lot of deps to do what their needs are.. so.. that's why i don't see big problem of including things like some of the two qs.. or other basic things like correct basic mime handling, here in the core package. Convenience and usability, over impracticality.
Small packages / unix philosophy starts to break when you realize that you really need few things to have some meaningful and functional final product, with common features. I'm not saying I'm okey with the mentioned problem how much v2 started to grow.

I'm for staying thin and this to remain in userland with some guides, but it also might be a good thing to include some basic detection for most common types, and not something huge and very generic like file-type.