Skip to content

Improved FAT32 support for ESP32 and allows Files to be used with SDC… #1528

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: public
Choose a base branch
from
Open

Improved FAT32 support for ESP32 and allows Files to be used with SDC… #1528

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 12 additions & 24 deletions documentation/files/files.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -366,19 +366,27 @@ On ESP32, the SPIFFS file system is mounted at a specified path and all files/di

The `File` class implements an optional FAT32 file system for the ESP32. Unlike SPIFFS, FAT32 file systems are not flat: they have directory structures and long filenames (up to 255 characters).

If the FAT32 file system has not been initialized then it is formatted when first used. As with SPIFFS, the `File` class only instantiates the FAT32 file system when necessary and it is automatically closed when not in use.
If the FAT32 file system has not been initialized, then it is formatted when first used. As with SPIFFS, the `File` class only instantiates the FAT32 file system when necessary, and it is automatically closed when not in use.

To enable the FAT32 file system, set the `fat32` [manifest define](https://github.com/Moddable-OpenSource/moddable/blob/public/documentation/tools/defines.md) to `1`:
There are several settings for the FAT32 file system in the [manifest define](https://github.com/Moddable-OpenSource/moddable/blob/public/documentation/tools/defines.md) .

```JSON
"defines": {
"file":{
"fat32": 1
"fat32": 1,
"root": "\"/sdcard\"",
"partition": "#userdata",
"sdcard": 0
}
}
```

The storage partition used by the default Moddable SDK build for ESP32 does not reserve a partition for FAT32. Therefore, it is necessary to use a different partition file in projects that use FAT32. To do that, set the `PARTITIONS_FILE` variable in the `build` section of the project manifest:
- You need to set `fat32` to to enable the FAT32 settings. Set this to 0 or remove it for SPIFFS file system.
- By default, the FAT32 file system is mounted at `/mod`. To change the default root, set the `root` define in the manifest. Do not include a trailing `/` in the root name.
- The default name for the FAT32 partition is `storage`. To use a different name, set the `partition` defined in the manifest.
- If the file system is not on the internal ESP32 Flash, it will need to be managed externally to the File's internal software. For example, this is required for SD card support, Set the `sdcard` define to 1 to bypass using the File class's software to mount, unmount and format the file system. You will need to manage the file system externally from the `File` class.

The storage partition used by the default Moddable SDK build for ESP32 does not reserve a partition for FAT32. Therefore, it is necessary to use a different partition file in projects that use FAT32. To do that, set the `PARTITIONS_FILE` variable in the `build` section of the project manifest to point at an ESP32 partition file:
Copy link
Collaborator

@phoddie phoddie Sep 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We didn't imagine someone wanting to use FAT32 for internal partitions. It should be easy enough to support a FAT32 internal partition without having to manually manage a PARTITIONS_FILE. In the mcconfig implementation here it could check the defines for "fat32" set to a non-zero value and use "fat32" instead of "spiffs".


```JSON
"build": {
Expand All @@ -397,26 +405,6 @@ xs, 0x40, 1, 0x310000, 0x040000,
settings, data, 1, 0x350000, 0x010000,
storage, data, fat, 0x360000, 0x090000,
```
The default name for the FAT32 partition is `storage`. To use a different name, set the `partition` define in the manifest:

```JSON
"defines": {
"file":{
"partition": "#userdata"
}
}
```

By default, the FAT32 file system is mounted at `/mod`. To change the default root, set the `root` define in the manifest:

```JSON
"defines": {
"file":{
"root": "/myroot/"
}
}
```

<a id="littlefs"></a>
#### littlefs
The [littlefs](https://github.com/littlefs-project/littlefs) file system is "a little fail-safe filesystem designed for microcontrollers." It provides a high reliability, hierarchical file system in a small code footprint (about 60 KB) using minimal memory (well under 1 KB) with a high degree of configurability. littlefs also supports long file names (up to 255 characters) and formats a new partition very quickly.
Expand Down
76 changes: 37 additions & 39 deletions modules/files/file/esp32/modFile.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,11 +47,6 @@
#else
#define MAX_FILENAME_LENGTH 12
#endif
#ifdef CONFIG_WL_SECTOR_SIZE
#define SECTOR_SIZE CONFIG_WL_SECTOR_SIZE
#else
#define SECTOR_SIZE 512
#endif
#else
#include "esp_spiffs.h"
#include "spiffs_config.h"
Expand All @@ -72,14 +67,14 @@ typedef struct {
char path[1];
} iteratorRecord, *iter;

static int startFS(void);
static int startFS(xsMachine *the);
static void stopFS(void);

static FILE *getFile(xsMachine *the)
{
FILE *result = xsmcGetHostData(xsThis);
if (!result)
xsUnknownError("closed");
xsUnknownError("closed file");
return result;
}

Expand All @@ -100,7 +95,7 @@ void xs_File(xsMachine *the)
uint8_t write = (argc < 2) ? 0 : xsmcToBoolean(xsArg(1));
char *path = xsmcToString(xsArg(0));

startFS();
startFS(the);

file = fopen(path, write ? "rb+" : "rb");
if (NULL == file) {
Expand Down Expand Up @@ -197,7 +192,7 @@ void xs_file_write(xsMachine *the)
}
result = fflush(file);
if (0 != result)
xsUnknownError("file flush failed");
xsUnknownError("file flush failed on write");
}

void xs_file_close(xsMachine *the)
Expand Down Expand Up @@ -235,7 +230,7 @@ void xs_file_delete(xsMachine *the)
int32_t result;
char *path = xsmcToString(xsArg(0));

startFS();
startFS(the);

result = unlink(path);

Expand All @@ -250,7 +245,7 @@ void xs_file_exists(xsMachine *the)
int32_t result;
char *path = xsmcToString(xsArg(0));

startFS();
startFS(the);

result = stat(path, &buf);

Expand All @@ -269,21 +264,21 @@ void xs_file_rename(xsMachine *the)
path = xsmcToString(xsArg(0));
if ('/' != toPath[0]) {
if (c_strchr(toPath + 1, '/'))
xsUnknownError("invalid to");
xsUnknownError("Invalid to on rename");

char *slash = c_strrchr(path, '/');
if (!slash)
xsUnknownError("invalid from");
xsUnknownError("Invalid from on rename");

size_t pathLength = slash - path + 1;
if (pathLength >= (c_strlen(path) + sizeof(toPath)))
xsUnknownError("path too long");
xsUnknownError("Path too long on rename");

c_strcpy(toPath, path);
xsmcToStringBuffer(xsArg(1), toPath + pathLength, sizeof(toPath) - pathLength);
}

startFS();
startFS(the);

result = rename(path, toPath);

Expand Down Expand Up @@ -326,7 +321,7 @@ void xs_File_Iterator(xsMachine *the)
#endif
d->rootPathLen = i;

startFS();
startFS(the);

if (NULL == (d->dir = opendir(d->path))) {
c_free(d);
Expand Down Expand Up @@ -367,7 +362,7 @@ void xs_file_iterator_next(xsMachine *the)
if (DT_REG == de->d_type) {
c_strcpy(d->path + d->rootPathLen, de->d_name);
if (-1 == stat(d->path, &buf))
xsUnknownError("stat error");
xsUnknownError("stat error on iterator next");
xsmcSetInteger(xsVar(0), buf.st_size);
xsmcSet(xsResult, xsID_length, xsVar(0));
}
Expand All @@ -385,7 +380,7 @@ void xs_file_system_info(xsMachine *the)
{
xsResult = xsmcNewObject();

startFS();
startFS(the);
size_t total = 0, used = 0;
esp_err_t ret;

Expand All @@ -394,23 +389,23 @@ void xs_file_system_info(xsMachine *the)
FATFS *fs;
DWORD fre_clust, fre_sect, tot_sect;
ret = f_getfree("0:", &fre_clust, &fs);
tot_sect = (fs->n_fatent - 2) * fs->csize;
fre_sect = fre_clust * fs->csize;
total = tot_sect * SECTOR_SIZE;
used = (tot_sect - fre_sect) * SECTOR_SIZE;
tot_sect = (fs->n_fatent - 2) * fs->csize; // First two FAT clusters are reserved
fre_sect = fre_clust * fs->csize; // Free clusters
total = tot_sect * fs->ssize; // Sector size (512, 1024, 2048 or 4096)
used = (tot_sect - fre_sect) * fs->ssize;
#else
ret = esp_spiffs_info(NULL, &total, &used);
#endif

stopFS();

if (ret != ESP_OK)
xsUnknownError("system info failed");
xsUnknownError("System info failed");

xsmcVars(1);
xsmcSetInteger(xsVar(0), total);
xsmcSetInteger(xsVar(0), (uint64_t)total);
Copy link
Collaborator

@phoddie phoddie Sep 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This won't work as you expect. An XS integer is a 32 bit signed value by definition. The file system values can be bigger than that. The correct fix is to use a number (floating point double).

xsmcSetNumber(xsVar(0), total);

That said, I'm not sure that size_t on ESP32 is bigger than 32 bits. You might want to check that and explicitly use a uint64_t instead, if it isn't already 64-bits.

xsmcSet(xsResult, xsID_total, xsVar(0));
xsmcSetInteger(xsVar(0), used);
xsmcSetInteger(xsVar(0), (uint64_t)used);
xsmcSet(xsResult, xsID_used, xsVar(0));
}

Expand All @@ -420,20 +415,21 @@ static uint16_t gUseCount;
static wl_handle_t wl_handle;
#endif

int startFS(void)
int startFS(xsMachine *the)
{
esp_err_t ret;
if (0 != gUseCount++)
return 0;

// If this is an SD card filesystem, then don't do the internal Flash partition filesystem
#if MODDEF_FILE_SDCARD != 1
#if MODDEF_FILE_FAT32
esp_vfs_fat_mount_config_t conf = {
.format_if_mount_failed = true,
.max_files = MODDEF_FILE_MAXOPEN,
.allocation_unit_size = 512
};

ret = esp_vfs_fat_spiflash_mount(MODDEF_FILE_ROOT, MODDEF_FILE_PARTITION, &conf, &wl_handle);
ret = esp_vfs_fat_spiflash_mount_rw_wl(MODDEF_FILE_ROOT, MODDEF_FILE_PARTITION, &conf, &wl_handle);
#else
//@@ these behaviors could be controlled by DEFINE properties
esp_vfs_spiffs_conf_t conf = {
Expand All @@ -451,55 +447,57 @@ int startFS(void)

if (ret != ESP_OK) {
if (ret == ESP_FAIL) {
modLog("Failed to mount or format filesystem");
xsUnknownError("Failed to mount or format filesystem");
} else if (ret == ESP_ERR_NO_MEM) {
modLog("Failed to allocate memory for filesystem");
xsUnknownError("Failed to allocate memory for filesystem");
} else if (ret == ESP_ERR_NOT_FOUND) {
modLog("Failed to find partition");
xsUnknownError("Failed to find partition");
} else if (ret == ESP_ERR_INVALID_STATE) {
modLog("Failed because filesystem was already registered");
xsUnknownError("Failed because filesystem was already registered");
} else {
modLog("Failed to initialize filesystem");
xsUnknownError("Failed to initialize filesystem");
}

gUseCount--;
Copy link
Collaborator

@phoddie phoddie Sep 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

xsUnknownError("foo") is equivalent to throw new Error("foo") in JavaScript and consequently will not return. That means that gUseCount will never be decremented in the case of an error, which will leave the file system permanently active. The decrement should happen earlier so it isn't bypassed when the exception is thrown.

}

#endif // if file sdcard is set to 1, the don't do the internal Flash partition filesystem
return 0;
}

void stopFS(void)
{
if (0 != --gUseCount)
return;

#if MODDEF_FILE_SDCARD != 1
#if MODDEF_FILE_FAT32
esp_vfs_fat_spiflash_unmount(MODDEF_FILE_ROOT, wl_handle);
esp_vfs_fat_spiflash_unmount_rw_wl(MODDEF_FILE_ROOT, wl_handle);
wl_handle = 0;
#else
esp_vfs_spiffs_unregister(NULL);
#endif
#endif // if file sdcard is set to 1, the don't do the internal Flash partition filesystem
}


void xs_directory_create(xsMachine *the)
{
char *path = xsmcToString(xsArg(0));

startFS();
startFS(the);
int result = mkdir(path, 0775);
if (result && (EEXIST != errno))
xsUnknownError("failed");
xsUnknownError("Directory create failed");
Copy link
Collaborator

@phoddie phoddie Sep 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure about this. We generally have concise messages for a couple reasons. First, to minimize code size. They add up. Second, the operation ("Directory create" here) should be pretty clear from the exception thrown. For example:

try {
	function myFunction() {
		throw new Error("boom");
	}
	myFunction();
}
catch (e) {
	trace(e.stack, "\n");
}

Outputs:

Error: boom
 at myFunction ($MODDABLE/examples/hellotest/main.js:3)
 at main ($MODDABLE/examples/hellotest/main.js:5)

But maybe there's a scenario where this helps that I'm overlooking. Can you explain the motivation?

stopFS();
}

void xs_directory_delete(xsMachine *the)
{
char *path = xsmcToString(xsArg(0));

startFS();
startFS(the);
int result = rmdir(path);
if (result && (ENOENT != errno))
xsUnknownError("failed");
xsUnknownError("Directory delete failed");
stopFS();
}