Skip to content

error: FileNotFound/error:Unexpected when running zig build on Windows, when zig project is on different filesystem than OS #16374

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
RadsammyT opened this issue Jul 10, 2023 · 19 comments
Labels
bug Observed behavior contradicts documented or intended behavior contributor friendly This issue is limited in scope and/or knowledge of Zig internals. os-windows standard library This issue involves writing Zig code for the standard library.
Milestone

Comments

@RadsammyT
Copy link

RadsammyT commented Jul 10, 2023

Zig Version

0.11.0-dev.3949+27a66191c

Steps to Reproduce and Observed Behavior

OS Name: Microsoft Windows 10 Home
Version: 10.0.19045 Build 19045
Arch: x86_64
Using skeeto/w64devkit shell with zig set in its PATH, using Zig w/ Command Prompt/Powershell does not make a difference.
Using 0.10.1 will compile its respective init-exe template correctly for me.
on the above listed version and on dev.3947+89396ff02, I get a simple error: FileNotFound when running zig build for the first time and future builds (where zig-cache is present) return error: Unexpected, even on a freshly created init-exe project.

log
F:/PROJECTS $ mkdir zig_bug
F:/PROJECTS $ cd zig_bug
F:/PROJECTS/zig_bug $ zig version
0.11.0-dev.3949+27a66191c
F:/PROJECTS/zig_bug $ zig init-exe
info: Created build.zig
info: Created src\main.zig
info: Next, try `zig build --help` or `zig build run`
F:/PROJECTS/zig_bug $ ll
total 4K
drwxrwxr-x    3 Radsammy Radsammy       0 Jul 10 12:23 ./
drwxrwxr-x   14 Radsammy Radsammy       0 Jun 18 19:41 ../
-rw-rw-r--    1 Radsammy Radsammy    3.1K Jul 10 12:23 build.zig
drwxrwxr-x    2 Radsammy Radsammy       0 Jul 10 12:23 src/
F:/PROJECTS/zig_bug $ cat build.zig
const std = @import("std");

// Although this function looks imperative, note that its job is to
// declaratively construct a build graph that will be executed by an external
// runner.
pub fn build(b: *std.Build) void {
    // Standard target options allows the person running `zig build` to choose
    // what target to build for. Here we do not override the defaults, which
    // means any target is allowed, and the default is native. Other options
    // for restricting supported target set are available.
    const target = b.standardTargetOptions(.{});

    // Standard optimization options allow the person running `zig build` to select
    // between Debug, ReleaseSafe, ReleaseFast, and ReleaseSmall. Here we do not
    // set a preferred release mode, allowing the user to decide how to optimize.
    const optimize = b.standardOptimizeOption(.{});

    const exe = b.addExecutable(.{
        .name = "zig_bug",
        // In this case the main source file is merely a path, however, in more
        // complicated build scripts, this could be a generated file.
        .root_source_file = .{ .path = "src/main.zig" },
        .target = target,
        .optimize = optimize,
    });

    // This declares intent for the executable to be installed into the
    // standard location when the user invokes the "install" step (the default
    // step when running `zig build`).
    b.installArtifact(exe);

    // This *creates* a Run step in the build graph, to be executed when another
    // step is evaluated that depends on it. The next line below will establish
    // such a dependency.
    const run_cmd = b.addRunArtifact(exe);

    // By making the run step depend on the install step, it will be run from the
    // installation directory rather than directly from within the cache directory.
    // This is not necessary, however, if the application depends on other installed
    // files, this ensures they will be present and in the expected location.
    run_cmd.step.dependOn(b.getInstallStep());

    // This allows the user to pass arguments to the application in the build
    // command itself, like this: `zig build run -- arg1 arg2 etc`
    if (b.args) |args| {
        run_cmd.addArgs(args);
    }

    // This creates a build step. It will be visible in the `zig build --help` menu,
    // and can be selected like this: `zig build run`
    // This will evaluate the `run` step rather than the default, which is "install".
    const run_step = b.step("run", "Run the app");
    run_step.dependOn(&run_cmd.step);

    // Creates a step for unit testing. This only builds the test executable
    // but does not run it.
    const unit_tests = b.addTest(.{
        .root_source_file = .{ .path = "src/main.zig" },
        .target = target,
        .optimize = optimize,
    });

    const run_unit_tests = b.addRunArtifact(unit_tests);

    // Similar to creating the run step earlier, this exposes a `test` step to
    // the `zig build --help` menu, providing a way for the user to request
    // running the unit tests.
    const test_step = b.step("test", "Run unit tests");
    test_step.dependOn(&run_unit_tests.step);
}
F:/PROJECTS/zig_bug $ zig build
error: FileNotFound
F:/PROJECTS/zig_bug $
//before cache
F:/PROJECTS/zig_bug $ zig build
error: FileNotFound

//after cache
F:/PROJECTS/zig_bug $ zig build
error: Unexpected

However, doing zig build-exe src/main.zig successfully compiles it.

F:/PROJECTS/zig_bug $ zig build-exe src/main.zig
F:/PROJECTS/zig_bug $ ./main.exe
All your codebase are belong to us.   
Run `zig build test` to run the tests.

To reproduce

I have 3 internal drives on my machine, C: (Windows OS, NTFS), D: (NTFS), and E: (exFAT). My zig installation is installed on E: which also houses a programming directory I do some of my work on. Theres also a F: (FAT32) USB flash drive which also houses zig. I am able to compile zig code on a init-exe template on C: and D: , but not E: nor F:.

One might be able to reproduce this bug by compiling a zig project on a different filesystem than the FS the OS was installed on.

Expected Behavior

zig build should compile just fine.

@RadsammyT RadsammyT added the bug Observed behavior contradicts documented or intended behavior label Jul 10, 2023
@RadsammyT
Copy link
Author

This error also persists when using dev.3947+89396ff02 to compile Zig on Windows following instructions on this wiki page (Option 1b).

E:\CODING WORKSPACE\zig>"%DEVKIT%\bin\zig.exe" build -p stage3 --search-prefix %DEVKIT% --zig-lib-dir lib -Dstatic-llvm -Duse-zig-libcxx -Dtarget=x86_64-windows-gnu
error: FileNotFound

E:\CODING WORKSPACE\zig>"%DEVKIT%\bin\zig.exe" build -p stage3 --search-prefix %DEVKIT% --zig-lib-dir lib -Dstatic-llvm -Duse-zig-libcxx -Dtarget=x86_64-windows-gnu
error: Unexpected

E:\CODING WORKSPACE\zig>

@kassane
Copy link
Contributor

kassane commented Jul 10, 2023

build.zig file is missing. zig build command requires build system file.
zig build-exe|lib|obj single build project file.

Reference:
https://ziglang.org/documentation/master/#Zig-Build-System

@squeek502
Copy link
Collaborator

Try outside of the w64devkit shell. Can't reproduce this locally.

@RadsammyT
Copy link
Author

RadsammyT commented Jul 11, 2023

@squeek502
Downloaded 0.11.0-dev.3952+82a9d5d78 binaries and added it to Windows' PATH, but it still won't find that file.


E:\zigging\test-proj>where zig
E:\zigging\zig.exe

E:\zigging\test-proj>zig version
0.11.0-dev.3952+82a9d5d78

E:\zigging\test-proj>cat build.zig
const std = @import("std");

// Although this function looks imperative, note that its job is to
// declaratively construct a build graph that will be executed by an external
// runner.
pub fn build(b: *std.Build) void {
    // Standard target options allows the person running `zig build` to choose
    // what target to build for. Here we do not override the defaults, which
    // means any target is allowed, and the default is native. Other options
    // for restricting supported target set are available.
    const target = b.standardTargetOptions(.{});

    // Standard optimization options allow the person running `zig build` to select
    // between Debug, ReleaseSafe, ReleaseFast, and ReleaseSmall. Here we do not
    // set a preferred release mode, allowing the user to decide how to optimize.
    const optimize = b.standardOptimizeOption(.{});

    const exe = b.addExecutable(.{
        .name = "test-proj",
        // In this case the main source file is merely a path, however, in more
        // complicated build scripts, this could be a generated file.
        .root_source_file = .{ .path = "src/main.zig" },
        .target = target,
        .optimize = optimize,
    });

    // This declares intent for the executable to be installed into the
    // standard location when the user invokes the "install" step (the default
    // step when running `zig build`).
    b.installArtifact(exe);

    // This *creates* a Run step in the build graph, to be executed when another
    // step is evaluated that depends on it. The next line below will establish
    // such a dependency.
    const run_cmd = b.addRunArtifact(exe);

    // By making the run step depend on the install step, it will be run from the
    // installation directory rather than directly from within the cache directory.
    // This is not necessary, however, if the application depends on other installed
    // files, this ensures they will be present and in the expected location.
    run_cmd.step.dependOn(b.getInstallStep());

    // This allows the user to pass arguments to the application in the build
    // command itself, like this: `zig build run -- arg1 arg2 etc`
    if (b.args) |args| {
        run_cmd.addArgs(args);
    }

    // This creates a build step. It will be visible in the `zig build --help` menu,
    // and can be selected like this: `zig build run`
    // This will evaluate the `run` step rather than the default, which is "install".
    const run_step = b.step("run", "Run the app");
    run_step.dependOn(&run_cmd.step);

    // Creates a step for unit testing. This only builds the test executable
    // but does not run it.
    const unit_tests = b.addTest(.{
        .root_source_file = .{ .path = "src/main.zig" },
        .target = target,
        .optimize = optimize,
    });

    const run_unit_tests = b.addRunArtifact(unit_tests);

    // Similar to creating the run step earlier, this exposes a `test` step to
    // the `zig build --help` menu, providing a way for the user to request
    // running the unit tests.
    const test_step = b.step("test", "Run unit tests");
    test_step.dependOn(&run_unit_tests.step);
}

E:\zigging\test-proj>cat src\main.zig
const std = @import("std");

pub fn main() !void {
    // Prints to stderr (it's a shortcut based on `std.io.getStdErr()`)
    std.debug.print("All your {s} are belong to us.\n", .{"codebase"});

    // stdout is for the actual output of your application, for example if you
    // are implementing gzip, then only the compressed bytes should be sent to
    // stdout, not any debugging messages.
    const stdout_file = std.io.getStdOut().writer();
    var bw = std.io.bufferedWriter(stdout_file);
    const stdout = bw.writer();

    try stdout.print("Run `zig build test` to run the tests.\n", .{});

    try bw.flush(); // don't forget to flush!
}

test "simple test" {
    var list = std.ArrayList(i32).init(std.testing.allocator);
    defer list.deinit(); // try commenting this out and see if zig detects the memory leak!
    try list.append(42);
    try std.testing.expectEqual(@as(i32, 42), list.pop());
}

E:\zigging\test-proj>rm -r zig-cache

E:\zigging\test-proj>zig build
error: FileNotFound

E:\zigging\test-proj>zig build
error: Unexpected

E:\zigging\test-proj>

From what I am seeing, its failing at LLD Linking.
Is there a way to get any stacktrace on the build errors in question? A way to find out what file its missing?
I'm again looking into compiling Zig on debug to get the stacktrace. hopefully this goes well.

@squeek502
Copy link
Collaborator

What does zig env print for you? Are there any non-ASCII characters in the global_cache_dir path?

(non-ASCII characters shouldn't be a problem but maybe there's been some regression involving that)

@RadsammyT
Copy link
Author

zig env output:

E:\CODING_WORKSPACE\zig\build>zig env
{
 "zig_exe": "E:\\zigging\\zig.exe",
 "lib_dir": "E:\\zigging\\lib",
 "std_dir": "E:\\zigging\\lib\\std",
 "global_cache_dir": "C:\\Users\\RadsammyT\\AppData\\Local\\zig",
 "version": "0.11.0-dev.3952+82a9d5d78",
 "target": "x86_64-windows.win10_fe...win10_fe-gnu"
}

On another note, I wasn't able to compile zig on both methods, (using existing zig installation & cmake/ninja), as they also can't find a file.

E:\CODING_WORKSPACE\zig\build>ninja
[20/20] Building stage3
FAILED: stage3/bin/zig E:/CODING_WORKSPACE/zig/build/stage3/bin/zig
cmd.exe /C "cd /D E:\CODING_WORKSPACE\zig && E:\CODING_WORKSPACE\zig\build\zig2.exe build --prefix E:/CODING_WORKSPACE/zig/build/stage3 --zig-lib-dir E:/CODING_WORKSPACE/zig/lib -Dconfig_h=E:/CODING_WORKSPACE/zig/build/config.h -Denable-llvm -Duse-zig-libcxx -Dtarget=native -Dcpu=native -Dversion-string=0.11.0-dev.3950+a75531073"
error: FileNotFound
ninja: build stopped: subcommand failed.

If possible, is someone able to provide a debug executable of Zig?

@emidoots
Copy link

you may be able to run zig using procmon to see which file it is trying to open

@RadsammyT
Copy link
Author

RadsammyT commented Jul 13, 2023

Procmon Logs
I gathered procmon logs for two versions of Zig, 0.10.1 (The version where my issue doesn't appear) and 0.11.0-dev.3952+82a9d5d78. I wasn't able to find what file it was failing to open so I am leaving both of those logs here.

@squeek502
Copy link
Collaborator

@RadsammyT here's a debug build you can try:

https://www.ryanliptak.com/misc/zig-windows-x86_64-debug-0.11.0-dev.3892+0a6cd257b.zip

@RadsammyT
Copy link
Author

RadsammyT commented Jul 13, 2023

@squeek502 Thanks for the build! Unfortunately the backtrace it provides isn't... helpful.

E:\zigging\test-proj>zig version
0.11.0-dev.3892+0a6cd257b

E:\zigging\test-proj>rm -r zig-cache

E:\zigging\test-proj>zig build
error: FileNotFound
???:?:?: 0x7ff7da434dab in ??? (zig.exe)
???:?:?: 0x7ff7da1bf9f8 in ??? (zig.exe)
???:?:?: 0x7ff7d9f9233c in ??? (zig.exe)
???:?:?: 0x7ff7d9f98f43 in ??? (zig.exe)
???:?:?: 0x7ff7d9e2aa70 in ??? (zig.exe)
???:?:?: 0x7ff7d9dc3ee9 in ??? (zig.exe)
???:?:?: 0x7ff7d9dc15b8 in ??? (zig.exe)

E:\zigging\test-proj>zig build
error.Unexpected NTSTATUS=0xc0000035
???:?:?: 0x7ff7da19f5d0 in ??? (zig.exe)
???:?:?: 0x7ff7d9f62481 in ??? (zig.exe)
???:?:?: 0x7ff7d9fb4537 in ??? (zig.exe)
???:?:?: 0x7ff7d9fb4295 in ??? (zig.exe)
???:?:?: 0x7ff7d9fc0588 in ??? (zig.exe)
???:?:?: 0x7ff7d9e294db in ??? (zig.exe)
???:?:?: 0x7ff7d9dc3e66 in ??? (zig.exe)
???:?:?: 0x7ff7d9dc156e in ??? (zig.exe)
???:?:?: 0x7ff7d9dc1234 in ??? (zig.exe)
???:?:?: 0x7ff7dbc71495 in ??? (zig.exe)
???:?:?: 0x7ff7dbc714fb in ??? (zig.exe)
???:?:?: 0x7ffaef1b7613 in ??? (KERNEL32.DLL)
???:?:?: 0x7ffaefb026f0 in ??? (ntdll.dll)
error: Unexpected
???:?:?: 0x7ff7d9fef53b in ??? (zig.exe)
???:?:?: 0x7ff7da19f7e7 in ??? (zig.exe)
???:?:?: 0x7ff7d9f624c7 in ??? (zig.exe)
???:?:?: 0x7ff7d9fb4558 in ??? (zig.exe)
???:?:?: 0x7ff7d9fb42ec in ??? (zig.exe)
???:?:?: 0x7ff7d9fc05a6 in ??? (zig.exe)
???:?:?: 0x7ff7d9e29740 in ??? (zig.exe)
???:?:?: 0x7ff7d9dc3ee9 in ??? (zig.exe)
???:?:?: 0x7ff7d9dc15b8 in ??? (zig.exe)

@squeek502
Copy link
Collaborator

Ah, that's unfortunate. At least we know the unexpected status is STATUS_OBJECT_NAME_COLLISION, though.

Weird solution that I'm not sure will work, but maybe try cloning the zig repo (at commit 0a6cd25) into C:\Users\Ryan\Programming\Zig\zig so that its paths match what's in the PDB.

@RadsammyT
Copy link
Author

RadsammyT commented Jul 13, 2023

Nope, no dice on the solution either, but while messing around a bit I was able to somehow compile init-exe on C:\Users\Ryan\Programming\Zig\zigtest (on cmd.exe as admin). I'm not kidding.

C:\Users\Ryan\Programming\Zig\zigtest>zig init-exe
info: Created build.zig
info: Created src\main.zig
info: Next, try `zig build --help` or `zig build run

C:\Users\Ryan\Programming\Zig\zigtest>zig build

C:\Users\Ryan\Programming\Zig\zigtest>ls
build.zig  src        zig-cache  zig-out

C:\Users\Ryan\Programming\Zig\zigtest>zig build run
All your codebase are belong to us.
Run `zig build test` to run the tests.

C:\Users\Ryan\Programming\Zig\zigtest>zig version
0.11.0-dev.3892+0a6cd257b

I have 3 internal drives on my machine, C: (Windows OS, NTFS), D: (NTFS), and E: (exFAT). My zig installation is installed on E: which also houses a programming directory I do some of my work on. Theres also a F: (FAT32) USB flash drive which also houses zig. I am able to compile zig code (same debug version from E:, and on init-exe template) on C: and D: , but not E: nor F: .
I'm personally thinking this whole issue might be a filesystem specific one, but I'm not sure.

@squeek502
Copy link
Collaborator

Ah, thanks for the info. Definitely a filesystem related bug; I'm able to reproduce it with a FAT32 flash drive and the stack trace is even broken for me.

error: FileNotFound
???:?:?: 0x7ff6955b35fb in ??? (zig.exe)
???:?:?: 0x7ff69534043c in ??? (zig.exe)
???:?:?: 0x7ff69510df2c in ??? (zig.exe)
???:?:?: 0x7ff695114bd3 in ??? (zig.exe)
???:?:?: 0x7ff694fa035e in ??? (zig.exe)
???:?:?: 0x7ff694f33ef9 in ??? (zig.exe)
???:?:?: 0x7ff694f315b8 in ??? (zig.exe)

@squeek502
Copy link
Collaborator

Actually, I was being dumb and used the wrong Zig. Here are the stack traces:

error: FileNotFound
C:\Users\Ryan\Programming\Zig\zig\lib\std\child_process.zig:1138:5: 0x7ff665354dab in windowsCreateProcessPathExt (zig.exe.obj)
    return unappended_err;
    ^
C:\Users\Ryan\Programming\Zig\zig\lib\std\child_process.zig:849:21: 0x7ff6650df9f8 in spawnWindows (zig.exe.obj)
                    return original_err;
                    ^
C:\Users\Ryan\Programming\Zig\zig\lib\std\child_process.zig:200:13: 0x7ff664eb233c in spawn (zig.exe.obj)
            return self.spawnWindows();
            ^
C:\Users\Ryan\Programming\Zig\zig\lib\std\child_process.zig:207:9: 0x7ff664eb8f43 in spawnAndWait (zig.exe.obj)
        try self.spawn();
        ^
C:\Users\Ryan\Programming\Zig\zig\src\main.zig:4524:22: 0x7ff664d4aa70 in cmdBuild (zig.exe.obj)
        const term = try child.spawnAndWait();
                     ^
C:\Users\Ryan\Programming\Zig\zig\src\main.zig:285:9: 0x7ff664ce3ee9 in mainArgs (zig.exe.obj)
        return cmdBuild(gpa, arena, cmd_args);
        ^
C:\Users\Ryan\Programming\Zig\zig\src\main.zig:213:5: 0x7ff664ce15b8 in main (zig.exe.obj)
    return mainArgs(gpa, arena, args);
error.Unexpected NTSTATUS=0xc0000035
C:\Users\Ryan\Programming\Zig\zig\lib\std\os.zig:2693:48: 0x7ff6650bf5d0 in renameatW (zig.exe.obj)
        else => return windows.unexpectedStatus(rc),
                                               ^
C:\Users\Ryan\Programming\Zig\zig\lib\std\os.zig:2558:25: 0x7ff664e82481 in renameat (zig.exe.obj)
        return renameatW(old_dir_fd, old_path_w.span(), new_dir_fd, new_path_w.span(), windows.TRUE);
                        ^
C:\Users\Ryan\Programming\Zig\zig\lib\std\fs.zig:1938:27: 0x7ff664ed4537 in rename (zig.exe.obj)
        return os.renameat(self.fd, old_sub_path, self.fd, new_sub_path);
                          ^
C:\Users\Ryan\Programming\Zig\zig\src\Package.zig:748:25: 0x7ff664ed4295 in renameTmpIntoCache (zig.exe.obj)
        cache_dir.rename(tmp_dir_sub_path, dest_dir_sub_path) catch |err| switch (err) {
                        ^
C:\Users\Ryan\Programming\Zig\zig\src\Package.zig:345:27: 0x7ff664ee0588 in createFilePkg (zig.exe.obj)
    try renameTmpIntoCache(cache_directory.handle, tmp_dir_sub_path, o_dir_sub_path);
                          ^
C:\Users\Ryan\Programming\Zig\zig\src\main.zig:4467:55: 0x7ff664d494db in cmdBuild (zig.exe.obj)
            const deps_pkg = try Package.createFilePkg(
                                                      ^
C:\Users\Ryan\Programming\Zig\zig\src\main.zig:285:24: 0x7ff664ce3e66 in mainArgs (zig.exe.obj)
        return cmdBuild(gpa, arena, cmd_args);
                       ^
C:\Users\Ryan\Programming\Zig\zig\src\main.zig:213:20: 0x7ff664ce156e in main (zig.exe.obj)
    return mainArgs(gpa, arena, args);
                   ^
C:\Users\Ryan\Programming\Zig\zig\lib\std\start.zig:507:101: 0x7ff664ce1234 in main (zig.exe.obj)
    return @call(.always_inline, callMainWithArgs, .{ @as(usize, @intCast(c_argc)), @as([*][*:0]u8, @ptrCast(c_argv)), envp });
                                                                                                    ^
???:?:?: 0x7ff666b91495 in ??? (zig.exe)
???:?:?: 0x7ff666b914fb in ??? (zig.exe)
???:?:?: 0x7fffd8567613 in ??? (KERNEL32.DLL)
???:?:?: 0x7fffd94c26b0 in ??? (ntdll.dll)
error: Unexpected
C:\Users\Ryan\Programming\Zig\zig\lib\std\os\windows.zig:2345:5: 0x7ff664f0f53b in unexpectedStatus (zig.exe.obj)
    return error.Unexpected;
    ^
C:\Users\Ryan\Programming\Zig\zig\lib\std\os.zig:2693:17: 0x7ff6650bf7e7 in renameatW (zig.exe.obj)
        else => return windows.unexpectedStatus(rc),
                ^
C:\Users\Ryan\Programming\Zig\zig\lib\std\os.zig:2558:9: 0x7ff664e824c7 in renameat (zig.exe.obj)
        return renameatW(old_dir_fd, old_path_w.span(), new_dir_fd, new_path_w.span(), windows.TRUE);
        ^
C:\Users\Ryan\Programming\Zig\zig\lib\std\fs.zig:1938:9: 0x7ff664ed4558 in rename (zig.exe.obj)
        return os.renameat(self.fd, old_sub_path, self.fd, new_sub_path);
        ^
C:\Users\Ryan\Programming\Zig\zig\src\Package.zig:763:25: 0x7ff664ed42ec in renameTmpIntoCache (zig.exe.obj)
            else => |e| return e,
                        ^
C:\Users\Ryan\Programming\Zig\zig\src\Package.zig:345:5: 0x7ff664ee05a6 in createFilePkg (zig.exe.obj)
    try renameTmpIntoCache(cache_directory.handle, tmp_dir_sub_path, o_dir_sub_path);
    ^
C:\Users\Ryan\Programming\Zig\zig\src\main.zig:4467:30: 0x7ff664d49740 in cmdBuild (zig.exe.obj)
            const deps_pkg = try Package.createFilePkg(
                             ^
C:\Users\Ryan\Programming\Zig\zig\src\main.zig:285:9: 0x7ff664ce3ee9 in mainArgs (zig.exe.obj)
        return cmdBuild(gpa, arena, cmd_args);
        ^
C:\Users\Ryan\Programming\Zig\zig\src\main.zig:213:5: 0x7ff664ce15b8 in main (zig.exe.obj)
    return mainArgs(gpa, arena, args);
    ^

@RadsammyT RadsammyT changed the title error: FileNotFound/error:Unexpected when running zig build on Windows error: FileNotFound/error:Unexpected when running zig build on Windows, when zig is on different filesystem than OS Jul 13, 2023
@RadsammyT RadsammyT changed the title error: FileNotFound/error:Unexpected when running zig build on Windows, when zig is on different filesystem than OS error: FileNotFound/error:Unexpected when running zig build on Windows, when zig project is on different filesystem than OS Jul 13, 2023
@kcbanner
Copy link
Contributor

This may be related: #16209

I consistently see these errors with a cleared cache.

@andrewrk andrewrk added the standard library This issue involves writing Zig code for the standard library. label Jul 22, 2023
@andrewrk andrewrk modified the milestone: friendl Jul 22, 2023
@andrewrk andrewrk added the contributor friendly This issue is limited in scope and/or knowledge of Zig internals. label Jul 22, 2023
@andrewrk andrewrk added this to the 0.13.0 milestone Jul 22, 2023
@andrewrk
Copy link
Member

Actually, I was being dumb and used the wrong Zig. Here are the stack traces:

Nice, thank you. This is a contributer-friendly issue at this point, just need to add more switch cases and error codes to the respective locations.

@squeek502
Copy link
Collaborator

squeek502 commented Jul 22, 2023

The bug here is that zig build fails on FAT filesystems. The unexpected status error is just a side-effect that can be fixed separately, but won't address the underlying issue.


I think I found the problem for the initial zig build error at least:

zig/lib/std/child_process.zig

Lines 1025 to 1030 in 32a1757

// NtQueryDirectoryFile returns results in order by filename, so the first result of
// the wildcard call will always be the unappended version if it exists. So, if found_name
// is not the unappended version, we can skip straight to trying versions with PATHEXT appended.
// TODO: This might depend on the filesystem, though; need to somehow verify that it always
// works this way.
if (found_name != null and windows.eqlIgnoreCaseWTF16(found_name.?, app_buf.items[0..app_name_len])) {

That TODO is the key here. On NTFS, if you give a path like E:\path\to\build.exe* to NtQueryDirectoryFile then the first result it returns will always be build.exe, but it looks like on FAT, that is not guaranteed (in this case it's returning build.exe.obj first), so Zig (incorrectly) assumes that build.exe doesn't exist.

So I think this little optimization needs to be removed, and instead we always need to check all the results of NtQueryDirectoryFile instead of just the first.

squeek502 added a commit to squeek502/zig that referenced this issue Jul 23, 2023
This fixes a regression caused by ziglang#13993

As an optimization, the first call to `NtQueryDirectoryFile` would only ask for a single result and assume that if the result returned did not match the app_name exactly, then the unappended app_name did not exist. However, this relied on the assumption that the unappended app_name would always be returned first, but that only seems to be the same on NTFS. On FAT filesystems, the order of returned files can be different, which meant that it could assume the unappended file doesn't exist when it actually does.

This commit fixes that by fully iterating the wildcard matches via `NtQueryDirectoryFile` and taking note of any unappended/PATHEXT-appended filenames it finds. In practice, this strategy does not introduce a speed regression compared to the previous (buggy) implementation.

Benchmark 1 (10 runs): winpathbench-master.exe
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           508ms ± 4.08ms     502ms …  517ms          1 (10%)        0%
  peak_rss           3.62MB ± 2.76KB    3.62MB … 3.63MB          0 ( 0%)        0%
Benchmark 2 (10 runs): winpathbench-fat32-fix.exe
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           500ms ± 21.4ms     480ms …  535ms          0 ( 0%)          -  1.5% ±  2.8%
  peak_rss           3.62MB ± 2.76KB    3.62MB … 3.63MB          0 ( 0%)          -  0.0% ±  0.1%

---

Partially addresses ziglang#16374 (it fixes `zig build` on FAT32 when no `zig-cache` is present)
squeek502 added a commit to squeek502/zig that referenced this issue Jul 23, 2023
This fixes a regression caused by ziglang#13993

As an optimization, the first call to `NtQueryDirectoryFile` would only ask for a single result and assume that if the result returned did not match the app_name exactly, then the unappended app_name did not exist. However, this relied on the assumption that the unappended app_name would always be returned first, but that only seems to be the case on NTFS. On FAT filesystems, the order of returned files can be different, which meant that it could assume the unappended file doesn't exist when it actually does.

This commit fixes that by fully iterating the wildcard matches via `NtQueryDirectoryFile` and taking note of any unappended/PATHEXT-appended filenames it finds. In practice, this strategy does not introduce a speed regression compared to the previous (buggy) implementation.

Benchmark 1 (10 runs): winpathbench-master.exe
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           508ms ± 4.08ms     502ms …  517ms          1 (10%)        0%
  peak_rss           3.62MB ± 2.76KB    3.62MB … 3.63MB          0 ( 0%)        0%
Benchmark 2 (10 runs): winpathbench-fat32-fix.exe
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           500ms ± 21.4ms     480ms …  535ms          0 ( 0%)          -  1.5% ±  2.8%
  peak_rss           3.62MB ± 2.76KB    3.62MB … 3.63MB          0 ( 0%)          -  0.0% ±  0.1%

---

Partially addresses ziglang#16374 (it fixes `zig build` on FAT32 when no `zig-cache` is present)
squeek502 added a commit to squeek502/zig that referenced this issue Jul 23, 2023
@squeek502
Copy link
Collaborator

squeek502 commented Jul 23, 2023

Can be closed once #16492 and #16498 are merged.

When #16499 and #16501 are merged as well, then the standard library tests will fully pass on FAT filesystems.

andrewrk pushed a commit that referenced this issue Jul 24, 2023
This fixes a regression caused by #13993

As an optimization, the first call to `NtQueryDirectoryFile` would only ask for a single result and assume that if the result returned did not match the app_name exactly, then the unappended app_name did not exist. However, this relied on the assumption that the unappended app_name would always be returned first, but that only seems to be the case on NTFS. On FAT filesystems, the order of returned files can be different, which meant that it could assume the unappended file doesn't exist when it actually does.

This commit fixes that by fully iterating the wildcard matches via `NtQueryDirectoryFile` and taking note of any unappended/PATHEXT-appended filenames it finds. In practice, this strategy does not introduce a speed regression compared to the previous (buggy) implementation.

Benchmark 1 (10 runs): winpathbench-master.exe
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           508ms ± 4.08ms     502ms …  517ms          1 (10%)        0%
  peak_rss           3.62MB ± 2.76KB    3.62MB … 3.63MB          0 ( 0%)        0%
Benchmark 2 (10 runs): winpathbench-fat32-fix.exe
  measurement          mean ± σ            min … max           outliers         delta
  wall_time           500ms ± 21.4ms     480ms …  535ms          0 ( 0%)          -  1.5% ±  2.8%
  peak_rss           3.62MB ± 2.76KB    3.62MB … 3.63MB          0 ( 0%)          -  0.0% ±  0.1%

---

Partially addresses #16374 (it fixes `zig build` on FAT32 when no `zig-cache` is present)
@squeek502
Copy link
Collaborator

This can now be closed.

@andrewrk andrewrk modified the milestones: 0.13.0, 0.11.0 Jul 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Observed behavior contradicts documented or intended behavior contributor friendly This issue is limited in scope and/or knowledge of Zig internals. os-windows standard library This issue involves writing Zig code for the standard library.
Projects
None yet
Development

No branches or pull requests

6 participants