std.fs.selfExePath converts drive letter to *:/ on secondary drive on windows

[patrik-gustavsson]

Zig Version

0.14.0-dev.1472+3929cac15

Steps to Reproduce and Observed Behavior

I am trying to get the std.fs.Dir to the self executable so that files relative to the executable can be loaded. The std.fs.cwd() cannot be used in this case since the executable can be run from any directory. I get a problem in windows when using multiple drives, it seems to only work for the main drive C:/ but the path to my other drive D:/ is converted to *:/ when using std.fs.selfExePath.

To reproduce:

const std = @import("std");
pub fn main() !void {
    var buffer = [_]u8{0} ** std.fs.max_path_bytes;
    const exe_path = try std.fs.selfExePath(&buffer);
    std.debug.print("ExePath: {s}\n", .{exe_path});
}

When located on C:/ it produces: ExePath: C:/...
When located on D:/ it produces: ExePath: *:/...

What I have gathered from other issues/proposals is that the problem lies in std.fs.Dir.realpath, this function is proposed to be removed in #19353 since it is buggy, but since the proposal is about removing realpath and not to fix it I created this issue.

I have circumvented this bug by not calling the selfExePath but instead use std.os.windows.peb().ProcessParameters.ImagePathName directly for windows builds and convert it to utf8 without using realpath in between, but I do not know if this approach is reliable.

What I can see from std.fs.openSelfExe it does not use realpath, is it really necessary to use realpath in selfExePath?
Is there another way that I haven't listed to get the absolute path or std.fs.Dir to the executable that is reliable?

Expected Behavior

When located on D:/ it should produce: ExePath: D:/...

[squeek502]

is it really necessary to use realpath in selfExePath?

Yes, this is done to follow symlinks: #16885

My guess is that this is something going wrong in std.os.windows.GetFinalPathNameByHandle. Could you make these changes to your Zig install's lib/std/os/windows.zig file, run your test program, and post the results?

diff --git a/lib/std/os/windows.zig b/lib/std/os/windows.zig
index 54a98c174f..46b76beae1 100644
--- a/lib/std/os/windows.zig
+++ b/lib/std/os/windows.zig
@@ -1286,6 +1286,8 @@ pub fn GetFinalPathNameByHandle(
         else => |e| return e,
     };
 
+    std.debug.print("final_path: {}\n", .{std.unicode.fmtUtf16Le(final_path)});
+
     switch (fmt.volume_name) {
         .Nt => {
             // the returned path is already in .Nt format
@@ -1367,6 +1369,8 @@ pub fn GetFinalPathNameByHandle(
                     @ptrCast(@alignCast(&output_buf[mount_point.SymbolicLinkNameOffset])),
                 )[0 .. mount_point.SymbolicLinkNameLength / 2];
 
+                std.debug.print("mount point symlink: {}\n", .{std.unicode.fmtUtf16Le(symlink)});
+
                 // Look for `\DosDevices\` prefix. We don't really care if there are more than one symlinks
                 // with traditional DOS drive letters, so pick the first one available.
                 var prefix_buf = std.unicode.utf8ToUtf16LeStringLiteral("\\DosDevices\\");

Here's the output when I run the test program with those changes:

> zig run selfexe.zig
final_path: \Device\HarddiskVolume4\Users\Ryan\AppData\Local\zig\o\0c62aef23144020ee5c08d8921b9c78b\selfexe.exe
mount point symlink: \DosDevices\C:
ExePath: C:\Users\Ryan\AppData\Local\zig\o\0c62aef23144020ee5c08d8921b9c78b\selfexe.exe

(feel free to redact any part of the path you don't want to share)

[patrik-gustavsson]

This is the result I get after editing the lib/std/os/windows.zig and running on the D:/ drive.

final_path: \Device\HarddiskVolume6\...
mount point symlink: \DosDevices\*:
ExePath: *:\...

When running on C:/ I get the same results as you did, retaining the drive letter.

[rootbeer]

I get C: and D: as expected on my two drives:

final_path: \Device\HarddiskVolume3\Users\patgw\selfexe.exe
mount point symlink: \DosDevices\C:
ExePath: C:\Users\patgw\selfexe.exe

final_path: \Device\HarddiskVolume7\pat\selfexe.exe
mount point symlink: \DosDevices\D:
ExePath: D:\pat\selfexe.exe

In Powershell you can query the list of mount points with Get-PSDrive -PSProvider FileSystem. Do you see a *: entry in there? Mine looks like:

> Get-PSDrive -PSProvider FileSystem

Name           Used (GB)     Free (GB) Provider      Root                                                                                                                                                                                                             CurrentLocation
----           ---------     --------- --------      ----                                                                                                                                                                                                             ---------------
C                1318.90       2482.91 FileSystem    C:\                                                                                                                                                         ProgramData\Microsoft\Windows\Start Menu\Programs\Windows PowerShell
D                3991.00       1597.90 FileSystem    D:\                                                                                                                                                                                                                             
F                   0.18        953.69 FileSystem    F:\                                                                                                                                                                                                                             
H                1211.77        886.23 FileSystem    H:\                                                                                                                                                                                                                             
N                 307.34        646.53 FileSystem    N:\                                                                                                                                                                                                                             
W                  98.04        590.78 FileSystem    \\cerberus\backups                                                                                                                                                                                                              
X                 119.55         27.91 FileSystem    \\cerberus\pat                                                                                                                                                                                                                  
Y                  44.78         33.79 FileSystem    \\cerberus\data                                                                                                                                                                                                                 
Z                 338.12        408.80 FileSystem    \\cerberus\scratch                                                                                                                                                                                                              

[squeek502]

This comment may end up being relevant (i.e. choosing the first entry may not always be a good idea):

zig/lib/std/os/windows.zig

Lines 1370 to 1371 in 074dd4d

	// Look for `\DosDevices\` prefix. We don't really care if there are more than one symlinks
	// with traditional DOS drive letters, so pick the first one available.

Will try to find some info on \DosDevices\*:

EDIT: Unable to find any info about *: so far.

[patrik-gustavsson]

I ran Get-PSDrive but found no * entry.

I did, however, insert a couple of usb-drives and they all seemed to work. It may be something to do with my secondary drive. I tried to change the drive letter but that didn't help.

[patrik-gustavsson]

I think I have narrowed down where the error comes from, the function ntdll.NtDeviceIoControlFile in lib.std.os.windows on line 364 seems to return back duplicate DosDevices in string, see below.

Input: \Device\HarddiskVolume6
Output for D:/ drive: \Device\HarddiskVolume6\DosDevices\*:\DosDevices\D:\??\Volume{guid}
Output for C:/ drive: \Device\HarddiskVolume3\DosDevices\C:\??\Volume{guid}

Didn't include everything, only focused on diff, there are some more bytes before and I think after as well.

It seems NtDeviceIoControlFile is deprecated, I will try with the replacement function DeviceIoControl and see if the error persists.

EDIT: same problem with DeviceIoControl

[patrik-gustavsson]

Finally found it, it seems to fetch the data from regedit under HKEY_LOCAL_MACHINE\SYSTEM\MountedDevices, for some reason I had the same drive connected to two keys: \DosDevices\*: and \DosDevices\D:

Since this had nothing to do with zig but my configuration on my computer I will close this issue.

EDIT: @squeek502 this was connected to what you mentioned with selecting the first entry, do not know in what circumstance more than one appears, but I seem to have gotten it at one point.

[squeek502]

Thanks for all the info. Worth noting that Zig is trying to emulate GetFinalPathNameByHandleW (see #1840), so when I get a chance I'll try reproducing the problem and then check what GetFinalPathNameByHandleW returns and go from there.

[relapids]

@squeek502 Fwiw GetFinalPathNameByHandleW does the mapping via IOCTLs to the MountMgr device.

VOLUME_NAME_DOS -> IOCTL_MOUNTMGR_QUERY_DOS_VOLUME_PATH (maybe could use RtlVolumeDeviceToDosName?)
VOLUME_NAME_NT -> Already in NT format
VOLUME_NAME_GUID -> IOCTL_MOUNTMGR_QUERY_POINTS (Not sure if there's an Rtl/Nt equivalent)

[squeek502]

@relapids the current implementation is doing exactly that:

zig/lib/std/os/windows.zig

Lines 1317 to 1446 in 0312391

	// Get DOS volume name. DOS volume names are actually symbolic link objects to the
	// actual NT volume. For example:
	// (NT) \Device\HarddiskVolume4 => (DOS) \DosDevices\C: == (DOS) C:
	const MIN_SIZE = @sizeOf(MOUNTMGR_MOUNT_POINT) + MAX_PATH;
	// We initialize the input buffer to all zeros for convenience since
	// `DeviceIoControl` with `IOCTL_MOUNTMGR_QUERY_POINTS` expects this.
	var input_buf: [MIN_SIZE]u8 align(@alignOf(MOUNTMGR_MOUNT_POINT)) = [_]u8{0} ** MIN_SIZE;
	var output_buf: [MIN_SIZE * 4]u8 align(@alignOf(MOUNTMGR_MOUNT_POINTS)) = undefined;
	// This surprising path is a filesystem path to the mount manager on Windows.
	// Source: https://stackoverflow.com/questions/3012828/using-ioctl-mountmgr-query-points
	// This is the NT namespaced version of \\.\MountPointManager
	const mgmt_path_u16 = std.unicode.utf8ToUtf16LeStringLiteral("\\??\\MountPointManager");
	const mgmt_handle = OpenFile(mgmt_path_u16, .{
	.access_mask = SYNCHRONIZE,
	.share_access = FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE,
	.creation = FILE_OPEN,
	}) catch |err| switch (err) {
	error.IsDir => return error.Unexpected,
	error.NotDir => return error.Unexpected,
	error.NoDevice => return error.Unexpected,
	error.AccessDenied => return error.Unexpected,
	error.PipeBusy => return error.Unexpected,
	error.PathAlreadyExists => return error.Unexpected,
	error.WouldBlock => return error.Unexpected,
	error.NetworkNotFound => return error.Unexpected,
	error.AntivirusInterference => return error.Unexpected,
	else => |e| return e,
	};
	defer CloseHandle(mgmt_handle);
	var input_struct: *MOUNTMGR_MOUNT_POINT = @ptrCast(&input_buf[0]);
	input_struct.DeviceNameOffset = @sizeOf(MOUNTMGR_MOUNT_POINT);
	input_struct.DeviceNameLength = @intCast(volume_name_u16.len * 2);
	@memcpy(input_buf[@sizeOf(MOUNTMGR_MOUNT_POINT)..][0 .. volume_name_u16.len * 2], @as([*]const u8, @ptrCast(volume_name_u16.ptr)));
	DeviceIoControl(mgmt_handle, IOCTL_MOUNTMGR_QUERY_POINTS, &input_buf, &output_buf) catch |err| switch (err) {
	error.AccessDenied => return error.Unexpected,
	else => |e| return e,
	};
	const mount_points_struct: *const MOUNTMGR_MOUNT_POINTS = @ptrCast(&output_buf[0]);
	const mount_points = @as(
	[*]const MOUNTMGR_MOUNT_POINT,
	@ptrCast(&mount_points_struct.MountPoints[0]),
	)[0..mount_points_struct.NumberOfMountPoints];
	for (mount_points) |mount_point| {
	const symlink = @as(
	[*]const u16,
	@ptrCast(@alignCast(&output_buf[mount_point.SymbolicLinkNameOffset])),
	)[0 .. mount_point.SymbolicLinkNameLength / 2];
	// Look for `\DosDevices\` prefix. We don't really care if there are more than one symlinks
	// with traditional DOS drive letters, so pick the first one available.
	var prefix_buf = std.unicode.utf8ToUtf16LeStringLiteral("\\DosDevices\\");
	const prefix = prefix_buf[0..prefix_buf.len];
	if (mem.startsWith(u16, symlink, prefix)) {
	const drive_letter = symlink[prefix.len..];
	if (out_buffer.len < drive_letter.len + file_name_u16.len) return error.NameTooLong;
	@memcpy(out_buffer[0..drive_letter.len], drive_letter);
	mem.copyForwards(u16, out_buffer[drive_letter.len..][0..file_name_u16.len], file_name_u16);
	const total_len = drive_letter.len + file_name_u16.len;
	// Validate that DOS does not contain any spurious nul bytes.
	if (mem.indexOfScalar(u16, out_buffer[0..total_len], 0)) |_| {
	return error.BadPathName;
	}
	return out_buffer[0..total_len];
	} else if (mountmgrIsVolumeName(symlink)) {
	// If the symlink is a volume GUID like \??\Volume{383da0b0-717f-41b6-8c36-00500992b58d},
	// then it is a volume mounted as a path rather than a drive letter. We need to
	// query the mount manager again to get the DOS path for the volume.
	// 49 is the maximum length accepted by mountmgrIsVolumeName
	const vol_input_size = @sizeOf(MOUNTMGR_TARGET_NAME) + (49 * 2);
	var vol_input_buf: [vol_input_size]u8 align(@alignOf(MOUNTMGR_TARGET_NAME)) = [_]u8{0} ** vol_input_size;
	// Note: If the path exceeds MAX_PATH, the Disk Management GUI doesn't accept the full path,
	// and instead if must be specified using a shortened form (e.g. C:\FOO~1\BAR~1\<...>).
	// However, just to be sure we can handle any path length, we use PATH_MAX_WIDE here.
	const min_output_size = @sizeOf(MOUNTMGR_VOLUME_PATHS) + (PATH_MAX_WIDE * 2);
	var vol_output_buf: [min_output_size]u8 align(@alignOf(MOUNTMGR_VOLUME_PATHS)) = undefined;
	var vol_input_struct: *MOUNTMGR_TARGET_NAME = @ptrCast(&vol_input_buf[0]);
	vol_input_struct.DeviceNameLength = @intCast(symlink.len * 2);
	@memcpy(@as([*]WCHAR, &vol_input_struct.DeviceName)[0..symlink.len], symlink);
	DeviceIoControl(mgmt_handle, IOCTL_MOUNTMGR_QUERY_DOS_VOLUME_PATH, &vol_input_buf, &vol_output_buf) catch |err| switch (err) {
	error.AccessDenied => return error.Unexpected,
	else => |e| return e,
	};
	const volume_paths_struct: *const MOUNTMGR_VOLUME_PATHS = @ptrCast(&vol_output_buf[0]);
	const volume_path = std.mem.sliceTo(@as(
	[*]const u16,
	&volume_paths_struct.MultiSz,
	)[0 .. volume_paths_struct.MultiSzLength / 2], 0);
	if (out_buffer.len < volume_path.len + file_name_u16.len) return error.NameTooLong;
	// `out_buffer` currently contains the memory of `file_name_u16`, so it can overlap with where
	// we want to place the filename before returning. Here are the possible overlapping cases:
	//
	// out_buffer: [filename]
	// dest: [___(a)___] [___(b)___]
	//
	// In the case of (a), we need to copy forwards, and in the case of (b) we need
	// to copy backwards. We also need to do this before copying the volume path because
	// it could overwrite the file_name_u16 memory.
	const file_name_dest = out_buffer[volume_path.len..][0..file_name_u16.len];
	const file_name_byte_offset = @intFromPtr(file_name_u16.ptr) - @intFromPtr(out_buffer.ptr);
	const file_name_index = file_name_byte_offset / @sizeOf(u16);
	if (volume_path.len > file_name_index)
	mem.copyBackwards(u16, file_name_dest, file_name_u16)
	else
	mem.copyForwards(u16, file_name_dest, file_name_u16);
	@memcpy(out_buffer[0..volume_path.len], volume_path);
	const total_len = volume_path.len + file_name_u16.len;
	// Validate that DOS does not contain any spurious nul bytes.
	if (mem.indexOfScalar(u16, out_buffer[0..total_len], 0)) |_| {
	return error.BadPathName;
	}
	return out_buffer[0..total_len];
	}
	}