Feature Request: Variable-size "long" for simplified p/invoke

[jherby2k]

To greatly simplify p/invoke code (particularly struct definitions, but also method signatures), it would be amazing if there was a "long" type that matched the OS's compiler definition of long. This would mean no more preprocessor definitions, and a single assembly that works with native libraries compiled for multiple OSes.

Here's an example of what you have to do in .net right now to p/invoke ogg_packet, defined here:

C code:

typedef struct {
  unsigned char *packet;
  long  bytes;
  long  b_o_s;
  long  e_o_s;
  ogg_int64_t  granulepos;
  ogg_int64_t  packetno;
} ogg_packet;

C# code:

[StructLayout(LayoutKind.Sequential)]
struct OggPacket
{
    IntPtr Packet;
#if WINDOWS             // long is always 4 bytes on Windows
    int Bytes;
    int BeginningOfStream;
    int EndOfStream;
#else                   // long matches the word length on Linux/OSX
    IntPtr Bytes;
    IntPtr BeginningOfStream;
    IntPtr EndOfStream;
#endif
    long GranulePosition;
    long PacketNumber;
}

Ugly, and I need to ship separate binaries for Windows and OSX/Linux.

To be clear, this is partly about word length and partly about what "long" means in compiled C code on Windows (vs basically every other OS):

Type	Windows 32-bit	Windows 64-bit	*nix 32-bit	*nix 64-bit
C# IntPtr	4	8	4	8
C# int (Int32)	4	4	4	4
C# long (Int64)	8	8	8	8
C long (no managed equivalent)	4	4	4	8

https://software.intel.com/en-us/articles/size-of-long-integer-type-on-different-architecture-and-os/

I get that this is technically up to the compiler, not the OS, but I believe the dust has settled on these sizes since the shift from 16->32->64 bits.

[jherby2k]

Thinking some more about implementation, it might make the most sense to just go with something like [MarshalAs(UnmanagedType.CLong)] applied to a regular Int32 (or UInt32), like so:

[StructLayout(LayoutKind.Sequential)]
struct OggPacket
{
    IntPtr Packet;
    [MarshalAs(UnmanagedType.CLong] int Bytes;
    [MarshalAs(UnmanagedType.CLong] int BeginningOfStream;
    [MarshalAs(UnmanagedType.CLong] int EndOfStream;
    long GranulePosition;
    long PacketNumber;
}

[EgorBo]

Looks like what you need is proposed here https://github.com/dotnet/corefxlab/blob/master/docs/specs/nativesized.md

[jherby2k]

@EgorBo That refers to types that vary in size by word length (32-bit, 64-bit etc.). Different request but easy to confuse (I originally used the term "native long" but changed it to try and avoid this confusion).

[Clockwork-Muse]

....so why isn't the native code considered "buggy", in that it's using a type with a variable size definition? Especially since the next set of fields has an explicit size?

(Frankly, this is one of my least favorite C/C++ "features", for mostly this reason - that, and in almost all cases the far more interesting information is the range of the domain type, not the size of the physical type)

[jherby2k]

The merits of using long isn’t really the point.. my example is a pre-existing API already present on most Linux distributions. And it isn’t particularly obscure - I can provide other examples.

[tannergooding]

[MarshalAs(UnmanagedType.CLong] int EndOfStream;

This isn't sufficient. This would not allow the upper 32-bits to be used on 64-bit Linux.
Likewise, using long could cause a loss of data on Windows, if managed code were to set the upper 32-bits.

If this were to be handled, it would likely need to be done by an explicit type that the runtime will dynamically size as needed (much like IntPtr or Vector<T>)

[jherby2k]

I suppose that’s potentially true, although in many (most?) cases the upper 32 bits are not particularly useful (considering that the code is assumed to work on platforms with 4 byte longs). I’m curious if there is any new code being written with longs. My assumption is that their occurance in code is a legacy holdover from when 64-bit architectures were a pipe dream.

Totally not against a “CLong” type, just playing this through. Any solution would be super helpful.

[tannergooding]

I would like to hope that most "modern/new" C/C++ code is being written to be cross-platform and cross-architecture aware and that they will use explicitly sized types (int32_t, int64_t, intptr_t, etc), rather than relying on the C/C++ language keywords (int, long, long long, etc). And looking around at various "newer" codebases that are explicitly meant to be cross-platform/architecture, they tend to do this (clang, llvm, vulkan, among others).

In an ideal world, many of these codebases (such as Ogg) would have been updated to use intptr_t instead of long when 64-bit computers first started coming out (and the difference between Unix and other platforms was realized). This would have ensured that both Unix and other platforms (such as Windows) had consistently sized structures. But that didn't happen, and left us with the current mess. Due to the differences in size, and common existence for older code-bases, I think its something that still needs some sort of answer, and having a specially recognized type that is the appropriate size on each platform seems like the best solution. This ensures that you have no potential for data loss (when dealt with properly) and gives you a single entry-point for the function interop, rather than needing to cross-compile everything.

[jkotas]

[MarshalAs(UnmanagedType.CLong] int EndOfStream;
This isn't sufficient. This would not allow the upper 32-bits to be used on 64-bit Linux.

I think [MarshalAs(UnmanagedType.CLong] on int or long would be sufficient. It would work just like any other interop marshaling and throw exceptions when the data does not fit.

If this were to be handled, it would likely need to be done by an explicit type that the runtime will dynamically size as needed

I do not think a special casing like this would be worth it in this case.

[tannergooding]

I do not think a special casing like this would be worth it in this case.

This keeps coming up (the most notable likely being from @migueldeicaza: https://github.com/dotnet/coreclr/issues/963), and the only "correct" solution, today, is to cross-compile.

It would work just like any other interop marshaling and throw exceptions when the data does not fit.

Which requires you to actually marshal the data, rather than allowing pinning or blitting of the value on 64-bit. If you expect to support 64-bit, you have to use long (or intptr), because you can't necessarily guarantee that the input (from either side) is always less than 32-bits on non-Windows. This also means that Windows is less efficient/etc.

We could potentially just handle this at the framework level, with #ifdef and a Unix vs Windows specific implementation (much like we do for IntPtr).

[jkotas]

dotnet/coreclr#963 is a different issue from this one. I think that paying the price to marshal the data is perfectly fine for CLong.

[jherby2k]

What kind of performance overhead are we talking about? Just the overhead of copying arrays of these things vs pinning, or would even my example be affected somewhat? I’d be reluctant to implement anything that slowed my solution down noticeably. Would probably stick with cross compiling in performance critical code if that is the case.

[jkotas]

would even my example be affected somewhat?

You example would get some extra instructions as well, in some cases at least.

MarshalAs(UnmanagedType.CLong) would have similar performance characteristics and limitations as passing bool over managed/unmanaged boundary. C# bool has to be converted to C++ bool, similar to how C# int or long has to be converted to C++ long here.

[jherby2k]

I think the best argument for not introducing a new type is that this issue relates to legacy / awkward code interop only. first-class support seems like overkill.

On the other hand, if adding the type isn’t too hard, the performance advantages might be nice. And it’s probably cleaner.

I will leave it up to you experts. both solutions are acceptable and I’d just like to see something as soon as possible!

Just want to make sure the attribute would deal with sign extension.

[jkotas]

I think the best argument for not introducing a new type is that this issue relates to legacy / awkward code interop only

+1

adding the type isn’t too hard, the performance advantages might be nice. And it’s probably cleaner.

Adding a first class type for this is complex. Many different parts of the system would have to know about it, depending on how much first class this type would be. I do not think it is cleaner from the architecture point of view at least. The interop specific constructs should stay within the interop subsystem.