Possible DefaultStorageProvider concurrency issue (System.IO.IOException)?

[daldridge-dt]

I'm using Analytics-CSharp v2.5.0 in an ASP.Net web application. Dependency injection setup is:

services.AddScoped(sp =>
{
    var persistentDataPath = Path.GetTempPath();
    var logger = sp.GetRequiredService<IDatadogLogger>();
    logger.LogInformation($"Segment Analytics is configured with persistentDataPath: {persistentDataPath}");
    
    var analytics = new Analytics(new Segment.Analytics.Configuration(
        settings.BusinessEvents.WriteKey,
        apiHost: settings.BusinessEvents.ApiHost,
        flushAt: settings.BusinessEvents.FlushAtEventCount,
        flushInterval: settings.BusinessEvents.FlushIntervalSec,
        storageProvider: new DefaultStorageProvider(persistentDataPath)));
    analytics.Add(new UserIdEnrichmentPlugin());
    Analytics.Logger = sp.GetRequiredService<SegmentLogger>();
    return analytics;
});

FlushAt=20 and FlushInterval=30 in this scenario.

The SegmentLogger class implements ISegmentLogger and forwards to DataDog. The UserIdEnrichmentPlugin mutates the RawEvents by lifting a userId in properties to the root of the event. (Neither should be relevant to the IO exception issue. I'm just describing them here for completeness.)

I have verified that file system permissions are correct, and the majority of the events we emit via Analytics.Track() are being received; however, we have roughly 16% event loss (never received), each corresponding with an error log similar to:

System.IO.IOException: The process cannot access the file '/tmp/segment.data/VZiEGasTFTYm557jKVM0Z8d38XxRuns8/events/VZiEGasTFTYm557jKVM0Z8d38XxRuns8-0' because it is being used by another process.
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Init(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Int64& fileLength, UnixFileMode& filePermissions)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, UnixFileMode openPermissions, Int64& fileLength, UnixFileMode& filePermissions, Boolean failForSymlink, Boolean& wasSymlink, Func`4 createOpenException)
   at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize, Nullable`1 unixCreateMode)
   at System.IO.FileInfo.Open(FileMode mode, FileAccess access)
   at Segment.Analytics.Utilities.FileEventStream.OpenOrCreate(String file, Boolean& newFile)
   at Segment.Analytics.Utilities.Storage.<>c__DisplayClass32_0.<<StoreEvent>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Segment.Analytics.Utilities.Storage.WithLock(Func`1 block)
   at Segment.Analytics.Utilities.Storage.StoreEvent(String event)
   at Segment.Analytics.Utilities.Storage.Write(StorageConstants key, String value)
   at Segment.Analytics.Utilities.EventPipeline.<Write>b__23_0()

This ASP.Net application should be the only process accessing /tmp/segment.* directories. (FWIW, this is a Docker image running on Kubernetes/AKS, using a base image of mcr.microsoft.com/dotnet/aspnet:8.0-jammy).

We're likely going to switch to InMemoryStorageProvider with flushAt=1 as a short-term "fix" to avoid losing events, but I'd like to understand the root cause of the System.IO.IOException. Do you have any suggestions on how to determine whether it's a defect in the storage provider versus an environment/host issue?

Thanks!

[wenxi-zeng]

hi @daldridge-dt, thanks for reporting this. we take event loss very seriously. will prioritize on this issue and get to back to you asap. in the meanwhile, yes, please use InMemoryStorageProvider as workaround.

[wenxi-zeng]

hi @daldridge-dt after investigation, the issue is likely caused by multiple instance of analytics using the same writekey. and with DefaultStorageProvider, all instances of analytics will write into the same file thus causing the problem (fyi, the SDK is thread-safe, but concurrent access to the same file from multiple processes is not safe). so the solutions would be as follow:

1. use Analytics in a singleton scope. this probably is the most performant way
2. write a custom storage provider to support multiple instances. you would want to
   • map each instance to a dedicated storage
   • map each page/endpoints to a dedicated instance
3. alternative to 2, write a database storage provider which is atomic and safe for concurrent access via multi-processes

I'd suggest to try the singleton approach to confirm that's the root cause in your environment, since it's a quick and simple change. once it can be confirmed that's the root cause, follow this example on customizing storage to support multiple instances.

since it's a valid scenario for all .net users, we'll add an example of MultiInstancesStorageProvider once we can confirm that's what causing your event loss, and will update our doc for .net use case accordingly.

[daldridge-dt]

Thanks @wenxi-zeng . I've confirmed both the Singleton instance, and in-memory storage provider both solve our issue 👍

[wenxi-zeng]

@daldridge-dt great to hear that! I'll close this ticket out and will update you once we got an example for multi-instance storage ready. thanks for bearing with us!

[remcoros]

If you make Analytics a singleton, don't you run into concurrency issues regarding Identify and Group calls? As far as I understand, the user and group is stored inside Analytics class, so this would mean (for a .NET Web API with concurrent requests from different users and groups) they all get mixed?

[wenxi-zeng]

@remcoros the legacy Analytics.NET requires a userId for every call. you would have to do the same for this SDK as well. the SDK has a internal state that's useful for client cases, but is no necessary for server use cases. please check the discussion below.
#75 (comment)

again we are updating the docs to include all these information, so it'd be less confusing in the future.