Invalid top array size (ref: 2224064, array size: 655360) file size: 0

[BlueCobold]

SDK and version

SDK : RealmKotlin
Version: 2.2.0 (Core 14.12.0)

Observations

• How frequent do the crash occur?
  Rarely
• Does it happen in production or during dev/test?
  Production
• Can the crash be reproduced by you?
  No
• Can you provide instructions for how we can reproduce it?
  No

Crash log / stacktrace

Failed to memory buffer:Invalid top array size (ref: 2224064, array size: 655360) file size: 0, read lock size: none, read lock version: none:
java.lang.IllegalStateException: [RLM_ERR_INVALID_DATABASE]: Failed to open Realm file at path "....": Failed to memory buffer:Invalid top array size (ref: 2224064, array size: 655360) file size: 0, read lock size: none, read lock version: none
at io.realm.kotlin.internal.interop.CoreErrorConverter.asThrowable(SourceFile:96)
at io.realm.kotlin.internal.interop.realmcJNI.realm_open(Native Method)
at io.realm.kotlin.internal.interop.realmc.realm_open(SourceFile:1)
at io.realm.kotlin.internal.interop.RealmInterop.realm_open(SourceFile:41)
at io.realm.kotlin.internal.ConfigurationImpl.openRealm$lambda$0(SourceFile:18)
at io.realm.kotlin.internal.ConfigurationImpl.a(SourceFile:1)
at io.realm.kotlin.internal.b.invoke(SourceFile:1)
at io.realm.kotlin.internal.interop.NativePointerKt.use(SourceFile:11)
at io.realm.kotlin.internal.ConfigurationImpl.openRealm$suspendImpl(SourceFile:20)
at io.realm.kotlin.internal.ConfigurationImpl.openRealm(SourceFile:1)
at io.realm.kotlin.internal.RealmImpl$1.invokeSuspend(SourceFile:138)
at kotlin.coroutines.jvm.internal.a.resumeWith(SourceFile:12)
at kotlinx.coroutines.DispatchedTask.run(SourceFile:129)
at kotlinx.coroutines.EventLoopImplBase.processNextEvent(SourceFile:82)
at kotlinx.coroutines.BlockingCoroutine.joinBlocking(SourceFile:27)
at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(SourceFile:77)
at kotlinx.coroutines.BuildersKt.runBlocking(SourceFile:1)
at io.realm.kotlin.internal.platform.CoroutineUtilsSharedJvmKt.runBlocking(SourceFile:11)
at io.realm.kotlin.internal.platform.CoroutineUtilsSharedJvmKt.runBlocking$default(SourceFile:7)
at io.realm.kotlin.internal.RealmImpl.\u003cinit\u003e(SourceFile:15)
at io.realm.kotlin.internal.RealmImpl.\u003cinit\u003e(SourceFile:1)
at io.realm.kotlin.internal.RealmImpl$Companion.create$io_realm_kotlin_library(SourceFile:9)
at io.realm.kotlin.Realm$Companion.open(SourceFile:10)

Steps & Code to Reproduce

I sadly do not have the file in question. However, a broken topRef means there are still bugs which make Realm not ACID and a reported file size of 0 means the error message is incorrect as well, because if it was 0 bytes in size, there would not be a readable topRef at all.

[sync-by-unito]

➤ PM Bot commented:

Jira ticket: RCORE-2260

[nicola-cab]

Hello @BlueCobold thanks for reporting this. Without the file however it is almost impossible to understand what happened.

Clearly, the top ref array is not valid:

realm-core/src/realm/group.cpp

Line 493 in c729fc8

void Group::validate_top_array(const Array& arr, const SlabAlloc& alloc, std::optional<size_t> read_lock_file_size,

I can only think about something bad happening when a transaction was committed.
Do you have more information on your side that can give us some tips about how to try to reproduce this issue?

[BlueCobold]

I'm afraid I can not provide more info that this crash report. I understand it's probably super hard to identify. I can say that access to the realm is temporary opened and closed in a single-threaded exclusive way. If it's impossible to find a reason for such an invalid top-ref, I'm sure you can at least correct the wrong error-message which claims the file-size to be 0, which is logically impossible if a top-ref can be read.

[jedelbo]

The file size we are referring to here is the logical file size that is stored in the top array. So if the top ref is pointing to some invalid array, that size can perfectly fine be 0.
Unfortunately we have not been able to find the reason why the top ref can be wrong. Can you tell on which platform this is happening? Historically we have only seen these kind of problems on iOS. So the theory is that something weird is going on in the file system - possibly if the platform crashes.

[BlueCobold]

Ah, I see. So it's a confusing message, because it says "array size: 655360), file size: 0".

It happened on Android.

[nicola-cab]

@jedelbo I may be wrong (because I don't recall in detail what we did). But as far as I remember, we only added the extra msync call for IOS and left the management of the dirty pages untouched for the other platforms. I repeat, I might be wrong on this.
Probably this is just speculation, though.

[sync-by-unito]

➤ jedelbo commented:

Without having the file, I can't see how we can move forward on this issue, I I will close it.

[BlueCobold]

@jedelbo The issue persists. I keep getting reports.

Failed to open Realm file at path "...": Failed to memory buffer:Invalid top array size (ref: 2882016, array size: 16777215) file size: 108086541928145056, read lock size: none, read lock version: none

[BlueCobold]

I had that error happen again locally today with a file I had opened in RealmStudio and trying to compact it externally from another C# process (Realm DotNet 20.0.0). And I can reproduce it. The file itself seems fine, though. As soon as I close RealmStudio (15.2.1) and compact the file again, it works without an issue.

Failed to memory buffer:Invalid top array size (ref: 2732888, array size: 16777215)
 file size: 9223372036854114051, read lock size: some(2732944), read lock version: some(2006)

[jedelbo]

@BlueCobold can you give a bit more detail on how to reproduce this error. I am also not sure where the error is seen.

[BlueCobold]

@jedelbo The last crash I report was caused by:

1. Opening an encrypted realm file in RealmStudio and keeping it open - just simply viewing it
2. Launching a C# project (probably any realm-core would do) to perform a compaction on said file
   Result:
   The C# process natively crashes with given the given log output.

As soon as I close RealmStudio, the C# process would compact the file without a crash.

I know it's not exactly identical to the original report I made, because the C# one includes a lock size while the original reported "none". But it's still claiming very weird file sizes as you can see.
I also cannot tell if the way to "reproduce" is the same reason why I keep getting crash reports from Android users, but it does read pretty much the same and since I do use compaction in my app, it is not entirely out of possibility. If it's not the reason, the given crash using RealmStudio and C# would still be very misleading and confusing. I would expect to either not be able to open the file due to being process-locked by RealmStudio or to result in a different kind of error.

PS: I am running on MacOS - M1 - so ARM CPU just as on Android devices.

[jedelbo]

@BlueCobold I am not sure what actually happens. It should not be possible to compact a file while another process has the file open. I don't have a Mac, so I can't tell if this test is failing on MacOS. I will try to get one of my colleagues to test this.

[jedelbo]

@ironage can you check if you can compact a file while it is kept open by RealmStudio? I assume you have an M1 device.

[ironage]

I ran the test on my M1 machine as outlined below and was not able to reproduce. I used Studio 15.2.1 and realm-core 14.13.3. I tried with an encrypted file, and non-encrypted file. In both cases, I observed the expected behaviour that compact does nothing (returns false) if the file is opened in Studio, or compact succeeds if Studio does not have the file open.

1. Opening an encrypted realm file in RealmStudio and keeping it open - just simply viewing it
2. Launching a C# project (probably any realm-core would do) to perform a compaction on said file

[jedelbo]

As we are not able to easily reproduce this, I don't think we have the resources to persue this issue further. If you can share a complete repro case, you are welcome to reopen the issue.