Dead Lock when @Serializable + companion object

[erkas-c]

Describe the bug

@Serializable
sealed class TestClass {
    @Serializable data class Test1(val a: Int): TestClass()
    @Serializable data class Test2(val b: Int): TestClass()

    companion object {
        val defaultTest1 = Test1(1)
        val defaultTest2 = Test2(2)
    }
}

The above code can cause a deadlock.
Since @Serializable uses LazyThreadSafetyMode, a lock is applied when creating the Test1, Test2, and Test3 classes.

coroutineScope {
    val taskA = async {
        Test2(10) // A 
    }
    val taskB = async {
        Test1(10) // B
    }
    taskA.await()
    taskB.await()
}

Executing the above code may result in a deadlock.
If a lock is applied to Test2 at point A and to Test3 at point B,
they will be waiting on defaultTest2 and defaultTest3 in the companion object.

One way to resolve this issue is to avoid creating Test* instances inside the companion object.
However, if a developer is unaware of this issue, they may struggle to debug it.

Therefore, I suggest generating an error or warning for this case.

To Reproduce
executing the above code

Expected behavior
generating an error or warning for this case. by compile time

Environment

• Kotlin version: 2.0.20
• Library version: 1.6.3
• Kotlin platforms: Android
• Gradle version: 8.10.2
• IDE version (if bug is related to the IDE) Android Studio Ladybug Feature Drop | 2024.2.2

[shanshin]

Hi, could you please clarify, have you encountered a deadlock or do you assume it exists due to the use of LazyThreadSafetyMode?

We use the PUBLICATION mode, which does not involve setting the lock, it uses the AtomicReferenceFieldUpdater.

[pdvrieze]

When looking at the example code I didn't see a deadlock, although there is likely to be an initialization loop causing uninitialised constants to be visible (lazy breaks such loops). Initialization is synchronized by the JVM though, and should not deadlock.

[erkas-c]

@shanshin
The Kotlin bytecode for the Companion object of TestClass includes the following instructions:

LINENUMBER 12 L2
    GETSTATIC kotlin/LazyThreadSafetyMode.PUBLICATION : Lkotlin/LazyThreadSafetyMode;
    INVOKEDYNAMIC invoke()Lkotlin/jvm/functions/Function0; [
      // handle kind 0x6 : INVOKESTATIC
      java/lang/invoke/LambdaMetafactory.metafactory(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;Ljava/lang/invoke/MethodType;Ljava/lang/invoke/MethodHandle;Ljava/lang/invoke/MethodType;)Ljava/lang/invoke/CallSite;
      // arguments:
      ()Ljava/lang/Object;, 
      // handle kind 0x6 : INVOKESTATIC
      org/TestClass._init_$_anonymous_()Lkotlinx/serialization/KSerializer;, 
      ()Lkotlinx/serialization/KSerializer;
    ]
    INVOKESTATIC kotlin/LazyKt.lazy (Lkotlin/LazyThreadSafetyMode;Lkotlin/jvm/functions/Function0;)Lkotlin/Lazy;
    PUTSTATIC org/TestClass.$cachedSerializer$delegate : Lkotlin/Lazy;
   L3

This code is only added when the @serializable annotation is present.

From this observation, it can be inferred that @serializable involves the use of thread locking.

The example code in the above discussion actually led to a deadlock situation.

In a scenario where two async tasks are running concurrently on a system with at least two CPU cores, a deadlock occurs during the initialization of the Companion object.

[erkas-c]

@pdvrieze
I cannot agree with the claim that the Companion object is synchronized by the JVM.

In Kotlin, an object class is not initialized at JVM load time but rather at runtime when it is first accessed.

In fact, the above code actually caused a deadlock, which resulted in all coroutine threads being blocked. This made it extremely difficult to identify the root cause.

Here is my hypothesized scenario for how the deadlock occurs:

1. Thread A attempts to create an instance of Test2 → Acquires a lock on Test2.
2. Thread B attempts to create an instance of Test1 → Acquires a lock on Test1.
3. Thread B attempts to initialize the Companion object.
4. Thread B tries to create defaultTest1, requiring an instance of Test1 → Succeeds.
5. Thread B tries to create defaultTest2, requiring an instance of Test2 → Blocked, as Test2 is locked by Thread A.
6. Thread A attempts to initialize the Companion object.
7. Thread A tries to create defaultTest1, requiring an instance of Test1 → Blocked, as Test1 is locked by Thread B.

This screenshot is from an actual app, showing the code in use. After the deadlock occurred, I paused execution in debug mode and confirmed that all coroutine threads were in a waiting state. Additionally, I verified that the last variable in the Companion object was null.

For reference, the variable being null was not caused by hitting a breakpoint at that point.

[shanshin]

@shanshin The Kotlin bytecode for the Companion object of TestClass includes the following instructions:

LINENUMBER 12 L2
    GETSTATIC kotlin/LazyThreadSafetyMode.PUBLICATION : Lkotlin/LazyThreadSafetyMode;
    INVOKEDYNAMIC invoke()Lkotlin/jvm/functions/Function0; [
      // handle kind 0x6 : INVOKESTATIC
      java/lang/invoke/LambdaMetafactory.metafactory(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;Ljava/lang/invoke/MethodType;Ljava/lang/invoke/MethodHandle;Ljava/lang/invoke/MethodType;)Ljava/lang/invoke/CallSite;
      // arguments:
      ()Ljava/lang/Object;, 
      // handle kind 0x6 : INVOKESTATIC
      org/TestClass._init_$_anonymous_()Lkotlinx/serialization/KSerializer;, 
      ()Lkotlinx/serialization/KSerializer;
    ]
    INVOKESTATIC kotlin/LazyKt.lazy (Lkotlin/LazyThreadSafetyMode;Lkotlin/jvm/functions/Function0;)Lkotlin/Lazy;
    PUTSTATIC org/TestClass.$cachedSerializer$delegate : Lkotlin/Lazy;
   L3

This code does not set locks, it only creates a delegate.
It is an analog of such a code:
$cachedSerializer$delegate = LazyKt.lazy(LazyThreadSafetyMode.PUBLICATION) { _init_$_anonymous_() }

$cachedSerializer$delegate.getValue is not called in the companion constructor, therefore, the value of LazyThreadSafetyMode does not affect anything.

[shanshin]

For reference, the variable being null was not caused by hitting a breakpoint at that point.

According to this screenshot, the thread is not stopped while waiting for some lock from $cachedSerializer$delegate, therefore it can be concluded that LazyThreadSafetyMode has nothing to do with a potential deadlock.

[pdvrieze]

@erkas-c

Here is my hypothesized scenario for how the deadlock occurs:

1. Thread A attempts to create an instance of Test2 → Acquires a lock on Test2.
2. Thread B attempts to create an instance of Test1 → Acquires a lock on Test1.
3. Thread B attempts to initialize the Companion object.
4. Thread B tries to create defaultTest1, requiring an instance of Test1 → Succeeds.
5. Thread B tries to create defaultTest2, requiring an instance of Test2 → Blocked, as Test2 is locked by Thread A.
6. Thread A attempts to initialize the Companion object.
7. Thread A tries to create defaultTest1, requiring an instance of Test1 → Blocked, as Test1 is locked by Thread B.

That is not quite what happens.

1. Thread A refers to class Test2. The JVM notices this class is not loaded yet, so it tries to load Test2 (with JVM built-in synchronization that includes not exposing the result of loading the class until the static initializer has run).
2. When loading Test2 the JVM needs to load parent class TestClass and it initiates loading of TestClass
3. TestClass contains the static variable INSTANCE of type TestClass$Companion, as this is not yet loaded loading TestClass$Companion is initiated
4. As TestClass$Companion contains attributes of types Test1 and Test2 they need to be loaded. Test2 is skipped (already being loaded - one of the initialization loops) and Test1 is initialized
5. For Test1 the parent type TestClass is already being loaded so initialisation suspends after initial loading
6. Test2 can now continue loading by loading the Lazy class (type parameters are erased so irrelevant)
7. At this point all types are loaded (but not initialized). There is no obvious order for class initialization, but as
8. The class initializers can now initialize the classes (that should all be loaded). The challenge is static members. These are either the companion instances or serializers. Serializers are implemented to be loaded lazily, and as such have no further initialization yet.

[erkas-c]

@shanshin, @pdvrieze
I appreciate both of your insights.

If my analysis of the thread deadlock caused by @Serializable is incorrect, then the question remains—why does the deadlock occur?

Here is a summary of the issue I encountered:

1. When @Serializable is not present, no deadlock occurs.
2. When @Serializable is present, the async tasks occasionally do not complete.

coroutineScope {
    val taskA = async {
        Test2(10) // A  
        // some code  
    }
    val taskB = async {
        Test1(10) // B  
        // some code  
    }
    taskA.await()
    taskB.await()
}

This means that the "some code" section in the tasks does not execute.

Based on this observation, I concluded that @Serializable causes thread locking and leads to a deadlock.

If there is no locking and no deadlock, then why does the "some code" section fail to execute?
I would like to hear your thoughts on this.
Thanks

[fzhinkin]

@erkas-c it is hard to tell what causes the issue without being able to reproduce it. Code from the summary section of this issue does not help as it works perfectly fine for anyone who attempted to run it. Perhaps, you can extract a trimmed-down reproducer from your project, so we can debug it and try to find the root cause?

Also, have you already tried to diagnose the issue by inspecting thread dumps created when the app hangs or by using a tool like JConsole's Detect deadlock?