Bug: TransactionLog BEGIN/END bitfields race between same-priority threads causes SIGABRT

[tfa2si]

Description

Severity

Critical — deterministic SIGABRT crash in real-time workloads (SCHED_FIFO >= 40)

Summary

S-CORE LoLa's TransactionLog::ReferenceTransactionBegin() asserts on BEGIN bit being TRUE when two threads at the same RT priority (e.g., RT thread + LoLa executor thread) concurrently access the same event slot via GetNewSamples(). The TransactionLogSlot uses non-atomic bitfields (uint8_t transaction_begin_ : 1; uint8_t transaction_end_ : 2;) with no synchronization, violating its own single-threaded assumption when Subscribe() spawns executor threads.

Root Cause

1. Non-atomic TransactionLogSlot

// transaction_log_slot.h
class TransactionLogSlot {
    std::uint8_t transaction_begin_ : 1;  // ← NOT atomic
    std::uint8_t transaction_end_ : 2;    // ← NOT atomic
};  

2. Unprotected BEGIN/END manipulation
`// transaction_log.cpp:117-122
void TransactionLog::ReferenceTransactionBegin(SlotIndexType slot_index) noexcept {
    TransactionLogSlot& slot = reference_count_slots_.at(slot_index);
    SCORE_LANGUAGE_FUTURECPP_PRECONDITION(!slot.GetTransactionBegin());  // ← ASSERTION
    WaitForTransactionEndToBecomeFalse(slot);
    slot.SetTransactionBegin(true);  // ← Plain write, no CAS
}`

3. Executor thread shares same TransactionLog
Subscribe() creates a LoLa executor/notification thread (see messaging/message_passing_service_instance.cpp:925-931). User code elevates this thread to SCHED_FIFO 80 via elevate_new_threads(). Both threads now access the same TransactionLog (allocated in SHM via ManagedMemoryResource).

4. Race condition at same priority
Executor Thread (SCHED_FIFO 80):     RT Thread (SCHED_FIFO 80):
ReferenceTransactionBegin(slot X)    [preempts — same priority, time-slices]
  → BEGIN=true                        ReferenceTransactionBegin(slot X)
                                      → PRECONDITION(!BEGIN) → FAIL → abort

**At same SCHED_FIFO priority, threads time-slice. Either thread can preempt the other mid-transaction, leaving BEGIN stuck TRUE.**
Why Existing Tests Miss This
S-CORE test suite runs at SCHED_OTHER — fair scheduling avoids preemption within transactions
S-CORE examples (e.g., sample_sender_receiver.cpp) use SCHED_OTHER
slot_collector.h warns "not thread-safe" but the TransactionLog has no such warning
AoU comment says "event sender has to be single-threaded/non-concurrent" — but consumer side (executor + user thread) is also multi-threaded
Reproduction
HFC Configuration: rt_priority: 80, rt_policy: SCHED_FIFO (mix config)

Subscribe() to any LoLa event
elevate_new_threads(priority=80, policy=SCHED_FIFO) — promotes executor thread
RT thread polls GetNewSamples() every 1-2ms
Executor thread also calls GetNewSamples() on event notifications (even without SetReceiveHandler, executor exists and can fire callbacks)
Both touch same TransactionLog → PRECONDITION fires → SIGABRT at ~cycle 300-500
Note: Crash also occurs with pure single-thread polling if elevate_new_threads() promotes the executor thread, even when user never calls SetReceiveHandler().

Workaround
Use SetReceiveHandler pattern — moves GetNewSamples() to a single executor thread. RT thread reads plain shared structs with zero S-CORE calls.

Affected Files
score/mw/com/impl/bindings/lola/transaction_log_slot.h — non-atomic bitfields
score/mw/com/impl/bindings/lola/transaction_log.cpp — PRECONDITION assertions, sleep_for()
score/mw/com/impl/bindings/lola/slot_collector.h — thread-safety warning (mentions slot_collector but not TransactionLog)


Additional Notes
sleep_for(10ms) in WaitForTransactionEndToBecomeFalse() is anti-pattern for RT systems
Design doc claims "wait-free algorithms" but TransactionLog path blocks
Crash is deterministic with SCHED_FIFO ≥ 40, timing-dependent (~300-500 cycles)

### Error Occurrence  Rate

None

### How to reproduce

_No response_

### Supporting Information

_No response_

### Classification

minor

### Affected Version

v0.3.0

### Category

- [ ] Safety Related
- [ ] Security Related

[tfa2si]

We hit a different symptom of this same bug: a silent freeze in the notification dispatch when a second subscriber joins a service instance where one subscriber uses pure polling.

Trigger Conditions

• Service instance has at least 2 events
• One subscriber (process A) uses pure polling (GetNewSamples with no SetReceiveHandler) on event 1
• Another subscriber (process B) calls Subscribe() on event 2 of the same service instance
• Result: process A's notification handler (for an event on a different service instance) deterministically stops firing

Our Exact Setup

• HfcZoneFeedback service instance has 2 events: zone_angle_batch (pure polling by HFC) and zone_stats (event-driven by telemetry)
• HFC's RT thread polls zone_angle_batch via GetNewSamples — no notification thread involved
• Telemetry calls zone_stats.Subscribe(64U) — this writes a TransactionLogNode via CAS into SHM
• zone_adapter (the skeleton process) has a different proxy (HfcZoneCommand on another service instance) with SetReceiveHandler
• Result: the instant telemetry subscribes, zone_adapter's zone_cmd handler stops firing. However, zone_adapter's own stats thread (which bypasses LoLa notifications and calls zone_stats.Send() directly) continues running fine.

Why Pure Polling Matters

With pure polling, the skeleton's notification thread for the polled event (zone_angle_batch) still exists in the LoLa executor thread pool even though SetReceiveHandler was never called. This executor thread shares the same TransactionLogSet SHM region. When Subscribe writes to proxy_transaction_logs_ via CAS for a different event in the same ServiceDataControl, the concurrent iteration by the executor thread reads corrupted state from non-atomic TransactionLogSlot bitfields.

Key Insight

The race requires the skeleton process to have an active LoLa executor thread (created by any Subscribe + SetReceiveHandler in that process, not necessarily on the same event) that iterates SHM while a concurrent RegisterProxyElement from another process writes to SHM.

Workaround Applied

Moved zone_stats to its own dedicated service instance (HfcZoneStats, serviceId 6666) with separate SHM segment. No shared TransactionLogSet across events = no race.

Test Results

With the workaround:

• zone_angle_batch.Send() — 1623 calls over 15 seconds
• zone_stats.Send() — 9 calls over 15 seconds
• Both running concurrently, zero freeze