Mempool shutdown crash

[pavel-kokolemin]

The GUI node crashes inside the mempool task if stopped while downloading blocks.
I can't reproduce the crash with node-daemon, but I don't think the crash is GUI specific.
It looks like mempool is trying to access chainstate after chainstate has been stooped.

Debug log:

[2023-07-03T03:59:35Z DEBUG p2p::sync] Broadcasting a new tip header 0x63b4…2bf7                                                                                                                   
[2023-07-03T03:59:35Z DEBUG p2p::sync::peer] Block (0xe75e…679b) from peer 1                                                                                                                       
[2023-07-03T03:59:35Z INFO  mempool::pool] mempool: Processing chainstate event NewTip(Id<Block>{0x63b4d51de19b86fe4382425e721e12a8b028b924f6270c4f00b51e40fc982bf7}, BlockHeight(16038))          
[2023-07-03T03:59:35Z INFO  mempool::pool] new tip: block Id<Block>{0x63b4d51de19b86fe4382425e721e12a8b028b924f6270c4f00b51e40fc982bf7} height BlockHeight(16038)                                  
[2023-07-03T03:59:35Z DEBUG p2p::sync::peer] Peer 2 sent a block that already exists (0x0997…c4a2)                                                                                                 
[2023-07-03T03:59:35Z DEBUG p2p::sync::peer] Block (0x1193…5ec7) from peer 2                                                                                                                       
[2023-07-03T03:59:35Z INFO  chainstate::detail] Processing block 0xe75e…679b, attempt #0                                                                                                           
[2023-07-03T03:59:35Z INFO  chainstate::detail] Updating status for block 0xe75e…679b, attempt #0                                                                                                  
[2023-07-03T03:59:35Z INFO  subsystem::manager] Subsystem mintlayer/ctrl-c terminated                                                                                                              
[2023-07-03T03:59:35Z INFO  subsystem::manager] Subsystem mintlayer/p2p terminated                                                                                                                 
[2023-07-03T03:59:35Z INFO  subsystem::manager] Subsystem mintlayer/rpc_test_functions terminated                                                                                                  
[2023-07-03T03:59:35Z ERROR blockprod::detail::job_manager] Error sending shutdown during job manager drop: SendError(Sender { inner: Some(Inner { state: State { is_complete: false, is_closed: false, is_rx_task_set: false, is_tx_task_set: false } }) })                                                                                                                                          
[2023-07-03T03:59:35Z INFO  subsystem::manager] Subsystem mintlayer/blockprod terminated                                                                                                           
[2023-07-03T03:59:35Z INFO  subsystem::manager] Subsystem mintlayer/rpc terminated                                                                                                                 
[2023-07-03T03:59:35Z INFO  chainstate::detail] New tip in chainstate 0xe75e…679b with height 16039, timestamp: 1688327545                                                                         
[2023-07-03T03:59:35Z ERROR subsystem::subsystem] Shutdown channel sender closed prematurely                                                                                                       
[2023-07-03T03:59:35Z ERROR utils::tap_error_log] Chainstate subscriber failed to send new tip: channel closed                                                                                     
[2023-07-03T03:59:35Z ERROR utils::tap_error_log] The new tip receiver closed: channel closed                                                                                                      
[2023-07-03T03:59:35Z ERROR utils::tap_error_log] Chainstate subscriber failed to send new tip: channel closed                                                                                     
[2023-07-03T03:59:35Z INFO  subsystem::manager] Subsystem mintlayer/chainstate terminated                                                                                                          
thread 'tokio-runtime-worker' panicked at 'Database error while reading utxos best block: Subsystem(ResultFetchFailed)', /home/admin0/rep/mintlayer/mintlayer-core/chainstate/tx-verifier/src/transaction_verifier/mod.rs:166:14                                                                                                                                                                      

Stack:

#11 0x0000555558c59563 in core::result::unwrap_failed () at library/core/src/result.rs:1651                                                                                                        
#12 0x000055555817c207 in core::result::Result<common::primitives::id::Id<common::chain::gen_block::GenBlock>, mempool::pool::tx_verifier::chainstate_handle::Error>::expect<common::primitives::id::Id<common::chain::gen_block::GenBlock>, mempool::pool::tx_verifier::chainstate_handle::Error> (self=..., msg="Database error while reading utxos best block")                                    
    at /rustc/78a6ac0a86c3feda182dea496c99fc7f866a22e9/library/core/src/result.rs:1033                                                                                                             
#13 0x00005555582adfb9 in tx_verifier::transaction_verifier::TransactionVerifier<alloc::sync::Arc<common::chain::config::ChainConfig>, mempool::pool::tx_verifier::chainstate_handle::ChainstateHandle, mempool::pool::tx_verifier::chainstate_handle::ChainstateHandle, mempool::pool::tx_verifier::chainstate_handle::ChainstateHandle>::new_generic<alloc::sync::Arc<common::chain::config::ChainConfig>, mempool::pool::tx_verifier::chainstate_handle::ChainstateHandle, mempool::pool::tx_verifier::chainstate_handle::ChainstateHandle, mempool::pool::tx_verifier::chainstate_handle::ChainstateHandle> (storage=..., chain_config=Arc(strong=4, weak=0) = {...}, utxos=..., accounting=..., verifier_config=...) at chainstate/tx-verifier/src/transaction_verifier/mod.rs:164                    
#14 0x000055555826105a in mempool::pool::tx_verifier::create (chain_config=Arc(strong=4, weak=0) = {...}, chainstate=...) at mempool/src/pool/tx_verifier/mod.rs:43
#15 0x000055555818c2eb in mempool::pool::Mempool<mempool::pool::memory_usage_estimator::StoreMemoryUsageEstimator>::reset<mempool::pool::memory_usage_estimator::StoreMemoryUsageEstimator> (
    self=0x7fffa8001f88) at mempool/src/pool/mod.rs:145
#16 0x000055555819600a in mempool::pool::reorg::refresh_mempool<mempool::pool::memory_usage_estimator::StoreMemoryUsageEstimator, core::iter::adapters::filter::Filter<core::iter::adapters::flatten::FlatMap<core::iter::adapters::rev::Rev<alloc::vec::into_iter::IntoIter<common::chain::block::Block, alloc::alloc::Global>>, alloc::vec::Vec<common::chain::transaction::signed_transaction::SignedTransaction, alloc::alloc::Global>, mempool::pool::reorg::{impl#0}::into_disconnected_transactions::{closure_env#1}>, mempool::pool::reorg::{impl#0}::into_disconnected_transactions::{closure_env#2}>> (mempool=0x7fffa8001f88, txs_to_insert=...) at mempool/src/pool/reorg.rs:146
#17 0x0000555558195f94 in mempool::pool::reorg::handle_new_tip<mempool::pool::memory_usage_estimator::StoreMemoryUsageEstimator> (mempool=0x7fffa8001f88, new_tip=...)
    at mempool/src/pool/reorg.rs:137
#18 0x0000555558193979 in mempool::pool::Mempool<mempool::pool::memory_usage_estimator::StoreMemoryUsageEstimator>::new_tip_set<mempool::pool::memory_usage_estimator::StoreMemoryUsageEstimator>
    (self=0x7fffa8001f88, block_id=..., block_height=...) at mempool/src/pool/mod.rs:957
#19 0x0000555558193812 in mempool::pool::Mempool<mempool::pool::memory_usage_estimator::StoreMemoryUsageEstimator>::process_chainstate_event<mempool::pool::memory_usage_estimator::StoreMemoryUsageEstimator> (self=0x7fffa8001f88, evt=...) at mempool/src/pool/mod.rs:950
#20 0x00005555582ed7a0 in mempool::interface::mempool_interface_impl::{impl#2}::run::{async_block#0} () at mempool/src/interface/mempool_interface_impl.rs:100
#21 0x000055555842fe89 in core::future::future::{impl#1}::poll<alloc::boxed::Box<(dyn core::future::future::Future<Output=()> + core::marker::Send), alloc::alloc::Global>> (self=..., 
    cx=0x7ffff7287a90) at /rustc/78a6ac0a86c3feda182dea496c99fc7f866a22e9/library/core/src/future/future.rs:125
#22 0x00005555564c6b76 in subsystem::manager::{impl#2}::add_raw_subsystem_with_config::{async_block#0}<dyn mempool::interface::mempool_interface::MempoolInterface, core::pin::Pin<alloc::boxed::Box<(dyn core::future::future::Future<Output=()> + core::marker::Send), alloc::alloc::Global>>, node_lib::runner::initialize::{async_fn#0}::{closure_env#0}> () at subsystem/src/manager.rs:186
#23 0x000055555842fe89 in core::future::future::{impl#1}::poll<alloc::boxed::Box<(dyn core::future::future::Future<Output=()> + core::marker::Send), alloc::alloc::Global>> (self=..., 
    cx=0x7ffff7287a90) at /rustc/78a6ac0a86c3feda182dea496c99fc7f866a22e9/library/core/src/future/future.rs:125
#24 0x000055555843ca01 in tokio::runtime::task::core::{impl#6}::poll::{closure#0}<core::pin::Pin<alloc::boxed::Box<(dyn core::future::future::Future<Output=()> + core::marker::Send), alloc::alloc::Global>>, alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>> (ptr=0x7fffe4006bd0)
    at /home/admin0/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.1/src/runtime/task/core.rs:223

[pavel-kokolemin]

I also got a deadlock on shutdown.
All subsystems were stopped:

[2023-08-08T10:17:02Z INFO  mempool::pool] mempool: Processing chainstate event NewTip(Id<Block>{0xe6f6420b6a8771a9abdc27a09e26192891d5363c7fe7a945f969f5ad5ee410c2}, BlockHeight(9042))           
[2023-08-08T10:17:02Z INFO  mempool::pool] new tip: block Id<Block>{0xe6f6420b6a8771a9abdc27a09e26192891d5363c7fe7a945f969f5ad5ee410c2} height BlockHeight(9042)                                   
^C[2023-08-08T10:18:40Z INFO  subsystem::manager] Ctrl-C signal received                                                                                                                           
[2023-08-08T10:18:40Z INFO  subsystem::manager] Subsystem mintlayer/ctrl-c terminated                                                                                                              
[2023-08-08T10:18:40Z INFO  subsystem::manager] Manager mintlayer shutting down                                                                                                                    
[2023-08-08T10:18:40Z INFO  subsystem::manager] Subsystem mintlayer/rpc terminated                                                                                                                 
[2023-08-08T10:18:40Z INFO  subsystem::manager] Subsystem mintlayer/rpc_test_functions terminated                                                                                                  
[2023-08-08T10:18:40Z INFO  subsystem::manager] Subsystem mintlayer/blockprod terminated                                                                                                           
[2023-08-08T10:18:40Z INFO  blockprod::detail::job_manager::jobs_container] Stopping block production job manager                                                                                  
[2023-08-08T10:18:40Z INFO  p2p::net::default_backend] Backend is shut down                                                                                                                        
[2023-08-08T10:18:40Z INFO  p2p] Sync manager is shut down                                                                                                                                         
[2023-08-08T10:18:40Z INFO  p2p] Peer manager is shut down                                                                                                                                         
[2023-08-08T10:18:40Z INFO  subsystem::manager] Subsystem mintlayer/p2p terminated                                                                                                                 
[2023-08-08T10:18:40Z INFO  subsystem::manager] Subsystem mintlayer/mempool terminated                                                                                                             
[2023-08-08T10:18:40Z INFO  subsystem::manager] Subsystem mintlayer/chainstate terminated                                                                                                          
[2023-08-08T10:18:40Z WARN  subsystem::manager] Manager mintlayer: ctrl-c subsystem is already down                                                                                                
[2023-08-08T10:18:40Z INFO  subsystem::manager] Manager mintlayer terminated                                                                                                                       
*** Deadlock happened here. ***

I think Tokio was waiting for blocking tasks to finish while some of the background mempool tasks were trying to access chainstate. I think it's the same problem as described above.
I'm attaching the call stacks.
gdb.txt