bug: Intermittent E2E Test failure due to WS client not reconnecting #1712
Description
What happened?
Test failed here
2025-12-11T17:48:53.6448210Z === RUN TestEthereumMultipartyE2ESuite/TestE2EFungibleTokensAsync
2025-12-11T17:48:53.6451798Z common.go:129: Blockchain provider: ethereum
2025-12-11T17:48:53.6452747Z common.go:153: Client 1: http://127.0.0.1:5000/api/v1
2025-12-11T17:48:53.6453536Z common.go:154: Client 2: http://127.0.0.1:5001/api/v1
2025-12-11T17:48:53.6454586Z restclient.go:109: 2025-12-11T17:48:53.644894204Z: ==> GET /namespaces/default/status map[]: null
2025-12-11T17:48:53.6462764Z restclient.go:116: 2025-12-11T17:48:53.646147255Z: <== 200
2025-12-11T17:48:53.6463838Z restclient.go:109: 2025-12-11T17:48:53.646171229Z: ==> GET /namespaces/default/status map[]: null
2025-12-11T17:48:53.6473316Z restclient.go:116: 2025-12-11T17:48:53.647199531Z: <== 200
2025-12-11T17:48:53.6474860Z restclient.go:109: 2025-12-11T17:48:53.647228295Z: ==> GET /namespaces/default/network/organizations map[sort:[created]]: null
2025-12-11T17:48:53.6481335Z restclient.go:116: 2025-12-11T17:48:53.648014132Z: <== 200
2025-12-11T17:48:53.6482633Z restclient.go:109: 2025-12-11T17:48:53.64804011Z: ==> GET /namespaces/default/network/organizations map[sort:[created]]: null
2025-12-11T17:48:53.6490437Z restclient.go:116: 2025-12-11T17:48:53.648857662Z: <== 200
2025-12-11T17:48:53.6492022Z restclient.go:109: 2025-12-11T17:48:53.648892197Z: ==> GET /namespaces/default/identities/86417da0-7daf-4c43-9e43-d74a39a7fbd6/verifiers map[type:[!=dx_peer_id]]: null
2025-12-11T17:48:53.6499534Z restclient.go:116: 2025-12-11T17:48:53.649794267Z: <== 200
2025-12-11T17:48:53.6501391Z restclient.go:109: 2025-12-11T17:48:53.649820576Z: ==> GET /namespaces/default/identities/9b5e7563-e04f-4c63-9ce8-76af60ad07f8/verifiers map[type:[!=dx_peer_id]]: null
2025-12-11T17:48:53.6507827Z restclient.go:116: 2025-12-11T17:48:53.650644263Z: <== 200
2025-12-11T17:48:53.6509296Z common.go:181: Org1: ID=86417da0-7daf-4c43-9e43-d74a39a7fbd6 DID=did:firefly:org/org_035c32 Key=0xd0650a969bb32331acce70c2e56b6895a3a84501
2025-12-11T17:48:53.6511466Z common.go:182: Org2: ID=9b5e7563-e04f-4c63-9ce8-76af60ad07f8 DID=did:firefly:org/org_fd56bd Key=0x0a37ab8758bd7d772e10335a61b0d81865b41489
2025-12-11T17:48:53.6526499Z tokens.go:56: Pool name: pool_e2e_9a3fb1dcd5
2025-12-11T17:48:53.6528557Z restclient.go:109: 2025-12-11T17:48:53.65210746Z: ==> POST /namespaces/default/tokens/pools map[confirm:[false] publish:[true]]: {"type":"fungible","name":"pool_e2e_9a3fb1dcd5","active":false,"tx":{"type":""},"published":false}
2025-12-11T17:48:53.6874710Z restclient.go:116: 2025-12-11T17:48:53.687253404Z: <== 202
2025-12-11T17:48:56.5583214Z Websocket 127.0.0.1:5001 closing, error: websocket: close 1006 (abnormal closure): unexpected EOF
2025-12-11T17:48:57.6547930Z Websocket 127.0.0.1:5000 event: default/blockchain_event_received/9e51becf-21b9-4103-ae74-5b0fa54d2c63 -> b59ac1b2-a7fe-4392-91ae-58cddc0be67c (tx=%!s(*core.Transaction=<nil>))
2025-12-11T17:48:57.6550378Z Websocket 127.0.0.1:5000 event: default/blockchain_event_received/96fd745a-1efc-452b-98e9-4cf813baefe2 -> cd3c26c3-9447-47af-ab80-f1c870c6d47f (tx=%!s(*core.Transaction=<nil>))
2025-12-11T17:48:57.6552968Z Websocket 127.0.0.1:5000 event: default/token_pool_confirmed/38829071-527e-48b8-bdd0-f6c2c003fb33 -> f1a9919d-a59a-4598-9bf0-4ba6c29a06b0 (tx=%!s(*core.Transaction=<nil>))
2025-12-11T17:48:57.6554634Z Websocket 127.0.0.1:5000 event: default/message_confirmed/39c2a1c6-8d48-41c2-93cc-60b404a32ca6 -> 4791e850-d335-40ca-bf03-c87289f9949c (tx=%!s(*core.Transaction=<nil>))
2025-12-11T17:48:57.6556587Z e2e.go:142: Ignored event '9e51becf-21b9-4103-ae74-5b0fa54d2c63' [blockchain_event_received]
2025-12-11T17:48:57.6557358Z e2e.go:142: Ignored event '96fd745a-1efc-452b-98e9-4cf813baefe2' [blockchain_event_received]
2025-12-11T17:48:57.6558417Z e2e.go:139: Detected 'token_pool_confirmed' event for ref 'f1a9919d-a59a-4598-9bf0-4ba6c29a06b0'
2025-12-11T17:48:57.6560047Z restclient.go:109: 2025-12-11T17:48:57.654866769Z: ==> GET /namespaces/default/tokens/pools map[created:[>1765475333644847117]]: null
2025-12-11T17:48:57.6561994Z restclient.go:116: 2025-12-11T17:48:57.656064577Z: <== 200
2025-12-11T17:57:51.5118730Z panic: test timed out after 10m0s
2025-12-11T17:57:51.5119587Z running tests:
2025-12-11T17:57:51.5120570Z TestEthereumMultipartyE2ESuite (10m0s)
2025-12-11T17:57:51.5122069Z TestEthereumMultipartyE2ESuite/TestE2EFungibleTokensAsync (8m58s)
What did you expect to happen?
Test to pass, this failed to do the client in the test not having proper reconnection, we should be using FireFly Common where possible
How can we reproduce it (as minimally and precisely as possible)?
Manually disconnect the client in the test and wait for the timeout
Anything else we need to know?
No response
OS version
Details
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here
# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here