svl simulator shut down every 20 seconds #1396

Joonyi0130 · 2021-04-27T08:28:46Z

when i start Simulations with 'Visual scenario editor' Template, it just play only 20 seconds.
everytime when i change scenarios, which is make with visual editor, it happened.
if i choose Random Traffic or Python API, it goes well after 20 sec
any idea what is wrong with my Simulation setting? Do i miss something?

EricBoiseLGSVL · 2021-04-27T17:54:24Z

Do you have a player.log we can look at? What OS, AD stack, GPU are you running? @KarByc ?

KarByc · 2021-04-27T17:59:52Z

@Joonyi0130
Is there any NPC with NPCWaypointBehaviour added to the scenario?

Joonyi0130 · 2021-04-28T00:08:49Z

@EricBoiseLGSVL i'm using ubuntu 20.04.1 and GTX1660 supper.
where can i get player.log? can't get with grep HAHA nothing searched

Joonyi0130 · 2021-04-28T00:10:46Z

@KarByc yes i made both NPCAgent and PedestrianAgent.
in Visual Editor, it plays longer than 20 seconds. it only happens in Simulations

Joonyi0130 · 2021-04-28T00:14:42Z

when reached 20 sec, there's an message on info tab in simulation and it shows like 0.2sec lol
so i recorded it, hope it helps to figure out what happens

KarByc · 2021-04-28T20:41:55Z

@Joonyi0130
Thanks for the feedback. Looks like the Player.log may help us with finding out the issue. Can you attach a Player.log file after a VSE scenario run?

GabrielZZZ · 2021-05-03T08:06:26Z

I encountered the same issue. @KarByc Here is my Player.log:

`[CONN] HTTP 200 OK
WARNING: Shader Unsupported: 'HDRP/Lit' - All passes removed
WARNING: Shader Did you use #pragma only_renderers and omit this platform?
WARNING: Shader Unsupported: 'Shader Graphs/VehicleLightsShader' - All passes removed
WARNING: Shader Did you use #pragma only_renderers and omit this platform?
found manifest at 1e531ff0-8ce5-4d26-9dda-bbd7a4e2c930/manifest.json
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /media/unnc/DATA1/svlsimulator-linux64-2021.1/simulator_Data/Mono/data-0x5185620.so
found manifest at 328b2afb-78c4-4c86-8a18-62015194a3fe/manifest.json
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /media/unnc/DATA1/svlsimulator-linux64-2021.1/simulator_Data/Mono/data-0x5bb61d0.so
found manifest at 7affd0d4-e9a5-4100-8a0f-e914c1176551/manifest.json
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /media/unnc/DATA1/svlsimulator-linux64-2021.1/simulator_Data/Mono/data-0x5bc3480.so
found manifest at ec968dc8-2e34-4307-8d3a-9bbffdf9e6cc/manifest.json
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /media/unnc/DATA1/svlsimulator-linux64-2021.1/simulator_Data/Mono/data-0x5bc5a90.so
found manifest at d7cc5876-aa06-45b6-9873-513959d3375f/manifest.json
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /media/unnc/DATA1/svlsimulator-linux64-2021.1/simulator_Data/Mono/data-0x5bc93e0.so
found manifest at a1c24125-22ab-4b5b-aeeb-4f6550d7b40b/manifest.json
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /media/unnc/DATA1/svlsimulator-linux64-2021.1/simulator_Data/Mono/data-0x5bce500.so
found manifest at 3a08ef80-03d6-4082-9dde-a001f8ba601b/manifest.json
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /media/unnc/DATA1/svlsimulator-linux64-2021.1/simulator_Data/Mono/data-0x5c1cab0.so
found manifest at f30ddcc6-7853-4282-8b0a-305d18827f03/manifest.json
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /media/unnc/DATA1/svlsimulator-linux64-2021.1/simulator_Data/Mono/data-0x5c2aea0.so
found manifest at e946ff05-6c77-4800-9e7e-3d246d48c74c/manifest.json
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /media/unnc/DATA1/svlsimulator-linux64-2021.1/simulator_Data/Mono/data-0x5c39a60.so
found manifest at 5bbf03d5-dd78-4505-85b0-7db70db41870/manifest.json
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /media/unnc/DATA1/svlsimulator-linux64-2021.1/simulator_Data/Mono/data-0x5c420c0.so
found manifest at d50ac8b1-0166-44a6-8499-5d8ecf7c8527/manifest.json
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /media/unnc/DATA1/svlsimulator-linux64-2021.1/simulator_Data/Mono/data-0x5c4aa80.so
[PROC][3596605][ERR] 2021-05-03 07:54:38,179 [INFO][dreamview] Starting Localization module...
[PROC][3596605][ERR] 2021-05-03 07:54:38,180 [INFO][dreamview] Starting Perception module...
[PROC][3596605][ERR] 2021-05-03 07:54:38,180 [INFO][dreamview] Starting Transform module...
[PROC][3596605][ERR] 2021-05-03 07:54:38,180 [INFO][dreamview] Starting Routing module...
[PROC][3596605][ERR] 2021-05-03 07:54:38,180 [INFO][dreamview] Starting Prediction module...
[PROC][3596605][ERR] 2021-05-03 07:54:38,181 [INFO][dreamview] Starting Planning module...
[PROC][3596605][ERR] 2021-05-03 07:54:38,181 [INFO][dreamview] Starting Traffic Light module...
[PROC][3596605][ERR] 2021-05-03 07:54:38,181 [INFO][dreamview] Starting Control module...
[PROC][3596605][ERR] 2021-05-03 07:54:38,182 [INFO][dreamview] Starting Recorder module...
[ANMGR] Initializing with TestReportId:45e0b255-3a6d-493a-b96c-7a7c3ee7bd17
[PROC][3596605][ERR] 2021-05-03 07:54:40,437 [INFO][dreamview] Control message received
[PROC][3596605][ERR] 2021-05-03 07:54:42,791 [INFO][run_vse] Starting scenario...
[PROC][3596605][ERR] 2021-05-03 07:54:55,148 [INFO][run_vse] All agents traversed their waypoints.
[PROC][3596605][ERR] 2021-05-03 07:54:55,149 [INFO][run_vse] Stopping simulation
[PROC][3596605][ERR] 2021-05-03 07:54:55,149 [INFO][run_vse] Scenario simulation ended.
5/3/2021 3:54:55 PM|Fatal|<>c__DisplayClass174_0.b__2:0|WebSocketSharp.WebSocketException: The header part of a frame could not be read.
at WebSocketSharp.WebSocketFrame.processHeader (System.Byte[] header) [0x0001f] in <84d0020558774b6c88b3203b756a09ec>:0
at WebSocketSharp.WebSocketFrame+<>c__DisplayClass73_0.b__0 (System.Byte[] bytes) [0x00000] in <84d0020558774b6c88b3203b756a09ec>:0
at WebSocketSharp.Ext+<>c__DisplayClass59_0.b__0 (System.IAsyncResult ar) [0x0008e] in <84d0020558774b6c88b3203b756a09ec>:0
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

[PROC][3596605][ERR]
[PROC][3596605][OUT]
[PROC][main] Process #3596605 exited with result 0
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

[LOADER] TestCase process exits: TestCaseFinishedArgs(ExitCode=0, output: 534 bytes error: 1300 bytes)
[LOADER] Stopping simulation on TestCase process exit
[LOADER] Cleanup volumes on TestCase process exit
[TC] Removing volumes for simulationId:ca9e9997-9f44-48b7-9956-b5f3dbbaa732
[LOADER] Update simulation status Running -> Stopping
[CONN] Updated simulation ca9e9997-9f44-48b7-9956-b5f3dbbaa732 with status:Stopping msg:
[CONN] POST /api/v1/simulations/ca9e9997-9f44-48b7-9956-b5f3dbbaa732/status
Deinitialized the Simulation Network data.
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

[LOADER] StopAsync: Terminating process
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

[PROC][3596605] Process is already exited
Unloading 2 Unused Serialized files (Serialized files now loaded: 17)
[ANMGR] Sending test report data id:45e0b255-3a6d-493a-b96c-7a7c3ee7bd17
[CONN] POST /api/v1/test-results/45e0b255-3a6d-493a-b96c-7a7c3ee7bd17
Fallback handler could not load library /media/unnc/DATA1/svlsimulator-linux64-2021.1/simulator_Data/Mono/libKernel32
Fallback handler could not load library /media/unnc/DATA1/svlsimulator-linux64-2021.1/simulator_Data/Mono/libKernel32.so
Fallback handler could not load library /media/unnc/DATA1/svlsimulator-linux64-2021.1/simulator_Data/Mono/Kernel32
Fallback handler could not load library /media/unnc/DATA1/svlsimulator-linux64-2021.1/simulator_Data/Mono/libKernel32
Fallback handler could not load library /media/unnc/DATA1/svlsimulator-linux64-2021.1/simulator_Data/Mono/libKernel32.so
Fallback handler could not load library /media/unnc/DATA1/svlsimulator-linux64-2021.1/simulator_Data/Mono/libKernel32
UnloadTime: 150.756305 ms
[LOADER] Update simulation status Stopping -> Idle
[CONN] Updated simulation ca9e9997-9f44-48b7-9956-b5f3dbbaa732 with status:Idle msg:
[CONN] POST /api/v1/simulations/ca9e9997-9f44-48b7-9956-b5f3dbbaa732/status
[CONN] HTTP 200 OK
Receiving response of OK System.Net.Http.StreamContent
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Unloading 837 unused Assets to reduce memory usage. Loaded Objects now: 6698.
Total: 45.491265 ms (FindLiveObjects: 0.895352 ms CreateObjectMapping: 0.675044 ms MarkObjects: 33.920383 ms DeleteObjects: 9.999193 ms)

Unloading 1 Unused Serialized files (Serialized files now loaded: 0)

Unloading 0 unused Assets to reduce memory usage. Loaded Objects now: 6687.
Total: 37.149180 ms (FindLiveObjects: 0.870221 ms CreateObjectMapping: 0.701224 ms MarkObjects: 35.553388 ms DeleteObjects: 0.023519 ms)

Unloading 0 Unused Serialized files (Serialized files now loaded: 0)
[CONN] HTTP 200 OK
Receiving response of OK System.Net.Http.StreamContent
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

[CONN] HTTP 200 OK
Receiving response of OK System.Net.Http.StreamContent
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)
`

EricBoiseLGSVL · 2021-05-03T17:13:37Z

Are you running the VSE scenario without connecting to AD stack?

GabrielZZZ · 2021-05-04T02:21:45Z

I ran it with Apollo Dreamview. I am certain there is a connection that the vehicles in dreamview moves simultaneously with the one in the simulator, and all the functions seems normal except that the perception and traffic light module are turned on automatically even under the modular testing scenario, just as #1370.

EricBoiseLGSVL · 2021-05-04T21:02:58Z

VSE based simulations only support Apollo 5.0 at the moment as it is based on dreamview-python-api.
See here

MisterioRemo · 2021-05-13T06:00:47Z

@EricBoiseLGSVL
Hello, I have a similar problem. I want to run VSE + AutowareAuto.
So, I start lgsvl_bridge and binary version of simulator on the same machine. The scenario starts playing, lgsvl_bridge subscribes to topics, but after ~20 seconds the scenario stops unexpectedly.
Part of my log file:


[CONN] HTTP 200 OK
InitializeOrResetSwapChain 924x1136 hdr=0 samples=1
WARNING: Shader Unsupported: 'HDRP/Lit' - All passes removed
WARNING: Shader Did you use #pragma only_renderers and omit this platform?
WARNING: Shader Unsupported: 'Shader Graphs/VehicleLightsShader' - All passes removed
WARNING: Shader Did you use #pragma only_renderers and omit this platform?
found manifest at 52f29c02-6966-4749-bab1-aa0456735740/manifest.json 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/data-0x5006ce0.so
found manifest at 328b2afb-78c4-4c86-8a18-62015194a3fe/manifest.json 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/data-0x50dc280.so
found manifest at 1e531ff0-8ce5-4d26-9dda-bbd7a4e2c930/manifest.json 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/data-0x5009f30.so
found manifest at aa5e3262-fde7-4445-b86e-0922ed5fedba/manifest.json 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/data-0x5012270.so
found manifest at 68ddcc86-1b72-447e-8b48-736917e07281/manifest.json 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/data-0x5022ce0.so
found manifest at e8bb3908-0059-4829-8cc3-4ba07dd915bf/manifest.json 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/data-0x503f930.so
found manifest at ec968dc8-2e34-4307-8d3a-9bbffdf9e6cc/manifest.json 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/data-0x50c5b80.so
found manifest at a1022620-f2a3-4cef-9146-d10eaa445386/manifest.json 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/data-0x50bf040.so
found manifest at 3a08ef80-03d6-4082-9dde-a001f8ba601b/manifest.json 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/data-0x50ca510.so
found manifest at 3a9b1c3a-3e5a-4b15-9b6c-3acbc3c937a0/manifest.json 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/data-0x50f5f10.so
found manifest at d50ac8b1-0166-44a6-8499-5d8ecf7c8527/manifest.json 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/data-0x50f88c0.so
found manifest at ad643739-d4e3-415a-bdf3-0ebcf4bc2f64/manifest.json 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/data-0x5109fa0.so
found manifest at f30ddcc6-7853-4282-8b0a-305d18827f03/manifest.json 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/data-0x51126a0.so
found manifest at 5bba1420-a7fd-40cc-887d-fe88cc6c49b7/manifest.json 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/data-0x5118ed0.so
found manifest at d7cc5876-aa06-45b6-9873-513959d3375f/manifest.json 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/data-0x5129760.so
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/libadvapi32.dll
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/libadvapi32.dll.so
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/advapi32.dll
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/libadvapi32
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/libadvapi32.so
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/advapi32
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/libadvapi32.dll
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/libadvapi32.dll.so
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/libadvapi32.dll
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/libadvapi32.dll
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/libadvapi32.dll.so
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/advapi32.dll
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/libadvapi32
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/libadvapi32.so
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/advapi32
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/libadvapi32.dll
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/libadvapi32.dll.so
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/libadvapi32.dll
[PROC][39883][ERR] 2021-05-12 12:33:54,907 [ERROR][run_vse] Original exception: [Errno 111] Connection refused
[PROC][39883][ERR] 2021-05-12 12:33:54,907 [ERROR][run_vse] Something went wrong with bridge / dreamview connection.
InitializeOrResetSwapChain 1024x768 hdr=0 samples=1
Waypoint timestamps absent or invalid, caluclating timestamps based on speed. 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

[PROC][39883][ERR] 2021-05-12 12:33:55,636 [INFO][run_vse] Starting scenario...
[ANMGR] Initializing with TestReportId:4c2f7ab5-0575-4731-8fc4-ed35117e16eb
[PROC][39883][ERR] 2021-05-12 12:34:27,316 [INFO][run_vse] Scenario simulation ended.
5/12/2021 3:34:27 PM|Fatal|<>c__DisplayClass174_0.<startReceiving>b__2:0|WebSocketSharp.WebSocketException: The header part of a frame could not be read.
                             at WebSocketSharp.WebSocketFrame.processHeader (System.Byte[] header) [0x0001f] in <84d0020558774b6c88b3203b756a09ec>:0 
                             at WebSocketSharp.WebSocketFrame+<>c__DisplayClass73_0.<readHeaderAsync>b__0 (System.Byte[] bytes) [0x00000] in <84d0020558774b6c88b3203b756a09ec>:0 
                             at WebSocketSharp.Ext+<>c__DisplayClass59_0.<ReadBytesAsync>b__0 (System.IAsyncResult ar) [0x0008e] in <84d0020558774b6c88b3203b756a09ec>:0  
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

[PROC][39883][OUT] 
[PROC][39883][ERR] 
[PROC][main] Process #39883 exited with result 0 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

[LOADER] TestCase process exits: TestCaseFinishedArgs(ExitCode=0, output: 507 bytes error: 640 bytes)
[LOADER] Stopping simulation on TestCase process exit
[LOADER] Cleanup volumes on TestCase process exit
[TC] Removing volumes for simulationId:032fcedc-5e3b-4523-a3e1-7e9efb7074e1
Deinitialized the Simulation Network data. 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

[LOADER] StopAsync: Terminating process 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

[PROC][39883] Process is already exited
Unloading 3 Unused Serialized files (Serialized files now loaded: 16)
[ANMGR] Sending test report data id:4c2f7ab5-0575-4731-8fc4-ed35117e16eb
[CONN] POST /api/v1/test-results/4c2f7ab5-0575-4731-8fc4-ed35117e16eb
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/libKernel32
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/libKernel32.so
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/Kernel32
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/libKernel32
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/libKernel32.so
Fallback handler could not load library /home/lui/Downloads/sim_linux_2021.1/simulator_Data/Mono/libKernel32
UnloadTime: 57.145063 ms
[LOADER] Update simulation status Starting -> Idle
[CONN] Updated simulation 032fcedc-5e3b-4523-a3e1-7e9efb7074e1 with status:Idle msg:
[CONN] POST /api/v1/simulations/032fcedc-5e3b-4523-a3e1-7e9efb7074e1/status

Unloading 416 unused Assets to reduce memory usage. Loaded Objects now: 6551.
Total: 14.474155 ms (FindLiveObjects: 0.375678 ms CreateObjectMapping: 0.184801 ms MarkObjects: 11.774025 ms  DeleteObjects: 2.139070 ms)

Unloading 1 Unused Serialized files (Serialized files now loaded: 0)
[CONN] HTTP 500 InternalServerError

Unloading 0 unused Assets to reduce memory usage. Loaded Objects now: 6512.
Total: 13.924347 ms (FindLiveObjects: 0.432112 ms CreateObjectMapping: 0.283634 ms MarkObjects: 13.182957 ms  DeleteObjects: 0.025082 ms)

Unloading 0 Unused Serialized files (Serialized files now loaded: 0)
[CONN] HTTP 200 OK
Receiving response of OK System.Net.Http.StreamContent 
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)


Unloading 0 unused Assets to reduce memory usage. Loaded Objects now: 6512.
Total: 13.369595 ms (FindLiveObjects: 0.550483 ms CreateObjectMapping: 0.426983 ms MarkObjects: 12.368738 ms  DeleteObjects: 0.022741 ms)

Unloading 0 Unused Serialized files (Serialized files now loaded: 0)

Unloading 0 unused Assets to reduce memory usage. Loaded Objects now: 6512.
Total: 10.193459 ms (FindLiveObjects: 0.314205 ms CreateObjectMapping: 0.154849 ms MarkObjects: 9.713740 ms  DeleteObjects: 0.010270 ms)

Unloading 0 Unused Serialized files (Serialized files now loaded: 0)

Unloading 0 unused Assets to reduce memory usage. Loaded Objects now: 6512.
Total: 13.472160 ms (FindLiveObjects: 0.391940 ms CreateObjectMapping: 0.224416 ms MarkObjects: 12.822638 ms  DeleteObjects: 0.032650 ms)

Unloading 0 Unused Serialized files (Serialized files now loaded: 0)

Unloading 0 unused Assets to reduce memory usage. Loaded Objects now: 6512.
Total: 10.644788 ms (FindLiveObjects: 0.295699 ms CreateObjectMapping: 0.155582 ms MarkObjects: 10.178879 ms  DeleteObjects: 0.014094 ms)

I try the same scenario but with Apollo 5.0 and it stops with the same error described above.

EricBoiseLGSVL · 2021-05-13T17:10:22Z

We are looking into this issue and hope to have a solution soon. Adding @KarByc

KarByc · 2021-05-13T18:28:35Z

@MisterioRemo

Thanks for using our VSERunner. About your issue there is some error with the bridge connection:

[PROC][39883][ERR] 2021-05-12 12:33:54,907 [ERROR][run_vse] Original exception: [Errno 111] Connection refused
[PROC][39883][ERR] 2021-05-12 12:33:54,907 [ERROR][run_vse] Something went wrong with bridge / dreamview connection.

Currently the VSERunner works only with the Apollo, we are working to support the AutowareAuto.

MisterioRemo · 2021-05-14T04:59:39Z

@KarByc

Thanks for using our VSERunner. About your issue there is some error with the bridge connection:

Yes, I noticed this error too. But judging by the lgsvl_bridge's log, it works fine. It subscribes to topics when the scenario starts, and unsubscribes when the scenario stops working.
But there is one oddity. I can see the topic's message through terminal ros2 topic echo /my_topic. But rviz2 and rqt almost never sees the simulator's topics when it plays scenario. In the lgsvl_bridge project I changed reliability policy from reliable to best effort and this mini problem was resolved.

Currently the VSERunner works only with the Apollo, we are working to support the AutowareAuto.

Thank you, I'm looking forward to it.

yejw5 · 2021-05-15T12:19:37Z

I have encountered the same error with Apollo 5.0. Look forward to the solution.

EricBoiseLGSVL · 2021-05-17T17:16:37Z

Do you have the connection refused error? What error do you have @yejw5 ?

yejw5 · 2021-05-18T02:27:21Z

Do you have the connection refused error? What error do you have @yejw5 ?

Sorry, I found that my logs were different. I have run tests in Ubuntu18.04 + Apollo5.0 + lgsvl2021.1, and got failed with lgsvl Player.log as follows:

[PROC][3587][ERR] 2021-05-18 02:18:08,965 [INFO][dreamview] Starting Transform module...
[PROC][3587][ERR] 2021-05-18 02:18:08,965 [INFO][dreamview] Starting Routing module...
[PROC][3587][ERR] 2021-05-18 02:18:08,965 [INFO][dreamview] Starting Prediction module...
[PROC][3587][ERR] 2021-05-18 02:18:08,965 [INFO][dreamview] Starting Planning module...
[PROC][3587][ERR] 2021-05-18 02:18:08,965 [INFO][dreamview] Starting Traffic Light module...
[PROC][3587][ERR] 2021-05-18 02:18:08,965 [INFO][dreamview] Starting Control module...
[PROC][3587][ERR] 2021-05-18 02:18:08,965 [INFO][dreamview] Starting Recorder module...
[ANMGR] Initializing with TestReportId:a6263248-16dc-4f65-bd7c-e3a951e34f18
[PROC][3587][ERR] 2021-05-18 02:18:14,153 [INFO][dreamview] Control message received
[PROC][3587][ERR] 2021-05-18 02:18:15,300 [INFO][run_vse] Starting scenario...
[PROC][3587][ERR] 2021-05-18 02:18:35,314 [INFO][run_vse] Scenario simulation ended.
2021/5/18 10:18:35|Fatal|<>c__DisplayClass174_0.<startReceiving>b__2:0|WebSocketSharp.WebSocketException: The header part of a frame could not be read.
                           at WebSocketSharp.WebSocketFrame.processHeader (System.Byte[] header) [0x0001f] in <84d0020558774b6c88b3203b756a09ec>:0
                           at WebSocketSharp.WebSocketFrame+<>c__DisplayClass73_0.<readHeaderAsync>b__0 (System.Byte[] bytes) [0x00000] in <84d0020558774b6c88b3203b756a09ec>:0
                           at WebSocketSharp.Ext+<>c__DisplayClass59_0.<ReadBytesAsync>b__0 (System.IAsyncResult ar) [0x0008e] in <84d0020558774b6c88b3203b756a09ec>:0
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

[PROC][3587][OUT]
[PROC][3587][ERR]
[PROC][main] Process #3587 exited with result 0
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

[LOADER] TestCase process exits: TestCaseFinishedArgs(ExitCode=0, output: 534 bytes error: 1166 bytes)
[LOADER] Stopping simulation on TestCase process exit
[LOADER] Cleanup volumes on TestCase process exit
[TC] Removing volumes for simulationId:75063b5b-5eaa-4da2-a75e-6466dbf5eb4e
[LOADER] Update simulation status Running -> Stopping
[CONN] Updated simulation 75063b5b-5eaa-4da2-a75e-6466dbf5eb4e with status:Stopping msg:
[CONN] POST /api/v1/simulations/75063b5b-5eaa-4da2-a75e-6466dbf5eb4e/status
Deinitialized the Simulation Network data.
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

[LOADER] StopAsync: Terminating process
(Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 35)

[PROC][3587] Process is already exited
Unloading 2 Unused Serialized files (Serialized files now loaded: 0)
[ANMGR] Sending test report data id:a6263248-16dc-4f65-bd7c-e3a951e34f18
[CONN] POST /api/v1/test-results/a6263248-16dc-4f65-bd7c-e3a951e34f18
UnloadTime: 7.215336 ms
[LOADER] Update simulation status Stopping -> Idle
[CONN] Updated simulation 75063b5b-5eaa-4da2-a75e-6466dbf5eb4e with status:Idle msg:
[CONN] POST /api/v1/simulations/75063b5b-5eaa-4da2-a75e-6466dbf5eb4e/status

Besides, the apollo bridge.sh prints:

I0518 10:17:59.558449  3534 global_data.cc:150] [cyber_bridge] host ip: X.X.X.X
E0518 10:18:36.339259  3534 client.cc:98] [cyber_bridge] Client write failed, disconnectingsystem:9

EricBoiseLGSVL · 2021-05-18T17:39:45Z

We have a solution for this issue and will be releasing a fix soon :)

yejw5 · 2021-05-19T00:54:37Z

We have a solution for this issue and will be releasing a fix soon :)

Thank you, I'm looking forward to it.

QZJGeorge · 2021-05-19T21:17:18Z

Same issue here. The simulation shuts down for no reason at about 20 seconds. The bridge did not report an error. Looking forward to a fix.

safdarzareef · 2021-11-30T16:08:44Z

Hi @EricBoiseLGSVL we are still running into a similar issue, even though Autoware AI connects (2D nav goal and moving the ego vehicle works) when we run a modified Python API test case, Red Light Runner for Apollo, and change the ego vehicle for Autoware AI to Jaguar2015XE, the exception is raised that the AD stack is not ready yet. Is there any fix for this?

EricBoiseLGSVL added help wanted Extra attention is needed question Further information is requested labels Apr 27, 2021

t0ny-peng mentioned this issue May 2, 2021

Simulator shuts down in headless mode + Python API after 30 seconds #1405

Open

EricBoiseLGSVL added the bug Something isn't working label May 13, 2021

EricBoiseLGSVL added next release Feature/bug fix will be included in next release and removed bug Something isn't working help wanted Extra attention is needed question Further information is requested labels May 18, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

svl simulator shut down every 20 seconds #1396

svl simulator shut down every 20 seconds #1396

Joonyi0130 commented Apr 27, 2021

EricBoiseLGSVL commented Apr 27, 2021 •

edited

Loading

KarByc commented Apr 27, 2021 •

edited

Loading

Joonyi0130 commented Apr 28, 2021

Joonyi0130 commented Apr 28, 2021

Joonyi0130 commented Apr 28, 2021

KarByc commented Apr 28, 2021

GabrielZZZ commented May 3, 2021

EricBoiseLGSVL commented May 3, 2021

GabrielZZZ commented May 4, 2021

EricBoiseLGSVL commented May 4, 2021

MisterioRemo commented May 13, 2021 •

edited

Loading

EricBoiseLGSVL commented May 13, 2021

KarByc commented May 13, 2021

MisterioRemo commented May 14, 2021 •

edited

Loading

yejw5 commented May 15, 2021

EricBoiseLGSVL commented May 17, 2021

yejw5 commented May 18, 2021

EricBoiseLGSVL commented May 18, 2021

yejw5 commented May 19, 2021

QZJGeorge commented May 19, 2021

safdarzareef commented Nov 30, 2021

svl simulator shut down every 20 seconds #1396

svl simulator shut down every 20 seconds #1396

Comments

Joonyi0130 commented Apr 27, 2021

EricBoiseLGSVL commented Apr 27, 2021 • edited Loading

KarByc commented Apr 27, 2021 • edited Loading

Joonyi0130 commented Apr 28, 2021

Joonyi0130 commented Apr 28, 2021

Joonyi0130 commented Apr 28, 2021

KarByc commented Apr 28, 2021

GabrielZZZ commented May 3, 2021

EricBoiseLGSVL commented May 3, 2021

GabrielZZZ commented May 4, 2021

EricBoiseLGSVL commented May 4, 2021

MisterioRemo commented May 13, 2021 • edited Loading

EricBoiseLGSVL commented May 13, 2021

KarByc commented May 13, 2021

MisterioRemo commented May 14, 2021 • edited Loading

yejw5 commented May 15, 2021

EricBoiseLGSVL commented May 17, 2021

yejw5 commented May 18, 2021

EricBoiseLGSVL commented May 18, 2021

yejw5 commented May 19, 2021

QZJGeorge commented May 19, 2021

safdarzareef commented Nov 30, 2021

EricBoiseLGSVL commented Apr 27, 2021 •

edited

Loading

KarByc commented Apr 27, 2021 •

edited

Loading

MisterioRemo commented May 13, 2021 •

edited

Loading

MisterioRemo commented May 14, 2021 •

edited

Loading