KeyError: 'latitude'

[ugrkm]

When I use PythonAPI with Apollo Dreamview on the svl simulator, simulation crashes. The error message is:

Traceback (most recent call last):
  File "src/test_sim.py", line 75, in <module>
    s = Scenario(scenario_options, connection, dv)
  File "/home/user/Testing-of-autonomous-software/src/scenario.py", line 182, in __init__
    self.dv.setup_apollo(*self.destination, self.modules)
  File "/home/user2/Documents/lgsvl/api/PythonAPI-2021.2/lgsvl/dreamview/dreamview.py", line 323, in setup_apollo
    self.enable_apollo(dest_x, dest_z, modules, coord_type=coord_type)
  File "/home/user2/Documents/lgsvl/api/PythonAPI-2021.2/lgsvl/dreamview/dreamview.py", line 290, in enable_apollo
    self.set_destination(dest_x, dest_z, coord_type=coord_type)
  File "/home/user2/Documents/lgsvl/api/PythonAPI-2021.2/lgsvl/dreamview/dreamview.py", line 57, in set_destination
    current_gps = self.sim.map_to_gps(current_pos)
  File "/home/user2/Documents/lgsvl/api/PythonAPI-2021.2/lgsvl/utils.py", line 23, in new_f
    return f(*args, **kwargs)
  File "/home/user2/Documents/lgsvl/api/PythonAPI-2021.2/lgsvl/simulator.py", line 230, in map_to_gps
    return GpsData(j["latitude"], j["longitude"], j["northing"], j["easting"], j["altitude"], j["orientation"])
KeyError: 'latitude'

The thing is, this happens when I run the simulation with a custom sensor-equipped car. In contrast, when I run the simulation with one of the default sensor configurations it runs without a problem. Furthermore, when I run the simulation again the problem disappears. So, it throws an error in every two runs. Any ideas?

[EricBoiseLGSVL]

When you run the simulation again with the custom sensor it works?
Can you post the custom sensor? Is it a GPS sensor?
What does the player.log in the application say?
What OS and sim version (looks like 2021.2 right?)
The fact it is crashing is very bad. usually a GPU memory issue. Nulls just stop the sim if caught or fail to load all assets. Crash means something is very wrong.

[heeen]

I believe @gorkemkrdmn was speaking figuratively when saying it crashed. @gorkemkrdmn can you provide us with the simulation player log ($HOME/.config/unity3d/LGElectronics/LGSVL Simulator*/Player.log)

[ugrkm]

Yes, crash was a figure of speech. I will provide the log files. I think for some reason pythonapi can’t acces the gps data once in a two run as I mentioned.

[marifdemirtas]

I'm having a very similar problem. I'm using a script to run multiple simulations in a for loop. In each iteration, new lgsvl.Simulator and dreamview.Connection objects are generated. In the first iteration, everything runs fine, but in the second iteration, the Python script halts with KeyError, the simulator itself does not crash but it stops as the sim.run call in the python script does not run. The exact error I get is:

Traceback (most recent call last):
  File "/home/user/my_test_script.py", line 165, in __init__
    dv.setup_apollo(*self.destination, modules)
  File "/home/user/lgsvl/api/PythonAPI-2021.2/lgsvl/dreamview/dreamview.py", line 323, in setup_apollo
    self.enable_apollo(dest_x, dest_z, modules, coord_type=coord_type)
  File "/home/user/lgsvl/api/PythonAPI-2021.2/lgsvl/dreamview/dreamview.py", line 290, in enable_apollo
    self.set_destination(dest_x, dest_z, coord_type=coord_type)
  File "/home/user/lgsvl/api/PythonAPI-2021.2/lgsvl/dreamview/dreamview.py", line 57, in set_destination
    current_gps = self.sim.map_to_gps(current_pos)
  File "/home/user/lgsvl/api/PythonAPI-2021.2/lgsvl/utils.py", line 23, in new_f
    return f(*args, **kwargs)
  File "/home/reyhanslab/Documents/lgsvl/api/PythonAPI-2021.2/lgsvl/simulator.py", line 227, in map_to_gps
    return GpsData(j["latitude"], j["longitude"], j["northing"], j["easting"], j["altitude"], j["orientation"])
KeyError: 'latitude'

The script that causes this problem is almost same as the example provided in the docs.

The setup is:

OS: Ubuntu 18.04
SVL release: 2021.2

The Player.log file is at: Player.log

[EricBoiseLGSVL]

Are you completely toggling off Apollo's modules? I'll ping @revati-naik to see if they have seen this error.

[lemketron]

In the first iteration, everything runs fine, but in the second iteration, the Python script halts with KeyError, the simulator itself does not crash but it stops as the sim.run call in the python script does not run.

This sounds like it could be a known issue where Apollo doesn't like the car to "teleport" (back) to the original spawn location; the fix (as @EricBoiseLGSVL alluded to above) is to restart the Apollo modules (specifically the Routing module, IIRC, or maybe the Planning module).

You can reproduce this problem by manually setting a destination in Dreamview and letting Apollo drive there, then hit F12 to reset the active (ego) vehicle location. You'll see that routing and planning do not plan a new route until (one or both) are restarted in the Dreamview module control view.

[EricBoiseLGSVL]

Thanks @lemketron . We also have a few fixes coming in the next release and it may help with the KeyError. I would also try this when it's out next week.

[heeen]

I wonder if API responses get out of sync somehow and the response is actually an object missing the key

[marifdemirtas]

@lemketron @EricBoiseLGSVL toggling just Routing and Planning did not have any effect, but toggling all modules caused a similar error, but interestingly on the previous line of the dreamview.py file, still in set_destination function. Here's the trace:

Traceback (most recent call last):
  File "/home/user/my_test_script.py", line 166, in __init__
    self.dv.setup_apollo(*self.destination, modules)
  File "/home/user/lgsvl/api/PythonAPI-2021.2/lgsvl/dreamview/dreamview.py", line 323, in setup_apollo
    self.enable_apollo(dest_x, dest_z, modules, coord_type=coord_type)
  File "/home/user/lgsvl/api/PythonAPI-2021.2/lgsvl/dreamview/dreamview.py", line 290, in enable_apollo
    self.set_destination(dest_x, dest_z, coord_type=coord_type)
  File "/home/user/lgsvl/api/PythonAPI-2021.2/lgsvl/dreamview/dreamview.py", line 56, in set_destination
    current_pos = self.ego.state.transform
  File "/home/user/lgsvl/api/PythonAPI-2021.2/lgsvl/agent.py", line 114, in state
    return AgentState.from_json(j)
  File "/home/user/lgsvl/api/PythonAPI-2021.2/lgsvl/utils.py", line 61, in from_json
    Transform.from_json(j["transform"]),
KeyError: 'transform'

I think @heeen may be right about the out of sync condition. I suppose it also explains how adding the disable_apollo call causes the error to move up by one line. I modified the dreamview.py file in the PythonAPI to print the response and the API response during the first run (no issues):
{'latitude': 37.4144506701616, 'longitude': -122.017114419075, 'northing': 4141303.63845825, 'easting': 586977.466850281, 'altitude': 0, 'orientation': 0}
The response in second run, where the issue happens:
{'events': [{'agent': '353a3b41-830c-469b-b0b4-5aa3cc9b738a', 'type': 'custom', 'kind': 'checkControl', 'context': {}}]}

Any ideas on how to debug this synchronization error? I am already using the clock sensor for communicating with Apollo.

[ugrkm]

@EricBoiseLGSVL the custom sensor that causes the problem is
OffRoadSensor.cs.zip. Any updates ?

[EricBoiseLGSVL]

Are you running this on a Mac? It is concerning that MacOSX folders are here with duplicate cs files. Also, are you creating a prefab for the class?

[heeen]

So that sensor sends an event using apimanager addcustom which eventually ends up in sendresult. I am thinking this will mix with the regular api call results and confuse the unexpecting receiver in agent state get.

[EricBoiseLGSVL]

We'll look into this, thanks for posting

[ugrkm]

@EricBoiseLGSVL I am working on the Mac but I run the code on linux (cloud). Yes, I am creating a prefab for the class.

[EricBoiseLGSVL]

Have you tried this with 2021.3? We have some issues with 2021.2 running multiple iterations in python.

[ugrkm]

@EricBoiseLGSVL Yes, I have tried this with 2021.3. The issue is still not resolved.

[EricBoiseLGSVL]

Are you trying to access the original vehicle ref after a reset? ApiManager on reset deletes and removes the vehicle and all sensors so a call to set destination on the old ref will fail. Have you tried setting the EGO state instead of a reset?

[heeen]

As a workaround, can you try to stop sending events using AddCustom when the simulation is paused?

e.g. if(Mathf.Approximately(Time.timeScale, 0.0f)) return;

[heeen]

can you post your python script? Events are sent as the response to the simulator/run api, but that call blocks, so I wonder how you can issue another api command after that that gets confused by the events message