possible bug with HERReplayBuffer under pyrtorch

[st2yang]

Hi,

I managed to run examples/tf/her_ddpg_fetchreach.py with tuned parameters (PR #1739 ). Then I try using pytorch, and something strange happens. It was working fine at first then crashes dramatically. The comparison between tf and torch.

The main modification is tf.policies.ContinuousMLPPolicy, tf.q_functions.ContinuousMLPQFunction -> torch.policies.DeterministicMLPPolicy, torch.q_functions.ContinuousMLPQFunction, while other parameters are kept the same.

[ryanjulian]

Great work!

Wow, this is a tough one. Do you know if this behavior persists across several random seeds?

DDPG can be a very unstable algorithm, and it's possible you're seeing some seed-dependent instability.

By the way, it's easy to share experiments from garage using tensorboard.dev. See the docs for full details.

Other ideas:

• Turning down the learning rate
• Perhaps the replay buffer becomes over-populated with positive (high-reward) examples and doesn't have negative (low-reward) examples? You could check this by logging the mean/median/min/max (or a histogram) of the rewards in the batches the algorithm samples from the replay buffer. The docs cover how to do that.
• Collecting negatives even when the agent is succeeding is one purpose for the exploration_policy. Does it improve if you increase sigma in the exploration policy, or use a different exploration policy?

[ryanjulian]

@krzentner @maliesa96

[st2yang]

@ryanjulian This behavior persists under the default seed. But I didn't test it with different seeds. I can try to test possible cases when I have more time.

[irisliucy]

@st2yang Would you share your experiment settings/ essential code snippets? That will help us to reproduce the issues and fix the potential bugs.

[st2yang]

@irisliucy Sorry for the late reply. Can you try adapting examples/tf/her_ddpg_fetchreach.py to run with pytorch? I just changed some tf-related to be pytorch things, and training got worse. I used very similar parameters when debugging.

If the bug persists, you should see something similar.