line 2 lead to imp_sampling=1

[xzhang2523]

log_action_probs = action_dists.log_prob(actions)

    imp_sampling = torch.exp(log_action_probs - log_action_probs.detach())
    # Change to torch.matmul
    reward_loss = -torch.mean(imp_sampling * reward_advs)

Since, log_action_probs - log_action_probs.detach()=0,
imp_sampling is a all one vector

[ajlangley]

I answered this question in your other issue about 0 KL divergence. Let me know if this is not a satisfactory answer.