what do actions in 'multi_post_retargeting_v3.pkl' means

[bb0928]

Hi, when i am trying to load your action sequence in your demonstration files, i founded that the hand pose are shaking randomly, which seems not the true actions. my code are as follow:

    data = pickle.load(open('./demonstrations/relocate-mustard_bottle.pkl', 'rb'))
    actions = data['seq_10/multi_post_retargeting_v3.pkl']['actions']
    # qpos = data['seq_100/multi_post_retargeting_v3.pkl']['sim_data']
    # Loop visualization
    for i in range(actions.shape[0]):

        # action = np.random.uniform(low, high)
        action = actions[i]
        obs, reward, done, _ = env.step(action)
        env.render()

could you please tell me what does the actions in .pkl means? does it correspond to the action defined in mujoco? thanks.

[yzqin]

@bb0928

Your observation is correct. The actions are derived from inverse dynamics calculations on the estimated pose sequence. Since these poses are estimated rather than manually annotated, they contain significant noise which leads to high cumulative errors in the action sequence.

This is why replaying the action sequence in simulation doesn't produce accurate results. For this reason, DexMV's imitation learning avoids trajectory-level methods in favor of approaches like DAPG. These methods process imitation data as individual state-action pairs, where noise has less impact. Sequential replay of actions would compound errors during simulation.

This accumulated error is an inherent limitation of video-based pose estimation - it cannot perfectly replicate physical dynamics. For applications requiring high precision without scaling concerns, teleoperation would be more suitable. When learning from video, it's best to avoid relying on detailed physics, especially for sequential actions.