Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability to use state+rgb+segmentation obs_mode #806

Closed
AlexandreBrown opened this issue Jan 25, 2025 · 0 comments · Fixed by #821
Closed

Add ability to use state+rgb+segmentation obs_mode #806

AlexandreBrown opened this issue Jan 25, 2025 · 0 comments · Fixed by #821
Labels
enhancement New feature or request

Comments

@AlexandreBrown
Copy link

AlexandreBrown commented Jan 25, 2025

Feature Description

Hi,
As discussed in the discord channel, it would be great to have the ability to use both state (privileged state information) and visual information (eg: rgb+segmentation).

Current Limitations

One can already use state + manually do env.render() to get rgb but it is currently not possible to get the segmentation data.

Use Case

The use case behind this is to train an expert policy using privileged state information so that we can then generate expert trajectories with rgb+segmentation data to obtain rgb segments of frames generated by an expert.

Current Workaround

The current workaround seems to be to :

  1. Train a policy using state obs_mode.
  2. Use the trained expert policy to collect and save trajectories using the RecordEpisode wrapper.
  3. Convert the state observations to rgb+segmentation using Maniskill conversion tool.
    Ex:
python -m mani_skill.trajectory.replay_trajectory \
  --traj-path my_trajectory.h5 \
  --save-traj \
  --target-control-mode my_control_mode \
  --obs-mode rgb+segmentation \
  --use-env-states \
  --num-procs 10
@AlexandreBrown AlexandreBrown changed the title Add ability to use sate+rgb+segmentation obs_mode Add ability to use state+rgb+segmentation obs_mode Jan 26, 2025
@StoneT2000 StoneT2000 added the enhancement New feature or request label Jan 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants