(🫵 Input needed) Think about how using a RLHF-style proxy reward on top of the VLM influences our plans

[Eugleo]

If we are to believe the RL-VLM-F paper, if we get to actual training, we might want to use the VLM to train a proxy reward function instead of using the VLM directly as a reward.

The reward is then based on querying the VLM for preferences over a pair of images (videos, in our case).

This fact (us caring about VLM preferences more than the actual numbers we get from it) might influence how we generate the trajectories #3 and visualize the results #5, hence I created this issue.

[Eugleo]

My thoughts:

1. We generate a bunch of trajectories where a task is and isn't being done.
2. For each trajectory we obtain a score from the model (e.g. similarity to the description of the task)
3. We report the raw scores in a nice plot (this is what we're doing now)
4. We find a principled way to use the raw scores of two videos to generate a preference label for the pair (-1, 0, 1)
5. We report the statistics of the preferences in a nice plot, too.

It seems that 4, 5 are (very) nice to have for the presentation, but depend on 1, 2, 3, so we should focus on those first anyway. However, rather than spending a lot of time on making 3 as nice as possible, for example, we could do 4, 5 instead.

[Dont-Care-Didnt-Ask]

I think there is no sense in creating preference pairs from raw scores of individually evaluated videos -- the whole point of using pairs is that humans (and supposedly VLMs) are often more consistent at comparing two images/videos, rather than assigning a score from 0 to 1. If we compute scores individually, we lose this effect.

Also, I think that if our data collection procedure is "gather pairs (trajectory, its thorough text description)", then we can easily pivot to comparison-based approach later.

[Eugleo]

I'm not sure I agree with the first part, though we probably do agree on all the practical issues (second part) which is what matters. Let's focus on 1, 2, 3 from my original comment and see about 4,5 later.

As to why I don't agree — even for feeding the model single examples I can see a model being bad for getting out raw numbers, e.g. because it is noisy, but being good at producing an ordering, esp. if we train another function on top of it (which presumably gets rid of some of the noise in the orderings, too).