MultiprocessingSampler is broken for tf

[ryanjulian]

Edit examples/tf/trpo_swimmer/ray_sampler.py to use MultiprocessingSampler and you will get:

ValueError: Variable GaussianMLPPolicy/GaussianMLPModel/dist_params/mean_network/hidden_0/kernel already exists, disallowed. Did you mean to set reuse=True or reuse=tf.AUTO_REUSE in VarScope?

[krzentner]

I don't think I can solve this issue this week. Let's just not allow TF + MultiprocessingSampler for now.

[yeukfu]

@krzentner @ryanjulian. multiprocessing uses os.fork() to start a process by default in Linux. This causes problems when we use TF, as fork will copy the tf.Graph() into sub process. There are two methods to address the issue:

• Start the sub processes by spawn instead of fork, which will create a fresh sub process. But using spawn needs to add the followings into the launcher.

from multiprocessing import freeze_support

if __name__ == '__main__':
    freeze_support()     # https://docs.python.org/3/library/multiprocessing.html#multiprocessing.freeze_support

    trpo_swimmer_ray_sampler(seed=100)

• Clear the graph in sub process at the beginning. I try to call tf.compat.v1.reset_default_graph() but failed because this cannot be called in a nested graph.

[krzentner]

Yeah, unfortunately there's no way to "clear the graph." If you ever create a TF graph in a process, you can never fork in that process again. Modifying the launcher is also really unpleasant, especially because neither click nor clize know how to handle this case automatically (even though they reasonably should be able to).

There is another option though: fork before the graph is created. This is tricky though. Optimally, we would only fork one process when MultiprocessingSampler is imported, then use it to create additional processes when needed. However, this is complicated by the MultiprocessingSampler currently using the mp.Process objects directly (not through a mp.Queue) when shutting down. Unfortunately, due to some unfortunate design mistakes in multiprocessing, it's impossible to transmit the mp.Process object to another process, so that logic would need to be partially re-written to make this option work.