Rebuild

Author: 9bow

docs/_downloads/032d653a4f5a9c1ec32b9fc7c989ffe1/seq2seq_translation_tutorial.ipynb

@@ -15,7 +15,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "\nSingle-Machine Model Parallel Best Practices\n================================\n**Author**: `Shen Li <https://mrshenli.github.io/>`_\n\nModel parallel is widely-used in distributed training\ntechniques. Previous posts have explained how to use\n`DataParallel <https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html>`_\nto train a neural network on multiple GPUs; this feature replicates the\nsame model to all GPUs, where each GPU consumes a different partition of the\ninput data. Although it can significantly accelerate the training process, it\ndoes not work for some use cases where the model is too large to fit into a\nsingle GPU. This post shows how to solve that problem by using **model parallel**,\nwhich, in contrast to ``DataParallel``, splits a single model onto different GPUs,\nrather than replicating the entire model on each GPU (to be concrete, say a model\n``m`` contains 10 layers: when using ``DataParallel``, each GPU will have a\nreplica of each of these 10 layers, whereas when using model parallel on two GPUs,\neach GPU could host 5 layers).\n\nThe high-level idea of model parallel is to place different sub-networks of a\nmodel onto different devices, and implement the ``forward`` method accordingly\nto move intermediate outputs across devices. As only part of a model operates\non any individual device, a set of devices can collectively serve a larger\nmodel. In this post, we will not try to construct huge models and squeeze them\ninto a limited number of GPUs. Instead, this post focuses on showing the idea\nof model parallel. It is up to the readers to apply the ideas to real-world\napplications.\n\n<div class=\"alert alert-info\"><h4>Note</h4><p>For distributed model parallel training where a model spans multiple\n    servers, please refer to\n    `Getting Started With Distributed RPC Framework <rpc_tutorial.html>`__\n    for examples and details.</p></div>\n\nBasic Usage\n-----------\n\n"
+        "\nSingle-Machine Model Parallel Best Practices\n================================\n**Author**: `Shen Li <https://mrshenli.github.io/>`_\n\nModel parallel is widely-used in distributed training\ntechniques. Previous posts have explained how to use\n`DataParallel <https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html>`_\nto train a neural network on multiple GPUs; this feature replicates the\nsame model to all GPUs, where each GPU consumes a different partition of the\ninput data. Although it can significantly accelerate the training process, it\ndoes not work for some use cases where the model is too large to fit into a\nsingle GPU. This post shows how to solve that problem by using **model parallel**,\nwhich, in contrast to ``DataParallel``, splits a single model onto different GPUs,\nrather than replicating the entire model on each GPU (to be concrete, say a model\n``m`` contains 10 layers: when using ``DataParallel``, each GPU will have a\nreplica of each of these 10 layers, whereas when using model parallel on two GPUs,\neach GPU could host 5 layers).\n\nThe high-level idea of model parallel is to place different sub-networks of a\nmodel onto different devices, and implement the ``forward`` method accordingly\nto move intermediate outputs across devices. As only part of a model operates\non any individual device, a set of devices can collectively serve a larger\nmodel. In this post, we will not try to construct huge models and squeeze them\ninto a limited number of GPUs. Instead, this post focuses on showing the idea\nof model parallel. It is up to the readers to apply the ideas to real-world\napplications.\n\n<div class=\"alert alert-info\"><h4>Note</h4><p>For distributed model parallel training where a model spans multiple\n    servers, please refer to\n    `Getting Started With Distributed RPC Framework <rpc_tutorial.html>`__\n    for examples and details.</p></div>\n\nBasic Usage\n-----------\n"
       ]
     },
     {
@@ -175,7 +175,7 @@
       "name": "python",
       "nbconvert_exporter": "python",
       "pygments_lexer": "ipython3",
-      "version": "3.6.7"
+      "version": "3.7.4"
     }
   },
   "nbformat": 4,