Split updating submodules and building their content

* Add `make pull-modules` and `make build-modules` commands
* Use these commands in GitHub Actions workflow
* Update the building instructions in README.md
* Update the documentation on adding new submodules

Author: NickVolynkin

.github/workflows/main.yml

@@ -19,8 +19,9 @@ jobs:
       TARANTOOL_UPDATE_URL: ${{secrets.TARANTOOL_UPDATE_URL}}
     steps:
       - uses: actions/checkout@v2
-      - run: bash update_submodules.sh
       - run: cmake .
+      - run: make pull-modules
+      - run: make build-modules
       - run: make json
       - run: make json-ru
       - run: make singlehtml

CMakeLists.txt

@@ -298,6 +298,20 @@ add_custom_target(epub-ru ALL
     COMMENT "Building Russian epub with Sphinx"
 )
 
+add_custom_target(pull-modules ALL
+    COMMAND /usr/bin/env bash
+        ./pull_submodules.sh
+    COMMENT "Update submodules to their latest commits"
+)
+
+
+add_custom_target(build-modules ALL
+    COMMAND /usr/bin/env bash
+        ./build_submodules.sh
+    COMMENT "Rebuild and copy content from submodules"
+)
+
+
 
 if (LATEX_FOUND)
     add_custom_target(pdf-ru ALL COMMENT "PDF generation")

README.md

@@ -6,39 +6,68 @@ Tarantool documentation source, published at https://www.tarantool.io/doc/.
 
 ## How to build Tarantool documentation using [Docker](https://www.docker.com)
 
-### Build docker image
+### Prepare for work
 
-```bash
-docker build -t tarantool-doc-builder .
-```
+First of all, pull the image for building the docs.
 
-### Build Tarantool documentation using *tarantool-doc-builder* image
-## NOTE: 
-> Run this command only if you don't have untracked files!
-  check it by `git status` 
-  Also failures during git submodule update can be fixed by:
-   ```bash
-   git submodule deinit -f .
-   git submodule update --init
-   ```
-
-Init and update submodules:
-```bash
-docker run --rm -it -v $(pwd):/doc tarantool-doc-builder sh -c "./update_submodules.sh"
-```
-or do it manually:
 ```bash
-git submodule init
-git submodule update
-git pull --recurse-submodules
-git submodule update --remote --merge
+docker pull tarantool/doc-builder
 ```
 
-Init make commands:
+Next, initialize a Makefile for your OS:
+
 ```bash
-docker run --rm -it -v $(pwd):/doc tarantool-doc-builder sh -c "cmake ."
+docker run --rm -it -v $(pwd):/doc tarantool/doc-builder sh -c "cmake ."
 ```
 
+### Update submodules and generate documentation sources from code
+
+A big part of documentation sources comes from several other projects,
+connected as Git submodules.
+To include their latest contents in the docs, run these two steps.
+
+1.  Update the submodules:
+
+    ```bash
+    docker run --rm -it -v $(pwd):/doc tarantool/doc-builder sh -c "make pull-modules"
+    ```
+    
+    This will initialize Git submodules and update them to the top of the stable
+    branch in each repository.
+    
+    You can also do without a Docker container:
+    
+    ```bash
+    git submodule update --init
+    git fetch --recurse-submodules
+    git submodule update --remote --checkout
+    ```
+   
+    `git submodule update` can sometimes fail, for example,
+    when you have changes in submodules' files.
+    You can reinitialize submodules to fix the problem.
+    
+    **Caution:** all untracked changes in submodules will be lost!
+
+        ```bash
+        git submodule deinit -f .
+        git submodule update --init
+        ```
+
+2.  Build the submodules content:
+
+    ```bash
+    docker run --rm -it -v $(pwd):/doc tarantool/doc-builder sh -c "make build-modules"
+    ```
+    
+    This command will do two things:
+
+    1. Generate documentation source files from the source code
+    2. Copy these files to the right places under the `./doc/` directory.
+
+    If you're editing submodules locally, repeat this step
+    to view the updated results.
+
 Now you're ready to build and preview the documentation locally.
 
 ### Build and run the documentation on your machine
@@ -49,7 +78,7 @@ Every time you make changes in the source files, it will rebuild the docs
 and refresh the browser page.
 
 ```bash
-docker run --rm -it -p 8000:8000 -v $(pwd):/doc tarantool-doc-builder sh -c "make autobuild"
+docker run --rm -it -p 8000:8000 -v $(pwd):/doc tarantool/doc-builder sh -c "make autobuild"
 ```
 
 First build will take some time.
@@ -60,8 +89,8 @@ and the browser tab with preview will reload automatically.
 You can also build the docs manually with `make html`,
 and then serve them using python3 built-in server:
 ```bash
-docker run --rm -it -v $(pwd):/doc tarantool-doc-builder sh -c "make html"
-docker run --rm -it -v $(pwd):/doc tarantool-doc-builder sh -c "make html-ru"
+docker run --rm -it -v $(pwd):/doc tarantool/doc-builder sh -c "make html"
+docker run --rm -it -v $(pwd):/doc tarantool/doc-builder sh -c "make html-ru"
 python3 -m http.server --directory output/html
 ```
 
@@ -73,24 +102,23 @@ python -m SimpleHTTPServer
 
 then go to [localhost:8000](http://localhost:8000) in your browser.
 
-
 There are other commands which can run 
-in the *tarantool-doc-builder* container:
+in the *tarantool/doc-builder* container:
 
 ```bash
-docker run --rm -it -v $(pwd):/doc tarantool-doc-builder sh -c "make html"
-docker run --rm -it -v $(pwd):/doc tarantool-doc-builder sh -c "make html-ru"
-docker run --rm -it -v $(pwd):/doc tarantool-doc-builder sh -c "make singlehtml"
-docker run --rm -it -v $(pwd):/doc tarantool-doc-builder sh -c "make singlehtml-ru"
-docker run --rm -it -v $(pwd):/doc tarantool-doc-builder sh -c "make pdf"
-docker run --rm -it -v $(pwd):/doc tarantool-doc-builder sh -c "make pdf-ru"
-docker run --rm -it -v $(pwd):/doc tarantool-doc-builder sh -c "make json"
-docker run --rm -it -v $(pwd):/doc tarantool-doc-builder sh -c "make json-ru"
-docker run --rm -it -v $(pwd):/doc tarantool-doc-builder sh -c "make epub"
-docker run --rm -it -v $(pwd):/doc tarantool-doc-builder sh -c "make epub-ru"
-docker run --rm -it -v $(pwd):/doc tarantool-doc-builder sh -c "make update-pot"
-docker run --rm -it -v $(pwd):/doc tarantool-doc-builder sh -c "make update-po"
-docker run --rm -it -v $(pwd):/doc tarantool-doc-builder sh -c "make update-po-force"
+docker run --rm -it -v $(pwd):/doc tarantool/doc-builder sh -c "make html"
+docker run --rm -it -v $(pwd):/doc tarantool/doc-builder sh -c "make html-ru"
+docker run --rm -it -v $(pwd):/doc tarantool/doc-builder sh -c "make singlehtml"
+docker run --rm -it -v $(pwd):/doc tarantool/doc-builder sh -c "make singlehtml-ru"
+docker run --rm -it -v $(pwd):/doc tarantool/doc-builder sh -c "make pdf"
+docker run --rm -it -v $(pwd):/doc tarantool/doc-builder sh -c "make pdf-ru"
+docker run --rm -it -v $(pwd):/doc tarantool/doc-builder sh -c "make json"
+docker run --rm -it -v $(pwd):/doc tarantool/doc-builder sh -c "make json-ru"
+docker run --rm -it -v $(pwd):/doc tarantool/doc-builder sh -c "make epub"
+docker run --rm -it -v $(pwd):/doc tarantool/doc-builder sh -c "make epub-ru"
+docker run --rm -it -v $(pwd):/doc tarantool/doc-builder sh -c "make update-pot"
+docker run --rm -it -v $(pwd):/doc tarantool/doc-builder sh -c "make update-po"
+docker run --rm -it -v $(pwd):/doc tarantool/doc-builder sh -c "make update-po-force"
 ```
 
 ## Localization
@@ -119,7 +147,7 @@ Upload translation sources any time when they have changed:
 
 ```bash
 # first, update the translation sources
-docker run --rm -it -v $(pwd):/doc tarantool-doc-builder sh -c "make update-pot"
+docker run --rm -it -v $(pwd):/doc tarantool/doc-builder sh -c "make update-pot"
 
 # next, upload them to Crowdin
 crowdin upload 
@@ -138,7 +166,7 @@ Then reformat them to see the real changes.
 
 ```bash
 crowdin download
-docker run --rm -it -v $(pwd):/doc tarantool-doc-builder sh -c "make reformat-po"
+docker run --rm -it -v $(pwd):/doc tarantool/doc-builder sh -c "make reformat-po"
 ```
 ## How to contribute
 

build_submodules.sh

@@ -2,13 +2,6 @@
 
 set -x
 
-mkdir -p ~/.ssh/
-ssh-keyscan github.com >> ~/.ssh/known_hosts
-git submodule init
-git submodule update
-git pull --recurse-submodules
-git submodule update --remote --merge
-
 project_root=$(pwd)
 echo "${project_root}"
 cartridge_root="${project_root}/modules/cartridge"

doc/contributing/docs/infra.rst

@@ -59,12 +59,12 @@ a submodule.
 
 .. _guidelines_doc_submodules_update:
 
-2. Update update_submodule.sh
+2. Update build_submodules.sh
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Next, we should define what directories and files are to be copied from
 the submodule repository into the documentation one before building
-documentation. These settings are defined in the ``update_submodule.sh`` file
+documentation. These settings are defined in the ``build_submodules.sh`` file
 which is in the root directory of the documentation repository.
 
 We can take examples of the already existing submodules to show the logic of
@@ -85,7 +85,7 @@ So, we need to:
 *   copy the entire content of the  ``./modules/metrics/doc/monitoring/`` directory to
     ``./doc/book/monitoring/``.
 
-The corresponding lines in the ``update_submodule.sh`` file are the following:
+The corresponding lines in the ``build_submodules.sh`` file are the following:
 
 ..  code-block:: bash
 
@@ -113,7 +113,7 @@ So, we need to:
 *   copy ``./modules/cartridge_cli/README.rst`` to
     ``./doc/book/cartridge/cartridge_cli/index.rst``.
 
-The corresponding settings in the ``update_submodule.sh`` file are the following:
+The corresponding settings in the ``build_submodules.st`` file are the following:
 
 ..  code-block:: bash
 

pull_submodules.sh

@@ -4,64 +4,6 @@ set -x
 
 mkdir -p ~/.ssh/
 ssh-keyscan github.com >> ~/.ssh/known_hosts
-git submodule init
-git submodule update
-git pull --recurse-submodules
-git submodule update --remote --merge
-
-project_root=$(pwd)
-echo "${project_root}"
-cartridge_root="${project_root}/modules/cartridge"
-rst_src="${project_root}/modules/cartridge/build.luarocks/build.rst"
-rst_dest="${project_root}/doc/book/cartridge"
-pot_src="${project_root}/modules/cartridge/build.luarocks/build.rst/locale"
-pot_dest="${project_root}/locale/book/cartridge"
-po_src="${project_root}/modules/cartridge/build.luarocks/build.rst/locale/ru/LC_MESSAGES"
-po_dest="${project_root}/locale/ru/LC_MESSAGES/book/cartridge"
-cartridge_cli_root="${project_root}/modules/cartridge-cli"
-cartridge_cli_dest="${rst_dest}/cartridge_cli"
-cartridge_cli_index_dest="${cartridge_cli_dest}/index.rst"
-monitoring_root="${project_root}/modules/metrics/doc/monitoring"
-monitoring_dest="${project_root}/doc/book"
-monitoring_grafana_root="${project_root}/modules/grafana-dashboard/doc/monitoring"
-luatest_root="${project_root}/modules/luatest/"
-luatest_dest="${project_root}/doc/reference/reference_rock/luatest"
-
-cartridge_kubernetes_root="${project_root}/modules/tarantool-operator/doc/cartridge_kubernetes_guide"
-cartridge_kubernetes_dest="${rst_dest}/"
-
-cd "${luatest_root}"
-ldoc --ext=rst --dir=rst --toctree="API" .
-cd "${luatest_dest}"
-yes | cp -fa "${luatest_root}/rst/." "${luatest_dest}"
-yes | cp "${luatest_root}/README.rst" "${luatest_dest}"
-
-mkdir -p "${monitoring_dest}"
-yes | cp -rf "${monitoring_root}" "${monitoring_dest}/"
-yes | cp -rf "${monitoring_grafana_root}" "${monitoring_dest}/"
-
-mkdir -p "${cartridge_cli_dest}"
-yes | cp -rf "${cartridge_cli_root}/README.rst" "${cartridge_cli_index_dest}"
-
-mkdir -p "${cartridge_kubernetes_dest}"
-yes | cp -rf "${cartridge_kubernetes_root}" "${cartridge_kubernetes_dest}"
-
-cd "${cartridge_root}" || exit
-tarantoolctl rocks install \
-  https://raw.githubusercontent.com/tarantool/LDoc/tarantool/ldoc-scm-2.rockspec \
-  --server=http://rocks.moonscript.org
-CMAKE_DUMMY_WEBUI=true tarantoolctl rocks make
-
-cd "${rst_src}" || exit
-mkdir -p "${rst_dest}"
-find . -iregex '.*\.\(rst\|png\)$' -exec cp -r --parents {} "${rst_dest}" \;
-
-cd "${pot_src}" || exit
-mkdir -p "${pot_dest}"
-find . -name '*.pot' -exec cp -r --parents {} "${pot_dest}" \;
-
-cd "${po_src}" || exit
-mkdir -p "${po_dest}"
-find . -name '*.po' -exec cp -r --parents {} "${po_dest}" \;
-
-sed -i -e '/Cartridge CLI<cartridge_cli\/index>/a\' -e '\ \ \ Cartridge Kubernetes guide<cartridge_kubernetes_guide/index>' "${rst_dest}/index.rst"
+git submodule update --init
+git fetch --recurse-submodules
+git submodule update --remote --checkout

update_submodules.sh

@@ -0,0 +1,7 @@
+#!/bin/bash
+
+# This file is kept for compatibility.
+# Run `make pull-modules build-modules` instead
+
+. ./pull_submodules.sh
+. ./build_submodules.sh