Jupyterlab extension for Mango widgets (bigdatagenomics#559)

* jupyter lab widgets work, renamed mango-viz to mango-pileup

.gitignore

@@ -25,11 +25,11 @@ workfiles/
 
 # metadata
 example-files/notebooks/metastore_db/*
-mango-viz/examples/.ipynb_checkpoints/*
+mango-pileup/examples/.ipynb_checkpoints/*
 */*/.ipynb_checkpoints/*
 
 # builds
-mango-viz/build/*
+mango-pileup/build/*
 
 # venvs
 venv/*
@@ -38,20 +38,20 @@ mango-python/adam/*
 mango-python/dist/*
 mango-python/build/*
 mango-python/bdgenomics.mango.egg-info/*
-mango-viz/metastore_db/*
-mango-viz/venv/*
+mango-pileup/metastore_db/*
+mango-pileup/venv/*
 *.bak
 
-mango-viz/bdgenomics/mango/pileup/static/*
-mango-viz/bdgenomics/mango/js/dist/*
-mango-viz/bdgenomics.mango.pileup.egg-info/*
+mango-pileup/bdgenomics/mango/pileup/static/*
+mango-pileup/bdgenomics/mango/js/dist/*
+mango-pileup/bdgenomics.mango.pileup.egg-info/*
 *.pyc
 
 example-files/metastore_db/*
 
 
-# mango-viz node modules
-mango-viz/bdgenomics/mango/js/node_modules/*
+# mango-pileup node modules
+mango-pileup/bdgenomics/mango/js/node_modules/*
 
 # docs _build
 docs/_build/*

docs/conf.py

@@ -40,7 +40,7 @@
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
 this_dir = os.path.dirname(os.path.abspath(__file__))
-mango_viz_dir = os.path.abspath(os.path.join(this_dir, '..','mango-viz'))
+mango_viz_dir = os.path.abspath(os.path.join(this_dir, '..','mango-pileup'))
 mango_python_dir = os.path.abspath(os.path.join(this_dir, '..','mango-python'))
 
 # make sure paths exists for API docs.

docs/installation/source.rst

@@ -46,15 +46,15 @@ To build and test `Mango’s Python bindings <#python>`__, first set environment
 .. code:: bash
 
    export SPARK_HOME=<PATH_TO_SPARK>
-   export MANGO_HOME=<PATH_TO_MANGO> 
-   
+   export MANGO_HOME=<PATH_TO_MANGO>
+
 Next, build Mango jars without running tests, by running the following command from the root of the Mango repo install directory:
 
 .. code:: bash
 
    mvn clean package -DskipTests
 
-Additionally, the PySpark dependencies must be on the Python module load path and the Mango JARs must be built and provided to PySpark. This can be done with the following bash commands: 
+Additionally, the PySpark dependencies must be on the Python module load path and the Mango JARs must be built and provided to PySpark. This can be done with the following bash commands:
 
 .. code:: bash
 
@@ -72,17 +72,17 @@ Next, install dependencies using the following commands:
    cd mango-python
    make prepare
    cd ..
-   cd mango-viz
+   cd mango-pileup
    make prepare
    cd ..
-   
-Finally, run ``mvn package`` again, this time enabling the ``python`` profile as well as tests:   
+
+Finally, run ``mvn package`` again, this time enabling the ``python`` profile as well as tests:
 
 
 .. code:: bash
 
     mvn package -P python
 
-This will enable the ``mango-python`` and ``mango-viz`` module as part of the Mango build.
+This will enable the ``mango-python`` and ``mango-pileup`` module as part of the Mango build.
 This module uses Maven to invoke a Makefile that builds a Python egg and
-runs tests. 
+runs tests.

docs/jupyterWidgets/usage.rst

@@ -3,12 +3,12 @@ Jupyter Widget Usage
 
 **Note**: Python 2.7 is `dropping support <https://www.anaconda.com/end-of-life-eol-for-python-2-7-is-coming-are-you-ready/>`__ January 1, 2020. For this reason, Mango no longer supports Python 2.
 
-The Mango widgets are Jupyter widgets built using `pileup.js <https://github.com/hammerlab/pileup.js>`__ .
+The Mango widgets are Jupyter widgets built using `pileup.js <https://github.com/hammerlab/pileup.js>`__.
 The widgets support visualizations for alignments, features,
-variants, and genotypes in a Jupyter Notebook.
+variants, and genotypes in a Jupyter Notebook and Jupyter lab version >2.0.
 
-Installation
-------------
+Installation for Jupyter notebook
+---------------------------------
 
 First, install and enable bdgenomics.mango.pileup, a Jupyter Widget:
 
@@ -17,18 +17,44 @@ First, install and enable bdgenomics.mango.pileup, a Jupyter Widget:
 
     pip install bdgenomics.mango.pileup
 
-    jupyter nbextension enable --py widgetsnbextension
-    jupyter nbextension install --py --symlink --user bdgenomics.mango.pileup
-    jupyter nbextension enable bdgenomics.mango.pileup --user --py
+    jupyter nbextension install --py --user bdgenomics.mango.pileup
+    jupyter nbextension install --py --user widgetsnbextension
 
+    jupyter nbextension enable --py --user widgetsnbextension
+    jupyter nbextension enable --py --user bdgenomics.mango.pileup
 
+**Note**: If you are using an conda environment, install extensions using ``--sys-prefix``:
 
-These tutorials show how to create a Jupyter pileup.js widget. An example notebook can be found in the `Mango Github repository <https://github.com/bigdatagenomics/mango/blob/master/mango-viz/examples/pileup-tutorial.ipynb>`__.
+.. code:: bash
+
+    jupyter nbextension install --py --sys-prefix bdgenomics.mango.pileup
+    jupyter nbextension install --py --sys-prefix widgetsnbextension
+
+    jupyter nbextension enable --py --sys-prefix widgetsnbextension
+    jupyter nbextension enable --py --sys-prefix bdgenomics.mango.pileup 
+
+This will install the bdgenomics.mango.pileup extension into your current conda environment.
+
+Installation for Jupyter lab
+----------------------------
+
+To use the Mango widgets in Jupyter lab, you will need the following requirements:
+
+- `Jupyter lab version > 2.0 <https://jupyterlab.readthedocs.io/en/stable/getting_started/installation.html>`__
+- `node > 12 <https://nodejs.org/en/download/>`__
+
+.. code:: bash
+
+    pip install bdgenomics.mango.pileup
+    jupyter labextension install @jupyter-widgets/jupyterlab-manager # install the Jupyter widgets extension
+    jupyter labextension install bdgenomics.mango.pileup
+
+These tutorials show how to create a Jupyter pileup.js widget. An example notebook can be found in the `Mango Github repository <https://github.com/bigdatagenomics/mango/blob/master/mango-pileup/examples/pileup-tutorial.ipynb>`__.
 
 Pileup Example
 --------------
 
-This example shows how to visualize alignments through a Jupyter widget. It is located in ``<PATH_TO_MANGO>/mango-viz/examples/pileup-tutorial.ipynb``.
+This example shows how to visualize alignments through a Jupyter widget.
 
 .. code:: python
 
@@ -106,4 +132,4 @@ This example shows how to visualize genotypes through a Jupyter widget.
     genotypes = pileup.PileupViewer(locus="chr22:21340030-21340150", reference="hg19", tracks=tracks)
     genotypes
 
-.. image:: ../img/jupyterWidgets/genotypeWidget.png
+.. image:: ../img/jupyterWidgets/genotypeWidget.png

docs/mangoPython/api.rst

@@ -1,4 +1,4 @@
-Mango Python API Documentation
-==============================
+Mango Pyspark API Documentation
+===============================
 
-.. automodule:: bdgenomics.mango
+.. automodule:: bdgenomics.mango

docs/mangoPython/example.rst

@@ -24,7 +24,7 @@ the open port assigned by Jupyter notebook. There are three notebooks that can b
 - `example-files/notebooks/mango-python-alignment.ipynb <https://github.com/bigdatagenomics/mango/blob/master/example-files/notebooks/mango-python-alignment.ipynb>`__
 - `example-files/notebooks/mango-python-coverage.ipynb <https://github.com/bigdatagenomics/mango/blob/master/example-files/notebooks/mango-python-coverage.ipynb>`__
 - `example-files/notebooks/mango-python-variants.ipynb <https://github.com/bigdatagenomics/mango/blob/master/example-files/notebooks/mango-python-variants.ipynb>`__
-- `example-files/notebooks/mango-viz.ipynb <https://github.com/bigdatagenomics/mango/blob/master/example-files/notebooks/mango-viz.ipynb>`__
+- `example-files/notebooks/mango-pileup.ipynb <https://github.com/bigdatagenomics/mango/blob/master/example-files/notebooks/mango-pileup.ipynb>`__
 
 
 Running the Mango Notebook with Parameters

mango-distribution/src/main/assembly/assembly.xml

@@ -36,7 +36,7 @@
       <fileMode>0755</fileMode>
     </fileSet>
     <fileSet>
-      <directory>../mango-viz/bdgenomics.mango.pileup.egg-info</directory>
+      <directory>../mango-pileup/bdgenomics.mango.pileup.egg-info</directory>
       <outputDirectory>bdgenomics.mango.pileup.egg-info</outputDirectory>
       <directoryMode>0755</directoryMode>
       <fileMode>0755</fileMode>
@@ -57,7 +57,7 @@
       <fileMode>0644</fileMode>
     </fileSet>
     <fileSet>
-      <directory>../mango-viz/dist</directory>
+      <directory>../mango-pileup/dist</directory>
       <includes>
         <include>bdgenomics.mango.pileup*.egg</include>
       </includes>

mango-viz/Makefile → mango-pileup/Makefile

@@ -35,24 +35,36 @@ node=node
 mango_version:=$(shell $(python) bdgenomics/mango/pileup/_version.py)
 sdist_name:=mango.pileup-$(mango_version).tar.gz
 tests=bdgenomics/mango/
+js_path=bdgenomics/mango/js
 extras=
 current_commit:=$(shell git log --pretty=oneline -n 1 -- $(pwd) | cut -f1 -d " ")
 dirty:=$(shell (git diff --exit-code && git diff --cached --exit-code) > /dev/null || printf -- --DIRTY)
 
+
+# Error if python and npm frontend versions are not equal
+py_frontend_version=$(shell ($(python) -c "import sys;sys.path.insert(1, 'bdgenomics/mango/pileup');import _version;print(_version.__frontend_version__[1:])"))
+npm_frontend_version=$(shell (grep version $(js_path)/package.json | cut -c 15- | rev | cut -c 3- | rev))
+
+ifneq ($(py_frontend_version), $(npm_frontend_version))
+$(error The python frontend versions-$(py_frontend_version)-and npm frontend versions-$(npm_frontend_version)-do not match.)
+endif
+
+
+
 green=\033[0;32m
 normal=\033[0m\n
 red=\033[0;31m
 
 develop:
 	rm -rf bdgenomics/mango/pileup/static
-	rm -rf bdgenomics/mango/js/dist
+	rm -rf $(js_path)/dist
 	rm -rf bdgenomics.mango.pileup.egg-info/
-	- $(pip) install -v -e .$(extras)
-	- $(jupyter) nbextension uninstall pileup
-	- $(jupyter) nbextension enable --py widgetsnbextension
-	- $(jupyter) nbextension install --py --symlink --user bdgenomics.mango.pileup
-	- $(jupyter) nbextension enable bdgenomics.mango.pileup --py --user
-
+	$(pip) install -v -e .$(extras)
+	$(jupyter) nbextension uninstall bdgenomics.mango.pileup
+	$(jupyter) nbextension enable --py widgetsnbextension
+	$(jupyter) nbextension install --py --symlink --user bdgenomics.mango.pileup
+	$(jupyter) nbextension enable bdgenomics.mango.pileup --py --user
+	$(jupyter) labextension install $(js_path)/ # install widget for jupyter lab
 
 sdist: dist/$(sdist_name)
 dist/$(sdist_name):
@@ -75,24 +87,35 @@ test: check_build_reqs
 	mkdir -p target
 	$(python) -m pytest -vv --junitxml target/pytest-reports/tests.xml $(tests)
 	# run lint and flow for js
-	pushd bdgenomics/mango/js \
-		&& npm run lint
+	pushd $(js_path) \
+		&& $(npm) run test
+
 
+npm:
+	pushd $(js_path) \
+		&& ./scripts/publish \
+		&& $(npm) publish
 
 pypi: clean sdist
 	set -x \
 	&& $(python) setup.py egg_info sdist bdist_egg \
 	&& twine check dist/* \
 	&& twine upload dist/*
-
+
+publish:
+	npm
+	pypi
+
 # clean
 clean:
-	rm -rf bdgenomics/mango/js/dist
+	rm -rf $(js_path)/dist
 	rm -rf bdgenomics.mango.pileup.egg-info/
 	rm -rf bdgenomics/mango/pileup/static/
 	rm -rf dist/
-	rm -rf bdgenomics/mango/js/node_modules
-	$(jupyter) nbextension uninstall pileup
+	rm -rf $(js_path)/node_modules
+	$(jupyter) nbextension uninstall bdgenomics.mango.pileup
+	$(jupyter) labextension uninstall bdgenomics.mango.pileup
+
 
 
 # check for node, jupyter and npm
@@ -112,5 +135,5 @@ prepare:
 	$(pip) install -r requirements.txt
 
 	# install node_modules
-	cd bdgenomics/mango/js; \
+	cd $(js_path); \
 		$(npm) install

mango-pileup/README.md

@@ -0,0 +1,60 @@
+# bdgenomics.mango.pileup
+
+bdgenomics.mango.pileup is a Jupyter widget that allows users to view genomic reads, variants and features in a Jupyter notebook or in Jupyter lab version >2.0.
+bdgenomics.mango.pileup builds off of [pileup.js](https://github.com/hammerlab/pileup.js).
+
+## Installation
+
+### from pip:
+
+    $ pip install bdgenomics.mango.pileup
+
+#### Enable widgets for Jupyter notebook:
+
+    $ jupyter nbextension enable --py --sys-prefix bdgenomics.mango.pileup  # can be skipped for notebook version 5.3 and above
+
+#### Install widgets for jupyter lab:
+
+    $ jupyter labextension install bdgenomics.mango.pileup
+
+## Running Examples
+
+    $ First run installation, explained above.
+    $ cd examples
+    $ jupyter notebook
+
+Or run in Jupyter lab:
+
+    $ jupyter lab
+
+### from Source:
+
+For a development installation (requires npm (version >= 3.8) and node (version >= 4.0)):
+
+    $ git clone https://github.com/bdgenomics/mango
+    $ cd mango-pileup
+    $ rm -r bdgenomics/mango/pileup/static
+
+Install bdgenomics.mango.pileup:
+
+    $ pip install -e .
+    $ jupyter nbextension install --py --symlink --sys-prefix bdgenomics.mango.pileup
+    $ jupyter nbextension enable --py --sys-prefix bdgenomics.mango.pileup]
+
+Note for developers: the --symlink argument on Linux or OS X allows one to modify the JavaScript code in-place. This feature is not available with Windows.
+
+
+To install the JupyterLab extension, install the Package, the Jupyter widgets extension, and the bdgenomics.mango.pileup extension:
+
+
+    $ pip install -e .
+    $ cd bdgenomics/mango/js/
+    $ jupyter labextension install @jupyter-widgets/jupyterlab-manager # install the Jupyter widgets extension
+    $ jupyter labextension install .
+
+
+#### Testing:
+
+mango-pileup has a Makefile with a ``test`` command to run tests for python and javascript:
+
+    $ make test

mango-pileup/RELEASE.md

@@ -0,0 +1,33 @@
+# Release on npm
+To release a new version of bdgenomics.mango.pileup on npm:
+
+- Update
+- cd bdgenomics/mango/js.
+- Update `version` in `package.json`. Commit this change.
+- Run `scripts/publish.sh`.
+- Run `npm publish`.
+- Push to github.
+
+If you are publishing a beta version, run:
+
+- `npm publish --tag beta`
+
+
+# Release on PyPI
+
+To release a new version of bdgenomics.mango.pileup on PyPI:
+
+Update version in _version.py (set release version, remove 'dev')
+Also, update the npm frontend version in _version.py to match (TODO LINK to npm)
+Update version in bdgenomics/mango/pileup/js/package.json
+make clean
+make sdist
+git add and git commit
+make sdist
+twine upload dist/*.tar.gz
+git tag -a X.X.X -m 'comment'
+
+Update _version.py (add 'dev' and increment minor)
+git add and git commit
+git push
+git push --tags

mango-pileup/bdgenomics.mango.pileup.json

@@ -0,0 +1,5 @@
+{
+  "load_extensions": {
+    "bdgenomics.mango.pileup/extension": true
+  }
+}