Authomatic (auth/login), Pyramid-friend comments, misc fixes

.gitignore

@@ -4,6 +4,7 @@
 *.class
 tmp
 *.tmp
+venv/
 backup/
 smasher/tax/gbif
 smasher/tax/ncbi
@@ -22,3 +23,4 @@ smasher/lib/json-simple-1.1.1.jar
 deploy/setup/CONFIG
 webapp/static/statistics/
 lint/
+private/

README.md

@@ -1,16 +1,16 @@
-opentree
-========
+webapp-tree-viewer
+==================
 
-This is the repository for the Open Tree of Life web applications, one of many subsystems making up the Open Tree of Life project code.
+This is the repository for the Open Tree of Life synthetic-tree viewer, one of many subsystems making up the Open Tree of Life project code. Previously, this was bundled in a common repo with the study-curation app, but we're breaking these apart as part of our conversion from web2py to Pyramid.
 
-For Open Tree of Life documentation, see [the germinator repository's wiki](https://github.com/OpenTreeOfLife/germinator/wiki). The 'deployment system' and web API documentation sources that formerly resided in this repository now live in the [germinator repository](https://github.com/OpenTreeOfLife/germinator).
+For Open Tree of Life documentation, see [the germinator repository's wiki](https://github.com/OpenTreeOfLife/germinator/wiki).
 
 The following instructions have not been reviewed in a long time. For local installation a better place to start might be [this wiki page](https://github.com/OpenTreeOfLife/opentree/wiki/Installing-a-local-curator-and-tree-browser-test-server).
 
 Installation
 ============
-See the phylografter instructions for
-more details about using web2py.
+
+**TBD - pending changes in converting from web2py to the Pyramid framework.**
 
 We strongly recommend using a virtual environment to manage the version of
 Python and installed modules. We're currently running opentree with Python
@@ -36,8 +36,25 @@ The included **requirements.txt** file lists known-good versions of all the requ
 python modules for opentree, plus a few convenience modules. To [install these modules 
 using pip](http://www.pip-installer.org/en/latest/cookbook.html#requirements-files), 
 
+The file **requirements.in** captures our intentions, typically following the
+latest version of each module unless frozen to address bugs or incompatibilities.
+
 <pre>
+# create a python3 venv inside the top-level folder
+cd ~/repos/webapp-tree-viewer
+export VENV="$(pwd)/venv"
+python3 -m venv $VENV
+
+# install the pinned set of known-good modules
 pip install -r requirements.txt
+
+# OR during development, try to update these using `requirements.in`
+pip install -r requirements.in
+
+# capture the resulting versions for production deploymentsl
+$VENV/bin/pip freeze -r requirements.in > requirements.txt
+# capture and save the full dependency tree for all modules
+$VENV/bin/pipdeptree > pipdeptree.out
 </pre>
 
 The contents of the webapp subdirectory are a web2py application.  Make a symbolic 
@@ -136,44 +153,7 @@ http://www.web2py.com/examples/default/download MTH used version 2.4.2 of web2py
 Subdirectories
 --------------
 
-mockup
-: JAR's hand-written html that mock up the design of the site.
-
-smasher
-: construction of synthetic taxonomy. See https://github.com/OpenTreeOfLife/opentree/wiki/Open-Tree-Taxonomy for details
-
-webapp/controllers
-: python code executed by web2py when a URL is successfully mapped to a controller.
-
-webapp/modules
-: python code that can be imported and used by the webpp, but is not exposed as controller
-
-webapp/models
-: code that describes the database structure used by the web app
-
-webapp/views
-: templates for the page content that is rendered in response to a query. The view is typically specific to a few controllers in the web app
-
-webapp/static
-: static content to be returned by the web app. Contains css, images and js subdirectories for commonly used items.
-
-webapp/private
-: the location to be used for storing installation-specific configuration information.
-
-webapp/cron
-: directory that stores commands to be executed periodically by the web2py framework when the app is running.
-
-webapp/databases
-: the location of the database files used by web2py.
-
-webapp/languages
-: web2py code for internationalization
-
-webapp/cache, webapp/databases, webapp/errors, webapp/sessions, webapp/uploads
-: directories used by web2py to store content associated with user's activities. Content here should not need to be committed to version control.
-
-curator/*
-: similar subdirectories to those in webapp/* above
+**TBD - pending changes in converting from web2py to the Pyramid framework.**
 
 Acknowledgements
 ----------------

curator/views/study/edit.html

@@ -1342,7 +1342,7 @@ <h4 title="Click to show mapping options" style="margin-top: 0;">
                     {{ pass }}
                     {{ except: }}
                             <option selected="selected">ERROR loading context values</option>
-                       {{ pass }}
+                    {{ pass }}
                     </select>
                     <div style="text-align: center; margin: 0 0 1em;">
                         <button class="btn btn-small Xbtn-info"

pipdeptree.out

@@ -0,0 +1,141 @@
+Authomatic==1.2.1
+bleach==5.0.1
+  - six [required: >=1.9.0, installed: 1.16.0]
+  - webencodings [required: Any, installed: 0.5.1]
+configparser==5.3.0
+cookiecutter==2.1.1
+  - binaryornot [required: >=0.4.4, installed: 0.4.4]
+    - chardet [required: >=3.0.2, installed: 5.0.0]
+  - click [required: >=7.0,<9.0.0, installed: 8.1.3]
+  - Jinja2 [required: >=2.7,<4.0.0, installed: 3.1.2]
+    - MarkupSafe [required: >=2.0, installed: 2.1.1]
+  - jinja2-time [required: >=0.2.0, installed: 0.2.0]
+    - arrow [required: Any, installed: 0.15.8]
+      - python-dateutil [required: Any, installed: 2.2]
+        - six [required: Any, installed: 1.16.0]
+    - jinja2 [required: Any, installed: 3.1.2]
+      - MarkupSafe [required: >=2.0, installed: 2.1.1]
+  - python-slugify [required: >=4.0.0, installed: 6.1.2]
+    - text-unidecode [required: >=1.3, installed: 1.3]
+  - pyyaml [required: >=5.3.1, installed: 6.0]
+  - requests [required: >=2.23.0, installed: 2.28.1]
+    - certifi [required: >=2017.4.17, installed: 2022.6.15.1]
+    - charset-normalizer [required: >=2,<3, installed: 2.1.1]
+    - idna [required: >=2.5,<4, installed: 3.3]
+    - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.12]
+locket==0.1.1
+Markdown==3.5.1
+  - importlib-metadata [required: >=4.4, installed: 7.0.0]
+    - zipp [required: >=0.5, installed: 3.17.0]
+pip==23.3.1
+pipdeptree==2.3.1
+pytest-cov==3.0.0
+  - coverage [required: >=5.2.1, installed: 6.4.4]
+  - pytest [required: >=4.6, installed: 7.1.3]
+    - attrs [required: >=19.2.0, installed: 22.1.0]
+    - iniconfig [required: Any, installed: 1.1.1]
+    - packaging [required: Any, installed: 21.3]
+      - pyparsing [required: >=2.0.2,!=3.0.5, installed: 3.0.9]
+    - pluggy [required: >=0.12,<2.0, installed: 1.0.0]
+    - py [required: >=1.8.2, installed: 1.11.0]
+    - tomli [required: >=1.0.0, installed: 2.0.1]
+python-jwt==3.3.0
+  - jwcrypto [required: >=0.8, installed: 0.9.1]
+    - cryptography [required: >=2.3, installed: 38.0.1]
+      - cffi [required: >=1.12, installed: 1.15.1]
+        - pycparser [required: Any, installed: 2.21]
+    - deprecated [required: Any, installed: 1.2.13]
+      - wrapt [required: >=1.10,<2, installed: 1.14.1]
+    - six [required: Any, installed: 1.16.0]
+synthetic-tree-viewer==0.0
+  - plaster-pastedeploy [required: Any, installed: 0.7]
+    - PasteDeploy [required: >=2.0, installed: 2.1.1]
+      - setuptools [required: Any, installed: 60.10.0]
+    - plaster [required: >=0.5, installed: 1.0]
+      - setuptools [required: Any, installed: 60.10.0]
+  - pyramid [required: Any, installed: 2.0]
+    - hupper [required: >=1.5, installed: 1.10.3]
+    - plaster [required: Any, installed: 1.0]
+      - setuptools [required: Any, installed: 60.10.0]
+    - plaster-pastedeploy [required: Any, installed: 0.7]
+      - PasteDeploy [required: >=2.0, installed: 2.1.1]
+        - setuptools [required: Any, installed: 60.10.0]
+      - plaster [required: >=0.5, installed: 1.0]
+        - setuptools [required: Any, installed: 60.10.0]
+    - setuptools [required: Any, installed: 60.10.0]
+    - translationstring [required: >=0.4, installed: 1.4]
+    - venusian [required: >=1.0, installed: 3.0.0]
+    - webob [required: >=1.8.3, installed: 1.8.7]
+    - zope.deprecation [required: >=3.5.0, installed: 4.4.0]
+      - setuptools [required: Any, installed: 60.10.0]
+    - zope.interface [required: >=3.8.0, installed: 5.4.0]
+      - setuptools [required: Any, installed: 60.10.0]
+  - pyramid-debugtoolbar [required: Any, installed: 4.9]
+    - Pygments [required: Any, installed: 2.13.0]
+    - pyramid [required: >=1.4, installed: 2.0]
+      - hupper [required: >=1.5, installed: 1.10.3]
+      - plaster [required: Any, installed: 1.0]
+        - setuptools [required: Any, installed: 60.10.0]
+      - plaster-pastedeploy [required: Any, installed: 0.7]
+        - PasteDeploy [required: >=2.0, installed: 2.1.1]
+          - setuptools [required: Any, installed: 60.10.0]
+        - plaster [required: >=0.5, installed: 1.0]
+          - setuptools [required: Any, installed: 60.10.0]
+      - setuptools [required: Any, installed: 60.10.0]
+      - translationstring [required: >=0.4, installed: 1.4]
+      - venusian [required: >=1.0, installed: 3.0.0]
+      - webob [required: >=1.8.3, installed: 1.8.7]
+      - zope.deprecation [required: >=3.5.0, installed: 4.4.0]
+        - setuptools [required: Any, installed: 60.10.0]
+      - zope.interface [required: >=3.8.0, installed: 5.4.0]
+        - setuptools [required: Any, installed: 60.10.0]
+    - pyramid-mako [required: >=0.3.1, installed: 1.1.0]
+      - Mako [required: >=1.1.0, installed: 1.2.2]
+        - MarkupSafe [required: >=0.9.2, installed: 2.1.1]
+      - pyramid [required: Any, installed: 2.0]
+        - hupper [required: >=1.5, installed: 1.10.3]
+        - plaster [required: Any, installed: 1.0]
+          - setuptools [required: Any, installed: 60.10.0]
+        - plaster-pastedeploy [required: Any, installed: 0.7]
+          - PasteDeploy [required: >=2.0, installed: 2.1.1]
+            - setuptools [required: Any, installed: 60.10.0]
+          - plaster [required: >=0.5, installed: 1.0]
+            - setuptools [required: Any, installed: 60.10.0]
+        - setuptools [required: Any, installed: 60.10.0]
+        - translationstring [required: >=0.4, installed: 1.4]
+        - venusian [required: >=1.0, installed: 3.0.0]
+        - webob [required: >=1.8.3, installed: 1.8.7]
+        - zope.deprecation [required: >=3.5.0, installed: 4.4.0]
+          - setuptools [required: Any, installed: 60.10.0]
+        - zope.interface [required: >=3.8.0, installed: 5.4.0]
+          - setuptools [required: Any, installed: 60.10.0]
+    - repoze.lru [required: Any, installed: 0.7]
+  - pyramid-jinja2 [required: Any, installed: 2.10]
+    - jinja2 [required: >=2.5.0,!=2.11.2,!=2.11.1,!=2.11.0, installed: 3.1.2]
+      - MarkupSafe [required: >=2.0, installed: 2.1.1]
+    - markupsafe [required: Any, installed: 2.1.1]
+    - pyramid [required: >=1.3.0, installed: 2.0]
+      - hupper [required: >=1.5, installed: 1.10.3]
+      - plaster [required: Any, installed: 1.0]
+        - setuptools [required: Any, installed: 60.10.0]
+      - plaster-pastedeploy [required: Any, installed: 0.7]
+        - PasteDeploy [required: >=2.0, installed: 2.1.1]
+          - setuptools [required: Any, installed: 60.10.0]
+        - plaster [required: >=0.5, installed: 1.0]
+          - setuptools [required: Any, installed: 60.10.0]
+      - setuptools [required: Any, installed: 60.10.0]
+      - translationstring [required: >=0.4, installed: 1.4]
+      - venusian [required: >=1.0, installed: 3.0.0]
+      - webob [required: >=1.8.3, installed: 1.8.7]
+      - zope.deprecation [required: >=3.5.0, installed: 4.4.0]
+        - setuptools [required: Any, installed: 60.10.0]
+      - zope.interface [required: >=3.8.0, installed: 5.4.0]
+        - setuptools [required: Any, installed: 60.10.0]
+    - zope.deprecation [required: Any, installed: 4.4.0]
+      - setuptools [required: Any, installed: 60.10.0]
+  - waitress [required: Any, installed: 2.1.2]
+WebTest==3.0.0
+  - beautifulsoup4 [required: Any, installed: 4.11.1]
+    - soupsieve [required: >1.2, installed: 2.3.2.post1]
+  - waitress [required: >=0.8.5, installed: 2.1.2]
+  - WebOb [required: >=1.2, installed: 1.8.7]

private/config.example

@@ -0,0 +1,62 @@
+# Copy this file to "config" (and replace tokens below) to make it active
+#
+# DO NOT CHANGE the formatting of assignment lines (removing spaces, etc) in
+# this file! The 'config.example' files are used by our deployment scripts to
+# build the final config for each web2py app. Changes here might keep it
+# from being built properly.
+[security]
+secure_sessions_with_HTTPS = false
+
+# paths for the study/status page (DEPRECATED)
+[paths]
+#nexsonsdir = /usr/share/open-tree/avatol_nexsons
+#study_to_status_script = /usr/share/open-tree/avatol_nexsons/nexson2treemach/study_nexson_to_status_html.py
+
+[apis]
+# Settings for a registered app on GitHub, for example on devtree:
+# <https://github.com/settings/apps/open-tree-tree-explorer-dev>
+github_app_id = YOUR_APP_ID_HERE
+github_client_id = YOUR_CLIENT_ID_HERE
+github_redirect_uri = YOUR_REDIRECT_URI_HERE
+# DON'T INCLUDE 'github_client_secret' value here. For better security, this is
+# loaded from a separate file and added to config data on-the-fly.
+#
+# github_app_installation_id can be found from the installing organization's page, e.g.
+# <https://github.com/organizations/OpenTreeOfLife/settings/installations>
+# (the installation ID is in the URL of the Configure button here)
+github_app_installation_id = YOUR_APP_INSTALLATION_ID_HERE
+
+# List public-facing base URL for treemachine and taxomachine services
+# (NOTE that these are used by both server- and client-side code)
+[domains]
+treemachine = http://opentree.myserver.org
+taxomachine = http://opentree.myserver.org
+oti = http://opentree.myserver.org
+opentree_api = http://opentree.myserver.org
+collections_api_base_url = https://devapi.opentreeoflife.org
+favorites_api_base_url = https://devapi.opentreeoflife.org
+# Cached versions of some APIs will speed up repeated calls (see below).
+# These use a simple web2py cache implemented in phylesystem-api
+CACHED_treemachine = https://opentree.myserver.org/cached
+CACHED_taxomachine = https://opentree.myserver.org/cached
+CACHED_oti = https://opentree.myserver.org/cached
+
+# Building on the URLs above, complete the public-facing URL for all methods
+# (again, these should work from both server- and client-side code)
+#
+# NOTE: In our example setup, all service methods are proxied through Apache as
+# URLs within the main opentree site. So we assume that partial paths '/db/data/' 
+# are handled in the ProxyPass statement.
+[method_urls]
+getDraftTreeID_url = {CACHED_treemachine_domain}/v3/tree_of_life/about
+getSyntheticTree_url = {CACHED_treemachine_domain}/v3/tree_of_life/subtree
+getDraftSubtree_url = {treemachine_domain}/v3/tree_of_life/subtree
+doTNRSForAutocomplete_url = {taxomachine_domain}/v3/tnrs/autocomplete_name
+getContextsJSON_url = {CACHED_taxomachine_domain}/v3/tnrs/contexts
+getSynthesisSourceList_url = {CACHED_treemachine_domain}/v3/tree_of_life/about
+findAllStudies_url = {CACHED_oti_domain}/v3/studies/find_studies
+singlePropertySearchForStudies_url = {oti_domain}/v3/studies/find_studies
+singlePropertySearchForTrees_url = {oti_domain}/v3/studies/find_trees
+getTaxonInfo_url = {taxomachine_domain}/v3/taxonomy/taxon_info
+# Include one phylesystem-api method to download NexSON from Bibliographic References page
+API_load_study_GET_url = {opentree_api_domain}/v3/study/{STUDY_ID}

requirements.in

@@ -0,0 +1,16 @@
+requests
+python-dateutil==2.2
+locket==0.1.1
+bleach
+jwcrypto==0.9.1
+python_jwt
+pyramid==2.0
+pipdeptree
+cookiecutter
+ConfigParser
+markdown
+authomatic
+uvicorn
+prettydate
+pytz
+pyramid_retry