Fix Django 1.6 support, XML-RPC handler, tar file dirs, doc link, etc. #14

aaaaalbert · 2016-05-16T15:16:23Z

Fixes #5 (XML-RPC interface) and #9 (vesselinfo directory mishap).

See the commit messages for details.

Omitting the build file list for now -- `build.py` doesn't completely supplant `preparetest` yet as it lacks a way to copy stuff to sub-directories of the build target dir...

…llerbuilder into new_build

This build file is needed to set up the RepyV2 runtime which the custominstallerbuilder requires to run. We need to do that because plain Python `import`'s from Repy libraries don't currently work. Future contributors, please check if this workaround is still needed every time progress is made on SeattleTestbed/portability#25, SeattleTestbed/portability#26, SeattleTestbed/portability#27, or SeattleTestbed/portability#28!

This makes `seash` in the Repy runtime runnable, and lets you generate user keys more easily than via the clearinghouse website.

I don't know why `simplejson` was ever `import`'ed `as json`. Now that it doesn't seem to be around anymore, let's make sure we have *some* kind of JSON support.

The SeattleTestbed/clearinghouse and SeattleTestbed/custominstallerbuilder Django projects are laid out somewhat differently (for reasons lost in history). I'll try to make them more similar as I go through the changes required for supporting Django 1.6. In this case, this comes at the expense of forcing the user to set up environment variables (`DJANGO_SETTINGS_MODULE`, `PYTHONPATH`) rather than trying to configure them automatically (or so it seems --- I'll find out when testing.) Now is not the time for me to do porting *and* debug clever logic at the same time, so I'm going for the simpler, non-autoconfiguring method.

Cf. SantiagoTorres@a23561d670ec028e5f7e93cf64042852c576e075.

Cf. SantiagoTorres@d5466218b2d216ff4cce809b64652fd07caa0da9 in website/settings.py

See SantiagoTorres@e519a16b19fe15c8a40c1a320cfe4051d8df7755 and the Django 1.5 release notes for reference.

Oops, new_build assumes `.` is the repo root.

Streamlines key generation

We only need that script from dist, but it was a 200MB download before.

We should also add an explanation who this user is supposed to be (the clearinghouse user, not the custominstallerbuilder user), and better separate the roles in the future.

…encies dist does still include its own requirement file lists we currently rely on. Needs more refactoring.

@SantiagoTorres

This follows the example of SeattleTestbed/custominstallerbuilder@da5351a (with due thanks to @SantiagoTorres who provided the initial implementation.)

The property `raw_post_data` of Django `request`s was renamed to `body`, see relevant Django docs at https://code.djangoproject.com/ticket/17323

Remove the prefix `./` from `config_dir_rel`. This keeps `tar` from creating *two* dirs in the Linux tarball (`seattle` and `.`) which could confuse users, cause them to extract only one, and find `vesselinfo` missing when running the installer.

The property `raw_post_data` of Django `request`s was renamed to `body`, see relevant Django docs at https://code.djangoproject.com/ticket/17323 . (This mirrors fix 04239c9 which should have gone to this development branch, not `master`.)

See note at https://docs.djangoproject.com/en/1.5/ref/templates/builtins/#std:templatetag-url: """ Warning Don’t forget to put quotes around the function path or pattern name! Changed in Django 1.5: The first parameter used not to be quoted, which was inconsistent with other template tags. Since Django 1.5, it is evaluated according to the usual rules: it can be a quoted string or a variable that will be looked up in the context. """

lukpueh · 2016-05-17T22:25:13Z

I am on it. But I spent half a day trying to actually set up CIB and CH to test the patches, and was forced to deal with other issues:

#15
SeattleTestbed/docs#14
#10 (comment)

I think the process and the docs to setup CH and CIB could be dramatically simplified. But I don't no if this is a pressing concern.

lukpueh · 2016-05-18T16:00:19Z

This PR just grew dramatically because I merged albert's django16support branch into his master (this PR's merge candidate).

Nevertheless, we can still focus on the last couple of commits as initially addressed by this PR, because in production we already use albert's django16support branch at commit 8bfafc2 (see production server git ouputs below) which is 7 commits behind HEAD of django16support (we want to merge HEAD).
Also in the CIB installation docs we instruct to checkout django16support.

I suggest that we update the docs and maybe also the production server's remote urls as soon as this has been reviewed and merged.

custominstallerbuilder@sensibilityclearinghouse:~/custominstallerbuilder$ git remote -v
origin  https://github.com/aaaaalbert/custominstallerbuilder.git (fetch)
origin  https://github.com/aaaaalbert/custominstallerbuilder.git (push)
custominstallerbuilder@sensibilityclearinghouse:~/custominstallerbuilder$ git branch
* django16support
custominstallerbuilder@sensibilityclearinghouse:~/custominstallerbuilder$ git log

commit 8bfafc2c04cce2eaf5ae853629ef5bd1d8330188
Author: aaaaalbert <[email protected]>
Date:   Mon Apr 13 16:03:17 2015 +0200

    Fix SeattleTestbed/custominstallerbuilder#9

    Remove the prefix `./` from `config_dir_rel`. This keeps `tar` from creating

lukpueh · 2016-05-18T21:39:58Z

xmlrpc/templates/xmlrpc-info.html

  <p>Interface with this page using an XML-RPC client, or use the 
  <a href="{% url builder %}">interactive builder</a>.
  Visit the Seattle project wiki for the 
  <a href="https://seattle.poly.edu/wiki/CustomInstallerBuilderApi">XML-RPC API</a>.</p>
 </div>
-{% endblock %}</div>
 {% endblock %}


Yep. We only need one {% endblock %}.

lukpueh · 2016-05-18T22:50:38Z

Here is how I tested this PR:

Installed CIB (c.f. CIB Installation)
Configured ~/custominstallerbuilder/local/settings.py (c.f. Create and configure local settings)
Created a keypair (cf. generatekeys.py)

Did not adapt the build scripts. Instead cloned, initialized and built installer-packaging repo

Changed the variables in ~/installer-packaging/RUNNABLE/rebuild_base_installers.py like so:

software_update_url = 'http://seattle.cs.washington.edu/couvb/updatesite/0.1/'
public_key_file = '/root/cib.publickey'
private_key_file = '/root/cib.privatekey'

base_installer_directory = '/home/cib/custominstallerbuilder/html/static/installers/base/'
base_installer_archive_dir = '/home/cib/custominstallerbuilder/html/static/installers/old_base_installers/'

user = 'cib'

Note: The CIB docs suggest to use http://blackbox.poly.edu/updatesite/ as software_update_url but it seemed to me that this url was outdated

Created base_installer and base_installer_archive_dir directories
Modified both ~/custominstallerbuilder/DEPENDENCIES/softwareupdater.py and ~/installer-packaging/RUNNABLE/seattle_repy/softwareupdater.py to match with the above generated publickey like explained in the docs (c.f. softwareupdater URL and Keys)

Created base installers:

$ python ~/installer-packaging/RUNNABLE/rebuild_base_installers.py <version as in nmmain.py>

Configured apache

I don't think CIB and installer-packaging are supposed to be used like this. What if we would only clone installer-packaging repo when building CIB and use the installer-packaging for both, creating the base installers and also to create a repy_runtime for the cib django app.

lukpueh · 2016-05-19T21:58:19Z

xmlrpc/views.py

@@ -56,6 +56,6 @@ def xmlrpc_handler(request):
    xmlrpc_handler.register_instance(PublicFunctions())

    response = HttpResponse(mimetype='application/xml')
-    response.write(xmlrpc_handler._marshaled_dispatch(request.raw_post_data))
+    response.write(xmlrpc_handler._marshaled_dispatch(request.body))


I could reproduce the error before this change. And can verify that it works after applying the fix. Also, since this is just a name refactor, documented in the Django docs, I would say this is good to go.

aaaaalbert · 2016-05-20T13:20:12Z

@lukpueh, asking for clarification of your last comment. Are you saying that the whole PR is good, or are you referring to the last bit in views.py only?

lukpueh · 2016-05-20T14:51:24Z

xmlrpc/templates/xmlrpc-info.html

@@ -3,6 +3,9 @@
 {% block content %}
 <div class="section">
  <h2>XML-RPC Interface</h2>
-  <p>Interface with this page using an XML-RPC client, or use the <a href="{% url builder %}">interactive builder</a>. Visit the Seattle project wiki for the <a href="https://seattle.cs.washington.edu/wiki/CustomInstallerBuilderAPI">XML-RPC API</a>.</p>
+  <p>Interface with this page using an XML-RPC client, or use the 
+  <a href="{% url "builder" %}">interactive builder</a>.


Seems to work. It finds the intended url which leads to the intended view which renders the intended html.

lukpueh · 2016-05-20T15:53:39Z

Clarification:
I just moved my inline-comments from the outdated diffs to the actual diffs that are going to be merged with this PR. All commits that this PR addressed initially are now tested and commented on.

How I tested this - Part 2:

Locally set up CIB (Ubuntu 15.10., Python 2.7.10, Django 1.8.3, @aaaaalbert's cib/master)
Locally set up CH (Ubuntu 15.10., Python 2.7.10, Django 1.8.3, @awwad's ch/fix_django)
Surfed to the parts of CIB/CH that invoked the code from the relevant commits, before and after the change.
Wrote comments on GitHub

Conclusion:
As stated earlier, I only reviewed the most recent diffs, those that were originally addressed by this PR. All the earlier commits are already used in production and therefor belong to master anyways.
Let's merge this PR!

Caveats:

CIB docs need to be updated (cf. [issue Can't find installer bundle #15(https://github.com/Custominstallerbuilder docs should use SeattleTestbed/installer-packaging docs#15)
self-signed certificates don't work anymore in python 2.7.9+ (cf. issue #16)

aaaaalbert · 2016-05-20T16:03:48Z

OK, merging!

aaaaalbert and others added 30 commits August 5, 2014 16:17

Adding init file list and script for new build process.

65a428f

Omitting the build file list for now -- `build.py` doesn't completely supplant `preparetest` yet as it lacks a way to copy stuff to sub-directories of the build target dir...

Cleaning up the init script.

1d1d53e

Added missing components

223cfaf

Merge branch 'new_build' of https://github.com/aaaaalbert/custominsta…

ed39028

…llerbuilder into new_build

Forgot to add resource repo...

7e21884

Adding Affix to custominstallerbuilder's config_build.txt

5505fe4

This makes `seash` in the Repy runtime runnable, and lets you generate user keys more easily than via the clearinghouse website.

Import any of json, simplejson.

d78c667

I don't know why `simplejson` was ever `import`'ed `as json`. Now that it doesn't seem to be around anymore, let's make sure we have *some* kind of JSON support.

Use RepyV2, dy_import_module_symbols rather than repyhelper

9beccc5

Workaround so that dylink finds all of the libraries.

23193c2

Updated init/build scripts and config

7336c9a

Need common repo too

8bbf897

Add target dirs to config_initialize

4fe7742

Update location of required functions from django.conf.urls

699c6d0

Cf. SantiagoTorres@a23561d670ec028e5f7e93cf64042852c576e075.

Use new template loader, Loader.

7b1083e

Cf. SantiagoTorres@d5466218b2d216ff4cce809b64652fd07caa0da9 in website/settings.py

Updating templates to Django>=1.5 {% url ...} syntax.

8337a21

See SantiagoTorres@e519a16b19fe15c8a40c1a320cfe4051d8df7755 and the Django 1.5 release notes for reference.

Replaced washington.edu by poly.edu

a8341e5

Fix path to DEPENDENCIES in config_build.txt

53440df

Reverting 53440df

b1a4f6a

Oops, new_build assumes `.` is the repo root.

Add generatekeys.py

92fa22c

Streamlines key generation

Adding Django csrf_token to HTTP post

51c256f

Moving in the rebuild_base_installers script from SeatteTestbed/dist

6c5eeb3

We only need that script from dist, but it was a 200MB download before.

Removing the dist repo from dependencies followin 6c5eeb3

53c4714

Refactor seattlegeni to $USER

3f8e808

We should also add an explanation who this user is supposed to be (the clearinghouse user, not the custominstallerbuilder user), and better separate the roles in the future.

Remove chdir workaround for dylink

9659e28

Reverting parent commit which removed SeattleTestbed/dist from depend…

24dd782

…encies dist does still include its own requirement file lists we currently rely on. Needs more refactoring.

Add WSGI dir and startup script

89bfa31

This follows the example of SeattleTestbed/custominstallerbuilder@da5351a (with due thanks to @SantiagoTorres who provided the initial implementation.)

Fix SeattleTestbed#5

04239c9

The property `raw_post_data` of Django `request`s was renamed to `body`, see relevant Django docs at https://code.djangoproject.com/ticket/17323

Fix SeattleTestbed#9

0e1573a

Remove the prefix `./` from `config_dir_rel`. This keeps `tar` from creating *two* dirs in the Linux tarball (`seattle` and `.`) which could confuse users, cause them to extract only one, and find `vesselinfo` missing when running the installer.

aaaaalbert added 6 commits April 13, 2015 16:03

Fix SeattleTestbed#9

8bfafc2

Remove the prefix `./` from `config_dir_rel`. This keeps `tar` from creating *two* dirs in the Linux tarball (`seattle` and `.`) which could confuse users, cause them to extract only one, and find `vesselinfo` missing when running the installer.

Fix SeattleTestbed#5

d3e0fae

The property `raw_post_data` of Django `request`s was renamed to `body`, see relevant Django docs at https://code.djangoproject.com/ticket/17323 . (This mirrors fix 04239c9 which should have gone to this development branch, not `master`.)

Fix link to XML-RPC API doc

8605571

Fix link to XML-RPC API doc

eb236bf

Remove duplicated lines from previous commit

a15af52

aaaaalbert assigned lukpueh May 16, 2016

lukpueh added 2 commits May 18, 2016 11:22

Merges django16support into master

a3170fb

Merge remote-tracking branch 'upstream/master'

736ad75

aaaaalbert changed the title ~~Fix XML-RPC handler, tar file dirs, doc link~~ Fix Django 1.6 support, XML-RPC handler, tar file dirs, doc link, etc. May 18, 2016

Re-delete rebuild_base_installers.sh

fa76193

lukpueh reviewed May 18, 2016
View reviewed changes

lukpueh reviewed May 19, 2016
View reviewed changes

This was referenced May 20, 2016

Need changes to find repy_runtime #10

Closed

Custominstallerbuilder docs should use SeattleTestbed/installer-packaging SeattleTestbed/docs#15

Open

lukpueh reviewed May 20, 2016
View reviewed changes

aaaaalbert merged commit 8750ffd into SeattleTestbed:master May 20, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix Django 1.6 support, XML-RPC handler, tar file dirs, doc link, etc. #14

Fix Django 1.6 support, XML-RPC handler, tar file dirs, doc link, etc. #14

aaaaalbert commented May 16, 2016

lukpueh commented May 17, 2016

lukpueh commented May 18, 2016 •

edited

Loading

lukpueh May 18, 2016

lukpueh commented May 18, 2016

lukpueh May 19, 2016

aaaaalbert commented May 20, 2016

lukpueh May 20, 2016

lukpueh commented May 20, 2016

aaaaalbert commented May 20, 2016

Fix Django 1.6 support, XML-RPC handler, tar file dirs, doc link, etc. #14

Fix Django 1.6 support, XML-RPC handler, tar file dirs, doc link, etc. #14

Conversation

aaaaalbert commented May 16, 2016

lukpueh commented May 17, 2016

lukpueh commented May 18, 2016 • edited Loading

lukpueh May 18, 2016

Choose a reason for hiding this comment

lukpueh commented May 18, 2016

lukpueh May 19, 2016

Choose a reason for hiding this comment

aaaaalbert commented May 20, 2016

lukpueh May 20, 2016

Choose a reason for hiding this comment

lukpueh commented May 20, 2016

aaaaalbert commented May 20, 2016

lukpueh commented May 18, 2016 •

edited

Loading