Experiment with rearranging the parts structure

Author: hjwp

Makefile

@@ -27,6 +27,7 @@ part%.forbook.asciidoc: part%.asciidoc
 book.html: part1.forbook.asciidoc
 book.html: part2.forbook.asciidoc
 book.html: part3.forbook.asciidoc
+book.html: part4.forbook.asciidoc
 book.html: $(SOURCES)
 
 

atlas.json

@@ -10,6 +10,7 @@
         "pre-requisite-installations.asciidoc",
         "video_plug.asciidoc",
         "acknowledgments.asciidoc",
+
         "part1.asciidoc",
         "chapter_01.asciidoc",
         "chapter_02_unittest.asciidoc",
@@ -18,19 +19,23 @@
         "chapter_05_post_and_database.asciidoc",
         "chapter_06_explicit_waits_1.asciidoc",
         "chapter_07_working_incrementally.asciidoc",
-        "part2.asciidoc",
         "chapter_08_prettification.asciidoc",
+
+        "part2.asciidoc",
         "chapter_09_docker.asciidoc",
         "chapter_10_production_readiness.asciidoc",
         "chapter_11_server_prep.asciidoc",
         "chapter_12_ansible.asciidoc",
+
+        "part3.asciidoc",
         "chapter_13_organising_test_files.asciidoc",
         "chapter_14_database_layer_validation.asciidoc",
         "chapter_15_simple_form.asciidoc",
         "chapter_16_advanced_forms.asciidoc",
+
+        "part4.asciidoc",
         "chapter_17_javascript.asciidoc",
         "chapter_18_second_deploy.asciidoc",
-        "part3.asciidoc",
         "chapter_19_spiking_custom_auth.asciidoc",
         "chapter_20_mocking_1.asciidoc",
         "chapter_21_mocking_2.asciidoc",
@@ -39,6 +44,7 @@
         "chapter_24_outside_in.asciidoc",
         "chapter_25_CI.asciidoc",
         "chapter_26_page_pattern.asciidoc",
+
         "chapter_27_hot_lava.asciidoc",
         "epilogue.asciidoc",
         "appendix_I_PythonAnywhere.asciidoc",

book.asciidoc

@@ -27,21 +27,23 @@ include::chapter_04_philosophy_and_refactoring.asciidoc[]
 include::chapter_05_post_and_database.asciidoc[]
 include::chapter_06_explicit_waits_1.asciidoc[]
 include::chapter_07_working_incrementally.asciidoc[]
+include::chapter_08_prettification.asciidoc[]
 
 include::part2.forbook.asciidoc[]
-include::chapter_08_prettification.asciidoc[]
 include::chapter_09_docker.asciidoc[]
 include::chapter_10_production_readiness.asciidoc[]
 include::chapter_11_server_prep.asciidoc[]
 include::chapter_12_ansible.asciidoc[]
+
+include::part3.forbook.asciidoc[]
 include::chapter_13_organising_test_files.asciidoc[]
 include::chapter_14_database_layer_validation.asciidoc[]
 include::chapter_15_simple_form.asciidoc[]
 include::chapter_16_advanced_forms.asciidoc[]
+
+include::part4.forbook.asciidoc[]
 include::chapter_17_javascript.asciidoc[]
 include::chapter_18_second_deploy.asciidoc[]
-
-include::part3.forbook.asciidoc[]
 include::chapter_19_spiking_custom_auth.asciidoc[]
 include::chapter_20_mocking_1.asciidoc[]
 include::chapter_21_mocking_2.asciidoc[]

chapter_09_docker.asciidoc

@@ -10,144 +10,31 @@ This will be the 9th chapter of the final book. The GitHub repo is available at
 If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the author at [email protected].
 ****
 
-[quote, 'http://bit.ly/2uhCXnH[Devops Borat]']
+[quote, Malvina Reynolds]']
 ______________________________________________________________
-Is all fun and game until you are need of put it in production.
+Little boxes, all the same
 ______________________________________________________________
 
-It's time to deploy the first version of our site and make it public.
-They say that if you wait until you feel _ready_ to ship,
-then you've waited too long.
-
-Is our site usable? Is it better than nothing? Can we make lists on it?
-Yes, yes, yes.
-
-No, you can't log in yet.
-No, you can't mark tasks as completed.
-But do we really need any of that stuff?
-Not really--and you can never be sure
-what your users are _actually_ going to do with your site
-once they get their hands on it.
-We think our users want to use the site for to-do lists,
-but maybe they actually want to use it
-to make "top 10 best fly-fishing spots" lists,
-for which you don't _need_ any kind of "mark completed" function.
-We won't know until we put it out there.
-
-Over the next couple of chapters we're going to go through
-and actually deploy our site to a real, live web server.
-
-You might be tempted to skip this bit--there's lots of daunting stuff in it,
-and maybe you think this isn't what you signed up for.
-But I _strongly_ urge you to give it a go.
-This is one of the sections of the book I'm most pleased with,
-and it's one that people often write to me
-saying they were really glad they stuck through it.
-
-If you've never done a server deployment before,
-it will demystify a whole world for you,
-and there's nothing like the feeling of seeing your site live
-on the actual internet.
-Give it a buzzword name like "DevOps"
-if that's what it takes to convince you it's worth it.
-
-.New content alert!  Pls send feedback.
-****
-As part of my work on the third edition of the book,
-I've made big changes to the deployment chapters,
-so this stuff is all very fresh.
+In this chapter, we'll start by adapting our FTs so that they can run against a container,
+And then we'll set about containerising our app,
+and get those tests passing our code running inside Docker.
 
-What that means is that I'd, really, really love feedback from readers.
-Please have a go at following along and let me know what you think!
-I'm [email protected], or you can open up
-https://github.com/hjwp/Book-TDD-Web-Dev-Python/issues[GitHub Issues]
-or Pull Requests.
+* We'll build a minimal Dockerfile with everything we need to run our site.
 
-Let me know how you got on, if you get stuck on anything,
-if any explanations don't make sense,
-or if any of the instructions don't work for you.
+* We'll learn how to build and run a container on our machine.
 
-I hope you enjoy the new version!
-****
+* We'll make a few changes to our source code layout. like using an _src_ folder.
 
-=== The Danger Areas of Deployment
-// XUAN: I read the title as meaning _TDD and_ the danger areas of deployment, but the following section seems to be talking about the danger areas of deployment?
-
-Deploying a site to a live web server can be a tricky topic.
-Oft-heard is the forlorn cry __"but it works on my machine!"__
-
-((("deployment", "danger areas of")))
-Some of the danger areas of deployment include:
-
-Networking::
-    Once we're off our own machine, networking issues come in:
-    making sure that DNS is routing our domain to the correct IP address for our server,
-    making sure our server is configured to listen to traffic coming in from the world,
-    making sure it's using the right ports,
-    and making sure any firewalls in the way are configured to let traffic through.
-
-Dependencies::
-    We need to make sure that the packages our software relies on
-    (Python, Django, and so on) are installed on the server,
-    and have the correct versions.
-
-The database::
-    There can be permissions and path issues,
-    and we need to be careful about preserving data between deploys.
-
-Static files (CSS, JavaScript, images, etc.)::
-    Web servers usually need special configuration for serving these.
-    ((("static files", "challenges of")))
-
-Security and Configuration::
-    Once we're on the public internet,
-    we need to worry more about security.
-    Various settings that are really useful for local development
-    (like the Django debug page)
-    become dangerous in production
-    (because they expose our source code in tracebacks).
-
-Reproducibility and Divergence between local dev and prod::
-    All of the above add up to differences between your local development environemnt
-    and the way code runs in production.
-    We want to be able to reproduce the way things work on our machine,
-    as closely as possible, in production (and vice-versa)
-    to give us as much confidence as possible that
-    "it works on my machine" means "it's going to work in production".
-
-
-One way to approach the problem is to get a server
-and start manually configuring and installing everything,
-hacking about until it works,
-and maybe think about automating things laterfootnote:[
-This was, more or less, the approach I took in earlier editions of the book.
-With a fair bit of testing thrown in of course.].
-
-But if there's one thing we've learned
-in the world of agile/lean software development,
-it's that taking smaller steps usually pays off.
-
-How can we take smaller, safer steps towards a production deployment?
-Can we _simulate_ the process of moving to a server
-so that we can iron out all the bugs
-before we actually take the plunge?
-Can we then make small changes one at a time,
-solving problems one by one,
-rather than having to bite off everything in one mouthful?
-Can we use our existing test suite to make sure things
-work on the server, as well as locally?
-
-Absolutely we can.  And from the title of the chapter,
-I'm sure you're already guessing that Docker is going
-to be part of the answer.
+* We'll start flushing out a few issues around networking and the database.
 
 
 === Docker, Containers and Virtualization
 
 Docker is a commercial product that wraps several free
 and open source technologies from the world of Linux,
 sometimes referred to as "containerization".
-(Feel free to skip this section if you already know all about Docker.)
+
+NOTE: Feel free to skip this section if you already know all about Docker.
 
 You may have already heard of the idea of "virtualization",
 which allows a single physical computer to pretend to be several machines.
@@ -176,10 +63,13 @@ So you can run multiple operating systems using separate VMs
 on the same physical box, as in <<virtualization-diagram>>.
 
 [[virtualization-diagram]]
-.Virtualenvs vs Virtual Machines
-image::images/virtualenv-vs-vm.png["A diagram showing a virtualenv running in an operating system, vs multiple virtual machines running different operating systems on a single real machine"]
+.Physical vs Virtual Machines
+image::images/virtualenv-vs-vm.png["A diagram showing a physical machine, with an operating system and a Python virtualenv running inside it, vs multiple virtual machines running different operating systems on a single real machine"]
 
 
+// TODO; remove virtualenvs from this diagram, they just confuse things.
+// add another diagram later to contrast venvs with dockers.
+
 Containerization works at the operating system level:
 it gives you multiple virtual operating systems that
 all run on a single real OS.footnote:[
@@ -207,11 +97,6 @@ If you're running Linux containers on a Mac or a PC,
 it's because you're actually running them on a Linux VM under the hood.].
 See <<containers-diagram>> for an illustration.
 
-[[containers-diagram]]
-.Containers Share a Kernel in the Host Operating System
-image::images/containers-diagram.png["Diagram showing one or more containers running on a single host operating system, showing that each container uses the kernel from the host OS, but is able to have its own filesystem, based on an image, but also possibly mounting directories from the host filesystem"]
-
-
 The upshot of this is that containers are much "cheaper".
 You can start one up in milliseconds,
 and you can run hundreds on the same machine.
@@ -224,6 +109,9 @@ NOTE: If you're new to all this, I know it's a lot to wrap your head around!
   Hopefully, following along with these chapters and seeing them working in practice
   will help you to better understand the theory.
 
+[[containers-diagram]]
+.Containers Share a Kernel in the Host Operating System
+image::images/containers-diagram.png["Diagram showing one or more containers running on a single host operating system, showing that each container uses the kernel from the host OS, but is able to have its own filesystem, based on an image, but also possibly mounting directories from the host filesystem"]
 
 
 ==== Why not just use a virtualenv?
@@ -246,6 +134,8 @@ and they don't need to understand the intricacies of any particular
 language's packaging systems.
 
 
+
+
 ==== Docker and your CV
 
 That's all well and good for the _theoretical_ justification,
@@ -265,97 +155,14 @@ For many working developers, a container image is the final artifact of their wo
 it's what they "deliver",
 and often the rest of the deployment process is something they rarely have to think about.
 
+In any case, without further ado, let's get into it!
 
 
 
-=== An Overview of Our Deployment Procedure
-
-Over the next three chapters, I'm going to go through _a_ deployment procedure.
-It isn't meant to be the _perfect_ deployment procedure,
-so please don't take it as being best practice,
-or a recommendation--it's meant to be an illustration,
-to show the kinds of issues involved in putting code into production,
-and where testing fits in.
-
-
-**This chapter: Containerizing our software**
-
-* Adapt our FTs so they can run against a container.
-
-* Build a minimal Dockerfile with everything we need to run our site.
-
-* Learn how to build and run a container on our machine.
-
-* Get a first cut of our code up and running inside Docker,
-  with passing tests.
-
-
-**<<chapter_10_production_readiness,Next chapter>>: Moving to a production-ready configuration**
-
-* Gradually, incrementally change the container configuration
-  to make it production-ready.
-
-* Regularly re-run the FTs to check we didn't break anything.
-
-* Address issues to do with the database, static files, secrets, and so on.
-
-
-**<<chapter_11_server_prep,Third chapter>>: Preparing our staging server and deployment tools**
-
-* We'll set up a "staging":footnote:[
-Some people prefer the term pre-prod or test environment.
-It's all the same idea.]
-server, using the same infrastructure a we plan to use for production.
-* Set up a real domain name and point it at this server
-* Install Ansible and flush out any networking issues
-
-
-**<<chapter_12_ansible,Final chapter>>: Preparing our staging server and deployment tools**
-
-* Gradually build up an Ansible playbook to deploy our containers on a real server.
-
-* Again, use our FTs to check for any problems.
-
-* Learn how to SSH in to the server to debug things,
-  where to find logs and other useful information.
-
-* Confidently deploy to production once we have a working deployment script for staging.
-
-
-
-=== TDD and Docker vs the Danger Areas of Deployment
-
-Hopefully you can start to see how the combination of TDD, Docker, Staging,
-and automation are going to help minimise the risk of the various "Danger Areas".
-
-* Containers will act as mini-servers
-  letting us flush out issues with dependencies, static files, and so on.
-  A key advantage is that they'll give us a way of getting faster feedback cycles,
-  because we can spin them up locally very quickly, and make changes quickly.
-
-* Our containers will package up both our Python and system dependencies,
-  including a production-ready web server and static files system.
-  as well as many production settings and configuration differences.
-  This minimises the difference between what we can test locally,
-  and what we will have on our servers.
-
-* Our FTs mean that we'll have a fully automated way of checking
-  that everything works.
-
-* Later, when we deploy our containers to a staging server,
-  we can run the FTs against that too.
-  It'll be slightly slower and might involve some fiddly compromises,
-  but it'll give us one more layer of reassurance.
-
-* Finally, by fully automating container creation and deployment,
-  and by testing the end results of both these things,
-  we maximise reproducibility, thus minimising the risk of deployment to production.
-
-
 === As Always, Start with a Test
 
 ((("environment variables")))
-Let's adapt our functional tests slightly
+Let's adapt our functional tests
 so that they can run against a standalone server,
 instead of the one that `LiveServerTestCase` creates for us.
 We'll do it by checking for an environment variable
@@ -496,7 +303,7 @@ Currently, all our code is source code really, so we move everything into _src_
 (we'll be seeing some new files appearing outside _src_ shortly).footnote:[
 A common thing to find outside of the _src_ folder is a folder called _tests_.
 We won't be doing that while we're relying on the standard Django test framework,
-but it's a good thing to do if you're using pytest, for example.]
+but it can be a good thing to do if you're using pytest, for example.]