Moved architecture docs to own file, with fixes

Author: spookylukey

ARCHITECTURE.rst

@@ -0,0 +1,116 @@
+Architecture of fluent-compiler
+-------------------------------
+
+The following is a brief, very high-level overview of what fluent-compiler does
+and the various layers.
+
+Our basic strategy is that we take an FTL file like this::
+
+   hello-user = Hello { $username }!
+
+   welcome = { hello-user } Welcome to { -app-name }.
+
+   -app-name = Acme CMS
+
+
+and compile it to Python functions, something roughly like this:
+
+.. code-block:: python
+
+   def hello_user(args, errors):
+       try:
+           username = args['username']
+       except KeyError:
+           username = "username"
+           errors.append(FluentReferenceError("Unknown external: username"))
+       return f"Hello {username}"
+
+
+   def welcome(args, errors):
+       return f"{hello_user(args, errors)} Welcome to Acme CMS."
+
+
+We then need to store these message functions in some dictionary-like object,
+to allow us to call them.
+
+.. code-block:: python
+
+   message_functions = {
+       'hello-user': hello_user,
+       'welcome': welcome,
+   }
+
+To actually format a message we have to do something like:
+
+.. code-block:: python
+
+   errors = []
+   formatted_message = message_functions['hello-user']({'username': 'guest'}, errors)
+   return formatted_message, errors
+
+Note a few things:
+
+* Each message becomes a Python function.
+* Message references are handled by calling other message functions.
+* We do lots of optimizations at compile time to heavily simplify the
+  expressions that are evaluated at runtime, including things like inlining
+  terms.
+* We have to handle possible errors in accordance with the Fluent philosophy.
+  Where possible we detect errors at compile time, in addition to the runtime
+  handling shown above.
+
+We do not, in fact, generate Python code as a string, but instead generate AST
+which we can convert to executable Python functions using the builtin functions
+`compile <https://docs.python.org/3/library/functions.html#compile>`_ and `exec
+<https://docs.python.org/3/library/functions.html#exec>`_.
+
+Layers
+~~~~~~
+
+The highest level code, which can be used as an entry point by users, is in
+``fluent_compiler.bundle``. The interface provided here, however, is meant
+mainly for demonstration purposes, since it is expected that in many
+circumstances the next level down will be used. For example, `django-ftl
+<https://github.com/django-ftl/django-ftl>`_ by-passes this module and uses the
+next layer down.
+
+The next layer is ``fluent_compiler.compiler``, which handles actual
+compilation, converting FTL expressions (i.e. FTL AST nodes) into Python code.
+The bulk of the FTL specific logic is found here. See especially the comments
+on ``compile_expr``.
+
+For generating Python code, it uses the classes provided by the
+``fluent_compiler.codegen`` module. These are simplified versions of various
+Python constructs, with an interface that makes it easy for the ``compiler``
+module to construct correct code without worrying about lower level details.
+
+The classes in the ``codegen`` module eventually need to produce AST objects
+that can be passed to Python’s builtin `compile
+<https://docs.python.org/3/library/functions.html?highlight=compile#compile>`_
+function. The stdlib `ast <https://docs.python.org/3/library/ast.html>`_ module
+has incompatible differences between different Python versions, so we abstract
+over these in ``fluent_compiler.ast_compat`` which allows the ``codegen`` module
+to almost entirely ignore the differences in AST for different Python.
+
+In addition to these modules, there are some runtime functions and types that
+are needed by the generated Python code, found in ``fluent_compiler.runtime``.
+
+The ``fluent_compiler.types`` module contains types for handling number/date
+formatting - these are used directly by users of ``fluent_compiler``, as well as
+internally for implementing things like the ``NUMBER`` and ``DATETIME`` builtin
+FTL functions.
+
+Other related level classes for the user are provided in
+``fluent_compiler.resource`` and ``fluent_compiler.escapers``.
+
+Tests
+~~~~~
+
+The highest level tests are in ``tests/format/``. These are essentially
+functional tests that ensure we produce correct output at runtime.
+
+In addition we have many tests of the lower layers of code. These include
+a lot of tests for our optimizations, many of which work at the level of
+examining the generated Python code.
+
+We also have benchmarking tests in ``tools``.

CONTRIBUTING.rst

@@ -47,120 +47,3 @@ Please submit fixes and features by:
 
 For new features it is often better to open an issue first to see if we agree
 that the feature is a good idea, before spending a lot of time implementing it.
-
-Architecture
-------------
-
-The following is a brief, very high-level overview of what fluent-compiler does
-and the various layers.
-
-Our basic strategy is that we take an FTL file like this::
-
-   hello-user = Hello { $username }!
-
-   -app-name = Acme CMS
-
-   welcome = { hello-user } Welcome to { -app-name }.
-
-
-and compile it to Python functions, something roughly like this:
-
-.. code-block:: python
-
-   def hello_user(args, errors):
-       try:
-           username = args['username']
-       except KeyError:
-           username = "username"
-           errors.append(FluentReferenceError("Unknown external: username"))
-       return f"Hello {username}"
-
-
-   def welcome(args, errors):
-       return f"{hello_user(args, errors)} Welcome to Acme CMS."
-
-
-We then need to store these message functions in some dictionary-like object,
-to allow us to call them.
-
-.. code-block:: python
-
-   message_functions = {
-       'hello-user': hello_user,
-       'welcome': welcome,
-   }
-
-To actually format a message with name ``'msg-name'`` and external arguments
-``args``, we have to do something like:
-
-.. code-block:: python
-
-   errors = []
-   formatted_message = message_functions['msg-name'](args, errors)
-   return formatted_message, errors
-
-Note a few things:
-
-* Each message becomes a Python function.
-* Message references are handled by calling other message functions.
-* We do lots of optimizations at compile time to heavily simplify the
-  expressions that are evaluated at runtime, including things like inlining
-  terms.
-* We have to handle possible errors in accordance with the Fluent philosophy.
-  Where possible we detect errors at compile time, in addition to the runtime
-  handling shown above.
-
-We do not, in fact, generate Python code as a string, but instead generate AST
-which we can convert to executable Python functions using the builtin functions
-`compile <https://docs.python.org/3/library/functions.html#compile>`_ and `exec
-<https://docs.python.org/3/library/functions.html#exec>`_.
-
-Layers
-~~~~~~
-
-The highest level code, which can be used as an entry point by users, is in
-``fluent_compiler.bundle``. The interface provided here, however, is meant
-mainly for demonstration purposes, since it is expected that in many
-circumstances the next level down will be used. For example, `django-ftl
-<https://github.com/django-ftl/django-ftl>`_ by-passes this module and uses the
-next layer down.
-
-The next layer is ``fluent_compiler.compiler``, which handles actual
-compilation, converting FTL expressions (i.e. FTL AST nodes) into Python code.
-The bulk of the FTL specific logic is found here. See especially the comments
-on ``compile_expr``.
-
-For generating Python code, it uses the classes provided by the
-``fluent_compiler.codegen`` module. These are simplified versions of various
-Python constructs, with an interface that makes it easy for the ``compiler``
-module to construct correct code without worrying about lower level details.
-
-The classes in the ``codegen`` module eventually need to produce AST objects
-that can be passed Python’s builtin ``compile`` function. The stdlib `ast
-<https://docs.python.org/3/library/ast.html>`_ module has incompatible
-differences between different Python versions, so we abstract over these in
-``fluent_compiler.ast_compat`` which allows the ``codegen`` module to almost
-entirely ignore the differences in AST for different Python.
-
-In addition to these modules, there are some runtime functions and types that
-are needed by the generated Python code, found in ``fluent_compiler.runtime``.
-
-The ``fluent_compiler.types`` module contains types for handling number/date
-formatting - these are used directly by users of ``fluent_compiler``, as well as
-internally for implementing things like the ``NUMBER`` and ``DATETIME`` builtin
-FTL functions.
-
-Other related level classes for the user are provided in
-``fluent_compiler.resource`` and ``fluent_compiler.escapers``.
-
-Tests
-~~~~~
-
-The highest level tests are in ``tests/format/``. These are essentially
-functional tests that ensures we produce correct output at runtime.
-
-In addition we have many tests of the lower layers of code. These include
-a lot of tests for our optimizations, many of which work at the level of
-examining the generated Python code.
-
-We also have benchmarking tests in ``tools``.

README.rst

@@ -58,7 +58,7 @@ incompatible changes easily.
 
 See the `issues list <https://github.com/django-ftl/fluent-compiler/issues>`_
 for planned features, and `CONTRIBUTING.rst <CONTRIBUTING.rst>`_ for information
-about how to contribute.
+about how to contribute, and the `architecture docs <ARCHITECTURE.rst>`_
 
 Background
 ----------

docs/contributing.rst

@@ -1 +1,3 @@
 .. include:: ../CONTRIBUTING.rst
+
+.. include:: ../ARCHITECTURE.rst

docs/index.rst

@@ -1,7 +1,3 @@
-.. fluent_compiler documentation master file, created by
-   sphinx-quickstart on Thu Jan 24 15:45:23 2019.
-   You can adapt this file completely to your liking, but it should at least
-   contain the root `toctree` directive.
 
 Welcome to fluent_compiler's documentation!
 ===========================================

src/fluent_compiler/compiler.py

@@ -1,5 +1,5 @@
 # The heart of the FTL -> Python compiler. See the architecture docs in
-# CONTRIBUTING.rst for the big picture, and comments on compile_expr below.
+# ARCHITECTURE.rst for the big picture, and comments on compile_expr below.
 from __future__ import absolute_import, unicode_literals
 
 import contextlib