Merge pull request #504 from minrk/return_exceptions

Add AsyncResult.get(return_exceptions=True), map(return_exceptions=True)

Author: minrk

docs/source/conf.py

@@ -117,7 +117,7 @@
 
 # The reST default role (used for this markup: `text`) to use for all
 # documents.
-# default_role = None
+default_role = 'literal'
 
 # If true, '()' will be appended to :func: etc. cross-reference text.
 # add_function_parentheses = True

docs/source/reference/details.rst

@@ -32,7 +32,7 @@ The following will fail:
 .. sourcecode:: ipython
 
     In [3]: A = numpy.zeros(2)
-    
+
     In [4]: def setter(a):
        ...:   a[0]=1
        ...:   return a
@@ -49,7 +49,7 @@ The :attr:`ndarray.flags.writeable` flag will tell you if you can write to an ar
 .. sourcecode:: ipython
 
     In [3]: A = numpy.zeros(2)
-    
+
     In [4]: def setter(a):
        ...:     """only copy read-only arrays"""
        ...:     if not a.flags.writeable:
@@ -59,7 +59,7 @@ The :attr:`ndarray.flags.writeable` flag will tell you if you can write to an ar
 
     In [5]: rc[0].apply_sync(setter, A)
     Out[5]: array([ 1.,  0.])
-    
+
     # note that results will also be read-only:
     In [6]: _.flags.writeable
     Out[6]: False
@@ -73,14 +73,14 @@ checking and waiting for 0MQ to finish with a buffer.
 .. sourcecode:: ipython
 
     In [5]: A = numpy.random.random((1024,1024))
-    
+
     In [6]: view.track=True
-    
+
     In [7]: ar = view.apply_async(lambda x: 2*x, A)
-    
+
     In [8]: ar.sent
     Out[8]: False
-    
+
     In [9]: ar.wait_on_send() # blocks until sent is True
 
 
@@ -114,7 +114,7 @@ An example of a function that uses a closure:
             # inner will have a closure
             return a
         return inner
-    
+
     f1 = f(1)
     f2 = f(2)
     f1() # returns 1
@@ -150,7 +150,7 @@ of the global namespace:
 .. sourcecode:: ipython
 
     In [10]: view.apply(lambda : a)
-    
+
     # is equivalent to
     In [11]: view.pull('a')
 
@@ -240,10 +240,11 @@ timeout : float/int or None
 execute and run
 ---------------
 
-For executing strings of Python code, :class:`DirectView`s also provide an :meth:`execute` and
-a :meth:`run` method, which rather than take functions and arguments, take Python strings.
+
+For executing strings of Python code, :class:`~.DirectView` s also provide an :meth:`~.DirectView.execute` and
+a :meth:`~.DirectView.run` method, which rather than take functions and arguments, take Python strings.
 `execute` takes a string of Python code to execute, and sends it to the Engine(s). `run`
-is the same as `execute`, but for a *file* rather than a string. It is a wrapper that
+is the same as `execute`, but for a *filename* rather than a string. It is a wrapper that
 does something very similar to ``execute(open(f).read())``.
 
 .. note::
@@ -292,7 +293,7 @@ the index directly to the client's :attr:`ids` list, so:
     # negative index
     In [2]: rc[-1]
     Out[2]: <DirectView 3>
-    
+
     # or slicing:
     In [3]: rc[::2]
     Out[3]: <DirectView [0,2]>
@@ -303,11 +304,11 @@ are always the same as:
 
     In [2]: rc[rc.ids[-1]]
     Out[2]: <DirectView 3>
-    
+
     In [3]: rc[rc.ids[::2]]
     Out[3]: <DirectView [0,2]>
 
-Also note that the slice is evaluated at the time of construction of the DirectView, so the 
+Also note that the slice is evaluated at the time of construction of the DirectView, so the
 targets will not change over time if engines are added/removed from the cluster.
 
 Execution via DirectView
@@ -320,7 +321,7 @@ For instance, to get the process ID of all your engines:
 .. sourcecode:: ipython
 
     In [5]: import os
-    
+
     In [6]: dview.apply_sync(os.getpid)
     Out[6]: [1354, 1356, 1358, 1360]
 
@@ -329,7 +330,7 @@ Or to see the hostname of the machine they are on:
 .. sourcecode:: ipython
 
     In [5]: import socket
-    
+
     In [6]: dview.apply_sync(socket.gethostname)
     Out[6]: ['tesla', 'tesla', 'edison', 'edison', 'edison']
 
@@ -407,23 +408,25 @@ AsyncResults
 ------------
 
 Our primary representation of the results of remote execution is the :class:`~.AsyncResult`
-object, based on the object of the same name in the built-in :mod:`multiprocessing.pool`
-module. Our version provides a superset of that interface.
+object, based on the object of the same name in the built-in :py:mod:`multiprocessing.pool`
+module.
+Our version provides a superset of that interface,
+and starting in 6.0 is a subclass of :class:`concurrent.futures.Future`.
 
-The basic principle of the AsyncResult is the encapsulation of one or more results not yet completed.  Execution methods (including data movement, such as push/pull) will all return
-AsyncResults when `block=False`.
+The basic principle of the AsyncResult is the encapsulation of one or more results not yet completed.
+Execution methods (including data movement, such as push/pull) will all return AsyncResults when `block=False`.
 
 The mp.pool.AsyncResult interface
 ---------------------------------
 
-The basic interface of the AsyncResult is exactly that of the AsyncResult in :mod:`multiprocessing.pool`, and consists of four methods:
+The basic interface of the AsyncResult is exactly that of the AsyncResult in :py:mod:`multiprocessing.pool`, and consists of four methods:
 
 .. AsyncResult spec directly from docs.python.org
 
 .. class:: AsyncResult
 
    The stdlib AsyncResult spec
-   
+
    .. method:: wait([timeout])
 
       Wait until the result is available or until *timeout* seconds pass. This
@@ -592,7 +595,7 @@ clear
 
 shutdown
 
-    You can also instruct engines (and the Controller) to terminate from a Client.  This 
+    You can also instruct engines (and the Controller) to terminate from a Client.  This
     can be useful when a job is finished, since you can shutdown all the processes with a
     single command.
 
@@ -632,15 +635,25 @@ and many parallel execution tools in Python, such as the built-in
 :py:class:`multiprocessing.Pool` object provide implementations of `map`. All View objects
 provide a :meth:`map` method as well, but the load-balanced and direct implementations differ.
 
-Views' map methods can be called on any number of sequences, but they can also take the `block`
-and `bound` keyword arguments, just like :meth:`~client.apply`, but *only as keywords*.
+Views' map methods can be called on any number of sequences,
+but they can also take keyword arguments to influence how the work is distributed.
+What keyword arguments are available depends on the view being used.
 
-.. sourcecode:: python
 
-    dview.map(*sequences, block=None)
+
+.. class:: ipyparallel.DirectView
+   :noindex:
+
+   .. automethod:: map
+
+.. class:: ipyparallel.LoadBalancedView
+   :noindex:
+
+   .. automethod:: map
+
+   .. automethod:: imap
 
 
-* iter, map_async, reduce
 
 Decorators and RemoteFunctions
 ==============================

docs/source/tutorial/asyncresult.rst

@@ -4,31 +4,36 @@
 The AsyncResult object
 ======================
 
-In non-blocking mode, :meth:`apply` submits the command to be executed and
-then returns a :class:`~.AsyncResult` object immediately. The
-AsyncResult object gives you a way of getting a result at a later
+In non-blocking mode, :meth:`~.View.apply`, :meth:`~.View.map`, and friends
+submit the command to be executed and then return an :class:`~.AsyncResult` object immediately.
+The AsyncResult object gives you a way of getting a result at a later
 time through its :meth:`get` method, but it also collects metadata
 on execution.
 
 
-Beyond multiprocessing's AsyncResult
+Beyond stdlib AsyncResult and Future
 ====================================
 
-.. Note::
+The :class:`AsyncResult` is a subclass of :py:class:`concurrent.futures.Future`.
+This means it can be integrated into existing async workflows,
+with e.g. :py:func:`asyncio.wrap_future`.
+It also extends the :py:class:`~.multiprocessing.AsyncResult` API.
 
-    The :class:`~.AsyncResult` object provides a superset of the interface in
-    :py:class:`multiprocessing.pool.AsyncResult`.  See the
-    `official Python documentation <https://docs.python.org/library/multiprocessing#multiprocessing.pool.AsyncResult>`_
-    for more on the basics of this interface.
 
-Our AsyncResult objects add a number of convenient features for working with
-parallel results, beyond what is provided by the original AsyncResult.
+.. seealso::
+
+    - :py:class:`multiprocessing.AsyncResult` API
+    - :py:class:`concurrent.futures.Future` API
+
+In addition to these common features,
+our AsyncResult objects add a number of convenient methods for working with parallel results,
+beyond what is provided by the standard library classes on which they are based.
 
 
 get_dict
 --------
 
-First, is :meth:`.AsyncResult.get_dict`, which pulls results as a dictionary
+:meth:`.AsyncResult.get_dict` pulls results as a dictionary,
 keyed by engine_id, rather than a flat list.  This is useful for quickly
 coordinating or distributing information about all of the engines.
 
@@ -48,7 +53,7 @@ as in IPython's :file:`examples/parallel/interengine` examples.
 Metadata
 ========
 
-ipyparallel tracks some metadata about the tasks, which is stored
+IPython Parallel tracks some metadata about the tasks, which is stored
 in the :attr:`.Client.metadata` dict.  The AsyncResult object gives you an
 interface for this information as well, including timestamps stdout/err,
 and engine IDs.
@@ -111,9 +116,9 @@ That is to say, if you treat an AsyncMapResult as if it were a list of your actu
 results, it should behave as you would expect, with the only difference being
 that you can start iterating through the results before they have even been computed.
 
-This lets you do a dumb version of map/reduce with the builtin Python functions,
+This lets you do a simple version of map/reduce with the builtin Python functions,
 and the only difference between doing this locally and doing it remotely in parallel
-is using the asynchronous view.map instead of the builtin map.
+is using the asynchronous ``view.map`` instead of the builtin ``map``.
 
 
 Here is a simple one-line RMS (root-mean-square) implemented with Python's builtin map/reduce.