Changeset 388 for python/vendor/current/Doc/howto

python/vendor/current/Doc/howto/cporting.rst

-              r2
+              r388
 .. highlightlang:: c
+********************************
+Porting Extension Modules to 3.0
+********************************
+.. _cporting-howto:
+*************************************
+Porting Extension Modules to Python 3
+*************************************
 :author: Benjamin Peterson
 …
 .. topic:: Abstract
+   Although changing the C-API was not one of Python 3.0's objectives, the many
+   Python level changes made leaving 2.x's API intact impossible.  In fact, some
+   changes such as :func:`int` and :func:`long` unification are more obvious on
+   the C level.  This document endeavors to document incompatibilities and how
+   they can be worked around.
+   Although changing the C-API was not one of Python 3's objectives,
+   the many Python-level changes made leaving Python 2's API intact
+   impossible.  In fact, some changes such as :func:`int` and
+   :func:`long` unification are more obvious on the C level.  This
+   document endeavors to document incompatibilities and how they can
+   be worked around.
 …
 =======================
 The easiest way to compile only some code for 3.0 is to check if
 :cmacro:`PY_MAJOR_VERSION` is greater than or equal to 3. ::
+The easiest way to compile only some code for Python 3 is to check
+if :c:macro:`PY_MAJOR_VERSION` is greater than or equal to 3. ::
    #if PY_MAJOR_VERSION >= 3
 …
 ======================
 Python 3.0 merged together some types with similar functions while cleanly
+Python 3 merged together some types with similar functions while cleanly
 separating others.
 …
 Python 3.0's :func:`str` (``PyString_*`` functions in C) type is equivalent to
+.x's :func:`unicode` (``PyUnicode_*``).  The old 8-bit string type has become
 :func:`bytes`.  Python 2.6 and later provide a compatibility header,
+Python 3's :func:`str` (``PyString_*`` functions in C) type is equivalent to
+Python 2's :func:`unicode` (``PyUnicode_*``).  The old 8-bit string type has
+become :func:`bytes`.  Python 2.6 and later provide a compatibility header,
 :file:`bytesobject.h`, mapping ``PyBytes`` names to ``PyString`` ones.  For best
 compatibility with 3.0, :ctype:`PyUnicode` should be used for textual data and
 :ctype:`PyBytes` for binary data.  It's also important to remember that
 :ctype:`PyBytes` and :ctype:`PyUnicode` in 3.0 are not interchangeable like
 :ctype:`PyString` and :ctype:`PyString` are in 2.x.  The following example shows
+best practices with regards to :ctype:`PyUnicode`, :ctype:`PyString`, and
 :ctype:`PyBytes`. ::
+compatibility with Python 3, :c:type:`PyUnicode` should be used for textual data and
+:c:type:`PyBytes` for binary data.  It's also important to remember that
+:c:type:`PyBytes` and :c:type:`PyUnicode` in Python 3 are not interchangeable like
+:c:type:`PyString` and :c:type:`PyUnicode` are in Python 2.  The following example
+shows best practices with regards to :c:type:`PyUnicode`, :c:type:`PyString`,
+and :c:type:`PyBytes`. ::
    #include "stdlib.h"
 …
 --------------------
+In Python 3.0, there is only one integer type.  It is called :func:`int` on the
+Python level, but actually corresponds to 2.x's :func:`long` type.  In the
+C-API, ``PyInt_*`` functions are replaced by their ``PyLong_*`` neighbors.  The
+best course of action here is using the ``PyInt_*`` functions aliased to
+``PyLong_*`` found in :file:`intobject.h`.  The abstract ``PyNumber_*`` APIs
+can also be used in some cases. ::
+   #include "Python.h"
+   #include "intobject.h"
+   static PyObject *
+   add_ints(PyObject *self, PyObject *args) {
+       int one, two;
+       PyObject *result;
+       if (!PyArg_ParseTuple(args, "ii:add_ints", &one, &two))
+           return NULL;
+       return PyInt_FromLong(one + two);
+   }
+Python 3 has only one integer type, :func:`int`.  But it actually
+corresponds to Python 2's :func:`long` type--the :func:`int` type
+used in Python 2 was removed.  In the C-API, ``PyInt_*`` functions
+are replaced by their ``PyLong_*`` equivalents.
 …
 ===============================
+Python 3.0 has a revamped extension module initialization system.  (See PEP
+:pep:`3121`.)  Instead of storing module state in globals, they should be stored
+in an interpreter specific structure.  Creating modules that act correctly in
+both 2.x and 3.0 is tricky.  The following simple example demonstrates how. ::
+Python 3 has a revamped extension module initialization system.  (See
+:pep:`3121`.)  Instead of storing module state in globals, they should
+be stored in an interpreter specific structure.  Creating modules that
+act correctly in both Python 2 and Python 3 is tricky.  The following
+simple example demonstrates how. ::
    #include "Python.h"
 …
+CObject replaced with Capsule
+=============================
+The :c:type:`Capsule` object was introduced in Python 3.1 and 2.7 to replace
+:c:type:`CObject`.  CObjects were useful,
+but the :c:type:`CObject` API was problematic: it didn't permit distinguishing
+between valid CObjects, which allowed mismatched CObjects to crash the
+interpreter, and some of its APIs relied on undefined behavior in C.
+(For further reading on the rationale behind Capsules, please see :issue:`5630`.)
+If you're currently using CObjects, and you want to migrate to 3.1 or newer,
+you'll need to switch to Capsules.
+:c:type:`CObject` was deprecated in 3.1 and 2.7 and completely removed in
+Python 3.2.  If you only support 2.7, or 3.1 and above, you
+can simply switch to :c:type:`Capsule`.  If you need to support Python 3.0,
+or versions of Python earlier than 2.7,
+you'll have to support both CObjects and Capsules.
+(Note that Python 3.0 is no longer supported, and it is not recommended
+for production use.)
+The following example header file :file:`capsulethunk.h` may
+solve the problem for you.  Simply write your code against the
+:c:type:`Capsule` API and include this header file after
+:file:`Python.h`.  Your code will automatically use Capsules
+in versions of Python with Capsules, and switch to CObjects
+when Capsules are unavailable.
+:file:`capsulethunk.h` simulates Capsules using CObjects.  However,
+:c:type:`CObject` provides no place to store the capsule's "name".  As a
+result the simulated :c:type:`Capsule` objects created by :file:`capsulethunk.h`
+behave slightly differently from real Capsules.  Specifically:
+  * The name parameter passed in to :c:func:`PyCapsule_New` is ignored.
+  * The name parameter passed in to :c:func:`PyCapsule_IsValid` and
+    :c:func:`PyCapsule_GetPointer` is ignored, and no error checking
+    of the name is performed.
+  * :c:func:`PyCapsule_GetName` always returns NULL.
+  * :c:func:`PyCapsule_SetName` always raises an exception and
+    returns failure.  (Since there's no way to store a name
+    in a CObject, noisy failure of :c:func:`PyCapsule_SetName`
+    was deemed preferable to silent failure here.  If this is
+    inconvenient, feel free to modify your local
+    copy as you see fit.)
+You can find :file:`capsulethunk.h` in the Python source distribution
+as :source:`Doc/includes/capsulethunk.h`.  We also include it here for
+your convenience:
+.. literalinclude:: ../includes/capsulethunk.h
 Other options
 =============
 …
 If you are writing a new extension module, you might consider `Cython
 <http://www.cython.org>`_.  It translates a Python-like language to C.  The
 extension modules it creates are compatible with Python 3.x and 2.x.
+extension modules it creates are compatible with Python 3 and Python 2.

python/vendor/current/Doc/howto/curses.rst

-              r2
+              r388
 messed up when the application dies without restoring the terminal to its
 previous state.  In Python this commonly happens when your code is buggy and
 raises an uncaught exception.  Keys are no longer be echoed to the screen when
+raises an uncaught exception.  Keys are no longer echoed to the screen when
 you type them, for example, which makes using the shell difficult.
 …
 window of a given size, returning the new window object. ::
    begin_x = 20 ; begin_y = 7
    height = 5 ; width = 40
+   begin_x = 20; begin_y = 7
+   height = 5; width = 40
    win = curses.newwin(height, width, begin_y, begin_x)
 …
    for y in range(0, 100):
        for x in range(0, 100):
+           try: pad.addch(y,x, ord('a') + (x*x+y*y) % 26 )
+           except curses.error: pass
+           try:
+               pad.addch(y,x, ord('a') + (x*x+y*y) % 26)
+           except curses.error:
+               pass
    #  Displays a section of the pad in the middle of the screen
    pad.refresh( 0,0, 5,5, 20,75)
+   pad.refresh(0,0, 5,5, 20,75)
 The :func:`refresh` call displays a section of the pad in the rectangle
 …
 attribute for each cell on the screen.
 An attribute is a integer, each bit representing a different attribute.  You can
+An attribute is an integer, each bit representing a different attribute.  You can
 try to display text with multiple attribute bits set, but curses doesn't
 guarantee that all the possible combinations are available, or that they're all
 …
    stdscr.refresh()
 The curses library also supports color on those terminals that provide it, The
+The curses library also supports color on those terminals that provide it. The
 most common such terminal is probably the Linux console, followed by color
 xterms.
 …
 An example, which displays a line of text using color pair 1::
    stdscr.addstr( "Pretty text", curses.color_pair(1) )
+   stdscr.addstr("Pretty text", curses.color_pair(1))
    stdscr.refresh()
 …
 with::
    stdscr.addstr(0,0, "RED ALERT!", curses.color_pair(1) )
+   stdscr.addstr(0,0, "RED ALERT!", curses.color_pair(1))
 Very fancy terminals can change the definitions of the actual colors to a given
 …
    while 1:
        c = stdscr.getch()
+       if c == ord('p'): PrintDocument()
+       elif c == ord('q'): break  # Exit the while()
+       elif c == curses.KEY_HOME: x = y = 0
+       if c == ord('p'):
+           PrintDocument()
+       elif c == ord('q'):
+           break  # Exit the while()
+       elif c == curses.KEY_HOME:
+           x = y = 0
 The :mod:`curses.ascii` module supplies ASCII class membership functions that
 …
 The ncurses FAQ: http://invisible-island.net/ncurses/ncurses.faq.html

python/vendor/current/Doc/howto/doanddont.rst

-              r2
+              r388
 valid, no more than having a smart lawyer makes a man innocent. Do not use it
 like that ever. Even in versions where it was accepted, it made the function
 execution slower, because the compiler could not be certain which names are
 local and which are global. In Python 2.1 this construct causes warnings, and
+execution slower, because the compiler could not be certain which names were
+local and which were global. In Python 2.1 this construct causes warnings, and
 sometimes even errors.
 …
 some module grows additional functions or classes.
 One of the most awful question asked on the newsgroup is why this code::
+One of the most awful questions asked on the newsgroup is why this code::
    f = open("www")
 …
 This is a "don't" which is much weaker than the previous "don't"s but is still
 something you should not do if you don't have good reasons to do that. The
 reason it is usually bad idea is because you suddenly have an object which lives
+reason it is usually a bad idea is because you suddenly have an object which lives
 in two separate namespaces. When the binding in one namespace changes, the
 binding in the other will not, so there will be a discrepancy between them. This
 …
 Python has the ``except:`` clause, which catches all exceptions. Since *every*
+error in Python raises an exception, this makes many programming errors look
+like runtime problems, and hinders the debugging process.
+The following code shows a great example::
+error in Python raises an exception, using ``except:`` can make many
+programming errors look like runtime problems, which hinders the debugging
+process.
+The following code shows a great example of why this is bad::
    try:
 …
        sys.exit("could not open file!")
+The second line triggers a :exc:`NameError` which is caught by the except
+clause. The program will exit, and you will have no idea that this has nothing
+to do with the readability of ``"file"``.
+The example above is better written ::
+The second line triggers a :exc:`NameError`, which is caught by the except
+clause. The program will exit, and the error message the program prints will
+make you think the problem is the readability of ``"file"`` when in fact
+the real error has nothing to do with ``"file"``.
+A better way to write the above is ::
    try:
        foo = opne("file") # will be changed to "open" as soon as we run it
+       foo = opne("file")
    except IOError:
        sys.exit("could not open file")
+There are some situations in which the ``except:`` clause is useful: for
+example, in a framework when running callbacks, it is good not to let any
+callback disturb the framework.
+When this is run, Python will produce a traceback showing the :exc:`NameError`,
+and it will be immediately apparent what needs to be fixed.
+.. index:: bare except, except; bare
+Because ``except:`` catches *all* exceptions, including :exc:`SystemExit`,
+:exc:`KeyboardInterrupt`, and :exc:`GeneratorExit` (which is not an error and
+should not normally be caught by user code), using a bare ``except:`` is almost
+never a good idea.  In situations where you need to catch all "normal" errors,
+such as in a framework that runs callbacks, you can catch the base class for
+all normal exceptions, :exc:`Exception`.  Unfortunately in Python 2.x it is
+possible for third-party code to raise exceptions that do not inherit from
+:exc:`Exception`, so in Python 2.x there are some cases where you may have to
+use a bare ``except:`` and manually re-raise the exceptions you don't want
+to catch.
 …
        return open(file).readline()
+Consider the case the file gets deleted between the time the call to
+:func:`os.path.exists` is made and the time :func:`open` is called. That means
+the last line will throw an :exc:`IOError`. The same would happen if *file*
+exists but has no read permission. Since testing this on a normal machine on
+existing and non-existing files make it seem bugless, that means in testing the
+results will seem fine, and the code will get shipped. Then an unhandled
+:exc:`IOError` escapes to the user, who has to watch the ugly traceback.
+Here is a better way to do it. ::
+Consider the case where the file gets deleted between the time the call to
+:func:`os.path.exists` is made and the time :func:`open` is called. In that
+case the last line will raise an :exc:`IOError`.  The same thing would happen
+if *file* exists but has no read permission.  Since testing this on a normal
+machine on existent and non-existent files makes it seem bugless, the test
+results will seem fine, and the code will get shipped.  Later an unhandled
+:exc:`IOError` (or perhaps some other :exc:`EnvironmentError`) escapes to the
+user, who gets to watch the ugly traceback.
+Here is a somewhat better way to do it. ::
    def get_status(file):
        try:
            return open(file).readline()
        except (IOError, OSError):
            print "file not found"
+       except EnvironmentError as err:
+           print "Unable to open file: {}".format(err)
            sys.exit(1)
+In this version, \*either\* the file gets opened and the line is read (so it
+works even on flaky NFS or SMB connections), or the message is printed and the
+application aborted.
+Still, :func:`get_status` makes too many assumptions --- that it will only be
+used in a short running script, and not, say, in a long running server. Sure,
+the caller could do something like ::
+In this version, *either* the file gets opened and the line is read (so it
+works even on flaky NFS or SMB connections), or an error message is printed
+that provides all the available information on why the open failed, and the
+application is aborted.
+However, even this version of :func:`get_status` makes too many assumptions ---
+that it will only be used in a short running script, and not, say, in a long
+running server. Sure, the caller could do something like ::
    try:
 …
        status = None
+So, try to make as few ``except`` clauses in your code --- those will usually be
+a catch-all in the :func:`main`, or inside calls which should always succeed.
+So, the best version is probably ::
+But there is a better way.  You should try to use as few ``except`` clauses in
+your code as you can --- the ones you do use will usually be inside calls which
+should always succeed, or a catch-all in a main function.
+So, an even better version of :func:`get_status()` is probably ::
    def get_status(file):
        return open(file).readline()
 The caller can deal with the exception if it wants (for example, if it  tries
+The caller can deal with the exception if it wants (for example, if it tries
 several files in a loop), or just let the exception filter upwards to *its*
 caller.
+The last version is not very good either --- due to implementation details, the
+file would not be closed when an exception is raised until the handler finishes,
+and perhaps not at all in non-C implementations (e.g., Jython). ::
+But the last version still has a serious problem --- due to implementation
+details in CPython, the file would not be closed when an exception is raised
+until the exception handler finishes; and, worse, in other implementations
+(e.g., Jython) it might not be closed at all regardless of whether or not
+an exception is raised.
+The best version of this function uses the ``open()`` call as a context
+manager, which will ensure that the file gets closed as soon as the
+function returns::
    def get_status(file):
+       fp = open(file)
+       try:
+       with open(file) as fp:
            return fp.readline()
-       finally:
-           fp.close()
 …
 :func:`splitext`.
+There are also many useful built-in functions people seem not to be aware of for
+some reason: :func:`min` and :func:`max` can find the minimum/maximum of any
+sequence with comparable semantics, for example, yet many people write their own
+:func:`max`/:func:`min`. Another highly useful function is :func:`reduce`. A
+classical use of :func:`reduce` is something like ::
+   import sys, operator
+   nums = map(float, sys.argv[1:])
+   print reduce(operator.add, nums)/len(nums)
+This cute little script prints the average of all numbers given on the command
+line. The :func:`reduce` adds up all the numbers, and the rest is just some
+pre- and postprocessing.
+On the same note, note that :func:`float`, :func:`int` and :func:`long` all
+accept arguments of type string, and so are suited to parsing --- assuming you
+are ready to deal with the :exc:`ValueError` they raise.
+There are also many useful built-in functions people seem not to be aware of
+for some reason: :func:`min` and :func:`max` can find the minimum/maximum of
+any sequence with comparable semantics, for example, yet many people write
+their own :func:`max`/:func:`min`. Another highly useful function is
+:func:`reduce` which can be used to repeatly apply a binary operation to a
+sequence, reducing it to a single value.  For example, compute a factorial
+with a series of multiply operations::
+   >>> n = 4
+   >>> import operator
+   >>> reduce(operator.mul, range(1, n+1))
+When it comes to parsing numbers, note that :func:`float`, :func:`int` and
+:func:`long` all accept string arguments and will reject ill-formed strings
+by raising an :exc:`ValueError`.

python/vendor/current/Doc/howto/functional.rst

-              r2
+              r388
 :Author: A. M. Kuchling
 :Release: 0.31
-(This is a first draft.  Please send comments/error reports/suggestions to
-amk@amk.ca.)
 In this document, we'll take a tour of Python's features suitable for
 …
   variants) and Haskell.
+The designers of some computer languages choose to emphasize one
+particular approach to programming.  This often makes it difficult to
+write programs that use a different approach.  Other languages are
+multi-paradigm languages that support several different approaches.
+Lisp, C++, and Python are multi-paradigm; you can write programs or
+libraries that are largely procedural, object-oriented, or functional
+in all of these languages.  In a large program, different sections
+might be written using different approaches; the GUI might be
+object-oriented while the processing logic is procedural or
+The designers of some computer languages choose to emphasize one particular
+approach to programming.  This often makes it difficult to write programs that
+use a different approach.  Other languages are multi-paradigm languages that
+support several different approaches.  Lisp, C++, and Python are
+multi-paradigm; you can write programs or libraries that are largely
+procedural, object-oriented, or functional in all of these languages.  In a
+large program, different sections might be written using different approaches;
+the GUI might be object-oriented while the processing logic is procedural or
 functional, for example.
 …
 and ``"not in"`` operators also support iterators: ``X in iterator`` is true if
 X is found in the stream returned by the iterator.  You'll run into obvious
 problems if the iterator is infinite; ``max()``, ``min()``, and ``"not in"``
+problems if the iterator is infinite; ``max()``, ``min()``
 will never return, and if the element X never appears in the stream, the
 ``"in"`` operator won't return either.
+``"in"`` and ``"not in"`` operators won't return either.
 Note that you can only go forward in an iterator; there's no way to get the
 …
 List comprehensions and generator expressions (short form: "listcomps" and
 "genexps") are a concise notation for such operations, borrowed from the
 functional programming language Haskell (http://www.haskell.org).  You can strip
+functional programming language Haskell (http://www.haskell.org/).  You can strip
 all the whitespace from a stream of strings with the following code::
 …
-The functional module
----------------------
-Collin Winter's `functional module <http://oakwinter.com/code/functional/>`__
-provides a number of more advanced tools for functional programming. It also
-reimplements several Python built-ins, trying to make them more intuitive to
-those used to functional programming in other languages.
-This section contains an introduction to some of the most important functions in
-``functional``; full documentation can be found at `the project's website
-<http://oakwinter.com/code/functional/documentation/>`__.
-``compose(outer, inner, unpack=False)``
-The ``compose()`` function implements function composition.  In other words, it
-returns a wrapper around the ``outer`` and ``inner`` callables, such that the
-return value from ``inner`` is fed directly to ``outer``.  That is, ::
-    >>> def add(a, b):
-    ...     return a + b
-    ...
-    >>> def double(a):
-    ...     return 2 * a
-    ...
-    >>> compose(double, add)(5, 6)
-is equivalent to ::
-    >>> double(add(5, 6))
-The ``unpack`` keyword is provided to work around the fact that Python functions
-are not always `fully curried <http://en.wikipedia.org/wiki/Currying>`__.  By
-default, it is expected that the ``inner`` function will return a single object
-and that the ``outer`` function will take a single argument. Setting the
-``unpack`` argument causes ``compose`` to expect a tuple from ``inner`` which
-will be expanded before being passed to ``outer``. Put simply, ::
-    compose(f, g)(5, 6)
-is equivalent to::
-    f(g(5, 6))
-while ::
-    compose(f, g, unpack=True)(5, 6)
-is equivalent to::
-    f(*g(5, 6))
-Even though ``compose()`` only accepts two functions, it's trivial to build up a
-version that will compose any number of functions. We'll use ``reduce()``,
-``compose()`` and ``partial()`` (the last of which is provided by both
-``functional`` and ``functools``). ::
-    from functional import compose, partial
-    multi_compose = partial(reduce, compose)
-We can also use ``map()``, ``compose()`` and ``partial()`` to craft a version of
-``"".join(...)`` that converts its arguments to string::
-    from functional import compose, partial
-    join = compose("".join, partial(map, str))
-``flip(func)``
-``flip()`` wraps the callable in ``func`` and causes it to receive its
-non-keyword arguments in reverse order. ::
-    >>> def triple(a, b, c):
-    ...     return (a, b, c)
-    ...
-    >>> triple(5, 6, 7)
-    (5, 6, 7)
-    >>>
-    >>> flipped_triple = flip(triple)
-    >>> flipped_triple(5, 6, 7)
-    (7, 6, 5)
-``foldl(func, start, iterable)``
-``foldl()`` takes a binary function, a starting value (usually some kind of
-'zero'), and an iterable.  The function is applied to the starting value and the
-first element of the list, then the result of that and the second element of the
-list, then the result of that and the third element of the list, and so on.
-This means that a call such as::
-    foldl(f, 0, [1, 2, 3])
-is equivalent to::
-    f(f(f(0, 1), 2), 3)
-``foldl()`` is roughly equivalent to the following recursive function::
-    def foldl(func, start, seq):
-        if len(seq) == 0:
-            return start
-        return foldl(func, func(start, seq[0]), seq[1:])
-Speaking of equivalence, the above ``foldl`` call can be expressed in terms of
-the built-in ``reduce`` like so::
-    reduce(f, [1, 2, 3], 0)
-We can use ``foldl()``, ``operator.concat()`` and ``partial()`` to write a
-cleaner, more aesthetically-pleasing version of Python's ``"".join(...)``
-idiom::
-    from functional import foldl, partial from operator import concat
-    join = partial(foldl, concat, "")
 Revision History and Acknowledgements
 =====================================
 …
 Mertz also wrote a 3-part series of articles on functional programming
 for IBM's DeveloperWorks site; see
+`part 1 <http://www-128.ibm.com/developerworks/library/l-prog.html>`__,
+`part 2 <http://www-128.ibm.com/developerworks/library/l-prog2.html>`__, and
+`part 3 <http://www-128.ibm.com/developerworks/linux/library/l-prog3.html>`__,
+`part 1 <http://www.ibm.com/developerworks/linux/library/l-prog/index.html>`__,
+`part 2 <http://www.ibm.com/developerworks/linux/library/l-prog2/index.html>`__, and
+`part 3 <http://www.ibm.com/developerworks/linux/library/l-prog3/index.html>`__,

python/vendor/current/Doc/howto/index.rst

-              r2
+              r388
    :maxdepth: 1
    advocacy.rst
+   pyporting.rst
    cporting.rst
    curses.rst
+   descriptor.rst
    doanddont.rst
    functional.rst
+   logging.rst
+   logging-cookbook.rst
    regex.rst
    sockets.rst
+   sorting.rst
    unicode.rst
    urllib2.rst
    webservers.rst
+   argparse.rst

python/vendor/current/Doc/howto/regex.rst

-              r2
+              r388
 :Author: A.M. Kuchling <amk@amk.ca>
-:Release: 0.05
 .. TODO:
 …
 in the rest of this HOWTO. ::
    . ^ $ * + ? { [ ] \ | ( )
+   . ^ $ * + ? { } [ ] \ | ( )
 The first metacharacters we'll look at are ``[`` and ``]``. They're used for
 …
 of characters that are often useful, such as the set of digits, the set of
 letters, or the set of anything that isn't whitespace.  The following predefined
+special sequences are available:
+special sequences are a subset of those available. The equivalent classes are
+for byte string patterns. For a complete list of sequences and expanded class
+definitions for Unicode string patterns, see the last part of
+:ref:`Regular Expression Syntax <re-syntax>`.
 ``\d``
 …
    >>> import re
    >>> p = re.compile('ab*')
    >>> print p
    <_sre.SRE_Pattern object at 80b4150>
+   >>> p  #doctest: +ELLIPSIS
+   <_sre.SRE_Pattern object at 0x...>
 :func:`re.compile` also accepts an optional *flags* argument, used to enable
 …
 :meth:`match` and :meth:`search` return ``None`` if no match can be found.  If
 they're successful, a ``MatchObject`` instance is returned, containing
+information about the match: where it starts and ends, the substring it matched,
 and more.
+they're successful, a :ref:`match object <match-objects>` instance is returned,
+containing information about the match: where it starts and ends, the substring
+it matched, and more.
 You can learn about this by interactively experimenting with the :mod:`re`
 module.  If you have Tkinter available, you may also want to look at
 :file:`Tools/scripts/redemo.py`, a demonstration program included with the
+:source:`Tools/scripts/redemo.py`, a demonstration program included with the
 Python distribution.  It allows you to enter REs and strings, and displays
 whether the RE matches or fails. :file:`redemo.py` can be quite useful when
 …
    >>> import re
    >>> p = re.compile('[a-z]+')
    >>> p
    <_sre.SRE_Pattern object at 80c3c28>
+   >>> p  #doctest: +ELLIPSIS
+   <_sre.SRE_Pattern object at 0x...>
 Now, you can try matching various strings against the RE ``[a-z]+``.  An empty
 …
 Now, let's try it on a string that it should match, such as ``tempo``.  In this
 case, :meth:`match` will return a :class:`MatchObject`, so you should store the
 result in a variable for later use. ::
+case, :meth:`match` will return a :ref:`match object <match-objects>`, so you
+should store the result in a variable for later use. ::
    >>> m = p.match('tempo')
    >>> print m
    <_sre.SRE_Match object at 80c4f68>
 Now you can query the :class:`MatchObject` for information about the matching
+string.   :class:`MatchObject` instances also have several methods and
 attributes; the most important ones are:
+   >>> m  #doctest: +ELLIPSIS
+   <_sre.SRE_Match object at 0x...>
+Now you can query the :ref:`match object <match-objects>` for information
+about the matching string.  :ref:`match object <match-objects>` instances
+also have several methods and attributes; the most important ones are:
 +------------------+--------------------------------------------+
 …
    >>> print p.match('::: message')
    None
    >>> m = p.search('::: message') ; print m
    <re.MatchObject instance at 80c9650>
+   >>> m = p.search('::: message'); print m  #doctest: +ELLIPSIS
+   <_sre.SRE_Match object at 0x...>
    >>> m.group()
    'message'
 …
    (4, 11)
+In actual programs, the most common style is to store the :class:`MatchObject`
+in a variable, and then check if it was ``None``.  This usually looks like::
+In actual programs, the most common style is to store the
+:ref:`match object <match-objects>` in a variable, and then check if it was
+``None``.  This usually looks like::
    p = re.compile( ... )
 …
 :meth:`findall` has to create the entire list before it can be returned as the
 result.  The :meth:`finditer` method returns a sequence of :class:`MatchObject`
 instances as an :term:`iterator`. [#]_ ::
+result.  The :meth:`finditer` method returns a sequence of
+:ref:`match object <match-objects>` instances as an :term:`iterator`. [#]_ ::
    >>> iterator = p.finditer('12 drummers drumming, 11 ... 10 ...')
    >>> iterator
    <callable-iterator object at 0x401833ac>
+   >>> iterator  #doctest: +ELLIPSIS
+   <callable-iterator object at 0x...>
    >>> for match in iterator:
    ...     print match.span()
 …
 take the same arguments as the corresponding pattern method, with
 the RE string added as the first argument, and still return either ``None`` or a
 :class:`MatchObject` instance. ::
+:ref:`match object <match-objects>` instance. ::
    >>> print re.match(r'From\s+', 'Fromage amk')
    None
    >>> re.match(r'From\s+', 'From amk Thu May 14 19:12:10 1998')
    <re.MatchObject instance at 80c5978>
+   >>> re.match(r'From\s+', 'From amk Thu May 14 19:12:10 1998')  #doctest: +ELLIPSIS
+   <_sre.SRE_Match object at 0x...>
 Under the hood, these functions simply create a pattern object for you
 …
 the definitions in one place, in a section of code that compiles all the REs
 ahead of time.  To take an example from the standard library, here's an extract
 from :file:`xmllib.py`::
+from the deprecated :mod:`xmllib` module::
    ref = re.compile( ... )
 …
    line, the RE to use is ``^From``. ::
       >>> print re.search('^From', 'From Here to Eternity')
       <re.MatchObject instance at 80c1520>
+      >>> print re.search('^From', 'From Here to Eternity')  #doctest: +ELLIPSIS
+      <_sre.SRE_Match object at 0x...>
       >>> print re.search('^From', 'Reciting From Memory')
       None
 …
    or any location followed by a newline character.     ::
       >>> print re.search('}$', '{block}')
       <re.MatchObject instance at 80adfa8>
+      >>> print re.search('}$', '{block}')  #doctest: +ELLIPSIS
+      <_sre.SRE_Match object at 0x...>
       >>> print re.search('}$', '{block} ')
       None
       >>> print re.search('}$', '{block}\n')
       <re.MatchObject instance at 80adfa8>
+      >>> print re.search('}$', '{block}\n')  #doctest: +ELLIPSIS
+      <_sre.SRE_Match object at 0x...>
    To match a literal ``'$'``, use ``\$`` or enclose it inside a character class,
 …
       >>> p = re.compile(r'\bclass\b')
       >>> print p.search('no class at all')
       <re.MatchObject instance at 80c8f28>
+      >>> print p.search('no class at all')  #doctest: +ELLIPSIS
+      <_sre.SRE_Match object at 0x...>
       >>> print p.search('the declassified algorithm')
       None
 …
       >>> print p.search('no class at all')
       None
       >>> print p.search('\b' + 'class' + '\b')
       <re.MatchObject instance at 80c3ee0>
+      >>> print p.search('\b' + 'class' + '\b')  #doctest: +ELLIPSIS
+      <_sre.SRE_Match object at 0x...>
    Second, inside a character class, where there's no use for this assertion,
 …
 to :meth:`group`, :meth:`start`, :meth:`end`, and :meth:`span`.  Groups are
 numbered starting with 0.  Group 0 is always present; it's the whole RE, so
 :class:`MatchObject` methods all have group 0 as their default argument.  Later
+we'll see how to express groups that don't capture the span of text that they
 match. ::
+:ref:`match object <match-objects>` methods all have group 0 as their default
+argument.  Later we'll see how to express groups that don't capture the span
+of text that they match. ::
    >>> p = re.compile('(a)b')
 …
 ``(?P<name>...)``.  *name* is, obviously, the name of the group.  Named groups
 also behave exactly like capturing groups, and additionally associate a name
 with a group.  The :class:`MatchObject` methods that deal with capturing groups
+all accept either integers that refer to the group by number or strings that
+contain the desired group's name.  Named groups are still given numbers, so you
 can retrieve information about a group in two ways::
+with a group.  The :ref:`match object <match-objects>` methods that deal with
+capturing groups all accept either integers that refer to the group by number
+or strings that contain the desired group's name.  Named groups are still
+given numbers, so you can retrieve information about a group in two ways::
    >>> p = re.compile(r'(?P<word>\b\w+\b)')
 …
 *replacement* can also be a function, which gives you even more control.  If
 *replacement* is a function, the function is called for every non-overlapping
 occurrence of *pattern*.  On each call, the function is  passed a
 :class:`MatchObject` argument for the match and can use this information to
 compute the desired replacement string and return it.
 In the following example, the replacement function translates  decimals into
+occurrence of *pattern*.  On each call, the function is passed a
+:ref:`match object <match-objects>` argument for the match and can use this
+information to compute the desired replacement string and return it.
+In the following example, the replacement function translates decimals into
 hexadecimal::
    >>> def hexrepl( match ):
+   >>> def hexrepl(match):
    ...     "Return the hex string for a decimal number"
    ...     value = int( match.group() )
+   ...     value = int(match.group())
    ...     return hex(value)
    ...
 …
 Not Using re.VERBOSE
+Using re.VERBOSE
 --------------------

python/vendor/current/Doc/howto/sockets.rst

-              r2
+              r388
+.. _socket-howto:
 ****************************
   Socket Programming HOWTO
 …
 Sockets
 =======
-Sockets are used nearly everywhere, but are one of the most severely
-misunderstood technologies around. This is a 10,000 foot overview of sockets.
-It's not really a tutorial - you'll still have work to do in getting things
-working. It doesn't cover the fine points (and there are a lot of them), but I
-hope it will give you enough background to begin using them decently.
 I'm only going to talk about INET sockets, but they account for at least 99% of
 …
 -------
 Of the various forms of IPC (*Inter Process Communication*), sockets are by far
+the most popular.  On any given platform, there are likely to be other forms of
+IPC that are faster, but for cross-platform communication, sockets are about the
 only game in town.
+Of the various forms of :abbr:`IPC (Inter Process Communication)`,
+sockets are by far the most popular.  On any given platform, there are
+likely to be other forms of IPC that are faster, but for
+cross-platform communication, sockets are about the only game in town.
 They were invented in Berkeley as part of the BSD flavor of Unix. They spread
 …
    s.connect(("www.mcmillan-inc.com", 80))
+When the ``connect`` completes, the socket ``s`` can now be used to send in a
+request for the text of this page. The same socket will read the reply, and then
+be destroyed. That's right - destroyed. Client sockets are normally only used
+for one exchange (or a small set of sequential exchanges).
+When the ``connect`` completes, the socket ``s`` can be used to send
+in a request for the text of the page. The same socket will read the
+reply, and then be destroyed. That's right, destroyed. Client sockets
+are normally only used for one exchange (or a small set of sequential
+exchanges).
 What happens in the web server is a bit more complex. First, the web server
 creates a "server socket". ::
+creates a "server socket"::
    #create an INET, STREAMing socket
 …
 A couple things to notice: we used ``socket.gethostname()`` so that the socket
+would be visible to the outside world. If we had used ``s.bind(('', 80))`` or
+``s.bind(('localhost', 80))`` or ``s.bind(('127.0.0.1', 80))`` we would still
+have a "server" socket, but one that was only visible within the same machine.
+would be visible to the outside world.  If we had used ``s.bind(('localhost',
+))`` or ``s.bind(('127.0.0.1', 80))`` we would still have a "server" socket,
+but one that was only visible within the same machine.  ``s.bind(('', 80))``
+specifies that the socket is reachable by any address the machine happens to
+have.
 A second thing to note: low number ports are usually reserved for "well known"
 …
 connections. If the rest of the code is written properly, that should be plenty.
 OK, now we have a "server" socket, listening on port 80. Now we enter the
+Now that we have a "server" socket, listening on port 80, we can enter the
 mainloop of the web server::
 …
 Now there are two sets of verbs to use for communication. You can use ``send``
 and ``recv``, or you can transform your client socket into a file-like beast and
 use ``read`` and ``write``. The latter is the way Java presents their sockets.
+use ``read`` and ``write``. The latter is the way Java presents its sockets.
 I'm not going to talk about it here, except to warn you that you need to use
 ``flush`` on sockets. These are buffered "files", and a common mistake is to
 …
 your output buffer.
 Now we come the major stumbling block of sockets - ``send`` and ``recv`` operate
+Now we come to the major stumbling block of sockets - ``send`` and ``recv`` operate
 on the network buffers. They do not necessarily handle all the bytes you hand
 them (or expect from them), because their major focus is handling the network
 …
 the process of closing) the connection.  You will not receive any more data on
 this connection. Ever.  You may be able to send data successfully; I'll talk
 about that some on the next page.
+more about this later.
 A protocol like HTTP uses a socket for only one transfer. The client sends a
 request, the reads a reply.  That's it. The socket is discarded. This means that
+request, then reads a reply.  That's it. The socket is discarded. This means that
 a client can detect the end of the reply by receiving 0 bytes.
 But if you plan to reuse your socket for further transfers, you need to realize
 that *there is no "EOT" (End of Transfer) on a socket.* I repeat: if a socket
+that *there is no* :abbr:`EOT (End of Transfer)` *on a socket.* I repeat: if a socket
 ``send`` or ``recv`` returns after handling 0 bytes, the connection has been
 broken.  If the connection has *not* been broken, you may wait on a ``recv``
 …
 ====================
 If you've understood the preceeding, you already know most of what you need to
+If you've understood the preceding, you already know most of what you need to
 know about the mechanics of using sockets. You'll still use the same calls, in
 much the same ways. It's just that, if you do it right, your app will be almost
 …
 In C, coding ``select`` is fairly complex. In Python, it's a piece of cake, but
 it's close enough to the C version that if you understand ``select`` in Python,
 you'll have little trouble with it in C. ::
+you'll have little trouble with it in C::
    ready_to_read, ready_to_write, in_error = \
 …
 reason to do otherwise.
 In return, you will get three lists. They have the sockets that are actually
+In return, you will get three lists. They contain the sockets that are actually
 readable, writable and in error. Each of these lists is a subset (possibly
+empty) of the corresponding list you passed in. And if you put a socket in more
+than one input list, it will only be (at most) in one output list.
+empty) of the corresponding list you passed in.
 If a socket is in the output readable list, you can be

python/vendor/current/Doc/howto/unicode.rst

-              r2
+              r388
 *****************
+:Release: 1.02
+This HOWTO discusses Python's support for Unicode, and explains various problems
+that people commonly encounter when trying to work with Unicode.
+:Release: 1.03
+This HOWTO discusses Python 2.x's support for Unicode, and explains
+various problems that people commonly encounter when trying to work
+with Unicode.  For the Python 3 version, see
+<http://docs.python.org/py3k/howto/unicode.html>.
 Introduction to Unicode
 …
    handle content with embedded zero bytes.
+Generally people don't use this encoding, instead choosing other encodings that
+are more efficient and convenient.
+Generally people don't use this encoding, instead choosing other
+encodings that are more efficient and convenient.  UTF-8 is probably
+the most commonly supported encoding; it will be discussed below.
 Encodings don't have to handle every possible Unicode character, and most
 …
 Python's Unicode Support
 ========================
+Python 2.x's Unicode Support
+============================
 Now that you've learned the rudiments of Unicode, we can look at Python's
 …
     >>> type(s)
     <type 'unicode'>
     >>> unicode('abcdef' + chr(255))
+    >>> unicode('abcdef' + chr(255))    #doctest: +NORMALIZE_WHITESPACE
     Traceback (most recent call last):
       File "<stdin>", line 1, in ?
+    ...
     UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 6:
                         ordinal not in range(128)
+    ordinal not in range(128)
 The ``errors`` argument specifies the response when the input string can't be
 …
 Unicode result).  The following examples show the differences::
     >>> unicode('\x80abc', errors='strict')
+    >>> unicode('\x80abc', errors='strict')     #doctest: +NORMALIZE_WHITESPACE
     Traceback (most recent call last):
       File "<stdin>", line 1, in ?
+        ...
     UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0:
                         ordinal not in range(128)
+    ordinal not in range(128)
     >>> unicode('\x80abc', errors='replace')
     u'\ufffdabc'
 …
     u'abc'
 Encodings are specified as strings containing the encoding's name.  Python 2.4
+Encodings are specified as strings containing the encoding's name.  Python 2.7
 comes with roughly 100 different encodings; see the Python Library Reference at
 :ref:`standard-encodings` for a list.  Some encodings
 …
 than 127 will cause an exception::
     >>> s.find('Was\x9f')
+    >>> s.find('Was\x9f')                   #doctest: +NORMALIZE_WHITESPACE
     Traceback (most recent call last):
+      File "<stdin>", line 1, in ?
+    UnicodeDecodeError: 'ascii' codec can't decode byte 0x9f in position 3: ordinal not in range(128)
+        ...
+    UnicodeDecodeError: 'ascii' codec can't decode byte 0x9f in position 3:
+    ordinal not in range(128)
     >>> s.find(u'Was\x9f')
     -1
 …
     >>> u.encode('utf-8')
     '\xea\x80\x80abcd\xde\xb4'
     >>> u.encode('ascii')
+    >>> u.encode('ascii')                       #doctest: +NORMALIZE_WHITESPACE
     Traceback (most recent call last):
+      File "<stdin>", line 1, in ?
+    UnicodeEncodeError: 'ascii' codec can't encode character '\ua000' in position 0: ordinal not in range(128)
+        ...
+    UnicodeEncodeError: 'ascii' codec can't encode character u'\ua000' in
+    position 0: ordinal not in range(128)
     >>> u.encode('ascii', 'ignore')
     'abcd'
 …
     >>> s = u"a\xac\u1234\u20ac\U00008000"
                ^^^^ two-digit hex escape
                    ^^^^^^ four-digit Unicode escape
                                ^^^^^^^^^^ eight-digit Unicode escape
+    ... #      ^^^^ two-digit hex escape
+    ... #          ^^^^^^ four-digit Unicode escape
+    ... #                      ^^^^^^^^^^ eight-digit Unicode escape
     >>> for c in s:  print ord(c),
     ...
 …
 When you run it with Python 2.4, it will output the following warning::
     amk:~$ python p263.py
+    amk:~$ python2.4 p263.py
     sys:1: DeprecationWarning: Non-ASCII character '\xe9'
          in file p263.py on line 2, but no encoding declared;
          see http://www.python.org/peps/pep-0263.html for details
+Python 2.5 and higher are stricter and will produce a syntax error::
+    amk:~$ python2.5 p263.py
+    File "/tmp/p263.py", line 2
+    SyntaxError: Non-ASCII character '\xc3' in file /tmp/p263.py
+      on line 2, but no encoding declared; see
+      http://www.python.org/peps/pep-0263.html for details
 …
 "Number, other", ``'Mn'`` is "Mark, nonspacing", and ``'So'`` is "Symbol,
 other".  See
 <http://unicode.org/Public/5.1.0/ucd/UCD.html#General_Category_Values> for a
+<http://www.unicode.org/reports/tr44/#General_Category_Values> for a
 list of category codes.
 …
 Version 1.02: posted August 16 2005.  Corrects factual errors.
+Version 1.03: posted June 20 2010.  Notes that Python 3.x is not covered,
+and that the HOWTO only covers 2.x.
+.. comment Describe Python 3.x support (new section? new document?)
 .. comment Additional topic: building Python w/ UCS2 or UCS4 support
 .. comment Describe obscure -U switch somewhere?

python/vendor/current/Doc/howto/urllib2.rst

-              r2
+              r388
     >>> data['language'] = 'Python'
     >>> url_values = urllib.urlencode(data)
     >>> print url_values
+    >>> print url_values  # The order may differ. #doctest: +SKIP
     name=Somebody+Here&language=Python&location=Northampton
     >>> url = 'http://www.example.com/example.cgi'
     >>> full_url = url + '?' + url_values
     >>> data = urllib2.open(full_url)
+    >>> data = urllib2.urlopen(full_url)
 Notice that the full URL is created by adding a ``?`` to the URL, followed by
 …
     >>> req = urllib2.Request('http://www.pretend_server.org')
     >>> try: urllib2.urlopen(req)
     >>> except URLError, e:
     >>>    print e.reason
     >>>
+    ... except URLError as e:
+    ...    print e.reason   #doctest: +SKIP
+    ...
     (4, 'getaddrinfo failed')
 …
     >>> req = urllib2.Request('http://www.python.org/fish.html')
     >>> try:
     >>>     urllib2.urlopen(req)
     >>> except HTTPError, e:
     >>>     print e.code
     >>>     print e.read()
     >>>
+    ...     urllib2.urlopen(req)
+    ... except urllib2.HTTPError as e:
+    ...     print e.code
+    ...     print e.read() #doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE
+    ...
     <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
         "http://www.w3.org/TR/html4/loose.dtd">
     <?xml-stylesheet href="./css/ht2html.css"
         type="text/css"?>
     <html><head><title>Error 404: File Not Found</title>
+    ...... etc...
+    <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+    ...
+    <title>Page Not Found</title>
+    ...
 Wrapping it Up
 …
     try:
         response = urlopen(req)
     except HTTPError, e:
+    except HTTPError as e:
         print 'The server couldn\'t fulfill the request.'
         print 'Error code: ', e.code
     except URLError, e:
+    except URLError as e:
         print 'We failed to reach a server.'
         print 'Reason: ', e.reason
 …
     try:
         response = urlopen(req)
     except URLError, e:
+    except URLError as e:
         if hasattr(e, 'reason'):
             print 'We failed to reach a server.'
 …
 When authentication is required, the server sends a header (as well as the 401
 error code) requesting authentication.  This specifies the authentication scheme
 and a 'realm'. The header looks like : ``Www-authenticate: SCHEME
+and a 'realm'. The header looks like : ``WWW-Authenticate: SCHEME
 realm="REALM"``.
 e.g. ::
     Www-authenticate: Basic realm="cPanel Users"
+    WWW-Authenticate: Basic realm="cPanel Users"
 …
     In the above example we only supplied our ``HTTPBasicAuthHandler`` to
     ``build_opener``. By default openers have the handlers for normal situations
+    -- ``ProxyHandler``, ``UnknownHandler``, ``HTTPHandler``,
+    -- ``ProxyHandler`` (if a proxy setting such as an :envvar:`http_proxy`
+    environment variable is set), ``UnknownHandler``, ``HTTPHandler``,
     ``HTTPDefaultErrorHandler``, ``HTTPRedirectHandler``, ``FTPHandler``,
     ``FileHandler``, ``HTTPErrorProcessor``.
 …
 **urllib2** will auto-detect your proxy settings and use those. This is through
+the ``ProxyHandler`` which is part of the normal handler chain. Normally that's
+a good thing, but there are occasions when it may not be helpful [#]_. One way
+to do this is to setup our own ``ProxyHandler``, with no proxies defined. This
+is done using similar steps to setting up a `Basic Authentication`_ handler : ::
+the ``ProxyHandler``, which is part of the normal handler chain when a proxy
+setting is detected.  Normally that's a good thing, but there are occasions
+when it may not be helpful [#]_. One way to do this is to setup our own
+``ProxyHandler``, with no proxies defined. This is done using similar steps to
+setting up a `Basic Authentication`_ handler : ::
     >>> proxy_support = urllib2.ProxyHandler({})

python/vendor/current/Doc/howto/webservers.rst

-              r2
+              r388
 .. topic:: Abstract
    This document shows how Python fits into the web.  It presents some ways on
    how to integrate Python with the web server and general practices useful for
+   This document shows how Python fits into the web.  It presents some ways
+   to integrate Python with a web server, and general practices useful for
    developing web sites.
 Programming for the Web has become a hot topic since the raise of the "Web 2.0",
+Programming for the Web has become a hot topic since the rise of "Web 2.0",
 which focuses on user-generated content on web sites.  It has always been
 possible to use Python for creating web sites, but it was a rather tedious task.
 Therefore, many so-called "frameworks" and helper tools were created to help
 developers creating sites faster and these sites being more robust.  This HOWTO
 describes some of the methods used to combine Python with a web server to create
 dynamic content.  It is not meant as a general introduction as this topic is far
+too broad to be covered in one single document.  However, a short overview of
 the most popular libraries is provided.
 .. seealso::
    While this HOWTO tries to give an overview over Python in the Web, it cannot
    always be as up to date as desired.  Web development in Python is moving
    forward rapidly, so the wiki page on `Web Programming
    <http://wiki.python.org/moin/WebProgramming>`_ might be more in sync with
+Therefore, many frameworks and helper tools have been created to assist
+developers in creating faster and more robust sites.  This HOWTO describes
+some of the methods used to combine Python with a web server to create
+dynamic content.  It is not meant as a complete introduction, as this topic is
+far too broad to be covered in one single document.  However, a short overview
+of the most popular libraries is provided.
+.. seealso::
+   While this HOWTO tries to give an overview of Python in the web, it cannot
+   always be as up to date as desired.  Web development in Python is rapidly
+   moving forward, so the wiki page on `Web Programming
+   <http://wiki.python.org/moin/WebProgramming>`_ may be more in sync with
    recent development.
 The low-level view
+The Low-Level View
 ==================
+.. .. image:: http.png
+When a user enters a web site, his browser makes a connection to the site's
+webserver (this is called the *request*).  The server looks up the file in the
+When a user enters a web site, their browser makes a connection to the site's
+web server (this is called the *request*).  The server looks up the file in the
 file system and sends it back to the user's browser, which displays it (this is
+the *response*).  This is roughly how the unterlying protocol, HTTP works.
+Now, dynamic web sites are not files in the file system, but rather programs
+which are run by the web server when a request comes in.  They can do all sorts
+of useful things, like display the postings of a bulletin board, show your
+mails, configurate software or just display the current time.  These programs
+can be written in about any programming language the server supports, so it is
+easy to use Python for creating dynamic web sites.
+As most of HTTP servers are written in C or C++, they cannot execute Python code
+in a simple way -- a bridge is needed between the server and the program.  These
+bridges or rather interfaces define how programs interact with the server.  In
+the past there have been numerous attempts to create the best possible
+interface, but there are only a few worth mentioning.
+Not every web server supports every interface.  Many web servers do support only
+old, now-obsolete interfaces.  But they can often be extended using some
+third-party modules to support new interfaces.
+the *response*).  This is roughly how the underlying protocol, HTTP, works.
+Dynamic web sites are not based on files in the file system, but rather on
+programs which are run by the web server when a request comes in, and which
+*generate* the content that is returned to the user.  They can do all sorts of
+useful things, like display the postings of a bulletin board, show your email,
+configure software, or just display the current time.  These programs can be
+written in any programming language the server supports.  Since most servers
+support Python, it is easy to use Python to create dynamic web sites.
+Most HTTP servers are written in C or C++, so they cannot execute Python code
+directly -- a bridge is needed between the server and the program.  These
+bridges, or rather interfaces, define how programs interact with the server.
+There have been numerous attempts to create the best possible interface, but
+there are only a few worth mentioning.
+Not every web server supports every interface.  Many web servers only support
+old, now-obsolete interfaces; however, they can often be extended using
+third-party modules to support newer ones.
 …
 ------------------------
+This interface is the oldest one, supported by nearly every web server out of
+the box.  Programs using CGI to communicate with their web server need to be
+started by the server for every request.  So, every request starts a new Python
+interpreter -- which takes some time to start up -- thus making the whole
+interface only usable for low load situations.
+The upside of CGI is that it is simple -- writing a program which uses CGI is a
+matter of about three lines of code.  But this simplicity comes at a price: it
+does very few things to help the developer.
+Writing CGI programs, while still possible, is not recommended anymore.  With
+WSGI (more on that later) it is possible to write programs that emulate CGI, so
+they can be run as CGI if no better option is available.
+This interface, most commonly referred to as "CGI", is the oldest, and is
+supported by nearly every web server out of the box.  Programs using CGI to
+communicate with their web server need to be started by the server for every
+request.  So, every request starts a new Python interpreter -- which takes some
+time to start up -- thus making the whole interface only usable for low load
+situations.
+The upside of CGI is that it is simple -- writing a Python program which uses
+CGI is a matter of about three lines of code.  This simplicity comes at a
+price: it does very few things to help the developer.
+Writing CGI programs, while still possible, is no longer recommended.  With
+:ref:`WSGI <WSGI>`, a topic covered later in this document, it is possible to write
+programs that emulate CGI, so they can be run as CGI if no better option is
+available.
 .. seealso::
 …
    * :mod:`cgi` -- Handling of user input in CGI scripts
    * :mod:`cgitb` -- Displays nice tracebacks when errors happen in of CGI
+   * :mod:`cgitb` -- Displays nice tracebacks when errors happen in CGI
      applications, instead of presenting a "500 Internal Server Error" message
 …
     print "Hello World!"
+You need to write this code into a file with a ``.py`` or ``.cgi`` extension,
+this depends on your web server configuration.  Depending on your web server
+configuration, this file may also need to be in a ``cgi-bin`` folder, for
+security reasons.
+Depending on your web server configuration, you may need to save this code with
+a ``.py`` or ``.cgi`` extension.  Additionally, this file may also need to be
+in a ``cgi-bin`` folder, for security reasons.
 You might wonder what the ``cgitb`` line is about.  This line makes it possible
 to display a nice traceback instead of just crashing and displaying an "Internal
 Server Error" in the user's browser.  This is useful for debugging, but it might
 risk exposing some confident data to the user.  Don't use it when the script is
 ready for production use.  Still, you should *always* catch exceptions, and
+risk exposing some confidential data to the user.  You should not use ``cgitb``
+in production code for this reason.  You should *always* catch exceptions, and
 display proper error pages -- end-users don't like to see nondescript "Internal
 Server Errors" in their browsers.
 …
 If you don't have your own web server, this does not apply to you.  You can
 check whether if works as-is and if not you need to talk to the administrator of
+your web server anyway. If it is a big hoster, you can try filing a ticket
 asking for Python support.
 If you're your own administrator or want to install it for testing purposes on
 your own computers, you have to configure it by yourself.  There is no one and
 single way on how to configure CGI, as there are many web servers with different
 configuration options.  The currently most widely used free web server is
 `Apache HTTPd <http://httpd.apache.org/>`_, Apache for short -- this is the one
+that most people use, it can be easily installed on nearly every system using
 the systems' package management.  But `lighttpd <http://www.lighttpd.net>`_ has
+been gaining attention since some time and is said to have a better performance.
+On many systems this server can also be installed using the package management,
 so manually compiling the web server is never needed.
 * On Apache you can take a look into the `Dynamic Content with CGI
+check whether it works as-is, and if not you will need to talk to the
+administrator of your web server. If it is a big host, you can try filing a
+ticket asking for Python support.
+If you are your own administrator or want to set up CGI for testing purposes on
+your own computers, you have to configure it by yourself.  There is no single
+way to configure CGI, as there are many web servers with different
+configuration options.  Currently the most widely used free web server is
+`Apache HTTPd <http://httpd.apache.org/>`_, or Apache for short. Apache can be
+easily installed on nearly every system using the system's package management
+tool.  `lighttpd <http://www.lighttpd.net>`_ is another alternative and is
+said to have better performance.  On many systems this server can also be
+installed using the package management tool, so manually compiling the web
+server may not be needed.
+* On Apache you can take a look at the `Dynamic Content with CGI
   <http://httpd.apache.org/docs/2.2/howto/cgi.html>`_ tutorial, where everything
   is described.  Most of the time it is enough just to set ``+ExecCGI``.  The
   tutorial also describes the most common gotchas that might arise.
 * On lighttpd you need to use the `CGI module
   <http://trac.lighttpd.net/trac/wiki/Docs%3AModCGI>`_ which can be configured
+  <http://redmine.lighttpd.net/wiki/lighttpd/Docs:ModCGI>`_\ , which can be configured
   in a straightforward way.  It boils down to setting ``cgi.assign`` properly.
 …
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Trying to use CGI sometimes leads to small annoyances that one might experience
+while trying to get these scripts to run.  Sometimes it happens that a seemingly
+correct script does not work as expected, which is caused by some small hidden
+reason that's difficult to spot.
+Some of these reasons are:
+* The Python script is not marked executable.  When CGI scripts are not
+  executable most of the web servers will let the user download it, instead of
+Using CGI sometimes leads to small annoyances while trying to get these
+scripts to run.  Sometimes a seemingly correct script does not work as
+expected, the cause being some small hidden problem that's difficult to spot.
+Some of these potential problems are:
+* The Python script is not marked as executable.  When CGI scripts are not
+  executable most web servers will let the user download it, instead of
   running it and sending the output to the user.  For CGI scripts to run
+  properly the ``+x`` bit needs to be set.  Using ``chmod a+x your_script.py``
+  might already solve the problem.
+* The line endings must be of Unix-type.  This is important because the web
+  server checks the first line of the script (called shebang) and tries to run
+  the program specified there.  It gets easily confused by Windows line endings
+  (Carriage Return & Line Feed, also called CRLF), so you have to convert the
+  file to Unix line endings (only Line Feed, LF).  This can be done
+  automatically by uploading the file via FTP in text mode instead of binary
+  mode, but the preferred way is just telling your editor to save the files with
+  Unix line endings.  Most proper editors support this.
+* Your web server must be able to read the file, you need to make sure the
+  permissions are fine.  Often the server runs as user and group ``www-data``,
+  so it might be worth a try to change the file ownership or making the file
+  world readable by using ``chmod a+r your_script.py``.
+* The webserver must be able to know that the file you're trying to access is a
+  CGI script.  Check the configuration of your web server, maybe there is some
+  mistake.
+* The path to the interpreter in the shebang (``#!/usr/bin/env python``) must be
+  currect.  This line calls ``/usr/bin/env`` to find Python, but it'll fail if
+  there is no ``/usr/bin/env``.  If you know where your Python is installed, you
+  can also use that path.  The commands ``whereis python`` and ``type -p
+  python`` might also help to find where it is installed.  Once this is known,
+  the shebang line can be changed accordingly: ``#!/usr/bin/python``.
+  properly on Unix-like operating systems, the ``+x`` bit needs to be set.
+  Using ``chmod a+x your_script.py`` may solve this problem.
+* On a Unix-like system, The line endings in the program file must be Unix
+  style line endings.  This is important because the web server checks the
+  first line of the script (called shebang) and tries to run the program
+  specified there.  It gets easily confused by Windows line endings (Carriage
+  Return & Line Feed, also called CRLF), so you have to convert the file to
+  Unix line endings (only Line Feed, LF).  This can be done automatically by
+  uploading the file via FTP in text mode instead of binary mode, but the
+  preferred way is just telling your editor to save the files with Unix line
+  endings.  Most editors support this.
+* Your web server must be able to read the file, and you need to make sure the
+  permissions are correct.  On unix-like systems, the server often runs as user
+  and group ``www-data``, so it might be worth a try to change the file
+  ownership, or making the file world readable by using ``chmod a+r
+  your_script.py``.
+* The web server must know that the file you're trying to access is a CGI script.
+  Check the configuration of your web server, as it may be configured
+  to expect a specific file extension for CGI scripts.
+* On Unix-like systems, the path to the interpreter in the shebang
+  (``#!/usr/bin/env python``) must be correct.  This line calls
+  ``/usr/bin/env`` to find Python, but it will fail if there is no
+  ``/usr/bin/env``, or if Python is not in the web server's path.  If you know
+  where your Python is installed, you can also use that full path.  The
+  commands ``whereis python`` and ``type -p python`` could help you find
+  where it is installed.  Once you know the path, you can change the shebang
+  accordingly: ``#!/usr/bin/python``.
 * The file must not contain a BOM (Byte Order Mark). The BOM is meant for
+  determining the byte order of UTF-16 encodings, but some editors write this
+  also into UTF-8 files.  The BOM interferes with the shebang line, so be sure
+  to tell your editor not to write the BOM.
+* :ref:`mod-python` might be making problems.  mod_python is able to handle CGI
+  scripts by itself, but it can also be a source for problems.  Be sure you
+  disable it.
+  determining the byte order of UTF-16 and UTF-32 encodings, but some editors
+  write this also into UTF-8 files.  The BOM interferes with the shebang line,
+  so be sure to tell your editor not to write the BOM.
+* If the web server is using :ref:`mod-python`, ``mod_python`` may be having
+  problems.  ``mod_python`` is able to handle CGI scripts by itself, but it can
+  also be a source of issues.
 …
 People coming from PHP often find it hard to grasp how to use Python in the web.
+Their first thought is mostly `mod_python <http://www.modpython.org/>`_ because
+they think that this is the equivalent to ``mod_php``.  Actually it is not
+really.  It does embed the interpreter into the Apache process, thus speeding up
+requests by not having to start a Python interpreter every request.  On the
+other hand, it is by far not "Python intermixed with HTML" as PHP often does.
+The Python equivalent of that is a template engine.  mod_python itself is much
+more powerful and gives more access to Apache internals.  It can emulate CGI, it
+can work an a "Python Server Pages" mode similar to JSP which is "HTML
+intermangled with Python" and it has a "Publisher" which destignates one file to
+accept all requests and decide on what to do then.
+But mod_python has some problems.  Unlike the PHP interpreter the Python
+interpreter uses caching when executing files, so when changing a file the whole
+web server needs to be re-started to update.  Another problem ist the basic
+concept -- Apache starts some child processes to handle the requests and
+unfortunately every child process needs to load the whole Python interpreter
+even if it does not use it.  This makes the whole web server slower.  Another
+problem is that as mod_python is linked against a specific version of
+``libpython``, it is not possible to switch from an older version to a newer
+(e.g. 2.4 to 2.5) without recompiling mod_python.  mod_python is also bound to
+the Apache web server, so programs written for mod_python cannot easily run on
+other web servers.
+These are the reasons why mod_python should be avoided when writing new
+programs.  In some circumstances it might be still a good idea to use mod_python
+for deployment, but WSGI makes it possible to run WSGI programs under mod_python
+as well.
+Their first thought is mostly `mod_python <http://www.modpython.org/>`_\ ,
+because they think that this is the equivalent to ``mod_php``.  Actually, there
+are many differences.  What ``mod_python`` does is embed the interpreter into
+the Apache process, thus speeding up requests by not having to start a Python
+interpreter for each request.  On the other hand, it is not "Python intermixed
+with HTML" in the way that PHP is often intermixed with HTML. The Python
+equivalent of that is a template engine.  ``mod_python`` itself is much more
+powerful and provides more access to Apache internals.  It can emulate CGI,
+work in a "Python Server Pages" mode (similar to JSP) which is "HTML
+intermingled with Python", and it has a "Publisher" which designates one file
+to accept all requests and decide what to do with them.
+``mod_python`` does have some problems.  Unlike the PHP interpreter, the Python
+interpreter uses caching when executing files, so changes to a file will
+require the web server to be restarted.  Another problem is the basic concept
+-- Apache starts child processes to handle the requests, and unfortunately
+every child process needs to load the whole Python interpreter even if it does
+not use it.  This makes the whole web server slower.  Another problem is that,
+because ``mod_python`` is linked against a specific version of ``libpython``,
+it is not possible to switch from an older version to a newer (e.g. 2.4 to 2.5)
+without recompiling ``mod_python``.  ``mod_python`` is also bound to the Apache
+web server, so programs written for ``mod_python`` cannot easily run on other
+web servers.
+These are the reasons why ``mod_python`` should be avoided when writing new
+programs.  In some circumstances it still might be a good idea to use
+``mod_python`` for deployment, but WSGI makes it possible to run WSGI programs
+under ``mod_python`` as well.
 …
 FastCGI and SCGI try to solve the performance problem of CGI in another way.
 Instead of embedding the interpreter into the web server, they create
 long-running processes which run in the background. There still is some module
+in the web server which makes it possible for the web server to "speak" with the
+background process.  As the background process is independent from the server,
+it can be written in any language of course also in Python.  The language just
 needs to have a library which handles the communication with the web server.
+long-running background processes. There is still a module in the web server
+which makes it possible for the web server to "speak" with the background
+process.  As the background process is independent of the server, it can be
+written in any language, including Python.  The language just needs to have a
+library which handles the communication with the webserver.
 The difference between FastCGI and SCGI is very small, as SCGI is essentially
 just a "simpler FastCGI".  But as the web server support for SCGI is limited
+just a "simpler FastCGI".  As the web server support for SCGI is limited,
 most people use FastCGI instead, which works the same way.  Almost everything
 that applies to SCGI also applies to FastCGI as well, so we'll only write about
+that applies to SCGI also applies to FastCGI as well, so we'll only cover
 the latter.
 These days, FastCGI is never used directly.  Just like ``mod_python`` it is only
+These days, FastCGI is never used directly.  Just like ``mod_python``, it is only
 used for the deployment of WSGI applications.
 …
    * `FastCGI, SCGI, and Apache: Background and Future
      <http://www.vmunix.com/mark/blog/archives/2006/01/02/fastcgi-scgi-and-apache-background-and-future/>`_
      is a discussion on why the concept of FastCGI and SCGI is better that that
+     is a discussion on why the concept of FastCGI and SCGI is better than that
      of mod_python.
 …
 ^^^^^^^^^^^^^^^^^^
 Depending on the web server you need to have a special module.
 * Apache has both `mod_fastcgi <http://www.fastcgi.com/>`_ and `mod_fcgid
   <http://fastcgi.coremail.cn/>`_.  ``mod_fastcgi`` is the original one, but it
   has some licensing issues that's why it is sometimes considered non-free.
   ``mod_fcgid`` is a smaller, compatible alternative. One of these modules needs
+Each web server requires a specific module.
+* Apache has both `mod_fastcgi <http://www.fastcgi.com/drupal/>`_ and `mod_fcgid
+  <http://httpd.apache.org/mod_fcgid/>`_.  ``mod_fastcgi`` is the original one, but it
+  has some licensing issues, which is why it is sometimes considered non-free.
+  ``mod_fcgid`` is a smaller, compatible alternative.  One of these modules needs
   to be loaded by Apache.
 * lighttpd ships its own `FastCGI module
+  <http://trac.lighttpd.net/trac/wiki/Docs%3AModFastCGI>`_ as well as an `SCGI
+  module <http://trac.lighttpd.net/trac/wiki/Docs%3AModSCGI>`_.
+* nginx also supports `FastCGI <http://wiki.nginx.org/NginxSimplePythonFCGI>`_.
+  <http://redmine.lighttpd.net/wiki/lighttpd/Docs:ModFastCGI>`_ as well as an
+  `SCGI module <http://redmine.lighttpd.net/wiki/lighttpd/Docs:ModSCGI>`_.
+* `nginx <http://nginx.org/>`_ also supports `FastCGI
+  <http://wiki.nginx.org/NginxSimplePythonFCGI>`_.
 Once you have installed and configured the module, you can test it with the
 …
    There is some documentation on `setting up Django with FastCGI
    <http://www.djangoproject.com/documentation/fastcgi/>`_, most of which can be
    reused for other WSGI-compliant frameworks and libraries.  Only the
    ``manage.py`` part has to be changed, the example used here can be used
    instead. Django does more or less the exact same thing.
+   <http://docs.djangoproject.com/en/dev/howto/deployment/fastcgi/>`_, most of
+   which can be reused for other WSGI-compliant frameworks and libraries.
+   Only the ``manage.py`` part has to be changed, the example used here can be
+   used instead.  Django does more or less the exact same thing.
 …
 --------
+`mod_wsgi <http://www.modwsgi.org/>`_ is an attempt to get rid of the low level
+gateways.  As FastCGI, SCGI, mod_python are mostly used to deploy WSGI
+applications anyway, mod_wsgi was started to directly embed WSGI aplications
+into the Apache web server.  The benefit from this approach is that WSGI
+applications can be deployed much easier as is is specially designed to host
+WSGI applications -- unlike the other low level methods which have glue code to
+host WSGI applications (like flup which was mentioned before).  The downside is
+that mod_wsgi is limited to the Apache web server, other servers would need
+`mod_wsgi <http://code.google.com/p/modwsgi/>`_ is an attempt to get rid of the
+low level gateways.  Given that FastCGI, SCGI, and mod_python are mostly used to
+deploy WSGI applications, mod_wsgi was started to directly embed WSGI applications
+into the Apache web server. mod_wsgi is specifically designed to host WSGI
+applications.  It makes the deployment of WSGI applications much easier than
+deployment using other low level methods, which need glue code.  The downside
+is that mod_wsgi is limited to the Apache web server; other servers would need
 their own implementations of mod_wsgi.
 It supports two modes: the embedded mode in which it integrates with the Apache
 process and the daemon mode which is more FastCGI-like.  Contrary to FastCGI,
 mod_wsgi handles the worker-processes by itself which makes administration
+mod_wsgi supports two modes: embedded mode, in which it integrates with the
+Apache process, and daemon mode, which is more FastCGI-like.  Unlike FastCGI,
+mod_wsgi handles the worker-processes by itself, which makes administration
 easier.
 …
 ===============
+WSGI was already mentioned several times so it has to be something important.
+In fact it really is, so now it's time to explain.
+The *Web Server Gateway Interface*, :pep:`333` or WSGI for short is currently
+the best possible way to Python web programming.  While it is great for
+programmers writing frameworks, the normal person does not need to get in direct
+contact with it.  But when choosing a framework for web development it is a good
+idea to take one which supports WSGI.
+The big profit from WSGI is the unification.  When your program is compatible
+with WSGI -- that means that your framework has support for WSGI, your program
+can be deployed on every web server interface for which there are WSGI wrappers.
+So you do not need to care about whether the user uses mod_python or FastCGI --
+with WSGI it just works on any gateway interface.  The Python standard library
+contains its own WSGI server :mod:`wsgiref`, which is a small web server that
+can be used for testing.
+A really great WSGI feature are the middlewares.  Middlewares are layers around
+your program which can add various functionality to it.  There is a `number of
+middlewares <http://wsgi.org/wsgi/Middleware_and_Utilities>`_ already available.
+For example, instead of writing your own session management (to identify a user
+in subsequent requests, as HTTP does not maintain state, so it does now know
+that the requests belong to the same user) you can just take one middleware,
+plug it in and you can rely an already existing functionality.  The same thing
+is compression -- say you want to compress your HTML using gzip, to save your
+server's bandwidth.  So you only need to plug-in a middleware and you're done.
+Authentication is also a problem easily solved using a middleware.
+So, generally -- although WSGI may seem complex, the initial phase of learning
+can be very rewarding as WSGI does already have solutions to many problems that
+might arise while writing web sites.
+WSGI has already been mentioned several times, so it has to be something
+important.  In fact it really is, and now it is time to explain it.
+The *Web Server Gateway Interface*,  or WSGI for short, is defined in
+:pep:`333` and is currently the best way to do Python web programming.  While
+it is great for programmers writing frameworks, a normal web developer does not
+need to get in direct contact with it.  When choosing a framework for web
+development it is a good idea to choose one which supports WSGI.
+The big benefit of WSGI is the unification of the application programming
+interface.  When your program is compatible with WSGI -- which at the outer
+level means that the framework you are using has support for WSGI -- your
+program can be deployed via any web server interface for which there are WSGI
+wrappers.  You do not need to care about whether the application user uses
+mod_python or FastCGI or mod_wsgi -- with WSGI your application will work on
+any gateway interface.  The Python standard library contains its own WSGI
+server, :mod:`wsgiref`, which is a small web server that can be used for
+testing.
+A really great WSGI feature is middleware.  Middleware is a layer around your
+program which can add various functionality to it.  There is quite a bit of
+`middleware <http://www.wsgi.org/en/latest/libraries.html>`_ already
+available.  For example, instead of writing your own session management (HTTP
+is a stateless protocol, so to associate multiple HTTP requests with a single
+user your application must create and manage such state via a session), you can
+just download middleware which does that, plug it in, and get on with coding
+the unique parts of your application.  The same thing with compression -- there
+is existing middleware which handles compressing your HTML using gzip to save
+on your server's bandwidth.  Authentication is another a problem easily solved
+using existing middleware.
+Although WSGI may seem complex, the initial phase of learning can be very
+rewarding because WSGI and the associated middleware already have solutions to
+many problems that might arise while developing web sites.
 …
 The code that is used to connect to various low level gateways like CGI or
 mod_python is called *WSGI server*.  One of these servers is ``flup`` which was
 already mentioned and supports FastCGI, SCGI as well as `AJP
+mod_python is called a *WSGI server*.  One of these servers is ``flup``, which
+supports FastCGI and SCGI, as well as `AJP
 <http://en.wikipedia.org/wiki/Apache_JServ_Protocol>`_.  Some of these servers
 are written in Python as ``flup`` is, but there also exist others which are
+are written in Python, as ``flup`` is, but there also exist others which are
 written in C and can be used as drop-in replacements.
 There are quite a lot of servers already available, so a Python web application
 can be deployed nearly everywhere.  This is one big advantage that Python has
 compared with other web techniques.
 .. seealso::
    A good overview of all WSGI-related code can be found in the `WSGI wiki
    <http://wsgi.org/wsgi>`_, which contains an extensive list of `WSGI servers
    <http://wsgi.org/wsgi/Servers>`_, which can be used by *every* application
+There are many servers already available, so a Python web application
+can be deployed nearly anywhere.  This is one big advantage that Python has
+compared with other web technologies.
+.. seealso::
+   A good overview of WSGI-related code can be found in the `WSGI homepage
+   <http://www.wsgi.org/en/latest/index.html>`_, which contains an extensive list of `WSGI servers
+   <http://www.wsgi.org/en/latest/servers.html>`_ which can be used by *any* application
    supporting WSGI.
 …
 --------------------
+What does WSGI give the web application developer?  Let's take a look on one
+long existing web application written in Python without using WSGI.
+One of the most widely used wiki software is `MoinMoin <http://moinmo.in/>`_.
+It was created in 2000, so it predates WSGI by about three years.  While it now
+includes support for WSGI, older versions needed separate code to run on CGI,
+mod_python, FastCGI and standalone.  Now, this all is possible by using WSGI and
+the already-written gateways.  For running with on FastCGI ``flup`` can be used,
+for running a standalone server :mod:`wsgiref` is the way to go.
+Model-view-controller
+What does WSGI give the web application developer?  Let's take a look at
+an application that's been around for a while, which was written in
+Python without using WSGI.
+One of the most widely used wiki software packages is `MoinMoin
+<http://moinmo.in/>`_.  It was created in 2000, so it predates WSGI by about
+three years.  Older versions needed separate code to run on CGI, mod_python,
+FastCGI and standalone.
+It now includes support for WSGI.  Using WSGI, it is possible to deploy
+MoinMoin on any WSGI compliant server, with no additional glue code.
+Unlike the pre-WSGI versions, this could include WSGI servers that the
+authors of MoinMoin know nothing about.
+Model-View-Controller
 =====================
 The term *MVC* is often heard in statements like "framework *foo* supports MVC".
+While MVC is not really something technical but rather organisational, many web
+frameworks use this model to help the developer to bring structure into his
+program.  Bigger web applications can have lots of code so it is a good idea to
+have structure in the program right from the beginnings.  That way, even users
+of other frameworks (or even languages, as MVC is nothing Python-specific) can
 understand the existing code easier, as they are already familiar with the
 structure.
+The term *MVC* is often encountered in statements such as "framework *foo*
+supports MVC".  MVC is more about the overall organization of code, rather than
+any particular API.  Many web frameworks use this model to help the developer
+bring structure to their program.  Bigger web applications can have lots of
+code, so it is a good idea to have an effective structure right from the beginning.
+That way, even users of other frameworks (or even other languages, since MVC is
+not Python-specific) can easily understand the code, given that they are
+already familiar with the MVC structure.
 MVC stands for three components:
+* The *model*.  This is the data that is meant to modify.  In Python frameworks
+  this component is often represented by the classes used by the
+  object-relational mapper.  So, all declarations go here.
+* The *model*.  This is the data that will be displayed and modified.  In
+  Python frameworks, this component is often represented by the classes used by
+  an object-relational mapper.
 * The *view*.  This component's job is to display the data of the model to the
+  user.  Typically this component is represented by the templates.
+  user.  Typically this component is implemented via templates.
 * The *controller*.  This is the layer between the user and the model.  The
+  controller reacts on user actions (like opening some specific URL) and tells
+  the model to modify the data if necessary.
+  controller reacts to user actions (like opening some specific URL), tells
+  the model to modify the data if necessary, and tells the view code what to
+  display,
 While one might think that MVC is a complex design pattern, in fact it is not.
 …
    (the model) from the user interaction logic (the controller) and the
    templates (the view).  That's why it is important not to write unnecessary
+   Python code in the templates -- it is against MVC and creates more chaos.
+.. seealso::
+   The english Wikipedia has an article about the `Model-View-Controller pattern
+   <http://en.wikipedia.org/wiki/Model-view-controller>`_, which includes a long
+   list of web frameworks for different programming languages.
+Ingredients for web sites
+=========================
+Web sites are complex constructs, so tools were created to help the web site
+developer to make his work maintainable.  None of these tools are in any way
+Python specific, they also exist for other programming languages as well.  Of
+course, developers are not forced to use these tools and often there is no
+"best" tool, but it is worth informing yourself before choosing something
+because of the big number of helpers that the developer can use.
+.. seealso::
+   People have written far more components that can be combined than these
+   presented here.  The Python wiki has a page about these components, called
+   Python code in the templates -- it works against the MVC model and creates
+   chaos in the code base, making it harder to understand and modify.
+.. seealso::
+   The English Wikipedia has an article about the `Model-View-Controller pattern
+   <http://en.wikipedia.org/wiki/Model-view-controller>`_.  It includes a long
+   list of web frameworks for various programming languages.
+Ingredients for Websites
+========================
+Websites are complex constructs, so tools have been created to help web
+developers make their code easier to write and more maintainable.  Tools like
+these exist for all web frameworks in all languages.  Developers are not forced
+to use these tools, and often there is no "best" tool.  It is worth learning
+about the available tools because they can greatly simplify the process of
+developing a web site.
+.. seealso::
+   There are far more components than can be presented here.  The Python wiki
+   has a page about these components, called
    `Web Components <http://wiki.python.org/moin/WebComponents>`_.
 …
 ---------
 Mixing of HTML and Python code is possible with some libraries.  While
+Mixing of HTML and Python code is made possible by a few libraries.  While
 convenient at first, it leads to horribly unmaintainable code.  That's why
 templates exist.  Templates are, in the simplest case, just HTML files with
 placeholders.  The HTML is sent to the user's browser after filling out the
+placeholders.  The HTML is sent to the user's browser after filling in the
 placeholders.
+Python already includes such simple templates::
+    # a simple template
+    template = "<html><body><h1>Hello %s!</h1></body></html>"
+    print template % "Reader"
+The Python standard library also includes some more advanced templates usable
+through :class:`string.Template`, but in HTML templates it is needed to use
+conditional and looping contructs like Python's *for* and *if*.  So, some
+*template engine* is needed.
+Now, Python has a lot of template engines which can be used with or without a
+`framework`_.  Some of these are using a plain-text programming language which
+is very easy to learn as it is quite limited while others use XML so the
+template output is always guaranteed to be valid XML.  Some `frameworks`_ ship
+their own template engine or recommend one particular.  If one is not yet sure,
+using these is a good idea.
+.. note::
+   While Python has quite a lot of different template engines it usually does
+   not make sense to use a homebrewed template system.  The time needed to
+   evaluate all templating systems is not really worth it, better invest the
+   time in looking through the most popular ones.  Some frameworks have their
+   own template engine or have a recommentation for one.  It's wise to use
+   these.
+   Popular template engines include:
+   * Mako
+   * Genshi
+   * Jinja
+.. seealso::
+   Lots of different template engines divide the attention between themselves
+   because it's easy to create them in Python.  The page `Templating
+Python already includes two ways to build simple templates::
+    >>> template = "<html><body><h1>Hello %s!</h1></body></html>"
+    >>> print template % "Reader"
+    <html><body><h1>Hello Reader!</h1></body></html>
+    >>> from string import Template
+    >>> template = Template("<html><body><h1>Hello ${name}</h1></body></html>")
+    >>> print template.substitute(dict(name='Dinsdale'))
+    <html><body><h1>Hello Dinsdale!</h1></body></html>
+To generate complex HTML based on non-trivial model data, conditional
+and looping constructs like Python's *for* and *if* are generally needed.
+*Template engines* support templates of this complexity.
+There are a lot of template engines available for Python which can be used with
+or without a `framework`_.  Some of these define a plain-text programming
+language which is easy to learn, partly because it is limited in scope.
+Others use XML, and the template output is guaranteed to be always be valid
+XML.  There are many other variations.
+Some `frameworks`_ ship their own template engine or recommend one in
+particular.  In the absence of a reason to use a different template engine,
+using the one provided by or recommended by the framework is a good idea.
+Popular template engines include:
+   * `Mako <http://www.makotemplates.org/>`_
+   * `Genshi <http://genshi.edgewall.org/>`_
+   * `Jinja <http://jinja.pocoo.org/2/>`_
+.. seealso::
+   There are many template engines competing for attention, because it is
+   pretty easy to create them in Python.  The page `Templating
    <http://wiki.python.org/moin/Templating>`_ in the wiki lists a big,
+   ever-growing number of these.
+   ever-growing number of these.  The three listed above are considered "second
+   generation" template engines and are a good place to start.
 …
 ----------------
+*Data persistence*, while sounding very complicated is just about storing data.
+This data might be the text of blog entries, the postings of a bulletin board or
+the text of a wiki page.  As always, there are different ways to store
+informations on a web server.
+Often relational database engines like `MySQL <http://www.mysql.com/>`_ or
+`PostgreSQL <http://www.postgresql.org/>`_ are used due to their good
+performance handling very large databases consisting of up to millions of
+entries.  These are *queried* using a language called `SQL
+<http://en.wikipedia.org/wiki/SQL>`_.  Python programmers in general do not like
+SQL too much, they prefer to work with objects.  It is possible to save Python
+objects into a database using a technology called `ORM
+<http://en.wikipedia.org/wiki/Object-relational_mapping>`_.  ORM translates all
+object-oriented access into SQL code under the hood, the user does not need to
+think about it.  Most `frameworks`_ use ORMs and it works quite well.
+A second possibility is using files that are saved on the hard disk (sometimes
+called flatfiles).  This is very easy, but is not too fast.  There is even a
+small database engine called `SQLite <http://www.sqlite.org/>`_ which is bundled
+with Python in the :mod:`sqlite` module and uses only one file.  This database
+can be used to store objects via an ORM and has no other dependencies.  For
+smaller sites SQLite is just enough.  But it is not the only way in which data
+can be saved into the file systems.  Sometimes normal, plain text files are
+enough.
+The third and least used possibility are so-called object oriented databases.
+These databases store the *actual objects* instead of the relations that
+OR-mapping creates between rows in a database.  This has the advantage that
+nearly all objects can be saven in a straightforward way, unlike in relational
+databases where some objects are very hard to represent with ORMs.
+`Frameworks`_ often give the users hints on which method to choose, it is
+usually a good idea to stick to these unless there are some special requirements
+which require to use the one method and not the other.
+*Data persistence*, while sounding very complicated, is just about storing data.
+This data might be the text of blog entries, the postings on a bulletin board or
+the text of a wiki page.  There are, of course, a number of different ways to store
+information on a web server.
+Often, relational database engines like `MySQL <http://www.mysql.com/>`_ or
+`PostgreSQL <http://www.postgresql.org/>`_ are used because of their good
+performance when handling very large databases consisting of millions of
+entries.  There is also a small database engine called `SQLite
+<http://www.sqlite.org/>`_, which is bundled with Python in the :mod:`sqlite3`
+module, and which uses only one file.  It has no other dependencies.  For
+smaller sites SQLite is just enough.
+Relational databases are *queried* using a language called `SQL
+<http://en.wikipedia.org/wiki/SQL>`_.  Python programmers in general do not
+like SQL too much, as they prefer to work with objects.  It is possible to save
+Python objects into a database using a technology called `ORM
+<http://en.wikipedia.org/wiki/Object-relational_mapping>`_ (Object Relational
+Mapping).  ORM translates all object-oriented access into SQL code under the
+hood, so the developer does not need to think about it.  Most `frameworks`_ use
+ORMs, and it works quite well.
+A second possibility is storing data in normal, plain text files (some
+times called "flat files").  This is very easy for simple sites,
+but can be difficult to get right if the web site is performing many
+updates to the stored data.
+A third possibility are object oriented databases (also called "object
+databases").  These databases store the object data in a form that closely
+parallels the way the objects are structured in memory during program
+execution.  (By contrast, ORMs store the object data as rows of data in tables
+and relations between those rows.)  Storing the objects directly has the
+advantage that nearly all objects can be saved in a straightforward way, unlike
+in relational databases where some objects are very hard to represent.
+`Frameworks`_ often give hints on which data storage method to choose.  It is
+usually a good idea to stick to the data store recommended by the framework
+unless the application has special requirements better satisfied by an
+alternate storage mechanism.
 .. seealso::
    * `Persistence Tools <http://wiki.python.org/moin/PersistenceTools>`_ lists
+     possibilities on how to save data in the file system, some of these modules
+     are part of the standard library
+     possibilities on how to save data in the file system.  Some of these
+     modules are part of the standard library
    * `Database Programming <http://wiki.python.org/moin/DatabaseProgramming>`_
+     helps on choosing a method on how to save the data
+   * `SQLAlchemy <http://www.sqlalchemy.org/>`_, the most powerful OR-Mapper for
+     Python and `Elixir <http://elixir.ematia.de/>`_ which makes it easier to
+     use
+     helps with choosing a method for saving data
+   * `SQLAlchemy <http://www.sqlalchemy.org/>`_, the most powerful OR-Mapper
+     for Python, and `Elixir <http://elixir.ematia.de/>`_, which makes
+     SQLAlchemy easier to use
    * `SQLObject <http://www.sqlobject.org/>`_, another popular OR-Mapper
    * `ZODB <https://launchpad.net/zodb>`_ and `Durus
      <http://www.mems-exchange.org/software/durus/>`_, two object oriented
 …
 ==========
+As web sites can easily become quite large, there are so-called frameworks which
+were created to help the developer with making these sites.  Although the most
+well-known framework is Ruby on Rails, Python does also have its own frameworks
+which are partly inspired by Rails or which were existing a long time before
+Rails.
+Two possible approaches to web frameworks exist: the minimalistic approach and
+the all-inclusive approach (somtimes called *full-stack*). Frameworks which are
+all-inclusive give you everything you need to start working, like a template
+engine, some way to save and access data in databases and many features more.
+Most users are best off using these as they are widely used by lots of other
+users and well documented in form of books and tutorials.  Other web frameworks
+go the minimalistic approach trying to be as flexible as possible leaving the
+user the freedom to choose what's best for him.
+The majority of users is best off with all-inclusive framewors.  They bring
+everything along so a user can just jump in and start to code.  While they do
+have some limitations they can fullfill 80% of what one will ever want to
+perfectly.  They consist of various components which are designed to work
+together as good as possible.
+The multitude of web frameworks written in Python demonstrates that it is really
+easy to write one.  One of the most well-known web applications written in
+Python is `Zope <http://www.zope.org/>`_ which can be regarded as some kind of
+big framework.  But Zope was not the only framework, there were some others
+which are by now nearly forgotten.  These do not need to be mentioned anymore,
+because most people that used them moved on to newer ones.
+The process of creating code to run web sites involves writing code to provide
+various services.  The code to provide a particular service often works the
+same way regardless of the complexity or purpose of the web site in question.
+Abstracting these common solutions into reusable code produces what are called
+"frameworks" for web development.  Perhaps the most well-known framework for
+web development is Ruby on Rails, but Python has its own frameworks.  Some of
+these were partly inspired by Rails, or borrowed ideas from Rails, but many
+existed a long time before Rails.
+Originally Python web frameworks tended to incorporate all of the services
+needed to develop web sites as a giant, integrated set of tools.  No two web
+frameworks were interoperable:  a program developed for one could not be
+deployed on a different one without considerable re-engineering work.  This led
+to the development of "minimalist" web frameworks that provided just the tools
+to communicate between the Python code and the http protocol, with all other
+services to be added on top via separate components.  Some ad hoc standards
+were developed that allowed for limited interoperability between frameworks,
+such as a standard that allowed different template engines to be used
+interchangeably.
+Since the advent of WSGI, the Python web framework world has been evolving
+toward interoperability based on the WSGI standard.  Now many web frameworks,
+whether "full stack" (providing all the tools one needs to deploy the most
+complex web sites) or minimalist, or anything in between, are built from
+collections of reusable components that can be used with more than one
+framework.
+The majority of users will probably want to select a "full stack" framework
+that has an active community.  These frameworks tend to be well documented,
+and provide the easiest path to producing a fully functional web site in
+minimal time.
 …
 -----------------------
+There is an incredible number of frameworks, so there is no way to describe them
+all.  It is not even necessary, as most of these frameworks are nothing special
+and everything that can be done with these can also be done with one of the
+popular ones.
+There are an incredible number of frameworks, so they cannot all be covered
+here.  Instead we will briefly touch on some of the most popular.
 …
 `Django <http://www.djangoproject.com/>`_ is a framework consisting of several
 tightly coupled elements which were written from scratch and work together very
 well.  It includes an ORM which is quite powerful while being simple to use and
+has a great online administration interface which makes it possible to edit the
+data in the database with a browser.  The template engine is text-based and is
+designed to be usable for page designers who cannot write Python.  It supports
 so-called template inheritance and filters (which work like Unix pipes).  Django
 has many handy features bundled, like creation of RSS feeds or generic views
 which make it possible to write web sites nearly without any Python code.
 It has a big, international community which has created many sites using Django.
 There are also quite a lot of add-on projects which extend Django's normal
+well.  It includes an ORM which is quite powerful while being simple to use,
+and has a great online administration interface which makes it possible to edit
+the data in the database with a browser.  The template engine is text-based and
+is designed to be usable for page designers who cannot write Python.  It
+supports template inheritance and filters (which work like Unix pipes).  Django
+has many handy features bundled, such as creation of RSS feeds or generic views,
+which make it possible to create web sites almost without writing any Python code.
+It has a big, international community, the members of which have created many
+web sites.  There are also a lot of add-on projects which extend Django's normal
 functionality.  This is partly due to Django's well written `online
 documentation <http://docs.djangoproject.com/>`_ and the `Django book
 …
 .. note::
    Although Django is an MVC-style framework, it calls the components
+   Although Django is an MVC-style framework, it names the elements
    differently, which is described in the `Django FAQ
    <http://www.djangoproject.com/documentation/faq/#django-appears-to-be-a-mvc-framework-but-you-call-the-controller-the-view-and-the-view-the-template-how-come-you-don-t-use-the-standard-names>`_.
+   <http://docs.djangoproject.com/en/dev/faq/general/#django-appears-to-be-a-mvc-framework-but-you-call-the-controller-the-view-and-the-view-the-template-how-come-you-don-t-use-the-standard-names>`_.
 …
 ^^^^^^^^^^
+The other popular web framework in Python is `TurboGears
+<http://www.turbogears.org/>`_.  It takes the approach of using already existing
+components and combining them with glue code to create a seamless experience.
+TurboGears gives the user more flexibility on which components to choose, the
+ORM can be switched between some easy to use but limited and complex but very
+powerful.  Same goes for the template engine.  One strong point about TurboGears
+is that the components that it consists of can be used easily in other projects
+without depending on TurboGears, for example the underlying web server CherryPy.
+Another popular web framework for Python is `TurboGears
+<http://www.turbogears.org/>`_.  TurboGears takes the approach of using already
+existing components and combining them with glue code to create a seamless
+experience.  TurboGears gives the user flexibility in choosing components. For
+example the ORM and template engine can be changed to use packages different
+from those used by default.
 The documentation can be found in the `TurboGears wiki
 …
 published, which is a good starting point.
+The plan for the next major version of TurboGears, version 2.0 is to switch to a
+more flexible base provided by another very flexible web framework called
+`Pylons <http://pylonshq.com/>`_.
+The newest version of TurboGears, version 2.0, moves even further in direction
+of WSGI support and a component-based architecture.  TurboGears 2 is based on
+the WSGI stack of another popular component-based web framework, `Pylons
+<http://pylonshq.com/>`_.
+Zope
+^^^^
+The Zope framework is one of the "old original" frameworks.  Its current
+incarnation in Zope2 is a tightly integrated full-stack framework.  One of its
+most interesting feature is its tight integration with a powerful object
+database called the `ZODB <https://launchpad.net/zodb>`_ (Zope Object Database).
+Because of its highly integrated nature, Zope wound up in a somewhat isolated
+ecosystem:  code written for Zope wasn't very usable outside of Zope, and
+vice-versa.  To solve this problem the Zope 3 effort was started.  Zope 3
+re-engineers Zope as a set of more cleanly isolated components.  This effort
+was started before the advent of the WSGI standard, but there is WSGI support
+for Zope 3 from the `Repoze <http://repoze.org/>`_ project.  Zope components
+have many years of production use behind them, and the Zope 3 project gives
+access to these components to the wider Python community.  There is even a
+separate framework based on the Zope components: `Grok
+<http://grok.zope.org/>`_.
+Zope is also the infrastructure used by the `Plone <http://plone.org/>`_ content
+management system, one of the most powerful and popular content management
+systems available.
 …
 ^^^^^^^^^^^^^^^^^^^^^^^^
+These two are of course not the only frameworks that are available, there are
+also some less-popular frameworks worth mentioning.
+One of these is the already mentioned Zope, which has been around for quite a
+long time.  With Zope 2.x having been known as rather un-pythonic, the newer
+Zope 3.x tries to change that and therefore gets more acceptance from Python
+programmers.  These efforts already showed results, there is a project which
+connects Zope with WSGI called `Repoze <http://repoze.org/>`_ and another
+project called `Grok <http://grok.zope.org/>`_ which makes it possible for
+"normal" Python programmers use the very mature Zope components.
+Of course these are not the only frameworks that are available.  There are
+many other frameworks worth mentioning.
 Another framework that's already been mentioned is `Pylons`_.  Pylons is much
 like TurboGears with ab even stronger emphasis on flexibility, which is bought
+like TurboGears, but with an even stronger emphasis on flexibility, which comes
 at the cost of being more difficult to use.  Nearly every component can be
 exchanged, which makes it necessary to use the documentation of every single
+component, because there are so many Pylons combinations possible that can
+satisfy every requirement.  Pylons builds upon `Paste
+component, of which there are many.  Pylons builds upon `Paste
 <http://pythonpaste.org/>`_, an extensive set of tools which are handy for WSGI.
 …
    Most frameworks also have their own mailing lists and IRC channels, look out
    for these on the projects' websites.  There is also a general "Python in the
+   for these on the projects' web sites.  There is also a general "Python in the
    Web" IRC channel on freenode called `#python.web
    <http://wiki.python.org/moin/PoundPythonWeb>`_.

Context Navigation

Legend:

python/vendor/current/Doc/howto/cporting.rst

python/vendor/current/Doc/howto/curses.rst

python/vendor/current/Doc/howto/doanddont.rst

python/vendor/current/Doc/howto/functional.rst

python/vendor/current/Doc/howto/index.rst

python/vendor/current/Doc/howto/regex.rst

python/vendor/current/Doc/howto/sockets.rst

python/vendor/current/Doc/howto/unicode.rst

python/vendor/current/Doc/howto/urllib2.rst

python/vendor/current/Doc/howto/webservers.rst

Download in other formats: