Changeset 391 for python/trunk/Doc/library/tokenize.rst

Timestamp:

Mar 19, 2014, 11:31:01 PM (11 years ago)

Author:

dmik

Message:

python: Merge vendor 2.7.6 to trunk.

Location:

python/trunk

Files:

• . (modified) (1 prop)
• Doc/library/tokenize.rst (modified) (4 diffs)

python/trunk/Doc/library/tokenize.rst

@@ -1 @@
-
 :mod:`tokenize` --- Tokenizer for Python source
 ===============================================
@@ -8 +7 @@
 .. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org>
 
+**Source code:** :source:`Lib/tokenize.py`
+
+--------------
 
 The :mod:`tokenize` module provides a lexical scanner for Python source code,
@@ -13 +15 @@
 well, making it useful for implementing "pretty-printers," including colorizers
 for on-screen displays.
+
+To simplify token stream handling, all :ref:`operators` and :ref:`delimiters`
+tokens are returned using the generic :data:`token.OP` token type.  The exact
+type can be determined by checking the token ``string`` field on the
+:term:`named tuple` returned from :func:`tokenize.tokenize` for the character
+sequence that identifies a specific operator token.
 
 The primary entry point is a :term:`generator`:
@@ -22 +30 @@
    :meth:`readline` method of built-in file objects (see section
    :ref:`bltin-file-objects`).  Each call to the function should return one line
-   of input as a string.
+   of input as a string. Alternately, *readline* may be a callable object that
+   signals completion by raising :exc:`StopIteration`.
 
    The generator produces 5-tuples with these members: the token type; the token