Functions
def	group

def	any

def	maybe

Variables
string	__author__ = 'Ka-Ping Yee <ping@lfw.org>'

	__credits__ = \

list	__all__ = [x for x in dir(token) if x[0] != '_']

	COMMENT = N_TOKENS

int	NL = N_TOKENS+1

string	Whitespace = r'[ \f\t]*'

string	Comment = r'#[^\r\n]*'

tuple	Ignore = Whitespace+any(r'\\\r?\n' + Whitespace)

string	Name = r'[a-zA-Z_]\w*'

string	Hexnumber = r'0[xX][\da-fA-F]*[lL]?'

string	Octnumber = r'0[0-7]*[lL]?'

string	Decnumber = r'[1-9]\d*[lL]?'

tuple	Intnumber = group(Hexnumber, Octnumber, Decnumber)

string	Exponent = r'[eE][-+]?\d+'

tuple	Pointfloat = group(r'\d+\.\d*', r'\.\d+')

string	Expfloat = r'\d+'

tuple	Floatnumber = group(Pointfloat, Expfloat)

tuple	Imagnumber = group(r'\d+[jJ]', Floatnumber + r'[jJ]')

tuple	Number = group(Imagnumber, Floatnumber, Intnumber)

string	Single = r"[^'\\](?:\\.[^'\\])*'"

string	Double = r'[^"\\](?:\\.[^"\\])*"'

string	Single3 = r"[^'\\](?:(?:\\.\|'(?!''))[^'\\])*'''"

string	Double3 = r'[^"\\](?:(?:\\.\|"(?!""))[^"\\])*"""'

Detailed Description

Tokenization help for Python programs.

generate_tokens(readline) is a generator that breaks a stream of
text into Python tokens.  It accepts a readline-like method which is called
repeatedly to get the next line of input (or "" for EOF).  It generates
5-tuples with these members:

the token type (see token.py)
the token (a string)
the starting (row, column) indices of the token (a 2-tuple of ints)
the ending (row, column) indices of the token (a 2-tuple of ints)
the original line (string)

It is designed to match the working of the Python tokenizer exactly, except
that it produces COMMENT tokens for comments and gives type OP for all
operators

Older entry points
tokenize_loop(readline, tokeneater)
tokenize(readline, tokeneater=printtoken)
are the same, except instead of generating tokens, tokeneater is a callback
function to which the 5 fields described above are passed as 5 arguments,
each time a new token is found.

Function Documentation

def tokenize.any ( choices)

Definition at line 45 of file tokenize.py.

45

def any(*choices): return apply(group, choices) + '*'

Functions

Variables

Detailed Description

Function Documentation

Variable Documentation