Chapter 4. The Power Of Introspection¶

Example 4.1. apihelper.py¶

If you have not already done so, you can download this and other examples ( http://diveintopython.org/download/diveintopython-examples-5.4.zip) used in this book.

def info(object, spacing=10, collapse=1): (1) (2) (3)
    """Print methods and doc strings.

    Takes module, class, list, dictionary, or string."""
    methodList = [method for method in dir(object) if callable(getattr(object, method))]
    processFunc = collapse and (lambda s: " ".join(s.split())) or (lambda s: s)
    print "\n".join(["%s %s" %
                      (method.ljust(spacing),
                       processFunc(str(getattr(object, method).__doc__)))
                     for method in methodList])

if __name__ == "__main__":                (4) (5)
    print info.__doc__

1. This module has one function, info. According to its function declaration, it takes three parameters: object, spacing, and collapse. The last two are actually optional parameters, as you’ll see shortly.
2. The info function has a multi-line doc string that succinctly describes the function’s purpose. Note that no return value is mentioned; this function will be used solely for its effects, rather than its value.
3. Code within the function is indented.
4. The if __name__ trick allows this program do something useful when run by itself, without interfering with its use as a module for other programs. In this case, the program simply prints out the doc string of the info function.
5. if statements use == for comparison, and parentheses are not required.

The info function is designed to be used by you, the programmer, while working in the Python IDE. It takes any object that has functions or methods (like a module, which has functions, or a list, which has methods) and prints out the functions and their doc strings.

Example 4.2. Sample Usage of apihelper.py¶

>>> from apihelper import info
>>> li = []
>>> info(li)
append     L.append(object) -- append object to end
count      L.count(value) -> integer -- return number of occurrences of value
extend     L.extend(list) -- extend list by appending list elements
index      L.index(value) -> integer -- return index of first occurrence of value
insert     L.insert(index, object) -- insert object before index
pop        L.pop([index]) -> item -- remove and return item at index (default last)
remove     L.remove(value) -- remove first occurrence of value
reverse    L.reverse() -- reverse *IN PLACE*
sort       L.sort([cmpfunc]) -- sort *IN PLACE*; if given, cmpfunc(x, y) -> -1, 0, 1

By default the output is formatted to be easy to read. Multi-line doc strings are collapsed into a single long line, but this option can be changed by specifying 0 for the collapse argument. If the function names are longer than 10 characters, you can specify a larger value for the spacing argument to make the output easier to read.

Example 4.3. Advanced Usage of apihelper.py¶

>>> import odbchelper
>>> info(odbchelper)
buildConnectionString Build a connection string from a dictionary Returns string.
>>> info(odbchelper, 30)
buildConnectionString          Build a connection string from a dictionary Returns string.
>>> info(odbchelper, 30, 0)
buildConnectionString          Build a connection string from a dictionary

Returns string.

Example 4.4. Valid Calls of info¶

info(odbchelper)                    (1)
info(odbchelper, 12)                (2)
info(odbchelper, collapse=0)        (3)
info(spacing=15, object=odbchelper) (4)

1. With only one argument, spacing gets its default value of 10 and collapse gets its default value of 1.
2. With two arguments, collapse gets its default value of 1.
3. Here you are naming the collapse argument explicitly and specifying its value. spacing still gets its default value of 10.
4. Even required arguments (like object, which has no default value) can be named, and named arguments can appear in any order.

This looks totally whacked until you realize that arguments are simply a dictionary. The “normal” method of calling functions without argument names is actually just a shorthand where Python matches up the values with the argument names in the order they’re specified in the function declaration. And most of the time, you’ll call functions the “normal” way, but you always have the additional flexibility if you need it.

Note: Calling Functions is Flexible The only thing you need to do to call a function is specify a value (somehow) for each required argument; the manner and order in which you do that is up to you.

Further Reading on Optional Arguments

• Python Tutorial (http://www.python.org/doc/current/tut/tut.html) discusses exactly when and how default arguments are evaluated (http:// www.python.org/doc/current/tut/node6.html#SECTION006710000000000000000), which matters when the default value is a list or an expression with side effects.

4.3.1. The type Function¶

The type function returns the datatype of any arbitrary object. The possible types are listed in the types module. This is useful for helper functions that can handle several types of data.

>>> type(1)           (1)
<type 'int'>
>>> li = []
>>> type(li)          (2)
<type 'list'>
>>> import odbchelper
>>> type(odbchelper)  (3)
<type 'module'>
>>> import types      (4)
>>> type(odbchelper) == types.ModuleType
True

4.3.2. The str Function¶

The str coerces data into a string. Every datatype can be coerced into a string.

>>> str(1)          (1)
'1'
>>> horsemen = ['war', 'pestilence', 'famine']
>>> horsemen
['war', 'pestilence', 'famine']
>>> horsemen.append('Powerbuilder')
>>> str(horsemen)   (2)
"['war', 'pestilence', 'famine', 'Powerbuilder']"
>>> str(odbchelper) (3)
"<module 'odbchelper' from 'c:\\docbook\\dip\\py\\odbchelper.py'>"
>>> str(None)       (4)
'None'

>>> li = []
>>> dir(li)           (1)
['append', 'count', 'extend', 'index', 'insert',
'pop', 'remove', 'reverse', 'sort']
>>> d = {}
>>> dir(d)            (2)
['clear', 'copy', 'get', 'has_key', 'items', 'keys', 'setdefault', 'update', 'values']
>>> import odbchelper
>>> dir(odbchelper)   (3)
['__builtins__', '__doc__', '__file__', '__name__', 'buildConnectionString']

>>> import string
>>> string.punctuation           (1)
'!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~'
>>> string.join                  (2)
<function join at 00C55A7C>
>>> callable(string.punctuation) (3)
False
>>> callable(string.join)        (4)
True
>>> print string.join.__doc__    (5)
join(list [,sep]) -> string

Return a string composed of the words in list, with
intervening occurrences of sep.  The default separator is a
single space.

(joinfields and join are synonymous)

4.3.3. Built-In Functions¶

type, str, dir, and all the rest of Python’s built-in functions are grouped into a special module called __builtin__. (That’s two underscores before and after.) If it helps, you can think of Python automatically executing from __builtin__ import * on startup, which imports all the “built-in” functions into the namespace so you can use them directly.

The advantage of thinking like this is that you can access all the built-in functions and attributes as a group by getting information about the __builtin__ module. And guess what, Python has a function called info. Try it yourself and skim through the list now. We’ll dive into some of the more important functions later. (Some of the built-in error classes, like AttributeError, should already look familiar.)

>>> from apihelper import info
>>> import __builtin__
>>> info(__builtin__, 20)
ArithmeticError      Base class for arithmetic errors.
AssertionError       Assertion failed.
AttributeError       Attribute not found.
EOFError             Read beyond end of file.
EnvironmentError     Base class for I/O related errors.
Exception            Common base class for all exceptions.
FloatingPointError   Floating point operation failed.
IOError              I/O operation failed.

Example 4.10. Introducing getattr¶

>>> li = ["Larry", "Curly"]
>>> li.pop                       (1)
<built-in method pop of list object at 010DF884>
>>> getattr(li, "pop")           (2)
<built-in method pop of list object at 010DF884>
>>> getattr(li, "append")("Moe") (3)
>>> li
["Larry", "Curly", "Moe"]
>>> getattr({}, "clear")         (4)
<built-in method clear of dictionary object at 00F113D4>
>>> getattr((), "pop")           (5)
Traceback (innermost last):
  File "<interactive input>", line 1, in ?
AttributeError: 'tuple' object has no attribute 'pop'

1. This gets a reference to the pop method of the list. Note that this is not calling the pop method; that would be li.pop(). This is the method itself.
2. This also returns a reference to the pop method, but this time, the method name is specified as a string argument to the getattr function. getattr is an incredibly useful built-in function that returns any attribute of any object. In this case, the object is a list, and the attribute is the pop method.
3. In case it hasn’t sunk in just how incredibly useful this is, try this: the return value of getattr is the method, which you can then call just as if you had said li.append(“Moe”) directly. But you didn’t call the function directly; you specified the function name as a string instead.
4. getattr also works on dictionaries.
5. In theory, getattr would work on tuples, except that tuples have no methods , so getattr will raise an exception no matter what attribute name you give.

4.4.1. getattr with Modules¶

getattr isn’t just for built-in datatypes. It also works on modules.

>>> import odbchelper
>>> odbchelper.buildConnectionString             (1)
<function buildConnectionString at 00D18DD4>
>>> getattr(odbchelper, "buildConnectionString") (2)
<function buildConnectionString at 00D18DD4>
>>> object = odbchelper
>>> method = "buildConnectionString"
>>> getattr(object, method)                      (3)
<function buildConnectionString at 00D18DD4>
>>> type(getattr(object, method))                (4)
<type 'function'>
>>> import types
>>> type(getattr(object, method)) == types.FunctionType
True
>>> callable(getattr(object, method))            (5)
True

4.4.2. getattr As a Dispatcher¶

A common usage pattern of getattr is as a dispatcher. For example, if you had a program that could output data in a variety of different formats, you could define separate functions for each output format and use a single dispatch function to call the right one.

For example, let’s imagine a program that prints site statistics in HTML, XML, and plain text formats. The choice of output format could be specified on the command line, or stored in a configuration file. A statsout module defines three functions, output_html, output_xml, and output_text. Then the main program defines a single output function, like this:

import statsout

def output(data, format="text"):                              (1)
    output_function = getattr(statsout, "output_%s" % format) (2)
    return output_function(data)                              (3)

import statsout

def output(data, format="text"):
    output_function = getattr(statsout, "output_%s" % format, statsout.output_text)
    return output_function(data) (1)

Example 4.14. Introducing List Filtering¶

>>> li = ["a", "mpilgrim", "foo", "b", "c", "b", "d", "d"]
>>> [elem for elem in li if len(elem) > 1]       (1)
['mpilgrim', 'foo']
>>> [elem for elem in li if elem != "b"]         (2)
['a', 'mpilgrim', 'foo', 'c', 'd', 'd']
>>> [elem for elem in li if li.count(elem) == 1] (3)
['a', 'mpilgrim', 'foo', 'c']

1. The mapping expression here is simple (it just returns the value of each element), so concentrate on the filter expression. As Python loops through the list, it runs each element through the filter expression. If the filter expression is true, the element is mapped and the result of the mapping expression is included in the returned list. Here, you are filtering out all the one-character strings, so you’re left with a list of all the longer strings.
2. Here, you are filtering out a specific value, b. Note that this filters all occurrences of b, since each time it comes up, the filter expression will be false.
3. count is a list method that returns the number of times a value occurs in a list. You might think that this filter would eliminate duplicates from a list, returning a list containing only one copy of each value in the original list. But it doesn’t, because values that appear twice in the original list (in this case, b and d) are excluded completely. There are ways of eliminating duplicates from a list, but filtering is not the solution.

Let’s get back to this line from apihelper.py:

methodList = [method for method in dir(object) if callable(getattr(object, method))]

This looks complicated, and it is complicated, but the basic structure is the

same. The whole filter expression returns a list, which is assigned to the methodList variable. The first half of the expression is the list mapping part. The mapping expression is an identity expression, which it returns the value of each element. dir(object) returns a list of object’s attributes and methods – that’s the list you’re mapping. So the only new part is the filter expression after the if.

The filter expression looks scary, but it’s not. You already know about callable, getattr, and in. As you saw in the previous section, the expression getattr(object, method) returns a function object if object is a module and method is the name of a function in that module.

So this expression takes an object (named object). Then it gets a list of the names of the object’s attributes, methods, functions, and a few other things. Then it filters that list to weed out all the stuff that you don’t care about. You do the weeding out by taking the name of each attribute/method/function and getting a reference to the real thing, via the getattr function. Then you check to see if that object is callable, which will be any methods and functions, both built-in (like the pop method of a list) and user-defined (like the buildConnectionString function of the odbchelper module). You don’t care about other attributes, like the __name__ attribute that’s built in to every module.

Further Reading on Filtering Lists

• Python Tutorial (http://www.python.org/doc/current/tut/tut.html) discusses another way to filter lists using the built-in filter function ( http://www.python.org/doc/current/tut/node7.html# SECTION007130000000000000000).

Example 4.15. Introducing and¶

>>> 'a' and 'b'         (1)
'b'
>>> '' and 'b'          (2)
''
>>> 'a' and 'b' and 'c' (3)
'c'

1. When using and, values are evaluated in a boolean context from left to right. 0, ‘’, [], (), {}, and None are false in a boolean context; everything else is true. Well, almost everything. By default, instances of classes are true in a boolean context, but you can define special methods in your class to make an instance evaluate to false. You’ll learn all about classes and special methods in Chapter 5. If all values are true in a boolean context, and returns the last value. In this case, and evaluates ‘a’, which is true, then ‘b’, which is true, and returns ‘b’.
2. If any value is false in a boolean context, and returns the first false value. In this case, ‘’ is the first false value.
3. All values are true, so and returns the last value, ‘c’.

Example 4.16. Introducing or¶

>>> 'a' or 'b'          (1)
'a'
>>> '' or 'b'           (2)
'b'
>>> '' or [] or {}      (3)
{}
>>> def sidefx():
...     print "in sidefx()"
...     return 1
>>> 'a' or sidefx()     (4)
'a'

1. When using or, values are evaluated in a boolean context from left to right, just like and. If any value is true, or returns that value immediately. In this case, ‘a’ is the first true value.
2. or evaluates ‘’, which is false, then ‘b’, which is true, and returns ‘b’.
3. If all values are false, or returns the last value. or evaluates ‘’, which is false, then [], which is false, then {}, which is false, and returns {}.
4. Note that or evaluates values only until it finds one that is true in a boolean context, and then it ignores the rest. This distinction is important if some values can have side effects. Here, the function sidefx is never called, because or evaluates ‘a’, which is true, and returns ‘a’ immediately.

If you’re a C hacker, you are certainly familiar with the bool ? a : b expression, which evaluates to a if bool is true, and b otherwise. Because of the way and and or work in Python, you can accomplish the same thing.

4.6.1. Using the and-or Trick¶

>>> a = "first"
>>> b = "second"
>>> 1 and a or b (1)
'first'
>>> 0 and a or b (2)
'second'

>>> a = ""
>>> b = "second"
>>> 1 and a or b         (1)
'second'

>>> a = ""
>>> b = "second"
>>> (1 and [a] or [b])[0] (1)
''

Example 4.20. Introducing lambda Functions¶

>>> def f(x):
...     return x*2
...
>>> f(3)
6
>>> g = lambda x: x*2  (1)
>>> g(3)
6
>>> (lambda x: x*2)(3) (2)
6

1. This is a lambda function that accomplishes the same thing as the normal function above it. Note the abbreviated syntax here: there are no parentheses around the argument list, and the return keyword is missing (it is implied, since the entire function can only be one expression). Also, the function has no name, but it can be called through the variable it is assigned to.
2. You can use a lambda function without even assigning it to a variable. This may not be the most useful thing in the world, but it just goes to show that a lambda is just an in-line function.

To generalize, a lambda function is a function that takes any number of arguments (including optional arguments) and returns the value of a single expression. lambda functions can not contain commands, and they can not contain more than one expression. Don’t try to squeeze too much into a lambda function; if you need something more complex, define a normal function instead and make it as long as you want.

Note: lambda is Optional lambda functions are a matter of style. Using them is never required; anywhere you could use them, you could define a separate normal function and use that instead. I use them in places where I want to encapsulate specific, non-reusable code without littering my code with a lot of little one-line functions.

4.7.1. Real-World lambda Functions¶

Here are the lambda functions in apihelper.py:

processFunc = collapse and (lambda s: " ".join(s.split())) or (lambda s: s)

Notice that this uses the simple form of the and-or trick, which is okay,

because a lambda function is always true in a boolean context. (That doesn’t mean that a lambda function can’t return a false value. The function is always true; its return value could be anything.)

Also notice that you’re using the split function with no arguments. You’ve already seen it used with one or two arguments, but without any arguments it splits on whitespace.

>>> s = "this   is\na\ttest"  (1)
>>> print s
this   is
a       test
>>> print s.split()           (2)
['this', 'is', 'a', 'test']
>>> print " ".join(s.split()) (3)
'this is a test'

processFunc = collapse and (lambda s: " ".join(s.split())) or (lambda s: s)

Example 4.22. Getting a doc string Dynamically¶

>>> import odbchelper
>>> object = odbchelper                   (1)
>>> method = 'buildConnectionString'      (2)
>>> getattr(object, method)               (3)
<function buildConnectionString at 010D6D74>
>>> print getattr(object, method).__doc__ (4)
Build a connection string from a dictionary of parameters.

Returns string.

1. In the info function, object is the object you’re getting help on, passed in as an argument.
2. As you’re looping through methodList, method is the name of the current method.
3. Using the getattr function, you’re getting a reference to the method function in the object module.
4. Now, printing the actual doc string of the method is easy.

The next piece of the puzzle is the use of str around the doc string. As you may recall, str is a built-in function that coerces data into a string. But a doc string is always a string, so why bother with the str function? The answer is that not every function has a doc string, and if it doesn’t, its __doc__ attribute is None.

Example 4.23. Why Use str on a doc string?¶

>>> >>> def foo(): print 2
>>> >>> foo()
2
>>> >>> foo.__doc__     (1)
>>> foo.__doc__ == None (2)
True
>>> str(foo.__doc__)    (3)
'None'

1. You can easily define a function that has no doc string, so its __doc__ attribute is None. Confusingly, if you evaluate the __doc__ attribute directly, the Python IDE prints nothing at all, which makes sense if you think about it, but is still unhelpful.

2. You can verify that the value of the __doc__ attribute is actually None by comparing it directly.

3. The str function takes the null value and returns a string representation of it, ‘None’.

   Note: Python vs. SQL: null value comparisons In SQL, you must use IS NULL instead of = NULL to compare a null value. In Python, you can use either == None or is None, but is None is faster.

Now that you are guaranteed to have a string, you can pass the string to processFunc, which you have already defined as a function that either does or doesn’t collapse whitespace. Now you see why it was important to use str to convert a None value into a string representation. processFunc is assuming a string argument and calling its split method, which would crash if you passed it None because None doesn’t have a split method.

Stepping back even further, you see that you’re using string formatting again to concatenate the return value of processFunc with the return value of method’s ljust method. This is a new string method that you haven’t seen before.

Example 4.24. Introducing ljust¶

>>> s = 'buildConnectionString'
>>> s.ljust(30) (1)
'buildConnectionString         '
>>> s.ljust(20) (2)
'buildConnectionString'

1. ljust pads the string with spaces to the given length. This is what the info function uses to make two columns of output and line up all the doc strings in the second column.
2. If the given length is smaller than the length of the string, ljust will simply return the string unchanged. It never truncates the string.

You’re almost finished. Given the padded method name from the ljust method and the (possibly collapsed) doc string from the call to processFunc, you concatenate the two and get a single string. Since you’re mapping methodList, you end up with a list of strings. Using the join method of the string “n”, you join this list into a single string, with each element of the list on a separate line, and print the result.

Example 4.25. Printing a List¶

>>> li = ['a', 'b', 'c']
>>> print "\n".join(li) (1)
a
b
c

1. This is also a useful debugging trick when you’re working with lists. And in Python, you’re always working with lists.

That’s the last piece of the puzzle. You should now understand this code.

print “n”.join([“%s %s” %

(method.ljust(spacing),

processFunc(str(getattr(object, method).__doc__)))

System Message: WARNING/2 (/home/gerard/environments/sphinx-0.5-with-patch/thehazeltree/source/diveintopython/4.rst, line 1144)

Block quote ends without a blank line; unexpected unindent.

for method in methodList])