source: python/vendor/Python-2.7.6/Demo/metaclasses/index.html

Last change on this file was 2, checked in by Yuri Dario, 15 years ago

Initial import for vendor code.
  • Property svn:eol-style set to native
File size: 20.1 KB
Line 
1<HTML>
2
3<HEAD>
4<TITLE>Metaclasses in Python 1.5</TITLE>
5</HEAD>
6
7<BODY BGCOLOR="FFFFFF">
8
9<H1>Metaclasses in Python 1.5</H1>
10<H2>(A.k.a. The Killer Joke :-)</H2>
11
12<HR>
13
14(<i>Postscript:</i> reading this essay is probably not the best way to
15understand the metaclass hook described here. See a <A
16HREF="meta-vladimir.txt">message posted by Vladimir Marangozov</A>
17which may give a gentler introduction to the matter. You may also
18want to search Deja News for messages with "metaclass" in the subject
19posted to comp.lang.python in July and August 1998.)
20
21<HR>
22
23<P>In previous Python releases (and still in 1.5), there is something
24called the ``Don Beaudry hook'', after its inventor and champion.
25This allows C extensions to provide alternate class behavior, thereby
26allowing the Python class syntax to be used to define other class-like
27entities. Don Beaudry has used this in his infamous <A
28HREF="http://maigret.cog.brown.edu/pyutil/">MESS</A> package; Jim
29Fulton has used it in his <A
30HREF="http://www.digicool.com/releases/ExtensionClass/">Extension
31Classes</A> package. (It has also been referred to as the ``Don
32Beaudry <i>hack</i>,'' but that's a misnomer. There's nothing hackish
33about it -- in fact, it is rather elegant and deep, even though
34there's something dark to it.)
35
36<P>(On first reading, you may want to skip directly to the examples in
37the section "Writing Metaclasses in Python" below, unless you want
38your head to explode.)
39
40<P>
41
42<HR>
43
44<P>Documentation of the Don Beaudry hook has purposefully been kept
45minimal, since it is a feature of incredible power, and is easily
46abused. Basically, it checks whether the <b>type of the base
47class</b> is callable, and if so, it is called to create the new
48class.
49
50<P>Note the two indirection levels. Take a simple example:
51
52<PRE>
53class B:
54 pass
55
56class C(B):
57 pass
58</PRE>
59
60Take a look at the second class definition, and try to fathom ``the
61type of the base class is callable.''
62
63<P>(Types are not classes, by the way. See questions 4.2, 4.19 and in
64particular 6.22 in the <A
65HREF="http://www.python.org/cgi-bin/faqw.py" >Python FAQ</A>
66for more on this topic.)
67
68<P>
69
70<UL>
71
72<LI>The <b>base class</b> is B; this one's easy.<P>
73
74<LI>Since B is a class, its type is ``class''; so the <b>type of the
75base class</b> is the type ``class''. This is also known as
76types.ClassType, assuming the standard module <code>types</code> has
77been imported.<P>
78
79<LI>Now is the type ``class'' <b>callable</b>? No, because types (in
80core Python) are never callable. Classes are callable (calling a
81class creates a new instance) but types aren't.<P>
82
83</UL>
84
85<P>So our conclusion is that in our example, the type of the base
86class (of C) is not callable. So the Don Beaudry hook does not apply,
87and the default class creation mechanism is used (which is also used
88when there is no base class). In fact, the Don Beaudry hook never
89applies when using only core Python, since the type of a core object
90is never callable.
91
92<P>So what do Don and Jim do in order to use Don's hook? Write an
93extension that defines at least two new Python object types. The
94first would be the type for ``class-like'' objects usable as a base
95class, to trigger Don's hook. This type must be made callable.
96That's why we need a second type. Whether an object is callable
97depends on its type. So whether a type object is callable depends on
98<i>its</i> type, which is a <i>meta-type</i>. (In core Python there
99is only one meta-type, the type ``type'' (types.TypeType), which is
100the type of all type objects, even itself.) A new meta-type must
101be defined that makes the type of the class-like objects callable.
102(Normally, a third type would also be needed, the new ``instance''
103type, but this is not an absolute requirement -- the new class type
104could return an object of some existing type when invoked to create an
105instance.)
106
107<P>Still confused? Here's a simple device due to Don himself to
108explain metaclasses. Take a simple class definition; assume B is a
109special class that triggers Don's hook:
110
111<PRE>
112class C(B):
113 a = 1
114 b = 2
115</PRE>
116
117This can be though of as equivalent to:
118
119<PRE>
120C = type(B)('C', (B,), {'a': 1, 'b': 2})
121</PRE>
122
123If that's too dense for you, here's the same thing written out using
124temporary variables:
125
126<PRE>
127creator = type(B) # The type of the base class
128name = 'C' # The name of the new class
129bases = (B,) # A tuple containing the base class(es)
130namespace = {'a': 1, 'b': 2} # The namespace of the class statement
131C = creator(name, bases, namespace)
132</PRE>
133
134This is analogous to what happens without the Don Beaudry hook, except
135that in that case the creator function is set to the default class
136creator.
137
138<P>In either case, the creator is called with three arguments. The
139first one, <i>name</i>, is the name of the new class (as given at the
140top of the class statement). The <i>bases</i> argument is a tuple of
141base classes (a singleton tuple if there's only one base class, like
142the example). Finally, <i>namespace</i> is a dictionary containing
143the local variables collected during execution of the class statement.
144
145<P>Note that the contents of the namespace dictionary is simply
146whatever names were defined in the class statement. A little-known
147fact is that when Python executes a class statement, it enters a new
148local namespace, and all assignments and function definitions take
149place in this namespace. Thus, after executing the following class
150statement:
151
152<PRE>
153class C:
154 a = 1
155 def f(s): pass
156</PRE>
157
158the class namespace's contents would be {'a': 1, 'f': &lt;function f
159...&gt;}.
160
161<P>But enough already about writing Python metaclasses in C; read the
162documentation of <A
163HREF="http://maigret.cog.brown.edu/pyutil/">MESS</A> or <A
164HREF="http://www.digicool.com/papers/ExtensionClass.html" >Extension
165Classes</A> for more information.
166
167<P>
168
169<HR>
170
171<H2>Writing Metaclasses in Python</H2>
172
173<P>In Python 1.5, the requirement to write a C extension in order to
174write metaclasses has been dropped (though you can still do
175it, of course). In addition to the check ``is the type of the base
176class callable,'' there's a check ``does the base class have a
177__class__ attribute.'' If so, it is assumed that the __class__
178attribute refers to a class.
179
180<P>Let's repeat our simple example from above:
181
182<PRE>
183class C(B):
184 a = 1
185 b = 2
186</PRE>
187
188Assuming B has a __class__ attribute, this translates into:
189
190<PRE>
191C = B.__class__('C', (B,), {'a': 1, 'b': 2})
192</PRE>
193
194This is exactly the same as before except that instead of type(B),
195B.__class__ is invoked. If you have read <A HREF=
196"http://www.python.org/cgi-bin/faqw.py?req=show&file=faq06.022.htp"
197>FAQ question 6.22</A> you will understand that while there is a big
198technical difference between type(B) and B.__class__, they play the
199same role at different abstraction levels. And perhaps at some point
200in the future they will really be the same thing (at which point you
201would be able to derive subclasses from built-in types).
202
203<P>At this point it may be worth mentioning that C.__class__ is the
204same object as B.__class__, i.e., C's metaclass is the same as B's
205metaclass. In other words, subclassing an existing class creates a
206new (meta)inststance of the base class's metaclass.
207
208<P>Going back to the example, the class B.__class__ is instantiated,
209passing its constructor the same three arguments that are passed to
210the default class constructor or to an extension's metaclass:
211<i>name</i>, <i>bases</i>, and <i>namespace</i>.
212
213<P>It is easy to be confused by what exactly happens when using a
214metaclass, because we lose the absolute distinction between classes
215and instances: a class is an instance of a metaclass (a
216``metainstance''), but technically (i.e. in the eyes of the python
217runtime system), the metaclass is just a class, and the metainstance
218is just an instance. At the end of the class statement, the metaclass
219whose metainstance is used as a base class is instantiated, yielding a
220second metainstance (of the same metaclass). This metainstance is
221then used as a (normal, non-meta) class; instantiation of the class
222means calling the metainstance, and this will return a real instance.
223And what class is that an instance of? Conceptually, it is of course
224an instance of our metainstance; but in most cases the Python runtime
225system will see it as an instance of a a helper class used by the
226metaclass to implement its (non-meta) instances...
227
228<P>Hopefully an example will make things clearer. Let's presume we
229have a metaclass MetaClass1. It's helper class (for non-meta
230instances) is callled HelperClass1. We now (manually) instantiate
231MetaClass1 once to get an empty special base class:
232
233<PRE>
234BaseClass1 = MetaClass1("BaseClass1", (), {})
235</PRE>
236
237We can now use BaseClass1 as a base class in a class statement:
238
239<PRE>
240class MySpecialClass(BaseClass1):
241 i = 1
242 def f(s): pass
243</PRE>
244
245At this point, MySpecialClass is defined; it is a metainstance of
246MetaClass1 just like BaseClass1, and in fact the expression
247``BaseClass1.__class__ == MySpecialClass.__class__ == MetaClass1''
248yields true.
249
250<P>We are now ready to create instances of MySpecialClass. Let's
251assume that no constructor arguments are required:
252
253<PRE>
254x = MySpecialClass()
255y = MySpecialClass()
256print x.__class__, y.__class__
257</PRE>
258
259The print statement shows that x and y are instances of HelperClass1.
260How did this happen? MySpecialClass is an instance of MetaClass1
261(``meta'' is irrelevant here); when an instance is called, its
262__call__ method is invoked, and presumably the __call__ method defined
263by MetaClass1 returns an instance of HelperClass1.
264
265<P>Now let's see how we could use metaclasses -- what can we do
266with metaclasses that we can't easily do without them? Here's one
267idea: a metaclass could automatically insert trace calls for all
268method calls. Let's first develop a simplified example, without
269support for inheritance or other ``advanced'' Python features (we'll
270add those later).
271
272<PRE>
273import types
274
275class Tracing:
276 def __init__(self, name, bases, namespace):
277 """Create a new class."""
278 self.__name__ = name
279 self.__bases__ = bases
280 self.__namespace__ = namespace
281 def __call__(self):
282 """Create a new instance."""
283 return Instance(self)
284
285class Instance:
286 def __init__(self, klass):
287 self.__klass__ = klass
288 def __getattr__(self, name):
289 try:
290 value = self.__klass__.__namespace__[name]
291 except KeyError:
292 raise AttributeError, name
293 if type(value) is not types.FunctionType:
294 return value
295 return BoundMethod(value, self)
296
297class BoundMethod:
298 def __init__(self, function, instance):
299 self.function = function
300 self.instance = instance
301 def __call__(self, *args):
302 print "calling", self.function, "for", self.instance, "with", args
303 return apply(self.function, (self.instance,) + args)
304
305Trace = Tracing('Trace', (), {})
306
307class MyTracedClass(Trace):
308 def method1(self, a):
309 self.a = a
310 def method2(self):
311 return self.a
312
313aninstance = MyTracedClass()
314
315aninstance.method1(10)
316
317print "the answer is %d" % aninstance.method2()
318</PRE>
319
320Confused already? The intention is to read this from top down. The
321Tracing class is the metaclass we're defining. Its structure is
322really simple.
323
324<P>
325
326<UL>
327
328<LI>The __init__ method is invoked when a new Tracing instance is
329created, e.g. the definition of class MyTracedClass later in the
330example. It simply saves the class name, base classes and namespace
331as instance variables.<P>
332
333<LI>The __call__ method is invoked when a Tracing instance is called,
334e.g. the creation of aninstance later in the example. It returns an
335instance of the class Instance, which is defined next.<P>
336
337</UL>
338
339<P>The class Instance is the class used for all instances of classes
340built using the Tracing metaclass, e.g. aninstance. It has two
341methods:
342
343<P>
344
345<UL>
346
347<LI>The __init__ method is invoked from the Tracing.__call__ method
348above to initialize a new instance. It saves the class reference as
349an instance variable. It uses a funny name because the user's
350instance variables (e.g. self.a later in the example) live in the same
351namespace.<P>
352
353<LI>The __getattr__ method is invoked whenever the user code
354references an attribute of the instance that is not an instance
355variable (nor a class variable; but except for __init__ and
356__getattr__ there are no class variables). It will be called, for
357example, when aninstance.method1 is referenced in the example, with
358self set to aninstance and name set to the string "method1".<P>
359
360</UL>
361
362<P>The __getattr__ method looks the name up in the __namespace__
363dictionary. If it isn't found, it raises an AttributeError exception.
364(In a more realistic example, it would first have to look through the
365base classes as well.) If it is found, there are two possibilities:
366it's either a function or it isn't. If it's not a function, it is
367assumed to be a class variable, and its value is returned. If it's a
368function, we have to ``wrap'' it in instance of yet another helper
369class, BoundMethod.
370
371<P>The BoundMethod class is needed to implement a familiar feature:
372when a method is defined, it has an initial argument, self, which is
373automatically bound to the relevant instance when it is called. For
374example, aninstance.method1(10) is equivalent to method1(aninstance,
37510). In the example if this call, first a temporary BoundMethod
376instance is created with the following constructor call: temp =
377BoundMethod(method1, aninstance); then this instance is called as
378temp(10). After the call, the temporary instance is discarded.
379
380<P>
381
382<UL>
383
384<LI>The __init__ method is invoked for the constructor call
385BoundMethod(method1, aninstance). It simply saves away its
386arguments.<P>
387
388<LI>The __call__ method is invoked when the bound method instance is
389called, as in temp(10). It needs to call method1(aninstance, 10).
390However, even though self.function is now method1 and self.instance is
391aninstance, it can't call self.function(self.instance, args) directly,
392because it should work regardless of the number of arguments passed.
393(For simplicity, support for keyword arguments has been omitted.)<P>
394
395</UL>
396
397<P>In order to be able to support arbitrary argument lists, the
398__call__ method first constructs a new argument tuple. Conveniently,
399because of the notation *args in __call__'s own argument list, the
400arguments to __call__ (except for self) are placed in the tuple args.
401To construct the desired argument list, we concatenate a singleton
402tuple containing the instance with the args tuple: (self.instance,) +
403args. (Note the trailing comma used to construct the singleton
404tuple.) In our example, the resulting argument tuple is (aninstance,
40510).
406
407<P>The intrinsic function apply() takes a function and an argument
408tuple and calls the function for it. In our example, we are calling
409apply(method1, (aninstance, 10)) which is equivalent to calling
410method(aninstance, 10).
411
412<P>From here on, things should come together quite easily. The output
413of the example code is something like this:
414
415<PRE>
416calling &lt;function method1 at ae8d8&gt; for &lt;Instance instance at 95ab0&gt; with (10,)
417calling &lt;function method2 at ae900&gt; for &lt;Instance instance at 95ab0&gt; with ()
418the answer is 10
419</PRE>
420
421<P>That was about the shortest meaningful example that I could come up
422with. A real tracing metaclass (for example, <A
423HREF="#Trace">Trace.py</A> discussed below) needs to be more
424complicated in two dimensions.
425
426<P>First, it needs to support more advanced Python features such as
427class variables, inheritance, __init__ methods, and keyword arguments.
428
429<P>Second, it needs to provide a more flexible way to handle the
430actual tracing information; perhaps it should be possible to write
431your own tracing function that gets called, perhaps it should be
432possible to enable and disable tracing on a per-class or per-instance
433basis, and perhaps a filter so that only interesting calls are traced;
434it should also be able to trace the return value of the call (or the
435exception it raised if an error occurs). Even the Trace.py example
436doesn't support all these features yet.
437
438<P>
439
440<HR>
441
442<H1>Real-life Examples</H1>
443
444<P>Have a look at some very preliminary examples that I coded up to
445teach myself how to write metaclasses:
446
447<DL>
448
449<DT><A HREF="Enum.py">Enum.py</A>
450
451<DD>This (ab)uses the class syntax as an elegant way to define
452enumerated types. The resulting classes are never instantiated --
453rather, their class attributes are the enumerated values. For
454example:
455
456<PRE>
457class Color(Enum):
458 red = 1
459 green = 2
460 blue = 3
461print Color.red
462</PRE>
463
464will print the string ``Color.red'', while ``Color.red==1'' is true,
465and ``Color.red + 1'' raise a TypeError exception.
466
467<P>
468
469<DT><A NAME=Trace></A><A HREF="Trace.py">Trace.py</A>
470
471<DD>The resulting classes work much like standard
472classes, but by setting a special class or instance attribute
473__trace_output__ to point to a file, all calls to the class's methods
474are traced. It was a bit of a struggle to get this right. This
475should probably redone using the generic metaclass below.
476
477<P>
478
479<DT><A HREF="Meta.py">Meta.py</A>
480
481<DD>A generic metaclass. This is an attempt at finding out how much
482standard class behavior can be mimicked by a metaclass. The
483preliminary answer appears to be that everything's fine as long as the
484class (or its clients) don't look at the instance's __class__
485attribute, nor at the class's __dict__ attribute. The use of
486__getattr__ internally makes the classic implementation of __getattr__
487hooks tough; we provide a similar hook _getattr_ instead.
488(__setattr__ and __delattr__ are not affected.)
489(XXX Hm. Could detect presence of __getattr__ and rename it.)
490
491<P>
492
493<DT><A HREF="Eiffel.py">Eiffel.py</A>
494
495<DD>Uses the above generic metaclass to implement Eiffel style
496pre-conditions and post-conditions.
497
498<P>
499
500<DT><A HREF="Synch.py">Synch.py</A>
501
502<DD>Uses the above generic metaclass to implement synchronized
503methods.
504
505<P>
506
507<DT><A HREF="Simple.py">Simple.py</A>
508
509<DD>The example module used above.
510
511<P>
512
513</DL>
514
515<P>A pattern seems to be emerging: almost all these uses of
516metaclasses (except for Enum, which is probably more cute than useful)
517mostly work by placing wrappers around method calls. An obvious
518problem with that is that it's not easy to combine the features of
519different metaclasses, while this would actually be quite useful: for
520example, I wouldn't mind getting a trace from the test run of the
521Synch module, and it would be interesting to add preconditions to it
522as well. This needs more research. Perhaps a metaclass could be
523provided that allows stackable wrappers...
524
525<P>
526
527<HR>
528
529<H2>Things You Could Do With Metaclasses</H2>
530
531<P>There are lots of things you could do with metaclasses. Most of
532these can also be done with creative use of __getattr__, but
533metaclasses make it easier to modify the attribute lookup behavior of
534classes. Here's a partial list.
535
536<P>
537
538<UL>
539
540<LI>Enforce different inheritance semantics, e.g. automatically call
541base class methods when a derived class overrides<P>
542
543<LI>Implement class methods (e.g. if the first argument is not named
544'self')<P>
545
546<LI>Implement that each instance is initialized with <b>copies</b> of
547all class variables<P>
548
549<LI>Implement a different way to store instance variables (e.g. in a
550list kept outside the instance but indexed by the instance's id())<P>
551
552<LI>Automatically wrap or trap all or certain methods
553
554<UL>
555
556<LI>for tracing
557
558<LI>for precondition and postcondition checking
559
560<LI>for synchronized methods
561
562<LI>for automatic value caching
563
564</UL>
565<P>
566
567<LI>When an attribute is a parameterless function, call it on
568reference (to mimic it being an instance variable); same on assignment<P>
569
570<LI>Instrumentation: see how many times various attributes are used<P>
571
572<LI>Different semantics for __setattr__ and __getattr__ (e.g. disable
573them when they are being used recursively)<P>
574
575<LI>Abuse class syntax for other things<P>
576
577<LI>Experiment with automatic type checking<P>
578
579<LI>Delegation (or acquisition)<P>
580
581<LI>Dynamic inheritance patterns<P>
582
583<LI>Automatic caching of methods<P>
584
585</UL>
586
587<P>
588
589<HR>
590
591<H4>Credits</H4>
592
593<P>Many thanks to David Ascher and Donald Beaudry for their comments
594on earlier draft of this paper. Also thanks to Matt Conway and Tommy
595Burnette for putting a seed for the idea of metaclasses in my
596mind, nearly three years ago, even though at the time my response was
597``you can do that with __getattr__ hooks...'' :-)
598
599<P>
600
601<HR>
602
603</BODY>
604
605</HTML>
Note: See TracBrowser for help on using the repository browser.