Meta-Classes in Python

Dynamic OOP and Meta-Classes in Python

Guy Wiener

Dept. of Applied Math and Computer ScienceWeizmann Institute for Science

24/5/2011

Guy Wiener (WIS) Dynamic OOP and Meta-Classes in Python 24/5/2011 1 / 57

Introduction

Outline

1 Introduction

2 OOP Hacking

3 Meta-Classes

4 Examples

5 Summary


Introduction

What is OOP?

The OOP Manifest (abridged)Alan Kay, 1993

Everything is an object

Computation is performed by objects communicating with eachother by sending and receiving messages

Each object has its own memory (state), which consists of otherobjects.

Every object is an instance of a class.

The class is the repository for behavior. All instances of thesame class can perform the same actions.

Classes are organized into a singly rooted tree, called theinheritance hierarchy.


Introduction

What is an object?

An object, according to the OOP manifest:State Memory

Behavior Receiving and sending messages

Identity Each object is unique


Introduction

Objects Can Change!

Changing objectsState Read, write, add and remove field

Behavior Create, add and replace methods

Identity Remains the same


Introduction

Static Objects and Classes

Objects and Classes

Field descriptors

Method desc. and code

Hierarchy information

Class

Fieldscontent

ObjectRun-timetype infor-mation

Example

x: int

y: String

foo() {. . . }

goo() {. . . }

A

5

“hello”

Instance of A


Introduction

Dynamic Objects and Classes

Objects and Classes

A dictionaryof methodsand staticfields

Class

A dictionary offields and boundmethods

Object

Classpointer

Example

“foo” ⇒ {. . . }

“goo” ⇒ {. . . }

A

“x” ⇒ 5

“y” ⇒ “hello”

Instance of A


OOP Hacking

Outline

1 Introduction

2 OOP Hacking

3 Meta-Classes

4 Examples

5 Summary


OOP Hacking

Everything is a Dictionary

Python’s modules, classes and objects = Dictionaries.

The dictionary holds:Classes Methods and class variables

Objects Bound methods and fields

The “.” operator = Access the inner dictionary

The dir command = Lists keys of the inner dictionary


OOP Hacking

Manipulating the Inner Dictionary

The inner dictionary is accessible and manipulable by string keys!

Inner dictionary manipulationgetattr(o, ’a’) Get the attribute a from the dictionary of o

setattr(o, ’a’, v) Set the attribute a of o to be v

obj.foo is same as getattr(obj, ’foo’)

obj.foo = val is same as setattr(obj, ’foo’, val)


OOP Hacking

Manipulating the Inner Dictionary (cont’d)

Example>>> a = A()

>>> a.x = 5

>>> dir(a)

[..., ’x’]

>>> getattr(a, ’x’)

5

>>> setattr(a, ’x’, 6)

>>> a.x

6


OOP Hacking

Adding Fields

Adding fields to an objectclass A: pass

>>> a = A()

>>> a.x = 5

>>> a.x

5

>>> setattr(a, ’y’, 6)

>>> a.y

6

Gain & LossGained Flexible object content

Lost Type safety


OOP Hacking

Adding Fields

Adding fields to an objectclass A: pass

>>> a = A()

>>> a.x = 5

>>> a.x

5

>>> setattr(a, ’y’, 6)

>>> a.y

6

Gain & LossGained Flexible object content

Lost Type safety


OOP Hacking

Replacing Methods

Swapping methodsclass A:

def f(self):

print "f"

def g(self):

print "g"

>>> a = A()

>>> a.f, a.g = a.g, a.f

>>> a.f()

g

>>> a.g()

f

Gain & LossGained Change an object

behavior at runtime

Lost Control flow analysis


OOP Hacking

Replacing Methods

Swapping methodsclass A:

def f(self):

print "f"

def g(self):

print "g"

>>> a = A()

>>> a.f, a.g = a.g, a.f

>>> a.f()

g

>>> a.g()

f

Gain & LossGained Change an object

behavior at runtime

Lost Control flow analysis


OOP Hacking

Replacing Methods for a Class

Swapping class methodsclass A:

def f(self):

print "f"

def g(self):

print "g"

>>> a = A()

>>> A.f, A.g = A.g, A.f

>>> a.f()

g

>>> a.g()

f

Gain & LossGained Change a class at

runtime

Lost Sanity


OOP Hacking

Replacing Methods for a Class

Swapping class methodsclass A:

def f(self):

print "f"

def g(self):

print "g"

>>> a = A()

>>> A.f, A.g = A.g, A.f

>>> a.f()

g

>>> a.g()

f

Gain & LossGained Change a class at

runtime

Lost Sanity


OOP Hacking

Classes are Return Values Too

A function that returns a classdef PersonClass(salut):

class Person(object):

def __init__(self, name):

self.name = name

def greet(self):

print salut, self.name

return Person


OOP Hacking

Classes are Return Values Too (cont’d)

Making classesMan = PersonClass("Mr.")

Woman = PersonClass("Ms.")

Doctor = PersonClass("Dr.")

a = [Woman("Rose Tylar"),

Woman("Amy Pond"),

Man("Mickey Smith"),

Man("Rory Williams"),

Doctor("Who")]

Outputfor c in a: c.greet()

>>> Ms. Rose Tylar

Ms. Amy Pond

Mr. Mickey Smith

Mr. Rory Williams

Dr. Who


Meta-Classes

Outline

1 Introduction

2 OOP Hacking

3 Meta-Classes

4 Examples

5 Summary


Meta-Classes

What is Meta-Programming?

Meta-ProgramA program that:

One of its inputs is a program (possibly itself)

Its output is a program


Meta-Classes

What is Meta-OOP?

Meta-ClassA class that creates classes

Objects are instances of classes

Classes are objects

Classes are instances of meta-classes


Meta-Classes

Object Systems

3-levels

Object

A Class Type

super: Type[]

An Object

Legend

instance ofextends

4-levels

Object

Type

super: Type[]

A MetaClass

A Class

An Object


Meta-Classes Meta-Classes in Python

Outline

1 Introduction

2 OOP Hacking

3 Meta-ClassesMeta-Classes in Python

4 ExamplesLoggingCountingDelegationAssociations

5 Summary



Meta-Classes in Python

The type classBase class for meta-classes

Creates new classes

Dynamic members become static members of the instance class

type. init (cls, name, bases, dict)

cls The class object itself (i.e., “self”)

name The name of the class

bases A list of the base classes

dict A dictionary of the methods and static fields of the class



Creating Classes from Meta-Classes

Declaring a new class programatically# Meta-Class

class Printable(type):

def whoami(self): print "I am ", self.__name__

>>> Foo = Printable(’Foo’,(),{}) # Empty new class

>>> Foo.whoami()

I am Foo

>>> Foo.__class__

<class ’Printable’>

>>> f = Foo() # Object

>>> f.__class__

<class ’Foo’>



The metaclass field

Class C is an instance of a meta-class M if:1 C has a static field metaclass

2 One of the ancestors classes of C is an instance of M

3 There is a global variable metaclass

4 Otherwise, the default meta-class type is used



metaclass Example

Declaring a meta-class with metaclassclass Bar:

__metaclass__ = Printable

def foo(self): print ’foo’

>>> Bar.whoami()

I am a Bar

>>> b = Bar()

>>> b.foo()

foo


Examples

Outline

1 Introduction

2 OOP Hacking

3 Meta-Classes

4 Examples

5 Summary


Examples Logging

Outline

1 Introduction

2 OOP Hacking



5 Summary


Examples Logging

Logging with Meta-Classes

A logger decoratordef log(name, f):

def ret(*args):

print "enter", name

f(*args)

print "exit", name

return ret


Examples Logging

Logging with Meta-Classes (cont’d)

A logger meta-classclass Logged(type):

def init (cls, name, bases, dict):

type. init (cls, name, bases, dict)

p = re.compile(cls. logmatch )

for attr, item in dict.items():

if callable(item) and p.match(attr):

setattr(cls, attr, log(attr, item))


Examples Logging

Logging with Meta-Classes (cont’d)

A logged classclass Test:

metaclass = Logged

logmatch = ’.*’

def foo(self):

print ’foo’

>>> t = Test()

>>> t.foo()

enter foo

foo

exit foo


Examples Counting

Outline

1 Introduction

2 OOP Hacking



5 Summary


Examples Counting

Counting Instances

Countable meta-classclass Countable(type):

def new_init(cls, init):

def ret(self, *args):

init(self, *args)

cls.count = cls.count + 1

return ret

def __init__(cls, name, bases, dct):

type.__init__(cls, name, bases, dct)

cls.count = 0

cls.__init__ = cls.new_init(cls.__init__)


Examples Counting

Counting Example

Some classesclass Person:

__metaclass__ = Countable

def __init__(self, name):

self.name = name

class Book:

__metaclass__ = Countable

def __init__(self, name, author):

self.name = name

self.author = author


Examples Counting

Counting Example (cont’d)

Testingrah = Person("Robert Anson Heinlin")

b1 = Book("Stranger in a Strange Land", rah)

b2 = Book("Starship Troopers", rah)

print Person.count, Book.count

>>> 1 2


Examples Delegation

Outline

1 Introduction

2 OOP Hacking



5 Summary


Examples Delegation

Automatic Delegation

Delegate

An object of class A thatdispatches all message ofclass B to a target field oftype B.

Writing a delegate class is a monotonous work

But it can be done automatically


Examples Delegation

Delegation Using Meta-Classes

A delegation function decoratordef dlgt(cls, method):


method(self.__tgt__, *args)

return instancemethod(ret, None, cls)

instancemethod takes a function, an object/None and a class and

returns a method of the class

Auxiliary – Print class namedef clsname(self):

return self.__class__.__name__


Examples Delegation

Delegation Using Meta-Classes (cont’d)

The Delegate meta-classclass Delegate(type):

def __init__(cls, name, bases, dict):

type.__init__(cls, name, bases, dict)

tgtclass = cls.__tgtclass__

for name in dir(tgtclass):

val = getattr(tgtclass, name)

if callable(val):

setattr(cls, name, dlgt(cls, val))


Examples Delegation


The delegated classclass A:

def bar(self):

print clsname(self), ’bar’

def baz(self):

print clsname(self), ’baz’


Examples Delegation


The delegating classclass B:

metaclass = Delegate

tgtclass = A

def init (self, tgt):

self. tgt = tgt

def boo(self):

print clsname(self), ’boo’


Examples Delegation


Delegation test>>> b = B(A())

>>> b.bar()

A bar

>>> b.baz()

A baz

>>> b.boo()

B boo


Examples Delegation

Simplified Syntax for with Meta-Classes with Args.Thanks to Yoav Goldberg

Use a function to create the meta-classdef delegate_of(tgtclass):

class Delegate(type):

def __init__(cls, name, bases, dict):

type.__init__(cls, name, bases, dict)

for name in dir(tgtclass):

val = getattr(tgtclass, name)

if callable(val):

setattr(cls, name, dlgt(cls,val))


Examples Delegation

Simplified Syntax for Meta-Classes with Args.Thanks to Yoav Goldberg

Use the function to generate a meta-classclass B:

metaclass = delegate_of(A)

def init (self, tgt):

self. tgt = tgt

def boo(self):

print clsname(self), ’boo’


Examples Associations

Outline

1 Introduction

2 OOP Hacking



5 Summary



Associations

A Bab

myA

0..m

myB

0..n

GoalsAggregation operations, like

a.getMyB().foreach().g()r = b.getMyA().filter().f().map().h()

Optimizations: Multithreading, caching

No textual code generation



Basic Aggregation

Replace all public methods with an aggregator wrapperclass Aggr(type):

def __init__(cls, name, bases, dct):

p = re.compile(’__.*__’)

type.__init__(cls, name, bases, {})

for key, item in dct.iteritems():

if callable(item) and not p.match(key):

m = instancemethod(cls.wrap(item), None, self)

setattr(self, key, m)

def wrap(cls, func):


return self.__aggr__(func, *args)

return ret



Basic Aggregation (cont’d)

Make aggrs. that extend impl and wrap ifce

class ListAggr(Aggr):

@staticmethod

def make(name, impl, ifce):

dct = {}

for key in dir(ifce):

dct[key] = getattr(ifce, key)

return ListAggr(name, (impl,), dct)

def __init__(self, name, bases, dct):

Aggr.__init__(self, name, bases, dct)

base = bases[0]

ctr = base.__init__ % Use impl’s constructor

self.__init__ = instancemethod(ctr, None, self)



List-Based Sequential Aggrs.

Store list in private fieldclass ListFunc(object):

def __init__(self, data):

self.__data__ = data

def onAdd(self, other): pass

def onRemove(self, other): pass



List-Based Sequential Aggrs. (cont’d)

Sequential Foreachclass ListSeqForeach(ListFunc):


ListFunc.__init__(self, data)

def __aggr__(self, func, *args):

for inst in self.__data__:

func(inst, *args)



List-Based Sequential Aggrs. (cont’d)

Sequential Mapclass ListSeqMap(ListFunc):




ret = []

for inst in self.__data__:

ret.append(func(inst, *args))

return ret



Binding It Together

Making an aggr. that extends seq. foreachdef makeForeachClass(cls):

return ListAggr.make(’...’, ListSeqForeach, cls)

ListFunc

ListSeqForeach

Aggr

ListAggr

ForeachClasscls



Optimizations

Multithreaded Foreachclass ListSpawnForeach(ListFunc):




n = max(len(self.__data__)/K, 1)

ts = []

for c in chunks(self.__data__, n):

t = ForeachThread(c, func, args)

ts.append(t)

t.start()

for tr in ts: tr.join()



Binding It Together, Multithreaded Version

Making an aggr. that extends parallel foreachdef makeForeachClass(cls):

return ListAggr.make(’...’, ListSpawnForeach, cls)

ListFunc

ListSpawnForeach

Aggr

ListAggr

ForeachClasscls



Links

Download complete codehttp://www.cs.bgu.ac.il/~gwiener/software/associations/

Read full paper

Proceedings of ENASE’2010 / By request


http://www.cs.bgu.ac.il/~gwiener/software/associations/

Summary

Outline

1 Introduction

2 OOP Hacking

3 Meta-Classes

4 Examples

5 Summary


Summary

Advantages of Meta-Classes

Turn this. . .

Dai-U

ka-HakalTru-Saka-Uka

-Wek

uU

ru-N

ecro

s Phaarazu-Urha-Ytahi

Ia

Into this


Summary

Thank You!

[email protected]


[email protected]

Documents

Meta-Classes in Python