SQLObject and Database Programming in Python

Ian Bicking

Imaginary Landscape

The Ongoing Example

Address book:

• person:
  • first_name
  • last_name
  • email (many emails for one person)
• email:
  • type
  • address

The DB API

Database programming in Python is based on the DB-API. Some supported databases:

• MySQL
• PostgreSQL (multiple drivers)
• SQLite
• Firebird (Interbase)
• MAXDB (SAP DB)
• Oracle, Sybase, MS SQL Server

DB API Example

Example:

import MySQLdb
conn = MySQLdb.connect(
    db='test', username='ianb')
cur = conn.cursor()
cur.execute(
    "SELECT id, first_name, last_name FROM person")
result = cur.fetchall()
for id, first_name, last_name in result:
    print "%2i %s, %s" % (id, last_name, first_name)

DB API

• Import the database module (MySQLdb, psycopg, sqlite, etc)
• Use module.connect(...) to create a connection.
• Use connection.cursor() to get a cursor. Cursors do all the work.
• Use cursor.execute(sql_query) to run something.
• Use cursor.fetchall() to get results.

DB API problems

1. Connections and cursors are tedious.
2. The databases don't all work the same; multiple SQL dialects.
3. Lots of time spent writing SQL.
4. Database updates can effect all the SQL you've written.
5. Unless you make abstractions...

Abstractions

• Every significant database application has a database abstraction layer.
• You write one whether you mean to or not.

Object-Relational Mappers

The table:

ORMs: Classes

The class:

class Person(SQLObject):
    # id is implicit
    first_name = StringCol()
    last_name = StringCol()

ORMs: Instances

An instance:

>>> john = Person.get(1)
>>> john.first_name
'John'

ORM: summary

• Every table is a class
• Every row is an instance
• Columns become attributes

Usage

__connection__ = 'postgres://pgsql@localhost/test'
class Person(SQLObject):
    first_name = StringCol()
    last_name = StringCol()
    emails = MultipleJoin('Email')

Usage (classes)

class Email(SQLObject):
    person = ForeignKey('Person')
    type = EnumCol(['home', 'work'])
    address = StringCol()

Usage (creating tables)

def createTables():
    for table in (Person, Email):
        table.createTable(ifNotExists=True)

Usage (instances)

>>> john = Person(first_name='John', last_name='Doe')
>>> email = Email(person=john, type='home', 
...     address='john@work.com')
>>> john.emails
[]

Usage (instances)

>>> tom = Person(first_name='Tom', last_name='Jones')
>>> tom is Person.get(tom.id)
True
>>> list(Person.selectBy(first_name='John'))
[]

Defining Classes

• The class name matches the table name
• The attributes match the column names
• (kind of...)

Defining Classes...

You can add extra methods:

class Person(SQLObject):
    ...
    def _get_name(self):
        return self.first_name + ' ' + self.last_name

>>> tom.name
'Tom Jones'

Automatic properties

• _get_attr() methods are called whenever the obj.attr attribute is called
• _set_attr() methods are called whenever the obj.attr = value statement is run
• _set_attr() is optional

Defining classes...

You can override columns:

class Person(SQLObject):
    ...
    last_name_searchable = StringCol()
    def _set_last_name(self, value):
        self._SO_set_last_name(self, value)
        self.last_name_lower = re.sub(r'[^a-zA-Z]', '', value).lower()

Defining classes...

You can fiddle with the naming:

class Person(SQLObject):
    _table = "people"
    first_name = StringCol(dbName="fname")
    ...

Connecting classes

Foreign Keys:

class Email(SQLObject):
    person = ForeignKey('Person')
    ...

Backreferences

The other side of one-to-many:

class Person(SQLObject):
    ...
    emails = MultipleJoin('Email')

Many-to-many

Many to many relationships imply a "hidden" table:

class Address(SQLObject):
    people = RelatedJoin('Person')
    ...
class Person(SQLObject):
    addresses = RelatedJoin('Address')

Many-to-many...

The intermediate table created:

CREATE TABLE address_person (
  address_id INT NOT NULL,
  person_id INT NOT NULL
);

Lazy classes

SQLObject can use reflection to figure out what columns your table has:

class Person(SQLObject):
    _fromDatabase = True

Instances

• Use Person.get(id) to retrieve a row
• Use Person(first_name=...) to insert a row (the new object is returned)
• Use aPerson.destroySelf() to delete a row

Updating

• aPerson.first_name = "new_name"
UPDATE is executed immediately
• Update multiple columns at once:
  aPerson.set(first_name="new_fname", last_name="new_lname")

Selecting

You can use Python expressions (kind of):

query = Person.q.first_name == "Joe"
Person.select(query)

Selecting...

• Class.q.column_name creates a magical object that creates queries.
• sqlrepr() turns these query objects into SQL (mostly hidden)
• sqlrepr(Person.q.first_name == "Joe", 'mysql') creates the SQL person.first_name = 'Joe'

Selecting...

Complicated joins are possible:

Person.select((Person.q.id == Email.q.personID)
              & (Email.q.address.startswith('joe')))

SELECT person.id, person.first_name, person.last_name
FROM person, email
WHERE person.id = email.person_id
      AND email.address LIKE 'joe%'

SelectResult

• You can slice them, and LIMIT and OFFSET statements are added to your query
• You can iterate through them, and avoid loading all objects into memory
• You can do things like select_result.count() to run aggregate functions
• To get a real list, you must do list(select_result)

Caching

• SQLObject caches everything
• Maybe too much (bad if you have multiple updaters)
• Needs to because there's no joins
• Has potential to be faster than ad hoc queries
• Thread safety or thread confusion?

Performance

• Not good if you are dealing with lots and lots of rows
• Inserts and selects substantially slower
• But for many applications you'll pay that price sooner or later anyway

The problem with ORMs

ORMs live in the world between Object Oriented programmers (and programs) and Relational programmers (and programs). Neither will be happy.

ORM Mismatch

• Python classes aren't tables
• Python instances aren't rows

ORM Mismatch: Classes

• Databases don't have inheritance
• Even when they do, no one uses it
• Some tables are too boring (e.g., intermediate tables for many-to-many relationships), or don't warrant the overhead

ORM Mismatch: Instances

• Instances get garbage collected, rows are forever
• Rows exist even when instances don't
• Instances may still exist when the row is gone
• In the relational calculus, rows don't have real identity

ORM Mismatch: Relations

• Things like joins don't make sense for objects, but are central to relations
• Views and stored procedures also don't make sense
• Aggregate functions and set operations aren't natural in Python (the for loop is natural)

Solving the question by avoidance

• SQLObject doesn't let you forget SQL
• You still have to think relationally
• But many relational concepts don't work either
• Advantage: it's easy

Solving the Mismatch

• There exist database restrictions; not every database schema maps well. (But most are fine)
• Sometimes you have to do things more slowly or more imperatively, compared to a purely relational technique
• There exist object restrictions; not every object-oriented concept maps well; e.g., no inheritance
• Inheritance isn't planned; this isn't transparent persistence

Some other alternatives

• SQLObject doesn't try to be too OO, so it works well with normal database schemas
• SQLObject doesn't have a compile step
• SQLObject isn't just a SQL wrapper
• SQLObject is reasonably complete

Other alternatives...

See the Python Wiki at: python.org/moin
The page is HigherLevelDatabaseProgramming.