Archive for the 'Uncategorized' Category


Recombining ZODB storages

I recently faced the task of joining back together a Plone site composed of 4 ZODB filestorages that had been (mostly through cavalier naïveté on my part) split asunder some time ago.

Normally I would probably just do a ZEXP export of each of the folders that lived in its own mountpoint, then remove the mountpoints and reimport the ZEXP files into the main database. However, that wasn’t going to work in this case because the database included some cross-database references.

Some background: Normally in Zope, mountpoints are the only place where one filestorage references another one, but the ZODB has some support for *any* object to link to any other object in any other database, and this can happen within Zope if you copy an object from one filestorage to another. This is generally bad, since the ZODB’s support for cross-database references is partial — when you pack one filestorage, the garbage collection routine doesn’t know about the cross-database references (unless you use zc.zodbdgc), so an object might get removed even if some other filestorage still refers to it, and you’ll get POSKeyErrors. Also, in ZODB 3.7.x, the code that handles packing doesn’t know about cross-database references, so you’ll get KeyError: ‘m’ or KeyError: ‘n’ while packing.

Well, this is what had happened to my multi-database, and I wanted to keep those cross-database references intact while I merged the site back into one monolithic filestorage. So I ended up adapting the ZEXP export code to:

  1. traverse cross-database references (the standard ZEXP export ignores them and will not include objects in different filestorages from the starting object),
  2. traverse ZODB mountpoints (removing them in the process),
  3. and rewrite all the oids to avoid collisions in the new merged database.

Here is the script I ended up with. If you need to use it, you should:

  1. Edit the final line to pass the object you want to start traversing from, and the filename you want to write the ZEXP dump to.
  2. Run the script using bin/instance run multiexport.py
"""Support for export of multidatabases."""

##############################################################################
#
# Based on the ZODB import/export code.
# Copyright (c) 2009 David Glick.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL).  A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE
#
##############################################################################

import logging
import cPickle, cStringIO
from ZODB.utils import p64, u64
from ZODB.ExportImport import export_end_marker
from ZODB.DemoStorage import DemoStorage

logger = logging.getLogger('multiexport')

def export_zexp(self, fname):
    context = self
    f = open(fname, 'wb')
    f.write('ZEXP')
    for oid, p in flatten_multidatabase(context):
        f.writelines((oid, p64(len(p)), p))
    f.write(export_end_marker)
    f.close()

def flatten_multidatabase(context):
    """Walk a multidatabase and yield rewritten pickles with oids for a single database"""
    base_oid = context._p_oid
    base_conn = context._p_jar
    dbs = base_conn.connections
    
    dummy_storage = DemoStorage()

    oids = [(base_conn._db.database_name, base_oid)]
    done_oids = {}
    # table to keep track of mapping old oids to new oids
    ooid_to_oid = {oids[0]: dummy_storage.new_oid()}
    while oids:
        # loop while references remain to objects we haven't exported yet
        (dbname, ooid) = oids.pop(0)
        if (dbname, ooid) in done_oids:
            continue
        done_oids[(dbname, ooid)] = True

        db = dbs[dbname]
        try:
            # get pickle
            p, serial = db._storage.load(ooid, db._version)
        except:
            logger.debug("broken reference for db %s, oid %s", (dbname, repr(ooid)),
                         exc_info=True)
        else:
            def persistent_load(ref):
                """ Remap a persistent id to a new ID and create a ghost for it.
                
                This is called by the unpickler for each reference found.
                """

                # resolve the reference to a database name and oid
                if isinstance(ref, tuple):
                    rdbname, roid = (dbname, ref[0])
                elif isinstance(ref, str):
                    rdbname, roid = (dbname, ref)
                else:
                    try:
                        ref_type, args = ref
                    except ValueError:
                        # weakref
                        return
                    else:
                        if ref_type in ('m', 'n'):
                            rdbname, roid = (args[0], args[1])
                        else:
                            return

                # traverse Products.ZODBMountpoint mountpoints to the mounted location
                rdb = dbs[rdbname]
                p, serial = rdb._storage.load(roid, rdb._version)
                klass = p.split()[0]
                if 'MountedObject' in klass:
                    mountpoint = rdb.get(roid)
                    # get the object with the root as a parent, then unwrap,
                    # since there's no API to get the unwrapped object
                    mounted = mountpoint._getOrOpenObject(app).aq_base
                    rdbname = mounted._p_jar._db.database_name
                    roid = mounted._p_oid

                if roid:
                    print '%s:%s -> %s:%s' % (dbname, u64(ooid), rdbname, u64(roid))
                    oids.append((rdbname, roid))

                try:
                    oid = ooid_to_oid[(rdbname, roid)]
                except KeyError:
                    # generate a new oid and associate it with this old db/oid
                    ooid_to_oid[(rdbname, roid)] = oid = dummy_storage.new_oid()
                return Ghost(oid)

            # do the repickling dance to rewrite references
            
            pfile = cStringIO.StringIO(p)
            unpickler = cPickle.Unpickler(pfile)
            unpickler.persistent_load = persistent_load

            newp = cStringIO.StringIO()
            pickler = cPickle.Pickler(newp, 1)
            pickler.persistent_id = persistent_id

            pickler.dump(unpickler.load())
            pickler.dump(unpickler.load())
            p = newp.getvalue()

            yield ooid_to_oid[(dbname, ooid)], p

class Ghost(object):
    __slots__ = ("oid",)
    def __init__(self, oid):
        self.oid = oid

def persistent_id(obj):
    if isinstance(obj, Ghost):
        return obj.oid

export_zexp(app.mysite, '/tmp/mysite.zexp')

Download multiexport.py

I’ve used this script with apparent success, but it has not been extensively tested and your mileage may of course vary.



on a new place to live

Hello! It’s been a long time since I’ve posted anything…sorry about that.

This past week I’ve been busy getting ready to move to my new apartment, ca. ten blocks south of the VS house. That move finally happened this morning. (Elapsed time from when Meryl arrived with the truck until we had everything unloaded at the new place: about half an hour. Although, keep in mind that the truck had already been half loaded, and the few bits of furniture I own were delivered in a previous trip.) I’ll post more pictures sometime when I’m less exhausted, but here’s one to tide you over. From the covered porch looking southwest out toward West Seattle:

West Seattle from David???s apartment

Isn’t it pretty? And here’s aiming a bit more directly West, toward downtown:

sunset over downtown Seattle

Notice how I strategically failed to show you any of the inside of the apartment in its current state. 🙂

Shout out to all my friends who are also getting settled in new surroundings!



on Ads to Beware Of

And I quote:

Guys Make $500 A Hour

Make $50 in 10 mins.

Err, hello, MATH, people! …I guess maybe you get a $200 bonus for lasting the hour?

There, happy, jrosei? No, seriously, sorry about the lack of bloggage, folks. (Is that a blog jam? Writer’s blog? (pronounce with German accent)) I must have been too caught up in that thing called “real life.” Forgive me. And maybe I’ll even post something substantive real soon now.



on Transition, Anticipation, and Preparation

I arrive in Seattle in forty days. Maybe its time to head out into the wilderness?



on My First Sacred Harp Singing

Today I had a really wonderful experience. I biked 4 miles with Matt and Sol out to the New Testament Baptist Church, for the annual Sacred Harp singing that takes place there. I knew that it would be a good day when I biked the last yards toward the small white church building listening to the powerful chords emanating from it.

Sacred harp singing is a tradition of communal singing that originated in the South of the US in the mid-1800s. It is shape-note singing tradition–each shape on the staff represents a particular note of the scale, and each song is first sung through once naming the notes instead of using the words. There is a also a big emphasis on community and participation–the people present take turns leading each song, and go to lengths to help new people get up to speed. The typical seating arrangement is in a hollow square with one part on each side, all facing the leader in the middle, so this adds to the atmosphere of community. Sacred harp singers can tend to be a little obsessive about this pasttime: they will travel across multiple states for the well-known singings, and note the following picture as well. 🙂

Vanity license plates at the Sacred Harp singing

One thing that was cool about the singing is that there were people from many different backgrounds present–not just Mennonites–all united by their love for the music. Most of the songs were new to me, although I knew a few, felt at home, and did fine sight-reading (thank you J.D. Smucker, Deb Brubaker, and Assembly Mennonite Church).

I’d definitely recommend this tradition to anyone who loves singing in harmony. If you want to know more or find a singing near you (they go on year round), see fasola.org



on Carbon Neutrality

Goshen College just committed to becoming a ‘carbon neutral’ campus and announced the establishment of an Ecological Stewardship Committee to oversee this commitment. I’m so excited!