zodb.py3migrate

Reason for this package

It is not possible to use a ZODB database with both Python 2 and Python 3. There are the following reasons for this behavior:

  1. Given you have a database created using Python 2. When reading this database using Python 3 all string objects will be read as Python 3 string objects (aka unicode) requiring them to be ASCII encoded. If actual binary data is stored in such a str object its class has to inherit zodbpickle.binary, so it will be unpickled as binary.
  2. Because of reason #1 ZODB 4.x used different magic bytes (the first bytes of the database file) for the different Python versions and does not allow to open a database with a Python version which does not match the magic bytes.

Upgrade workflow

  1. Install zodb.py3migrate into your project, so it has access to your persistent classes when reading objects from the ZODB.

  2. Run the script using:

    bin/zodb-py3migrate-analyze path/to/Data.fs
    
    • The script shows a list of attributes and keys grouped by their class that need to be converted because they are binary strings with an encoding different from ASCII. You have to decide for each attribute if it really is a binary string (e. g. an image), or if it should be text, i. e. converted to unicode.

      • Attributes that should remain binary strings must become an instance of zodbpickle.binary so ZODB knows that this string are to be treated as binary data.
      • Since ZODB will read all non-marked binary strings as unicode, all text fields to unicode and adjust the class accordingly.
    • Your ZODB is Python 3 ready if the script does not output any attribute, since all binary strings were either converted to unicode or marked as binary.

    • Note

      The displayed total number of objects in the ZODB is only an approximation as returned by the FileStorage API.

  3. Convert binary attributes in your code base to Python 3.

    • Mark actual binary attributes with zodbpickle.binary. This way they will become compatible with Python 2 and Python 3, so you can continue using Python 2 without writing incompatible data.
    • If you only need Python 3 compatibility it is sufficient to use bytes.
  4. Convert the PythonScript objects inside the ZODB to be Python 3 compatible code. (Find all of them using the Find tab in the ZMI.)

  5. Convert existing data in the ZODB for Python 3 compatibility by calling:

    bin/zodb-py3migrate-convert path/to/Data.fs --config=convert.ini
    
    • Warning

      This call changes the database file in place, so only call it on a copy of your live ZODB.

    • Example for conversion config file, i. e. convert.ini in example call above.

      [zodbpickle.binary]
      foo.bar.Baz.image
      
      [utf-8]
      foo.bar.Baz.text
      foo.bar.Baz.title
      BTrees.OOBTree.OOBTree['7b6d22fa-594e']
      persistent.mapping.PersistentMapping['number-1']
      
      [latin-1]
      foo.bar.Baz.legacy
      persistent.list.PersistentList[1]
      
    • All entries of the section [zodbpickle.binary] will be wrapped into zodbpickle.binary, whereas all entries of other sections will be decoded to unicode using the section name as encoding.

    • You might use the result of the analyse call described in step #2 to build the config file.

    • To convert a value inside a BTree or in a PersistentMapping, the repr() of the key must be given in square brackets. (See example above.)

    • To convert a value inside a PersistentList, the index of the value must be given in square brackets. (See example above.)

    • Note

      • The conversion only changes values, but not keys, since this would break the application code.
      • Converting values in a BTree.*.*TreeSet is not (yet) supported.
      • Converting values nested in non-persistent objects is not supported.
    • After converting binary strings to zodbpickle.binary, your application needs the zodbpickle package as install dependency. Otherwise the converted objects will be broken.

  6. Convert the magic bytes in the database file to claim Python 3 compatibility using:

    bin/zodb-py3migrate-magic path/to/Data.fs Python3
    

(Use Python2 as second argument to revert back to Python 2.)

Warning

This call changes the database file in place, so only call it on a copy of your live ZODB.

Example calls

Analyze the ZODB:

$ bin/zodb-py3migrate-analyze path/to/Data.fs
Found 3 binary fields: (number of occurrences)
  foo.bar.Baz.image is string (12)
  foo.bar.Baz.legacy is iterable (1)
  foo.bar.Baz.text is string (23)

Convert the ZODB:

$ bin/zodb-py3migrate-convert path/to/Data.fs --config=convert.ini
Converted 2 binary fields: (number of occurrences)
  foo.bar.Baz.image (12)
  foo.bar.Baz.text (23)