Blobstorage errors when implementing Zope Replication Services (ZRS) on Plone 4.3.2

Question

I would like to use ZRS to replicate data between two servers (zopeserver-1 and zopeserver-2), with the ZEO on zopeserver-1 being primary, and replicating to the secondary ZEO on zopeserver-2. Each server will have two Zope clients, both pointing to the primary ZEO on zopeserver-1.

After running into problems with blobs when trying to make our existing configuration work with ZRS, I created vanilla Plone 4.3.2 instances on two servers to verify that I experienced the same issues. The non-vanilla parts of the buildout.cfg are:

Primary

eggs =
    ...
    zc.zrs

[zeoserver]
<= zeoserver_base
recipe = plone.recipe.zeoserver[zrs]
zeo-address = 8100
replicate-to = 5000

[client1]
<= client_base
recipe = plone.recipe.zope2instance
zeo-address = zopeserver-1:${zeoserver:zeo-address}
http-address = 8081

[client2]
<= client_base
recipe = plone.recipe.zope2instance
zeo-address = zopeserver-1:${zeoserver:zeo-address}
http-address = 8082

Secondary

eggs =
    ...
    zc.zrs

[zeoserver]
<= zeoserver_base
recipe = plone.recipe.zeoserver[zrs]
replicate-from = zopeserver-1:5000
keep-alive-delay = 60
zeo-address = 8100
read-only = on

[client1]
<= client_base
recipe = plone.recipe.zope2instance
zeo-address = zopeserver-1:${zeoserver:zeo-address}
http-address = 8081

[client2]
<= client_base
recipe = plone.recipe.zope2instance
zeo-address = zopeserver-1:${zeoserver:zeo-address}
http-address = 8082

The "Picked Versions" from buildout are:

[versions]
Twisted = 13.2.0
zc.zrs = 2.4.4

When I try to create Plone File objects using the Zope clients on the secondary server. The traceback I receive is:

Traceback (innermost last):
  Module ZPublisher.Publish, line 146, in publish
  Module Zope2.App.startup, line 301, in commit
  Module transaction._manager, line 89, in commit
  Module transaction._transaction, line 329, in commit
  Module transaction._transaction, line 446, in _commitResources
  Module ZODB.Connection, line 781, in tpc_vote
  Module ZEO.ClientStorage, line 1098, in tpc_vote
  Module ZEO.ClientStorage, line 929, in _check_serials
IOError: [Errno 2] No such file or directory: '/usr/local/plone/zeocluster/var/blobstorage/0x00/0x00/0x00/0x00/0x00/0x00/0x00/0xea/0x006ObqSw.tmp-'

because this 0x006ObqSw.tmp- file is created on the blobstorage on the secondary server, but not the primary server.

It looks like the blobs are replicated to the secondary ZEO correctly when created by a Zope client on the primary server, but it's not possible to create a file using a Zope client on the secondary server because the ZEO on the primary server can't find the .tmp file.

If I add a shared-blob = off under [client 1] and [client 2] on the secondary, I receive the error:

ValueError: Directory layout `zeocache` selected for blob directory /usr/local/plone/zeocluster/var/blobstorage/, but marker found for layout `bushy`

Removing the contents of /usr/local/plone/zeocluster/var/blobstorage to allow it to create a zeocache layout allows file creation, but this streams all of the blobs through ZEO. My understanding is that this reduces performance, and it doesn't replicate the primary's blobstorage, which defeats half the purpose of replication.

I see the note in this question:

Plone Switching to ZRS using plone.recipe.zeoserver on Plone 4.3.1

about setting the Zope clients to read-only as well as the secondary ZEO server, but unfortunately that prevents us from using PloneFormGen's Save Data adapter, which we use extensively across our public sites.

Based on this experience, my thoughts on approaching this issue are:

• NFS mount blobstorage from the primary on the secondary, and have the secondary ZEO write to a parallel blobstorage-replicated folder that can be renamed for failover.
• Have the secondary ZEO replicate blobs to the blobstorage folder, but point the secondary Zope clients at a separate blobstorage-zeocache folder with shared-blob = off

Am I missing a really simple ZRS concept or configuration? Which is entirely possible!

Answer 1

Secondary servers need to be read-only and writes only happen to primary server. ZRS does not do MASTER-MASTER replication.

Perhaps you can route POST requests coming in to the primary server zeoclients since they potentially write to the database.

Check out https://pypi.python.org/pypi/wildcard.readonly for handling write-on-read issues.