More interactive ZODB packing

Question

Current ZMI management "Pack database" functionality is little rough.

1) Could it be possible to have some kind of progress indicator for web UI? E.g. one telling how many minutes/hours are left giving at least some kind of estimate

2) How does ZODB packing affect the responsivity of the site? Are all transactions blocked?

3) Any command line scripts with progress indicator available so you could do this from a ZEO command line client?

4) At least some kind of log markers to logout output... [INFO] 30% done... 3:15 to go

Answer 1

Packing Anatomy

ZODB FileStorage packing is process of selective copying of data from one file to another one (only transactions that are "younger" then specified age). Before this copying starts some soft of index is built in memory to aid in process. Thus whole ZODB packing contains following steps:

1. Building pack index
2. Copying transactions to temporary file
3. Appending transactions that were performed after packing started
4. Replacing original FileStorage with packed one and reopening it in read/write mode

I'm usually monitoring the process by combination of top, vmstat/dstat, watch ls -la var/filestorage.

As Geir mentioned, you can have separate ZEO client dedicated to packing. This was reasonable as thread you invoked packing from blocked until packing finished. Now there is no need to if you use ZEO. ZEO server provides zeopack utility that connects directly to ZEO (no need for dedicated ZEO client) and initiates FileStorage packing. One of the benefits is no need for password, just proper permissions to access ZEO control socket.

Packing progress

As packing is performed by ZEO server (even not server but FileStorage itself), possibility of proper communication of progress to ZEO client is limited. ZEO protocol was not designed to communicate that type of information.

IMHO FileStorage itself could be more verbose in communicating through log file what it is doing right now. Some kind of progress could be built in. And if you feel like need the progress indicator, then you can design some kind of feedback channel through logging module back to ZEO-client/Zope-instance to be communicated back to browser.

Performance while packing

As FileStorage packing is quite intensive disk operation, it reduces throughoutput of disk subsystem. Additionally it expunges disk cache (in case of larger FileStorage), that impact disk performance even after packing finished, as caches should be warmed up again. Possible improvements that lead to longer packing time but smaller impact on system in FileStorage are:

• reverting to O_DIRECT operations (not to touch file cache)
• reducing disk scheduling priority (ionice on Linux) for thread performing the packing
• throttling packing speed

Answer 2

The recommended way of doing packing for large sites is to run it on a separate ZEO instance dedicated to such tasks — and that is not listening for http requests at all.

That will also remove the need for any of the featuers requested.

Answer 3

1) There is no such indicator and it would be possibly hard to implement one (I would love to see at least some progress indicator through the Zope logging system)

2) not blocked but depending on the packing phase you may see high IO and CPU usage

3) no

4) no