use of archive_command in PostgreSQL streaming replication

Question

When using streaming replication can someone please explain the purpose of archive_command and restore_command in PostgreSQL?

As i studied in streaming replication secondary server read and apply the partially filled WAL files.suppose i have my wal segment location in pg_xlog and using archive_command i am copying this to my local archive directory say /arclogs.

So if secondary server is going to read the partially filled archive logs from pg_xlog over the network then what's the use of files kept in /arclogs. and also the files will be sent to /arclogs only when they will be 16 mb?

I'm new to PostgreSQL & your help will be appericated.

Answer 1

The master will normally only retain a limited amount of WAL in pg_xlog, controlled by the master's wal_keep_segments setting. If the replica is too slow or disconnected for too long, the master will delete those transaction logs to ensure it can continue running without running out of disk space.

If that happens the replica has no way to catch up to the master, since it needs a continuous and gap-free stream of WAL.

So you can:

• Enable WAL archiving (archive_command and archive_mode) as a fallback, so the replica can switch to replaying WAL from archives if the master deletes WAL it needs from its pg_xlog. The replica fetches the WAL with its restore_command. Importantly, the archived WAL does not need to be on the same machine as the master, and usually isn't.

or

• Use a physical replication slot (primary_slot_name in recovery.conf) to connect the replica to the master. If a slot is used, the master knows what WAL the replica requires even when the replica is disconnected. So it won't remove WAL still needed by a replica from pg_xlog. But the downside is that pg_xlog can fill up if a replica is down for too long, causing the master to fail due to lack of disk space.

or

• Do neither, and allow replicas to fail if they fall too far behind. Then re-create them from a new base backup if this happens.

The documentation really needs an overview piece to put all this together.

WAL archiving has an additional benefit: If you make a base backup of the server you can use it, plus WAL archives, to do a point-in-time restore of the master. This lets you recover data from things like accidental table drops. PgBarman is one of the tools that can help you with this.