How to correct configuration for firewalld and docker/nginx?

Question

I have a CentOS 7 server which was running happily for 600+ days until it was rebooted recently, after which incoming web requests were receiving HTTP523 (Origin Is Unreachable) error codes (via Cloudflare, if that makes a difference?) unless I stopped the firewalld service. Things run fine without firewalld, but I'd rather not leave it disabled!

I've tried stopping docker and firewalld and restarting them in various sequences, but the same 523 error occurs unless I stop firewalld.

/var/log/firewalld contains a few warnings that might help:

• WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i br-8acb606a3b50 -o br-8acb606a3b50 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
• WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
• WARNING: AllowZoneDrifting is enabled. This is considered a n insecure configuration option. It will be removed in a future release. Please consider disabling it now.
• WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D PREROUTING -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables v1.4.21: Couldn't load target 'DOCKER':No such file or directory
• WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D PREROUTING' failed: iptables: Bad rule (does a matching rule exist in that chain?).
• WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -D OUTPUT' failed: iptables: Bad rule (does a matching rule exist in that chain?)
• WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -t nat -F DOCKER' failed: iptables: No chain/target/match by that name.

I've found seemingly conflicting advice around the place regarding any manual configuration/commands required:

1. firewall-cmd --permanent --zone=trusted --add-interface=docker0 on a CentOS forum
2. firewall-cmd --zone=trusted --remove-interface=docker0 --permanent on the offical Docker docs -- surely that's the opposite of the above?
3. a bunch of manual firewall-cmd commands on a Docker github issue -- surely all of that isn't required?
4. this one looks promising -- nmcli, NetworkManager and firewall-cmd --permanent --zone=trusted --change-interface=docker0

I don't fully understand where the br-8acb606a3b50 interface comes from, or whether I need to do anything to configure it as well as docker0 if I use a solution like 4. above? It was all working fine automatically for years until the reboot!

Are some magic firewalld incantations now required (and why?!) or is there some way I can get the system to get back into the correct auto/default configuration it was in prior to rebooting?

$ docker -v
Docker version 20.10.5, build 55c4c88
$ firewall-cmd --version
0.6.3
$ firewall-cmd --get-zones
block dmz docker drop external home internal public trusted work

Answer 1

To recap the chat investigation, this particular problem wasn't related to Docker and containers. The problem was in firewalld not having rules for NGINX running as a proxy for containers on the host. The solution was to add permanent firewalld rules for HTTP and HTTPS traffic:

sudo firewall-cmd --permanent --zone=public --add-service=http
sudo firewall-cmd --permanent --zone=public --add-service=https
sudo firewall-cmd --reload

Warning messages like this one:

WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w10 -D FORWARD -i br-8acb606a3b50 -o br-8acb606a3b50 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?)

... can appear during normal operation, when Docker attempts to delete a rule without checking its existence first. In other words, containers can be running smoothly even when there are warnings like this.