neutron-openvswitch-agent eventually fails listing/creating high count of ports on network node

Question

I have been trying to create huge number of network,subnet and vrfs so as to have huge number of ports, but neutron ovs agent is failing with the following error,

2018-06-28 10:23:48.632 40956 ERROR neutron.agent.ovsdb.native.commands 
2018-06-28 10:23:48.632 40956 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-f6ed7ec3-bfb6-4cfc-84c0-f9c3632972d8 - - - - -] Error while processing VIF ports
2018-06-28 10:23:48.632 40956 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2018-06-28 10:23:48.632 40956 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2037, in rpc_loop
2018-06-28 10:23:48.632 40956 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     failed_devices, failed_ancillary_devices))
2018-06-28 10:23:48.632 40956 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 153, in wrapper
2018-06-28 10:23:48.632 40956 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return f(*args, **kwargs)
2018-06-28 10:23:48.632 40956 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1854, in process_port_info
2018-06-28 10:23:48.632 40956 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     updated_ports_copy)
2018-06-28 10:23:48.632 40956 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 153, in wrapper
2018-06-28 10:23:48.632 40956 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return f(*args, **kwargs)
2018-06-28 10:23:48.632 40956 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1348, in scan_ports
2018-06-28 10:23:48.632 40956 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     updated_ports.update(self.check_changed_vlans())
2018-06-28 10:23:48.632 40956 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 153, in wrapper
2018-06-28 10:23:48.632 40956 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return f(*args, **kwargs)
2018-06-28 10:23:48.632 40956 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1371, in check_changed_vlans
2018-06-28 10:23:48.632 40956 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_tags = self.int_br.get_port_tag_dict()
2018-06-28 10:23:48.632 40956 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/common/ovs_lib.py", line 520, in get_port_tag_dict
2018-06-28 10:23:48.632 40956 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     'Port', columns=['name', 'tag'], if_exists=True)
2018-06-28 10:23:48.632 40956 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/common/ovs_lib.py", line 434, in get_ports_attributes
2018-06-28 10:23:48.632 40956 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     execute(check_error=check_error, log_errors=log_errors))
2018-06-28 10:23:48.632 40956 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/ovsdb/native/commands.py", line 43, in execute
2018-06-28 10:23:48.632 40956 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     ctx.reraise = False
2018-06-28 10:23:48.632 40956 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-06-28 10:23:48.632 40956 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.force_reraise()
2018-06-28 10:23:48.632 40956 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-06-28 10:23:48.632 40956 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     six.reraise(self.type_, self.value, self.tb)
2018-06-28 10:23:48.632 40956 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/ovsdb/native/commands.py", line 36, in execute
2018-06-28 10:23:48.632 40956 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     txn.add(self)
2018-06-28 10:23:48.632 40956 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/ovsdb/api.py", line 79, in __exit__
2018-06-28 10:23:48.632 40956 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.result = self.commit()
2018-06-28 10:23:48.632 40956 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/ovsdb/impl_idl.py", line 73, in commit
2018-06-28 10:23:48.632 40956 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     'timeout': self.timeout})
2018-06-28 10:23:48.632 40956 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent TimeoutException: Commands [DbListCommand(if_exists=True, records=[u'veth-lexcluum', u'veth-gzqqzaff', u'qr-74b785b6-77', u'tapf663ec4e-71', u'tape3527585-10', u'veth-lmjofnhl', u'qr-86cfd08d-30', u'tapb47283c9-fa', u'tapa1e26f22-ff', u'tap62bc5816-a6', u'veth-vsrtxhtp', u'qr-bd0f3539-e2', u'qr-6c256b7d-d9', u'qr-caf09ee0-dc', u'veth-fzhjayzq', u'veth-onxitcmi', u't-ylbcpoqb', u'qr-907897c3-4b', u'qr-67b3170e-ce', u'tap0d3b076e-6b', u'veth-bfgklmcy', u't-jtnwrurg', u'qr-35e039a5-5a', u'veth-iiooipfe', u'veth-qwxpdebw', u'veth-vbphdjif', u'tap2ff3fe26-35', u'tap3e3ec578-4d', u'tap73368a6c-0e', u'tap26127b91-3c', u't-wnzdsozp', u'veth-zzhyuwvo', u't-xbuggpjm', u'qr-375c485a-34', u'qr-0b4b2157-e6', u't-nkoramgq', u't-ksydecwm', u'qr-b2ea57df-59', u'tapb60d1b9d-cf', u'qr-632b2471-65', u'qr-8b47ba76-d1', u'veth-iplqlddt', u'tap349aecb5-04', u'qr-29914f8a-c0', u'veth-radbxzgn', u'qr-a3c03a93-f6', u't-pmyashdw', u't-teuuhdck', u'veth-blcivytk', u't-gwlsedkm', u'tapccac6f5a-f9', u't-brbumiiz', u'qr-6f741a6d-ef', u'qr-15a87b74-25', u'tap9fc838bb-a7', u't-lyttuyef', u'veth-xctlfqji', u'qr-aa4c37b2-46', u'qr-b6d57014-1d', u'veth-ibggatfo', u'tap0094f643-83', u'veth-kohluxat', u'tape608c910-72', u'tapa7ce37d4-00', u'qr-92be728a-c5', u'veth-rqznjhpp', u'tapef166ca1-1b', u'qr-627325ad-e1', u'qr-b67ca002-27', u'qr-390e0ce7-6c', u'veth-kmqtkkcs', u'tapdfeae3f9-5e', u'tapb536b5ab-c8', u't-chmtfieo', u'tap7be43aed-a7', u'qr-fad4ac66-4f', u'qr-20bf99b4-d6', u'tap615b8953-8e', u'qr-78ce3890-18', u't-mozroeqv', u'qr-efb1e219-7b', u'tap69813600-17', u'veth-xjwgfxoi', u'qr-e6df95ac-f9', u'veth-jpnilokb', u'tap45a091cd-9c', u't-lsphadyp', u'tap94007772-35', u'veth-cxvxkygu', u'veth-jihpxdtl', u't-ndtcxquw', u'veth-ucdnimwt', u't-osghmslr', u'qr-5c0f127c-e1', u'veth-ujszstfh', u'qr-00e163e8-4a', u'qr-ff690cf7-db', u'tapa4ae06e2-5c', u'qr-5e040c30-1a', u't-ryszaaru', u'veth-zbhqqbiu', u'tap637f2138-97', u'veth-jvxpppok', u'tapc8bbfd9a-c6', u'veth-ezjiqcrz', u'veth-trrlvkyh', u'qr-5f94db6e-f6', u'qr-7b6f90fe-c2', u'tap28565c49-23', u'veth-nphlokph', u't-rbohxcxg', u'qr-1dd15807-1b', u'veth-hkfxwphb', u't-ulbnhpof', u't-psyekduy', u'veth-iwdjgmhs', u'tapc642dc95-55', u'qr-33275a2a-6d', u't-vvotxigu', u'tap0344aa65-48', u'qr-2134ce25-90', u'tap9d08ec93-ec', u't-ijinhkki', u'veth-eoupyqjy', u't-oshzipdi', u't-ymeyglwm', u'veth-qiihqapc', u't-twuaavoe', u't-qlgmqeyf', u'tap18d19d59-e0', u't-gzzdugte', u't-ewkwvlam', u'veth-nwpvohty', u'tap2cf0724b-bb', u't-pnklxzyk', u'veth-wotayuph', u't-bvgprqev', u't-ucdnimwt', u'tapbe380541-3a', u'tap8d60d33e-40', u'qr-c8c53593-62', u'qr-f9b46990-a5', u't-onxitcmi', u'tap476085ce-33', u'qr-72ea9b70-7d', u't-lexcluum', u'qr-04162bfd-e8'..............
..........
s=['name', 'tag'])] exceeded timeout 10 seconds
2018-06-28 10:23:48.632 40956 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent

Answer 1

Try with a smaller network. Until it works, then try more, until it stops working. So you'll find the nice spot. Or the number that makes it no longer work.