RabbitMQ - vhost '/' is down for user 'XYZ'. even after user has all access

Question

I am using RabbitMQ version 3.7.17

As my AWS hard disk was completely occupied(100% full). Due to which all the services stopped working
Solution to this: I extended AWS server memory and than tried to start all the API services after that it started throwing error. (Post this it started giving error)

Connection.open: (541) INTERNAL_ERROR - access to vhost '/' refused for user 'XYZ': vhost '/' is down

Restarted RabbitmMQ server using the below code still it was giving error:

sudo service rabbitmq-server restart

If I checked the permission for my user using:

sudo rabbitmqctl list_permissions --vhost /

Response shows that user has all the access.

Listing permissions for vhost "/" ...
user    configure   write   read
XYZ .*  .*  .*

Thank You.

Answer 1

As the Memory was full the RabbitMQ that was processing was not completed which resulted it in error in vhost.

When tried to restart vhost sudo rabbitmqctl restart_vhost it got error:

ERROR:

Failed to start vhost '/' on node 'rabbit@ip-172-31-16-172'Reason: {:shutdown, {:failed_to_start_child, :rabbit_vhost_process, {:error, {{{:function_clause, [{:rabbit_queue_index, :journal_minus_segment1, [{{true, <<230, 140, 82, 5, 193, 81, 136, 75, 11, 91, 31, 232, 119, 30, 99, 112, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 144>>, <<131, 104, 6, 100, 0, 13, 98, 97, 115, 105, 99, 95, 109, 101, 115, 115, 97, 103, 101, 104, 4, 100, 0, 8, 114, 101, 115, 111, 117, 114, ...>>}, :no_del, :no_ack}, {{true, <<230, 140, 82, 5, 193, 81, 136, 75, 11, 91, 31, 232, 119, 30, 99, 112, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 144>>, <<131, 104, 6, 100, 0, 13, 98, 97, 115, 105, 99, 95, 109, 101, 115, 115, 97, 103, 101, 104, 4, 100, 0, 8, 114, 101, 115, 111, 117, ...>>}, :del, :no_ack}], [file: 'src/rabbit_queue_index.erl', line: 1231]}, {:rabbit_queue_index, :"-journal_minus_segment/3-fun-0-", 4, [file: 'src/rabbit_queue_index.erl', line: 1208]}, {:array, :sparse_foldl_3, 7, [file: 'array.erl', line: 1684]}, {:array, :sparse_foldl_2, 9, [file: 'array.erl', line: 1678]}, {:rabbit_queue_index, :"-recover_journal/1-fun-0-", 1, [file: 'src/rabbit_queue_index.erl', line: 915]}, {:lists, :map, 2, [file: 'lists.erl', line: 1239]}, {:rabbit_queue_index, :segment_map, 2, [file: 'src/rabbit_queue_index.erl', line: 1039]}, {:rabbit_queue_index, :recover_journal, 1, [file: 'src/rabbit_queue_index.erl', line: 906]}]}, {:gen_server2, :call, [#PID<10397.473.0>, :out, :infinity]}}, {:child, :undefined, :msg_store_persistent, {:rabbit_msg_store, :start_link, [:msg_store_persistent, '/var/lib/rabbitmq/mnesia/rabbit@ip-172-31-16-172/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L', [], {#Function<2.32138423/1 in :rabbit_queue_index>, {:start, [{:resource, "/", :queue, "xx_queue"}, {:resource, "/", :queue, "app_xxx_queue"}, {:resource, "/", :queue, "default"}, {:resource, "/", :queue, "xx_priority_queue"}, {:resource, "/", :queue, "xxx_queue"}, {:resource, "/", :queue, "xxxx_queue"}, {:resource, "/", :queue, "yyy_queue"}, {:resource, "/", :queue, "zzz_queue"}, {:resource, "/", :queue, "aaa_queue"}]}}]}, :transient, 30000, :worker, [:rabbit_msg_store]}}}}}

STEPS TO SOLVE IT

1. Stop your app node by below command.

sudo rabbitmqctl stop_app

2. Reset your node by below command. Removes the node from any cluster it belongs to, removes all data from the management database, such as configured users and vhosts, and deletes all persistent messages.(Be careful while using it.)

sudo rabbitmqctl reset

3. Start your Node by below command.

sudo rabbitmqctl start_app

4. Restart your vhost by below commad.

sudo rabbitmqctl restart_vhost

And if you are using some application that is depended on RabbitMQ. Such as I using celery you will have to restart them again.

This was the link that helped me to solve it.