How to fix ‘Consumer “async.operations.all” skipped as required connection “amqp” is not configured. Unknown connection name amqp’ in Magento 2

Magento 2 relies on message queues for some functions.

In a nutshell, this whole approach allows Magento to process some events in a timely, efficient and asynchronous way.

Message consumers are logical groups for of message processors.
They are defined within Magento code (usually declared in .xml files).

To get a list of consumers, you can run bin/magento queue:consumers:list. The default list as of Magento 2.3.4:

product_action_attribute.update
product_action_attribute.website.update
exportProcessor
codegeneratorProcessor
async.operations.all

Each consumer does message processing for queues known to it.

Example of how message queues work

So for example, if you run a built-in export from Magento admin, a message is created in queue_message MySQL table.
This message will contain requested export details.

Then, in a default setup, within a minute, your Magento cron job, will run a task consumers_runner, which runs subprocesses for each defined consumer.
So one of them, exportProcessor will check that MySQL table, and pick up the message about export, and create the export file for you.
All this happens in the background, without any wait for admin page, so there are no timeouts or other issues experienced in admin.

In this simple example, MySQL is the message exchanger software.
A message exchanger is simply put, a place where messages are stored until they are consumed.

RabbitMQ and message queues

Some features need a more sophisticated message exchanger than MySQL.

The Bulk API is one of the Magento features that relies on message queues.
However, it’s different in that it requires RabbitMQ to be the message exchanger.

Whether you use the bulk API endpoints or not, unless you configure RabbitMQ and set up its connection details, your var/log/system.log will be riddled with the message:

main.INFO: Consumer “async.operations.all” skipped as required connection “amqp” is not configured. Unknown connection name amqp [] []

There are quite many bad solutions for this message all over the internet. For example, The answer from Rafael Corrêa Gomes (and any to the point that mentions either 'cron_run' => false or 'consumers' => ['async.operations.all'] in the configuration are quite outrageous because they make many folks copy-paste and destroy their message queues from functioning.

cron_run set to false in configuration means that the message consumers are not going to be launched by the Magento cron. It means that you will have to run them using other means like SupervisorD or SystemD. And if not (and those answers don’t mention that at all), you will have broken stuff beyond words “message queues”, which includes, at very minimum, data export in Magento admin.

The documentation that is being referred to when posting such configuration has incorrect heading below this section with words:

Specific configuration

instead of

Sample configuration

It is a sample, and not a standard. Neither it is a recommendation to solve anything.

But, while the docs are at fault, it doesn’t excuse skipping the lines and not reading further, then posting destructive solutions online.

The real solution is either disabling the Bulk API which most installations don’t use or, configure RabbitMQ if you want the bulk API (highly unlikely, depends on specific use case).

Solution 1 (acceptable in most cases)

If you don’t intend to use bulk API (in all probability, you don’t care, because this is applicable to stores with a lot of visitors / specific use case),
you can disable the related module, and this, in turn, will disable the related “async.operations.all” consumer.

php bin/magento module:disable Magento_WebapiAsync

No more annoying message, at the cost of a disabled function, which didn’t work either way.

Solution 2. Setup and configure RabbitMQ

Step 1. Install RabbitMQ

If you have a powerful server, you may choose to set up RabbitMQ on the same host where you run Magento 2.

Magento 2 supports only the fairly recent branch of RabbitMQ, 3.8.x.

We will use the official RabbitMQ RPM repository to set it up.

The following commands will allow you to install RabbitMQ on either CentOS/RHEL 7 or CentOS/RHEL 8.

First, you need to run the following:

curl -s https://packagecloud.io/install/repositories/rabbitmq/erlang/script.rpm.sh | sudo bash

This brings in the required Erlang version (which is too old in the base repositories).

The next commands will be the same for both CentOS 7 and 8:

sudo yum -y install epel-release 
curl -s https://packagecloud.io/install/repositories/rabbitmq/rabbitmq-server/script.rpm.sh | sudo bash
sudo dnf -y install rabbitmq-server
sudo systemctl enable --now rabbitmq-server

If you haven’t configured FQDN as your hostname or it otherwise does not resolve to an IP, you might receive:

Job for rabbitmq-server.service failed because the control process exited with error code.

and the following in /var/log/messages:

HOSTNAME rabbitmq-server[11726]: ERROR: epmd error for host HOSTNAME: timeout (timed out)

The solution (and this is what you have to do in a well-configured server) is either defining FQDN as the hostname and/or ensuring entry for hostname in /etc/hosts.

You can check the status of your RabbitMQ node by running sudo rabbitmqctl status.

The default configuration file location is /etc/rabbitmq/rabbitmq.conf.
You will want to create it in case you intend to use a cluster setup.

For single-server setups where Magento 2 and RabbitMQ are on the same machine, it is sufficient to use the default RabbitMQ credentials: guest / guest.

Step 2. Configure Magento 2 with RabbitMQ connection details

Edit your app/etc/env.php file with connection details applicable to your RabbitMQ instance.

    'queue' =>  [
        'amqp' =>  [
            'host' => '127.0.0.1', // IP address of RabbitMQ instance. Change it, if it is not running on the same machine with Magento
            'port' => '5672', // Port on which RabbitMQ running. 5672 is default port
            'user' => 'guest', // RabbitMQ user name
            'password' => 'guest', // RabbitMQ password
            'virtualhost' => '/' // The virtual host for connecting to RabbitMQ. The default is /.
        ],
    ],

Verify that you have positioned the above config correctly in the file (it should be coming just above the trailing ]; by running php -l app/etc/env.php.

Then, run bin/magento setup:upgrade --keep-generated to apply the changes and create the required queues in RabbitMQ.

Verify your setup. Run:

bin/magento queue:consumers:start async.operations.all

If it stays on without error, it means that the queue consumer has successfully established a connection to the message broker (RabbitMQ) and is waiting for a message to appear.
You can cancel it out with Ctrl+C now. It will be launched later automatically by cron.

You can then examine the output of sudo rabbitmqctl status.
It will indicate a positive number of connections, e.g.:

Totals

Connection count: 1
Queue count: 1
Virtual host count: 1

Finally, minutes later, after you’re sure that Magento cron has run, you can check the process list:

ps aux | grep async.operations.all

You will see the consumer process:

/usr/bin/php path/to/magento2/bin/magento queue:consumers:start async.operations.all –pid-file-path=…

It is normal that this process stays on all the time.

That’s it! Now both Bulk API feature is functional and there is no annoying message about it.