DNS caching and beyond, in CentOS/RHEL and Fedora

Setting the right nameservers

We all know that Cloudflare is the leading and fastest DNS provider.
However, it does have uptime issues, so you should use more nameservers for fault tolerance.
Provided that your connection name in NetworkManager is eth0, you can use both Cloudflare and Google DNS nameservers like this:

nmcli connection modify eth0 ipv4.dns 1.1.1.1,1.0.0.1,8.8.8.8,8.8.4.4

By default, RHEL performs no DNS caching at all. Why this is so, is beyond mystery.

One could only guess that the rationale for the lack of DNS caching in RHEL is the arguable efficiency for those systems which aren’t network connected or simply don’t need to make any DNS lookups.

There are of course such cases where you don’t need (many) DNS resolutions. I can think of:

• a dedicated DB server
• a private server where all hosts are listed in the hosts file

Those systems will likely issue zero to none DNS lookups while running, and DNS cache isn’t really a thing for them.

But for the most intents of running either a desktop or server RHEL machines, you will absolutely benefit from a DNS cache.

Enabling DNS cache in RHEL 7 and 8 is easy thanks to the dnsmasq integration of NetworkManager.

The dnsmasq is a very lightweight caching DNS forwarder that runs great even on the tiniest hardware like your very own home router.

I won’t torture you with long instructions on how to enable the DNS cache. It’s really quick and goes down to:

yum -y install dnsmasq

cat << 'EOF' | sudo tee /etc/NetworkManager/conf.d/dns.conf 
[main]
dns=dnsmasq
EOF

systemctl reload NetworkManager

You have just made your machine already faster by running these.

For more details and fine-tuning, read on.

NetworkManager and dnsmasq

Let’s explain what happened when we ran the above commands to enable DNS caching.

In the first bit, we have installed the very essence of DNS caching – the dnsmasq program.

Then we write out a file, /etc/NetworkManager/conf.d/dns.conf, with contents telling NetworkManager to enable and use its dnsmasq plugin. Then we reload the NetworkManager configuration to apply our changes.

This, in turn, starts a private instance of the dnsmasq program, which is bound to the loopback interface, 127.0.0.1, and listening on standard DNS port, 53.

It doesn’t end there. NetworkManager now updated /etc/resolv.conf and put nameserver 127.0.0.1 so that the whole operating system will perform DNS lookups against its dnsmasq instance.

The dnsmasq itself will use whatever nameservers you had setup in NetworkManager explicitly, or the ones provided by DHCP requests.

Very clean and beautiful integration.

Verify dnsmasq is working

Simply perform a DNS lookup using dig, against 127.0.0.1”

# yum -y install bind-utils
dig +short example.com @127.0.0.1

If the output looks like a valid IP address or a list of IP addresses, then dnsmasq is working OK.

You can also check that DNS caching is working. Perform a resolution against another domain by running the following command twice:

time getent hosts foo.example.com

Observe real timing in the output reduced for the subsequent queries. E.g. first request yields:

real   0m0.048s
user   0m0.006s
sys    0m0.006s

Subsequent requests yield:

real   0m0.009s
user   0m0.006s
sys    0m0.002s

See what kind of DNS requests your system makes

To see what DNS request your system makes, you can temporarily enable the logging of queries. Note that this will clear the DNS cache because dnsmasq will be restarted:

echo log-queries | sudo tee -a /etc/NetworkManager/dnsmasq.d/log.conf
sudo systemctl reload NetworkManager

You can then tail or less the /var/log/messages file which will have information of requests being made. Example, on the web server that is using PaperTrail’s remote_syslog:

dnsmasq[20802]: forwarded logs6.papertrailapp.com to 2606:4700:4700::1001
dnsmasq[20802]: reply logs6.papertrailapp.com is 169.46.82.182
dnsmasq[20802]: reply logs6.papertrailapp.com is 169.46.82.183
dnsmasq[20802]: reply logs6.papertrailapp.com is 169.46.82.184
dnsmasq[20802]: reply logs6.papertrailapp.com is 169.46.82.185

This approach may be used for finding what external sites your server communicates with.

Once you’re done, don’t forget to turn off the logging:

sudo rm /etc/NetworkManager/dnsmasq.d/log.conf
sudo systemctl reload NetworkManager

How well is dnsmasq doing on your system

The dnsmasq manpage has this to say:

When it receives a SIGUSR1, dnsmasq writes statistics to the system log. It writes the cache size, the number of names which have had to removed from the cache before they expired in order to
make room for new names and the total number of names that have been inserted into the cache. The number of cache hits and misses and the number of authoritative queries answered are also given.

So we can collect DNS query stats easily:

sudo pkill --signal USR1 dnsmasq && sudo tail /var/log/messages | grep dnsmasq

The output may include, for example:

dnsmasq[31949]: cache size 400, 0/60 cache insertions re-used unexpired cache entries.
queries forwarded 30, queries answered locally 60

The 0 in 0/60 stands for “zero cache evictions”. So this number indicates that cache size is adequate. It should be as low as possible.
If that number is high, it means that cache size may be not large enough.

We also see that 30 DNS lookups were forwarded over to upstream nameservers (misses), while 60 were satisfied directly by cache (hits).

Gathering stats like this will work well in case you only have one instance of dnsmasq. Sometimes you have more than one (e.g. libvirt may run one of its own).

It is more reliable to use the statistical information of dnsmasq that is exposed, not surprisingly, via DNS 🙂 The commands:

dig +short chaos txt hits.bind
dig +short chaos txt misses.bind

… give you hits and misses, respectively.

With some command line magic, you can easily calculate your DNS cache hit ratio:

# yum -y install bc
echo "scale=2; $(dig +short chaos txt hits.bind)*100/($(dig +short chaos txt hits.bind)+$(dig +short chaos txt misses.bind))" | \
  sed 's@"@@g' | bc

The output is a percentage of DNS requests that were satisfied by DNS cache, e.g.: 80.95%.

Adding custom DNS entries

Sometimes you want to add your own DNS entries. This may be needed for adding development domains, e.g. example.local or overriding/blocking existing DNS entry.
You can choose to use /etc/hosts as usual. For dnsmasq to respect entries from it, create the file /etc/NetworkManager/dnsmasq.d/custom.conf with contents:

addn-hosts=/etc/hosts

Then reload the NetworkManager service.

Alternatively, you can directly add entries to dnsmasq, by adding them to the same file, e.g.:

address=/hexagon-analytics.com/0.0.0.0
address=/stats.pusher.com/0.0.0.0

Tuning the cache size

The default cache size of dnsmasq instance that is run by NetworkManager is 400.
This is a decent default for web servers.

For a desktop machine, you may want to increase it by large. This will assist with much less home router strain and faster network experience, especially if you’re a Chrome user. This browser does DNS caching of its own, but only as long as 1 minute – the issue that is discarded as a “feature”.

So to set DNS cache size to 20k, run:

echo cache-size=20000 | sudo tee -a /etc/NetworkManager/dnsmasq.d/cache.conf
sudo systemctl reload NetworkManager

dnsmasq and your desktop

To expand the topic of the desktop use of dnsmasq, you can also leverage it to block tracking scripts and for speeding up your browsing experience:

sudo curl https://raw.githubusercontent.com/dvershinin/lightweight-dnsmasq-blocklist/master/list.txt \
  --output /etc/NetworkManager/dnsmasq.d/blocklist.conf
sudo systemctl reload NetworkManager

Finally, you may also want to improve the DNS speed by ensuring minimum TTL for DNS records that have it set too low.

echo min-cache-ttl=1800 | sudo tee -a /etc/NetworkManager/dnsmasq.d/cache.conf
sudo systemctl reload NetworkManager

This will ensure that even if a DNS record is configured with, e.g. 2 minutes TTL on remote nameserver, dnsmasq will still cache it for 30 minutes.

Note that this is acceptable for desktop machines, but not for web servers.

Different DNS servers for different domains

Suppose that your system is set to resolve everything via Cloudflare’s DNS network. However, it is known for censorship. An example of this was when

dig +short securepay.tinkoff.ru @1.1.1.1 returned no results in the USA only.

To set things up in a way that specific domains are resolved via different DNS servers, e.g. Yandex DNS, use this as an example:

echo "server=/securepay.tinkoff.ru/77.88.8.8" | sudo tee -a /etc/NetworkManager/dnsmasq.d/custom.conf
sudo systemctl reload NetworkManager

For the record, Yandex DNS uses these nameservers:

• 77.88.8.8
• 77.88.8.1

It is also recommend to add strict-order at the top of /etc/NetworkManager/dnsmasq.d/custom.conf.
This will ensure that dnsmasq will use provided nameservers sequentially. This is useful when you specify the first nameservers as fastest, e.g. Cloudflare’s. But in case it doesn’t respond, dnsmasq will use other, potentially slower nameservers.

Phew, now I think that’s about it for dnsmasq today. Enjoy your faster DNS and be sure to subscribe to our Twitter for more fine articles 🙂