Server Setup

DNS caching and beyond, in CentOS/RHEL 7 and 8

by , , revisited on


We have by far the largest RPM repository with dynamic stable NGINX modules and VMODs for Varnish 4.1 and 6.0 LTS. If you want to install nginx, Varnish and lots of useful modules for them, this is your one stop repository to get all performance related software.
You have to maintain an active subscription in order to be able to use the repository!

By default, RHEL performs no DNS caching at all. Why this is so, is beyond mystery.

One could only guess that the rationale for lack of DNS caching in RHEL is the arguable efficiency for those systems which aren’t network connected or simply don’t need to make any DNS lookups.

There are of course such cases where you don’t need (many) DNS resolutions. I can think of:

  • a dedicated DB server
  • a private server where all hosts are listed in the hosts file

Those systems will likely issue zero to none DNS lookups while running, and DNS cache isn’t really a thing for them.

But for the most intents of running either a desktop or server RHEL machines, you will absolutely benefit from a DNS cache.

Enabling DNS cache in RHEL 7 and 8 is easy thanks to dnsmasq integration of NetworkManager.

The dnsmasq is a very lightweight caching DNS forwarder which runs great even on the tiniest hardware like your very own home router.

I won’t torture you with long instructions on how to enable the DNS cache. It’s really quick and goes down to:

yum -y install dnsmasq

cat << 'EOF' | sudo tee /etc/NetworkManager/conf.d/dns.conf 
[main]
dns=dnsmasq
EOF

systemctl reload NetworkManager

You have just made your machine already faster by running these.

For more details and fine-tuning, read on.

NetworkManager and dnsmasq

Let’s explain what happened when we ran the above commands to enable DNS caching.

In the first bit, we have installed the very essential of DNS caching – dnsmasq program.

Then we write out a file, /etc/NetworkManager/conf.d/dns.conf, with contents telling NetworkManager to enable and use its dnsmasq plugin. Then we reload NetworkManager configuration to apply our changes.

This, in turn, starts a private instance of dnsmasq program, which is bound to the loopback interface, 127.0.0.1 and listening on standard DNS port, 53.

It doesn’t end there. NetworkManager now updated /etc/resolv.conf and put nameserver 127.0.0.1 so that the whole operating system will perform DNS lookups against its dnsmasq instance.

The dnsmasq itself will use whatever nameservers you had setup in NetworkManager explicitly, or the ones provided by DHCP requests.

Very clean and beautiful integration.

Verify dnsmasq is working

Simply perform a DNS lookup using dig, against 127.0.0.1

# yum -y install bind-utils
dig +short example.com @127.0.0.1

If the output looks like a valid IP address or a list of IP addresses, then dnsmasq is working OK.

You can also check that DNS caching is working. Perform a resolution against another domain by running the following command twice:

time getent hosts foo.example.com

Observe real timing in the output reduced for the subsequent queries. E.g. first request yields:

real   0m0.048s
user   0m0.006s
sys    0m0.006s

Subsequent requests yield:

real   0m0.009s
user   0m0.006s
sys    0m0.002s

See what kind of DNS requests your system makes

To see what DNS request your system makes, you can temporarily enable logging of queries. Note that this will clear DNS cache because dnsmasq will be restarted:

echo log-queries | sudo tee -a /etc/NetworkManager/dnsmasq.d/log.conf
sudo systemctl reload NetworkManager

You can then tail or less the /var/log/messages file which will have information of requests being made. Example, on the web server that is using PaperTrail’s remote_syslog:

dnsmasq[20802]: forwarded logs6.papertrailapp.com to 2606:4700:4700::1001
dnsmasq[20802]: reply logs6.papertrailapp.com is 169.46.82.182
dnsmasq[20802]: reply logs6.papertrailapp.com is 169.46.82.183
dnsmasq[20802]: reply logs6.papertrailapp.com is 169.46.82.184
dnsmasq[20802]: reply logs6.papertrailapp.com is 169.46.82.185

This approach may be used for finding what external sites your server communicates with.

Once you’re done, don’t forget to turn off the logging:

sudo rm /etc/NetworkManager/dnsmasq.d/log.conf
sudo systemctl reload NetworkManager

How well is dnsmasq doing on your system

The dnsmasq manpage has this to say:

When it receives a SIGUSR1, dnsmasq writes statistics to the system log. It writes the cache size, the number of names which have had to removed from the cache before they expired in order to
make room for new names and the total number of names that have been inserted into the cache. The number of cache hits and misses and the number of authoritative queries answered are also given.

So we can collect DNS query stats easily:

sudo pkill --signal USR1 dnsmasq && sudo tail /var/log/messages | grep dnsmasq

The output may include, for example:

dnsmasq[31949]: cache size 400, 0/60 cache insertions re-used unexpired cache entries.
queries forwarded 30, queries answered locally 60

The 0 in 0/60 stands for “zero cache evictions”. So this number indicates that cache size is adequate. It should be as low as possible.
If that number is high, it means that cache size maybe not large enough.

We also see that 30 DNS lookups were forwarded over to upstream nameservers (misses), while 60 were satisfied directly by cache (hits).

Gathering stats like this will work well in case you only have one instance of dnsmasq. Sometimes you have more than one (e.g. libvirt may run one of its own).

It is more reliable to use the statistical information of dnsmasq that is exposed, not surprisingly, via DNS 🙂 The commands:

dig +short chaos txt hits.bind
dig +short chaos txt misses.bind

… give you hits and misses, respectively.

With some command line magic, you can easily calculate your DNS cache hit-ratio:

# yum -y install bc
echo "scale=2; $(dig +short chaos txt hits.bind)*100/($(dig +short chaos txt hits.bind)+$(dig +short chaos txt misses.bind))" | \
  sed 's@"@@g' | bc

The output is a percentage of DNS requests that were satisfied by DNS cache, e.g.: 80.95%.

Tuning the cache size

The default cache size of dnsmasq instance that is run by Networkmanager is 400.
This is a decent default for web servers.

For a desktop machine, you may want to increase it by large. This will assist with much less home router strain and faster network experience, especially if you’re a Chrome user. This browser does DNS caching of its own, but only as long as 1 minute – the issue that is discarded as a “feature”.

So to set DNS cache size to 20k, run:

echo cache-size=20000 | sudo tee -a /etc/NetworkManager/dnsmasq.d/cache.conf
sudo systemctl reload NetworkManager

dnsmasq and your desktop

To expand the topic of the desktop use of dnsmasq, you can also leverage it to block tracking scripts and for speeding up your browsing experience:

sudo curl https://raw.githubusercontent.com/aghorler/lightweight-dnsmasq-blocklist/master/list.txt \
  --output /etc/NetworkManager/dnsmasq.d/blocklist.conf
sudo systemctl reload NetworkManager

Finally, you may also want to improve the DNS speed by ensuring minimum TTL for DNS records that have it set too low.

echo min-cache-ttl=1800 | sudo tee -a /etc/NetworkManager/dnsmasq.d/cache.conf
sudo systemctl reload NetworkManager

This will ensure that even if a DNS record is configured with, e.g. 2 minutes TTL on remote nameserver, dnsmasq will still cache it for 30 minutes.

Note that this is acceptable for desktop machines, but not for web servers:

Phew, now I think that’s about it for dnsmasq today. Enjoy your faster DNS and be sure to subscribe for our Twitter for more fine articles 🙂

  1. Alex

    Good health to you, Danila!
    I understand that DNSMasq is faster and more lightweight than BIND, so it’s better for small web site on VPS (OpenVZ + Webmin)?
    Thank you very much for your web site and repository!
    It’s a really great help for non-professionals like me… so your work definitely makes the world a bit better. =)
    Будь здоров, и пусть поют птицы.

    Reply
    • Danila Vershinin

      Hi Alex,

      Yep, DNSMasq is great for all intents, especially for running on small servers.
      It is only a caching/forwarding DNS nameserver, it can’t host DNS entries.

      BIND is more appropriate if you’re a hosting company or for some very dubious reason would want to host DNS entries on your own server.
      There are many DNS hosting servers out there that will do things just better.

      You can think of DNS as incoming and outgoing information, which it is:

      • The outgoing is the DNS lookups that your server makes. E.g. when you run a simple yum update, the server will have to talk to a few other domains and resolve their IPs. This is example area that will improve when you setup DNSMasq, because it will cache those lookups, which doesn’t happen by default.
      • The incoming DNS is when someone resolves your domain, e.g. opening your site in their browser. For this, you need software/service to host information about which IP your domain is hosted on. The right thing as per specs is having at least 2 nameservers (machines running “incoming DNS software” like BIND) for hosting your domain DNS records. So it’s much better “outsourcing” the job of hosting DNS recommends to third party services like Cloudflare/Hurricane Electric, that will do it properly and in a way that it also works faster.

      Thanks!

      Reply

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.