Rocky Linux 8: NFSv4 + fscache always writes to cache but never reads back from the cache

by Danila Vershinin, August 5, 2021

GetPageSpeed keeps NGINX and Varnish servers fast and secure: performance modules and builds, plus security fixes within days, all through routine dnf, yum or apt upgrades. It maintains the largest RPM repository of NGINX module packages and Varnish VMODs, with APT packages for Debian and Ubuntu.
Free to install; commercial production use requires a subscription. Without one, a small license notice appears on HTML pages after 30 days.

Operating System and Software

Rocky Linux 8
NFS4 with fscache
- NFSv4 with ‘fsc’ mount option and cachefilesd service running

Problem

An NFSv4 mount with fscache enabled never reads back from the cache
We can see this is we mount the share, use ‘dd’ to read a file from the share, drop caches to drop the page cache, then use ‘dd’ again to read the file a second time. On the second ‘dd’ there should be no NFS operations recorded, since the reads should come from fscache not over the wire. Instead there is 5 reads again, the same amount as after the first ‘dd’.

rhel8 # mount -o vers=4,fsc nfsserver:/export /mnt/nfsserver
rhel8 # dd if=/mnt/nfsserver/file of=/dev/null
rhel8 # grep READ: /proc/self/mountstats
            READ: 5 5 0 1120 1049096 0 2 4
rhel8 # echo 3 >/proc/sys/vm/drop_caches
rhel8 # dd if=/mnt/nfsserver/file of=/dev/null
rhel8 # grep READ: /proc/self/mountstats
            READ: 10 10 0 2240 2098192 5 8 15

How to Fix

Rocky Linux 8

The issue is being tracked with bugzilla 1793560: Bug 1793560 – [Rocky Linux 8] fs-cache always writes data but never reads from fs-cache when using NFSv4 (Rocky Linux 8 8.1). As of Wed, June 03 2020, the status of 1793560 is ON_QA. A patch has been built and under test for the next minor release of Rocky Linux 8.

Workarounds

Use NFSv3 with fscache

Origin of the Problem

Fixed by the following commit

Author: Dave Wysochanski <dwysocha@redhat.com>
Date:   Thu Apr 16 06:06:08 2020 -0400

    NFSv4: Fix fscache cookie aux_data to ensure change_attr is included

    Commit 402cb8dda949 ("fscache: Attach the index key and aux data to
    the cookie") added the aux_data and aux_data_len to parameters to
    fscache_acquire_cookie(), and updated the callers in the NFS client.
    In the process it modified the aux_data to include the change_attr,
    but missed adding change_attr to a couple places where aux_data was
    used.  Specifically, when opening a file and the change_attr is not
    added, the following attempt to lookup an object will fail inside
    cachefiles_check_object_xattr() = -116 due to
    nfs_fscache_inode_check_aux() failing memcmp on auxdata and returning
    FSCACHE_CHECKAUX_OBSOLETE.

    Fix this by adding nfs_fscache_update_auxdata() to set the auxdata
    from all relevant fields in the inode, including the change_attr.

    Fixes: 402cb8dda949 ("fscache: Attach the index key and aux data to the cookie")
    Signed-off-by: Dave Wysochanski <dwysocha@redhat.com>
    Signed-off-by: David Howells <dhowells@redhat.com>

Diagnostic Steps

Check the /proc/self/mountstats file for NFS READ operations incrementing when the reads should come from fscache.

Danila Vershinin

Founder & Lead Engineer

NGINX configuration and optimizationLinux system administrationWeb performance engineering

10+ years NGINX experience • Maintainer of GetPageSpeed RPM repository • Contributor to open-source NGINX modules

GitHub →LinkedIn →

Solutions