If an NFS share becomes unmounted, data could potentially be written to the unmounted directory on the local drive and cause two problems:
- If the the NFS share is not reconnected, the local drive can eventually and unexpectedly become full.
- If the NFS share is remounted, the data that was written on the local drive will not be seen in the mounted directory but it will continue to use space on the local drive.
Environment
I was experiencing this on my Frigate setup running in a Docker container on a Jetson Nano. Frigate is configured to save the videos to an NFS share on a server that needed to be rebooted from time to time. This would sometimes lead to the Jetson Nano’s limited drive space getting filled up. And then when the NFS share is remounted, the files that were stored locally on the Jetson Nano would be missing from the NFS mounted directory.
Solution
To fix this and prevent it from occurring, I had to change the local mount point on the Jetson Nano to be immutable, which will prevent any writes to that directory unless it is successfully mounted to the NFS server.
First, become root:
sudo su
Assuming you have Frigate running in a docker container, stop Frigate with the following command:
docker stop frigate
Unmount the NFS mount point:
umount /mnt/frigate_media_mount
Set the mount point to immutable:
chattr +i /mnt/frigate_media_mount
Test to see if a file can be written to the directory of the mount point when the NFS share is disconnected:
cd /mnt/frigate_media_mount
touch hello
# This should return an error
Next, assuming your mount point is setup in fstab, remount with the following command:
mount -a
Test again, this time to confirm that files can be written to the directory when the NFS is mounted:
cd /mnt/frigate_media_mount
touch hello
# This should work and not return an error.
The directory for the mount point is now configured so that no files will be written to the directory if the NFS share is not mounted, and Frigate will return errors when it tries to write the files locally.
You might want to consider also setting up a monitoring solution to catch the errors and remount the NFS share, which is beyond the scope of this guide.
Leave a Reply