This shows how to set up a read/write layer on top of a read-only DwarFS image, incredibly handy if you need to modify or extend the contents of your read-only image temporarily.
My primary use case is storing highly compressed NATS dumps in a DwarFS image. Then, I set up a read/write layer to potentially alter these dumps as needed. Once the changes are no longer needed, I can simply wipe the read/write layer to return to the original NATS dumps.
Create a DwarFS image
Create a DwarFS image with the NATS dumps. This is a one-time operation.
mkdwfs -i nats -o nats-2023.dwarfs
Prepare directories
Create a set of directories for the original data overlay.
mkdir nats-2023 && cd nats-2023
mkdir nats-ro
mkdir nats-rw
mkdir nats-work
mkdir nats
Mount the DwarFS image
Mount the DwarFS image. -o allow_root is necessary to ensure overlayfs has
access to the mounted file system. You might need to adjust /etc/fuse.conf by
uncommenting or adding user_allow_other.
dwarfs ../nats-2023.dwarfs nats-ro -o allow_root
Now set up overlayfs:
sudo mount -t overlay overlay -o lowerdir=nats-ro,upperdir=nats-rw,workdir=nats-work nats
You should now have access to a writable version of your DwarFS image at
nats-2023/nats.
Unmount the overlayfs
When you're done with the changes, unmount the overlayfs:
sudo umount nats
Keep the changes
You can go further than this. Suppose you have different sets of modifications you regularly want to apply to the base DwarFS image. In that case, you can build a new DwarFS image from the read-write directory after unmounting the overlayfs and selectively add this by passing a colon-separated list to the lowerdir option when setting up the overlayfs mount:
sudo mount -t overlay overlay -o lowerdir=nats-ro:additional-modules nats
If you want this merged overlay to be writable, just add in the upperdir and workdir options from before again.
Script it
Here's a script that simplifies the process.
#!/usr/bin/env bash
set -o errexit
set -o pipefail
set -o nounset
# Default directory setup
display_full_usage() {
cat <<EOF
NAME
$(basename "$0") - Manages DwarFS images.
COMMANDS
create <source-dir> [output-path] [-c, --check-perms]
Creates a compressed read-only DwarFS image from 'source-dir' dir.
Optionally checks for files without group write permissions.
mount <mount-point> <image-path>
Mounts the 'image-path' image to 'mount-point' and sets up a writable overlay.
umount <mount-point>
Unmounts the DwarFS and overlay filesystems at 'mount-point'.
OPTIONS
-h, --help
Displays this help message and exits.
-c, --check-perms
Checks if the source directory has files without 'g+w' permissions.
EOF
}
command=""
args=()
check_perms=0
parse_args() {
local stop_processing_options=0
# Parse command-line arguments
while [[ $# -gt 0 ]]; do
if [[ $stop_processing_options -eq 1 ]]; then
args+=("$1")
shift
continue
fi
case $1 in
-h | --help)
display_full_usage
exit 0
;;
-c | --check-perms)
check_perms=1
shift
;;
--)
stop_processing_options=1
shift
;;
-*)
echo "Unknown option: $1"
exit 1
;;
*)
if [[ -z "$command" ]]; then
command="$1"
else
args+=("$1")
fi
shift
;;
esac
done
if [[ -z "${command:-}" ]]; then
echo "error: no command specified"
display_full_usage
exit 1
fi
}
check_dependencies() {
if ! command -v mkdwarfs &>/dev/null; then
echo "error: mkdwarfs is not installed. Install dwarfs: https://github.com/mhx/dwarfs"
exit 1
fi
if ! command -v dwarfs &>/dev/null; then
echo "error: dwarfs is not installed. Install dwarfs: https://github.com/mhx/dwarfs"
exit 1
fi
}
create_image() {
local source_dir="$1"
local default_output_path
default_output_path="$(basename "${source_dir}".dwarfs)"
local output_path="${2:-$default_output_path}"
# check if output path ends with .dwarfs else append it
if [[ ! "$output_path" =~ \.dwarfs$ ]]; then
output_path="${output_path}.dwarfs"
fi
if [[ $check_perms -eq 1 ]]; then
if ! check_file_permissions "$source_dir"; then
echo "Permission check failed. Some files don't have 'g+w' permissions."
echo "Run the following command to fix permissions:"
echo "chmod -R g+w $source_dir"
exit 1
fi
fi
echo "Creating DwarFS image from '${source_dir}' directory..."
mkdwarfs -i "$source_dir" -o "${output_path}"
echo "DwarFS image created at ${output_path}"
}
check_file_permissions() {
local dir="$1"
if find "$dir" -type f ! -perm /g+w -print -quit | grep -q '.'; then
return 1
fi
return 0
}
mount_image() {
local mount_point="$1"
local image_path="$2"
echo "Mounting DwarFS image..."
local read_only_dir="${mount_point}-ro"
local read_write_dir="${mount_point}-rw"
local work_dir="${mount_point}-work"
mkdir -p "$read_only_dir" "$read_write_dir" "$work_dir" "$mount_point"
dwarfs "$image_path" "$read_only_dir" -o allow_root
sudo mount -t overlay overlay -o lowerdir="$read_only_dir",upperdir="$read_write_dir",workdir="$work_dir" "$mount_point"
echo "DwarFS image mounted at $mount_point"
}
unmount_image() {
local mount_point="$1"
echo "Unmounting overlay and DwarFS image at '${mount_point}'..."
sudo umount "$mount_point"
sudo umount "${mount_point}-ro"
echo "Unmounted DwarFS image at '${mount_point}'"
}
execute_command() {
case $command in
create)
if [[ ${#args[@]} -lt 1 ]]; then
echo "err: create command requires one argument for source directory."
display_full_usage
exit 1
fi
create_image "${args[@]}"
;;
mount)
if [[ ${#args[@]} -ne 2 ]]; then
echo "err: mount command requires exactly two arguments: mount point and image path."
display_full_usage
exit 1
fi
mount_image "${args[0]}" "${args[1]}"
;;
unmount | umount)
if [[ ${#args[@]} -ne 1 ]]; then
echo "err: unmount command requires exactly one argument for mount point."
display_full_usage
exit 1
fi
unmount_image "${args[0]}"
;;
*)
echo "Invalid command: $command"
display_full_usage
exit 1
;;
esac
}
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
parse_args "$@"
check_dependencies
execute_command
fi
I have it saved as compressfs and use it like this:
compressfs -h
NAME
compressfs - Manages DwarFS images.
COMMANDS
create <source-dir> [output-path]
Creates a compressed read-only DwarFS image from 'source-dir' dir.
mount <image-path> <mount-point>
Mounts the 'image-path' image to 'mount-point' and sets up a writable overlay.
umount <mount-point>
Unmounts the DwarFS and overlay filesystems at 'mount-point'.
OPTIONS
-h, --help
Displays this help message and exits.
Create a DwarFS image:
compressfs create nats-2023
Mount the DwarFS image:
compressfs mount mounts/nats-2023 nats-2023.dwarfs
Unmount the DwarFS image:
compressfs umount mounts/nats-2023