Introduction

PJDFSTest is a file system test suite focused on POSIX compliance, primarily for FreeBSD file systems. It was originally written to validate the ZFS port to FreeBSD, but it now supports multiple operating systems and file systems.

This is a complete rewrite of the original test suite in Rust, as part of the Google Summer of Code 2022 program (https://summerofcode.withgoogle.com/archive/2022/projects/6XPYWLzJ).

NOTE: The documentation is still a work-in-progress

Build

cd rust
cargo run

Run as root

cd rust
cargo build && sudo ./target/debug/pjdfstest

Contributing

Please read the CONTRIBUTING.md file on how to contribute to this project. In addition to this book, you can also find the crate documentation by running cargo doc --open in the rust directory or by visiting the documentation.

Getting started

The test suite is as file system agnostic as possible and tries to comply with the POSIX specification. Typically, tests which make use of non-POSIX features are opt-in and only tests for syscalls which must be available on every POSIX system are ran. It can be configured with the configuration file, by specifying additional features supported by the file system/operating system.

Command-line interface

pjdfstest [OPTIONS] [--] TEST_PATTERNS

• -h, --help - Print help message
• -c, --configuration-file CONFIGURATION-FILE - Path of the configuration file
• -l, --list-features - List opt-in features
• -e, --exact - Match names exactly
• -v, --verbose - Verbose mode
• -p, --path PATH - Path where the test suite will be executed
• [--] TEST_PATTERNS - Filter tests which match against the provided patterns

Example: pjdfstest -c pjdfstest.toml chmod

Filter tests

It is possible to filter which tests should be run by specifying which parts should match. Tests are usually identified by syscall and optionally the file type on which it operates.

Rootless running

The test suite can be run without privileges. However, not all tests can be completed without privileges, therefore the coverage will be incomplete. For example, tests which need to switch users will not be run.

Dummy users/groups

The test suite needs dummy users and groups to be set up. This should be handled automatically when installing it via a package, but they need to be created otherwise. By default, the users (with the same name for the group associated to each of them) to create are:

• nobody
• tests
• pjdfstest

It is also possible to specify other users with the configuration file.

Create users

FreeBSD

cat <<EOF | adduser -w none -S -f -
pjdfstest::::::Dummy User for pjdfstest:/nonexistent:/sbin/nologin:
EOF

Linux

cat <<EOF | newusers
tests:x:::Dummy User for pjdfstest:/:/usr/bin/nologin
pjdfstest:x:::Dummy User for pjdfstest:/:/usr/bin/nologin
EOF

Compatibility

The test suite is designed to be compatible with multiple operating systems and file systems. It is primarily focused on POSIX compliance for FreeBSD file systems, but is also compatible with other operating systems and file systems in addition of supplementary tests.

Supported operating systems

Active support

The test suite has been tested and is actively maintained on the following operating systems:

• FreeBSD (main development platform)
  • UFS
  • ZFS
• Linux
  • ext4

Experimental support

The test suite has not been tested on the following operating systems, but should work:

• MacOS
• NetBSD
• OpenBSD
• Solaris
• DragonFly BSD
• Illumos
• Android
• iOS

Missing tests

The test suite is a complete rewrite of the original test suite in Rust. Many tests have been ported, but some tests are still missing.

Configuration file

The test runner can read a configuration file. For now, only the TOML format is supported. Its path can be specified by using the -c PATH flag.

Sections

[features]

Some features are not available for every file system. For tests requiring such features, the execution becomes opt-in. The user can enable their execution, by adding the corresponding feature as a key in this section. A list of these opt-in features is provided when executing the runner with -l argument.

For example, with posix_fallocate:

[features]
posix_fallocate = {}

# Can also be specified by using key notation
# [features.posix_fallocate]

Feature list

The following features can be enabled but do not require any additional configuration:

• chflags - The chflags syscall is available
• nfsv4_acls - NFSv4 style Access Control Lists are available
• posix_fallocate - The posix_fallocate syscall is available
• rename_ctime - rename changes st_ctime on success (POSIX does not require a file system to update a file's ctime when it gets renamed, but some file systems choose to do it anyway)
• stat_st_birthtime - struct stat contains an st_birthtime field
• chflags_sf_snapshot - The SF_SNAPSHOT flag can be set with chflags
• utime_now - The UTIME_NOW constant is available
• utimensat - The utimensat syscall is available

Following features require additional configuration.

file_flags

Some tests are related to file flags. However, not all file systems and operating systems support all flags. To give a sufficient level of granularity, each supported flag can be specified in the configuration with the file_flags array.

[features]
posix_fallocate = {}
file_flags = ["UF_IMMUTABLE"]

secondary_fs

Some tests require a secondary file system. This can be specified in the configuration with the secondary_fs key, but also with the secondary_fs argument. The argument takes precedence over the configuration.

[features]
secondary_fs = "/mnt/ISO"

[dummy_auth]

This section allows to modify the mechanism for switching users, which is required by some tests.

[dummy_auth]
entries = [
  ["nobody", "nobody"],
  # nogroup instead for some Linux distros
  # ["nobody", "nogroup"],
  ["tests", "tests"],
  ["pjdfstest", "pjdfstest"],
]

• entries - An entry is composed of a username and its associated group. Exactly 3 entries need to be specified if the default ones cannot be used.

[settings]

[settings]
naptime = 0.001
allow_remount = false

• naptime - The duration for a "short" sleep. It should be greater than the timestamp granularity of the file system under test. The default value is 1 second.
• allow_remount - If set to true, the runner will run the EROFS tests, which require to remount the file system on which pjdsfstest is run as read-only.

Structure

The package is made of the tests, and a test runner to launch them.

Tests (tests/)

To present how tests are organized, we take the chmod syscall as example.

There is a separate module for each syscall being tested. Within each of those modules, there may be either a single file, or a separate file for each aspect of the syscall.

The hierarchy is like this:

graph TD
TG[Syscall module<br /><i>chmod</i>] --> TC1[Aspect<br /><i>errno</i>]

TC1 --> TC1F1[Test case]
TC1 --> TC1F2[Test case]
TC1 --> TC1F3[Test case]
TC1 --> TC1F4[Test case]

TG --> TC2[Aspect<br /><i>permission</i>]

TC2 --> TC2F1[Test case]
TC2 --> TC2F2[Test case]

Layout

src/tests
├── chmod (syscall)
│   ├── errno.rs (aspect)
│   └── permission.rs (aspect)
├── mod.rs (glues syscalls together)
└── chmod.rs (syscall declaration and simple test cases)

tests/mod.rs

All the modules for the test groups should be declared in this file.

pub mod chmod;

Syscall module

A syscall module contains test cases related to a specific syscall. Its declaration should be in the <syscall_name>.rs file at the root of the tests/ directory. Common syscall-specific helpers can go here.

Aspect

An optional aspect module contains test cases that all relate to a common aspect of the syscall. Here "aspect" is a subjective area of related functionality. The aspect module may be either:

• in a single file, which contains all the test functions,
• in a folder, which contains multiple modules for the test functions, in which the case is declared.

Except in the case of a very large set of test functions, the first style should be preferred.

Test case

Each test case exercises a minimal piece of the syscall's functionality. Each must be registered with the test_case! macro.

crate::test_case! {
    /// open do not update parent directory ctime and mtime fields if
    /// the file previously existed.
    exists_no_update
}
fn exists_no_update(ctx: &mut TestContext) {
    let file = ctx.create(FileType::Regular).unwrap();

    assert_times_unchanged()
        .path(ctx.base_path(), CTIME | MTIME)
        .execute(ctx, false, || {
            assert!(open_wrapper(&file, Mode::from_bits_truncate(0o755)).is_ok());
        });
}

Test runner (main.rs)

The test runner has to run the tests, and provide a command-line interface to allow the user to modify how the tests should be run. It takes the tests from the specified test groups.

Test declaration

Test cases have the same structure than usual Rust tests, that is unwraping Results and using assertion macros (assert and assert_eq), the exception being that it should take a &mut TestContext parameter. It might also take a FileType argument if required. It also needs an additional declaration with the test_case! macro alongside the function, with the function name being the only mandatory argument.

For example:

// chmod/00.t:L58
crate::test_case! {
    /// chmod updates ctime when it succeeds
    update_ctime => [Regular, Dir, Fifo, Block, Char, Socket]
}
fn update_ctime(ctx: &mut TestContext, f_type: FileType) {
    let path = ctx.create(f_type).unwrap();
    assert_ctime_changed(ctx, &path, || {
        assert!(chmod(&path, Mode::from_bits_truncate(0o111)).is_ok());
    });
}

All the structures and functions needed are documented in the pjdfstest crate, which you can obtain by running cargo doc --open in the rust directory or by visiting the documentation.

Test context

The TestContext struct is a helper struct which provides methods to create files, sleep, change user, etc. It is passed as a parameter to the test functions and should be used to interact with the system in order to ensure that the tests are isolated and do not interfere with each other.

Serialization

When a test case needs to be run in a serialized manner, the SerializedTestContext struct should be used instead. It provides additional methods to change the user, group, supplementary groups, or umask of the process. Please see the Serialized test cases section for more information.

Assertions

The assert and assert_eq macros should be used to check the results of the tests. The assert macro should be used when the test is checking a condition, while the assert_eq macro should be used when the test is checking that two values are equal. In addition to these macros, the suite provides some additional assertion functions which should be used when appropriate. The tests module documentation provides a list of these functions.

Description

It is possible to provide doc comments which will be used as documentation for developers but also be displayed to users when they run the test. The doc comments should be written in the test_case! declaration, before anything. For example:

crate::test_case! {
    /// The file mode of a newly created file should not affect whether
    /// posix_fallocate will work, only the create args
    /// https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=154873
    affected_only_create_flags, serialized, root, FileSystemFeature::PosixFallocate
}

Parameterization

It is possible to give additional parameters to the test case macro, to modify the execution of the tests or add requirements.

File-system exclusive features

Some features are not available for every file system. For tests requiring such features, the execution becomes opt-in. A variant of the FileSystemFeature enum corresponding to this feature should be specified after potential root requirement and before guards. Multiple features can be specified, separated by a comma ,.

For example:

#[cfg(target_os = "freebsd")]
crate::test_case! {eperm_immutable_flag, FileSystemFeature::Chflags, FileSystemFeature::PosixFallocate ...}

Adding features

New features can be added to the FileSystemFeature enum. A description of the feature should be provided as documentation for both developers and users.

Guards

It is possible to specify "guards", which are functions which checks if a requirement is met and return an error if not so the test will be skipped. They can be specified by appending function names after a ; separator, after potential root requirement and features.

The function has to take a &Config argument which contains the current configuration and a &Path which represents the parent folder of the potential test context which would be created.

Guard signature

/// Function which indicates if the test should be skipped by returning an error.
pub type Guard = fn(&Config, &Path) -> anyhow::Result<()>;

Example

fn has_reasonable_link_max(_: &Config, base_path: &Path) -> anyhow::Result<()> {
    let link_max = pathconf(base_path, nix::unistd::PathconfVar::LINK_MAX)?
        .ok_or_else(|| anyhow::anyhow!("Failed to get LINK_MAX value"))?;

    if link_max >= LINK_MAX_LIMIT {
        anyhow::bail!("LINK_MAX value is too high ({link_max}, expected smaller than {LINK_MAX_LIMIT}");
    }

    Ok(())
}

crate::test_case! {
    /// link returns EMLINK if the link count of the file named by name1 would exceed {LINK_MAX}
    link_count_max; has_reasonable_link_max
}
...

Root privileges

Some tests may need root privileges to run. To declare that a test function require such privileges, root should be added to its declaration. For example:

crate::test_case!{change_perm, root}

The root requirement is automatically added for privileged file types, namely block and char.

File types

Some test cases need to test over different file types. The file types should be added at the end of the test case declaration, within brackets and with a fat arrow before (=> [Regular]). The test function should also accept a FileType parameter to operate on.

For example:

crate::test_case! {change_perm, root, FileSystemFeature::Chflags => [Regular, Fifo, Block, Char, Socket]}
fn change_perm(ctx: &mut TestContext, f_type: FileType) {

Platform-specific features

Some features (like lchmod) are not supported on every operating system. When a test make use of such feature, it is possible to restrain its compilation to the supported operating systems, with the attribute #[cfg(feature_name)]. It is also possible to apply this attribute on an aspect or even a syscall module. For example:

#[cfg(lchmod)]
mod lchmod;

To declare it, the feature and its requirements have to be specified in the build.rs file using the usual conditional compilation syntax. Then, the feature should be added to the cfg_aliases! macro. With lchmod, we would get:

    cfg_aliases! {
        ...
        lchmod: { any(target_os = "netbsd", target_os = "freebsd", target_os = "dragonfly") },
        ...
    }

Serialized test cases

Some test cases need functions only available when they are run serialized, especially when they affect the whole process. An example is changing user (SerializedTestContext::as_user). To have access to these functions, the test should be declared with a SerializedTestContext parameter in place of TestContext and the serialized keyword should be prepended before features and root requirement.

For example:

crate::test_case! {
    /// link changes neither ctime of file nor ctime or mtime of parent when it fails
    // link/00.t#77
    unchanged_ctime_fails, serialized, root => [Regular, Fifo, Block, Char, Socket]
}
fn unchanged_ctime_fails(ctx: &mut SerializedTestContext, ft: FileType) {
    let file = ctx.create(ft).unwrap();
    let new_path = ctx.gen_path();

    let user = ctx.get_new_user();
    assert_times_unchanged()
        .path(&file, CTIME)
        .path(ctx.base_path(), CTIME | MTIME)
        .execute(ctx, false, || {
            ctx.as_user(user, None, || {
                assert!(matches!(
                    link(&file, &new_path),
                    Err(Errno::EPERM | Errno::EACCES)
                ));
            })
        });
}