zfs send -R broken on 2.0.1 and Linux 5.10 when piped to /dev/null #11445

aerusso · 2021-01-08T14:46:08Z

System information

Type	Version/Name
Distribution Name	Debian
Distribution Version	sid
Linux Kernel	5.10.4
Architecture	amd64
ZFS Version	2.0.1

Describe the problem you're observing

Many ZTS failures related to send:

FAIL cli_root/zfs_send/zfs_send-b (expected PASS)
FAIL cli_root/zfs_send/zfs_send_003_pos (expected PASS)
FAIL cli_root/zfs_send/zfs_send_005_pos (expected PASS)
FAIL cli_root/zfs_send/zfs_send_encrypted (expected PASS)
FAIL cli_root/zfs_send/zfs_send_raw (expected PASS)
FAIL pyzfs/pyzfs_unittest (expected PASS)
FAIL redacted_send/redacted_negative (expected PASS)
FAIL removal/removal_with_send (expected PASS)

This is not reproducible on 5.9.

Describe how to reproduce the problem

truncate -s 512M /tmp/loopdev0
losetup loop0 /tmp/loopdev0
zpool create testpool /dev/loop0
zfs snapshot -r testpool@snap
zfs send -R testpool@snap >/dev/null

Include any warning/errors/backtraces from the system logs

This gives an error:

internal error: warning: cannot send 'testpool@snap': Invalid argument

The send command results a log in dmesg:

kernel: kernel write not supported for file /null (pid: 2241 comm: zfs)

The text was updated successfully, but these errors were encountered:

behlendorf · 2021-01-08T19:07:09Z

I've starting looking in to this and the short version is that the kernel kernel_write() and kernel_read() functions can no longer access the /dev/null and /dev/zero virtual devices because they do not register any *_iter operations. The good news is that it should only effect those two devices. Any filesystem which supports the *_iter operations (and most do) will continue to work fine.

For background here's the relevant kernel commit which adds the new check to __kernel_write() and a link to the upstream discussion regarding this change. We'll have to give some thought as the the best way to handle this.

aerusso · 2021-01-09T14:43:21Z

Thanks, this is far less concerning than I originally feared.

The ZTS now passes on 5.10 with 2.0.1. If there's a need for it, I can provide the patches I used to avoid piping to /dev/null.

behlendorf · 2021-01-11T17:24:53Z

That would be helpful to at least keep the CI happy until we settle on a solution we're happy with. Would you mind opening a PR.

As described in openzfs#11445, the kernel interface kernel_write and kernel_read cannot access /dev/null and /dev/zero. This causes ZTS failures (for positive tests) and may mask errors (for negative tests). Until the underlying bug is addressed, clean up the output from the ZTS by avoiding directly piping to or from /dev/null and /dev/zero. For /dev/zero input, simply use a pipe: cat </dev/zero | ... . Because of pipe failure semantics, we cannot simply pipe the output to cat, and then redirect it to /dev/null. Use a temporary file under $TESTDIR for output in favor of /dev/null. Signed-off-by: Antonio Russo <aerusso@aerusso.net>

As described in openzfs#11445, the kernel interface kernel_write and kernel_read cannot access /dev/null and /dev/zero. This causes ZTS failures (for positive tests) and may mask errors (for negative tests). Until the underlying bug is addressed, clean up the output from the ZTS by avoiding directly piping to or from /dev/null and /dev/zero. For /dev/zero input, simply use a pipe: cat </dev/zero | ... . Because of pipe failure semantics, we cannot simply pipe the output to cat, and then redirect it to /dev/null. Use a temporary file under $TEST_BASE_DIR for output in favor of /dev/null. Signed-off-by: Antonio Russo <aerusso@aerusso.net>

As described in openzfs#11445, the kernel interface kernel_{read,write} no longer act on special devices. In the ZTS, zfs send and receive are tested by piping to these devices, leading to spurious failures (for positive tests) and may mask errors (for negative tests). Until a more permanent mechanism to address this deficiency is developed, clean up the output from the ZTS by avoiding directly piping to or from /dev/null and /dev/zero. For /dev/zero input, simply use a pipe: `cat </dev/zero |` . However, for /dev/null output, the shell semantics for pipe failures means that zfs send error codes will be masked by the successful `| cat >/dev/null` command execution. In that case, use a temporary file under $TEST_BASE_DIR for output in favor. Signed-off-by: Antonio Russo <aerusso@aerusso.net>

As described in #11445, the kernel interface kernel_{read,write} no longer act on special devices. In the ZTS, zfs send and receive are tested by piping to these devices, leading to spurious failures (for positive tests) and may mask errors (for negative tests). Until a more permanent mechanism to address this deficiency is developed, clean up the output from the ZTS by avoiding directly piping to or from /dev/null and /dev/zero. For /dev/zero input, simply use a pipe: `cat </dev/zero |` . However, for /dev/null output, the shell semantics for pipe failures means that zfs send error codes will be masked by the successful `| cat >/dev/null` command execution. In that case, use a temporary file under $TEST_BASE_DIR for output in favor. Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Attila Fülöp <attila@fueloep.org> Signed-off-by: Antonio Russo <aerusso@aerusso.net> Closes #11478

As described in openzfs#11445, the kernel interface kernel_{read,write} no longer act on special devices. In the ZTS, zfs send and receive are tested by piping to these devices, leading to spurious failures (for positive tests) and may mask errors (for negative tests). Until a more permanent mechanism to address this deficiency is developed, clean up the output from the ZTS by avoiding directly piping to or from /dev/null and /dev/zero. For /dev/zero input, simply use a pipe: `cat </dev/zero |` . However, for /dev/null output, the shell semantics for pipe failures means that zfs send error codes will be masked by the successful `| cat >/dev/null` command execution. In that case, use a temporary file under $TEST_BASE_DIR for output in favor. Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Attila Fülöp <attila@fueloep.org> Signed-off-by: Antonio Russo <aerusso@aerusso.net> Closes openzfs#11478

As described in #11445, the kernel interface kernel_{read,write} no longer act on special devices. In the ZTS, zfs send and receive are tested by piping to these devices, leading to spurious failures (for positive tests) and may mask errors (for negative tests). Until a more permanent mechanism to address this deficiency is developed, clean up the output from the ZTS by avoiding directly piping to or from /dev/null and /dev/zero. For /dev/zero input, simply use a pipe: `cat </dev/zero |` . However, for /dev/null output, the shell semantics for pipe failures means that zfs send error codes will be masked by the successful `| cat >/dev/null` command execution. In that case, use a temporary file under $TEST_BASE_DIR for output in favor. Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Attila Fülöp <attila@fueloep.org> Signed-off-by: Antonio Russo <aerusso@aerusso.net> Closes #11478

As described in openzfs#11445, the kernel interface kernel_{read,write} no longer act on special devices. In the ZTS, zfs send and receive are tested by piping to these devices, leading to spurious failures (for positive tests) and may mask errors (for negative tests). Until a more permanent mechanism to address this deficiency is developed, clean up the output from the ZTS by avoiding directly piping to or from /dev/null and /dev/zero. For /dev/zero input, simply use a pipe: `cat </dev/zero |` . However, for /dev/null output, the shell semantics for pipe failures means that zfs send error codes will be masked by the successful `| cat >/dev/null` command execution. In that case, use a temporary file under $TEST_BASE_DIR for output in favor. Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Attila Fülöp <attila@fueloep.org> Signed-off-by: Antonio Russo <aerusso@aerusso.net> Closes openzfs#11478

After the changes documented in openzfs#11445, writing to certain files errored out with an opaque error message. Unfortunately, the path of least resistance to fixing this seemed to be sticking a pipe into the path of zfs send, along with a thread to copy data to stdout. Signed-off-by: Rich Ercolani <rincebrain@gmail.com>

As described in openzfs#11445, the kernel interface kernel_{read,write} no longer act on special devices. In the ZTS, zfs send and receive are tested by piping to these devices, leading to spurious failures (for positive tests) and may mask errors (for negative tests). Until a more permanent mechanism to address this deficiency is developed, clean up the output from the ZTS by avoiding directly piping to or from /dev/null and /dev/zero. For /dev/zero input, simply use a pipe: `cat </dev/zero |` . However, for /dev/null output, the shell semantics for pipe failures means that zfs send error codes will be masked by the successful `| cat >/dev/null` command execution. In that case, use a temporary file under $TEST_BASE_DIR for output in favor. Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Attila Fülöp <attila@fueloep.org> Signed-off-by: Antonio Russo <aerusso@aerusso.net> Closes openzfs#11478

After the changes documented in openzfs#11445, writing to certain files errored out with an opaque error message. Unfortunately, the path of least resistance to fixing this seemed to be sticking a pipe into the path of zfs send, along with a thread to copy data to stdout. Signed-off-by: Rich Ercolani <rincebrain@gmail.com>

By introducing lzc_send_wrapper() and routing all ZFS_IOC_SEND* users through it, we fix a Linux 5.10-introduced bug (see comment) This is all /transparent/ to the users API, ABI, and usage-wise, and disabled on FreeBSD and if the output is already a pipe, and transparently nestable (i.e. zfs_send_one() is wrapped, but so is lzc_send_redacted() it calls to ‒ this wouldn't be strictly necessary if ZFS_IOC_SEND_PROGRESS wasn't strictly denominational w.r.t. the descriptor the send is happening on) Supersedes openzfs#11992 Closes openzfs#11445 Co-authored-by: Rich Ercolani <rincebrain@gmail.com> Signed-off-by: Ahelenia Ziemiańska <nabijaczleweli@nabijaczleweli.xyz>

Unfortunately, openzfs#11445 means while we fail gracefully now, we still fail, unless people want to implement a complex workaround just to support /dev/null. So let's just use the cheap workaround in a test for now. Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: John Kennedy <john.kennedy@delphix.com> Signed-off-by: Rich Ercolani <rincebrain@gmail.com> Closes openzfs#12872

By introducing lzc_send_wrapper() and routing all ZFS_IOC_SEND* users through it, we fix a Linux 5.10-introduced bug (see comment) This is all /transparent/ to the users API, ABI, and usage-wise, and disabled on FreeBSD and if the output is already a pipe, and transparently nestable (i.e. zfs_send_one() is wrapped, but so is lzc_send_redacted() it calls to ‒ this wouldn't be strictly necessary if ZFS_IOC_SEND_PROGRESS wasn't strictly denominational w.r.t. the descriptor the send is happening on) Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Paul Dagnelie <pcd@delphix.com> Co-authored-by: Rich Ercolani <rincebrain@gmail.com> Signed-off-by: Ahelenia Ziemiańska <nabijaczleweli@nabijaczleweli.xyz> Closes openzfs#11445 Closes openzfs#13133

Unfortunately, openzfs#11445 means while we fail gracefully now, we still fail, unless people want to implement a complex workaround just to support /dev/null. So let's just use the cheap workaround in a test for now. Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: John Kennedy <john.kennedy@delphix.com> Signed-off-by: Rich Ercolani <rincebrain@gmail.com> Closes openzfs#12872

By introducing lzc_send_wrapper() and routing all ZFS_IOC_SEND* users through it, we fix a Linux 5.10-introduced bug (see comment) This is all /transparent/ to the users API, ABI, and usage-wise, and disabled on FreeBSD and if the output is already a pipe, and transparently nestable (i.e. zfs_send_one() is wrapped, but so is lzc_send_redacted() it calls to ‒ this wouldn't be strictly necessary if ZFS_IOC_SEND_PROGRESS wasn't strictly denominational w.r.t. the descriptor the send is happening on) Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Reviewed-by: Paul Dagnelie <pcd@delphix.com> Co-authored-by: Rich Ercolani <rincebrain@gmail.com> Signed-off-by: Ahelenia Ziemiańska <nabijaczleweli@nabijaczleweli.xyz> Closes openzfs#11445 Closes openzfs#13133

aerusso added Status: Triage Needed New issue which needs to be triaged Type: Defect Incorrect behavior (e.g. crash, hang) labels Jan 8, 2021

behlendorf removed the Status: Triage Needed New issue which needs to be triaged label Jan 8, 2021

aerusso changed the title ~~zfs send -R broken on 2.0.1 and Linux 5.10~~ zfs send -R broken on 2.0.1 and Linux 5.10 when piped to /dev/null Jan 9, 2021

aerusso mentioned this issue Jan 18, 2021

zts: avoid piping to special devices #11478

Merged

13 tasks

This was referenced May 2, 2021

zfs send on readonly pools broke in 2.0 #11985

Closed

Added error for writing to /dev/ on Linux. #11991

Merged

rincebrain mentioned this issue May 3, 2021

Refactored lzc_send_resume_redacted with a thread. #11992

Closed

13 tasks

rincebrain mentioned this issue Jun 8, 2021

ZFS on Linux null pointer dereference #11679

Open

glueckself mentioned this issue Aug 2, 2021

permanent errors (ereport.fs.zfs.authentication) reported after syncoid snapshot/send workload #11688

Open

behlendorf mentioned this issue Oct 8, 2021

Fail invalid incremental recursive send gracefully #12533

Merged

13 tasks

behlendorf closed this as completed in 3a909fe Mar 8, 2022

jgoerzen mentioned this issue Jun 19, 2022

zfs receive deadlocks when zstdcat piped to it #13571

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

zfs send -R broken on 2.0.1 and Linux 5.10 when piped to /dev/null #11445

zfs send -R broken on 2.0.1 and Linux 5.10 when piped to /dev/null #11445

aerusso commented Jan 8, 2021

behlendorf commented Jan 8, 2021

aerusso commented Jan 9, 2021

behlendorf commented Jan 11, 2021

zfs send -R broken on 2.0.1 and Linux 5.10 when piped to /dev/null #11445

zfs send -R broken on 2.0.1 and Linux 5.10 when piped to /dev/null #11445

Comments

aerusso commented Jan 8, 2021

System information

Describe the problem you're observing

Describe how to reproduce the problem

Include any warning/errors/backtraces from the system logs

behlendorf commented Jan 8, 2021

aerusso commented Jan 9, 2021

behlendorf commented Jan 11, 2021