[systemd-devel] Using btrfs subvolumes with systemd

Discussion:

Florent Peterschmitt

2016-06-08 18:48:07 UTC

Hi,

I have a recurrent problem at boot time, when mounting filesystems.

First :

o This problem was on systemd v229 and is still here with v230
o I use BTRFS for everything
o I have 6 subvolumes
o I use a subvolume as root filesystem, then mount other subvolumes
o I use Gentoo as main distribution.

Here are some outputs:

# btrfs subvolume list /
ID 257 gen 90 top level 5 path archlinux
ID 258 gen 4296 top level 5 path gentoo
ID 259 gen 2715 top level 5 path lib.modules
ID 260 gen 4225 top level 5 path SteamLibrary
ID 261 gen 613 top level 5 path ubuntu
ID 262 gen 2875 top level 5 path usr.src

# cat /etc/fstab
/dev/mapper/root / btrfs autodefrag,noatime,subvol=gentoo 0 0
tmpfs /var/tmp/portage/ tmpfs defaults 0 0
UUID=98c23f22-566e-4a17-b663-3c0ce781a897 /boot ext4 defaults,noatime,nodev 0 0
/dev/mapper/root /lib64/modules btrfs autodefrag,noatime,subvol=lib.modules 0 0
/dev/mapper/root /usr/src btrfs autodefrag,noatime,subvol=usr.src 0 0
/dev/mapper/root /home/florent/SteamLibrary btrfs autodefrag,noatime,subvol=SteamLibrary 0 0

/dev/cdrom /mnt/cdrom auto noauto,ro 0 0
/dev/fd0 /mnt/floppy auto noauto 0 0

As you can see, /dev/mapper/root is the btrfs filesystem, created with "mkfs.btrfs
/dev/mapper/root", nothing special. Same for subvolumes.

Here is the problem:

Each time i boot, this error happen:
# journalctl -xe
[...]
systemd[1]: dev-mapper-root.device: Job dev-mapper-root.device/start timed out.
systemd[1]: Timed out waiting for device /dev/mapper/root.
[...]

After StartExecTimeout (reduced to 10s…), i'm dropped into an emergency shell. There, i fire "mount -a", then quit, and the system boots.

But only gentoo does continue booting, with ArchLinux (v230 too), i'm simply dropped into emergency shell forever until i mask usr-src.mount and lib-modules.mount before quitting!

Is this a miss-configuration or a problem with systemd's device handling?

Thanks.

Florent Peterschmitt

Florent Peterschmitt

2016-06-08 19:11:11 UTC

Permalink

Post by Florent Peterschmitt
Hi,
I have a recurrent problem at boot time, when mounting filesystems.
o This problem was on systemd v229 and is still here with v230
o I use BTRFS for everything
o I have 6 subvolumes
o I use a subvolume as root filesystem, then mount other subvolumes
o I use Gentoo as main distribution.
# btrfs subvolume list /
ID 257 gen 90 top level 5 path archlinux
ID 258 gen 4296 top level 5 path gentoo
ID 259 gen 2715 top level 5 path lib.modules
ID 260 gen 4225 top level 5 path SteamLibrary
ID 261 gen 613 top level 5 path ubuntu
ID 262 gen 2875 top level 5 path usr.src
# cat /etc/fstab
/dev/mapper/root / btrfs autodefrag,noatime,subvol=gentoo 0 0
tmpfs /var/tmp/portage/ tmpfs defaults 0 0
UUID=98c23f22-566e-4a17-b663-3c0ce781a897 /boot ext4 defaults,noatime,nodev 0 0
/dev/mapper/root /lib64/modules btrfs autodefrag,noatime,subvol=lib.modules 0 0
/dev/mapper/root /usr/src btrfs autodefrag,noatime,subvol=usr.src 0 0
/dev/mapper/root /home/florent/SteamLibrary btrfs autodefrag,noatime,subvol=SteamLibrary 0 0
/dev/cdrom /mnt/cdrom auto noauto,ro 0 0
/dev/fd0 /mnt/floppy auto noauto 0 0
As you can see, /dev/mapper/root is the btrfs filesystem, created with "mkfs.btrfs
/dev/mapper/root", nothing special. Same for subvolumes.
# journalctl -xe
[...]
systemd[1]: dev-mapper-root.device: Job dev-mapper-root.device/start timed out.
systemd[1]: Timed out waiting for device /dev/mapper/root.
[...]
After StartExecTimeout (reduced to 10s…), i'm dropped into an emergency shell. There, i fire "mount
-a", then quit, and the system boots.
But only gentoo does continue booting, with ArchLinux (v230 too), i'm simply dropped into emergency
shell forever until i mask usr-src.mount and lib-modules.mount before quitting!
Is this a miss-configuration or a problem with systemd's device handling?
Thanks.
Florent Peterschmitt

I forgot to mention that i use this workaround:

ln -s /dev/mapper/root /etc/dev-usr-src
ln -s /dev/mapper/root /etc/dev-lib-modules
ln -s /dev/mapper/root /etc/dev-steam

Then adapt /etc/fstab to use /etc/dev-* instead of /dev/mapper/root

Florent Peterschmitt

Chris Murphy

2016-06-08 20:34:10 UTC

Permalink

On Wed, Jun 8, 2016 at 12:48 PM, Florent Peterschmitt

Post by Florent Peterschmitt
Hi,
I have a recurrent problem at boot time, when mounting filesystems.
o This problem was on systemd v229 and is still here with v230
o I use BTRFS for everything
o I have 6 subvolumes
o I use a subvolume as root filesystem, then mount other subvolumes
o I use Gentoo as main distribution.
# btrfs subvolume list /
ID 257 gen 90 top level 5 path archlinux
ID 258 gen 4296 top level 5 path gentoo
ID 259 gen 2715 top level 5 path lib.modules
ID 260 gen 4225 top level 5 path SteamLibrary
ID 261 gen 613 top level 5 path ubuntu
ID 262 gen 2875 top level 5 path usr.src
# cat /etc/fstab
/dev/mapper/root / btrfs autodefrag,noatime,subvol=gentoo 0 0
tmpfs /var/tmp/portage/ tmpfs defaults 0 0
UUID=98c23f22-566e-4a17-b663-3c0ce781a897 /boot ext4 defaults,noatime,nodev 0 0
/dev/mapper/root /lib64/modules btrfs autodefrag,noatime,subvol=lib.modules 0 0
/dev/mapper/root /usr/src btrfs autodefrag,noatime,subvol=usr.src 0 0
/dev/mapper/root /home/florent/SteamLibrary btrfs autodefrag,noatime,subvol=SteamLibrary 0 0
/dev/cdrom /mnt/cdrom auto noauto,ro 0 0
/dev/fd0 /mnt/floppy auto noauto 0 0
As you can see, /dev/mapper/root is the btrfs filesystem, created with "mkfs.btrfs
/dev/mapper/root", nothing special. Same for subvolumes.
# journalctl -xe
[...]
systemd[1]: dev-mapper-root.device: Job dev-mapper-root.device/start timed out.
systemd[1]: Timed out waiting for device /dev/mapper/root.
[...]
After StartExecTimeout (reduced to 10s…), i'm dropped into an emergency shell. There, i fire "mount -a", then quit, and the system boots.
But only gentoo does continue booting, with ArchLinux (v230 too), i'm simply dropped into emergency shell forever until i mask usr-src.mount and lib-modules.mount before quitting!
Is this a miss-configuration or a problem with systemd's device handling?

I take it /dev/mapper/root is a dmcrypt device?

If you can add systemd.log_level=debug at boot time as a boot
parameter (remove the symlink modification so the boot fails as it has
been), mount -a, then post the output from 'journalctl -b -o
short-monotonic > journal.log' somewhere, that might provide a bit
more context.

That it works for gentoo and not arch makes me wonder if this is
actually an issue in the initramfs. You could also add rd.debug as a
boot parameter, but the combination of both debugs makes things super
slow and the journal output will be quite large.

--
Chris Murphy

Lennart Poettering

2016-06-09 09:53:48 UTC

Permalink

Post by Florent Peterschmitt
# journalctl -xe
[...]
systemd[1]: dev-mapper-root.device: Job dev-mapper-root.device/start timed out.
systemd[1]: Timed out waiting for device /dev/mapper/root.
[...]

This appears unrelated to the fact that btrfs is used, and most likely
is caused by borked DM logic. Maybe you built the DM tools without
proper udev support? They used to have a compile-time option there,
which defaulted to off, which makes DM incompatible with systemd. Most
distributions don't bother, and always set it, but given that you use
Gentoo I figure yours might not, dunno.

Lennart

--
Lennart Poettering, Red Hat

Florent Peterschmitt

2016-06-09 11:10:04 UTC

Permalink

Post by Lennart Poettering

Post by Florent Peterschmitt
# journalctl -xe
[...]
systemd[1]: dev-mapper-root.device: Job dev-mapper-root.device/start timed out.
systemd[1]: Timed out waiting for device /dev/mapper/root.
[...]

Well, as asked before, /dev/mapper/root is effectively a dmcrypt device, opened with cryptsetup and built with udev support.

Also, I should add that the root filesystem is mounted, it's only subsequent mount points that are failing, after the root filesystem is mounted (the "gentoo" btrfs subvolume).

Just in case i'm unclear, here is how it happend:

1. initramfs boots and cryptsetup opens the crypt device to /dev/mapper/root (cryptsetup luksOpen /dev/sda1 root)
2. /dev/mapper/root is mounted on the new root, using the "gentoo" subvolume (root=/dev/mapper/root rootflags=subvol=gentoo)
3. initramfs starts systemd (init=/usr/lib/systemd/systemd)
4. systemd remounts /dev/mapper/root in rw successfully
5. the autogenerated mount units from fstab fail

My position on that problem is that systemd only looks for the device file's path to generated mount units, and somehow a conflict appears somewhere. With the symlinks it does generate some etc-dev-usr-src, etc-dev-lib-modules... mount units.

But, as requested before, i'll send log files to give you some hints.

Also, I think it could be nice to setup a virtual machine with no dmcrypt, juste some /dev/vda1, btrfs and two subvolumes.

Florent Peterschmitt

Lennart Poettering

2016-06-09 13:31:09 UTC

Permalink

Post by Florent Peterschmitt
My position on that problem is that systemd only looks for the
device file's path to generated mount units, and somehow a conflict
appears somewhere. With the symlinks it does generate some
etc-dev-usr-src, etc-dev-lib-modules... mount units.

Well, as soon as something is mounted it's mounted, systemd accepts
that.

Try "udevadm info /dev/mapper/root". That should show you what udev
knows about the device. Then try "systemctl status
/dev/mapper/root". That will show you what systemd knows about it.

Lennart

--
Lennart Poettering, Red Hat

Florent Peterschmitt

2016-06-09 20:30:11 UTC

Permalink

Post by Chris Murphy
On Wed, Jun 8, 2016 at 12:48 PM, Florent Peterschmitt

I take it /dev/mapper/root is a dmcrypt device?
If you can add systemd.log_level=debug at boot time as a boot
parameter (remove the symlink modification so the boot fails as it has
been), mount -a, then post the output from 'journalctl -b -o
short-monotonic > journal.log' somewhere, that might provide a bit
more context.
That it works for gentoo and not arch makes me wonder if this is
actually an issue in the initramfs. You could also add rd.debug as a
boot parameter, but the combination of both debugs makes things super
slow and the journal output will be quite large.
--
Chris Murphy

The log with systemd.log_level=debug at boot is located here: http://data.peterschmitt.fr/systemd.log

Here is the output from udevadm about /dev/mapper/root:

# udevadm info /dev/mapper/root
P: /devices/virtual/block/dm-0
N: dm-0
E: DEVNAME=/dev/dm-0
E: DEVPATH=/devices/virtual/block/dm-0
E: DEVTYPE=disk
E: DM_UDEV_DISABLE_DISK_RULES_FLAG=1
E: DM_UDEV_DISABLE_OTHER_RULES_FLAG=1
E: DM_UDEV_DISABLE_SUBSYSTEM_RULES_FLAG=1
E: MAJOR=253
E: MINOR=0
E: SUBSYSTEM=block
E: SYSTEMD_READY=0
E: TAGS=:systemd:
E: USEC_INITIALIZED=18904677

Florent Peterschmitt

Andrei Borzenkov

2016-06-10 06:06:27 UTC

Permalink

Post by Florent Peterschmitt
The log with systemd.log_level=debug at boot is located here: http://data.peterschmitt.fr/systemd.log

Unfortunately this log while being verbose does not add any useful
information.

Post by Florent Peterschmitt
# udevadm info /dev/mapper/root
P: /devices/virtual/block/dm-0
N: dm-0
E: DEVNAME=/dev/dm-0
E: DEVPATH=/devices/virtual/block/dm-0
E: DEVTYPE=disk
E: DM_UDEV_DISABLE_DISK_RULES_FLAG=1
E: DM_UDEV_DISABLE_OTHER_RULES_FLAG=1
E: DM_UDEV_DISABLE_SUBSYSTEM_RULES_FLAG=1
E: MAJOR=253
E: MINOR=0
E: SUBSYSTEM=block
E: SYSTEMD_READY=0

Which explains why systemd ignores this device.

Look in initrd udev rules for SYSTEMD_READY assignment and post all
rules that do it.

Florent Peterschmitt

2016-06-10 08:01:03 UTC

Permalink

Post by Andrei Borzenkov

Post by Florent Peterschmitt
http://data.peterschmitt.fr/systemd.log

Unfortunately this log while being verbose does not add any useful
information.

Which explains why systemd ignores this device.
Look in initrd udev rules for SYSTEMD_READY assignment and post all
rules that do it.

cat initrd | cpio -idv && grep -rn SYSTEMD_READY returns nothing.

Perhaps genkernel (the tool used to make an initrd) is really not compatible with systemd.

This remembers me that "genkernel-next" exists and has an "UDEV=" variable, but didn't used it because the btrfs binary wasn't embedded.

I'll check it again, also i could try with dracut.

Thanks for your time, i'll give news about this.

Florent Peterschmitt

Andrei Borzenkov

2016-06-10 08:36:50 UTC

Permalink

Post by Florent Peterschmitt

Post by Andrei Borzenkov

Post by Florent Peterschmitt
http://data.peterschmitt.fr/systemd.log

Unfortunately this log while being verbose does not add any useful
information.

Which explains why systemd ignores this device.
Look in initrd udev rules for SYSTEMD_READY assignment and post all
rules that do it.

cat initrd | cpio -idv && grep -rn SYSTEMD_READY returns nothing.

SYSTEMD_READY must be set explicitly. There must be rule that does it.

Most likely your distro is using early cpio image, which means initrd is
concatenation of two archives - one plain cpio with processor firmware,
another compressed cpio with actual initramfs. That also explains that
you did not need to uncompress it.

Dracut has helper script to skip early cpio. May be your toolset offers
something similar.

Post by Florent Peterschmitt
Perhaps genkernel (the tool used to make an initrd) is really not compatible with systemd.
This remembers me that "genkernel-next" exists and has an "UDEV=" variable, but didn't used it because the btrfs binary wasn't embedded.
I'll check it again, also i could try with dracut.
Thanks for your time, i'll give news about this.
Florent Peterschmitt

Lennart Poettering

2016-06-10 13:31:45 UTC

Permalink

So, the question is why this is set for your device. It could be set
for two reasons: because the DM rules decide so, or because the btrfs
rules decide so.

I am not a DM pro, so I am not going to comment on that, but if
it's the btrfs udev rules, then it would be this file:

/usr/lib/udev/rules.d/64-btrfs.rules

Which sets SYSTEMD_READY=0 in case the btrfs ioctls claim that the
device isn't ready yet to be mounted.

That said, I'd still bet the DM setup of yours is hosed, and the btrfs
stuff works fine. If the DM stuff would work correctly you'd have
properties like DM_NAME or DM_UUID on the device, but you do not.

Lennart

--
Lennart Poettering, Red Hat

Florent Peterschmitt

2016-06-10 13:48:38 UTC

Permalink

Post by Lennart Poettering

So, the question is why this is set for your device. It could be set
for two reasons: because the DM rules decide so, or because the btrfs
rules decide so.
I am not a DM pro, so I am not going to comment on that, but if
/usr/lib/udev/rules.d/64-btrfs.rules
Which sets SYSTEMD_READY=0 in case the btrfs ioctls claim that the
device isn't ready yet to be mounted.
That said, I'd still bet the DM setup of yours is hosed, and the btrfs
stuff works fine. If the DM stuff would work correctly you'd have
properties like DM_NAME or DM_UUID on the device, but you do not.
Lennart
--
Lennart Poettering, Red Hat

I have no udev rules at all in this initramfs, and as I said in another mail, i'll try with genkernel-next that embeds udev (there is a specific option for that) and do stuff with SYSTEMD_READY.

Florent Peterschmitt

Lennart Poettering

2016-06-10 16:26:18 UTC

Permalink

Post by Florent Peterschmitt
I have no udev rules at all in this initramfs, and as I said in
another mail, i'll try with genkernel-next that embeds udev (there is
a specific option for that) and do stuff with SYSTEMD_READY.

I am not sure DM is capable of initializing first without udev and
then with.

The other code should be fine though, i.e. normally "udev trigger" is
all that's necessary to make devices appear. It's just that DM is a
bit weird...

Lennart

--
Lennart Poettering, Red Hat

Florent Peterschmitt

2016-06-10 18:23:28 UTC

Permalink

Post by Lennart Poettering

I am not sure DM is capable of initializing first without udev and
then with.
The other code should be fine though, i.e. normally "udev trigger" is
all that's necessary to make devices appear. It's just that DM is a
bit weird...
Lennart
--
Lennart Poettering, Red Hat

Well, using dracut with crypt, systemd and btrfs modules (among others) works just fine.

The genkernel-next scripts weren't setting SYSTEMD_READY=1 anywhere, only SYSTEMD_READY=0… and for crypt devices apparently.

dracut has a udev rule that set ENV{SYSTEMD_READY}=1

I guess I should build my own initrd to learn precisely how it works.

Florent Peterschmitt