Discussion:
Bug#958910: debci: 'debci setup -a armhf' fails to set up an lxc container on an amd64 host
(too old to reply)
Sven Geuer
2020-04-26 16:20:02 UTC
Permalink
Package: debci
Version: 2.11
Severity: normal

Dear Maintainer,

I intended to set up an armhf lxc container by running

debci setup -a armhf

as root on my amd64 system. The installation process terminated with

[...]
Timed out waiting for container to boot
lxc-stop: autopkgtest-unstable-armhf.new: tools/lxc_stop.c: main: 191
autopkgtest-unstable-armhf.new is not running
lxc-destroy: autopkgtest-unstable-armhf.new: tools/lxc_destroy.c: main: 271
Destroyed container autopkgtest-unstable-armhf.new


In /var/log/syslog I encountered these lines probably of relevance

Apr 26 17:16:25 e580sg kernel: [14086.041927] audit: type=1400
audit(1587914185.032:51): apparmor="STATUS" operation="profile_load"
profile="/usr/bin/lxc-start" name="lxc-autopkgtest-unstable-armhf.new_</var/lib
/lxc>" pid=88028 comm="apparmor_parser"
Apr 26 17:16:25 e580sg NetworkManager[686]: <info> [1587914185.0606] manager:
(vethSB9884): new Veth device (/org/freedesktop/NetworkManager/Devices/22)
Apr 26 17:16:25 e580sg kernel: [14086.064571] br0: port 2(vethS5LJSK) entered
blocking state
Apr 26 17:16:25 e580sg kernel: [14086.064573] br0: port 2(vethS5LJSK) entered
disabled state
Apr 26 17:16:25 e580sg kernel: [14086.064617] device vethS5LJSK entered
promiscuous mode
Apr 26 17:16:25 e580sg kernel: [14086.064692] br0: port 2(vethS5LJSK) entered
blocking state
Apr 26 17:16:25 e580sg kernel: [14086.064693] br0: port 2(vethS5LJSK) entered
forwarding state
Apr 26 17:16:25 e580sg kernel: [14086.066265] br0: port 2(vethS5LJSK) entered
disabled state
Apr 26 17:16:25 e580sg systemd-udevd[88029]: ethtool: autonegotiation is unset
or enabled, the speed and duplex are not writable.
Apr 26 17:16:25 e580sg NetworkManager[686]: <info> [1587914185.0618] manager:
(vethS5LJSK): new Veth device (/org/freedesktop/NetworkManager/Devices/23)
Apr 26 17:16:25 e580sg systemd-udevd[88029]: Using default interface naming
scheme 'v245'.
Apr 26 17:16:25 e580sg systemd-udevd[88029]: Could not set Alias=, MACAddress=
or MTU= on vethSB9884: No such device
Apr 26 17:16:25 e580sg systemd-udevd[88029]: vethSB9884: Could not apply link
config, ignoring: No such device
Apr 26 17:16:25 e580sg systemd-udevd[88030]: ethtool: autonegotiation is unset
or enabled, the speed and duplex are not writable.
Apr 26 17:16:25 e580sg systemd-udevd[88030]: Using default interface naming
scheme 'v245'.
Apr 26 17:16:25 e580sg kernel: [14086.092692] eth0: renamed from vethSB9884
Apr 26 17:16:25 e580sg gnome-shell[2393]: Removing a network device that was
not added
Apr 26 17:16:25 e580sg NetworkManager[686]: <info> [1587914185.1205] device
(vethS5LJSK): carrier: link connected
Apr 26 17:16:25 e580sg NetworkManager[686]: <info> [1587914185.1209] device
(br0): carrier: link connected
Apr 26 17:16:25 e580sg kernel: [14086.124443] IPv6: ADDRCONF(NETDEV_CHANGE):
eth0: link becomes ready
Apr 26 17:16:25 e580sg kernel: [14086.124480] IPv6: ADDRCONF(NETDEV_CHANGE):
vethS5LJSK: link becomes ready
Apr 26 17:16:25 e580sg kernel: [14086.124553] br0: port 2(vethS5LJSK) entered
blocking state
Apr 26 17:16:25 e580sg kernel: [14086.124555] br0: port 2(vethS5LJSK) entered
forwarding state
Apr 26 17:16:25 e580sg kernel: [14086.152175] audit: type=1400
audit(1587914185.144:52): apparmor="DENIED" operation="mount" info="failed
flags match" error=-13 profile="/usr/bin/lxc-start" name="/proc/sys/kerne
l/random/boot_id" pid=88031 comm="lxc-start" srcname="/dev/.lxc-boot-id"
flags="rw, bind"
Apr 26 17:16:25 e580sg kernel: [14086.154374] Not activating Mandatory Access
Control as /sbin/tomoyo-init does not exist.
Apr 26 17:16:25 e580sg kernel: [14086.248964] br0: port 2(vethS5LJSK) entered
disabled state
Apr 26 17:16:25 e580sg kernel: [14086.250781] device vethS5LJSK left
promiscuous mode
Apr 26 17:16:25 e580sg kernel: [14086.250785] br0: port 2(vethS5LJSK) entered
disabled state
Apr 26 17:16:25 e580sg NetworkManager[686]: <info> [1587914185.2731] device
(vethS5LJSK): released from master device br0
Apr 26 17:16:25 e580sg gnome-shell[2393]: Removing a network device that was
not added
Apr 26 17:16:25 e580sg kernel: [14086.394930] audit: type=1400
audit(1587914185.384:53): apparmor="STATUS" operation="profile_remove"
profile="/usr/bin/lxc-start" name="lxc-autopkgtest-unstable-
armhf.new_</var/lib/lxc>" pid=88076 comm="apparmor_parser"

These are the installed qemu packages

$ LANG=C dpkg -l 'qemu-*' | grep -v '^un'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-========================-==============-============-======================================================================
ii qemu-efi-aarch64 0.0~20200229-2 all UEFI firmware for
64-bit ARM virtual machines
ii qemu-efi-arm 0.0~20200229-2 all UEFI firmware for
32-bit ARM virtual machines
ii qemu-kvm 1:4.2-6 amd64 QEMU Full
virtualization on x86 hardware
ii qemu-system 1:4.2-6 amd64 QEMU full system
emulation binaries
ii qemu-system-arm 1:4.2-6 amd64 QEMU full system
emulation binaries (arm)
ii qemu-system-common 1:4.2-6 amd64 QEMU full system
emulation binaries (common files)
ii qemu-system-data 1:4.2-6 all QEMU full system
emulation (data files)
ii qemu-system-gui 1:4.2-6 amd64 QEMU full system
emulation binaries (user interface and audio support)
ii qemu-system-mips 1:4.2-6 amd64 QEMU full system
emulation binaries (mips)
ii qemu-system-misc 1:4.2-6 amd64 QEMU full system
emulation binaries (miscellaneous)
ii qemu-system-ppc 1:4.2-6 amd64 QEMU full system
emulation binaries (ppc)
ii qemu-system-sparc 1:4.2-6 amd64 QEMU full system
emulation binaries (sparc)
ii qemu-system-x86 1:4.2-6 amd64 QEMU full system
emulation binaries (x86)
ii qemu-user-static 1:4.2-6 amd64 QEMU user mode
emulation binaries (static version)
ii qemu-utils 1:4.2-6 amd64 QEMU utilities

Building armhf packages via pbuilder works flawlessly, so I assume my qemu
setup being functional.

When aborting the installation right before the timeout occurs I can chroot
into the container's rootfs. But lxc-start fails.

Please look into this.

Regards,
Sven



-- System Information:
Debian Release: bullseye/sid
APT prefers testing
APT policy: (500, 'testing')
Architecture: amd64 (x86_64)

Kernel: Linux 5.5.0-2-amd64 (SMP w/8 CPU cores)
Kernel taint flags: TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE
Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8), LANGUAGE=de_DE.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages debci depends on:
ii adduser 3.118
ii amqp-tools 0.10.0-1
ii bsdmainutils 11.1.2+b1
ii curl 7.68.0-1
ii dctrl-tools 2.24-3+b1
ii debian-archive-keyring 2019.1
ii debootstrap 1.0.123
ii devscripts 2.20.2
ii distro-info 0.23
ii fonts-font-awesome 5.0.10+really4.7.0~dfsg-1
ii jq 1.6-1
ii libjs-bootstrap 3.4.1+dfsg-1
ii libjs-jquery 3.3.1~dfsg-3
ii libjs-jquery-flot 0.8.3+dfsg-1
ii moreutils 0.63-1+b1
ii netcat-traditional 1.10-41.1+b1
ii parallel 20161222-1.1
ii patchutils 0.3.4-2+b1
ii rsync 3.1.3-8
ii ruby 1:2.7+1
ii ruby-activerecord 2:5.2.4.1+dfsg-2
ii ruby-bunny 2.14.4-3
ii ruby-kaminari-activerecord 1.0.1-5
ii ruby-pg 1.1.3-3+b3
ii ruby-sinatra 2.0.8.1-2
ii ruby-sinatra-contrib 2.0.8.1-2
ii ruby-sqlite3 1.4.2-2+b2
ii ruby-thor 0.20.3-2
ii sudo 1.8.31p1-1

Versions of packages debci recommends:
ii ntp [time-daemon] 1:4.2.8p14+dfsg-2

Versions of packages debci suggests:
ii apt-cacher-ng 3.4-1

-- Configuration Files:
/etc/sudoers.d/debci [Errno 13] Keine Berechtigung: '/etc/sudoers.d/debci'

-- no debconf information
Sven Geuer
2020-09-20 18:50:01 UTC
Permalink
Dear Maintainer,

I am missing someone has taken a look into this issue. I'd appreciate
to get some feedback.

Please let me know what further input from my side may be helpful.

Regards,
Sven
Antonio Terceiro
2020-09-21 15:10:01 UTC
Permalink
Control: tag -1 + moreinfo
Post by Sven Geuer
Dear Maintainer,
I am missing someone has taken a look into this issue. I'd appreciate
to get some feedback.
Please let me know what further input from my side may be helpful.
I just tried it here, and it just worked for me. Are you able to start
regular lxc containers?
Sven Geuer
2020-09-25 20:20:01 UTC
Permalink
Hello Antonio,

I have no issue with containers of the native amd64 architecture of my
system. Setting up and using a i386 container also works flawlessly. I
have been using them even before trying to setup a container of a
foreign architecture.

I seem to have a general issue to setup containers of a foreign
architecture, armhf is a mere example. I repeated a armhf setup. It
keeps failing, while the error looks different now compared to my first
post. It presents itself as a network access error:

[...]
Running setup script /usr/share/autopkgtest/setup-commands/setup-
testbed...
/usr/bin/sh: Attempting to set up Debian/Ubuntu apt sources
automatically
/usr/bin/sh: Distribution assumed to resemble Debian
Err:1 http://deb.debian.org/debian unstable InRelease
Temporary failure resolving 'deb.debian.org'
Reading package lists...
W: Failed to fetch
http://deb.debian.org/debian/dists/unstable/InRelease Temporary
failure resolving 'deb.debian.org'
[...]

Please see attach a complete log of the setup run. Hope this helps to
track down what going on.

Thanks,
Sven
Post by Antonio Terceiro
Control: tag -1 + moreinfo
Post by Sven Geuer
Dear Maintainer,
I am missing someone has taken a look into this issue. I'd
appreciate
to get some feedback.
Please let me know what further input from my side may be helpful.
I just tried it here, and it just worked for me. Are you able to start
regular lxc containers?
Antonio Terceiro
2020-09-26 13:10:02 UTC
Permalink
Post by Sven Geuer
Hello Antonio,
I have no issue with containers of the native amd64 architecture of my
system. Setting up and using a i386 container also works flawlessly. I
have been using them even before trying to setup a container of a
foreign architecture.
I seem to have a general issue to setup containers of a foreign
architecture, armhf is a mere example. I repeated a armhf setup. It
keeps failing, while the error looks different now compared to my first
[...]
Running setup script /usr/share/autopkgtest/setup-commands/setup-
testbed...
/usr/bin/sh: Attempting to set up Debian/Ubuntu apt sources
automatically
/usr/bin/sh: Distribution assumed to resemble Debian
Err:1 http://deb.debian.org/debian unstable InRelease
Temporary failure resolving 'deb.debian.org'
Reading package lists...
W: Failed to fetch
http://deb.debian.org/debian/dists/unstable/InRelease Temporary
failure resolving 'deb.debian.org'
[...]
Please see attach a complete log of the setup run. Hope this helps to
track down what going on.
I already knew you couldn't setup a debci container, that's why I asked
if you are able to create and start a plain lxc container in a foreign
architecture, so we can track down whether the issue is in debci, lxc,
or something else that is broken on your end (since it works for me).

e.g. does this work:

sudo lxc-create --template=debian --name=armhf -- --arch=armhf
sudo lxc-start --name=armhf
sudo lxc-attach --name=armhf

?
Sven Geuer
2020-09-26 13:50:02 UTC
Permalink
Hello Antonio,
Post by Antonio Terceiro
sudo lxc-create --template=debian --name=armhf -- --arch=armhf
sudo lxc-start --name=armhf
sudo lxc-attach --name=armhf
?
It seems to work in some way. But there are some errors at the end of
the installation process. Also, lxc-stop takes several seconds to
complete and it comes back with an error.

Here's the relevant output:

***@e580sg:~# lxc-create --template=debian --name=armhf -- --
arch=armhf
debootstrap ist /usr/sbin/debootstrap
Checking cache download in /var/cache/lxc/debian/rootfs-stable-armhf
...
Downloading debian minimal ...
I: Target architecture can be executed
I: Retrieving InRelease
I: Checking Release signature
I: Valid Release signature (key id
6D33866EDD8FFA41C0143AEDDCC9EFBF77E11517)
I: Retrieving Packages
I: Validating Packages
I: Resolving dependencies of required packages...
I: Resolving dependencies of base packages...
I: Checking component main on http://deb.debian.org/debian...
[...]
I: Base system installed successfully.
Download complete.
Copying rootfs to /var/lib/lxc/armhf/rootfs...Generating locales (this
might take a while)...
de_DE.UTF-8... done
de_DE.UTF-8... done
Generation complete.
update-rc.d: error: cannot find a LSB script for checkroot.sh
update-rc.d: error: cannot find a LSB script for umountfs
Failed to disable unit, unit hwclock.sh.service does not exist.
update-rc.d: error: cannot find a LSB script for hwclockfirst.sh
Creating SSH2 RSA key; this may take some time ...
2048 SHA256:eyTvbzzRV0YzdRVk9sFLWp9dzK0nRuUPgN2k6QMpkMY ***@e580sg
(RSA)
Creating SSH2 ECDSA key; this may take some time ...
256 SHA256:CfbEORbaL5SptfEnhIQDdg3M/3tNkRPX1mkL5FwUyOE ***@e580sg
(ECDSA)
Creating SSH2 ED25519 key; this may take some time ...
256 SHA256:rGf9JbAsSf8Go+5CZGLQH5Vag4reEEey/PjEWfVl8Vg ***@e580sg
(ED25519)
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.

Current default time zone: 'Etc/UTC'
Local time is now: Sat Sep 26 13:31:23 UTC 2020.
Universal Time is now: Sat Sep 26 13:31:23 UTC 2020.

***@e580sg:~# lxc-start --name=armhf
***@e580sg:~# lxc-attach --name=armhf
***@armhf:~# uname -a
Linux armhf 5.8.0-2-amd64 #1 SMP Debian 5.8.10-1 (2020-09-19) armv7l
GNU/Linux
***@armhf:~# exit
exit
***@e580sg:~# lxc-stop --name=armhf
lxc-stop: armhf: commands_utils.c: lxc_cmd_sock_rcv_state: 72 Resource
temporarily unavailable - Failed to receive message

Hope this helps.

Sven
Sven Geuer
2020-09-26 14:20:01 UTC
Permalink
Further tests show that networking does not seem to work:

***@e580sg:~# lxc-start -n armhf
***@e580sg:~# lxc-attach -n armhf
***@armhf:~# LANG=C apt update
Err:1 http://security.debian.org stable/updates InRelease
Temporary failure resolving 'security.debian.org'
Err:2 http://deb.debian.org/debian stable InRelease
Temporary failure resolving 'deb.debian.org'
Reading package lists... Done
Building dependency tree... Done
All packages are up to date.
W: Failed to fetch http://deb.debian.org/debian/dists/stable/InRelease
Temporary failure resolving 'deb.debian.org'
W: Failed to fetch
http://security.debian.org/dists/stable/updates/InRelease Temporary
failure resolving 'security.debian.org'
W: Some index files failed to download. They have been ignored, or old
ones used instead.
***@armhf:~# exit
exit
***@e580sg:~# lxc-stop -n armhf
lxc-stop: armhf: commands_utils.c: lxc_cmd_sock_rcv_state: 72 Resource
temporarily unavailable - Failed to receive message
Antonio Terceiro
2020-09-26 16:00:02 UTC
Permalink
Post by Sven Geuer
Err:1 http://security.debian.org stable/updates InRelease
Temporary failure resolving 'security.debian.org'
Err:2 http://deb.debian.org/debian stable InRelease
Temporary failure resolving 'deb.debian.org'
Reading package lists... Done
Building dependency tree... Done
All packages are up to date.
W: Failed to fetch http://deb.debian.org/debian/dists/stable/InRelease
Temporary failure resolving 'deb.debian.org'
W: Failed to fetch
http://security.debian.org/dists/stable/updates/InRelease Temporary
failure resolving 'security.debian.org'
W: Some index files failed to download. They have been ignored, or old
ones used instead.
exit
lxc-stop: armhf: commands_utils.c: lxc_cmd_sock_rcv_state: 72 Resource
temporarily unavailable - Failed to receive message
Does networking work in n ative plain lxc container? What does your
/etc/lxc/default.conf look like?
Sven Geuer
2020-09-26 17:30:02 UTC
Permalink
Post by Antonio Terceiro
Does networking work in n ative plain lxc container?
Networking works with plain native containers as is does with debci
generated native ones.

***@e580sg:~# lxc-start -n amd64
***@e580sg:~# lxc-attach -n amd64
***@amd64:~# LANG=C apt update
Get:1 http://security.debian.org stable/updates InRelease [65.4 kB]
Hit:2 http://deb.debian.org/debian stable InRelease
Get:3 http://deb.debian.org/debian stable/main Translation-en [5968 kB]
Get:4 http://security.debian.org stable/updates/main amd64 Packages
[233 kB]
Get:5 http://security.debian.org stable/updates/main Translation-en
[125 kB]
Fetched 6392 kB in 5s (1329
kB/s)
Reading package lists... Done
Building dependency tree... Done
All packages are up to date.
***@amd64:~# exit
exit
***@e580sg:~# lxc-stop -n amd64

Note: lxc-stop terminates quickly and without any error message.
Post by Antonio Terceiro
What does your/etc/lxc/default.conf look like?
lxc.net.0.type = veth
lxc.net.0.link = br0
lxc.net.0.flags = up
lxc.apparmor.profile = generated
lxc.apparmor.allow_nesting = 1

Sven
Sven Geuer
2020-09-27 11:10:01 UTC
Permalink
I tracked the issue down to the following point:

Networking works only for native containers with my usual local setup
made up of a bridge configured in /etc/network/interfaces, dnsmasq
listing on that bridge and arno-iptables-firewall doing the
masquerading and forwarding.

Networking works for native and armhf containers with using lxc-net and
my firewall disabled.

Networking fails for native and armhf containers with using lxc-net and
my firewall enabled.

So my questions are:

What's the difference between using lcx-net alone and my home-grown
setup (which works perfectly well also for qemu and virtualbox VMs)
with regards to non-native containers?

Does lxc-net anything specific regarding a containers architecture?

Should this be a bug against lxc then?

Off-topic: At present my email provider does not handle sender
validiation properly, so the mail server for debian.org refuses my
mails. Luckily they are accepted at bugs.debian.org.

Sven
Sven Geuer
2020-09-27 13:50:02 UTC
Permalink
I solved the issue for my environment. I observed the armhf container
did not receive dnsmasq's UDP replies. Examining /usr/libexec/lxc/lxc-
net I identified this line

iptables $use_iptables_lock -t mangle -A POSTROUTING -o ${LXC_BRIDGE}
-p udp -m udp --dport 68 -j CHECKSUM --checksum-fill

as something special and added its equivalent to my firewall
configuration. Since then networking works perfectly well also for the
armhf container.

I believe the special purpose of this iptables rule should be pointed
out not sure where.

Antonio, our questions helped me to look in the right direction. Thanks
for your patience.

Loading...