Discussion:
[Lxc-users] systemd inside LXC
John
2012-10-19 13:29:55 UTC
Permalink
Hello, I'm in the middle of a migration from init to systemd. I've
completed the transition of my host environment and my 6 existing
containers continue to work as expected (they all use sysvinit
internally). I've started work on a systemd container and am getting
some odd effects.

First off, if I use systemd-nspawn to start the container, it starts
fine. I can log in and halt it and all goes as expected. If, however I
use lxc-start, it clobbers my desktop, which is running in another
container.

So I have 2 problems: (a) the container does not boot and (b) it manages
to effect changes in another container.

I've been searching the 'net for most of this morning looking for
information on using systemd inside a container.

I'm using Arch Linux (3.6.2-1-ARCH) with LXC 0.8.0-rc2. Arch now uses
systemd by default.

To try to test this, I created a basic container and this exhibits the
same problems:

$ mkarchroot test base

Starting with systemd-nspawn works fine:
$ systemd-nspawn -D test/ /sbin/init

Starting with LXC does not:
$ lxc-create -n test -f test.conf
$ lxc-start -n test

The file test.conf contains these two lines:

lxc.utsname = test2
lxc.rootfs = /srv/lxc/test

When I start the container in LXC, all that happens is that my X session
dies (this is running in another container). The X session re-starts but
the keyboard does not work. I have to connect using another machine to
kill the test container and re-start my desktop container. I can't see
anything starting inside the test container.

I'd be grateful for any help and/or pointers in the right direction so I
can complete this transition to systemd.

Many thanks,
John
Serge Hallyn
2012-10-19 15:51:01 UTC
Permalink
Post by John
Hello, I'm in the middle of a migration from init to systemd. I've
completed the transition of my host environment and my 6 existing
containers continue to work as expected (they all use sysvinit
internally). I've started work on a systemd container and am getting
some odd effects.
First off, if I use systemd-nspawn to start the container, it starts
fine. I can log in and halt it and all goes as expected. If, however I
use lxc-start, it clobbers my desktop, which is running in another
container.
So I have 2 problems: (a) the container does not boot and (b) it manages
to effect changes in another container.
I've been searching the 'net for most of this morning looking for
information on using systemd inside a container.
I'm using Arch Linux (3.6.2-1-ARCH) with LXC 0.8.0-rc2. Arch now uses
systemd by default.
To try to test this, I created a basic container and this exhibits the
$ mkarchroot test base
$ systemd-nspawn -D test/ /sbin/init
$ lxc-create -n test -f test.conf
$ lxc-start -n test
lxc.utsname = test2
lxc.rootfs = /srv/lxc/test
Add:

lxc.network.type = empty

If you don't have any lxc.network.type sections, then the container
shares network with the host, and so the container talks to the host's
systemd. (same with upstart)
Post by John
When I start the container in LXC, all that happens is that my X session
dies (this is running in another container). The X session re-starts but
the keyboard does not work. I have to connect using another machine to
kill the test container and re-start my desktop container. I can't see
anything starting inside the test container.
I'd be grateful for any help and/or pointers in the right direction so I
can complete this transition to systemd.
Many thanks,
John
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________
Lxc-users mailing list
Lxc-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lxc-users
Michael H. Warfield
2012-10-20 23:41:59 UTC
Permalink
Serge,

I'm going to top post here simply because this is going to go off in a
different direction and bringing in an old thread but it is related...

Back on February 14 you responded to a message about Fedora 16 in a
container, which is something I've been trying to do and I had run into
that posters problems. You responded with this:

Subject: Re: [Lxc-users] fedora 16 under lxc
now all my efforts have not succeedd to get getty on tty1 to start
unmasking udev did something different
it created all the /dev devices
and made getty start but it started on the hosts's tty not on the container's
could someone shed some light here?
lxc-start creates some ptys and bind mounts them onto the guest's
/dev/{console,tty{1,2,3,4}}. It sounds like fedora's init is mounting
over the /dev set up by lxc causing a new /dev/tty to be created as
chardev 4:{1-4}. Devices namespaces would help this. We're hoping to
discuss design for those at next UDS, but those will come after user
namespaces. In the mean time, you'll need to make sure that the guest
does not mount over /dev, and does not remount /dev/pts.
-serge
That got me thinking and started into looking deeper into systemd, which
Fedora 16 and above uses and why it may be related here. I've made
Fedora 16 work in the past by installing upstart and disabling systemd
but that really becomes impractical in Fedora 17 because they're
including so few of the compatibility scripts. Yes, you are right, the
Fedora's init (systemd) is mounting something on /dev like this:

devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=1784160k,nr_inodes=446040,mode=755)

This is very bad for the reasons you pointed out in Feb. Looking at the
source code for systemd, this is hard coded into the binary and is not
configurable.

systemd-37/src/mount-setup.c:
--
/* The first three entries we might need before SELinux is up. The
* other ones we can delay until SELinux is loaded. */
#define N_EARLY_MOUNT 3

static const MountPoint mount_table[] = {
{ "proc", "/proc", "proc", NULL, MS_NOSUID|MS_NOEXEC|MS_NODEV, true },
{ "sysfs", "/sys", "sysfs", NULL, MS_NOSUID|MS_NOEXEC|MS_NODEV, true },
{ "devtmpfs", "/dev", "devtmpfs", "mode=755", MS_NOSUID, true },
{ "tmpfs", "/dev/shm", "tmpfs", "mode=1777", MS_NOSUID|MS_NODEV, true },
{ "devpts", "/dev/pts", "devpts", "mode=620,gid=" STRINGIFY(TTY_GID), MS_NOSUID|MS_NOEXEC, false },
{ "tmpfs", "/run", "tmpfs", "mode=755", MS_NOSUID|MS_NODEV, true },
{ "tmpfs", "/sys/fs/cgroup", "tmpfs", "mode=755", MS_NOSUID|MS_NOEXEC|MS_NODEV, false },
{ "cgroup", "/sys/fs/cgroup/systemd", "cgroup", "none,name=systemd", MS_NOSUID|MS_NOEXEC|MS_NODEV, false },
};
--
Short of building a custom systemd, I don't know how to fix that problem
and I suspect this OP is going to run into this same thing (container
taking over host's console) and might explain some of what he's seeing.
Several of these look like they could cause problems (like /dev/pts in
there). I've really reached an impasse at getting systemd (at least
Fedora 16 and 17) to work in a container without screwing up the host.
Prohibiting mounts entirely in the container might work but I suspect
(having read some systemd error messages) systemd is going to have some
serious heartburn there.

Thoughts?

Regards,
Mike
Hello, I'm in the middle of a migration from init to systemd. I've
completed the transition of my host environment and my 6 existing
containers continue to work as expected (they all use sysvinit
internally). I've started work on a systemd container and am getting
some odd effects.
First off, if I use systemd-nspawn to start the container, it starts
fine. I can log in and halt it and all goes as expected. If, however I
use lxc-start, it clobbers my desktop, which is running in another
container.
So I have 2 problems: (a) the container does not boot and (b) it manages
to effect changes in another container.
I've been searching the 'net for most of this morning looking for
information on using systemd inside a container.
I'm using Arch Linux (3.6.2-1-ARCH) with LXC 0.8.0-rc2. Arch now uses
systemd by default.
To try to test this, I created a basic container and this exhibits the
$ mkarchroot test base
$ systemd-nspawn -D test/ /sbin/init
$ lxc-create -n test -f test.conf
$ lxc-start -n test
lxc.utsname = test2
lxc.rootfs = /srv/lxc/test
lxc.network.type = empty
If you don't have any lxc.network.type sections, then the container
shares network with the host, and so the container talks to the host's
systemd. (same with upstart)
When I start the container in LXC, all that happens is that my X session
dies (this is running in another container). The X session re-starts but
the keyboard does not work. I have to connect using another machine to
kill the test container and re-start my desktop container. I can't see
anything starting inside the test container.
I'd be grateful for any help and/or pointers in the right direction so I
can complete this transition to systemd.
Many thanks,
John
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________
Lxc-users mailing list
Lxc-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lxc-users
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________
Lxc-users mailing list
Lxc-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lxc-users
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121020/9506b7ae/attachment.pgp>
Serge Hallyn
2012-10-21 19:49:54 UTC
Permalink
Post by Michael H. Warfield
Serge,
I'm going to top post here simply because this is going to go off in a
different direction and bringing in an old thread but it is related...
Back on February 14 you responded to a message about Fedora 16 in a
container, which is something I've been trying to do and I had run into
Subject: Re: [Lxc-users] fedora 16 under lxc
now all my efforts have not succeedd to get getty on tty1 to start
unmasking udev did something different
it created all the /dev devices
and made getty start but it started on the hosts's tty not on the container's
could someone shed some light here?
lxc-start creates some ptys and bind mounts them onto the guest's
/dev/{console,tty{1,2,3,4}}. It sounds like fedora's init is mounting
over the /dev set up by lxc causing a new /dev/tty to be created as
chardev 4:{1-4}. Devices namespaces would help this. We're hoping to
discuss design for those at next UDS, but those will come after user
namespaces. In the mean time, you'll need to make sure that the guest
does not mount over /dev, and does not remount /dev/pts.
-serge
That got me thinking and started into looking deeper into systemd, which
Fedora 16 and above uses and why it may be related here. I've made
Fedora 16 work in the past by installing upstart and disabling systemd
but that really becomes impractical in Fedora 17 because they're
including so few of the compatibility scripts. Yes, you are right, the
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=1784160k,nr_inodes=446040,mode=755)
This is very bad for the reasons you pointed out in Feb. Looking at the
source code for systemd, this is hard coded into the binary and is not
configurable.
--
/* The first three entries we might need before SELinux is up. The
* other ones we can delay until SELinux is loaded. */
#define N_EARLY_MOUNT 3
static const MountPoint mount_table[] = {
{ "proc", "/proc", "proc", NULL, MS_NOSUID|MS_NOEXEC|MS_NODEV, true },
{ "sysfs", "/sys", "sysfs", NULL, MS_NOSUID|MS_NOEXEC|MS_NODEV, true },
{ "devtmpfs", "/dev", "devtmpfs", "mode=755", MS_NOSUID, true },
{ "tmpfs", "/dev/shm", "tmpfs", "mode=1777", MS_NOSUID|MS_NODEV, true },
{ "devpts", "/dev/pts", "devpts", "mode=620,gid=" STRINGIFY(TTY_GID), MS_NOSUID|MS_NOEXEC, false },
{ "tmpfs", "/run", "tmpfs", "mode=755", MS_NOSUID|MS_NODEV, true },
{ "tmpfs", "/sys/fs/cgroup", "tmpfs", "mode=755", MS_NOSUID|MS_NOEXEC|MS_NODEV, false },
{ "cgroup", "/sys/fs/cgroup/systemd", "cgroup", "none,name=systemd", MS_NOSUID|MS_NOEXEC|MS_NODEV, false },
};
--
Short of building a custom systemd, I don't know how to fix that problem
and I suspect this OP is going to run into this same thing (container
taking over host's console) and might explain some of what he's seeing.
Several of these look like they could cause problems (like /dev/pts in
there). I've really reached an impasse at getting systemd (at least
Fedora 16 and 17) to work in a container without screwing up the host.
Prohibiting mounts entirely in the container might work but I suspect
(having read some systemd error messages) systemd is going to have some
serious heartburn there.
Thoughts?
IIRC, simply having apparmor(/selinux) refuse the mount of /dev by the
container should work, i.e. systemd was not going to fail as a result.

-serge
Michael H. Warfield
2012-10-21 21:27:50 UTC
Permalink
Post by Serge Hallyn
Post by Michael H. Warfield
Serge,
I'm going to top post here simply because this is going to go off in a
different direction and bringing in an old thread but it is related...
Back on February 14 you responded to a message about Fedora 16 in a
container, which is something I've been trying to do and I had run into
Subject: Re: [Lxc-users] fedora 16 under lxc
now all my efforts have not succeedd to get getty on tty1 to start
unmasking udev did something different
it created all the /dev devices
and made getty start but it started on the hosts's tty not on the container's
could someone shed some light here?
lxc-start creates some ptys and bind mounts them onto the guest's
/dev/{console,tty{1,2,3,4}}. It sounds like fedora's init is mounting
over the /dev set up by lxc causing a new /dev/tty to be created as
chardev 4:{1-4}. Devices namespaces would help this. We're hoping to
discuss design for those at next UDS, but those will come after user
namespaces. In the mean time, you'll need to make sure that the guest
does not mount over /dev, and does not remount /dev/pts.
-serge
That got me thinking and started into looking deeper into systemd, which
Fedora 16 and above uses and why it may be related here. I've made
Fedora 16 work in the past by installing upstart and disabling systemd
but that really becomes impractical in Fedora 17 because they're
including so few of the compatibility scripts. Yes, you are right, the
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=1784160k,nr_inodes=446040,mode=755)
This is very bad for the reasons you pointed out in Feb. Looking at the
source code for systemd, this is hard coded into the binary and is not
configurable.
--
/* The first three entries we might need before SELinux is up. The
* other ones we can delay until SELinux is loaded. */
#define N_EARLY_MOUNT 3
static const MountPoint mount_table[] = {
{ "proc", "/proc", "proc", NULL, MS_NOSUID|MS_NOEXEC|MS_NODEV, true },
{ "sysfs", "/sys", "sysfs", NULL, MS_NOSUID|MS_NOEXEC|MS_NODEV, true },
{ "devtmpfs", "/dev", "devtmpfs", "mode=755", MS_NOSUID, true },
{ "tmpfs", "/dev/shm", "tmpfs", "mode=1777", MS_NOSUID|MS_NODEV, true },
{ "devpts", "/dev/pts", "devpts", "mode=620,gid=" STRINGIFY(TTY_GID), MS_NOSUID|MS_NOEXEC, false },
{ "tmpfs", "/run", "tmpfs", "mode=755", MS_NOSUID|MS_NODEV, true },
{ "tmpfs", "/sys/fs/cgroup", "tmpfs", "mode=755", MS_NOSUID|MS_NOEXEC|MS_NODEV, false },
{ "cgroup", "/sys/fs/cgroup/systemd", "cgroup", "none,name=systemd", MS_NOSUID|MS_NOEXEC|MS_NODEV, false },
};
--
Short of building a custom systemd, I don't know how to fix that problem
and I suspect this OP is going to run into this same thing (container
taking over host's console) and might explain some of what he's seeing.
Several of these look like they could cause problems (like /dev/pts in
there). I've really reached an impasse at getting systemd (at least
Fedora 16 and 17) to work in a container without screwing up the host.
Prohibiting mounts entirely in the container might work but I suspect
(having read some systemd error messages) systemd is going to have some
serious heartburn there.
Thoughts?
IIRC, simply having apparmor(/selinux) refuse the mount of /dev by the
container should work, i.e. systemd was not going to fail as a result.
I'm not sure how that would work or if that would work in the case where
you didn't have selinux in the host kernel or you were crossing apparmor
in the container and selinux in the host or vice-versa.

In any case, I'm hitting the systemd-devel list looking to raise their
awareness of the problem and including this list and the fedora list.
If someone wants to mention it on the Arch Linux list, please do, I
don't participate over there.
Post by Serge Hallyn
-serge
Thanks
Regards,
Mike
Post by Serge Hallyn
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________
Lxc-users mailing list
Lxc-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lxc-users
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121021/9a1b9dd7/attachment.pgp>
Michael H. Warfield
2012-10-22 02:21:01 UTC
Permalink
Post by Serge Hallyn
Post by Michael H. Warfield
Serge,
...
Post by Serge Hallyn
Post by Michael H. Warfield
Short of building a custom systemd, I don't know how to fix that problem
and I suspect this OP is going to run into this same thing (container
taking over host's console) and might explain some of what he's seeing.
Several of these look like they could cause problems (like /dev/pts in
there). I've really reached an impasse at getting systemd (at least
Fedora 16 and 17) to work in a container without screwing up the host.
Prohibiting mounts entirely in the container might work but I suspect
(having read some systemd error messages) systemd is going to have some
serious heartburn there.
Thoughts?
IIRC, simply having apparmor(/selinux) refuse the mount of /dev by the
container should work, i.e. systemd was not going to fail as a result.
Hopefully, you've seen the message from Kay Sievers cc'ed to this list
from my post to the systemd-devel list. Looks like they have a
mechanism in place to do this...

http://www.freedesktop.org/wiki/Software/systemd/ContainerInterface

First step appears to be to set a container=LXC (or some other short
string) before invoking init in the container. Is there a mechanism to
do this?

Might look over the rest of their recommendation and see if there's
anything else we need to do. Looks like there might be some additional
mounts (some read-only) in there that need to be handled in lxc-start as
well.
Post by Serge Hallyn
-serge
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________
Lxc-users mailing list
Lxc-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lxc-users
Regards,
Mike
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121021/48f9fc56/attachment.pgp>
Michael H. Warfield
2012-10-22 03:29:46 UTC
Permalink
Serge,
Post by Michael H. Warfield
Post by Serge Hallyn
Post by Michael H. Warfield
Serge,
...
Post by Serge Hallyn
Post by Michael H. Warfield
Short of building a custom systemd, I don't know how to fix that problem
and I suspect this OP is going to run into this same thing (container
taking over host's console) and might explain some of what he's seeing.
Several of these look like they could cause problems (like /dev/pts in
there). I've really reached an impasse at getting systemd (at least
Fedora 16 and 17) to work in a container without screwing up the host.
Prohibiting mounts entirely in the container might work but I suspect
(having read some systemd error messages) systemd is going to have some
serious heartburn there.
Thoughts?
IIRC, simply having apparmor(/selinux) refuse the mount of /dev by the
container should work, i.e. systemd was not going to fail as a result.
Hopefully, you've seen the message from Kay Sievers cc'ed to this list
from my post to the systemd-devel list. Looks like they have a
mechanism in place to do this...
http://www.freedesktop.org/wiki/Software/systemd/ContainerInterface
First step appears to be to set a container=LXC (or some other short
string) before invoking init in the container. Is there a mechanism to
do this?
Might look over the rest of their recommendation and see if there's
anything else we need to do. Looks like there might be some additional
mounts (some read-only) in there that need to be handled in lxc-start as
well.
Tried simply exporting the container=LXC variable, the HOSTNAME and a
couple of others in there. Confirmed in upstart mode that the variables
did propagate but in systemd mode it just hung with 0 processes and left
an unremovable "busy" cgroup directory when I tried to "lxc-stop" it.
BUT... Something obviously behaved differently. It didn't try to grab
the console and commit other heinous on the system like it did before
with systemd. Maybe need to look closer at those mount requirements.
Post by Michael H. Warfield
Post by Serge Hallyn
-serge
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________
Lxc-users mailing list
Lxc-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lxc-users
Regards,
Mike
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________ Lxc-users mailing list Lxc-users at lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121021/c1d09b95/attachment.pgp>
Serge Hallyn
2012-10-22 13:09:49 UTC
Permalink
Post by Michael H. Warfield
Post by Serge Hallyn
Post by Michael H. Warfield
Serge,
...
Post by Serge Hallyn
Post by Michael H. Warfield
Short of building a custom systemd, I don't know how to fix that problem
and I suspect this OP is going to run into this same thing (container
taking over host's console) and might explain some of what he's seeing.
Several of these look like they could cause problems (like /dev/pts in
there). I've really reached an impasse at getting systemd (at least
Fedora 16 and 17) to work in a container without screwing up the host.
Prohibiting mounts entirely in the container might work but I suspect
(having read some systemd error messages) systemd is going to have some
serious heartburn there.
Thoughts?
IIRC, simply having apparmor(/selinux) refuse the mount of /dev by the
container should work, i.e. systemd was not going to fail as a result.
Hopefully, you've seen the message from Kay Sievers cc'ed to this list
from my post to the systemd-devel list. Looks like they have a
mechanism in place to do this...
http://www.freedesktop.org/wiki/Software/systemd/ContainerInterface
Saw the email, haven't yet read the page, thanks.
Post by Michael H. Warfield
First step appears to be to set a container=LXC (or some other short
string) before invoking init in the container. Is there a mechanism to
do this?
We've been setting 'container=lxc' since before system existed :) It's
hardcoded in lxc_start.c using putenv.c. I don't think we want to make
it runtime configurable, but a build-time (configure) flag would be fine.
Post by Michael H. Warfield
Might look over the rest of their recommendation and see if there's
anything else we need to do. Looks like there might be some additional
mounts (some read-only) in there that need to be handled in lxc-start as
well.
Thanks, Michael!

-serge
Serge Hallyn
2012-10-22 14:12:16 UTC
Permalink
Post by Serge Hallyn
Post by Michael H. Warfield
Post by Serge Hallyn
Post by Michael H. Warfield
Serge,
...
Post by Serge Hallyn
Post by Michael H. Warfield
Short of building a custom systemd, I don't know how to fix that problem
and I suspect this OP is going to run into this same thing (container
taking over host's console) and might explain some of what he's seeing.
Several of these look like they could cause problems (like /dev/pts in
there). I've really reached an impasse at getting systemd (at least
Fedora 16 and 17) to work in a container without screwing up the host.
Prohibiting mounts entirely in the container might work but I suspect
(having read some systemd error messages) systemd is going to have some
serious heartburn there.
Thoughts?
IIRC, simply having apparmor(/selinux) refuse the mount of /dev by the
container should work, i.e. systemd was not going to fail as a result.
Hopefully, you've seen the message from Kay Sievers cc'ed to this list
from my post to the systemd-devel list. Looks like they have a
mechanism in place to do this...
http://www.freedesktop.org/wiki/Software/systemd/ContainerInterface
Saw the email, haven't yet read the page, thanks.
So based on that page, what we do (set 'container=lxc') should already be
sufficient.

-serge
Michael H. Warfield
2012-10-22 14:37:46 UTC
Permalink
Post by Serge Hallyn
Post by Serge Hallyn
Post by Michael H. Warfield
Post by Serge Hallyn
Post by Michael H. Warfield
Serge,
...
Post by Serge Hallyn
Post by Michael H. Warfield
Short of building a custom systemd, I don't know how to fix that problem
and I suspect this OP is going to run into this same thing (container
taking over host's console) and might explain some of what he's seeing.
Several of these look like they could cause problems (like /dev/pts in
there). I've really reached an impasse at getting systemd (at least
Fedora 16 and 17) to work in a container without screwing up the host.
Prohibiting mounts entirely in the container might work but I suspect
(having read some systemd error messages) systemd is going to have some
serious heartburn there.
Thoughts?
IIRC, simply having apparmor(/selinux) refuse the mount of /dev by the
container should work, i.e. systemd was not going to fail as a result.
Hopefully, you've seen the message from Kay Sievers cc'ed to this list
from my post to the systemd-devel list. Looks like they have a
mechanism in place to do this...
http://www.freedesktop.org/wiki/Software/systemd/ContainerInterface
Saw the email, haven't yet read the page, thanks.
So based on that page, what we do (set 'container=lxc') should already be
sufficient.
For that step yes. I'm hearing that they also need tmpfs mounted
on /dev, for some reason, and then bind mounting appropriate ttys and
creating devices. It's mentioned on that page and mentioned in another
reply. I'm going down the list of mounts that are detailed out now.
Several of those steps (UUID and HOSTNAME) seem optional.
Post by Serge Hallyn
-serge
Regards,
Mike
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121022/9226cfb3/attachment.pgp>
Michael H. Warfield
2012-10-22 20:05:20 UTC
Permalink
Serge,
Post by Serge Hallyn
Post by Serge Hallyn
Post by Michael H. Warfield
Post by Serge Hallyn
Post by Michael H. Warfield
Serge,
...
Post by Serge Hallyn
Post by Michael H. Warfield
Short of building a custom systemd, I don't know how to fix that problem
and I suspect this OP is going to run into this same thing (container
taking over host's console) and might explain some of what he's seeing.
Several of these look like they could cause problems (like /dev/pts in
there). I've really reached an impasse at getting systemd (at least
Fedora 16 and 17) to work in a container without screwing up the host.
Prohibiting mounts entirely in the container might work but I suspect
(having read some systemd error messages) systemd is going to have some
serious heartburn there.
Thoughts?
IIRC, simply having apparmor(/selinux) refuse the mount of /dev by the
container should work, i.e. systemd was not going to fail as a result.
Hopefully, you've seen the message from Kay Sievers cc'ed to this list
from my post to the systemd-devel list. Looks like they have a
mechanism in place to do this...
http://www.freedesktop.org/wiki/Software/systemd/ContainerInterface
Saw the email, haven't yet read the page, thanks.
So based on that page, what we do (set 'container=lxc') should already be
sufficient.
Thanks to the dude asking a libvirt-lxc question on the list, I was let
to a page that let to a page that led to some discussion you were having
back in March with Ramez Hanna on this very subject, "Re: [Lxc-users]
f16 update"...

http://www.mail-archive.com/lxc-users at lists.sourceforge.net/msg03263.html

This would look to be the kludge to make a workaround for this problem,
I'm just not sure how to make it happen. Given you already found the
answer that the device for /dev has to be different than the device for
the parent, what should we do.

I tried this in the config...

lxc.mount.entry=tmpfs /var/lib/lxc/private/Alcove/dev tmpfs defaults 0 0

Maybe I got that entry wrong but it didn't do anything (and would have
resulted in other failures later as I found out). A similar mount entry
for a shared filesystem worked just fine.

Trying an empty /dev fails because it's missing EVERYTHING so starting
out with a tmpfs without populating it is not the answer either.

The correct answer would be to mount a tmpfs file system and then
populate it before firing up systemd in the container. I don't see how
to do that. A bind mount isn't going to work either, for the reasons
you stated in that thread, it ends up on the same device. Having
another mount on another real device would be a workaround but a really
bad kludge that would complicate things immensely.
Post by Serge Hallyn
-serge
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121022/2550f66a/attachment.pgp>
Serge Hallyn
2012-10-22 20:14:52 UTC
Permalink
Post by Michael H. Warfield
Serge,
Post by Serge Hallyn
Post by Serge Hallyn
Post by Michael H. Warfield
Post by Serge Hallyn
Post by Michael H. Warfield
Serge,
...
Post by Serge Hallyn
Post by Michael H. Warfield
Short of building a custom systemd, I don't know how to fix that problem
and I suspect this OP is going to run into this same thing (container
taking over host's console) and might explain some of what he's seeing.
Several of these look like they could cause problems (like /dev/pts in
there). I've really reached an impasse at getting systemd (at least
Fedora 16 and 17) to work in a container without screwing up the host.
Prohibiting mounts entirely in the container might work but I suspect
(having read some systemd error messages) systemd is going to have some
serious heartburn there.
Thoughts?
IIRC, simply having apparmor(/selinux) refuse the mount of /dev by the
container should work, i.e. systemd was not going to fail as a result.
Hopefully, you've seen the message from Kay Sievers cc'ed to this list
from my post to the systemd-devel list. Looks like they have a
mechanism in place to do this...
http://www.freedesktop.org/wiki/Software/systemd/ContainerInterface
Saw the email, haven't yet read the page, thanks.
So based on that page, what we do (set 'container=lxc') should already be
sufficient.
Thanks to the dude asking a libvirt-lxc question on the list, I was let
to a page that let to a page that led to some discussion you were having
back in March with Ramez Hanna on this very subject, "Re: [Lxc-users]
f16 update"...
http://www.mail-archive.com/lxc-users at lists.sourceforge.net/msg03263.html
thanks, I knew we'd been over some of this, but couldn't find my logs of
it.
Post by Michael H. Warfield
This would look to be the kludge to make a workaround for this problem,
I'm just not sure how to make it happen. Given you already found the
answer that the device for /dev has to be different than the device for
the parent, what should we do.
I tried this in the config...
lxc.mount.entry=tmpfs /var/lib/lxc/private/Alcove/dev tmpfs defaults 0 0
How about just a devtmpfs? We actually now do this by default (as of very
recently) in ubuntu by adding

devtmpfs dev devtmpfs defaults 0 0

to /var/lib/lxc/$container/fstab

Or, are you on an older kernel which doesn't have devtmpfs?
Post by Michael H. Warfield
Maybe I got that entry wrong but it didn't do anything (and would have
resulted in other failures later as I found out). A similar mount entry
for a shared filesystem worked just fine.
Trying an empty /dev fails because it's missing EVERYTHING so starting
out with a tmpfs without populating it is not the answer either.
The correct answer would be to mount a tmpfs file system and then
populate it before firing up systemd in the container. I don't see how
to do that. A bind mount isn't going to work either, for the reasons
you stated in that thread, it ends up on the same device. Having
another mount on another real device would be a workaround but a really
bad kludge that would complicate things immensely.
Post by Serge Hallyn
-serge
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
Michael H. Warfield
2012-10-22 20:48:45 UTC
Permalink
Post by Serge Hallyn
Post by Michael H. Warfield
Serge,
Post by Serge Hallyn
Post by Serge Hallyn
Post by Michael H. Warfield
Post by Serge Hallyn
Post by Michael H. Warfield
Serge,
...
Post by Serge Hallyn
Post by Michael H. Warfield
Short of building a custom systemd, I don't know how to fix that problem
and I suspect this OP is going to run into this same thing (container
taking over host's console) and might explain some of what he's seeing.
Several of these look like they could cause problems (like /dev/pts in
there). I've really reached an impasse at getting systemd (at least
Fedora 16 and 17) to work in a container without screwing up the host.
Prohibiting mounts entirely in the container might work but I suspect
(having read some systemd error messages) systemd is going to have some
serious heartburn there.
Thoughts?
IIRC, simply having apparmor(/selinux) refuse the mount of /dev by the
container should work, i.e. systemd was not going to fail as a result.
Hopefully, you've seen the message from Kay Sievers cc'ed to this list
from my post to the systemd-devel list. Looks like they have a
mechanism in place to do this...
http://www.freedesktop.org/wiki/Software/systemd/ContainerInterface
Saw the email, haven't yet read the page, thanks.
So based on that page, what we do (set 'container=lxc') should already be
sufficient.
Thanks to the dude asking a libvirt-lxc question on the list, I was let
to a page that let to a page that led to some discussion you were having
back in March with Ramez Hanna on this very subject, "Re: [Lxc-users]
f16 update"...
http://www.mail-archive.com/lxc-users at lists.sourceforge.net/msg03263.html
thanks, I knew we'd been over some of this, but couldn't find my logs of
it.
Post by Michael H. Warfield
This would look to be the kludge to make a workaround for this problem,
I'm just not sure how to make it happen. Given you already found the
answer that the device for /dev has to be different than the device for
the parent, what should we do.
I tried this in the config...
lxc.mount.entry=tmpfs /var/lib/lxc/private/Alcove/dev tmpfs defaults 0 0
How about just a devtmpfs? We actually now do this by default (as of very
recently) in ubuntu by adding
devtmpfs dev devtmpfs defaults 0 0
NO! That's the problem! That leads to the container connecting to the
hosts console and other devices and committing random acts of terrorism.
That's exactly the case we are trying to avoid and which is causing the
problems to begin with. That's what systemd is doing if it doesn't
find /dev mounted to begin with.
Post by Serge Hallyn
to /var/lib/lxc/$container/fstab
Or, are you on an older kernel which doesn't have devtmpfs?
Post by Michael H. Warfield
Maybe I got that entry wrong but it didn't do anything (and would have
resulted in other failures later as I found out). A similar mount entry
for a shared filesystem worked just fine.
Trying an empty /dev fails because it's missing EVERYTHING so starting
out with a tmpfs without populating it is not the answer either.
The correct answer would be to mount a tmpfs file system and then
populate it before firing up systemd in the container. I don't see how
to do that. A bind mount isn't going to work either, for the reasons
you stated in that thread, it ends up on the same device. Having
another mount on another real device would be a workaround but a really
bad kludge that would complicate things immensely.
Post by Serge Hallyn
-serge
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
Regards,
Mike
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121022/0a5ad4da/attachment.pgp>
Serge Hallyn
2012-10-22 21:21:50 UTC
Permalink
Post by Michael H. Warfield
Post by Serge Hallyn
Post by Michael H. Warfield
Serge,
Post by Serge Hallyn
Post by Serge Hallyn
Post by Michael H. Warfield
Post by Serge Hallyn
Post by Michael H. Warfield
Serge,
...
Post by Serge Hallyn
Post by Michael H. Warfield
Short of building a custom systemd, I don't know how to fix that problem
and I suspect this OP is going to run into this same thing (container
taking over host's console) and might explain some of what he's seeing.
Several of these look like they could cause problems (like /dev/pts in
there). I've really reached an impasse at getting systemd (at least
Fedora 16 and 17) to work in a container without screwing up the host.
Prohibiting mounts entirely in the container might work but I suspect
(having read some systemd error messages) systemd is going to have some
serious heartburn there.
Thoughts?
IIRC, simply having apparmor(/selinux) refuse the mount of /dev by the
container should work, i.e. systemd was not going to fail as a result.
Hopefully, you've seen the message from Kay Sievers cc'ed to this list
from my post to the systemd-devel list. Looks like they have a
mechanism in place to do this...
http://www.freedesktop.org/wiki/Software/systemd/ContainerInterface
Saw the email, haven't yet read the page, thanks.
So based on that page, what we do (set 'container=lxc') should already be
sufficient.
Thanks to the dude asking a libvirt-lxc question on the list, I was let
to a page that let to a page that led to some discussion you were having
back in March with Ramez Hanna on this very subject, "Re: [Lxc-users]
f16 update"...
http://www.mail-archive.com/lxc-users at lists.sourceforge.net/msg03263.html
thanks, I knew we'd been over some of this, but couldn't find my logs of
it.
Post by Michael H. Warfield
This would look to be the kludge to make a workaround for this problem,
I'm just not sure how to make it happen. Given you already found the
answer that the device for /dev has to be different than the device for
the parent, what should we do.
I tried this in the config...
lxc.mount.entry=tmpfs /var/lib/lxc/private/Alcove/dev tmpfs defaults 0 0
How about just a devtmpfs? We actually now do this by default (as of very
recently) in ubuntu by adding
devtmpfs dev devtmpfs defaults 0 0
NO! That's the problem! That leads to the container connecting to the
hosts console and other devices and committing random acts of terrorism.
No, it shouldn't, because lxc sets up the console after doing the mounts.

-serge
Michael H. Warfield
2012-10-22 21:36:59 UTC
Permalink
Post by Serge Hallyn
Post by Michael H. Warfield
Post by Serge Hallyn
Post by Michael H. Warfield
Serge,
Post by Serge Hallyn
Post by Serge Hallyn
Post by Michael H. Warfield
Post by Serge Hallyn
Post by Michael H. Warfield
Serge,
...
Post by Serge Hallyn
Post by Michael H. Warfield
Short of building a custom systemd, I don't know how to fix that problem
and I suspect this OP is going to run into this same thing (container
taking over host's console) and might explain some of what he's seeing.
Several of these look like they could cause problems (like /dev/pts in
there). I've really reached an impasse at getting systemd (at least
Fedora 16 and 17) to work in a container without screwing up the host.
Prohibiting mounts entirely in the container might work but I suspect
(having read some systemd error messages) systemd is going to have some
serious heartburn there.
Thoughts?
IIRC, simply having apparmor(/selinux) refuse the mount of /dev by the
container should work, i.e. systemd was not going to fail as a result.
Hopefully, you've seen the message from Kay Sievers cc'ed to this list
from my post to the systemd-devel list. Looks like they have a
mechanism in place to do this...
http://www.freedesktop.org/wiki/Software/systemd/ContainerInterface
Saw the email, haven't yet read the page, thanks.
So based on that page, what we do (set 'container=lxc') should already be
sufficient.
Thanks to the dude asking a libvirt-lxc question on the list, I was let
to a page that let to a page that led to some discussion you were having
back in March with Ramez Hanna on this very subject, "Re: [Lxc-users]
f16 update"...
http://www.mail-archive.com/lxc-users at lists.sourceforge.net/msg03263.html
thanks, I knew we'd been over some of this, but couldn't find my logs of
it.
Post by Michael H. Warfield
This would look to be the kludge to make a workaround for this problem,
I'm just not sure how to make it happen. Given you already found the
answer that the device for /dev has to be different than the device for
the parent, what should we do.
I tried this in the config...
lxc.mount.entry=tmpfs /var/lib/lxc/private/Alcove/dev tmpfs defaults 0 0
How about just a devtmpfs? We actually now do this by default (as of very
recently) in ubuntu by adding
devtmpfs dev devtmpfs defaults 0 0
NO! That's the problem! That leads to the container connecting to the
hosts console and other devices and committing random acts of terrorism.
No, it shouldn't, because lxc sets up the console after doing the mounts.
Maybe it shouldn't but that appears to be what is happening and even you
remarked that maybe the problem was something doing a remount of /dev
after entering the container...

I see your point though. If you did that mount after LXC set up the
console, then systemd wouldn't set it up and would drop into its more
restricted mode. That MIGHT help but you still have the entire dev
space of the host exposed to the guest which is what you were talking
about before wrt namespaces on devices. It might help. Would it be the
answer? Given that we've restricted access to those nodes in the
config, maybe yes. I'm just not so sure. Will give it a shot though.

Strange, though, my earlier effort at tmpfs on dev had no effect. Will
give it a shot.
Post by Serge Hallyn
-serge
Regards,
Mike
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121022/8394d54d/attachment.pgp>
Serge Hallyn
2012-10-22 21:58:13 UTC
Permalink
Post by Michael H. Warfield
Post by Serge Hallyn
Post by Michael H. Warfield
Post by Serge Hallyn
Post by Michael H. Warfield
Serge,
Post by Serge Hallyn
Post by Serge Hallyn
Post by Michael H. Warfield
Post by Serge Hallyn
Post by Michael H. Warfield
Serge,
...
Post by Serge Hallyn
Post by Michael H. Warfield
Short of building a custom systemd, I don't know how to fix that problem
and I suspect this OP is going to run into this same thing (container
taking over host's console) and might explain some of what he's seeing.
Several of these look like they could cause problems (like /dev/pts in
there). I've really reached an impasse at getting systemd (at least
Fedora 16 and 17) to work in a container without screwing up the host.
Prohibiting mounts entirely in the container might work but I suspect
(having read some systemd error messages) systemd is going to have some
serious heartburn there.
Thoughts?
IIRC, simply having apparmor(/selinux) refuse the mount of /dev by the
container should work, i.e. systemd was not going to fail as a result.
Hopefully, you've seen the message from Kay Sievers cc'ed to this list
from my post to the systemd-devel list. Looks like they have a
mechanism in place to do this...
http://www.freedesktop.org/wiki/Software/systemd/ContainerInterface
Saw the email, haven't yet read the page, thanks.
So based on that page, what we do (set 'container=lxc') should already be
sufficient.
Thanks to the dude asking a libvirt-lxc question on the list, I was let
to a page that let to a page that led to some discussion you were having
back in March with Ramez Hanna on this very subject, "Re: [Lxc-users]
f16 update"...
http://www.mail-archive.com/lxc-users at lists.sourceforge.net/msg03263.html
thanks, I knew we'd been over some of this, but couldn't find my logs of
it.
Post by Michael H. Warfield
This would look to be the kludge to make a workaround for this problem,
I'm just not sure how to make it happen. Given you already found the
answer that the device for /dev has to be different than the device for
the parent, what should we do.
I tried this in the config...
lxc.mount.entry=tmpfs /var/lib/lxc/private/Alcove/dev tmpfs defaults 0 0
How about just a devtmpfs? We actually now do this by default (as of very
recently) in ubuntu by adding
devtmpfs dev devtmpfs defaults 0 0
NO! That's the problem! That leads to the container connecting to the
hosts console and other devices and committing random acts of terrorism.
No, it shouldn't, because lxc sets up the console after doing the mounts.
Maybe it shouldn't but that appears to be what is happening and even you
remarked that maybe the problem was something doing a remount of /dev
after entering the container...
I see your point though. If you did that mount after LXC set up the
console, then systemd wouldn't set it up and would drop into its more
restricted mode. That MIGHT help but you still have the entire dev
space of the host exposed to the guest which is what you were talking
about before wrt namespaces on devices. It might help. Would it be the
answer? Given that we've restricted access to those nodes in the
config, maybe yes. I'm just not so sure. Will give it a shot though.
Strange, though, my earlier effort at tmpfs on dev had no effect. Will
give it a shot.
Thanks again for looking into this - I know just how frustrating it
can be! :)

-serge
Michael H. Warfield
2012-10-22 22:05:13 UTC
Permalink
<Trimming some overhead we've seen enough of...>
Post by Serge Hallyn
Post by Michael H. Warfield
Post by Serge Hallyn
How about just a devtmpfs? We actually now do this by default (as of very
recently) in ubuntu by adding
devtmpfs dev devtmpfs defaults 0 0
NO! That's the problem! That leads to the container connecting to the
hosts console and other devices and committing random acts of terrorism.
No, it shouldn't, because lxc sets up the console after doing the mounts.
Damn, dude! That worked! That kludge rang da bell. Of course, I also
discovered the boneheaded typo I had in attempting the tmpfs mount in
the process. :-P I now have a container running systemd up and running
with Fedora 17 in it.

I'm not sure I'm totally happy with it.

Because of doing the devtmpfs thing, the guest can immediately see
things like removable drives coming and going and might, presumably, be
able to mount them. Not thrilled with that from a security standpoint.
Would also mean the guests could access things like my permanent
forensic CDs that are in the CD drives. I guess that can be restricted
in the config but still makes me a bit uncomfortable that the guest has
complete visibility into the hosts dev system.

Another gotcha, albeit a much more minor one... When systemd drops into
this mode, you no longer have vty consoles available so lxc-console
won't work. That's actually on their page.

I remember seeing this:
--
If systemd detects it is run in a container it will spawn a single shell
on /dev/console, and not care about VTs or multiple gettys on VTs
--
Suboptimal but a small price to pay I suppose.
Post by Serge Hallyn
-serge
Regards,
Mike
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121022/35587526/attachment.pgp>
Michael H. Warfield
2012-10-22 22:08:52 UTC
Permalink
Post by Michael H. Warfield
<Trimming some overhead we've seen enough of...>
Post by Serge Hallyn
Post by Michael H. Warfield
Post by Serge Hallyn
How about just a devtmpfs? We actually now do this by default (as of very
recently) in ubuntu by adding
devtmpfs dev devtmpfs defaults 0 0
NO! That's the problem! That leads to the container connecting to the
hosts console and other devices and committing random acts of terrorism.
No, it shouldn't, because lxc sets up the console after doing the mounts.
Damn, dude! That worked! That kludge rang da bell. Of course, I also
discovered the boneheaded typo I had in attempting the tmpfs mount in
the process. :-P I now have a container running systemd up and running
with Fedora 17 in it.
Forgot to include the entry I added to the config file to make it all
workie...

lxc.mount.entry=devtmpfs /srv/lxc/rootfs/dev devtmpfs defaults 0 0

That /srv/lxc/rootfs is my common bind mount for all my containers.
Post by Michael H. Warfield
I'm not sure I'm totally happy with it.
Because of doing the devtmpfs thing, the guest can immediately see
things like removable drives coming and going and might, presumably, be
able to mount them. Not thrilled with that from a security standpoint.
Would also mean the guests could access things like my permanent
forensic CDs that are in the CD drives. I guess that can be restricted
in the config but still makes me a bit uncomfortable that the guest has
complete visibility into the hosts dev system.
Another gotcha, albeit a much more minor one... When systemd drops into
this mode, you no longer have vty consoles available so lxc-console
won't work. That's actually on their page.
--
If systemd detects it is run in a container it will spawn a single shell
on /dev/console, and not care about VTs or multiple gettys on VTs
--
Suboptimal but a small price to pay I suppose.
Post by Serge Hallyn
-serge
Regards,
Mike
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121022/e6e3b374/attachment.pgp>
Michael H. Warfield
2012-10-22 22:37:04 UTC
Permalink
Post by Michael H. Warfield
<Trimming some overhead we've seen enough of...>
Post by Serge Hallyn
Post by Michael H. Warfield
Post by Serge Hallyn
How about just a devtmpfs? We actually now do this by default (as of very
recently) in ubuntu by adding
devtmpfs dev devtmpfs defaults 0 0
NO! That's the problem! That leads to the container connecting to the
hosts console and other devices and committing random acts of terrorism.
No, it shouldn't, because lxc sets up the console after doing the mounts.
Damn, dude! That worked! That kludge rang da bell. Of course, I also
discovered the boneheaded typo I had in attempting the tmpfs mount in
the process. :-P I now have a container running systemd up and running
with Fedora 17 in it.
I'm not sure I'm totally happy with it.
Because of doing the devtmpfs thing, the guest can immediately see
things like removable drives coming and going and might, presumably, be
able to mount them. Not thrilled with that from a security standpoint.
Would also mean the guests could access things like my permanent
forensic CDs that are in the CD drives. I guess that can be restricted
in the config but still makes me a bit uncomfortable that the guest has
complete visibility into the hosts dev system.
Another downside. Container does not shut down cleanly...

init 0 inside the container...

In lxc-start -

Unmounting file systems.
Could not remount as read-only /: Device or resource busy
Not all file systems unmounted, 1 left.
Detaching loop devices.
Could not delete loopback /dev/loop7: Operation not permitted
Could not delete loopback /dev/loop6: Operation not permitted
Could not delete loopback /dev/loop5: Operation not permitted
Could not delete loopback /dev/loop4: Operation not permitted
Could not delete loopback /dev/loop3: Operation not permitted
Could not delete loopback /dev/loop2: Operation not permitted
Could not delete loopback /dev/loop1: Operation not permitted
Could not delete loopback /dev/loop0: Operation not permitted
Not all loop devices detached, 8 left.
Cannot finalize remaining file systems and devices, giving up.
Exiting container.
lxc-start: Device or resource busy - failed to remove cgroup '/sys/fs/cgroup/systemd/Alcove'

Not good. The tasks file is empty but... Can't get rid of it.
"Operation not permitted".

Sigh...

Getting closer though. Much closer.
Post by Michael H. Warfield
Another gotcha, albeit a much more minor one... When systemd drops into
this mode, you no longer have vty consoles available so lxc-console
won't work. That's actually on their page.
--
If systemd detects it is run in a container it will spawn a single shell
on /dev/console, and not care about VTs or multiple gettys on VTs
--
Suboptimal but a small price to pay I suppose.
Post by Serge Hallyn
-serge
Regards,
Mike
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121022/80f987a5/attachment.pgp>
Michael H. Warfield
2012-10-22 22:55:45 UTC
Permalink
Post by Michael H. Warfield
Post by Michael H. Warfield
<Trimming some overhead we've seen enough of...>
Post by Serge Hallyn
Post by Michael H. Warfield
Post by Serge Hallyn
How about just a devtmpfs? We actually now do this by default (as of very
recently) in ubuntu by adding
devtmpfs dev devtmpfs defaults 0 0
NO! That's the problem! That leads to the container connecting to the
hosts console and other devices and committing random acts of terrorism.
No, it shouldn't, because lxc sets up the console after doing the mounts.
Damn, dude! That worked! That kludge rang da bell. Of course, I also
discovered the boneheaded typo I had in attempting the tmpfs mount in
the process. :-P I now have a container running systemd up and running
with Fedora 17 in it.
I'm not sure I'm totally happy with it.
Because of doing the devtmpfs thing, the guest can immediately see
things like removable drives coming and going and might, presumably, be
able to mount them. Not thrilled with that from a security standpoint.
Would also mean the guests could access things like my permanent
forensic CDs that are in the CD drives. I guess that can be restricted
in the config but still makes me a bit uncomfortable that the guest has
complete visibility into the hosts dev system.
Another downside. Container does not shut down cleanly...
And another...

Container seems to hang if lxc-start is run in disconnected mode
(lxc-start -d -o {log}). Starts up fine with a console that's connected
to pty's but not to a log it seems...
Post by Michael H. Warfield
init 0 inside the container...
In lxc-start -
Unmounting file systems.
Could not remount as read-only /: Device or resource busy
Not all file systems unmounted, 1 left.
Detaching loop devices.
Could not delete loopback /dev/loop7: Operation not permitted
Could not delete loopback /dev/loop6: Operation not permitted
Could not delete loopback /dev/loop5: Operation not permitted
Could not delete loopback /dev/loop4: Operation not permitted
Could not delete loopback /dev/loop3: Operation not permitted
Could not delete loopback /dev/loop2: Operation not permitted
Could not delete loopback /dev/loop1: Operation not permitted
Could not delete loopback /dev/loop0: Operation not permitted
Not all loop devices detached, 8 left.
Cannot finalize remaining file systems and devices, giving up.
Exiting container.
lxc-start: Device or resource busy - failed to remove cgroup '/sys/fs/cgroup/systemd/Alcove'
Not good. The tasks file is empty but... Can't get rid of it.
"Operation not permitted".
Sigh...
Getting closer though. Much closer.
Post by Michael H. Warfield
Another gotcha, albeit a much more minor one... When systemd drops into
this mode, you no longer have vty consoles available so lxc-console
won't work. That's actually on their page.
--
If systemd detects it is run in a container it will spawn a single shell
on /dev/console, and not care about VTs or multiple gettys on VTs
--
Suboptimal but a small price to pay I suppose.
Post by Serge Hallyn
-serge
Regards,
Mike
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121022/7488b77d/attachment.pgp>
Stéphane Graber
2012-10-23 15:22:54 UTC
Permalink
Post by Michael H. Warfield
<Trimming some overhead we've seen enough of...>
Post by Serge Hallyn
Post by Michael H. Warfield
Post by Serge Hallyn
How about just a devtmpfs? We actually now do this by default (as of very
recently) in ubuntu by adding
devtmpfs dev devtmpfs defaults 0 0
NO! That's the problem! That leads to the container connecting to the
hosts console and other devices and committing random acts of terrorism.
No, it shouldn't, because lxc sets up the console after doing the mounts.
Damn, dude! That worked! That kludge rang da bell. Of course, I also
discovered the boneheaded typo I had in attempting the tmpfs mount in
the process. :-P I now have a container running systemd up and running
with Fedora 17 in it.
I'm not sure I'm totally happy with it.
Because of doing the devtmpfs thing, the guest can immediately see
things like removable drives coming and going and might, presumably, be
able to mount them. Not thrilled with that from a security standpoint.
Would also mean the guests could access things like my permanent
forensic CDs that are in the CD drives. I guess that can be restricted
in the config but still makes me a bit uncomfortable that the guest has
complete visibility into the hosts dev system.
That's actually similar to what Ubuntu has had for the past few releases
as we're running udevd in the container.

Basically all the block devices of the host and any hotplugged device
will appear in /dev but our default configuration is to only allow
"mknod"ing them, not read or write to them.

So the end result is basically the same as if they weren't there to
start with, except that for those that are actually allowed, they then
behave like they'd on the host by showing up when added and disappearing
when removed without any manual interaction.

It's not ideal, but it's safe. For the ideal implementation, we'll need
to wait for the device namespace.
Post by Michael H. Warfield
Another gotcha, albeit a much more minor one... When systemd drops into
this mode, you no longer have vty consoles available so lxc-console
won't work. That's actually on their page.
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________
Lxc-users mailing list
Lxc-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lxc-users
--
St?phane Graber
Ubuntu developer
http://www.ubuntu.com

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 897 bytes
Desc: OpenPGP digital signature
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121023/67ac829f/attachment.pgp>
John
2012-10-21 18:56:43 UTC
Permalink
Post by Serge Hallyn
lxc.network.type = empty
If you don't have any lxc.network.type sections, then the container
shares network with the host, and so the container talks to the host's
systemd. (same with upstart)
Thanks for the reply, I will try that tomorrow. I am sorry I wasn't
around to check for replies before now. One question though... I
actually want a separate network in the container (hence using veth) so
it has its own address distinct from the host. Are you saying that I
can't do this any more?

I've also read the later replies and they seem to be saying that this
simply does not work (systemd inside a container). Given its
proliferation into other distros (I'm on Arch and that's the reason I am
looking at this now), where does systemd come in the priorities of LXC?

I really hope we can get this working, as LXC has so far worked very
well for me.

Thanks,
John
Serge Hallyn
2012-10-21 20:00:10 UTC
Permalink
Post by John
Post by Serge Hallyn
lxc.network.type = empty
If you don't have any lxc.network.type sections, then the container
shares network with the host, and so the container talks to the host's
systemd. (same with upstart)
Thanks for the reply, I will try that tomorrow. I am sorry I wasn't
around to check for replies before now. One question though... I
actually want a separate network in the container (hence using veth) so
it has its own address distinct from the host. Are you saying that I
can't do this any more?
Not at all. But if you're saying you have a 'lxc.network.type = veth'
in your container config, then what I said doesn't apply anyway. It
sounds like the remount of /dev which Micheal mentioned is in fact your
real problem!
Post by John
I've also read the later replies and they seem to be saying that this
simply does not work (systemd inside a container). Given its
proliferation into other distros (I'm on Arch and that's the reason I am
looking at this now), where does systemd come in the priorities of LXC?
Where does LXC come in the priorities of systemd? :)

(my point being that it might be far easier to patch systemd to make
the filesystems to mount configurable, versus implementing a devices
namespace in the kernel so that lxc can work around it)

But, lxc is open source, as is the kernel (and systemd) - when you send
patches, your priorities influence its priorities.
Post by John
I really hope we can get this working, as LXC has so far worked very
well for me.
-serge
Loading...