[Lxc-users] Unable to run systemd in an LXC / cgroup container.

[Michael H. Warfield]

Hello,

This is being directed to the systemd-devel community but I'm cc'ing the
lxc-users community and the Fedora community on this for their input as
well. I know it's not always good to cross post between multiple lists
but this is of interest to all three communities who may have valuable
input.

I'm new to this particular list, just having joined after tracking a
problem down to some systemd internals...

Several people over the last year or two on the lxc-users list have been
discussions trying to run certain distros (notably Fedora 16 and above,
recent Arch Linux and possibly others) in LXC containers, virualizing
entire servers this way. This is very similar to Virtuoso / OpenVZ only
it's using the native Linux cgroups for the containers (primary reason I
dumped OpenVZ was to avoid their custom patched kernels). These recent
distros have switched to systemd for the main init process and this has
proven to be disastrous for those of us using LXC and trying to install
or update our containers.

To put it bluntly, it doesn't work and causes all sorts of problems on
the host.

To summarize the problem... The LXC startup binary sets up various
things for /dev and /dev/pts for the container to run properly and this
works perfectly fine for SystemV start-up scripts and/or Upstart.
Unfortunately, systemd has mounts of devtmpfs on /dev and devpts
on /dev/pts which then break things horribly. This is because the
kernel currently lacks namespaces for devices and won't for some time to
come (in design). When devtmpfs gets mounted over top of /dev in the
container, it then hijacks the hosts console tty and several other
devices which had been set up through bind mounts by LXC and should have
been LEFT ALONE.

Yes! I recognize that this problem with devtmpfs and lack of namespaces
is a potential security problem anyways that could (and does) cause
serious container-to-host problems. We're just not going to get that
fixed right away in the linux cgroups and namespaces.

How do we work around this problem in systemd where it has hard coded
mounts in the binary that we can't override or configure? Or is it
there and I'm just missing it trying to examine the sources? That's how
I found where the problem lay.

Regards,
Mike
--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121021/a5600a29/attachment.pgp>

[Michael H. Warfield]

Post by Michael H. Warfield
This is being directed to the systemd-devel community but I'm cc'ing the
lxc-users community and the Fedora community on this for their input as
well. I know it's not always good to cross post between multiple lists
but this is of interest to all three communities who may have valuable
input.
I'm new to this particular list, just having joined after tracking a
problem down to some systemd internals...
Several people over the last year or two on the lxc-users list have been
discussions trying to run certain distros (notably Fedora 16 and above,
recent Arch Linux and possibly others) in LXC containers, virualizing
entire servers this way. This is very similar to Virtuoso / OpenVZ only
it's using the native Linux cgroups for the containers (primary reason I
dumped OpenVZ was to avoid their custom patched kernels). These recent
distros have switched to systemd for the main init process and this has
proven to be disastrous for those of us using LXC and trying to install
or update our containers.
To put it bluntly, it doesn't work and causes all sorts of problems on
the host.
To summarize the problem... The LXC startup binary sets up various
things for /dev and /dev/pts for the container to run properly and this
works perfectly fine for SystemV start-up scripts and/or Upstart.
Unfortunately, systemd has mounts of devtmpfs on /dev and devpts
on /dev/pts which then break things horribly. This is because the
kernel currently lacks namespaces for devices and won't for some time to
come (in design). When devtmpfs gets mounted over top of /dev in the
container, it then hijacks the hosts console tty and several other
devices which had been set up through bind mounts by LXC and should have
been LEFT ALONE.
Yes! I recognize that this problem with devtmpfs and lack of namespaces
is a potential security problem anyways that could (and does) cause
serious container-to-host problems. We're just not going to get that
fixed right away in the linux cgroups and namespaces.
How do we work around this problem in systemd where it has hard coded
mounts in the binary that we can't override or configure? Or is it
there and I'm just missing it trying to examine the sources? That's how
I found where the problem lay.

http://www.freedesktop.org/wiki/Software/systemd/ContainerInterface

A very long ways, yeah. That looks like it could be just what we've
been looking for. Just gotta figure out how to set that environment
variable but that's up to a couple of others to comment on in the
lxc-users list. Then we'll see where we go from there.

Many thanks!

Kay

Regards,
Mike
--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121021/d31479f5/attachment.pgp>

[John]

Post by Michael H. Warfield

Post by Michael H. Warfield
This is being directed to the systemd-devel community but I'm cc'ing the
lxc-users community and the Fedora community on this for their input as
well. I know it's not always good to cross post between multiple lists
but this is of interest to all three communities who may have valuable
input.
I'm new to this particular list, just having joined after tracking a
problem down to some systemd internals...
Several people over the last year or two on the lxc-users list have been
discussions trying to run certain distros (notably Fedora 16 and above,
recent Arch Linux and possibly others) in LXC containers, virualizing
entire servers this way. This is very similar to Virtuoso / OpenVZ only
it's using the native Linux cgroups for the containers (primary reason I
dumped OpenVZ was to avoid their custom patched kernels). These recent
distros have switched to systemd for the main init process and this has
proven to be disastrous for those of us using LXC and trying to install
or update our containers.
To put it bluntly, it doesn't work and causes all sorts of problems on
the host.
To summarize the problem... The LXC startup binary sets up various
things for /dev and /dev/pts for the container to run properly and this
works perfectly fine for SystemV start-up scripts and/or Upstart.
Unfortunately, systemd has mounts of devtmpfs on /dev and devpts
on /dev/pts which then break things horribly. This is because the
kernel currently lacks namespaces for devices and won't for some time to
come (in design). When devtmpfs gets mounted over top of /dev in the
container, it then hijacks the hosts console tty and several other
devices which had been set up through bind mounts by LXC and should have
been LEFT ALONE.
Yes! I recognize that this problem with devtmpfs and lack of namespaces
is a potential security problem anyways that could (and does) cause
serious container-to-host problems. We're just not going to get that
fixed right away in the linux cgroups and namespaces.
How do we work around this problem in systemd where it has hard coded
mounts in the binary that we can't override or configure? Or is it
there and I'm just missing it trying to examine the sources? That's how
I found where the problem lay.

http://www.freedesktop.org/wiki/Software/systemd/ContainerInterface

A very long ways, yeah. That looks like it could be just what we've
been looking for. Just gotta figure out how to set that environment
variable but that's up to a couple of others to comment on in the
lxc-users list. Then we'll see where we go from there.
Many thanks!

Kay

Regards,
Mike

I've just performed a very quick check on my Arch Linux system here.

on host (running systemd):
# cat /proc/1/environ
TERM=linuxRD_TIMESTAMP=

In a container (running sysvinit):
# cat /proc/1/environ
STY=623.systemd-lithiumTERM=screenTERMCAP=SC|screen|VT 100/ANSI X3.64
virtual terminal:\
:DO=\E[%dB:LE=\E[%dD:RI=\E[%dC:UP=\E[%dA:bs:bt=\E[Z:\
:cd=\E[J:ce=\E[K:cl=\E[H\E[J:cm=\E[%i%d;%dH:ct=\E[3g:\
:do=^J:nd=\E[C:pt:rc=\E8:rs=\Ec:sc=\E7:st=\EH:up=\EM:\
:le=^H:bl=^G:cr=^M:it#8:ho=\E[H:nw=\EE:ta=^I:is=\E)0:\
:li#24:co#80:am:xn:xv:LP:sr=\EM:al=\E[L:AL=\E[%dL:\
:cs=\E[%i%d;%dr:dl=\E[M:DL=\E[%dM:dc=\E[P:DC=\E[%dP:\
:im=\E[4h:ei=\E[4l:mi:IC=\E[%d@:ks=\E[?1h\E=:\
:ke=\E[?1l\E>:vi=\E[?25l:ve=\E[34h\E[?25h:vs=\E[34l:\
:ti=\E[?1049h:te=\E[?1049l:k0=\E[10~:k1=\EOP:k2=\EOQ:\
:k3=\EOR:k4=\EOS:k5=\E[15~:k6=\E[17~:k7=\E[18~:\
:k8=\E[19~:k9=\E[20~:k;=\E[21~:F1=\E[23~:F2=\E[24~:\
:kh=\E[1~:@1=\E[1~:kH=\E[4~:@7=\E[4~:kN=\E[6~:kP=\E[5~:\
:kI=\E[2~:kD=\E[3~:ku=\EOA:kd=\EOB:kr=\EOC:kl=\EOD:WINDOW=0SHELL=/bin/shPATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binLANG=en_GB.UTF-8container=lxc

So it looks like that "container" environment variable is already set on
PID1

Regards,
John

[Michael H. Warfield]

Post by John

Post by Michael H. Warfield

Post by Michael H. Warfield
This is being directed to the systemd-devel community but I'm cc'ing the
lxc-users community and the Fedora community on this for their input as
well. I know it's not always good to cross post between multiple lists
but this is of interest to all three communities who may have valuable
input.
I'm new to this particular list, just having joined after tracking a
problem down to some systemd internals...
Several people over the last year or two on the lxc-users list have been
discussions trying to run certain distros (notably Fedora 16 and above,
recent Arch Linux and possibly others) in LXC containers, virualizing
entire servers this way. This is very similar to Virtuoso / OpenVZ only
it's using the native Linux cgroups for the containers (primary reason I
dumped OpenVZ was to avoid their custom patched kernels). These recent
distros have switched to systemd for the main init process and this has
proven to be disastrous for those of us using LXC and trying to install
or update our containers.
To put it bluntly, it doesn't work and causes all sorts of problems on
the host.
To summarize the problem... The LXC startup binary sets up various
things for /dev and /dev/pts for the container to run properly and this
works perfectly fine for SystemV start-up scripts and/or Upstart.
Unfortunately, systemd has mounts of devtmpfs on /dev and devpts
on /dev/pts which then break things horribly. This is because the
kernel currently lacks namespaces for devices and won't for some time to
come (in design). When devtmpfs gets mounted over top of /dev in the
container, it then hijacks the hosts console tty and several other
devices which had been set up through bind mounts by LXC and should have
been LEFT ALONE.
Yes! I recognize that this problem with devtmpfs and lack of namespaces
is a potential security problem anyways that could (and does) cause
serious container-to-host problems. We're just not going to get that
fixed right away in the linux cgroups and namespaces.
How do we work around this problem in systemd where it has hard coded
mounts in the binary that we can't override or configure? Or is it
there and I'm just missing it trying to examine the sources? That's how
I found where the problem lay.

http://www.freedesktop.org/wiki/Software/systemd/ContainerInterface

A very long ways, yeah. That looks like it could be just what we've
been looking for. Just gotta figure out how to set that environment
variable but that's up to a couple of others to comment on in the
lxc-users list. Then we'll see where we go from there.
Many thanks!

Kay

Regards,
Mike

I've just performed a very quick check on my Arch Linux system here.
# cat /proc/1/environ
TERM=linuxRD_TIMESTAMP=
# cat /proc/1/environ
STY=623.systemd-lithiumTERM=screenTERMCAP=SC|screen|VT 100/ANSI X3.64
virtual terminal:\
:DO=\E[%dB:LE=\E[%dD:RI=\E[%dC:UP=\E[%dA:bs:bt=\E[Z:\
:cd=\E[J:ce=\E[K:cl=\E[H\E[J:cm=\E[%i%d;%dH:ct=\E[3g:\
:do=^J:nd=\E[C:pt:rc=\E8:rs=\Ec:sc=\E7:st=\EH:up=\EM:\
:le=^H:bl=^G:cr=^M:it#8:ho=\E[H:nw=\EE:ta=^I:is=\E)0:\
:li#24:co#80:am:xn:xv:LP:sr=\EM:al=\E[L:AL=\E[%dL:\
:cs=\E[%i%d;%dr:dl=\E[M:DL=\E[%dM:dc=\E[P:DC=\E[%dP:\
:ke=\E[?1l\E>:vi=\E[?25l:ve=\E[34h\E[?25h:vs=\E[34l:\
:ti=\E[?1049h:te=\E[?1049l:k0=\E[10~:k1=\EOP:k2=\EOQ:\
:k3=\EOR:k4=\EOS:k5=\E[15~:k6=\E[17~:k7=\E[18~:\
:k8=\E[19~:k9=\E[20~:k;=\E[21~:F1=\E[23~:F2=\E[24~:\
:kI=\E[2~:kD=\E[3~:ku=\EOA:kd=\EOB:kr=\EOC:kl=\EOD:WINDOW=0SHELL=/bin/shPATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binLANG=en_GB.UTF-8container=lxc
So it looks like that "container" environment variable is already set on
PID1

Yeah, I saw that myself last night. Testing that out and it's still not
working here (although it doesn't seem to be grabbing the host console
now) if I use systemd but upstart fires right up and I see that
container variable set. Looked like a number of mounts listed on that
wiki page. Maybe something is missing. Right now it's just hanging
trying to start the container and, when I subsequently try to shut the
container down it results in a hung resource and it can't delete the
cgroups directory because it's busy. Only thing I did was change the
link to /sbin/init from upstart to systemd and it's now dead and I'll
have to reboot the host to free the resource. :-P

Post by John
Regards,
John

Regards,
Mike
--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121022/1227a20c/attachment.pgp>

[Michael H. Warfield]

Post by Michael H. Warfield
To summarize the problem... The LXC startup binary sets up various
things for /dev and /dev/pts for the container to run properly and this
works perfectly fine for SystemV start-up scripts and/or Upstart.
Unfortunately, systemd has mounts of devtmpfs on /dev and devpts
on /dev/pts which then break things horribly. This is because the
kernel currently lacks namespaces for devices and won't for some time to
come (in design). When devtmpfs gets mounted over top of /dev in the
container, it then hijacks the hosts console tty and several other
devices which had been set up through bind mounts by LXC and should have
been LEFT ALONE.

Please initialize a minimal tmpfs on /dev. systemd will then work fine.

My containers have a reasonable /dev that work with Upstart just fine
but they are not on tmpfs. Is mounting tmpfs on /dev and recreating
that minimal /dev required?

Well, it can be any kind of mount really. Just needs to be a mount. And
the idea is to use tmpfs for this.
What /dev are you currently using? It's probably not a good idea to
reuse the hosts' /dev, since it contains so many device nodes that
should not be accessible/visible to the container.

Got it. And that explains the problems we're seeing but also what I'm
seeing in some libvirt-lxc related pages, which is a separate and
distinct project in spite of the similarities in the name...

http://wiki.1tux.org/wiki/Lxc/Installation#Additional_notes

Unfortunately, in our case, merely getting a mount in there is a
complication in that it also has to be populated but, at least, we
understand the problem set now.

systemd will make use of pre-existing mounts if they exist, and only
mount something new if they don't exist.

So you're saying that, if we have something mounted on /dev, that's what
prevents systemd from mounting devtmpfs on /dev?

Yes.

But, I have systemd running on my host system (F17) and containers with
sysvinit or upstart inits are all starting just fine. That sounds like
it should impact all containers as pivot_root() is issued before systemd
in the container is started. Or am I missing something here? That
sounds like a problem for Serge and others to investigate further. I'll
see about trying that workaround though.

The "shared" issue is F18, and it's about running LXC on a systemd
system, not about running systemd inside of LXC.

Whew! I'll deal with F18 when I need to deal with F18. That explains
why my F17 hosts are running and gives Serge and others a chance to
address this, forewarned. Thanks for that info.

Lennart
--
Lennart Poettering - Red Hat, Inc.

Regards,
Mike
--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121022/2cb63b19/attachment.pgp>

[Michael H. Warfield]

Sorry for taking a few days to get back on this. I was delivering a
guest lecture up at Fordham University last Tuesday so I was out of
pocket a couple of days or I would have responded sooner...

Post by Michael H. Warfield

Post by Michael H. Warfield
To summarize the problem... The LXC startup binary sets up various
things for /dev and /dev/pts for the container to run properly and this
works perfectly fine for SystemV start-up scripts and/or Upstart.
Unfortunately, systemd has mounts of devtmpfs on /dev and devpts
on /dev/pts which then break things horribly. This is because the
kernel currently lacks namespaces for devices and won't for some time to
come (in design). When devtmpfs gets mounted over top of /dev in the
container, it then hijacks the hosts console tty and several other
devices which had been set up through bind mounts by LXC and should have
been LEFT ALONE.

Please initialize a minimal tmpfs on /dev. systemd will then work fine.

My containers have a reasonable /dev that work with Upstart just fine
but they are not on tmpfs. Is mounting tmpfs on /dev and recreating
that minimal /dev required?

Well, it can be any kind of mount really. Just needs to be a mount. And
the idea is to use tmpfs for this.
What /dev are you currently using? It's probably not a good idea to
reuse the hosts' /dev, since it contains so many device nodes that
should not be accessible/visible to the container.

Got it. And that explains the problems we're seeing but also what I'm
seeing in some libvirt-lxc related pages, which is a separate and
distinct project in spite of the similarities in the name...
http://wiki.1tux.org/wiki/Lxc/Installation#Additional_notes
Unfortunately, in our case, merely getting a mount in there is a
complication in that it also has to be populated but, at least, we
understand the problem set now.

Ok... Serge and I were corresponding on the lxc-users list and he had a
suggestion that worked but I consider to be a bit of a sub-optimal
workaround. Ironically, it was to mount devtmpfs on /dev. We don't
(currently) have a method to auto-populate a tmpfs mount with the needed
devices and this provided it. It does have a problem that makes me
uncomfortable in that the container now has visibility into the
hosts /dev system. I'm a security expert and I'm not comfortable with
that "solution" even with the controls we have. We can control access
but still, not happy with that.

I now have a container that starts with systemd running more or less
properly. We do have some problems with the convention that has been
set up, however.

When running in this mode, you run on the console and you don't spawn
getty's on the tty's. There seems to be a problem with this.

In this mode, if I manually start the container in a terminal window,
that eventually results in a login prompt there. Under sysvinit and
upstart I don't get that and can detach.

If I run lxc-console (which attaches to one of the vtys) it gives me
nothing. Under sysvinit and upstart I get vty login prompts because
they have started getty on those vtys. This is important in case
network access has not started for one reason or another and the
container was started detached in the background.

If I start lxc-start in detached mode (-d -o {logfile}) lxc-start
redirects the system console to the log file and goes daemon. In this
case, the systemd container hangs and never starts.

I SUSPECT the hang condition is something to do with systemd trying to
start and interactive console on /dev/console, which sysvinit and
upstart do not do. Maybe we have to do something different with the
redirects in this case, but it's not working consistent with the other
packages. We should also start appropriate gettys on those vtys if they
are configured. Maybe start the getty's if the tty? exists up to a
configured limit (and don't restart if they immediately fail) and
obviously don't start them if they don't. It then gives up control over
that process. Also don't start a login on /dev/console if you DO start
a getty? That would make your behavior congruent with that of the other
two systems.

I've got some more problems relating to shutting down containers, some
of which may be related to mounting tmpfs on /run to which /var/run is
symlinked to. We're doing halt / restart detection by monitoring utmp
in that directory but it looks like utmp isn't even in that directory
anymore and mounting tmpfs on it was always problematical. We may have
to have a more generic method to detect when a container has shut down
or is restarting in that case. I'm also finding we end up with dangling
resources where we can't remove to cgroup directories after a halt and
that creates a serious problem I have to investigate further. Not sure
if it's a host problem running on F17 or it something to do with running
systemd in a container but I can not shut down this particular container
and subsequently restart it without restarting the entire host. Not
good is an understatement.

Regards,
Mike

Post by Michael H. Warfield

systemd will make use of pre-existing mounts if they exist, and only
mount something new if they don't exist.

So you're saying that, if we have something mounted on /dev, that's what
prevents systemd from mounting devtmpfs on /dev?

Yes.

But, I have systemd running on my host system (F17) and containers with
sysvinit or upstart inits are all starting just fine. That sounds like
it should impact all containers as pivot_root() is issued before systemd
in the container is started. Or am I missing something here? That
sounds like a problem for Serge and others to investigate further. I'll
see about trying that workaround though.

The "shared" issue is F18, and it's about running LXC on a systemd
system, not about running systemd inside of LXC.

Whew! I'll deal with F18 when I need to deal with F18. That explains
why my F17 hosts are running and gives Serge and others a chance to
address this, forewarned. Thanks for that info.

Lennart
--
Lennart Poettering - Red Hat, Inc.

Regards,
Mike

--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121025/cbdaa3fd/attachment.pgp>

[Serge Hallyn]

Post by Michael H. Warfield
Sorry for taking a few days to get back on this. I was delivering a
guest lecture up at Fordham University last Tuesday so I was out of
pocket a couple of days or I would have responded sooner...

Post by Michael H. Warfield

Post by Michael H. Warfield
To summarize the problem... The LXC startup binary sets up various
things for /dev and /dev/pts for the container to run properly and this
works perfectly fine for SystemV start-up scripts and/or Upstart.
Unfortunately, systemd has mounts of devtmpfs on /dev and devpts
on /dev/pts which then break things horribly. This is because the
kernel currently lacks namespaces for devices and won't for some time to
come (in design). When devtmpfs gets mounted over top of /dev in the
container, it then hijacks the hosts console tty and several other
devices which had been set up through bind mounts by LXC and should have
been LEFT ALONE.

Please initialize a minimal tmpfs on /dev. systemd will then work fine.

My containers have a reasonable /dev that work with Upstart just fine
but they are not on tmpfs. Is mounting tmpfs on /dev and recreating
that minimal /dev required?

Well, it can be any kind of mount really. Just needs to be a mount. And
the idea is to use tmpfs for this.
What /dev are you currently using? It's probably not a good idea to
reuse the hosts' /dev, since it contains so many device nodes that
should not be accessible/visible to the container.

Got it. And that explains the problems we're seeing but also what I'm
seeing in some libvirt-lxc related pages, which is a separate and
distinct project in spite of the similarities in the name...
http://wiki.1tux.org/wiki/Lxc/Installation#Additional_notes
Unfortunately, in our case, merely getting a mount in there is a
complication in that it also has to be populated but, at least, we
understand the problem set now.

Ok... Serge and I were corresponding on the lxc-users list and he had a
suggestion that worked but I consider to be a bit of a sub-optimal
workaround. Ironically, it was to mount devtmpfs on /dev. We don't

Oh, sorry - I take back that suggestion :)

Note that we have mount hooks, so templates could install a mount hook to
mount a tmpfs onto /dev and populate it.

Or, if everyone is going to need it, we could just add a 'lxc.populatedevs = 1'
option which does that without needing a hook.

devtmpfs should not be used in containers :)

-serge

[Michael H. Warfield]

Post by Serge Hallyn

Post by Michael H. Warfield
Sorry for taking a few days to get back on this. I was delivering a
guest lecture up at Fordham University last Tuesday so I was out of
pocket a couple of days or I would have responded sooner...

Post by Michael H. Warfield

Post by Michael H. Warfield
To summarize the problem... The LXC startup binary sets up various
things for /dev and /dev/pts for the container to run properly and this
works perfectly fine for SystemV start-up scripts and/or Upstart.
Unfortunately, systemd has mounts of devtmpfs on /dev and devpts
on /dev/pts which then break things horribly. This is because the
kernel currently lacks namespaces for devices and won't for some time to
come (in design). When devtmpfs gets mounted over top of /dev in the
container, it then hijacks the hosts console tty and several other
devices which had been set up through bind mounts by LXC and should have
been LEFT ALONE.

Please initialize a minimal tmpfs on /dev. systemd will then work fine.

My containers have a reasonable /dev that work with Upstart just fine
but they are not on tmpfs. Is mounting tmpfs on /dev and recreating
that minimal /dev required?

Well, it can be any kind of mount really. Just needs to be a mount. And
the idea is to use tmpfs for this.
What /dev are you currently using? It's probably not a good idea to
reuse the hosts' /dev, since it contains so many device nodes that
should not be accessible/visible to the container.

Got it. And that explains the problems we're seeing but also what I'm
seeing in some libvirt-lxc related pages, which is a separate and
distinct project in spite of the similarities in the name...
http://wiki.1tux.org/wiki/Lxc/Installation#Additional_notes
Unfortunately, in our case, merely getting a mount in there is a
complication in that it also has to be populated but, at least, we
understand the problem set now.

Ok... Serge and I were corresponding on the lxc-users list and he had a
suggestion that worked but I consider to be a bit of a sub-optimal
workaround. Ironically, it was to mount devtmpfs on /dev. We don't

Oh, sorry - I take back that suggestion :)

Well, it worked (sort of) and reinforced what the problem was and where
the solution lay so there's no need to be sorry for it. We learned and
we know why it's not the right solution. This is good. We made a lot
of progress on this just in the last week. This is very good.

Post by Serge Hallyn
Note that we have mount hooks, so templates could install a mount hook to
mount a tmpfs onto /dev and populate it.

Ah, now that is interesting. I haven't looked at that before. I need
to explore that further.

Post by Serge Hallyn
Or, if everyone is going to need it, we could just add a 'lxc.populatedevs = 1'
option which does that without needing a hook.

Eventually, with Fedora (and later RHEL / CentOS / SL), Arch Linux, and
others going to systemd, I think this is going to be needed sooner than
later.

Post by Serge Hallyn
devtmpfs should not be used in containers :)

Concur!

Post by Serge Hallyn
-serge

Regards,
Mike
--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121025/dbdb3154/attachment.pgp>

[Michael H. Warfield]

Hey Serge,

Post by Serge Hallyn

Post by Michael H. Warfield
Sorry for taking a few days to get back on this. I was delivering a
guest lecture up at Fordham University last Tuesday so I was out of
pocket a couple of days or I would have responded sooner...

Post by Michael H. Warfield

Post by Michael H. Warfield
To summarize the problem... The LXC startup binary sets up various
things for /dev and /dev/pts for the container to run properly and this
works perfectly fine for SystemV start-up scripts and/or Upstart.
Unfortunately, systemd has mounts of devtmpfs on /dev and devpts
on /dev/pts which then break things horribly. This is because the
kernel currently lacks namespaces for devices and won't for some time to
come (in design). When devtmpfs gets mounted over top of /dev in the
container, it then hijacks the hosts console tty and several other
devices which had been set up through bind mounts by LXC and should have
been LEFT ALONE.

Please initialize a minimal tmpfs on /dev. systemd will then work fine.

My containers have a reasonable /dev that work with Upstart just fine
but they are not on tmpfs. Is mounting tmpfs on /dev and recreating
that minimal /dev required?

Well, it can be any kind of mount really. Just needs to be a mount. And
the idea is to use tmpfs for this.
What /dev are you currently using? It's probably not a good idea to
reuse the hosts' /dev, since it contains so many device nodes that
should not be accessible/visible to the container.

Got it. And that explains the problems we're seeing but also what I'm
seeing in some libvirt-lxc related pages, which is a separate and
distinct project in spite of the similarities in the name...
http://wiki.1tux.org/wiki/Lxc/Installation#Additional_notes
Unfortunately, in our case, merely getting a mount in there is a
complication in that it also has to be populated but, at least, we
understand the problem set now.

Ok... Serge and I were corresponding on the lxc-users list and he had a
suggestion that worked but I consider to be a bit of a sub-optimal
workaround. Ironically, it was to mount devtmpfs on /dev. We don't

Oh, sorry - I take back that suggestion :)
Note that we have mount hooks, so templates could install a mount hook to
mount a tmpfs onto /dev and populate it.

Ok... I've done some cursory search and turned up nothing but some
comments about "pre mount hooks". Where is the documentation about this
feature and how I might use / implement it? Some examples would
probably suffice. Is there a require release version of lxc-utils?

Post by Serge Hallyn
Or, if everyone is going to need it, we could just add a 'lxc.populatedevs = 1'
option which does that without needing a hook.
devtmpfs should not be used in containers :)
-serge

Regards,
Mike
--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121025/77b8535d/attachment.pgp>

[Michael H. Warfield]

Post by Michael H. Warfield
Hey Serge,

...

Post by Michael H. Warfield

Post by Serge Hallyn
Oh, sorry - I take back that suggestion :)
Note that we have mount hooks, so templates could install a mount hook to
mount a tmpfs onto /dev and populate it.

Ok... I've done some cursory search and turned up nothing but some
comments about "pre mount hooks". Where is the documentation about this
feature and how I might use / implement it? Some examples would
probably suffice. Is there a require release version of lxc-utils?

I think I found what I needed in the changelog here:

http://www.mail-archive.com/lxc-devel at lists.sourceforge.net/msg01490.html

I'll play with it and report back.

Post by Michael H. Warfield

Post by Serge Hallyn
Or, if everyone is going to need it, we could just add a 'lxc.populatedevs = 1'
option which does that without needing a hook.
devtmpfs should not be used in containers :)
-serge

Regards,
Mike
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________ Lxc-users mailing list Lxc-users at lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users

--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121025/67a050ed/attachment.pgp>

[Serge Hallyn]

Post by Michael H. Warfield

Post by Michael H. Warfield
Hey Serge,

...

Post by Michael H. Warfield

Post by Serge Hallyn
Oh, sorry - I take back that suggestion :)
Note that we have mount hooks, so templates could install a mount hook to
mount a tmpfs onto /dev and populate it.

Ok... I've done some cursory search and turned up nothing but some
comments about "pre mount hooks". Where is the documentation about this
feature and how I might use / implement it? Some examples would
probably suffice. Is there a require release version of lxc-utils?

http://www.mail-archive.com/lxc-devel at lists.sourceforge.net/msg01490.html
I'll play with it and report back.

Also the "Lifecycle management hooks" section in
https://help.ubuntu.com/12.10/serverguide/lxc.html

Note that I'm thinking that having lxc-start guess how to fill in /dev
is wrong, because different distros and even different releases of the
same distros have different expectations. For instance ubuntu lucid
wants /dev/shm to be a directory, while precise+ wants a symlink. So
somehow the template should get involved, be it by adding a hook, or
simply specifying a configuration file which lxc uses internally to
decide how to create /dev.

Personally I'd prefer if /dev were always populated by the templates,
and containers (i.e. userspace) didn't mount a fresh tmpfs for /dev.
But that does complicate userspace, and we've seen it in debian/ubuntu
as well (i.e. at certain package upgrades which rely on /dev being
cleared after a reboot).

-serge

[Michael H. Warfield]

Post by Serge Hallyn

Post by Michael H. Warfield

Post by Michael H. Warfield
Hey Serge,

...

Post by Michael H. Warfield

Post by Serge Hallyn
Oh, sorry - I take back that suggestion :)
Note that we have mount hooks, so templates could install a mount hook to
mount a tmpfs onto /dev and populate it.

Ok... I've done some cursory search and turned up nothing but some
comments about "pre mount hooks". Where is the documentation about this
feature and how I might use / implement it? Some examples would
probably suffice. Is there a require release version of lxc-utils?

http://www.mail-archive.com/lxc-devel at lists.sourceforge.net/msg01490.html
I'll play with it and report back.

Also the "Lifecycle management hooks" section in
https://help.ubuntu.com/12.10/serverguide/lxc.html

This isn't working...

Based on what was in both of those articles, I added this entry to
another container (Plover) to test...

lxc.hook.mount = /var/lib/lxc/Plover/mount

When I run "lxc-start -n Plover", I see this:

[root at forest ~]# lxc-start -n Plover
lxc-start: unknow key lxc.hook.mount
lxc-start: failed to read configuration file

I'm running the latest rc...

[root at forest ~]# rpm -qa | grep lxc
lxc-0.8.0.rc2-1.fc16.x86_64
lxc-libs-0.8.0.rc2-1.fc16.x86_64
lxc-doc-0.8.0.rc2-1.fc16.x86_64

Is it something in git that hasn't made it to a release yet?

Post by Serge Hallyn
Note that I'm thinking that having lxc-start guess how to fill in /dev
is wrong, because different distros and even different releases of the
same distros have different expectations. For instance ubuntu lucid
wants /dev/shm to be a directory, while precise+ wants a symlink. So
somehow the template should get involved, be it by adding a hook, or
simply specifying a configuration file which lxc uses internally to
decide how to create /dev.

I agree this needs to be by some sort of convention or template that we
can adjust.

Post by Serge Hallyn
Personally I'd prefer if /dev were always populated by the templates,
and containers (i.e. userspace) didn't mount a fresh tmpfs for /dev.
But that does complicate userspace, and we've seen it in debian/ubuntu
as well (i.e. at certain package upgrades which rely on /dev being
cleared after a reboot).
-serge

Regards,
Mike
--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121025/7ecc0253/attachment.pgp>

[Michael H. Warfield]

Post by Michael H. Warfield

Post by Serge Hallyn

Post by Michael H. Warfield

Post by Michael H. Warfield
Hey Serge,

...

Post by Michael H. Warfield

Post by Serge Hallyn
Oh, sorry - I take back that suggestion :)
Note that we have mount hooks, so templates could install a mount hook to
mount a tmpfs onto /dev and populate it.

Ok... I've done some cursory search and turned up nothing but some
comments about "pre mount hooks". Where is the documentation about this
feature and how I might use / implement it? Some examples would
probably suffice. Is there a require release version of lxc-utils?

http://www.mail-archive.com/lxc-devel at lists.sourceforge.net/msg01490.html
I'll play with it and report back.

Also the "Lifecycle management hooks" section in
https://help.ubuntu.com/12.10/serverguide/lxc.html

This isn't working...
Based on what was in both of those articles, I added this entry to
another container (Plover) to test...
lxc.hook.mount = /var/lib/lxc/Plover/mount
[root at forest ~]# lxc-start -n Plover
lxc-start: unknow key lxc.hook.mount
lxc-start: failed to read configuration file
I'm running the latest rc...
[root at forest ~]# rpm -qa | grep lxc
lxc-0.8.0.rc2-1.fc16.x86_64
lxc-libs-0.8.0.rc2-1.fc16.x86_64
lxc-doc-0.8.0.rc2-1.fc16.x86_64
Is it something in git that hasn't made it to a release yet?

nm... I see it. It's in git and hasn't made it to a release. I'm
working on a git build to test now. If this is something that solves
some of this, we need to move things along here and get these things
moved out. According to git, 0.8.0rc2 was 7 months ago? What's the
show stoppers here?

Post by Michael H. Warfield

Post by Serge Hallyn
Note that I'm thinking that having lxc-start guess how to fill in /dev
is wrong, because different distros and even different releases of the
same distros have different expectations. For instance ubuntu lucid
wants /dev/shm to be a directory, while precise+ wants a symlink. So
somehow the template should get involved, be it by adding a hook, or
simply specifying a configuration file which lxc uses internally to
decide how to create /dev.

I agree this needs to be by some sort of convention or template that we
can adjust.

Post by Serge Hallyn
Personally I'd prefer if /dev were always populated by the templates,
and containers (i.e. userspace) didn't mount a fresh tmpfs for /dev.
But that does complicate userspace, and we've seen it in debian/ubuntu
as well (i.e. at certain package upgrades which rely on /dev being
cleared after a reboot).
-serge

Regards,
Mike

Mike
--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121025/242b3146/attachment.pgp>

[Michael H. Warfield]

Adding in the lxc-devel list.

Post by Michael H. Warfield

Post by Michael H. Warfield

Post by Serge Hallyn

Post by Michael H. Warfield

Post by Michael H. Warfield
Hey Serge,

...

Post by Michael H. Warfield

Post by Serge Hallyn
Oh, sorry - I take back that suggestion :)
Note that we have mount hooks, so templates could install a mount hook to
mount a tmpfs onto /dev and populate it.

Ok... I've done some cursory search and turned up nothing but some
comments about "pre mount hooks". Where is the documentation about this
feature and how I might use / implement it? Some examples would
probably suffice. Is there a require release version of lxc-utils?

http://www.mail-archive.com/lxc-devel at lists.sourceforge.net/msg01490.html
I'll play with it and report back.

Also the "Lifecycle management hooks" section in
https://help.ubuntu.com/12.10/serverguide/lxc.html

This isn't working...
Based on what was in both of those articles, I added this entry to
another container (Plover) to test...
lxc.hook.mount = /var/lib/lxc/Plover/mount
[root at forest ~]# lxc-start -n Plover
lxc-start: unknow key lxc.hook.mount
lxc-start: failed to read configuration file
I'm running the latest rc...
[root at forest ~]# rpm -qa | grep lxc
lxc-0.8.0.rc2-1.fc16.x86_64
lxc-libs-0.8.0.rc2-1.fc16.x86_64
lxc-doc-0.8.0.rc2-1.fc16.x86_64
Is it something in git that hasn't made it to a release yet?

nm... I see it. It's in git and hasn't made it to a release. I'm
working on a git build to test now. If this is something that solves
some of this, we need to move things along here and get these things
moved out. According to git, 0.8.0rc2 was 7 months ago? What's the
show stoppers here?

While the git repo says 7 months ago, the date stamp on the
lxc-0.8.0-rc2 tarball is from July 10, so about 3-1/2 months ago.
Sounds like we've accumulated some features (like the hooks) we are
going to need like months ago to deal with this systemd debacle. How
close are we to either 0.8.0rc3 or 0.8.0? Any blockers or are we just
waiting on some more features?

Post by Michael H. Warfield

Post by Michael H. Warfield

Post by Serge Hallyn
Note that I'm thinking that having lxc-start guess how to fill in /dev
is wrong, because different distros and even different releases of the
same distros have different expectations. For instance ubuntu lucid
wants /dev/shm to be a directory, while precise+ wants a symlink. So
somehow the template should get involved, be it by adding a hook, or
simply specifying a configuration file which lxc uses internally to
decide how to create /dev.

I agree this needs to be by some sort of convention or template that we
can adjust.

Post by Serge Hallyn
Personally I'd prefer if /dev were always populated by the templates,
and containers (i.e. userspace) didn't mount a fresh tmpfs for /dev.
But that does complicate userspace, and we've seen it in debian/ubuntu
as well (i.e. at certain package upgrades which rely on /dev being
cleared after a reboot).
-serge

Regards,
Mike

--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121026/a9c90afb/attachment.pgp>

[Serge Hallyn]

Post by Michael H. Warfield
Adding in the lxc-devel list.

Post by Michael H. Warfield

Post by Michael H. Warfield

Post by Serge Hallyn

Post by Michael H. Warfield

Post by Michael H. Warfield
Hey Serge,

...

Post by Michael H. Warfield

Post by Serge Hallyn
Oh, sorry - I take back that suggestion :)
Note that we have mount hooks, so templates could install a mount hook to
mount a tmpfs onto /dev and populate it.

Ok... I've done some cursory search and turned up nothing but some
comments about "pre mount hooks". Where is the documentation about this
feature and how I might use / implement it? Some examples would
probably suffice. Is there a require release version of lxc-utils?

http://www.mail-archive.com/lxc-devel at lists.sourceforge.net/msg01490.html
I'll play with it and report back.

Also the "Lifecycle management hooks" section in
https://help.ubuntu.com/12.10/serverguide/lxc.html

This isn't working...
Based on what was in both of those articles, I added this entry to
another container (Plover) to test...
lxc.hook.mount = /var/lib/lxc/Plover/mount
[root at forest ~]# lxc-start -n Plover
lxc-start: unknow key lxc.hook.mount
lxc-start: failed to read configuration file
I'm running the latest rc...
[root at forest ~]# rpm -qa | grep lxc
lxc-0.8.0.rc2-1.fc16.x86_64
lxc-libs-0.8.0.rc2-1.fc16.x86_64
lxc-doc-0.8.0.rc2-1.fc16.x86_64
Is it something in git that hasn't made it to a release yet?

nm... I see it. It's in git and hasn't made it to a release. I'm
working on a git build to test now. If this is something that solves
some of this, we need to move things along here and get these things
moved out. According to git, 0.8.0rc2 was 7 months ago? What's the
show stoppers here?

While the git repo says 7 months ago, the date stamp on the
lxc-0.8.0-rc2 tarball is from July 10, so about 3-1/2 months ago.
Sounds like we've accumulated some features (like the hooks) we are
going to need like months ago to deal with this systemd debacle. How
close are we to either 0.8.0rc3 or 0.8.0? Any blockers or are we just
waiting on some more features?

Daniel has simply been too busy. St?phane has made a new branch which
cherrypicks 50 bugfixes for 0.8.0, with the remaining patches (about
twice as many) left for 0.9.0. I'm hoping we get 0.8.0 next week :)

[Michael H. Warfield]

Post by Serge Hallyn

Post by Michael H. Warfield

Post by Michael H. Warfield
nm... I see it. It's in git and hasn't made it to a release. I'm
working on a git build to test now. If this is something that solves
some of this, we need to move things along here and get these things
moved out. According to git, 0.8.0rc2 was 7 months ago? What's the
show stoppers here?

While the git repo says 7 months ago, the date stamp on the
lxc-0.8.0-rc2 tarball is from July 10, so about 3-1/2 months ago.
Sounds like we've accumulated some features (like the hooks) we are
going to need like months ago to deal with this systemd debacle. How
close are we to either 0.8.0rc3 or 0.8.0? Any blockers or are we just
waiting on some more features?

Daniel has simply been too busy.

Don't I know THAT feeling all too well. Over on the Samba Team (where
I'm the chief security consultant on the team) we're all too busy with
juggling our domain and our web cert. On top of that, I've got my day
job (of course). On top of that, I've got about six other OpenSource
projects I'm juggling (including this one). On top of that, I've got a
consulting customer that's going through fits. And the beat goes on.

I'll test out things as fast as I can. I need this. This suddenly got
very interesting as soon as we had a thread to pick at on the systemd
ball of yarn.

Post by Serge Hallyn
St?phane has made a new branch which
cherrypicks 50 bugfixes for 0.8.0, with the remaining patches (about
twice as many) left for 0.9.0. I'm hoping we get 0.8.0 next week :)

I'm hoping the hook patches are in that cherry picked basket. We really
need them if that's what it takes to make this work. Looking forward to
it. :-)=)

I'm going to look further into this whole redirect /dev/console to a log
hang thing. That's not good and may need to be resolved soon as well.
I can live with losing the vty's although I disagree with St?phan's
arguments. They (systemd) are behaving significantly different from
sysvinit and upstart and they claim they want to be transparent? Not.
No matter. We need to make that work properly as well, agree with them
or disagree with them.

Regards,
Mike
--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121026/fcdb0d89/attachment.pgp>

[Michael H. Warfield]

Post by Serge Hallyn

Post by Michael H. Warfield
Adding in the lxc-devel list.

Post by Michael H. Warfield

Post by Michael H. Warfield

Post by Serge Hallyn

Post by Michael H. Warfield

Post by Michael H. Warfield
Hey Serge,

...

Post by Michael H. Warfield

Post by Serge Hallyn
Oh, sorry - I take back that suggestion :)
Note that we have mount hooks, so templates could install a mount hook to
mount a tmpfs onto /dev and populate it.

Ok... I've done some cursory search and turned up nothing but some
comments about "pre mount hooks". Where is the documentation about this
feature and how I might use / implement it? Some examples would
probably suffice. Is there a require release version of lxc-utils?

http://www.mail-archive.com/lxc-devel at lists.sourceforge.net/msg01490.html
I'll play with it and report back.

Also the "Lifecycle management hooks" section in
https://help.ubuntu.com/12.10/serverguide/lxc.html

This isn't working...
Based on what was in both of those articles, I added this entry to
another container (Plover) to test...
lxc.hook.mount = /var/lib/lxc/Plover/mount
[root at forest ~]# lxc-start -n Plover
lxc-start: unknow key lxc.hook.mount
lxc-start: failed to read configuration file
I'm running the latest rc...
[root at forest ~]# rpm -qa | grep lxc
lxc-0.8.0.rc2-1.fc16.x86_64
lxc-libs-0.8.0.rc2-1.fc16.x86_64
lxc-doc-0.8.0.rc2-1.fc16.x86_64
Is it something in git that hasn't made it to a release yet?

nm... I see it. It's in git and hasn't made it to a release. I'm
working on a git build to test now. If this is something that solves
some of this, we need to move things along here and get these things
moved out. According to git, 0.8.0rc2 was 7 months ago? What's the
show stoppers here?

While the git repo says 7 months ago, the date stamp on the
lxc-0.8.0-rc2 tarball is from July 10, so about 3-1/2 months ago.
Sounds like we've accumulated some features (like the hooks) we are
going to need like months ago to deal with this systemd debacle. How
close are we to either 0.8.0rc3 or 0.8.0? Any blockers or are we just
waiting on some more features?

Daniel has simply been too busy. St?phane has made a new branch which
cherrypicks 50 bugfixes for 0.8.0, with the remaining patches (about
twice as many) left for 0.9.0. I'm hoping we get 0.8.0 next week :)

Trying to build latest from git. This is not good...

checking sys/apparmor.h usability... no
checking sys/apparmor.h presence... no
checking for sys/apparmor.h... no
configure: error: You must install the AppArmor development package in
order to compile lxc

What am I suppose to do on Fedora where we don't have that package? Is
it available in another repo somewhere? I'm looking and not finding.

Regards,
Mike

Post by Serge Hallyn
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________
Lxc-users mailing list
Lxc-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lxc-users

--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121027/a5e23660/attachment.pgp>

[Michael H. Warfield]

Post by Michael H. Warfield

Post by Serge Hallyn

Post by Michael H. Warfield
Adding in the lxc-devel list.

Post by Michael H. Warfield

Post by Michael H. Warfield

Post by Serge Hallyn

Post by Michael H. Warfield

Post by Michael H. Warfield
Hey Serge,

...

Post by Michael H. Warfield

Post by Serge Hallyn
Oh, sorry - I take back that suggestion :)
Note that we have mount hooks, so templates could install a mount hook to
mount a tmpfs onto /dev and populate it.

Ok... I've done some cursory search and turned up nothing but some
comments about "pre mount hooks". Where is the documentation about this
feature and how I might use / implement it? Some examples would
probably suffice. Is there a require release version of lxc-utils?

http://www.mail-archive.com/lxc-devel at lists.sourceforge.net/msg01490.html
I'll play with it and report back.

Also the "Lifecycle management hooks" section in
https://help.ubuntu.com/12.10/serverguide/lxc.html

This isn't working...
Based on what was in both of those articles, I added this entry to
another container (Plover) to test...
lxc.hook.mount = /var/lib/lxc/Plover/mount
[root at forest ~]# lxc-start -n Plover
lxc-start: unknow key lxc.hook.mount
lxc-start: failed to read configuration file
I'm running the latest rc...
[root at forest ~]# rpm -qa | grep lxc
lxc-0.8.0.rc2-1.fc16.x86_64
lxc-libs-0.8.0.rc2-1.fc16.x86_64
lxc-doc-0.8.0.rc2-1.fc16.x86_64
Is it something in git that hasn't made it to a release yet?

nm... I see it. It's in git and hasn't made it to a release. I'm
working on a git build to test now. If this is something that solves
some of this, we need to move things along here and get these things
moved out. According to git, 0.8.0rc2 was 7 months ago? What's the
show stoppers here?

While the git repo says 7 months ago, the date stamp on the
lxc-0.8.0-rc2 tarball is from July 10, so about 3-1/2 months ago.
Sounds like we've accumulated some features (like the hooks) we are
going to need like months ago to deal with this systemd debacle. How
close are we to either 0.8.0rc3 or 0.8.0? Any blockers or are we just
waiting on some more features?

Daniel has simply been too busy. St?phane has made a new branch which
cherrypicks 50 bugfixes for 0.8.0, with the remaining patches (about
twice as many) left for 0.9.0. I'm hoping we get 0.8.0 next week :)

Trying to build latest from git. This is not good...
checking sys/apparmor.h usability... no
checking sys/apparmor.h presence... no
checking for sys/apparmor.h... no
configure: error: You must install the AppArmor development package in
order to compile lxc
What am I suppose to do on Fedora where we don't have that package? Is
it available in another repo somewhere? I'm looking and not finding.

nm... I see that --enable-apparmor is defaulted to on. I just had to
add an option to --disable-apparmor. Sorry for the noise.

Post by Michael H. Warfield
Regards,
Mike

Mike

Post by Michael H. Warfield

Post by Serge Hallyn
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________
Lxc-users mailing list
Lxc-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lxc-users

------------------------------------------------------------------------------
WINDOWS 8 is here.
Millions of people. Your app in 30 days.
Visit The Windows 8 Center at Sourceforge for all your go to resources.
http://windows8center.sourceforge.net/
join-generation-app-and-make-money-coding-fast/
_______________________________________________ Lxc-users mailing list Lxc-users at lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users

--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121027/32a059dd/attachment.pgp>

[Michael H. Warfield]

/me erasing everything at this point and taking off the systemd crew,
since this will have no relevance to them...

Testing the hook feature out using git rev (finally got it built)...

I added this line to my config...

lxc.mount.entry=tmpfs /srv/lxc/private/Plover/dev.tmp tmpfs defaults 0 0
lxc.hook.mount = /var/lib/lxc/Plover/mount

In /var/lib/lxc/Plover/mount I have this:
--
rsync -avAH /srv/lxc/private/Plover/dev.template/. /srv/lxc/private/Plover/dev.tmp/
--
(This is just testing out the concepts.

If I understand this correctly, lxc.hook.pre-mount runs BEFORE the
mounting takes place and lxc.hook.mount takes place after the mount.

Problem is, the result of that rsync is not showing up in the mounted
tmpfs file system but is showing up in the underlying parent file system
as if it were run pre-mount. Something not right here...

Regards,
Mike
--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121027/01313e8b/attachment.pgp>

[Michael H. Warfield]

Post by Michael H. Warfield
/me erasing everything at this point and taking off the systemd crew,
since this will have no relevance to them...
Testing the hook feature out using git rev (finally got it built)...
I added this line to my config...
lxc.mount.entry=tmpfs /srv/lxc/private/Plover/dev.tmp tmpfs defaults 0 0
lxc.hook.mount = /var/lib/lxc/Plover/mount
--
rsync -avAH /srv/lxc/private/Plover/dev.template/. /srv/lxc/private/Plover/dev.tmp/
--
(This is just testing out the concepts.
If I understand this correctly, lxc.hook.pre-mount runs BEFORE the
mounting takes place and lxc.hook.mount takes place after the mount.
Problem is, the result of that rsync is not showing up in the mounted
tmpfs file system but is showing up in the underlying parent file system
as if it were run pre-mount. Something not right here...

I changed it to "lxc.hook.start = /srv/lxc/mount" (where I put the
script in the container) which then works but that then requires the
template and the command to be in the container. Suboptimal to say the
least. But it gives me a way to test this tmpfs thing out.

I also noticed that the .start hook runs, it appears, after caps are
dropped and I see a lot of bitching about mknod on certain devices. I
had to thrown an exit 0 into that script so it would continue in spite
of the errors but, now, I can refine my template based on what it could
create.

Post by Michael H. Warfield
Regards,
Mike

Regards,
Mike
--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121027/af8a8939/attachment.pgp>

[Michael H. Warfield]

Post by Michael H. Warfield

Post by Michael H. Warfield
/me erasing everything at this point and taking off the systemd crew,
since this will have no relevance to them...
Testing the hook feature out using git rev (finally got it built)...
I added this line to my config...
lxc.mount.entry=tmpfs /srv/lxc/private/Plover/dev.tmp tmpfs defaults 0 0
lxc.hook.mount = /var/lib/lxc/Plover/mount
--
rsync -avAH /srv/lxc/private/Plover/dev.template/. /srv/lxc/private/Plover/dev.tmp/
--
(This is just testing out the concepts.
If I understand this correctly, lxc.hook.pre-mount runs BEFORE the
mounting takes place and lxc.hook.mount takes place after the mount.
Problem is, the result of that rsync is not showing up in the mounted
tmpfs file system but is showing up in the underlying parent file system
as if it were run pre-mount. Something not right here...

I changed it to "lxc.hook.start = /srv/lxc/mount" (where I put the
script in the container) which then works but that then requires the
template and the command to be in the container. Suboptimal to say the
least. But it gives me a way to test this tmpfs thing out.
I also noticed that the .start hook runs, it appears, after caps are
dropped and I see a lot of bitching about mknod on certain devices. I
had to thrown an exit 0 into that script so it would continue in spite
of the errors but, now, I can refine my template based on what it could
create.

Crap. I've got a catch-22 here... This is going to take some work.

Yes, we can create the /dev directory with tmpfs from a template.
Problem is that /dev/pts does not exist at the time we need to mount the
devpts on /dev/pts for the pty's so that hurls chunks and dies. We
can't create the /dev/ directory contents prior to mounting in the
pre-mount hook because we won't have tmpfs in place at the time. We
have to get tmpfs mounted on /dev and then create /dev/pts and then
mount the ptys in there. There has to be a mkdir in between those two
mount actions. Simplest solution would seem to be to add some logic to
the mount logic that says "test if directory exists and, if not, create
it." I'm not sure of the consequences of that, though.

I don't see a way to make this happen with hooks. It's almost like we
need and on-mount per mount hook.

Post by Michael H. Warfield

Post by Michael H. Warfield
Regards,
Mike

Regards,
Mike

Regards,
Mike
--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121027/76dcfbec/attachment.pgp>

[Serge Hallyn]

Post by Michael H. Warfield

Post by Michael H. Warfield

Post by Michael H. Warfield
/me erasing everything at this point and taking off the systemd crew,
since this will have no relevance to them...
Testing the hook feature out using git rev (finally got it built)...
I added this line to my config...
lxc.mount.entry=tmpfs /srv/lxc/private/Plover/dev.tmp tmpfs defaults 0 0
lxc.hook.mount = /var/lib/lxc/Plover/mount
--
rsync -avAH /srv/lxc/private/Plover/dev.template/. /srv/lxc/private/Plover/dev.tmp/
--
(This is just testing out the concepts.
If I understand this correctly, lxc.hook.pre-mount runs BEFORE the
mounting takes place and lxc.hook.mount takes place after the mount.
Problem is, the result of that rsync is not showing up in the mounted
tmpfs file system but is showing up in the underlying parent file system
as if it were run pre-mount. Something not right here...

I changed it to "lxc.hook.start = /srv/lxc/mount" (where I put the
script in the container) which then works but that then requires the
template and the command to be in the container. Suboptimal to say the
least. But it gives me a way to test this tmpfs thing out.
I also noticed that the .start hook runs, it appears, after caps are
dropped and I see a lot of bitching about mknod on certain devices. I
had to thrown an exit 0 into that script so it would continue in spite
of the errors but, now, I can refine my template based on what it could
create.

Crap. I've got a catch-22 here... This is going to take some work.

Hey,

I've got a rather minimal patch (appended below) to add the support for
mounting and populating a minimal /dev working. (A few hours were wasted
due to my not knowing that upstart was going to issue mounted-dev even though
/dev was mounted before upstart started - and the mounted-dev hook deletes
and recreates all consoles. GAH)

Post by Michael H. Warfield
Yes, we can create the /dev directory with tmpfs from a template.
Problem is that /dev/pts does not exist at the time we need to mount the
devpts on /dev/pts for the pty's so that hurls chunks and dies. We
can't create the /dev/ directory contents prior to mounting in the
pre-mount hook because we won't have tmpfs in place at the time. We
have to get tmpfs mounted on /dev and then create /dev/pts and then
mount the ptys in there. There has to be a mkdir in between those two
mount actions. Simplest solution would seem to be to add some logic to
the mount logic that says "test if directory exists and, if not, create
it." I'm not sure of the consequences of that, though.
I don't see a way to make this happen with hooks. It's almost like we
need and on-mount per mount hook.

Should be moot given my patch, which I intend to push this week, but why
couldn't a lxc.hook.mount do the whole thing, mount /dev and and populate
it? I wasn't thinking a lxc.hook.start, for the reasons you encountered,
but I assume you tried lxc.hook.mount and it failed?

Patch below:

Index: lxc-qp/src/lxc/conf.c
===================================================================
--- lxc-qp.orig/src/lxc/conf.c 2012-10-27 17:24:50.768383000 -0500
+++ lxc-qp/src/lxc/conf.c 2012-10-28 05:44:07.871228322 -0500
@@ -619,7 +619,7 @@
}

if (mount(pty_info->name, lxcpath, "none", MS_BIND, 0)) {
- WARN("failed to mount '%s'->'%s'",
+ SYSERROR("failed to mount '%s'->'%s'",
pty_info->name, path);
continue;
}
@@ -636,7 +636,7 @@
}
} else {
if (mount(pty_info->name, path, "none", MS_BIND, 0)) {
- WARN("failed to mount '%s'->'%s'",
+ SYSERROR("failed to mount '%s'->'%s'",
pty_info->name, path);
continue;
}
@@ -842,6 +842,67 @@
return 0;
}

+struct lxc_devs {
+ char *name;
+ mode_t mode;
+ int maj;
+ int min;
+};
+
+struct lxc_devs lxc_devs[] = {
+ { "null", S_IFCHR | S_IRWXU | S_IRWXG | S_IRWXO, 1, 3 },
+ { "zero", S_IFCHR | S_IRWXU | S_IRWXG | S_IRWXO, 1, 5 },
+ { "full", S_IFCHR | S_IRWXU | S_IRWXG | S_IRWXO, 1, 7 },
+ { "urandom", S_IFCHR | S_IRWXU | S_IRWXG | S_IRWXO, 1, 9 },
+ { "random", S_IFCHR | S_IRWXU | S_IRWXG | S_IRWXO, 1, 8 },
+ { "tty", S_IFCHR | S_IRWXU | S_IRWXG | S_IRWXO, 5, 0 },
+ { "console", S_IFCHR | S_IRUSR | S_IWUSR, 5, 1 },
+};
+
+/*
+ * Do we want to add options for max size of /dev and a file to
+ * specify which devices to create?
+ */
+static int setup_autodev(char *root)
+{
+ int ret;
+ struct lxc_devs *d;
+ char path[MAXPATHLEN];
+ int i;
+
+ INFO("Creating and populating /dev under %s\n", root);
+ ret = snprintf(path, MAXPATHLEN, "%s/dev", root);
+ if (ret < 0 || ret > MAXPATHLEN)
+ return -1;
+ ret = mount("none", path, "tmpfs", 0, "size=100000");
+ if (ret) {
+ SYSERROR("Failed to mount /dev at %s\n", root);
+ return -1;
+ }
+ for (i = 0; i < sizeof(lxc_devs) / sizeof(lxc_devs[0]); i++) {
+ d = &lxc_devs[i];
+ ret = snprintf(path, MAXPATHLEN, "%s/dev/%s", root, d->name);
+ if (ret < 0 || ret >= MAXPATHLEN)
+ return -1;
+ ret = mknod(path, d->mode, makedev(d->maj, d->min));
+ if (ret) {
+ SYSERROR("Error creating %s\n", d->name);
+ return -1;
+ }
+ }
+ ret = snprintf(path, MAXPATHLEN, "%s/dev/pts", root);
+ if (ret < 0 || ret >= MAXPATHLEN)
+ return -1;
+ ret = mkdir(path, S_IRWXU | S_IRGRP | S_IXGRP | S_IROTH | S_IXOTH);
+ if (ret) {
+ SYSERROR("Failed to create /dev/pts in container");
+ return -1;
+ }
+
+ INFO("Populated /dev under %s\n", root);
+ return 0;
+}
+
static int setup_rootfs(const struct lxc_rootfs *rootfs)
{
if (!rootfs->path)
@@ -2208,6 +2269,13 @@
return -1;
}

+ if (lxc_conf->autodev) {
+ if (setup_autodev(lxc_conf->rootfs.mount)) {
+ ERROR("failed to set up /dev in the container");
+ return -1;
+ }
+ }
+
if (setup_mount(&lxc_conf->rootfs, lxc_conf->fstab, name)) {
ERROR("failed to setup the mounts for '%s'", name);
return -1;
Index: lxc-qp/src/lxc/conf.h
===================================================================
--- lxc-qp.orig/src/lxc/conf.h 2012-10-27 17:24:50.768383000 -0500
+++ lxc-qp/src/lxc/conf.h 2012-10-27 17:24:50.768383000 -0500
@@ -227,6 +227,7 @@
struct lxc_list hooks[NUM_LXC_HOOKS];
char *seccomp; // filename with the seccomp rules
int maincmd_fd;
+ int autodev; // if 1, mount and fill a /dev at start
};

int run_lxc_hooks(const char *name, char *hook, struct lxc_conf *conf);
Index: lxc-qp/src/lxc/confile.c
===================================================================
--- lxc-qp.orig/src/lxc/confile.c 2012-10-27 17:24:50.768383000 -0500
+++ lxc-qp/src/lxc/confile.c 2012-10-27 17:24:50.768383000 -0500
@@ -77,6 +77,7 @@
static int config_seccomp(const char *, char *, struct lxc_conf *);
static int config_includefile(const char *, char *, struct lxc_conf *);
static int config_network_nic(const char *, char *, struct lxc_conf *);
+static int config_autodev(const char *, char *, struct lxc_conf *);

typedef int (*config_cb)(const char *, char *, struct lxc_conf *);

@@ -118,6 +119,7 @@
{ "lxc.console", config_console },
{ "lxc.seccomp", config_seccomp },
{ "lxc.include", config_includefile },
+ { "lxc.autodev", config_autodev },
};

static const size_t config_size = sizeof(config)/sizeof(struct lxc_config_t);
@@ -853,6 +855,16 @@

return 0;
}
+
+static int config_autodev(const char *key, char *value,
+ struct lxc_conf *lxc_conf)
+{
+ int v = atoi(value);
+
+ lxc_conf->autodev = v;
+
+ return 0;
+}

static int config_aa_profile(const char *key, char *value,
struct lxc_conf *lxc_conf)

[Michael H. Warfield]

Post by Serge Hallyn

Post by Michael H. Warfield

Post by Michael H. Warfield

Post by Michael H. Warfield
/me erasing everything at this point and taking off the systemd crew,
since this will have no relevance to them...
Testing the hook feature out using git rev (finally got it built)...
I added this line to my config...
lxc.mount.entry=tmpfs /srv/lxc/private/Plover/dev.tmp tmpfs defaults 0 0
lxc.hook.mount = /var/lib/lxc/Plover/mount
--
rsync -avAH /srv/lxc/private/Plover/dev.template/. /srv/lxc/private/Plover/dev.tmp/
--
(This is just testing out the concepts.
If I understand this correctly, lxc.hook.pre-mount runs BEFORE the
mounting takes place and lxc.hook.mount takes place after the mount.
Problem is, the result of that rsync is not showing up in the mounted
tmpfs file system but is showing up in the underlying parent file system
as if it were run pre-mount. Something not right here...

I changed it to "lxc.hook.start = /srv/lxc/mount" (where I put the
script in the container) which then works but that then requires the
template and the command to be in the container. Suboptimal to say the
least. But it gives me a way to test this tmpfs thing out.
I also noticed that the .start hook runs, it appears, after caps are
dropped and I see a lot of bitching about mknod on certain devices. I
had to thrown an exit 0 into that script so it would continue in spite
of the errors but, now, I can refine my template based on what it could
create.

Crap. I've got a catch-22 here... This is going to take some work.

Hey,
I've got a rather minimal patch (appended below) to add the support for
mounting and populating a minimal /dev working. (A few hours were wasted
due to my not knowing that upstart was going to issue mounted-dev even though
/dev was mounted before upstart started - and the mounted-dev hook deletes
and recreates all consoles. GAH)

Post by Michael H. Warfield
Yes, we can create the /dev directory with tmpfs from a template.
Problem is that /dev/pts does not exist at the time we need to mount the
devpts on /dev/pts for the pty's so that hurls chunks and dies. We
can't create the /dev/ directory contents prior to mounting in the
pre-mount hook because we won't have tmpfs in place at the time. We
have to get tmpfs mounted on /dev and then create /dev/pts and then
mount the ptys in there. There has to be a mkdir in between those two
mount actions. Simplest solution would seem to be to add some logic to
the mount logic that says "test if directory exists and, if not, create
it." I'm not sure of the consequences of that, though.
I don't see a way to make this happen with hooks. It's almost like we
need and on-mount per mount hook.

Should be moot given my patch, which I intend to push this week, but why
couldn't a lxc.hook.mount do the whole thing, mount /dev and and populate
it? I wasn't thinking a lxc.hook.start, for the reasons you encountered,
but I assume you tried lxc.hook.mount and it failed?

See my other comment about lxc.hook.mount. I tried using it to
populate /dev but it showed up in the parent of the mount undeneath the
tmpfs mount. It was like it ran pre-mount. I tried it for several
different combinations and couldn't get it to go. Would still have the
problem with mounting /dev/pts which would take place before the mount
hook at run to mount the file system and populate it. That actually
MIGHT work (gotta think on this now) if I used lxc.hook.pre-mount and
mounted tmpfs over /dev, and populated it but then I run into a problem
where I was using a bind-mount for the rootfs. Might still work. I'll
test your patch out first though.
<snip>

Regards,
Mike
--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121028/d1006bd3/attachment.pgp>

[Michael H. Warfield]

Post by Serge Hallyn
Should be moot given my patch, which I intend to push this week, but why
couldn't a lxc.hook.mount do the whole thing, mount /dev and and populate
it? I wasn't thinking a lxc.hook.start, for the reasons you
encountered,
but I assume you tried lxc.hook.mount and it failed?

Patch failed against 0.8.0rc2 and git root. Even with loose patching
for whitespace, had failures...

This was against git:

[mhw at forest lxc]$ patch -p1 -l < ../lxc-autodev.patch
patching file src/lxc/conf.c
Hunk #1 succeeded at 616 (offset -3 lines).
Hunk #2 succeeded at 633 (offset -3 lines).
Hunk #3 succeeded at 839 (offset -3 lines).
Hunk #4 succeeded at 2203 (offset -66 lines).
patching file src/lxc/conf.h
Hunk #1 FAILED at 227.
1 out of 1 hunk FAILED -- saving rejects to file src/lxc/conf.h.rej
patching file src/lxc/confile.c
Hunk #1 FAILED at 77.
Hunk #2 FAILED at 118.
Hunk #3 succeeded at 854 with fuzz 2 (offset 1 line).
2 out of 3 hunks FAILED -- saving rejects to file src/lxc/confile

Version to patch to?

I'm trying to manually integrate those failed hunks now. Shouldn't be
too difficult for only three.

Regards,
Mike
--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121028/b900e802/attachment.pgp>

[Michael H. Warfield]

Post by Serge Hallyn

Post by Michael H. Warfield

Post by Michael H. Warfield

Post by Michael H. Warfield
/me erasing everything at this point and taking off the systemd crew,
since this will have no relevance to them...
Testing the hook feature out using git rev (finally got it built)...
I added this line to my config...
lxc.mount.entry=tmpfs /srv/lxc/private/Plover/dev.tmp tmpfs defaults 0 0
lxc.hook.mount = /var/lib/lxc/Plover/mount
--
rsync -avAH /srv/lxc/private/Plover/dev.template/. /srv/lxc/private/Plover/dev.tmp/
--
(This is just testing out the concepts.
If I understand this correctly, lxc.hook.pre-mount runs BEFORE the
mounting takes place and lxc.hook.mount takes place after the mount.
Problem is, the result of that rsync is not showing up in the mounted
tmpfs file system but is showing up in the underlying parent file system
as if it were run pre-mount. Something not right here...

I changed it to "lxc.hook.start = /srv/lxc/mount" (where I put the
script in the container) which then works but that then requires the
template and the command to be in the container. Suboptimal to say the
least. But it gives me a way to test this tmpfs thing out.
I also noticed that the .start hook runs, it appears, after caps are
dropped and I see a lot of bitching about mknod on certain devices. I
had to thrown an exit 0 into that script so it would continue in spite
of the errors but, now, I can refine my template based on what it could
create.

Crap. I've got a catch-22 here... This is going to take some work.

Hey,
I've got a rather minimal patch (appended below) to add the support for
mounting and populating a minimal /dev working. (A few hours were wasted
due to my not knowing that upstart was going to issue mounted-dev even though
/dev was mounted before upstart started - and the mounted-dev hook deletes
and recreates all consoles. GAH)

I am happy to report that, after manually editing my git head branch to
patch in the failed hunks, I was able to build it and test it and my
Fedora 17 systemd based container fired right up after adding the
lxc.autodev = 1 parameter to the config file. Yeah!!!!

I did run into one gotcha, but one I can live with. I had been bind
mounting the private root file system to another directory and then
using that as the rootfs like this:
--
lxc.rootfs = /srv/lxc/rootfs
lxc.mount.entry=/srv/lxc/private/Alcove /srv/lxc/rootfs none bind.shared 0 0
lxc.mount.entry=/home/shared /srv/lxc/private/Alcove/srv/shared none ro,bind 0 0

lxc.autodev = 1
--
This did not work and I got the startup error that it can not mount
to /dev because it doesn't exist.

I believe I can see why... You're doing the autodev populate prior to
any of the mounts being performed, so that "private" root file system is
not bound to the directory at that time.

Drop that bind mount for the rootfs and this then worked like a charm:
--
lxc.rootfs = /srv/lxc/private/Alcove
lxc.mount.entry=/home/shared /srv/lxc/private/Alcove/srv/shared none ro,bind 0 0

lxc.autodev = 1
--
I think that rootfs directory bind was an effort to more fully match the
OpenVZ behavior but also trying to deal with some of the read-only
problems were where having in the past with shutdowns. If it won't
work, it won't work and I won't miss it.

I did see some errors setting up that dev...
--
[root at forest mhw]# lxc-start -n Alcove
lxc-start: No such file or directory - failed to mount '/dev/pts/59'->'/usr/lib64/lxc/rootfs/dev/tty1'
lxc-start: No such file or directory - failed to mount '/dev/pts/60'->'/usr/lib64/lxc/rootfs/dev/tty2'
lxc-start: No such file or directory - failed to mount '/dev/pts/61'->'/usr/lib64/lxc/rootfs/dev/tty3'
lxc-start: No such file or directory - failed to mount '/dev/pts/62'->'/usr/lib64/lxc/rootfs/dev/tty4'
lxc-start: No such file or directory - failed to mount '/dev/pts/63'->'/usr/lib64/lxc/rootfs/dev/tty5'
lxc-start: No such file or directory - failed to mount '/dev/pts/64'->'/usr/lib64/lxc/rootfs/dev/tty6'
systemd 44 running in system mode. (+PAM +LIBWRAP +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP; fedora)

Welcome to Fedora 17 (Beefy Miracle)!
--
Not sure what that's all about but, since systemd isn't going to start
getty's on the tty? interfaces anyways, it probably doesn't make much
difference.

Regards,
Mike

Post by Serge Hallyn

Post by Michael H. Warfield
Yes, we can create the /dev directory with tmpfs from a template.
Problem is that /dev/pts does not exist at the time we need to mount the
devpts on /dev/pts for the pty's so that hurls chunks and dies. We
can't create the /dev/ directory contents prior to mounting in the
pre-mount hook because we won't have tmpfs in place at the time. We
have to get tmpfs mounted on /dev and then create /dev/pts and then
mount the ptys in there. There has to be a mkdir in between those two
mount actions. Simplest solution would seem to be to add some logic to
the mount logic that says "test if directory exists and, if not, create
it." I'm not sure of the consequences of that, though.
I don't see a way to make this happen with hooks. It's almost like we
need and on-mount per mount hook.

Should be moot given my patch, which I intend to push this week, but why
couldn't a lxc.hook.mount do the whole thing, mount /dev and and populate
it? I wasn't thinking a lxc.hook.start, for the reasons you encountered,
but I assume you tried lxc.hook.mount and it failed?
Index: lxc-qp/src/lxc/conf.c
===================================================================
--- lxc-qp.orig/src/lxc/conf.c 2012-10-27 17:24:50.768383000 -0500
+++ lxc-qp/src/lxc/conf.c 2012-10-28 05:44:07.871228322 -0500
@@ -619,7 +619,7 @@
}
if (mount(pty_info->name, lxcpath, "none", MS_BIND, 0)) {
- WARN("failed to mount '%s'->'%s'",
+ SYSERROR("failed to mount '%s'->'%s'",
pty_info->name, path);
continue;
}
@@ -636,7 +636,7 @@
}
} else {
if (mount(pty_info->name, path, "none", MS_BIND, 0)) {
- WARN("failed to mount '%s'->'%s'",
+ SYSERROR("failed to mount '%s'->'%s'",
pty_info->name, path);
continue;
}
@@ -842,6 +842,67 @@
return 0;
}
+struct lxc_devs {
+ char *name;
+ mode_t mode;
+ int maj;
+ int min;
+};
+
+struct lxc_devs lxc_devs[] = {
+ { "null", S_IFCHR | S_IRWXU | S_IRWXG | S_IRWXO, 1, 3 },
+ { "zero", S_IFCHR | S_IRWXU | S_IRWXG | S_IRWXO, 1, 5 },
+ { "full", S_IFCHR | S_IRWXU | S_IRWXG | S_IRWXO, 1, 7 },
+ { "urandom", S_IFCHR | S_IRWXU | S_IRWXG | S_IRWXO, 1, 9 },
+ { "random", S_IFCHR | S_IRWXU | S_IRWXG | S_IRWXO, 1, 8 },
+ { "tty", S_IFCHR | S_IRWXU | S_IRWXG | S_IRWXO, 5, 0 },
+ { "console", S_IFCHR | S_IRUSR | S_IWUSR, 5, 1 },
+};
+
+/*
+ * Do we want to add options for max size of /dev and a file to
+ * specify which devices to create?
+ */
+static int setup_autodev(char *root)
+{
+ int ret;
+ struct lxc_devs *d;
+ char path[MAXPATHLEN];
+ int i;
+
+ INFO("Creating and populating /dev under %s\n", root);
+ ret = snprintf(path, MAXPATHLEN, "%s/dev", root);
+ if (ret < 0 || ret > MAXPATHLEN)
+ return -1;
+ ret = mount("none", path, "tmpfs", 0, "size=100000");
+ if (ret) {
+ SYSERROR("Failed to mount /dev at %s\n", root);
+ return -1;
+ }
+ for (i = 0; i < sizeof(lxc_devs) / sizeof(lxc_devs[0]); i++) {
+ d = &lxc_devs[i];
+ ret = snprintf(path, MAXPATHLEN, "%s/dev/%s", root, d->name);
+ if (ret < 0 || ret >= MAXPATHLEN)
+ return -1;
+ ret = mknod(path, d->mode, makedev(d->maj, d->min));
+ if (ret) {
+ SYSERROR("Error creating %s\n", d->name);
+ return -1;
+ }
+ }
+ ret = snprintf(path, MAXPATHLEN, "%s/dev/pts", root);
+ if (ret < 0 || ret >= MAXPATHLEN)
+ return -1;
+ ret = mkdir(path, S_IRWXU | S_IRGRP | S_IXGRP | S_IROTH | S_IXOTH);
+ if (ret) {
+ SYSERROR("Failed to create /dev/pts in container");
+ return -1;
+ }
+
+ INFO("Populated /dev under %s\n", root);
+ return 0;
+}
+
static int setup_rootfs(const struct lxc_rootfs *rootfs)
{
if (!rootfs->path)
@@ -2208,6 +2269,13 @@
return -1;
}
+ if (lxc_conf->autodev) {
+ if (setup_autodev(lxc_conf->rootfs.mount)) {
+ ERROR("failed to set up /dev in the container");
+ return -1;
+ }
+ }
+
if (setup_mount(&lxc_conf->rootfs, lxc_conf->fstab, name)) {
ERROR("failed to setup the mounts for '%s'", name);
return -1;
Index: lxc-qp/src/lxc/conf.h
===================================================================
--- lxc-qp.orig/src/lxc/conf.h 2012-10-27 17:24:50.768383000 -0500
+++ lxc-qp/src/lxc/conf.h 2012-10-27 17:24:50.768383000 -0500
@@ -227,6 +227,7 @@
struct lxc_list hooks[NUM_LXC_HOOKS];
char *seccomp; // filename with the seccomp rules
int maincmd_fd;
+ int autodev; // if 1, mount and fill a /dev at start
};
int run_lxc_hooks(const char *name, char *hook, struct lxc_conf *conf);
Index: lxc-qp/src/lxc/confile.c
===================================================================
--- lxc-qp.orig/src/lxc/confile.c 2012-10-27 17:24:50.768383000 -0500
+++ lxc-qp/src/lxc/confile.c 2012-10-27 17:24:50.768383000 -0500
@@ -77,6 +77,7 @@
static int config_seccomp(const char *, char *, struct lxc_conf *);
static int config_includefile(const char *, char *, struct lxc_conf *);
static int config_network_nic(const char *, char *, struct lxc_conf *);
+static int config_autodev(const char *, char *, struct lxc_conf *);
typedef int (*config_cb)(const char *, char *, struct lxc_conf *);
@@ -118,6 +119,7 @@
{ "lxc.console", config_console },
{ "lxc.seccomp", config_seccomp },
{ "lxc.include", config_includefile },
+ { "lxc.autodev", config_autodev },
};
static const size_t config_size = sizeof(config)/sizeof(struct lxc_config_t);
@@ -853,6 +855,16 @@
return 0;
}
+
+static int config_autodev(const char *key, char *value,
+ struct lxc_conf *lxc_conf)
+{
+ int v = atoi(value);
+
+ lxc_conf->autodev = v;
+
+ return 0;
+}
static int config_aa_profile(const char *key, char *value,
struct lxc_conf *lxc_conf)
------------------------------------------------------------------------------
WINDOWS 8 is here.
Millions of people. Your app in 30 days.
Visit The Windows 8 Center at Sourceforge for all your go to resources.
http://windows8center.sourceforge.net/
join-generation-app-and-make-money-coding-fast/
_______________________________________________
Lxc-devel mailing list
Lxc-devel at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lxc-devel

--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121028/0e803b9a/attachment.pgp>

[Serge Hallyn]

Post by Michael H. Warfield

Post by Serge Hallyn

Post by Michael H. Warfield

Post by Michael H. Warfield

Post by Michael H. Warfield
/me erasing everything at this point and taking off the systemd crew,
since this will have no relevance to them...
Testing the hook feature out using git rev (finally got it built)...
I added this line to my config...
lxc.mount.entry=tmpfs /srv/lxc/private/Plover/dev.tmp tmpfs defaults 0 0
lxc.hook.mount = /var/lib/lxc/Plover/mount
--
rsync -avAH /srv/lxc/private/Plover/dev.template/. /srv/lxc/private/Plover/dev.tmp/
--
(This is just testing out the concepts.
If I understand this correctly, lxc.hook.pre-mount runs BEFORE the
mounting takes place and lxc.hook.mount takes place after the mount.
Problem is, the result of that rsync is not showing up in the mounted
tmpfs file system but is showing up in the underlying parent file system
as if it were run pre-mount. Something not right here...

I changed it to "lxc.hook.start = /srv/lxc/mount" (where I put the
script in the container) which then works but that then requires the
template and the command to be in the container. Suboptimal to say the
least. But it gives me a way to test this tmpfs thing out.
I also noticed that the .start hook runs, it appears, after caps are
dropped and I see a lot of bitching about mknod on certain devices. I
had to thrown an exit 0 into that script so it would continue in spite
of the errors but, now, I can refine my template based on what it could
create.

Crap. I've got a catch-22 here... This is going to take some work.

Hey,
I've got a rather minimal patch (appended below) to add the support for
mounting and populating a minimal /dev working. (A few hours were wasted
due to my not knowing that upstart was going to issue mounted-dev even though
/dev was mounted before upstart started - and the mounted-dev hook deletes
and recreates all consoles. GAH)

I am happy to report that, after manually editing my git head branch to

Sorry, it was against the ubuntu quantal package. I've been in the air
without onboard wifi, so working with what i had at hand.

Post by Michael H. Warfield
patch in the failed hunks, I was able to build it and test it and my
Fedora 17 systemd based container fired right up after adding the
lxc.autodev = 1 parameter to the config file. Yeah!!!!
I did run into one gotcha, but one I can live with. I had been bind
mounting the private root file system to another directory and then
--
lxc.rootfs = /srv/lxc/rootfs
lxc.mount.entry=/srv/lxc/private/Alcove /srv/lxc/rootfs none bind.shared 0 0
lxc.mount.entry=/home/shared /srv/lxc/private/Alcove/srv/shared none ro,bind 0 0
lxc.autodev = 1
--
This did not work and I got the startup error that it can not mount
to /dev because it doesn't exist.

Hm, yeah. If you do need to play a game like this, you might be best
off using a pre-mount hook for that.

Post by Michael H. Warfield
I believe I can see why... You're doing the autodev populate prior to
any of the mounts being performed, so that "private" root file system is
not bound to the directory at that time.
--
lxc.rootfs = /srv/lxc/private/Alcove
lxc.mount.entry=/home/shared /srv/lxc/private/Alcove/srv/shared none ro,bind 0 0
lxc.autodev = 1
--
I think that rootfs directory bind was an effort to more fully match the
OpenVZ behavior but also trying to deal with some of the read-only
problems were where having in the past with shutdowns. If it won't
work, it won't work and I won't miss it.
I did see some errors setting up that dev...
--
[root at forest mhw]# lxc-start -n Alcove
lxc-start: No such file or directory - failed to mount '/dev/pts/59'->'/usr/lib64/lxc/rootfs/dev/tty1'
lxc-start: No such file or directory - failed to mount '/dev/pts/60'->'/usr/lib64/lxc/rootfs/dev/tty2'
lxc-start: No such file or directory - failed to mount '/dev/pts/61'->'/usr/lib64/lxc/rootfs/dev/tty3'
lxc-start: No such file or directory - failed to mount '/dev/pts/62'->'/usr/lib64/lxc/rootfs/dev/tty4'
lxc-start: No such file or directory - failed to mount '/dev/pts/63'->'/usr/lib64/lxc/rootfs/dev/tty5'
lxc-start: No such file or directory - failed to mount '/dev/pts/64'->'/usr/lib64/lxc/rootfs/dev/tty6'
systemd 44 running in system mode. (+PAM +LIBWRAP +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP; fedora)
Welcome to Fedora 17 (Beefy Miracle)!
--
Not sure what that's all about but, since systemd isn't going to start
getty's on the tty? interfaces anyways, it probably doesn't make much
difference.

Oh, I see. Yeah, in the !lxc.ttydir case, when we created our own /dev
we should create the tty files. I need to fix that.

Of course in your case since systemd isn't going to start getty's on
them, you should not have the lxc.tty = 6 in your container config,
which it looks like you still do?

[Michael H. Warfield]

Post by Serge Hallyn

Post by Michael H. Warfield

Post by Serge Hallyn
I've got a rather minimal patch (appended below) to add the support for
mounting and populating a minimal /dev working. (A few hours were wasted
due to my not knowing that upstart was going to issue mounted-dev even though
/dev was mounted before upstart started - and the mounted-dev hook deletes
and recreates all consoles. GAH)

I am happy to report that, after manually editing my git head branch to

Sorry, it was against the ubuntu quantal package. I've been in the air
without onboard wifi, so working with what i had at hand.

Oh, I figured it was a package mismatch. Wasn't too terribly difficult
to patch those hunks in and kick out a diff against git.

Post by Serge Hallyn

Post by Michael H. Warfield
patch in the failed hunks, I was able to build it and test it and my
Fedora 17 systemd based container fired right up after adding the
lxc.autodev = 1 parameter to the config file. Yeah!!!!
I did run into one gotcha, but one I can live with. I had been bind
mounting the private root file system to another directory and then
--
lxc.rootfs = /srv/lxc/rootfs
lxc.mount.entry=/srv/lxc/private/Alcove /srv/lxc/rootfs none bind.shared 0 0
lxc.mount.entry=/home/shared /srv/lxc/private/Alcove/srv/shared none ro,bind 0 0
lxc.autodev = 1
--
This did not work and I got the startup error that it can not mount
to /dev because it doesn't exist.

Hm, yeah. If you do need to play a game like this, you might be best
off using a pre-mount hook for that.

Yeah, I don't think I "need to play a game like this" anymore. I'd have
to go back through some old old E-Mails to see why I did that before. I
seem to recall we were playing with all sorts of bind mount options for
some PRIVATE thing or another. It may not be necessary at all any
longer. IAC, it's minor to switch it back. I seem to recall switching
back and forth using bind mounts several times back when that got done
that way.

I may play with the pre-mount hook just for giggles and see how that
might work as well. Any idea why I was experiencing the problem with
the mount hook when trying to populate /dev? I know it wouldn't have
worked because of the /dev/pts mount but I have more heartburn in that
it looks like it ran too early and the mount on /dev had not even taken
place at that time.

Post by Serge Hallyn

Post by Michael H. Warfield
I believe I can see why... You're doing the autodev populate prior to
any of the mounts being performed, so that "private" root file system is
not bound to the directory at that time.
--
lxc.rootfs = /srv/lxc/private/Alcove
lxc.mount.entry=/home/shared /srv/lxc/private/Alcove/srv/shared none ro,bind 0 0
lxc.autodev = 1
--
I think that rootfs directory bind was an effort to more fully match the
OpenVZ behavior but also trying to deal with some of the read-only
problems were where having in the past with shutdowns. If it won't
work, it won't work and I won't miss it.
I did see some errors setting up that dev...
--
[root at forest mhw]# lxc-start -n Alcove
lxc-start: No such file or directory - failed to mount '/dev/pts/59'->'/usr/lib64/lxc/rootfs/dev/tty1'
lxc-start: No such file or directory - failed to mount '/dev/pts/60'->'/usr/lib64/lxc/rootfs/dev/tty2'
lxc-start: No such file or directory - failed to mount '/dev/pts/61'->'/usr/lib64/lxc/rootfs/dev/tty3'
lxc-start: No such file or directory - failed to mount '/dev/pts/62'->'/usr/lib64/lxc/rootfs/dev/tty4'
lxc-start: No such file or directory - failed to mount '/dev/pts/63'->'/usr/lib64/lxc/rootfs/dev/tty5'
lxc-start: No such file or directory - failed to mount '/dev/pts/64'->'/usr/lib64/lxc/rootfs/dev/tty6'
systemd 44 running in system mode. (+PAM +LIBWRAP +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP; fedora)
Welcome to Fedora 17 (Beefy Miracle)!
--
Not sure what that's all about but, since systemd isn't going to start
getty's on the tty? interfaces anyways, it probably doesn't make much
difference.

Oh, I see. Yeah, in the !lxc.ttydir case, when we created our own /dev
we should create the tty files. I need to fix that.

Cool. Once again... Looks like we got some real progress here with
this one. I've still got more testing to do, undoing some of my changes
in the container itself and making sure it all still works.

Also looks like I can stop and restart one of these containers now
without the hung cgroup directory. I suspected it was some stray
devices behind that. This is good.

Post by Serge Hallyn
Of course in your case since systemd isn't going to start getty's on
them, you should not have the lxc.tty = 6 in your container config,
which it looks like you still do?

Yeah. I was taking it one step at a time. I wish they WOULD fire up
some getty's on those tty's since that basically makes lxc-console kinda
useless and the one on /dev/console is kinda useless in disconnected
mode with the console redirected to a file. Maybe we need some what for
lxc-console to be able to grab that? I'll have to look deeper at
systemd and figure out if it can be parameterized or conditionalized in
some way. ITMT, I probably will just turn them off.

Many thanks!

Regards,
Mike
--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121028/d93e415b/attachment.pgp>

[Serge Hallyn]

Quoting Michael H. Warfield (mhw at WittsEnd.com):
...

Post by Michael H. Warfield
Yeah, I don't think I "need to play a game like this" anymore. I'd have
to go back through some old old E-Mails to see why I did that before. I
seem to recall we were playing with all sorts of bind mount options for
some PRIVATE thing or another. It may not be necessary at all any
longer. IAC, it's minor to switch it back. I seem to recall switching
back and forth using bind mounts several times back when that got done
that way.
I may play with the pre-mount hook just for giggles and see how that
might work as well. Any idea why I was experiencing the problem with
the mount hook when trying to populate /dev? I know it wouldn't have

The only idea I have is that perhaps your root is MS_SHARED by default?
Can you post the script you were using and the container config?

Post by Michael H. Warfield
worked because of the /dev/pts mount but I have more heartburn in that
it looks like it ran too early and the mount on /dev had not even taken
place at that time.

Post by Serge Hallyn

Post by Michael H. Warfield
I believe I can see why... You're doing the autodev populate prior to
any of the mounts being performed, so that "private" root file system is
not bound to the directory at that time.
--
lxc.rootfs = /srv/lxc/private/Alcove
lxc.mount.entry=/home/shared /srv/lxc/private/Alcove/srv/shared none ro,bind 0 0
lxc.autodev = 1
--
I think that rootfs directory bind was an effort to more fully match the
OpenVZ behavior but also trying to deal with some of the read-only
problems were where having in the past with shutdowns. If it won't
work, it won't work and I won't miss it.
I did see some errors setting up that dev...
--
[root at forest mhw]# lxc-start -n Alcove
lxc-start: No such file or directory - failed to mount '/dev/pts/59'->'/usr/lib64/lxc/rootfs/dev/tty1'
lxc-start: No such file or directory - failed to mount '/dev/pts/60'->'/usr/lib64/lxc/rootfs/dev/tty2'
lxc-start: No such file or directory - failed to mount '/dev/pts/61'->'/usr/lib64/lxc/rootfs/dev/tty3'
lxc-start: No such file or directory - failed to mount '/dev/pts/62'->'/usr/lib64/lxc/rootfs/dev/tty4'
lxc-start: No such file or directory - failed to mount '/dev/pts/63'->'/usr/lib64/lxc/rootfs/dev/tty5'
lxc-start: No such file or directory - failed to mount '/dev/pts/64'->'/usr/lib64/lxc/rootfs/dev/tty6'
systemd 44 running in system mode. (+PAM +LIBWRAP +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP; fedora)
Welcome to Fedora 17 (Beefy Miracle)!
--
Not sure what that's all about but, since systemd isn't going to start
getty's on the tty? interfaces anyways, it probably doesn't make much
difference.

Oh, I see. Yeah, in the !lxc.ttydir case, when we created our own /dev
we should create the tty files. I need to fix that.

Cool. Once again... Looks like we got some real progress here with
this one. I've still got more testing to do, undoing some of my changes
in the container itself and making sure it all still works.
Also looks like I can stop and restart one of these containers now
without the hung cgroup directory. I suspected it was some stray
devices behind that. This is good.

Post by Serge Hallyn
Of course in your case since systemd isn't going to start getty's on
them, you should not have the lxc.tty = 6 in your container config,
which it looks like you still do?

Yeah. I was taking it one step at a time. I wish they WOULD fire up
some getty's on those tty's since that basically makes lxc-console kinda

As I recall, you can specify gettys to start on any tty by creating a
magical symlink, something like

Post by Michael H. Warfield
useless and the one on /dev/console is kinda useless in disconnected
mode with the console redirected to a file. Maybe we need some what for
lxc-console to be able to grab that? I'll have to look deeper at
systemd and figure out if it can be parameterized or conditionalized in
some way. ITMT, I probably will just turn them off.
Many thanks!
Regards,
Mike
--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!

[Michael H. Warfield]

Post by Serge Hallyn
...

Post by Michael H. Warfield
Yeah, I don't think I "need to play a game like this" anymore. I'd have
to go back through some old old E-Mails to see why I did that before. I
seem to recall we were playing with all sorts of bind mount options for
some PRIVATE thing or another. It may not be necessary at all any
longer. IAC, it's minor to switch it back. I seem to recall switching
back and forth using bind mounts several times back when that got done
that way.
I may play with the pre-mount hook just for giggles and see how that
might work as well. Any idea why I was experiencing the problem with
the mount hook when trying to populate /dev? I know it wouldn't have

The only idea I have is that perhaps your root is MS_SHARED by default?
Can you post the script you were using and the container config?

Another point on the curve... The documentation says that "pre-mount"
takes place before the mount occurs and "mount takes place after the
mount occurs. Only problem is that "pre-mount" is not being recognized.

lxc-start 1351627853.032 ERROR lxc_confile - unknown key lxc.hook.pre-mount

This is the same binaries from git that recognize lxc.hook.mount so I'm
assuming the doco and the code don't match at this point.

Even without my original bind mount, if I have a mount hook that does
something in a newly mounted tmpfs directory, it doesn't show up in that
directory it shows up in the parent directory as if it ran before the
mounts took place. I could put the mount in the hook.mount script and
then do it but it's seriously acting like the pre-mount hook isn't even
there (parameter unknown) and the mount hook is running before the
mounts are complete.

Simple exerts from some test scripts doing, really, nothing but testing
sequencing...

Ok... Lets try this. I won't post entire configs but... For machine
Alcove (my testbed)...
--
lxc.tty = 6
lxc.pts = 64

lxc.rootfs = /srv/lxc/private/Alcove
lxc.mount.entry=none /srv/lxc/private/Alcove/dev.tmp tmpfs defaults 0 0
lxc.mount.entry=/home/shared /srv/lxc/private/Alcove/srv/shared none ro,bind 0 0

lxc.autodev = 1

# lxc.hook.pre-mount = /var/lib/lxc/Alcove/pre-mount
lxc.hook.mount = /var/lib/lxc/Alcove/mount

lxc.mount.entry=shmfs /var/lib/lxc/private/Alcove/dev/shm tmpfs mode=0644 0 0
--

Now /var/lib/lxc/Alcove/mount:

--

#!/bin/sh -

touch /srv/lxc/private/Alcove/dev.tmp/mounted

--

In that directory on the host fs I have this:

[root at forest mhw]# touch /srv/lxc/private/Alcove/dev.tmp/no-mounted
[root at forest mhw]# ls /srv/lxc/private/Alcove/dev.tmp/
no-mounted

Now, when I start the container, the tmpfs should get mounted
on /dev.tmp in the container (relative to the container rootfs) and
should have the single file "mounted" in it while the parent file system
back on the host should have the single file "not-mounted" in it.

Let's see... lxc-start -n Alcove...

In the container...

[mhw at alcove ~]$ mount | grep tmpfs
none on /dev type tmpfs (rw,relatime,seclabel,size=100k)
none on /dev.tmp type tmpfs (rw,relatime,seclabel)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,seclabel,mode=755)
tmpfs on /media type tmpfs (rw,nosuid,nodev,noexec,relatime,seclabel,mode=755)

Looks like the mount took place. I have tmpfs on /dev.tmp

[mhw at alcove ~]$ ls /dev.tmp/
[mhw at alcove ~]$

Opps... Where did the file end up?

Let's look on the host...

[mhw at forest ~]$ ls /srv/lxc/private/Alcove/dev.tmp/
mounted no-mounted

Arg... Wrong answer. It ended up in the parent file system before
tmpfs was mounted. But, the documentation says hook.mount runs after
the mounts have completed.

There's something wrong here or I am badly mistaken in my
understanding... (Probably the later, I'll admit.)

Regards,
Mike

Post by Serge Hallyn

Post by Michael H. Warfield
worked because of the /dev/pts mount but I have more heartburn in that
it looks like it ran too early and the mount on /dev had not even taken
place at that time.

Post by Serge Hallyn

Post by Michael H. Warfield
I believe I can see why... You're doing the autodev populate prior to
any of the mounts being performed, so that "private" root file system is
not bound to the directory at that time.
--
lxc.rootfs = /srv/lxc/private/Alcove
lxc.mount.entry=/home/shared /srv/lxc/private/Alcove/srv/shared none ro,bind 0 0
lxc.autodev = 1
--
I think that rootfs directory bind was an effort to more fully match the
OpenVZ behavior but also trying to deal with some of the read-only
problems were where having in the past with shutdowns. If it won't
work, it won't work and I won't miss it.
I did see some errors setting up that dev...
--
[root at forest mhw]# lxc-start -n Alcove
lxc-start: No such file or directory - failed to mount '/dev/pts/59'->'/usr/lib64/lxc/rootfs/dev/tty1'
lxc-start: No such file or directory - failed to mount '/dev/pts/60'->'/usr/lib64/lxc/rootfs/dev/tty2'
lxc-start: No such file or directory - failed to mount '/dev/pts/61'->'/usr/lib64/lxc/rootfs/dev/tty3'
lxc-start: No such file or directory - failed to mount '/dev/pts/62'->'/usr/lib64/lxc/rootfs/dev/tty4'
lxc-start: No such file or directory - failed to mount '/dev/pts/63'->'/usr/lib64/lxc/rootfs/dev/tty5'
lxc-start: No such file or directory - failed to mount '/dev/pts/64'->'/usr/lib64/lxc/rootfs/dev/tty6'
systemd 44 running in system mode. (+PAM +LIBWRAP +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP; fedora)
Welcome to Fedora 17 (Beefy Miracle)!
--
Not sure what that's all about but, since systemd isn't going to start
getty's on the tty? interfaces anyways, it probably doesn't make much
difference.

Oh, I see. Yeah, in the !lxc.ttydir case, when we created our own /dev
we should create the tty files. I need to fix that.

Cool. Once again... Looks like we got some real progress here with
this one. I've still got more testing to do, undoing some of my changes
in the container itself and making sure it all still works.
Also looks like I can stop and restart one of these containers now
without the hung cgroup directory. I suspected it was some stray
devices behind that. This is good.

Post by Serge Hallyn
Of course in your case since systemd isn't going to start getty's on
them, you should not have the lxc.tty = 6 in your container config,
which it looks like you still do?

Yeah. I was taking it one step at a time. I wish they WOULD fire up
some getty's on those tty's since that basically makes lxc-console kinda

As I recall, you can specify gettys to start on any tty by creating a
magical symlink, something like

Post by Michael H. Warfield
useless and the one on /dev/console is kinda useless in disconnected
mode with the console redirected to a file. Maybe we need some what for
lxc-console to be able to grab that? I'll have to look deeper at
systemd and figure out if it can be parameterized or conditionalized in
some way. ITMT, I probably will just turn them off.
Many thanks!
Regards,
Mike
--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!

--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121030/617a07e5/attachment.pgp>

[Serge Hallyn]

Can you tell me the exact git tree and branch you are using?

The results you're getting don't make sense to me... Hoping I can find
a simple answer.

-serge

[Michael H. Warfield]

Post by Serge Hallyn
Can you tell me the exact git tree and branch you are using?

I'm using head. I'm not specifying a tree.

Post by Serge Hallyn
The results you're getting don't make sense to me... Hoping I can find
a simple answer.

Me too.

Post by Serge Hallyn
-serge

Regards,
Mike
--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121031/9b34ef70/attachment.pgp>

[Serge Hallyn]

Post by Michael H. Warfield

Post by Serge Hallyn
Can you tell me the exact git tree and branch you are using?

I'm using head. I'm not specifying a tree.

?

I'm not sure what you mean - are you using git://github.com/lxc/lxc, or
the tree on lxc.sf.net?

Post by Michael H. Warfield

Post by Serge Hallyn
The results you're getting don't make sense to me... Hoping I can find
a simple answer.

Me too.

Post by Serge Hallyn
-serge

Regards,
Mike
--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!

[Michael H. Warfield]

Post by Serge Hallyn

Post by Michael H. Warfield

Post by Serge Hallyn
Can you tell me the exact git tree and branch you are using?

I'm using head. I'm not specifying a tree.

?
I'm not sure what you mean - are you using git://github.com/lxc/lxc, or
the tree on lxc.sf.net?

IOW, I'm not using a branch in the tree. I'm using the main trunk.

Created my tree with -

git clone git://github.com/lxc/lxc

So the former, not the later.

Post by Serge Hallyn

Post by Michael H. Warfield

Post by Serge Hallyn
The results you're getting don't make sense to me... Hoping I can find
a simple answer.

Me too.

Post by Serge Hallyn
-serge

Mike
--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121031/a24a4f72/attachment.pgp>

[Serge Hallyn]

Post by Michael H. Warfield

Post by Serge Hallyn
...

Post by Michael H. Warfield
Yeah, I don't think I "need to play a game like this" anymore. I'd have
to go back through some old old E-Mails to see why I did that before. I
seem to recall we were playing with all sorts of bind mount options for
some PRIVATE thing or another. It may not be necessary at all any
longer. IAC, it's minor to switch it back. I seem to recall switching
back and forth using bind mounts several times back when that got done
that way.
I may play with the pre-mount hook just for giggles and see how that
might work as well. Any idea why I was experiencing the problem with
the mount hook when trying to populate /dev? I know it wouldn't have

The only idea I have is that perhaps your root is MS_SHARED by default?
Can you post the script you were using and the container config?

Another point on the curve... The documentation says that "pre-mount"
takes place before the mount occurs and "mount takes place after the
mount occurs. Only problem is that "pre-mount" is not being recognized.

I have
serge at serge-ThinkPad-X130e:~$ cat /var/lib/lxc/premount
#!/bin/bash
echo 'hi there' > /tmp/hw
serge at serge-ThinkPad-X130e:~$ head -2 /var/lib/lxc/q4/config
lxc.hook.pre-mount = /var/lib/lxc/premount
lxc.network.type=veth
serge at serge-ThinkPad-X130e:~$ sudo lxc-start -n q4 -d
serge at serge-ThinkPad-X130e:~$ cat /tmp/hw
hi there

(This is using lxc 0.8.0~rc1-4ubuntu39 in ubuntu quantal)

-serge

[Michael H. Warfield]

Post by Serge Hallyn

Post by Michael H. Warfield
I did see some errors setting up that dev...
--
[root at forest mhw]# lxc-start -n Alcove
lxc-start: No such file or directory - failed to mount '/dev/pts/59'->'/usr/lib64/lxc/rootfs/dev/tty1'
lxc-start: No such file or directory - failed to mount '/dev/pts/60'->'/usr/lib64/lxc/rootfs/dev/tty2'
lxc-start: No such file or directory - failed to mount '/dev/pts/61'->'/usr/lib64/lxc/rootfs/dev/tty3'
lxc-start: No such file or directory - failed to mount '/dev/pts/62'->'/usr/lib64/lxc/rootfs/dev/tty4'
lxc-start: No such file or directory - failed to mount '/dev/pts/63'->'/usr/lib64/lxc/rootfs/dev/tty5'
lxc-start: No such file or directory - failed to mount '/dev/pts/64'->'/usr/lib64/lxc/rootfs/dev/tty6'
systemd 44 running in system mode. (+PAM +LIBWRAP +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP; fedora)
Welcome to Fedora 17 (Beefy Miracle)!
--
Not sure what that's all about but, since systemd isn't going to start
getty's on the tty? interfaces anyways, it probably doesn't make much
difference.

Oh, I see. Yeah, in the !lxc.ttydir case, when we created our own /dev
we should create the tty files. I need to fix that.

Well... I'm not sure I understand what you mean by that.
The /dev/pts/* entries do does exist in the host system. But the bind
mount fails saying they do not exist. Not sure I understand what the
problem is here but I would like them connected even if systemd doesn't
start getty's on them. I've used them for other purposes in the past.

Post by Serge Hallyn
Of course in your case since systemd isn't going to start getty's on
them, you should not have the lxc.tty = 6 in your container config,
which it looks like you still do?

Actually, I've decided this is worthy of debugging and there may be
other ways to start a getty (or something else) on that tty. It really
should work.

Regards,
Mike
--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121030/76cfb0a5/attachment.pgp>

[Serge Hallyn]

Post by Michael H. Warfield

Post by Serge Hallyn

Post by Michael H. Warfield
I did see some errors setting up that dev...
--
[root at forest mhw]# lxc-start -n Alcove
lxc-start: No such file or directory - failed to mount '/dev/pts/59'->'/usr/lib64/lxc/rootfs/dev/tty1'
lxc-start: No such file or directory - failed to mount '/dev/pts/60'->'/usr/lib64/lxc/rootfs/dev/tty2'
lxc-start: No such file or directory - failed to mount '/dev/pts/61'->'/usr/lib64/lxc/rootfs/dev/tty3'
lxc-start: No such file or directory - failed to mount '/dev/pts/62'->'/usr/lib64/lxc/rootfs/dev/tty4'
lxc-start: No such file or directory - failed to mount '/dev/pts/63'->'/usr/lib64/lxc/rootfs/dev/tty5'
lxc-start: No such file or directory - failed to mount '/dev/pts/64'->'/usr/lib64/lxc/rootfs/dev/tty6'
systemd 44 running in system mode. (+PAM +LIBWRAP +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP; fedora)
Welcome to Fedora 17 (Beefy Miracle)!
--
Not sure what that's all about but, since systemd isn't going to start
getty's on the tty? interfaces anyways, it probably doesn't make much
difference.

Oh, I see. Yeah, in the !lxc.ttydir case, when we created our own /dev
we should create the tty files. I need to fix that.

Well... I'm not sure I understand what you mean by that.
The /dev/pts/* entries do does exist in the host system. But the bind
mount fails saying they do not exist. Not sure I understand what the

In Ubuntu, we use lxc.ttydir = lxc. That means the actual /dev/ttyN in the
container is a symlink to /dev/lxc/ttyN (to allow package upgrades which
insist on removing /dev/ttyN to succeed - they will fail if /dev/ttyN is
mounted over).

When /dev was pre-populated, /dev/ttyN existed. But when we are populating
it, it does not. So before we try tomount /dev/pts/NN from the host onto
/dev/ttyN in the container, we have to create a file to bind mount over.
I didn't put that in the patch. Yet.

Post by Michael H. Warfield
problem is here but I would like them connected even if systemd doesn't
start getty's on them. I've used them for other purposes in the past.

Post by Serge Hallyn
Of course in your case since systemd isn't going to start getty's on
them, you should not have the lxc.tty = 6 in your container config,
which it looks like you still do?

Actually, I've decided this is worthy of debugging and there may be
other ways to start a getty (or something else) on that tty. It really
should work.
Regards,
Mike
--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________
Lxc-users mailing list
Lxc-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lxc-users

[Michael H. Warfield]

Post by Serge Hallyn

Post by Michael H. Warfield

Post by Serge Hallyn

Post by Michael H. Warfield
I did see some errors setting up that dev...
--
[root at forest mhw]# lxc-start -n Alcove
lxc-start: No such file or directory - failed to mount '/dev/pts/59'->'/usr/lib64/lxc/rootfs/dev/tty1'
lxc-start: No such file or directory - failed to mount '/dev/pts/60'->'/usr/lib64/lxc/rootfs/dev/tty2'
lxc-start: No such file or directory - failed to mount '/dev/pts/61'->'/usr/lib64/lxc/rootfs/dev/tty3'
lxc-start: No such file or directory - failed to mount '/dev/pts/62'->'/usr/lib64/lxc/rootfs/dev/tty4'
lxc-start: No such file or directory - failed to mount '/dev/pts/63'->'/usr/lib64/lxc/rootfs/dev/tty5'
lxc-start: No such file or directory - failed to mount '/dev/pts/64'->'/usr/lib64/lxc/rootfs/dev/tty6'
systemd 44 running in system mode. (+PAM +LIBWRAP +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP; fedora)
Welcome to Fedora 17 (Beefy Miracle)!
--
Not sure what that's all about but, since systemd isn't going to start
getty's on the tty? interfaces anyways, it probably doesn't make much
difference.

Oh, I see. Yeah, in the !lxc.ttydir case, when we created our own /dev
we should create the tty files. I need to fix that.

Well... I'm not sure I understand what you mean by that.
The /dev/pts/* entries do does exist in the host system. But the bind
mount fails saying they do not exist. Not sure I understand what the

In Ubuntu, we use lxc.ttydir = lxc. That means the actual /dev/ttyN in the
container is a symlink to /dev/lxc/ttyN (to allow package upgrades which
insist on removing /dev/ttyN to succeed - they will fail if /dev/ttyN is
mounted over).
When /dev was pre-populated, /dev/ttyN existed. But when we are populating
it, it does not. So before we try tomount /dev/pts/NN from the host onto
/dev/ttyN in the container, we have to create a file to bind mount over.
I didn't put that in the patch. Yet.

Got it! You and I both did the same thing. I'm thinking from the view
of Fedora and you're thinking from the view of Ubuntu. Got it. We'll
get this right yet. :-)=)

Post by Serge Hallyn

Post by Michael H. Warfield
problem is here but I would like them connected even if systemd doesn't
start getty's on them. I've used them for other purposes in the past.

Post by Serge Hallyn
Of course in your case since systemd isn't going to start getty's on
them, you should not have the lxc.tty = 6 in your container config,
which it looks like you still do?

Actually, I've decided this is worthy of debugging and there may be
other ways to start a getty (or something else) on that tty. It really
should work.
Regards,
Mike
--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________
Lxc-users mailing list
Lxc-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lxc-users

--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121030/3132ab24/attachment.pgp>

[Serge Hallyn]

Post by Serge Hallyn

Post by Michael H. Warfield

Post by Michael H. Warfield
Ok... I've done some cursory search and turned up nothing but some
comments about "pre mount hooks". Where is the documentation about this
feature and how I might use / implement it? Some examples would
probably suffice. Is there a require release version of lxc-utils?

http://www.mail-archive.com/lxc-devel at lists.sourceforge.net/msg01490.html
I'll play with it and report back.

Also the "Lifecycle management hooks" section in
https://help.ubuntu.com/12.10/serverguide/lxc.html
Note that I'm thinking that having lxc-start guess how to fill in /dev
is wrong, because different distros and even different releases of the
same distros have different expectations. For instance ubuntu lucid
wants /dev/shm to be a directory, while precise+ wants a symlink. So
somehow the template should get involved, be it by adding a hook, or
simply specifying a configuration file which lxc uses internally to
decide how to create /dev.

/dev/shm can be created/mounted/symlinked by the OS in the
container. This is nothing LXC should care about.
My recommendation for LXC would be to unconditionally pre-mount /dev as
tmpfs, and add exactly the device nodes /dev/null, /dev/zero, /dev/full,
/dev/urandom, /dev/random, /dev/tty, /dev/ptmx to it. That is the
minimal set you need to boot a machine. All further
submounts/symlinks/dirs can be created by the OS boot logic in the
container.

I'm thinking we'll do that, optionally. Templates (including fedora
and ubuntu) can simply always set the option to mount and fill /dev.
Others (like busybox and mini-sshd) won't.

http://www.freedesktop.org/wiki/Software/systemd/ContainerInterface
It would be good if LXC would do the same in order to minimize the
manual user configuration necessary.
Lennart

Agreed it simplifies things for full system containers with modern distros.

thanks,
-serge

[Michael H. Warfield]

Post by Michael H. Warfield
I've got some more problems relating to shutting down containers, some
of which may be related to mounting tmpfs on /run to which /var/run is
symlinked to. We're doing halt / restart detection by monitoring utmp
in that directory but it looks like utmp isn't even in that directory
anymore and mounting tmpfs on it was always problematical. We may have
to have a more generic method to detect when a container has shut down
or is restarting in that case.

I can't parse this. The system call reboot() is virtualized for
containers just fine and the container managaer (i.e. LXC) can check for
that easily.

The problem we have had was with differentiating between reboot and halt
to either shut the container down cold or restarted it. You say
"easily" and yet we never came up with an "easy" solution and monitored
utmp instead for the next runlevel change. What is your "easy" solution
for that problem?

Lennart
--
Lennart Poettering - Red Hat, Inc.

Regards,
Mike
--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121025/280a7725/attachment.pgp>

[Serge Hallyn]

Post by Michael H. Warfield

Post by Michael H. Warfield
I've got some more problems relating to shutting down containers, some
of which may be related to mounting tmpfs on /run to which /var/run is
symlinked to. We're doing halt / restart detection by monitoring utmp
in that directory but it looks like utmp isn't even in that directory
anymore and mounting tmpfs on it was always problematical. We may have
to have a more generic method to detect when a container has shut down
or is restarting in that case.

I can't parse this. The system call reboot() is virtualized for
containers just fine and the container managaer (i.e. LXC) can check for
that easily.

The problem we have had was with differentiating between reboot and halt
to either shut the container down cold or restarted it. You say
"easily" and yet we never came up with an "easy" solution and monitored
utmp instead for the next runlevel change. What is your "easy" solution
for that problem?

I think you're on older kernels, where we had to resort to that. Pretty
recently Daniel Lezcano's patch was finally accepted upstream, which lets
a container call reboot() and lets the parent of init tell whether it
called reboot or shutdown by looking at wTERMSIG(status).

-serge

[Michael H. Warfield]

Post by Serge Hallyn

Post by Michael H. Warfield

Post by Michael H. Warfield
I've got some more problems relating to shutting down containers, some
of which may be related to mounting tmpfs on /run to which /var/run is
symlinked to. We're doing halt / restart detection by monitoring utmp
in that directory but it looks like utmp isn't even in that directory
anymore and mounting tmpfs on it was always problematical. We may have
to have a more generic method to detect when a container has shut down
or is restarting in that case.

I can't parse this. The system call reboot() is virtualized for
containers just fine and the container managaer (i.e. LXC) can check for
that easily.

The problem we have had was with differentiating between reboot and halt
to either shut the container down cold or restarted it. You say
"easily" and yet we never came up with an "easy" solution and monitored
utmp instead for the next runlevel change. What is your "easy" solution
for that problem?

I think you're on older kernels, where we had to resort to that. Pretty
recently Daniel Lezcano's patch was finally accepted upstream, which lets
a container call reboot() and lets the parent of init tell whether it
called reboot or shutdown by looking at wTERMSIG(status).

Now THAT is wonderful news! I hadn't realized that had been accepted.
So we no longer need to rely on the old utmp kludge?

Post by Serge Hallyn
-serge

Regards,
Mike
--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121025/54137328/attachment.pgp>

[Serge Hallyn]

Post by Michael H. Warfield

Post by Serge Hallyn

Post by Michael H. Warfield

Post by Michael H. Warfield
I've got some more problems relating to shutting down containers, some
of which may be related to mounting tmpfs on /run to which /var/run is
symlinked to. We're doing halt / restart detection by monitoring utmp
in that directory but it looks like utmp isn't even in that directory
anymore and mounting tmpfs on it was always problematical. We may have
to have a more generic method to detect when a container has shut down
or is restarting in that case.

I can't parse this. The system call reboot() is virtualized for
containers just fine and the container managaer (i.e. LXC) can check for
that easily.

The problem we have had was with differentiating between reboot and halt
to either shut the container down cold or restarted it. You say
"easily" and yet we never came up with an "easy" solution and monitored
utmp instead for the next runlevel change. What is your "easy" solution
for that problem?

I think you're on older kernels, where we had to resort to that. Pretty
recently Daniel Lezcano's patch was finally accepted upstream, which lets
a container call reboot() and lets the parent of init tell whether it
called reboot or shutdown by looking at wTERMSIG(status).

Now THAT is wonderful news! I hadn't realized that had been accepted.
So we no longer need to rely on the old utmp kludge?

Yup :) It was very liberating, in terms of what containers can do with
mounting.

[Michael H. Warfield]

Post by Michael H. Warfield

Post by Michael H. Warfield
http://wiki.1tux.org/wiki/Lxc/Installation#Additional_notes
Unfortunately, in our case, merely getting a mount in there is a
complication in that it also has to be populated but, at least, we
understand the problem set now.

Ok... Serge and I were corresponding on the lxc-users list and he had a
suggestion that worked but I consider to be a bit of a sub-optimal
workaround. Ironically, it was to mount devtmpfs on /dev. We don't
(currently) have a method to auto-populate a tmpfs mount with the needed
devices and this provided it. It does have a problem that makes me
uncomfortable in that the container now has visibility into the
hosts /dev system. I'm a security expert and I'm not comfortable with
that "solution" even with the controls we have. We can control access
but still, not happy with that.

That's a pretty bad idea, access control to the device nodes in devtmpfs
is controlled by the host's udev instance. That means if your group/user
lists in the container and the host differ you already lost. Also access
control in udev is dynamic, due to stuff like uaccess and similar. You
really don't want to to have that into the container, i.e. where device
change ownership all the time with UIDs/GIDs that make no sense at all
in the container.

Concur.

In general I think it's a good idea not to expose any "real" devices to
the container, but only the "virtual" ones that are programming
APIs. That means: no /dev/sda, or /dev/ttyS0, but /dev/null, /dev/zero,
/dev/random, /dev/urandom. And creating the latter in a tmpfs is quite
simple.

Post by Michael H. Warfield
If I run lxc-console (which attaches to one of the vtys) it gives me
nothing. Under sysvinit and upstart I get vty login prompts because
they have started getty on those vtys. This is important in case
network access has not started for one reason or another and the
container was started detached in the background.

http://www.freedesktop.org/wiki/Software/systemd/ContainerInterface

Sorry. This is unacceptable. We need some way that these will be
active and you will be consistent with other containers.

If LXC mounts ptys on top of the VT devices that's a really bad idea
too, since /dev/tty1 and friends expose a number of APIs beyond the mere
tty device that you cannot emulate with that. It includes files in /sys,
as well as /dev/vcs and /dev/vcsa, various ioctls, and so on. Heck, even
the most superficial of things, the $TERM variable will be
incorrect. LXC shouldn't do that.

REGARDLESS. I'm in this situation now testing what I thought was a hang
condition (which is proving to be something else). I started a
container detached redirecting the console to a file (a parameter I was
missing) and the log to another file (which I had been doing). But, for
some reason, sshd is not starting up. I have no way to attach to the
bloody console of the container and I have no getty's on a vty I can
attach to using lxc-console and I can't remote access a container which,
for all other intents and purposes, appears to be running fine.
Parameterize this bloody thing so we can have control over it.

LXC really shouldn't pretent a pty was a VT tty, it's not. From the
libvirt guys it has been proposed that we introduce a new env var to
pass to PID 1 of the container, that simply lists ptys to start gettys
on. That way we don't pretend anything about ttys that the ttys can't
hold and have a clean setup.

Post by Michael H. Warfield
I SUSPECT the hang condition is something to do with systemd trying to
start and interactive console on /dev/console, which sysvinit and
upstart do not do.

Yes, this is documented, please see the link I already posted, and which
I linked above a second time.

Post by Michael H. Warfield
I've got some more problems relating to shutting down containers, some
of which may be related to mounting tmpfs on /run to which /var/run is
symlinked to. We're doing halt / restart detection by monitoring utmp
in that directory but it looks like utmp isn't even in that directory
anymore and mounting tmpfs on it was always problematical. We may have
to have a more generic method to detect when a container has shut down
or is restarting in that case.

I can't parse this. The system call reboot() is virtualized for
containers just fine and the container managaer (i.e. LXC) can check for
that easily.
Lennart
--
Lennart Poettering - Red Hat, Inc.

--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121026/0138a56c/attachment.pgp>

[Michael H. Warfield]

Post by Michael H. Warfield

Post by Michael H. Warfield

Post by Michael H. Warfield
http://wiki.1tux.org/wiki/Lxc/Installation#Additional_notes
Unfortunately, in our case, merely getting a mount in there is a
complication in that it also has to be populated but, at least, we
understand the problem set now.

Ok... Serge and I were corresponding on the lxc-users list and he had a
suggestion that worked but I consider to be a bit of a sub-optimal
workaround. Ironically, it was to mount devtmpfs on /dev. We don't
(currently) have a method to auto-populate a tmpfs mount with the needed
devices and this provided it. It does have a problem that makes me
uncomfortable in that the container now has visibility into the
hosts /dev system. I'm a security expert and I'm not comfortable with
that "solution" even with the controls we have. We can control access
but still, not happy with that.

That's a pretty bad idea, access control to the device nodes in devtmpfs
is controlled by the host's udev instance. That means if your group/user
lists in the container and the host differ you already lost. Also access
control in udev is dynamic, due to stuff like uaccess and similar. You
really don't want to to have that into the container, i.e. where device
change ownership all the time with UIDs/GIDs that make no sense at all
in the container.

Concur.

In general I think it's a good idea not to expose any "real" devices to
the container, but only the "virtual" ones that are programming
APIs. That means: no /dev/sda, or /dev/ttyS0, but /dev/null, /dev/zero,
/dev/random, /dev/urandom. And creating the latter in a tmpfs is quite
simple.

Post by Michael H. Warfield
If I run lxc-console (which attaches to one of the vtys) it gives me
nothing. Under sysvinit and upstart I get vty login prompts because
they have started getty on those vtys. This is important in case
network access has not started for one reason or another and the
container was started detached in the background.

http://www.freedesktop.org/wiki/Software/systemd/ContainerInterface

Sorry. This is unacceptable. We need some way that these will be
active and you will be consistent with other containers.

If LXC mounts ptys on top of the VT devices that's a really bad idea
too, since /dev/tty1 and friends expose a number of APIs beyond the mere
tty device that you cannot emulate with that. It includes files in /sys,
as well as /dev/vcs and /dev/vcsa, various ioctls, and so on. Heck, even
the most superficial of things, the $TERM variable will be
incorrect. LXC shouldn't do that.

REGARDLESS. I'm in this situation now testing what I thought was a hang
condition (which is proving to be something else). I started a
container detached redirecting the console to a file (a parameter I was
missing) and the log to another file (which I had been doing). But, for
some reason, sshd is not starting up. I have no way to attach to the
bloody console of the container and I have no getty's on a vty I can
attach to using lxc-console and I can't remote access a container which,
for all other intents and purposes, appears to be running fine.
Parameterize this bloody thing so we can have control over it.

Here's another weirdism that's in your camp...

The reason that sshd did not start was because the network did not start
(IPv6 was up but IPv4 was not and the startup of several services failed
as a consequence). Trying to restart the network manually resulted in
this:

[root at alcove mhw]# ifdown eth0
./network-functions: line 237: cd: /var/run/netreport: No such file or directory
[root at alcove mhw]# ifup eth0
./network-functions: line 237: cd: /var/run/netreport: No such file or directory
[root at alcove mhw]# ls /var/run/
dbus messagebus.pid rpcbind.sock systemd user
log mount syslogd.pid udev

What the hell is this? /var/run is symlinked to /run and is mounted
with a tmpfs.

So I created that directory and could ifup the the network and start
sshd. So I did a little check on the run levels... Hmmm... F17
container (Alcove) in an F17 host (Forest). WHAT is going ON here? Is
this why the network didn't start?

[root at forest mhw]# runlevel
N 5

[root at alcove mhw]# runlevel
unknown

[root at alcove mhw]# chkconfig

Note: This output shows SysV services only and does not include native
systemd services. SysV configuration data might be overridden by native
systemd configuration.

modules_dep 0:off 1:off 2:on 3:on 4:on 5:on 6:off
netconsole 0:off 1:off 2:off 3:off 4:off 5:off 6:off
network 0:off 1:off 2:off 3:on 4:off 5:off 6:off


Mike
--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121026/bad0ba39/attachment.pgp>

[Colin Guthrie]

'Twas brillig, and Michael H. Warfield at 26/10/12 18:18 did gyre and

Post by Michael H. Warfield
What the hell is this? /var/run is symlinked to /run and is mounted
with a tmpfs.

Yup, that's how /var/run and /run is being handled these days.

It provides a consistent space to pass info from the initrd over to the
main system and has various other uses also.

If you want to ensure files are created in this folder, just drop a
config file in to /usr/lib/tmpfiles.d/ in the package in question. See
man systemd-tmpfiles for more info.

Could be some packages are not fully upgraded to this concept in F17. As
a non-fedora user, I can't really comment on that specifically.

Col
--
Colin Guthrie
gmane(at)colin.guthr.ie
http://colin.guthr.ie/

Day Job:
Tribalogic Limited http://www.tribalogic.net/
Open Source:
Mageia Contributor http://www.mageia.org/
PulseAudio Hacker http://www.pulseaudio.org/
Trac Hacker http://trac.edgewall.org/

[Michael H. Warfield]

Post by Colin Guthrie
'Twas brillig, and Michael H. Warfield at 26/10/12 18:18 did gyre and

Post by Michael H. Warfield
What the hell is this? /var/run is symlinked to /run and is mounted
with a tmpfs.

Yup, that's how /var/run and /run is being handled these days.
It provides a consistent space to pass info from the initrd over to the
main system and has various other uses also.

Interesting. I hadn't considered that aspect of it before. Very
interesting.

Post by Colin Guthrie
If you want to ensure files are created in this folder, just drop a
config file in to /usr/lib/tmpfiles.d/ in the package in question. See
man systemd-tmpfiles for more info.

NOW THAT is something else I needed to know about! Thank you very very
much! Learned something new. This whole thing has been a massive
learning experience getting this container kick started.

Post by Colin Guthrie
Could be some packages are not fully upgraded to this concept in F17. As
a non-fedora user, I can't really comment on that specifically.

As it turns out, the kernel has had some of our patches applied that I
wasn't aware of vis-a-vis reboot/halt and this should no longer be an
issue. I'm still struggling with the tmpfs on /dev thing and have run
into a catch-22 with regards to that. I can mount tmpfs on /dev just
fine and can populate it just fine in a post mount hook but, then, we're
trying to mount a devpts file system on /dev/pts before we've had a
chance to populate it and it's then crashing on the mount. Sigh... I
think that's going to now have to wait for Serge or Daniel to comment
on.

Post by Colin Guthrie
Col
--
Colin Guthrie
gmane(at)colin.guthr.ie
http://colin.guthr.ie/
Tribalogic Limited http://www.tribalogic.net/
Mageia Contributor http://www.mageia.org/
PulseAudio Hacker http://www.pulseaudio.org/
Trac Hacker http://trac.edgewall.org/

Regards,
Mike
--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121027/971626a0/attachment.pgp>

[Michael H. Warfield]

Post by Michael H. Warfield
I SUSPECT the hang condition is something to do with systemd trying to
start and interactive console on /dev/console, which sysvinit and
upstart do not do.

Yes, this is documented, please see the link I already posted, and which
I linked above a second time.

This may have been my fault. I was using the -o option to lxc-start
(output logfile) and failed to specify the -c (console output redirect)
option. It seems to fire up nicely (albeit with other problems) with
that additional option. Continuing my research.

Post by Michael H. Warfield
I've got some more problems relating to shutting down containers, some
of which may be related to mounting tmpfs on /run to which /var/run is
symlinked to. We're doing halt / restart detection by monitoring utmp
in that directory but it looks like utmp isn't even in that directory
anymore and mounting tmpfs on it was always problematical. We may have
to have a more generic method to detect when a container has shut down
or is restarting in that case.

I can't parse this. The system call reboot() is virtualized for
containers just fine and the container managaer (i.e. LXC) can check for
that easily.

Apparently, in recent kernels, we can. Unfortunately, I'm still finding
that I can not restart a container I have previously halted. I have no
problem with sysvinit and upstart systems on this host, so it is a
container problem peculiar to systemd containers. Continuing to
research that problem.

Lennart
--
Lennart Poettering - Red Hat, Inc.

Regards,
Mike
--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121026/795f5cd2/attachment.pgp>

[Michael H. Warfield]

Post by Michael H. Warfield

Post by Michael H. Warfield
I SUSPECT the hang condition is something to do with systemd trying to
start and interactive console on /dev/console, which sysvinit and
upstart do not do.

Yes, this is documented, please see the link I already posted, and which
I linked above a second time.

This may have been my fault. I was using the -o option to lxc-start
(output logfile) and failed to specify the -c (console output redirect)
option. It seems to fire up nicely (albeit with other problems) with
that additional option. Continuing my research.

Confirming. Using the -c option for the console file works.
Unfortunately, thanks to no getty's on the ttys so lxc-console does not
work and no way to connect to that console redirect and the failure of
the network to start, I'm still trying to figure out just what is face
planting in a container I can not access. :-/=/ Punch out the punch
list one PUNCH at at time here.

Post by Michael H. Warfield

Post by Michael H. Warfield
I've got some more problems relating to shutting down containers, some
of which may be related to mounting tmpfs on /run to which /var/run is
symlinked to. We're doing halt / restart detection by monitoring utmp
in that directory but it looks like utmp isn't even in that directory
anymore and mounting tmpfs on it was always problematical. We may have
to have a more generic method to detect when a container has shut down
or is restarting in that case.

I can't parse this. The system call reboot() is virtualized for
containers just fine and the container managaer (i.e. LXC) can check for
that easily.

Apparently, in recent kernels, we can. Unfortunately, I'm still finding
that I can not restart a container I have previously halted. I have no
problem with sysvinit and upstart systems on this host, so it is a
container problem peculiar to systemd containers. Continuing to
research that problem.

Lennart
--
Lennart Poettering - Red Hat, Inc.

Regards,
Mike

--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121026/9482d2ed/attachment.pgp>

[John]

Post by Michael H. Warfield

Post by Michael H. Warfield

Post by Michael H. Warfield
I SUSPECT the hang condition is something to do with systemd trying to
start and interactive console on /dev/console, which sysvinit and
upstart do not do.

Yes, this is documented, please see the link I already posted, and which
I linked above a second time.

This may have been my fault. I was using the -o option to lxc-start
(output logfile) and failed to specify the -c (console output redirect)
option. It seems to fire up nicely (albeit with other problems) with
that additional option. Continuing my research.

Confirming. Using the -c option for the console file works.
Unfortunately, thanks to no getty's on the ttys so lxc-console does not
work and no way to connect to that console redirect and the failure of
the network to start, I'm still trying to figure out just what is face
planting in a container I can not access. :-/=/ Punch out the punch
list one PUNCH at at time here.

Post by Michael H. Warfield

Post by Michael H. Warfield
I've got some more problems relating to shutting down containers, some
of which may be related to mounting tmpfs on /run to which /var/run is
symlinked to. We're doing halt / restart detection by monitoring utmp
in that directory but it looks like utmp isn't even in that directory
anymore and mounting tmpfs on it was always problematical. We may have
to have a more generic method to detect when a container has shut down
or is restarting in that case.

I can't parse this. The system call reboot() is virtualized for
containers just fine and the container managaer (i.e. LXC) can check for
that easily.

Apparently, in recent kernels, we can. Unfortunately, I'm still finding
that I can not restart a container I have previously halted. I have no
problem with sysvinit and upstart systems on this host, so it is a
container problem peculiar to systemd containers. Continuing to
research that problem.

Lennart
--
Lennart Poettering - Red Hat, Inc.

Regards,
Mike

------------------------------------------------------------------------------
WINDOWS 8 is here.
Millions of people. Your app in 30 days.
Visit The Windows 8 Center at Sourceforge for all your go to resources.
http://windows8center.sourceforge.net/
join-generation-app-and-make-money-coding-fast/
_______________________________________________
Lxc-users mailing list
Lxc-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lxc-users

I raised the question about LXC/systemd a while back and have been
trying to follow the conversation but I have to admit it's going
somewhat over my head. I've also been away on another piece of work but
would now like to understand where things lie with LXC and systemd
inside a container.

I have just updated my system to 0.8.0 and I can't see any changes to
make a systemd container work. Are there changes in 0.8.0 ?

If so, I'd be grateful for some guidance on what I need to do to to my
configuration to make it work.

I'm also happy to help test this if I can. If it helps I am on Arch Linux.

many thanks everyone.

John

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121204/59b93712/attachment.html>

[Michael H. Warfield]

Post by John

Post by Michael H. Warfield

Post by Michael H. Warfield

Post by Michael H. Warfield
I SUSPECT the hang condition is something to do with systemd trying to
start and interactive console on /dev/console, which sysvinit and
upstart do not do.

Yes, this is documented, please see the link I already posted, and which
I linked above a second time.

This may have been my fault. I was using the -o option to lxc-start
(output logfile) and failed to specify the -c (console output redirect)
option. It seems to fire up nicely (albeit with other problems) with
that additional option. Continuing my research.

Confirming. Using the -c option for the console file works.
Unfortunately, thanks to no getty's on the ttys so lxc-console does not
work and no way to connect to that console redirect and the failure of
the network to start, I'm still trying to figure out just what is face
planting in a container I can not access. :-/=/ Punch out the punch
list one PUNCH at at time here.

Post by Michael H. Warfield

Post by Michael H. Warfield
I've got some more problems relating to shutting down containers, some
of which may be related to mounting tmpfs on /run to which /var/run is
symlinked to. We're doing halt / restart detection by monitoring utmp
in that directory but it looks like utmp isn't even in that directory
anymore and mounting tmpfs on it was always problematical. We may have
to have a more generic method to detect when a container has shut down
or is restarting in that case.

I can't parse this. The system call reboot() is virtualized for
containers just fine and the container managaer (i.e. LXC) can check for
that easily.

Apparently, in recent kernels, we can. Unfortunately, I'm still finding
that I can not restart a container I have previously halted. I have no
problem with sysvinit and upstart systems on this host, so it is a
container problem peculiar to systemd containers. Continuing to
research that problem.

Lennart
--
Lennart Poettering - Red Hat, Inc.

Regards,
Mike

------------------------------------------------------------------------------
WINDOWS 8 is here.
Millions of people. Your app in 30 days.
Visit The Windows 8 Center at Sourceforge for all your go to resources.
http://windows8center.sourceforge.net/
join-generation-app-and-make-money-coding-fast/
_______________________________________________
Lxc-users mailing list
Lxc-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lxc-users

I raised the question about LXC/systemd a while back and have been
trying to follow the conversation but I have to admit it's going
somewhat over my head. I've also been away on another piece of work but
would now like to understand where things lie with LXC and systemd
inside a container.

Ok... I'll try to answer some of them...

Post by John
I have just updated my system to 0.8.0 and I can't see any changes to
make a systemd container work. Are there changes in 0.8.0 ?

There are very significant changes in 0.8.0 but, unfortunately, not the
ones you need to get systemd to work in a container. We've been testing
a lot of these and they are in git but they are not in a release yet.
Hopefully soon, just not yet.

Post by John
If so, I'd be grateful for some guidance on what I need to do to to my
configuration to make it work.

Right now, you'll have to build from git.

Post by John
I'm also happy to help test this if I can. If it helps I am on Arch Linux.

There are two problems. One is systemd in an lxc container. I think we
have a rope on this one and it's tied down. The other is the more
recent (195+) versions of systemd in the host that throw the "pivot
root" errors. That has not been addressed as yet. I use Fedora. Right
now, I have Fedora 17 hosts with Fedora 17 containers. Fedora 18
(currently in beta) host (systemd 195) is going to be a train wreck
until we sort the pivot root problem. I don't know what you have with
Arch Linux. You'll have to tell us what versions of systemd you are
running.

Post by John
many thanks everyone.
John

Mike
--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121204/faa12549/attachment.pgp>

[C Anthony Risinger]

[...]

Post by Michael H. Warfield

Post by John
I'm also happy to help test this if I can. If it helps I am on Arch Linux.

There are two problems. One is systemd in an lxc container. I think we
have a rope on this one and it's tied down. The other is the more
recent (195+) versions of systemd in the host that throw the "pivot
root" errors. That has not been addressed as yet. I use Fedora. Right
now, I have Fedora 17 hosts with Fedora 17 containers. Fedora 18
(currently in beta) host (systemd 195) is going to be a train wreck
until we sort the pivot root problem. I don't know what you have with
Arch Linux. You'll have to tell us what versions of systemd you are
running.

we (Arch) are currently running systemd 196... so i guess... :sadface:

[John]

Post by Michael H. Warfield

Post by John
I raised the question about LXC/systemd a while back and have been
trying to follow the conversation but I have to admit it's going
somewhat over my head. I've also been away on another piece of work but
would now like to understand where things lie with LXC and systemd
inside a container.

Ok... I'll try to answer some of them...

Thanks Mike, much appreciated.

Post by Michael H. Warfield

Post by John
I have just updated my system to 0.8.0 and I can't see any changes to
make a systemd container work. Are there changes in 0.8.0 ?

There are very significant changes in 0.8.0 but, unfortunately, not the
ones you need to get systemd to work in a container. We've been testing
a lot of these and they are in git but they are not in a release yet.
Hopefully soon, just not yet.

Post by John
If so, I'd be grateful for some guidance on what I need to do to to my
configuration to make it work.

Right now, you'll have to build from git.

I will go away and do a git build later today.
I presume that would be from git://lxc.git.sourceforge.net/gitroot/lxc/lxc.

Post by Michael H. Warfield

Post by John
I'm also happy to help test this if I can. If it helps I am on Arch Linux.

There are two problems. One is systemd in an lxc container. I think we
have a rope on this one and it's tied down. The other is the more
recent (195+) versions of systemd in the host that throw the "pivot
root" errors. That has not been addressed as yet. I use Fedora. Right
now, I have Fedora 17 hosts with Fedora 17 containers. Fedora 18
(currently in beta) host (systemd 195) is going to be a train wreck
until we sort the pivot root problem. I don't know what you have with
Arch Linux. You'll have to tell us what versions of systemd you are
running.

Ah yes, the "pivot root" problem. I have worked around this for the time
being by doing a "mount --make-rprivate /". I created a systemd service
on the host as an "after" dependency on "systemd-remount-fs.service" to
do this. I believe this is ok in the short term (it appears to work ok
for me).

If I rebuild lxc from git, should I then expect my existing systemd
container to work or is there anything else that I need to do ?

My versions:

lxc version: 0.8.0
Linux hydrogen 3.6.8-1-ARCH #1 SMP PREEMPT Mon Nov 26 22:10:40 CET 2012
x86_64 GNU/Linux
systemd 196

Post by Michael H. Warfield

Post by John
many thanks everyone.
John

Mike

Thanks, I really appreciate the help.

[Serge Hallyn]

Post by John

Post by Michael H. Warfield

Post by John
If so, I'd be grateful for some guidance on what I need to do to to my
configuration to make it work.

Right now, you'll have to build from git.

I will go away and do a git build later today.
I presume that would be from git://lxc.git.sourceforge.net/gitroot/lxc/lxc.

No, git://github.com/lxc/lxc.git #staging

-serge

[Michael H. Warfield]

Post by John

Post by Michael H. Warfield

Post by John
I raised the question about LXC/systemd a while back and have been
trying to follow the conversation but I have to admit it's going
somewhat over my head. I've also been away on another piece of work but
would now like to understand where things lie with LXC and systemd
inside a container.

Ok... I'll try to answer some of them...

Thanks Mike, much appreciated.

Post by Michael H. Warfield

Post by John
I have just updated my system to 0.8.0 and I can't see any changes to
make a systemd container work. Are there changes in 0.8.0 ?

There are very significant changes in 0.8.0 but, unfortunately, not the
ones you need to get systemd to work in a container. We've been testing
a lot of these and they are in git but they are not in a release yet.
Hopefully soon, just not yet.

Post by John
If so, I'd be grateful for some guidance on what I need to do to to my
configuration to make it work.

Right now, you'll have to build from git.

I will go away and do a git build later today.
I presume that would be from git://lxc.git.sourceforge.net/gitroot/lxc/lxc.

Post by Michael H. Warfield

Post by John
I'm also happy to help test this if I can. If it helps I am on Arch Linux.

There are two problems. One is systemd in an lxc container. I think we
have a rope on this one and it's tied down. The other is the more
recent (195+) versions of systemd in the host that throw the "pivot
root" errors. That has not been addressed as yet. I use Fedora. Right
now, I have Fedora 17 hosts with Fedora 17 containers. Fedora 18
(currently in beta) host (systemd 195) is going to be a train wreck
until we sort the pivot root problem. I don't know what you have with
Arch Linux. You'll have to tell us what versions of systemd you are
running.

Ah yes, the "pivot root" problem. I have worked around this for the time
being by doing a "mount --make-rprivate /". I created a systemd service
on the host as an "after" dependency on "systemd-remount-fs.service" to
do this. I believe this is ok in the short term (it appears to work ok
for me).

Hmmm... I was thinking someone ran into some problems doing that and
causing problems with the /dev/pts mounts or some such. Good to note if
that worked for you. I'm about to start playing with Fedora 18 Beta
where I expect problems. I'll try that out.

Post by John
If I rebuild lxc from git, should I then expect my existing systemd
container to work or is there anything else that I need to do ?

Yeah, one other thing (in addition to following Serge's advice regarding
git and #stage)...

You have to add an option to the config file for your systemd
containers.

lxc.autodev = 1

Post by John
lxc version: 0.8.0
Linux hydrogen 3.6.8-1-ARCH #1 SMP PREEMPT Mon Nov 26 22:10:40 CET 2012
x86_64 GNU/Linux
systemd 196

Post by Michael H. Warfield

Post by John
many thanks everyone.
John

Mike

Thanks, I really appreciate the help.

Regards,
Mike
--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121205/00486801/attachment.pgp>

[Serge Hallyn]

Post by Michael H. Warfield
You have to add an option to the config file for your systemd
containers.
lxc.autodev = 1

Phrasing it this way makes me wonder, should lxc look for
'$rootfs/dev/console' and automatically set lxc.autodev if
that is not found?

(Right now if lxc.autodev is 1 then the tmpfs /dev is mounted
before all the lxc.mount.entries and /var/lib/lxc/$c/fstab
entries, but I can't right now think of a reason why it has
to stay that way. If we were to always set lxc.autodev if
/dev is empty, we'd want to make sure any separate /dev has
been mounted, of course.)

-serge

[Michael H. Warfield]

Post by Serge Hallyn

Post by Michael H. Warfield
You have to add an option to the config file for your systemd
containers.
lxc.autodev = 1

Phrasing it this way makes me wonder, should lxc look for
'$rootfs/dev/console' and automatically set lxc.autodev if
that is not found?

I'm of two minds here (which, in my case, is a reduction in force and
those two minds are going "where did everybody go?").

That might be an idea for jogging the default. If you don't have
'$rootfs/dev/console' then set it. In that obvious case, you need it.
If you do have it, obey what's on the config file?

We have a bit of a chicken and egg situation. It's not just auto
populating /dev but it's also mounting a ramfs partition on it. I'm not
sure I'm comfortable with the level of random acts of terrorism that
systemd has proven to be capable of if someone accidentally leaves
a .../dev/console in their file system so we don't then mount ramfs on
dev and we don't then auto populate dev.

But... The same situation exists if the user doesn't manually provide
the autodev option. But... I would not want us to switch based on the
existence of systemd either.

Maybe there is some other way we can autodetect this that doesn't depend
on those static devices?

Overall, I like that idea. It helps idiot proof the configuration
better.

Post by Serge Hallyn
(Right now if lxc.autodev is 1 then the tmpfs /dev is mounted
before all the lxc.mount.entries and /var/lib/lxc/$c/fstab
entries, but I can't right now think of a reason why it has
to stay that way. If we were to always set lxc.autodev if
/dev is empty, we'd want to make sure any separate /dev has
been mounted, of course.)

Concur. I think this is a good idea, we just have to watch some of the
corner cases. The one I fear the most is the one where someone (ME!)
does a yum upgrade of a container that then becomes systemd (F14 -> F15)
where it use to be Upstart with a static /dev populated. Boom. Flash
of light, mushroom cloud on the horizon. But, again, if I don't fix the
bloody config, I'm screwed as well. Self inflicted injury. I see that.

I don't see a downside to adding it. I'm just a little nudgey about
relying on it. On the balance... I'm in favor, yeah.

Post by Serge Hallyn
-serge

Regards,
Mike
--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121205/e299d792/attachment.pgp>

[John]

Post by Michael H. Warfield
You have to add an option to the config file for your systemd
containers.
lxc.autodev = 1

I would like to understand a bit more about what this option does and
learn the correct way of creating devices inside a container.

With autodev, if I understand correctly, LXC creates a 100Kb tmpfs for
/dev, overmounting any existing /dev. it creates a pts subdirectory plus
the devices listed in sutuct lxc_devs (src/lxc/conf.c) - null, zero,
full, urandom, random, tty and console.

What do I do if I need more than those devices in /dev? To date, I have
manually used mknod to create devices during the process of creating a
rootfs (i.e. I create them beforehand, on the host). I see the comment
in the source about providing a file, so I guess this is being thought
about?

I like being able to do things in the main config file, so perhaps
supporting options like "lxc.dev = name mask type maj min" ?

Also, I can't work out what the autodev option does that allows systemd
to work ? It's a bit over my head but I'd like to understand if I can.
What's the difference between /dev that is on the rootfs and a /dev that
is created and over-mounted?

systemd inside container = issues so far:
- creating devices in /dev
- no vty devices (cannot use lxc-console)

Regards,
John

[Serge Hallyn]

Post by John

Post by Michael H. Warfield
You have to add an option to the config file for your systemd
containers.
lxc.autodev = 1

I would like to understand a bit more about what this option does
and learn the correct way of creating devices inside a container.
With autodev, if I understand correctly, LXC creates a 100Kb tmpfs
for /dev, overmounting any existing /dev. it creates a pts
subdirectory plus the devices listed in sutuct lxc_devs
(src/lxc/conf.c) - null, zero, full, urandom, random, tty and
console.
What do I do if I need more than those devices in /dev? To date, I
have manually used mknod to create devices during the process of
creating a rootfs (i.e. I create them beforehand, on the host). I
see the comment in the source about providing a file, so I guess
this is being thought about?
I like being able to do things in the main config file, so perhaps
supporting options like "lxc.dev = name mask type maj min" ?

Yup, in either the commit msg or the RFC email I suggested we would
probably want to add that. I think it's a good idea. I just didn't
do it :) Does someone want to write that patch?

Post by John
Also, I can't work out what the autodev option does that allows
systemd to work ? It's a bit over my head but I'd like to understand

It's because systemd checks whether /dev is a separate filesystem from
/ or not. If it not, then it mounts its own /dev, hiding the console
which lxc has created, which is a unix98 pty which lxc-console will
attach to.

In fact it's more dangerous than that - systemd will (I'm pretty sure)
mount /dev as devtmpfs type, which means it's a shared mount with the
host, so changes made by the container to /dev will be reflected on the
host's /dev.

Post by John
if I can. What's the difference between /dev that is on the rootfs
and a /dev that is created and over-mounted?

Create a fedora 14 container. Look at /dev/console and /dev/tty1 -
/dev/tty4 in container and on the host. They're different.

Post by John
- creating devices in /dev
- no vty devices (cannot use lxc-console)
Regards,
John

[John]

On 05/12/12 14:55, Michael H. Warfield wrote:
[...]

Post by Michael H. Warfield

Post by John
Ah yes, the "pivot root" problem. I have worked around this for the time
being by doing a "mount --make-rprivate /". I created a systemd service
on the host as an "after" dependency on "systemd-remount-fs.service" to
do this. I believe this is ok in the short term (it appears to work ok
for me).

Hmmm... I was thinking someone ran into some problems doing that and
causing problems with the /dev/pts mounts or some such. Good to note if
that worked for you. I'm about to start playing with Fedora 18 Beta
where I expect problems. I'll try that out.

Post by John
If I rebuild lxc from git, should I then expect my existing systemd
container to work or is there anything else that I need to do ?

Yeah, one other thing (in addition to following Serge's advice regarding
git and #stage)...
You have to add an option to the config file for your systemd
containers.
lxc.autodev = 1

Ok got that. I used git://github.com/lxc/lxc.git #staging. Built and
installed ok. Existing containers running. When I try to create a new
one, with or without the autodev like you suggest, I get the below:

# lxc-create -n test2 -f test2.conf
lxc-create: unknown template ''
lxc-create: aborted

I checked and the above create does work with 0.8.0. I realise it's
probably a glitch caused by something unrelated and which will probably
be fixed quite quickly. I may try a re-build in the morning.


Next, I manually edited /var/lib/lxc/test/config to add lxc.autodev to
it but attempting to start the container gave me this:

# lxc-start -n test2
lxc-start: No such file or directory - failed to mount 'devshm' on
'/usr/lib/lxc/rootfs//dev/shm'

I had an instruction in the config to mount devshm so I removed that and
could then start the container up successfully. I got a login prompt and
can log in. Lovely!

I now need to run some more tests here but I can confirm that the
staging build will allow a container to start on my Arch system.

FYI (Arch - specific): I used a modified copy of the lxc-git PKGBUILD
(https://aur.archlinux.org/packages/lx/lxc-git/PKGBUILD) to build
lxc#staging. I only changed the git root to be
"git://github.com/lxc/lxc.git".

ps. I Just did an lxc-destroy while testing and it appears to now be
destructive. That took me by surprise.

Regards,
John

[Serge Hallyn]

Post by John
[...]

Post by Michael H. Warfield

Post by John
Ah yes, the "pivot root" problem. I have worked around this for the time
being by doing a "mount --make-rprivate /". I created a systemd service
on the host as an "after" dependency on "systemd-remount-fs.service" to
do this. I believe this is ok in the short term (it appears to work ok
for me).

Hmmm... I was thinking someone ran into some problems doing that and
causing problems with the /dev/pts mounts or some such. Good to note if
that worked for you. I'm about to start playing with Fedora 18 Beta
where I expect problems. I'll try that out.

Post by John
If I rebuild lxc from git, should I then expect my existing systemd
container to work or is there anything else that I need to do ?

Yeah, one other thing (in addition to following Serge's advice regarding
git and #stage)...
You have to add an option to the config file for your systemd
containers.
lxc.autodev = 1

Ok got that. I used git://github.com/lxc/lxc.git #staging. Built and
installed ok. Existing containers running. When I try to create a new
# lxc-create -n test2 -f test2.conf
lxc-create: unknown template ''
lxc-create: aborted

You have to specify a template, i.e. '-t debian'.

[John]

Post by Serge Hallyn
You have to specify a template, i.e. '-t debian'.

Oh. I wasn't using a template.

Up to now, I have an existing root fileyststem, say
/srv/lxc/mycontainer.x86_64 that is pointed to by my configuration file,
say mycontainer.conf, by its lxc.rootfs entry. I have seen lxc-create as
merely inserting the config from mycontainer.conf into
/var/lib/lxc/mycontainer/config and nothing more.

I haven't used a template script to create a container because I've got
my own that I have been using ever since I first started using lxc
(there were no templates back then, well not for arch anyway!).

I've always done a destroy/create to update the LXC configuration for a
container. This now seems to be the wrong way given destroy removes the
rootfs and create expects a template. What's the new way ?

I've looked at the man page for lxc-create but am none the wiser. How do
I now create a container (or just update the config) for an existing
root filesystem ?

Thanks,
John

[Serge Hallyn]

Post by John

Post by Serge Hallyn
You have to specify a template, i.e. '-t debian'.

Oh. I wasn't using a template.
Up to now, I have an existing root fileyststem, say
/srv/lxc/mycontainer.x86_64 that is pointed to by my configuration
file, say mycontainer.conf, by its lxc.rootfs entry. I have seen
lxc-create as merely inserting the config from mycontainer.conf into
/var/lib/lxc/mycontainer/config and nothing more.
I haven't used a template script to create a container because I've
got my own that I have been using ever since I first started using
lxc (there were no templates back then, well not for arch anyway!).
I've always done a destroy/create to update the LXC configuration
for a container. This now seems to be the wrong way given destroy
removes the rootfs and create expects a template. What's the new way
?
I've looked at the man page for lxc-create but am none the wiser.
How do I now create a container (or just update the config) for an
existing root filesystem ?

Hm, I see. Yeah this behavior likely changed with the introduction
of custom template paths.

Perhaps we should allow '-t none' for exactly your use case.

St?phane?

-serge

[John]

Post by Serge Hallyn

Post by John

Post by Serge Hallyn
You have to specify a template, i.e. '-t debian'.

Oh. I wasn't using a template.
Up to now, I have an existing root fileyststem, say
/srv/lxc/mycontainer.x86_64 that is pointed to by my configuration
file, say mycontainer.conf, by its lxc.rootfs entry. I have seen
lxc-create as merely inserting the config from mycontainer.conf into
/var/lib/lxc/mycontainer/config and nothing more.
I haven't used a template script to create a container because I've
got my own that I have been using ever since I first started using
lxc (there were no templates back then, well not for arch anyway!).
I've always done a destroy/create to update the LXC configuration
for a container. This now seems to be the wrong way given destroy
removes the rootfs and create expects a template. What's the new way
?
I've looked at the man page for lxc-create but am none the wiser.
How do I now create a container (or just update the config) for an
existing root filesystem ?

Hm, I see. Yeah this behavior likely changed with the introduction
of custom template paths.
Perhaps we should allow '-t none' for exactly your use case.
St?phane?
-serge

Or perhaps, allow leaving off the -t unless you want to work with a
template ?
(kind of like it's been to date). Would that not work ?

[Stéphane Graber]

Post by John

Post by Serge Hallyn

Post by John

Post by Serge Hallyn
You have to specify a template, i.e. '-t debian'.

Oh. I wasn't using a template.
Up to now, I have an existing root fileyststem, say
/srv/lxc/mycontainer.x86_64 that is pointed to by my configuration
file, say mycontainer.conf, by its lxc.rootfs entry. I have seen
lxc-create as merely inserting the config from mycontainer.conf into
/var/lib/lxc/mycontainer/config and nothing more.
I haven't used a template script to create a container because I've
got my own that I have been using ever since I first started using
lxc (there were no templates back then, well not for arch anyway!).
I've always done a destroy/create to update the LXC configuration
for a container. This now seems to be the wrong way given destroy
removes the rootfs and create expects a template. What's the new way
?
I've looked at the man page for lxc-create but am none the wiser.
How do I now create a container (or just update the config) for an
existing root filesystem ?

Hm, I see. Yeah this behavior likely changed with the introduction
of custom template paths.
Perhaps we should allow '-t none' for exactly your use case.
St?phane?
-serge

Or perhaps, allow leaving off the -t unless you want to work with a
template ?
(kind of like it's been to date). Would that not work ?

Yeah, that makes sense, I'll fix it.

Basically allow for "-t none" and have it default to that when not
specified, that should essentially revert to the previous behaviour.
--
St?phane Graber
Ubuntu developer
http://www.ubuntu.com

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 899 bytes
Desc: OpenPGP digital signature
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121206/7917ef51/attachment.pgp>

[John]

Post by Stéphane Graber

Post by John

Post by Serge Hallyn

Post by John

Post by Serge Hallyn
You have to specify a template, i.e. '-t debian'.

Oh. I wasn't using a template.
Up to now, I have an existing root fileyststem, say
/srv/lxc/mycontainer.x86_64 that is pointed to by my configuration
file, say mycontainer.conf, by its lxc.rootfs entry. I have seen
lxc-create as merely inserting the config from mycontainer.conf into
/var/lib/lxc/mycontainer/config and nothing more.
I haven't used a template script to create a container because I've
got my own that I have been using ever since I first started using
lxc (there were no templates back then, well not for arch anyway!).
I've always done a destroy/create to update the LXC configuration
for a container. This now seems to be the wrong way given destroy
removes the rootfs and create expects a template. What's the new way
?
I've looked at the man page for lxc-create but am none the wiser.
How do I now create a container (or just update the config) for an
existing root filesystem ?

Hm, I see. Yeah this behavior likely changed with the introduction
of custom template paths.
Perhaps we should allow '-t none' for exactly your use case.
St?phane?
-serge

Or perhaps, allow leaving off the -t unless you want to work with a
template ?
(kind of like it's been to date). Would that not work ?

Yeah, that makes sense, I'll fix it.
Basically allow for "-t none" and have it default to that when not
specified, that should essentially revert to the previous behaviour.

While on the subject, any reason for lxc-destroy now being destructive?
This in, my opinion, is a significant behavioural change and I did
actually unwittingly delete one of my containers last night. Luckily it
was just a test one :)

Can we make lxc-destroy work like it did before (or provide a cmdline
option to make it so) ?

I don't know how else to update lxc config without doing a
destroy/create cycle (except for hand-editing
/var/lib/mycontainer/config but I expect that's verboten).

sorry - going off topic for the original thread.

J

[Dan Kegel]

Post by John
While on the subject, any reason for lxc-destroy now being destructive?

Wait, isn't that the point? It's in the name and all.

When was it ever nondestructive?

[John]

Post by Dan Kegel

Post by John
While on the subject, any reason for lxc-destroy now being destructive?

Wait, isn't that the point? It's in the name and all.
When was it ever nondestructive?

It only destroyed the configuration in /var/lib and never deleted the
root filesystem until very recently (0.8.0, I guess).

[Serge Hallyn]

Post by John

Post by Dan Kegel

Post by John
While on the subject, any reason for lxc-destroy now being destructive?

Wait, isn't that the point? It's in the name and all.
When was it ever nondestructive?

It only destroyed the configuration in /var/lib and never deleted the
root filesystem until very recently (0.8.0, I guess).

Was your rootfs a symbolic link by chance? I'm guessing commit
55116c42e767ce795f796fc51cd2ef7d76cf18af is what you're seeing. Before
that it did remove the rootfs, but if your rootfs was a symlink it
happened to not do it. That wasn't by intent.

Perhaps lxc-destroy should take a flag to not delete the rootfs? Not
sure...

[John]

Post by Serge Hallyn

Post by John

Post by Dan Kegel

Post by John
While on the subject, any reason for lxc-destroy now being destructive?

Wait, isn't that the point? It's in the name and all.
When was it ever nondestructive?

It only destroyed the configuration in /var/lib and never deleted the
root filesystem until very recently (0.8.0, I guess).

Was your rootfs a symbolic link by chance? I'm guessing commit
55116c42e767ce795f796fc51cd2ef7d76cf18af is what you're seeing. Before
that it did remove the rootfs, but if your rootfs was a symlink it
happened to not do it. That wasn't by intent.
Perhaps lxc-destroy should take a flag to not delete the rootfs? Not
sure...

Ah, I can now see what is wrong. It isn't down to symlinks but beacuase
my rootfs isn't under /var/lib/lxc.

Looking at that commit, I can see that the remove (on line 126) deletes
"$lxc_path/$lxc_name" but does not explictly remove "$rootdev". The new
code added at line 122 does indeed remove $rootdev.

In my case I have my container rootfs in a directory called
"/srv/test.i686" (i.e not underneath $lxc_path - /var/lib/lxc). I guess
the design assumes that a template is used to create a container and
that it would put the rootfs beneath /var/lib/test.

So the commit fixes an anomaly but leaves me unsure of a couple of things:

1. what is the correct way to update a container config without removing
the rootfs. I have always used destroy/create to do this but that,
clearly, won't work if the destroy phase removes the rootfs. I like
being able to separately manage the rootfs from its configuration.

2. is it wrong to have the rootfs outside of the /var/lib/lxc ? I have a
small /var but use a large dedicated partition for my root filesystem
directories. I suspect I need to look at using per-container lvm
volumes, something that makes sense but I haven't delved into yet.

I would value having options to preserve the rootfs when doing
lxc-destroy and for lxc-create to use an existing rootfs (i.e. instead
of a template).

Thanks very much for the help.

[Serge Hallyn]

Post by John

Post by Serge Hallyn

Post by John

Post by Dan Kegel

Post by John
While on the subject, any reason for lxc-destroy now being destructive?

Wait, isn't that the point? It's in the name and all.
When was it ever nondestructive?

It only destroyed the configuration in /var/lib and never deleted the
root filesystem until very recently (0.8.0, I guess).

Was your rootfs a symbolic link by chance? I'm guessing commit
55116c42e767ce795f796fc51cd2ef7d76cf18af is what you're seeing. Before
that it did remove the rootfs, but if your rootfs was a symlink it
happened to not do it. That wasn't by intent.
Perhaps lxc-destroy should take a flag to not delete the rootfs? Not
sure...

Ah, I can now see what is wrong. It isn't down to symlinks but
beacuase my rootfs isn't under /var/lib/lxc.
Looking at that commit, I can see that the remove (on line 126)
deletes "$lxc_path/$lxc_name" but does not explictly remove
"$rootdev". The new code added at line 122 does indeed remove
$rootdev.
In my case I have my container rootfs in a directory called
"/srv/test.i686" (i.e not underneath $lxc_path - /var/lib/lxc). I
guess the design assumes that a template is used to create a
container and that it would put the rootfs beneath /var/lib/test.
1. what is the correct way to update a container config without
removing the rootfs. I have always used destroy/create to do this
but that, clearly, won't work if the destroy phase removes the
rootfs. I like being able to separately manage the rootfs from its
configuration.

This I don't really understand - I've always done it by hand. What
exactly is made easier by doing destroy/create? Maybe we can reproduce
that with an lxc-update or something... Especially if we can then
have lxc-update expand variables and take a list of containers to
update to batch the operations. Though still right now I would just
default to a bash loop calling sed...

Post by John
2. is it wrong to have the rootfs outside of the /var/lib/lxc ? I

Not if you're doing it right :) That's in fact why the --dir option
was added. But that's exactly why the lxc-destroy behavior should
be the same with the rootfs in our out of /var/lib/lxc/$container.

Post by John
have a small /var but use a large dedicated partition for my root
filesystem directories. I suspect I need to look at using
per-container lvm volumes, something that makes sense but I haven't
delved into yet.

If it's working for you and you don't need separate volumes for fast
snapshotted clones, I wouldn't change it. We need to make sure to
support your use case.

Post by John
I would value having options to preserve the rootfs when doing
lxc-destroy and for lxc-create to use an existing rootfs (i.e.
instead of a template).

Ok, I don't *really* want to make lxc-destroy not delete the rootfs
just if it is outside of /var/lib/lxc/$container... On the one hand
I can see people could do that specifically in the hopes of making
it outlive the container. On the other hand I could see people doing
it only because they are short on disk space ending up running out of
disk space because they lost track of where the undeleted rootfs's
were.

Maybe

lxc-destroy -k -n p1

for "--keep" (don't delete the rootfs)?

-serge

[John]

Post by Serge Hallyn

Post by John

Post by Serge Hallyn

Post by John

Post by Dan Kegel

Post by John
While on the subject, any reason for lxc-destroy now being destructive?

Wait, isn't that the point? It's in the name and all.
When was it ever nondestructive?

It only destroyed the configuration in /var/lib and never deleted the
root filesystem until very recently (0.8.0, I guess).

Was your rootfs a symbolic link by chance? I'm guessing commit
55116c42e767ce795f796fc51cd2ef7d76cf18af is what you're seeing. Before
that it did remove the rootfs, but if your rootfs was a symlink it
happened to not do it. That wasn't by intent.
Perhaps lxc-destroy should take a flag to not delete the rootfs? Not
sure...

Ah, I can now see what is wrong. It isn't down to symlinks but
beacuase my rootfs isn't under /var/lib/lxc.
Looking at that commit, I can see that the remove (on line 126)
deletes "$lxc_path/$lxc_name" but does not explictly remove
"$rootdev". The new code added at line 122 does indeed remove
$rootdev.
In my case I have my container rootfs in a directory called
"/srv/test.i686" (i.e not underneath $lxc_path - /var/lib/lxc). I
guess the design assumes that a template is used to create a
container and that it would put the rootfs beneath /var/lib/test.
1. what is the correct way to update a container config without
removing the rootfs. I have always used destroy/create to do this
but that, clearly, won't work if the destroy phase removes the
rootfs. I like being able to separately manage the rootfs from its
configuration.

This I don't really understand - I've always done it by hand. What
exactly is made easier by doing destroy/create? Maybe we can reproduce
that with an lxc-update or something... Especially if we can then
have lxc-update expand variables and take a list of containers to
update to batch the operations. Though still right now I would just
default to a bash loop calling sed...

I always treated /var/lib/lxc as "internal". From the early days,
/etc/lxc was suggested as a configuration directory and where the
original configuration would lie. Using lxc-create copied that config
into /var/lxc. This, in my mind, meant that I shouldn't mess with the
config inside /var/lib/lxc but should instead modify /etc/lxc and then
do a destroy/create. I may have been living on a mis-premise all this
time but that's how I've been using it.

[...]

Post by Serge Hallyn

Post by John
I would value having options to preserve the rootfs when doing
lxc-destroy and for lxc-create to use an existing rootfs (i.e.
instead of a template).

Ok, I don't *really* want to make lxc-destroy not delete the rootfs
just if it is outside of /var/lib/lxc/$container... On the one hand
I can see people could do that specifically in the hopes of making
it outlive the container. On the other hand I could see people doing
it only because they are short on disk space ending up running out of
disk space because they lost track of where the undeleted rootfs's
were.
Maybe
lxc-destroy -k -n p1
for "--keep" (don't delete the rootfs)?

yes, that would work.

Post by Serge Hallyn
-serge

[Serge Hallyn]

Post by John

Post by Serge Hallyn

Post by John

Post by Serge Hallyn

Post by John

Post by Dan Kegel

Post by John
While on the subject, any reason for lxc-destroy now being destructive?

Wait, isn't that the point? It's in the name and all.
When was it ever nondestructive?

It only destroyed the configuration in /var/lib and never deleted the
root filesystem until very recently (0.8.0, I guess).

Was your rootfs a symbolic link by chance? I'm guessing commit
55116c42e767ce795f796fc51cd2ef7d76cf18af is what you're seeing. Before
that it did remove the rootfs, but if your rootfs was a symlink it
happened to not do it. That wasn't by intent.
Perhaps lxc-destroy should take a flag to not delete the rootfs? Not
sure...

Ah, I can now see what is wrong. It isn't down to symlinks but
beacuase my rootfs isn't under /var/lib/lxc.
Looking at that commit, I can see that the remove (on line 126)
deletes "$lxc_path/$lxc_name" but does not explictly remove
"$rootdev". The new code added at line 122 does indeed remove
$rootdev.
In my case I have my container rootfs in a directory called
"/srv/test.i686" (i.e not underneath $lxc_path - /var/lib/lxc). I
guess the design assumes that a template is used to create a
container and that it would put the rootfs beneath /var/lib/test.
1. what is the correct way to update a container config without
removing the rootfs. I have always used destroy/create to do this
but that, clearly, won't work if the destroy phase removes the
rootfs. I like being able to separately manage the rootfs from its
configuration.

This I don't really understand - I've always done it by hand. What
exactly is made easier by doing destroy/create? Maybe we can reproduce
that with an lxc-update or something... Especially if we can then
have lxc-update expand variables and take a list of containers to
update to batch the operations. Though still right now I would just
default to a bash loop calling sed...

I always treated /var/lib/lxc as "internal". From the early days,
/etc/lxc was suggested as a configuration directory and where the
original configuration would lie. Using lxc-create copied that config
into /var/lxc. This, in my mind, meant that I shouldn't mess with the
config inside /var/lib/lxc but should instead modify /etc/lxc and then
do a destroy/create. I may have been living on a mis-premise all this
time but that's how I've been using it.

No actually I like that. I've not done it, but it seems sound.

Post by John

Post by Serge Hallyn

Post by John
I would value having options to preserve the rootfs when doing
lxc-destroy and for lxc-create to use an existing rootfs (i.e.
instead of a template).

Ok, I don't *really* want to make lxc-destroy not delete the rootfs
just if it is outside of /var/lib/lxc/$container... On the one hand
I can see people could do that specifically in the hopes of making
it outlive the container. On the other hand I could see people doing
it only because they are short on disk space ending up running out of
disk space because they lost track of where the undeleted rootfs's
were.
Maybe
lxc-destroy -k -n p1
for "--keep" (don't delete the rootfs)?

yes, that would work.

Great - if someone gets to writing that patch before I do, +1.

-serge

[John]

Hi everyone,

I haven't yet tried this yet but thought I'd ask first because it seems
a bit crazy, but...

Is it possible to configure a container so that it has a bridge in it ?

Reason for asking is this: I have a container that runs a desktop
environment. I want to run VirtualBox on that desktop, which I have
installed and it's working fine with NAT networking. However, I realise
I need a bridge for any vm to be able to do anything useful on the network.

So, can I configure a container so that it has a bridged network adapter
so that I can select it as a "bridged adapter" in VirtualBox's vm
network settings, either the same bridge as on the host or another one
within the container that is somehow connected to the lan ?

Many thanks,
John
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 483 bytes
Desc: not available
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20130105/dc289587/attachment.pgp>

[Serge Hallyn]

Post by John
Hi everyone,
I haven't yet tried this yet but thought I'd ask first because it

Why not? :)

Post by John
seems a bit crazy, but...

Not at all.

Post by John
Is it possible to configure a container so that it has a bridge in it ?

Yup. The container has a full l2 networking stack. You can create a
bridge and bridge eth0 with no problems.

-serge

[Michael H. Warfield]

Post by Michael H. Warfield
I've got some more problems relating to shutting down containers, some
of which may be related to mounting tmpfs on /run to which /var/run is
symlinked to. We're doing halt / restart detection by monitoring utmp
in that directory but it looks like utmp isn't even in that directory
anymore and mounting tmpfs on it was always problematical. We may have
to have a more generic method to detect when a container has shut down
or is restarting in that case.

I can't parse this. The system call reboot() is virtualized for
containers just fine and the container managaer (i.e. LXC) can check for
that easily.

I strongly suspect that the condition I'm dealing with (not being able
to restart the container) is an artifact of the devtmpfs kludge. I'm
seeing some errors relating to /dev/loop* busy that seems to be related
to the hung resources resulting in the inability to remove the zombie
container. Disregard until I can get further information following a
switch to a template based setup.

Lennart

Regards,
Mike
--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121026/100a6a85/attachment.pgp>

[Michael H. Warfield]

Note that there are reports that LXC has issues with the fact that newer
systemd enables shared mount propagation for all mounts by default (this
should actually be beneficial for containers as this ensures that new
mounts appear in the containers). LXC when run on such a system fails as
soon as it tries to use pivot_root(), as that is incompatible with
shared mount propagation. The needs fixing in LXC: it should use MS_MOVE
or MS_BIND to place the new root dir in / instead. A short term
work-around is to simply remount the root tree to private before
invoking LXC.

In another thread, Serge had some heartburn over this shared mount
propagation which then rang a bell in my head about past problems we
have seen.

...
This was from another threat with the systemd guys.

Note that there are reports that LXC has issues with the fact that
newer
systemd enables shared mount propagation for all mounts by default
(this
should actually be beneficial for containers as this ensures that new
mounts appear in the containers). LXC when run on such a system fails

MS_SLAVE does this as well. MS_SHARED means container mounts also
propagate into the host, which is less desirable in most cases.

Here's where we've seen some problems in the past. It's not just mounts
that are propagated but remounts as well. The problem arose that some
of us had our containers on a separate partition. When we would shut a
container down, that container tried to remount its file systems ro
which then propagated back into the host causing the hosts file system
to be ro (doesn't happen if you are running on the host's root fs for
the containers) and from there across into the other containers.

Are you using MS_SHARED or MS_SLAVE for this? If you are using
MS_SHARED do you create a potential security problem where actions in
the container can bleed into the state of the host and into other
containers. That's highly undesirable. If a mount in a propagates back
into the host and is then reflected to another container sharing that
same mount tree (I have shared partitions specific to that sort of
thing) does that create an information disclosure situation of one
container mounts a new file system and the other container sees the new
mount? I don't know if the mount propagation would reflect back up the
shared tree or not but I have certainly seen remounts do this. I don't
see that as desirable. Maybe I'm misunderstand how this is suppose to
work but I intend to test out those scenarios when I have a chance. I
do know that, when testing that ro problem, I was able to remount a
partition ro in one container and it would switch in the host and the
other container and I could the remount it rw in the other container and
have it propagate back. Not good.

Can you offer any clarity on this?

as
soon as it tries to use pivot_root(), as that is incompatible with
shared mount propagation. The needs fixing in LXC: it should use
MS_MOVE
or MS_BIND to place the new root dir in / instead. A short term

Actually not quite sure how this would work. It should be possible
to set up a set of conditions to work around this, but the kernel
checks at do_pivotroot are pretty harsh - mnt->mnt_parent of both
the new root and current root have to be not shared. So perhaps
we actually first chroot into a dir whose parent is non-shared,
then pivot_root from there? :)
(Simple chroot in place of pivot_root still does not suffice, not
only because of chroot escapes, but also different results in
/proc/pid/mountinfo and friends)

Comments on Serge's points?

At this point, we see where this will become problematical in Fedora 18
but appears to already be problematical in NixOS that another user is
running and which containers systemd 195 in the host.

We've had problems with chroot in the past due to chroot escapes and
other problems years ago as Serge mentioned.

Lennart
--
Lennart Poettering - Red Hat, Inc.

Regards,
Mike
--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121106/ad61fe80/attachment.pgp>

[Benoit Lourdelet]

Hello,

On fedora 18 Linux localhost.localdomain 3.6.7-5.fc18.x86_64 #1 SMP Tue
Nov 20 19:40:08 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
LXC version 0.8.0

After creating a container with lxc-fedora:

lxc.utsname = test1-18
lxc.tty = 4
lxc.pts = 1024
lxc.rootfs = /var/lib/lxc/test1-18/rootfs
lxc.mount = /var/lib/lxc/test1-18/fstab

# When using LXC with apparmor, uncomment the next line to run unconfined:
#lxc.aa_profile = unconfined

#cgroups
lxc.cgroup.devices.deny = a
# /dev/null and zero
lxc.cgroup.devices.allow = c 1:3 rwm
lxc.cgroup.devices.allow = c 1:5 rwm
# consoles
lxc.cgroup.devices.allow = c 5:1 rwm
lxc.cgroup.devices.allow = c 5:0 rwm
lxc.cgroup.devices.allow = c 4:0 rwm
lxc.cgroup.devices.allow = c 4:1 rwm
# /dev/{,u}random
lxc.cgroup.devices.allow = c 1:9 rwm
lxc.cgroup.devices.allow = c 1:8 rwm
lxc.cgroup.devices.allow = c 136:* rwm
lxc.cgroup.devices.allow = c 5:2 rwm
# rtc
lxc.cgroup.devices.allow = c 254:0 rwm



The container starts but always sends the prompt to the system console.

[root at localhost test1-18]# /usr/local/bin/lxc-info -n test1-18
state: RUNNING
pid: 16713



Trying to access the console, does not give a prompt :


/usr/local/bin/lxc-console -n test1-18

Type <Ctrl+a q> to exit the console, <Ctrl+a Ctrl+a> to enter Ctrl+a itself




Log file is as follow :

lxc-start 1354793173.737 INFO lxc_apparmor - apparmor_load -
apparmor is disabled
lxc-start 1354793173.738 DEBUG lxc_conf - allocated pty
'/dev/pts/4' (4/5)
lxc-start 1354793173.738 DEBUG lxc_conf - allocated pty
'/dev/pts/5' (6/7)
lxc-start 1354793173.738 DEBUG lxc_conf - allocated pty
'/dev/pts/6' (8/9)
lxc-start 1354793173.738 DEBUG lxc_conf - allocated pty
'/dev/pts/7' (10/11)
lxc-start 1354793173.738 INFO lxc_conf - tty's configured
lxc-start 1354793173.739 DEBUG lxc_console - using '/dev/tty' as
console
lxc-start 1354793173.739 DEBUG lxc_start - sigchild handler set
lxc-start 1354793173.739 INFO lxc_start - 'test1-18' is
initialized
lxc-start 1354793173.751 DEBUG lxc_start - Not dropping
cap_sys_boot or watching utmp

lxc-start 1354793173.751 INFO lxc_conf - opened
/var/lib/lxc/test1-18/rootfs.hold as fd 18

lxc-start 1354793173.751 DEBUG lxc_cgroup - checking '/' (rootfs)
lxc-start 1354793173.751 DEBUG lxc_cgroup - checking '/proc'
(proc)
lxc-start 1354793173.751 DEBUG lxc_cgroup - checking '/sys'
(sysfs)
lxc-start 1354793173.751 DEBUG lxc_cgroup - checking '/dev'
(devtmpfs)
lxc-start 1354793173.751 DEBUG lxc_cgroup - checking
'/sys/kernel/security' (securityfs)
lxc-start 1354793173.751 DEBUG lxc_cgroup - checking
'/sys/fs/selinux' (selinuxfs)
lxc-start 1354793173.751 DEBUG lxc_cgroup - checking '/dev/shm'
(tmpfs)
lxc-start 1354793173.752 DEBUG lxc_cgroup - checking '/dev/pts'
(devpts)
lxc-start 1354793173.752 DEBUG lxc_cgroup - checking '/run'
(tmpfs)
lxc-start 1354793173.752 DEBUG lxc_cgroup - checking
'/sys/fs/cgroup' (tmpfs)
lxc-start 1354793173.752 DEBUG lxc_cgroup - checking
'/sys/fs/cgroup/systemd' (cgroup)
lxc-start 1354793173.752 DEBUG lxc_cgroup - checking
'/sys/fs/cgroup/cpuset' (cgroup)
lxc-start 1354793173.752 INFO lxc_cgroup - [1] found cgroup
mounted at
'/sys/fs/cgroup/cpuset',opts='rw,nosuid,nodev,noexec,relatime,cpuset,clone_
children'
lxc-start 1354793173.752 DEBUG lxc_cgroup - get_init_cgroup:
found init cgroup for subsys (null) at /


lxc-start 1354793173.752 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/cpuset has flags 0x2
lxc-start 1354793173.752 INFO lxc_cgroup - created cgroup
'/sys/fs/cgroup/cpuset//lxc/test1-18'
lxc-start 1354793173.752 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/cpuset has flags 0x2
lxc-start 1354793173.752 DEBUG lxc_cgroup - get_init_cgroup:
found init cgroup for subsys (null) at /


lxc-start 1354793173.765 DEBUG lxc_cgroup - checking
'/sys/fs/cgroup/cpu,cpuacct' (cgroup)
lxc-start 1354793173.765 INFO lxc_cgroup - [2] found cgroup
mounted at
'/sys/fs/cgroup/cpu,cpuacct',opts='rw,nosuid,nodev,noexec,relatime,cpuacct,
cpu,clone_children'
lxc-start 1354793173.765 DEBUG lxc_cgroup - get_init_cgroup:
found init cgroup for subsys (null) at /


lxc-start 1354793173.765 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/cpu,cpuacct has flags 0x2
lxc-start 1354793173.765 INFO lxc_cgroup - created cgroup
'/sys/fs/cgroup/cpu,cpuacct//lxc/test1-18'
lxc-start 1354793173.765 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/cpu,cpuacct has flags 0x2
lxc-start 1354793173.765 DEBUG lxc_cgroup - get_init_cgroup:
found init cgroup for subsys (null) at /


lxc-start 1354793173.775 DEBUG lxc_cgroup - checking
'/sys/fs/cgroup/memory' (cgroup)
lxc-start 1354793173.775 INFO lxc_cgroup - [3] found cgroup
mounted at
'/sys/fs/cgroup/memory',opts='rw,nosuid,nodev,noexec,relatime,memory,clone_
children'
lxc-start 1354793173.775 DEBUG lxc_cgroup - get_init_cgroup:
found init cgroup for subsys (null) at /


lxc-start 1354793173.775 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/memory has flags 0x2
lxc-start 1354793173.775 INFO lxc_cgroup - created cgroup
'/sys/fs/cgroup/memory//lxc/test1-18'
lxc-start 1354793173.775 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/memory has flags 0x2
lxc-start 1354793173.775 DEBUG lxc_cgroup - get_init_cgroup:
found init cgroup for subsys (null) at /


lxc-start 1354793173.783 DEBUG lxc_cgroup - checking
'/sys/fs/cgroup/devices' (cgroup)
lxc-start 1354793173.783 INFO lxc_cgroup - [4] found cgroup
mounted at
'/sys/fs/cgroup/devices',opts='rw,nosuid,nodev,noexec,relatime,devices,clon
e_children'
lxc-start 1354793173.783 DEBUG lxc_cgroup - get_init_cgroup:
found init cgroup for subsys (null) at /


lxc-start 1354793173.783 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/devices has flags 0x2
lxc-start 1354793173.783 INFO lxc_cgroup - created cgroup
'/sys/fs/cgroup/devices//lxc/test1-18'
lxc-start 1354793173.783 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/devices has flags 0x2
lxc-start 1354793173.783 DEBUG lxc_cgroup - get_init_cgroup:
found init cgroup for subsys (null) at /


lxc-start 1354793173.788 DEBUG lxc_cgroup - checking
'/sys/fs/cgroup/freezer' (cgroup)
lxc-start 1354793173.788 INFO lxc_cgroup - [5] found cgroup
mounted at
'/sys/fs/cgroup/freezer',opts='rw,nosuid,nodev,noexec,relatime,freezer,clon
e_children'
lxc-start 1354793173.788 DEBUG lxc_cgroup - get_init_cgroup:
found init cgroup for subsys (null) at /


lxc-start 1354793173.788 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/freezer has flags 0x2
lxc-start 1354793173.788 INFO lxc_cgroup - created cgroup
'/sys/fs/cgroup/freezer//lxc/test1-18'
lxc-start 1354793173.788 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/freezer has flags 0x2
lxc-start 1354793173.788 DEBUG lxc_cgroup - get_init_cgroup:
found init cgroup for subsys (null) at /


lxc-start 1354793173.793 DEBUG lxc_cgroup - checking
'/sys/fs/cgroup/net_cls' (cgroup)
lxc-start 1354793173.793 INFO lxc_cgroup - [6] found cgroup
mounted at
'/sys/fs/cgroup/net_cls',opts='rw,nosuid,nodev,noexec,relatime,net_cls,clon
e_children'
lxc-start 1354793173.793 DEBUG lxc_cgroup - get_init_cgroup:
found init cgroup for subsys (null) at /


lxc-start 1354793173.793 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/net_cls has flags 0x2
lxc-start 1354793173.793 INFO lxc_cgroup - created cgroup
'/sys/fs/cgroup/net_cls//lxc/test1-18'
lxc-start 1354793173.793 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/net_cls has flags 0x2
lxc-start 1354793173.793 DEBUG lxc_cgroup - get_init_cgroup:
found init cgroup for subsys (null) at /


lxc-start 1354793173.798 DEBUG lxc_cgroup - checking
'/sys/fs/cgroup/blkio' (cgroup)
lxc-start 1354793173.798 INFO lxc_cgroup - [7] found cgroup
mounted at
'/sys/fs/cgroup/blkio',opts='rw,nosuid,nodev,noexec,relatime,blkio,clone_ch
ildren'
lxc-start 1354793173.798 DEBUG lxc_cgroup - get_init_cgroup:
found init cgroup for subsys (null) at /


lxc-start 1354793173.798 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/blkio has flags 0x2
lxc-start 1354793173.798 INFO lxc_cgroup - created cgroup
'/sys/fs/cgroup/blkio//lxc/test1-18'
lxc-start 1354793173.798 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/blkio has flags 0x2
lxc-start 1354793173.798 DEBUG lxc_cgroup - get_init_cgroup:
found init cgroup for subsys (null) at /


lxc-start 1354793173.803 DEBUG lxc_cgroup - checking
'/sys/fs/cgroup/perf_event' (cgroup)
lxc-start 1354793173.803 INFO lxc_cgroup - [8] found cgroup
mounted at
'/sys/fs/cgroup/perf_event',opts='rw,nosuid,nodev,noexec,relatime,perf_even
t,clone_children'
lxc-start 1354793173.803 DEBUG lxc_cgroup - get_init_cgroup:
found init cgroup for subsys (null) at /


lxc-start 1354793173.803 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/perf_event has flags 0x2
lxc-start 1354793173.803 INFO lxc_cgroup - created cgroup
'/sys/fs/cgroup/perf_event//lxc/test1-18'
lxc-start 1354793173.803 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/perf_event has flags 0x2
lxc-start 1354793173.803 DEBUG lxc_cgroup - get_init_cgroup:
found init cgroup for subsys (null) at /


lxc-start 1354793173.808 DEBUG lxc_cgroup - checking
'/var/lib/nfs/rpc_pipefs' (rpc_pipefs)
lxc-start 1354793173.808 DEBUG lxc_cgroup - checking '/' (ext4)
lxc-start 1354793173.808 DEBUG lxc_cgroup - checking
'/proc/sys/fs/binfmt_misc' (autofs)
lxc-start 1354793173.808 DEBUG lxc_cgroup - checking
'/dev/mqueue' (mqueue)
lxc-start 1354793173.808 DEBUG lxc_cgroup - checking
'/sys/kernel/debug' (debugfs)
lxc-start 1354793173.808 DEBUG lxc_cgroup - checking
'/dev/hugepages' (hugetlbfs)
lxc-start 1354793173.808 DEBUG lxc_cgroup - checking '/tmp'
(tmpfs)
lxc-start 1354793173.808 DEBUG lxc_cgroup - checking
'/sys/kernel/config' (configfs)
lxc-start 1354793173.808 DEBUG lxc_cgroup - checking '/boot'
(ext4)
lxc-start 1354793173.808 DEBUG lxc_cgroup - checking '/home'
(ext4)
lxc-start 1354793173.808 DEBUG lxc_cgroup - checking '/boot/efi'
(vfat)
lxc-start 1354793173.808 DEBUG lxc_cgroup - checking
'/sys/fs/fuse/connections' (fusectl)
lxc-start 1354793173.808 DEBUG lxc_cgroup - checking
'/proc/sys/fs/binfmt_misc' (binfmt_misc)
lxc-start 1354793173.808 DEBUG lxc_cgroup - checking
'/run/user/1000/gvfs' (fuse.gvfsd-fuse)
lxc-start 1354793173.808 INFO lxc_conf - 'test1-18' hostname has
been setup
lxc-start 1354793173.808 DEBUG lxc_conf - mounted
'/var/lib/lxc/test1-18/rootfs' on '/usr/local/lib/lxc/rootfs'
lxc-start 1354793173.808 DEBUG lxc_conf - mounted 'proc' on
'/usr/local/lib/lxc/rootfs/proc', type 'proc'
lxc-start 1354793173.808 DEBUG lxc_conf - mounted 'sysfs' on
'/usr/local/lib/lxc/rootfs/sys', type 'sysfs'
lxc-start 1354793173.808 INFO lxc_conf - mount points have been
setup
lxc-start 1354793173.809 INFO lxc_conf - mount points have been
setup
lxc-start 1354793173.809 DEBUG lxc_cgroup - lxc_cgroup_path_get:
called for subsys devices name test1-18

lxc-start 1354793173.809 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/devices has flags 0x2
lxc-start 1354793173.809 DEBUG lxc_cgroup - get_init_cgroup:
found init cgroup for subsys devices at /


lxc-start 1354793173.809 DEBUG lxc_cgroup - using cgroup mounted
at '/sys/fs/cgroup/devices//lxc'
lxc-start 1354793173.809 DEBUG lxc_cgroup - lxc_cgroup_path_get:
returning /sys/fs/cgroup/devices//lxc/test1-18 for subsystem devices.deny
lxc-start 1354793173.809 DEBUG lxc_conf - cgroup 'devices.deny'
set to 'a'
lxc-start 1354793173.809 DEBUG lxc_cgroup - lxc_cgroup_path_get:
called for subsys devices name test1-18

lxc-start 1354793173.809 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/devices has flags 0x2
lxc-start 1354793173.809 DEBUG lxc_cgroup - get_init_cgroup:
found init cgroup for subsys devices at /


lxc-start 1354793173.809 DEBUG lxc_cgroup - using cgroup mounted
at '/sys/fs/cgroup/devices//lxc'
lxc-start 1354793173.809 DEBUG lxc_cgroup - lxc_cgroup_path_get:
returning /sys/fs/cgroup/devices//lxc/test1-18 for subsystem devices.allow
lxc-start 1354793173.809 DEBUG lxc_conf - cgroup 'devices.allow'
set to 'c 1:3 rwm'
lxc-start 1354793173.809 DEBUG lxc_cgroup - lxc_cgroup_path_get:
called for subsys devices name test1-18

lxc-start 1354793173.809 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/devices has flags 0x2
lxc-start 1354793173.809 DEBUG lxc_cgroup - get_init_cgroup:
found init cgroup for subsys devices at /


lxc-start 1354793173.809 DEBUG lxc_cgroup - using cgroup mounted
at '/sys/fs/cgroup/devices//lxc'
lxc-start 1354793173.809 DEBUG lxc_cgroup - lxc_cgroup_path_get:
returning /sys/fs/cgroup/devices//lxc/test1-18 for subsystem devices.allow
lxc-start 1354793173.809 DEBUG lxc_conf - cgroup 'devices.allow'
set to 'c 1:5 rwm'
lxc-start 1354793173.809 DEBUG lxc_cgroup - lxc_cgroup_path_get:
called for subsys devices name test1-18

lxc-start 1354793173.809 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/devices has flags 0x2
lxc-start 1354793173.809 DEBUG lxc_cgroup - get_init_cgroup:
found init cgroup for subsys devices at /


lxc-start 1354793173.809 DEBUG lxc_cgroup - using cgroup mounted
at '/sys/fs/cgroup/devices//lxc'
lxc-start 1354793173.809 DEBUG lxc_cgroup - lxc_cgroup_path_get:
returning /sys/fs/cgroup/devices//lxc/test1-18 for subsystem devices.allow
lxc-start 1354793173.809 DEBUG lxc_conf - cgroup 'devices.allow'
set to 'c 5:1 rwm'
lxc-start 1354793173.809 DEBUG lxc_cgroup - lxc_cgroup_path_get:
called for subsys devices name test1-18

lxc-start 1354793173.810 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/devices has flags 0x2
lxc-start 1354793173.810 DEBUG lxc_cgroup - get_init_cgroup:
found init cgroup for subsys devices at /


lxc-start 1354793173.810 DEBUG lxc_cgroup - using cgroup mounted
at '/sys/fs/cgroup/devices//lxc'
lxc-start 1354793173.810 DEBUG lxc_cgroup - lxc_cgroup_path_get:
returning /sys/fs/cgroup/devices//lxc/test1-18 for subsystem devices.allow
lxc-start 1354793173.810 DEBUG lxc_conf - cgroup 'devices.allow'
set to 'c 5:0 rwm'
lxc-start 1354793173.810 DEBUG lxc_cgroup - lxc_cgroup_path_get:
called for subsys devices name test1-18

lxc-start 1354793173.810 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/devices has flags 0x2
lxc-start 1354793173.810 DEBUG lxc_cgroup - get_init_cgroup:
found init cgroup for subsys devices at /


lxc-start 1354793173.810 DEBUG lxc_cgroup - using cgroup mounted
at '/sys/fs/cgroup/devices//lxc'
lxc-start 1354793173.810 DEBUG lxc_cgroup - lxc_cgroup_path_get:
returning /sys/fs/cgroup/devices//lxc/test1-18 for subsystem devices.allow
lxc-start 1354793173.810 DEBUG lxc_conf - cgroup 'devices.allow'
set to 'c 4:0 rwm'
lxc-start 1354793173.810 DEBUG lxc_cgroup - lxc_cgroup_path_get:
called for subsys devices name test1-18

lxc-start 1354793173.810 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/devices has flags 0x2
lxc-start 1354793173.810 DEBUG lxc_cgroup - get_init_cgroup:
found init cgroup for subsys devices at /


lxc-start 1354793173.810 DEBUG lxc_cgroup - using cgroup mounted
at '/sys/fs/cgroup/devices//lxc'
lxc-start 1354793173.810 DEBUG lxc_cgroup - lxc_cgroup_path_get:
returning /sys/fs/cgroup/devices//lxc/test1-18 for subsystem devices.allow
lxc-start 1354793173.810 DEBUG lxc_conf - cgroup 'devices.allow'
set to 'c 4:1 rwm'
lxc-start 1354793173.810 DEBUG lxc_cgroup - lxc_cgroup_path_get:
called for subsys devices name test1-18

lxc-start 1354793173.810 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/devices has flags 0x2
lxc-start 1354793173.810 DEBUG lxc_cgroup - get_init_cgroup:
found init cgroup for subsys devices at /


lxc-start 1354793173.810 DEBUG lxc_cgroup - using cgroup mounted
at '/sys/fs/cgroup/devices//lxc'
lxc-start 1354793173.810 DEBUG lxc_cgroup - lxc_cgroup_path_get:
returning /sys/fs/cgroup/devices//lxc/test1-18 for subsystem devices.allow
lxc-start 1354793173.810 DEBUG lxc_conf - cgroup 'devices.allow'
set to 'c 1:9 rwm'
lxc-start 1354793173.810 DEBUG lxc_cgroup - lxc_cgroup_path_get:
called for subsys devices name test1-18

lxc-start 1354793173.810 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/devices has flags 0x2
lxc-start 1354793173.810 DEBUG lxc_cgroup - get_init_cgroup:
found init cgroup for subsys devices at /


lxc-start 1354793173.810 DEBUG lxc_cgroup - using cgroup mounted
at '/sys/fs/cgroup/devices//lxc'
lxc-start 1354793173.810 DEBUG lxc_cgroup - lxc_cgroup_path_get:
returning /sys/fs/cgroup/devices//lxc/test1-18 for subsystem devices.allow
lxc-start 1354793173.810 DEBUG lxc_conf - cgroup 'devices.allow'
set to 'c 1:8 rwm'
lxc-start 1354793173.810 DEBUG lxc_cgroup - lxc_cgroup_path_get:
called for subsys devices name test1-18

lxc-start 1354793173.810 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/devices has flags 0x2
lxc-start 1354793173.810 DEBUG lxc_cgroup - get_init_cgroup:
found init cgroup for subsys devices at /


lxc-start 1354793173.810 DEBUG lxc_cgroup - using cgroup mounted
at '/sys/fs/cgroup/devices//lxc'
lxc-start 1354793173.810 DEBUG lxc_cgroup - lxc_cgroup_path_get:
returning /sys/fs/cgroup/devices//lxc/test1-18 for subsystem devices.allow
lxc-start 1354793173.810 DEBUG lxc_conf - cgroup 'devices.allow'
set to 'c 136:* rwm'
lxc-start 1354793173.810 DEBUG lxc_cgroup - lxc_cgroup_path_get:
called for subsys devices name test1-18

lxc-start 1354793173.810 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/devices has flags 0x2
lxc-start 1354793173.810 DEBUG lxc_cgroup - get_init_cgroup:
found init cgroup for subsys devices at /


lxc-start 1354793173.810 DEBUG lxc_cgroup - using cgroup mounted
at '/sys/fs/cgroup/devices//lxc'
lxc-start 1354793173.810 DEBUG lxc_cgroup - lxc_cgroup_path_get:
returning /sys/fs/cgroup/devices//lxc/test1-18 for subsystem devices.allow
lxc-start 1354793173.810 DEBUG lxc_conf - cgroup 'devices.allow'
set to 'c 5:2 rwm'
lxc-start 1354793173.811 DEBUG lxc_cgroup - lxc_cgroup_path_get:
called for subsys devices name test1-18

lxc-start 1354793173.811 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/devices has flags 0x2
lxc-start 1354793173.811 DEBUG lxc_cgroup - get_init_cgroup:
found init cgroup for subsys devices at /


lxc-start 1354793173.811 DEBUG lxc_cgroup - using cgroup mounted
at '/sys/fs/cgroup/devices//lxc'
lxc-start 1354793173.811 DEBUG lxc_cgroup - lxc_cgroup_path_get:
returning /sys/fs/cgroup/devices//lxc/test1-18 for subsystem devices.allow
lxc-start 1354793173.811 DEBUG lxc_conf - cgroup 'devices.allow'
set to 'c 254:0 rwm'
lxc-start 1354793173.811 INFO lxc_conf - cgroup has been setup
lxc-start 1354793173.811 INFO lxc_conf - console has been setup
lxc-start 1354793173.811 INFO lxc_conf - 4 tty(s) has been setup
lxc-start 1354793173.811 DEBUG lxc_conf - created
'/usr/local/lib/lxc/rootfs/lxc_putold' directory
lxc-start 1354793173.811 DEBUG lxc_conf - mountpoint for old
rootfs is '/usr/local/lib/lxc/rootfs/lxc_putold'
lxc-start 1354793173.811 DEBUG lxc_conf - pivot_root syscall to
'/usr/local/lib/lxc/rootfs' successful
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/dev/shm'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/dev/pts'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/dev/mqueue'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/dev/hugepages'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/proc/sys/fs/binfmt_misc'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/sys/kernel/security'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/sys/fs/selinux'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/sys/fs/cgroup/systemd'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/sys/fs/cgroup/cpuset'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/sys/fs/cgroup/cpu,cpuacct'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/sys/fs/cgroup/memory'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/sys/fs/cgroup/devices'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/sys/fs/cgroup/freezer'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/sys/fs/cgroup/net_cls'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/sys/fs/cgroup/blkio'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/sys/fs/cgroup/perf_event'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/sys/kernel/debug'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/sys/kernel/config'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/sys/fs/fuse/connections'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/run/user/1000/gvfs'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/tmp'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/boot/efi'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/home'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/dev'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/sys/fs/cgroup'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/run'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/boot'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/sys'
lxc-start 1354793173.812 INFO lxc_conf - lazy unmount of
'/lxc_putold'
lxc-start 1354793173.812 WARN lxc_conf - failed to unmount
'/lxc_putold/proc'
lxc-start 1354793173.813 INFO lxc_conf - created new pts instance
lxc-start 1354793173.813 DEBUG lxc_conf - capabilities has been
setup
lxc-start 1354793173.813 NOTICE lxc_conf - 'test1-18' is setup.
lxc-start 1354793173.813 NOTICE lxc_start - exec'ing '/sbin/init'
lxc-start 1354793173.813 NOTICE lxc_start - '/sbin/init' started
with pid '17431'
lxc-start 1354793173.813 WARN lxc_start - invalid pid for SIGCHLD





tahnks

Benoit

[Michael H. Warfield]

Post by Michael H. Warfield
Hello,
On fedora 18 Linux localhost.localdomain 3.6.7-5.fc18.x86_64 #1 SMP Tue
Nov 20 19:40:08 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
LXC version 0.8.0

This is currently a known problem with running systemd in a container.
I haven't actually tested Fedora 18 yet and I'm sort of surprised you
didn't run into the pivot root problem associated with that version of
systemd in the host. Fixes for the console collision problem are in git
and being worked on right now. Looks like we're targetting 0.9.0 for
the systemd in a container problem and may have an alpha out for it in a
few days. If you run into the pivot root problem, there's a workaround
for that mentioned in some other threads.

Regards,
Mike

Post by Michael H. Warfield
lxc.utsname = test1-18
lxc.tty = 4
lxc.pts = 1024
lxc.rootfs = /var/lib/lxc/test1-18/rootfs
lxc.mount = /var/lib/lxc/test1-18/fstab
#lxc.aa_profile = unconfined
#cgroups
lxc.cgroup.devices.deny = a
# /dev/null and zero
lxc.cgroup.devices.allow = c 1:3 rwm
lxc.cgroup.devices.allow = c 1:5 rwm
# consoles
lxc.cgroup.devices.allow = c 5:1 rwm
lxc.cgroup.devices.allow = c 5:0 rwm
lxc.cgroup.devices.allow = c 4:0 rwm
lxc.cgroup.devices.allow = c 4:1 rwm
# /dev/{,u}random
lxc.cgroup.devices.allow = c 1:9 rwm
lxc.cgroup.devices.allow = c 1:8 rwm
lxc.cgroup.devices.allow = c 136:* rwm
lxc.cgroup.devices.allow = c 5:2 rwm
# rtc
lxc.cgroup.devices.allow = c 254:0 rwm
The container starts but always sends the prompt to the system console.
[root at localhost test1-18]# /usr/local/bin/lxc-info -n test1-18
state: RUNNING
pid: 16713
/usr/local/bin/lxc-console -n test1-18
Type <Ctrl+a q> to exit the console, <Ctrl+a Ctrl+a> to enter Ctrl+a itself
lxc-start 1354793173.737 INFO lxc_apparmor - apparmor_load -
apparmor is disabled
lxc-start 1354793173.738 DEBUG lxc_conf - allocated pty
'/dev/pts/4' (4/5)
lxc-start 1354793173.738 DEBUG lxc_conf - allocated pty
'/dev/pts/5' (6/7)
lxc-start 1354793173.738 DEBUG lxc_conf - allocated pty
'/dev/pts/6' (8/9)
lxc-start 1354793173.738 DEBUG lxc_conf - allocated pty
'/dev/pts/7' (10/11)
lxc-start 1354793173.738 INFO lxc_conf - tty's configured
lxc-start 1354793173.739 DEBUG lxc_console - using '/dev/tty' as
console
lxc-start 1354793173.739 DEBUG lxc_start - sigchild handler set
lxc-start 1354793173.739 INFO lxc_start - 'test1-18' is
initialized
lxc-start 1354793173.751 DEBUG lxc_start - Not dropping
cap_sys_boot or watching utmp
lxc-start 1354793173.751 INFO lxc_conf - opened
/var/lib/lxc/test1-18/rootfs.hold as fd 18
lxc-start 1354793173.751 DEBUG lxc_cgroup - checking '/' (rootfs)
lxc-start 1354793173.751 DEBUG lxc_cgroup - checking '/proc'
(proc)
lxc-start 1354793173.751 DEBUG lxc_cgroup - checking '/sys'
(sysfs)
lxc-start 1354793173.751 DEBUG lxc_cgroup - checking '/dev'
(devtmpfs)
lxc-start 1354793173.751 DEBUG lxc_cgroup - checking
'/sys/kernel/security' (securityfs)
lxc-start 1354793173.751 DEBUG lxc_cgroup - checking
'/sys/fs/selinux' (selinuxfs)
lxc-start 1354793173.751 DEBUG lxc_cgroup - checking '/dev/shm'
(tmpfs)
lxc-start 1354793173.752 DEBUG lxc_cgroup - checking '/dev/pts'
(devpts)
lxc-start 1354793173.752 DEBUG lxc_cgroup - checking '/run'
(tmpfs)
lxc-start 1354793173.752 DEBUG lxc_cgroup - checking
'/sys/fs/cgroup' (tmpfs)
lxc-start 1354793173.752 DEBUG lxc_cgroup - checking
'/sys/fs/cgroup/systemd' (cgroup)
lxc-start 1354793173.752 DEBUG lxc_cgroup - checking
'/sys/fs/cgroup/cpuset' (cgroup)
lxc-start 1354793173.752 INFO lxc_cgroup - [1] found cgroup
mounted at
'/sys/fs/cgroup/cpuset',opts='rw,nosuid,nodev,noexec,relatime,cpuset,clone_
children'
found init cgroup for subsys (null) at /
lxc-start 1354793173.752 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/cpuset has flags 0x2
lxc-start 1354793173.752 INFO lxc_cgroup - created cgroup
'/sys/fs/cgroup/cpuset//lxc/test1-18'
lxc-start 1354793173.752 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/cpuset has flags 0x2
found init cgroup for subsys (null) at /
lxc-start 1354793173.765 DEBUG lxc_cgroup - checking
'/sys/fs/cgroup/cpu,cpuacct' (cgroup)
lxc-start 1354793173.765 INFO lxc_cgroup - [2] found cgroup
mounted at
'/sys/fs/cgroup/cpu,cpuacct',opts='rw,nosuid,nodev,noexec,relatime,cpuacct,
cpu,clone_children'
found init cgroup for subsys (null) at /
lxc-start 1354793173.765 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/cpu,cpuacct has flags 0x2
lxc-start 1354793173.765 INFO lxc_cgroup - created cgroup
'/sys/fs/cgroup/cpu,cpuacct//lxc/test1-18'
lxc-start 1354793173.765 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/cpu,cpuacct has flags 0x2
found init cgroup for subsys (null) at /
lxc-start 1354793173.775 DEBUG lxc_cgroup - checking
'/sys/fs/cgroup/memory' (cgroup)
lxc-start 1354793173.775 INFO lxc_cgroup - [3] found cgroup
mounted at
'/sys/fs/cgroup/memory',opts='rw,nosuid,nodev,noexec,relatime,memory,clone_
children'
found init cgroup for subsys (null) at /
lxc-start 1354793173.775 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/memory has flags 0x2
lxc-start 1354793173.775 INFO lxc_cgroup - created cgroup
'/sys/fs/cgroup/memory//lxc/test1-18'
lxc-start 1354793173.775 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/memory has flags 0x2
found init cgroup for subsys (null) at /
lxc-start 1354793173.783 DEBUG lxc_cgroup - checking
'/sys/fs/cgroup/devices' (cgroup)
lxc-start 1354793173.783 INFO lxc_cgroup - [4] found cgroup
mounted at
'/sys/fs/cgroup/devices',opts='rw,nosuid,nodev,noexec,relatime,devices,clon
e_children'
found init cgroup for subsys (null) at /
lxc-start 1354793173.783 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/devices has flags 0x2
lxc-start 1354793173.783 INFO lxc_cgroup - created cgroup
'/sys/fs/cgroup/devices//lxc/test1-18'
lxc-start 1354793173.783 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/devices has flags 0x2
found init cgroup for subsys (null) at /
lxc-start 1354793173.788 DEBUG lxc_cgroup - checking
'/sys/fs/cgroup/freezer' (cgroup)
lxc-start 1354793173.788 INFO lxc_cgroup - [5] found cgroup
mounted at
'/sys/fs/cgroup/freezer',opts='rw,nosuid,nodev,noexec,relatime,freezer,clon
e_children'
found init cgroup for subsys (null) at /
lxc-start 1354793173.788 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/freezer has flags 0x2
lxc-start 1354793173.788 INFO lxc_cgroup - created cgroup
'/sys/fs/cgroup/freezer//lxc/test1-18'
lxc-start 1354793173.788 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/freezer has flags 0x2
found init cgroup for subsys (null) at /
lxc-start 1354793173.793 DEBUG lxc_cgroup - checking
'/sys/fs/cgroup/net_cls' (cgroup)
lxc-start 1354793173.793 INFO lxc_cgroup - [6] found cgroup
mounted at
'/sys/fs/cgroup/net_cls',opts='rw,nosuid,nodev,noexec,relatime,net_cls,clon
e_children'
found init cgroup for subsys (null) at /
lxc-start 1354793173.793 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/net_cls has flags 0x2
lxc-start 1354793173.793 INFO lxc_cgroup - created cgroup
'/sys/fs/cgroup/net_cls//lxc/test1-18'
lxc-start 1354793173.793 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/net_cls has flags 0x2
found init cgroup for subsys (null) at /
lxc-start 1354793173.798 DEBUG lxc_cgroup - checking
'/sys/fs/cgroup/blkio' (cgroup)
lxc-start 1354793173.798 INFO lxc_cgroup - [7] found cgroup
mounted at
'/sys/fs/cgroup/blkio',opts='rw,nosuid,nodev,noexec,relatime,blkio,clone_ch
ildren'
found init cgroup for subsys (null) at /
lxc-start 1354793173.798 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/blkio has flags 0x2
lxc-start 1354793173.798 INFO lxc_cgroup - created cgroup
'/sys/fs/cgroup/blkio//lxc/test1-18'
lxc-start 1354793173.798 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/blkio has flags 0x2
found init cgroup for subsys (null) at /
lxc-start 1354793173.803 DEBUG lxc_cgroup - checking
'/sys/fs/cgroup/perf_event' (cgroup)
lxc-start 1354793173.803 INFO lxc_cgroup - [8] found cgroup
mounted at
'/sys/fs/cgroup/perf_event',opts='rw,nosuid,nodev,noexec,relatime,perf_even
t,clone_children'
found init cgroup for subsys (null) at /
lxc-start 1354793173.803 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/perf_event has flags 0x2
lxc-start 1354793173.803 INFO lxc_cgroup - created cgroup
'/sys/fs/cgroup/perf_event//lxc/test1-18'
lxc-start 1354793173.803 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/perf_event has flags 0x2
found init cgroup for subsys (null) at /
lxc-start 1354793173.808 DEBUG lxc_cgroup - checking
'/var/lib/nfs/rpc_pipefs' (rpc_pipefs)
lxc-start 1354793173.808 DEBUG lxc_cgroup - checking '/' (ext4)
lxc-start 1354793173.808 DEBUG lxc_cgroup - checking
'/proc/sys/fs/binfmt_misc' (autofs)
lxc-start 1354793173.808 DEBUG lxc_cgroup - checking
'/dev/mqueue' (mqueue)
lxc-start 1354793173.808 DEBUG lxc_cgroup - checking
'/sys/kernel/debug' (debugfs)
lxc-start 1354793173.808 DEBUG lxc_cgroup - checking
'/dev/hugepages' (hugetlbfs)
lxc-start 1354793173.808 DEBUG lxc_cgroup - checking '/tmp'
(tmpfs)
lxc-start 1354793173.808 DEBUG lxc_cgroup - checking
'/sys/kernel/config' (configfs)
lxc-start 1354793173.808 DEBUG lxc_cgroup - checking '/boot'
(ext4)
lxc-start 1354793173.808 DEBUG lxc_cgroup - checking '/home'
(ext4)
lxc-start 1354793173.808 DEBUG lxc_cgroup - checking '/boot/efi'
(vfat)
lxc-start 1354793173.808 DEBUG lxc_cgroup - checking
'/sys/fs/fuse/connections' (fusectl)
lxc-start 1354793173.808 DEBUG lxc_cgroup - checking
'/proc/sys/fs/binfmt_misc' (binfmt_misc)
lxc-start 1354793173.808 DEBUG lxc_cgroup - checking
'/run/user/1000/gvfs' (fuse.gvfsd-fuse)
lxc-start 1354793173.808 INFO lxc_conf - 'test1-18' hostname has
been setup
lxc-start 1354793173.808 DEBUG lxc_conf - mounted
'/var/lib/lxc/test1-18/rootfs' on '/usr/local/lib/lxc/rootfs'
lxc-start 1354793173.808 DEBUG lxc_conf - mounted 'proc' on
'/usr/local/lib/lxc/rootfs/proc', type 'proc'
lxc-start 1354793173.808 DEBUG lxc_conf - mounted 'sysfs' on
'/usr/local/lib/lxc/rootfs/sys', type 'sysfs'
lxc-start 1354793173.808 INFO lxc_conf - mount points have been
setup
lxc-start 1354793173.809 INFO lxc_conf - mount points have been
setup
called for subsys devices name test1-18
lxc-start 1354793173.809 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/devices has flags 0x2
found init cgroup for subsys devices at /
lxc-start 1354793173.809 DEBUG lxc_cgroup - using cgroup mounted
at '/sys/fs/cgroup/devices//lxc'
returning /sys/fs/cgroup/devices//lxc/test1-18 for subsystem devices.deny
lxc-start 1354793173.809 DEBUG lxc_conf - cgroup 'devices.deny'
set to 'a'
called for subsys devices name test1-18
lxc-start 1354793173.809 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/devices has flags 0x2
found init cgroup for subsys devices at /
lxc-start 1354793173.809 DEBUG lxc_cgroup - using cgroup mounted
at '/sys/fs/cgroup/devices//lxc'
returning /sys/fs/cgroup/devices//lxc/test1-18 for subsystem devices.allow
lxc-start 1354793173.809 DEBUG lxc_conf - cgroup 'devices.allow'
set to 'c 1:3 rwm'
called for subsys devices name test1-18
lxc-start 1354793173.809 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/devices has flags 0x2
found init cgroup for subsys devices at /
lxc-start 1354793173.809 DEBUG lxc_cgroup - using cgroup mounted
at '/sys/fs/cgroup/devices//lxc'
returning /sys/fs/cgroup/devices//lxc/test1-18 for subsystem devices.allow
lxc-start 1354793173.809 DEBUG lxc_conf - cgroup 'devices.allow'
set to 'c 1:5 rwm'
called for subsys devices name test1-18
lxc-start 1354793173.809 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/devices has flags 0x2
found init cgroup for subsys devices at /
lxc-start 1354793173.809 DEBUG lxc_cgroup - using cgroup mounted
at '/sys/fs/cgroup/devices//lxc'
returning /sys/fs/cgroup/devices//lxc/test1-18 for subsystem devices.allow
lxc-start 1354793173.809 DEBUG lxc_conf - cgroup 'devices.allow'
set to 'c 5:1 rwm'
called for subsys devices name test1-18
lxc-start 1354793173.810 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/devices has flags 0x2
found init cgroup for subsys devices at /
lxc-start 1354793173.810 DEBUG lxc_cgroup - using cgroup mounted
at '/sys/fs/cgroup/devices//lxc'
returning /sys/fs/cgroup/devices//lxc/test1-18 for subsystem devices.allow
lxc-start 1354793173.810 DEBUG lxc_conf - cgroup 'devices.allow'
set to 'c 5:0 rwm'
called for subsys devices name test1-18
lxc-start 1354793173.810 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/devices has flags 0x2
found init cgroup for subsys devices at /
lxc-start 1354793173.810 DEBUG lxc_cgroup - using cgroup mounted
at '/sys/fs/cgroup/devices//lxc'
returning /sys/fs/cgroup/devices//lxc/test1-18 for subsystem devices.allow
lxc-start 1354793173.810 DEBUG lxc_conf - cgroup 'devices.allow'
set to 'c 4:0 rwm'
called for subsys devices name test1-18
lxc-start 1354793173.810 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/devices has flags 0x2
found init cgroup for subsys devices at /
lxc-start 1354793173.810 DEBUG lxc_cgroup - using cgroup mounted
at '/sys/fs/cgroup/devices//lxc'
returning /sys/fs/cgroup/devices//lxc/test1-18 for subsystem devices.allow
lxc-start 1354793173.810 DEBUG lxc_conf - cgroup 'devices.allow'
set to 'c 4:1 rwm'
called for subsys devices name test1-18
lxc-start 1354793173.810 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/devices has flags 0x2
found init cgroup for subsys devices at /
lxc-start 1354793173.810 DEBUG lxc_cgroup - using cgroup mounted
at '/sys/fs/cgroup/devices//lxc'
returning /sys/fs/cgroup/devices//lxc/test1-18 for subsystem devices.allow
lxc-start 1354793173.810 DEBUG lxc_conf - cgroup 'devices.allow'
set to 'c 1:9 rwm'
called for subsys devices name test1-18
lxc-start 1354793173.810 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/devices has flags 0x2
found init cgroup for subsys devices at /
lxc-start 1354793173.810 DEBUG lxc_cgroup - using cgroup mounted
at '/sys/fs/cgroup/devices//lxc'
returning /sys/fs/cgroup/devices//lxc/test1-18 for subsystem devices.allow
lxc-start 1354793173.810 DEBUG lxc_conf - cgroup 'devices.allow'
set to 'c 1:8 rwm'
called for subsys devices name test1-18
lxc-start 1354793173.810 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/devices has flags 0x2
found init cgroup for subsys devices at /
lxc-start 1354793173.810 DEBUG lxc_cgroup - using cgroup mounted
at '/sys/fs/cgroup/devices//lxc'
returning /sys/fs/cgroup/devices//lxc/test1-18 for subsystem devices.allow
lxc-start 1354793173.810 DEBUG lxc_conf - cgroup 'devices.allow'
set to 'c 136:* rwm'
called for subsys devices name test1-18
lxc-start 1354793173.810 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/devices has flags 0x2
found init cgroup for subsys devices at /
lxc-start 1354793173.810 DEBUG lxc_cgroup - using cgroup mounted
at '/sys/fs/cgroup/devices//lxc'
returning /sys/fs/cgroup/devices//lxc/test1-18 for subsystem devices.allow
lxc-start 1354793173.810 DEBUG lxc_conf - cgroup 'devices.allow'
set to 'c 5:2 rwm'
called for subsys devices name test1-18
lxc-start 1354793173.811 DEBUG lxc_cgroup - cgroup
/sys/fs/cgroup/devices has flags 0x2
found init cgroup for subsys devices at /
lxc-start 1354793173.811 DEBUG lxc_cgroup - using cgroup mounted
at '/sys/fs/cgroup/devices//lxc'
returning /sys/fs/cgroup/devices//lxc/test1-18 for subsystem devices.allow
lxc-start 1354793173.811 DEBUG lxc_conf - cgroup 'devices.allow'
set to 'c 254:0 rwm'
lxc-start 1354793173.811 INFO lxc_conf - cgroup has been setup
lxc-start 1354793173.811 INFO lxc_conf - console has been setup
lxc-start 1354793173.811 INFO lxc_conf - 4 tty(s) has been setup
lxc-start 1354793173.811 DEBUG lxc_conf - created
'/usr/local/lib/lxc/rootfs/lxc_putold' directory
lxc-start 1354793173.811 DEBUG lxc_conf - mountpoint for old
rootfs is '/usr/local/lib/lxc/rootfs/lxc_putold'
lxc-start 1354793173.811 DEBUG lxc_conf - pivot_root syscall to
'/usr/local/lib/lxc/rootfs' successful
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/dev/shm'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/dev/pts'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/dev/mqueue'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/dev/hugepages'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/proc/sys/fs/binfmt_misc'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/sys/kernel/security'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/sys/fs/selinux'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/sys/fs/cgroup/systemd'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/sys/fs/cgroup/cpuset'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/sys/fs/cgroup/cpu,cpuacct'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/sys/fs/cgroup/memory'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/sys/fs/cgroup/devices'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/sys/fs/cgroup/freezer'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/sys/fs/cgroup/net_cls'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/sys/fs/cgroup/blkio'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/sys/fs/cgroup/perf_event'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/sys/kernel/debug'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/sys/kernel/config'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/sys/fs/fuse/connections'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/run/user/1000/gvfs'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/tmp'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/boot/efi'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/home'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/dev'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/sys/fs/cgroup'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/run'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/boot'
lxc-start 1354793173.812 DEBUG lxc_conf - umounted
'/lxc_putold/sys'
lxc-start 1354793173.812 INFO lxc_conf - lazy unmount of
'/lxc_putold'
lxc-start 1354793173.812 WARN lxc_conf - failed to unmount
'/lxc_putold/proc'
lxc-start 1354793173.813 INFO lxc_conf - created new pts instance
lxc-start 1354793173.813 DEBUG lxc_conf - capabilities has been
setup
lxc-start 1354793173.813 NOTICE lxc_conf - 'test1-18' is setup.
lxc-start 1354793173.813 NOTICE lxc_start - exec'ing '/sbin/init'
lxc-start 1354793173.813 NOTICE lxc_start - '/sbin/init' started
with pid '17431'
lxc-start 1354793173.813 WARN lxc_start - invalid pid for SIGCHLD
tahnks
Benoit
------------------------------------------------------------------------------
LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial
Remotely access PCs and mobile devices and provide instant support
Improve your efficiency, and focus on delivering more value-add services
Discover what IT Professionals Know. Rescue delivers
http://p.sf.net/sfu/logmein_12329d2d
_______________________________________________
Lxc-users mailing list
Lxc-users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lxc-users

--
Michael H. Warfield (AI4NB) | [phone] | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | [phone] | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20121206/4c563779/attachment.pgp>

[Serge Hallyn]

The short answer is, if login prompt is going to the system console then
that means /dev/console was recreated in the container as 5:1.

What you probably want to do is grab the lxc version from
git://github.com/lxc/lxc.git#staging and set lxc.autodev=1 in your
container config. Then systemd in the container will see that
/dev is mounted and not try to re-mount/re-populate it.

See other threads here or in lxc-devel for other issues you can expect
to see.

-serge