Discussion:
[lxc-devel] [Lxc-users] Working LXC templates?
Michael H. Warfield
2013-09-04 13:40:49 UTC
Permalink
This issue really belongs on -devel since it's a template development
issue that really impacts all the template writers.
Thx all for the replies,
- lxc-version returns 0.8.0. Looking around, there might be a more current
unstable, but AFAIK it's the most recently released stable.
This is not going to work. 0.8.0 will not support systemd in a
container, which all recent supported versions of Fedora are going to
require. 0.8.0 may be the more recently released stable "FROM OpenSuse"
but it is not the more recently release stable from LXC. The most
recently released stable from LXC is 0.9.0 and even that doesn't have
some of the necessary patches to the Fedora Template. Your best bet
would be to built from the stage branch in git.

You may need to wait until we release 1.0.0 and I'll take some of your
thoughts into consideration for the Fedora template but I have no way to
test them on OpenSuse at this time. Once we release 1.0.0, you'll still
have the problem with what OpenSuse releases as their stable. We have
no control over what they decide and do.
- I'd have to re-run to get the SSL error again but I think I've described
its error accurately, no further explanation except that the identity of
the remote server cannot be authenticated. This would lead me to guess that
the server is not registered properly with a public CA (eg using a CA root
that isn't in a bootstrap Fedora) so guessing that perhaps an option should
be offered that allows over-riding authentication? SSL encryption of course
should still be implemented for security.
Well, I can give you an argument if the error was described accurately
enough. I didn't see any site names I could test to see what the root
CA is. Without that, I can't tell you why you're seeing that error. I
understand fully what the error is (having my own private CA for private
activities) but I can't determine the origin without knowing the source.

<WAG - Wild Ass Guess>

I suspect that their certs are properly signed by a CA particular to the
Fedora Project and properly contained in the Fedora rpm root store, so
it may really be yet another cross distribution issue that depends on
the distribution peculiar packages and configurations.

Since there's probably no need (I see no need) for the Fedora
Repositories to be "registered properly with a public CA" (and pay the
extra expense), I would say the term "properly" is missued in this case.
The rpms are all signed by the Fedora Project and their gpg key so,
integrity shouldn't be an example unless someone intercepts the original
rootfs build and provides a trojaned package with the gpg keys.

</WAG>

Since this is an inter-distribution issue, I'm not sure what the proper
solution would be (assuming my WAG is true) or what LXC can do to
address it. I also don't know why Ubuntu / Debian is not experiencing
this problem either. But, without some example names of specific sites
exhibiting this problem (I don't run OpenSuse) I have no way to
investigate further.

Yeah, we could probably add an option to the template to ignore the SSL
check or to use and alternate rootca store (if we can avoid the
catch-22) but it may be better to investigate a more generic,
distribution agnostic, solution to these types of problems.

I do think it is an issue with the whole "distribution agnostic
template" problem that may require some help from the distros or some
innovative ideas of how we can bootstrap distros using distro agnostic
tools (like stone knives and bear skins style install of the rootfs
using nothing more than tar, gzip, gpg, and curl or wget).
- The lxc-create issue is definitely there. At first, I encountered it
using the openSUSE YAST LXC container applet. but then when I invoked
lxc-create from a console, but the "help" verifies it supports few options
and not these. But, as I described if the template requires parameters,
it's also possible to simply provide them in the script instead of at
runtime as command switches (but might not be apparent at first to someone
reading your script).
Tony
Regards,
Mike
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20130904/a1db6e69/attachment.pgp>
Natanael Copa
2013-09-04 17:52:34 UTC
Permalink
On Wed, 04 Sep 2013 09:40:49 -0400
Post by Michael H. Warfield
I do think it is an issue with the whole "distribution agnostic
template" problem that may require some help from the distros or some
innovative ideas of how we can bootstrap distros using distro agnostic
tools (like stone knives and bear skins style install of the rootfs
using nothing more than tar, gzip, gpg, and curl or wget).
This would be very nice. I have not had success with any templates
except the debian on Alpine Linux. Debian works because we build a
debootstrap package. Ubuntu template did not work because it uses
'arch' command which we don't have. (ok, should be trivial to implement
if we want it bad enough - and I haven't tested current git templates)

However, the alpine template in current git should work on any distro.

Here is what we do:
* download static apk-tools (package manager) and the package with the
public keys used for package signature checking.

* unpack the the package manager and public keys package with tar. The
package format is basically .tar.gz with some files in the beginning
with metadata, so the .apk files can be extracted with tar -zx.

* verify that the public keys are unmodified against a sha256 sum that
is embedded in the template script.

* verify that the static binary is unmodified using the public key and
openssl. The apk-tools-static package includes a signature for the
static binary.

* use the verified static package manager to install a rootfs. The
package manager will use the previously downloaded pub keys.

This should work on any x86/x86_64 distro with tar, gzip, openssl and
wget.


-nc
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20130904/6936312f/attachment.pgp>
Tony Su
2013-09-08 23:08:39 UTC
Permalink
After putting some thought into this,
IMO LXC badly needs a universal tool with the following features

- A single script should be used to run on <any> HostOS, creating
<any> supported Container OS. Although this would make the script
extremely large, IMO it would actually be easier to maintain in the
long run. Unknown at this point is how long it would take to load, but
once loaded I would expect the script to run very fast.

- Flexible enough to support any packaging system, and perhaps even
non-package systems if someone should present a strong case or is
willing to put in the work

- Use the available HostOS distro packages whenever possible. Do not
require packages from other distros or cross-platform packages. IMO
this means less reliance on supporting non-mainline packages, keeps
package and functionality support simple, understandable and
responsibility clear.

- The script should be clearly modular, with sections clearly marked
for the functionality in that section.

- The script as currently envisioned should be a single file to reduce
missing "script dependencies." Any calls should generally be only to
internal functions or to mainstream packages provided by the distro.
Separate custom scripts should be avoided because otherwise the
dependency script(s) may be hard to locate.

To this end,
I've done the intial baby steps towards architecting a new universal
lxc-create template script that should have all the above features and
likely within a week or so post an early alpha on my github for
anyonne to criticize/contribute/modify.

In the meantime, I would be interested to know if anyone is
successfully doing what the title of this message thread describes...
Creating a Container with a different GuestOS than the HostOS, and if
you could either provide a link or description to where the template
script can be found (plus, as Nathaniel Copa described anything
special and extraordinary you had to do).

It would be done a lot faster except that this is being done in my spare time.
Now, if someone who deals with LXC a lot wanted to hire me to do this
and maybe more... :)

Tony
Post by Natanael Copa
On Wed, 04 Sep 2013 09:40:49 -0400
Post by Michael H. Warfield
I do think it is an issue with the whole "distribution agnostic
template" problem that may require some help from the distros or some
innovative ideas of how we can bootstrap distros using distro agnostic
tools (like stone knives and bear skins style install of the rootfs
using nothing more than tar, gzip, gpg, and curl or wget).
This would be very nice. I have not had success with any templates
except the debian on Alpine Linux. Debian works because we build a
debootstrap package. Ubuntu template did not work because it uses
'arch' command which we don't have. (ok, should be trivial to implement
if we want it bad enough - and I haven't tested current git templates)
However, the alpine template in current git should work on any distro.
* download static apk-tools (package manager) and the package with the
public keys used for package signature checking.
* unpack the the package manager and public keys package with tar. The
package format is basically .tar.gz with some files in the beginning
with metadata, so the .apk files can be extracted with tar -zx.
* verify that the public keys are unmodified against a sha256 sum that
is embedded in the template script.
* verify that the static binary is unmodified using the public key and
openssl. The apk-tools-static package includes a signature for the
static binary.
* use the verified static package manager to install a rootfs. The
package manager will use the previously downloaded pub keys.
This should work on any x86/x86_64 distro with tar, gzip, openssl and
wget.
-nc
Michael H. Warfield
2013-09-09 00:33:16 UTC
Permalink
With all due respect...
Post by Tony Su
After putting some thought into this,
IMO LXC badly needs a universal tool with the following features
- A single script should be used to run on <any> HostOS, creating
<any> supported Container OS. Although this would make the script
extremely large, IMO it would actually be easier to maintain in the
long run.
Actually, no. From my experience (30+ years in software development),
it would turn into a morass.

The problem here is that the maintainer(s) would then need to understand
how each and every distribution is installed and how it would be
installed on each and every distribution. It would distill the worse of
all the problems we have now in the templates into one great big dung
pile. It would rapidly become unmaintainable. The "extremely large" is
the red letter warning that it will become unmaintainable as fewer and
fewer people understand what this great big huge blob does.

I've written perl scripts that count 3K+ lines long and have to have
over 50% of the lines be comments just so people can read it (ok - it's
perl, what can I say). The template system with small, individually
maintained, distribution templates are vastly more maintainable. I'm
pretty good with the Fedora stuff. I'm not about to tell the
maintainers of the Ubuntu or OpenSuse stuff how to do their thing. We
can formulate rules and test each other's templates, but we can not
write or maintain them. But, we can be good at maintaining our own
templates.
Post by Tony Su
Unknown at this point is how long it would take to load, but
once loaded I would expect the script to run very fast.
Extremely doubtful, in my experience, and the "run very fast" is not the
point and not the source of your problems. From a systems analysis
basis, a function which is run rarely can run very inefficiently if it
does the right thing every time and is superior to something that runs
efficiently but missing way too many corner cases and is a complex
gemish-gamash to maintain because it's a great big hairball. I don't
care about "run very fast". I care about "does the right thing" in this
case. Plus, if it's the Fedora create that's busted, you're limited in
the domain of script code you have to look at, even if you are
unfamiliar with the code over all.
Post by Tony Su
- Flexible enough to support any packaging system, and perhaps even
non-package systems if someone should present a strong case or is
willing to put in the work
It'll never happen. It would take a huge design team, conversant in all
those packaging systems (I'm pretty damn good in yum and rpm, so-so in
pkg and apt-get, and absolutely suck at yast and zypper) and then it
would be "design by committee". A committee is a life form with six or
more legs AND NO BRAIN.
Post by Tony Su
- Use the available HostOS distro packages whenever possible. Do not
require packages from other distros or cross-platform packages. IMO
this means less reliance on supporting non-mainline packages, keeps
package and functionality support simple, understandable and
responsibility clear.
Agreed. But that is an argument for a modularized system with small,
distro specific, components melded with a small modularized system with
distro independent components.

BTW... You think trying to install Fedora on OpenSuse is bad? Try
installing OpenSuse on Fedora (or Ubuntu). It's a disaster. That
template won't even come off the dime (no offense to the author - we
just don't have the tools in our distros). Here's what you get...
--
[root at forest mhw]# lxc-create -n OpenSuse -t opensuse

lxc-create: No config file specified, using the default config /etc/lxc/default.conf
/usr/share/lxc/templates/lxc-opensuse: line 369: type: zypper: not found
'zypper' command is missing
lxc-create: failed to execute template 'opensuse'
lxc-create: aborted
--
Epic fail. There is no zypper on Fedora.
Post by Tony Su
- The script should be clearly modular, with sections clearly marked
for the functionality in that section.
And that argues against one great big script (which would have to be
collectively maintained) and for smaller components (which could be
independently maintained).
Post by Tony Su
- The script as currently envisioned should be a single file to reduce
missing "script dependencies." Any calls should generally be only to
internal functions or to mainstream packages provided by the distro.
Separate custom scripts should be avoided because otherwise the
dependency script(s) may be hard to locate.
Totally disagree as my experience leads me to believe this ultimately
leads to bit-rot and deterioration in the maintainability where nobody
knows who wrote what line and why and nobody will touch it to avoid the
blame of breaking someone else's code.

Nope... Not going to happen (IMNSHO). I've run into this even with the
Fedora template. Several times I've posted to the -devel list asking
who wrote this or that line and "what were you thinking" only to get no
answer. Now raise that to the N'th power where N is the number of
authors (or lines) in a given file.
Post by Tony Su
To this end,
I've done the intial baby steps towards architecting a new universal
lxc-create template script that should have all the above features and
likely within a week or so post an early alpha on my github for
anyonne to criticize/contribute/modify.
Right now, I'm proceeding down an avenue that may lead me to generating
a template that should work everywhere. The effort is to create an
intermediary RTE (Run Time Environment) based on the netinst (Network
Install) CD images present in all Fedora releases and many of the others
I have examined. I may even resort to using the RedHat / Fedora
Anaconda kickstart system to create a minimal RTE rootfs and using that
to create the container cache rootfs. The RTE could even be cached
alone with the individual roots for the containers. It's decidedly NOT
efficient (about a 200MB-300MB download if you can not install
"natively") but it should work (still working out the bugs and
dependencies) but only has to be done once for all the containers of a
given distro.

Is this applicable to other templates? Maybe yes, maybe no. Can others
copy the technique of creating an intermediary RTE to facilitate the
initial build of the rootfs? Hell yes (I hope so). But, it's up to the
individual template maintainers. At this point, I'm not sure if Serge,
Daniel, and Stephane and others consider me a maintainer for the Fedora
template or just an (very) annoying contributor (and agent provocatour)
but I wouldn't be offended or annoyed either way.

If you think you can do this, I'm more than willing to test and
critique. Maybe you're better at this than I am. Give it your best
shot. If you can come up with a way to build the Fedora rootfs without
recourse to rpm or yum or anaconda or the netinst CD (or prebuild,
non-distro distributed tarballs), I would love to hear your thoughts.
And, no, just because you have rpm on OpenSuse, it doesn't mean it will
install Fedora, unfortunately. One of the things I'm concerned about is
versioning in the rpm databases (on all rpm based distros) and if even a
distro RTE can build a true rootfs with the correct rpm database
version. Sigh...
Post by Tony Su
In the meantime, I would be interested to know if anyone is
successfully doing what the title of this message thread describes...
Creating a Container with a different GuestOS than the HostOS, and if
you could either provide a link or description to where the template
script can be found (plus, as Nathaniel Copa described anything
special and extraordinary you had to do).
I have done it manually down through the years for many distros on
several distros under both OpenVZ and LXC with varying success. It's
invariably a painful experience, often having to resort to the netinst
CD's or other precreated containers. It's interesting that even OpenVZ
seems to have abandoned their original templating scheme and largely
gone with a pile of pre-built template tarballs on their website. If we
can avoid recourse to that situation (and it has already been suggested
by more than one), we would be much better off. I don't want to go down
the pre-build tarball road, myself.
Post by Tony Su
It would be done a lot faster except that this is being done in my spare time.
Now, if someone who deals with LXC a lot wanted to hire me to do this
and maybe more... :)
Honestly (and no offense - EVERYONE STOP LAUGHING) if you think you can
do it, have at it, knock yourself out. But I won't support it. In my
mind, it will end up being one great big unsupportable hairball of
functions and side effects and dependencies with too many fingers in the
pie for each distro's piece of the pie. Too many years I've seen things
like that turn in to an exercise in autoerotism that is just not worth
the frustration. If you want to take on the sole task of maintaining it
for every distro - more power to you. I couldn't do it. Who will
maintain it?

Keep it small and modularized into separate files with specific
functionality with very tightly controlled interfaces to reduce
interactions and side effects. That way, I can take responsibility for
my files and you for yours. When I pass my pieces along, I hope I've
commented things enough for someone else to take up the flame or flame
me if I failed.

If you can truly come up with a distro agnostic way of doing this
without requiring significant new support from the (arbitrary) guest
distros, my hat is off to you. I'd love to see it.
Post by Tony Su
Tony
Apologies to anyone who feels I'm off base or too harsh here.

Regards,
Mike
Post by Tony Su
Post by Natanael Copa
On Wed, 04 Sep 2013 09:40:49 -0400
Post by Michael H. Warfield
I do think it is an issue with the whole "distribution agnostic
template" problem that may require some help from the distros or some
innovative ideas of how we can bootstrap distros using distro agnostic
tools (like stone knives and bear skins style install of the rootfs
using nothing more than tar, gzip, gpg, and curl or wget).
This would be very nice. I have not had success with any templates
except the debian on Alpine Linux. Debian works because we build a
debootstrap package. Ubuntu template did not work because it uses
'arch' command which we don't have. (ok, should be trivial to implement
if we want it bad enough - and I haven't tested current git templates)
However, the alpine template in current git should work on any distro.
* download static apk-tools (package manager) and the package with the
public keys used for package signature checking.
* unpack the the package manager and public keys package with tar. The
package format is basically .tar.gz with some files in the beginning
with metadata, so the .apk files can be extracted with tar -zx.
* verify that the public keys are unmodified against a sha256 sum that
is embedded in the template script.
* verify that the static binary is unmodified using the public key and
openssl. The apk-tools-static package includes a signature for the
static binary.
* use the verified static package manager to install a rootfs. The
package manager will use the previously downloaded pub keys.
This should work on any x86/x86_64 distro with tar, gzip, openssl and
wget.
-nc
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20130908/5594b2fb/attachment.pgp>
Natanael Copa
2013-09-09 06:58:54 UTC
Permalink
On Sun, 08 Sep 2013 20:33:16 -0400
Post by Michael H. Warfield
With all due respect...
Post by Tony Su
After putting some thought into this,
IMO LXC badly needs a universal tool with the following features
- A single script should be used to run on <any> HostOS, creating
<any> supported Container OS. Although this would make the script
extremely large, IMO it would actually be easier to maintain in the
long run.
Actually, no. From my experience (30+ years in software development),
it would turn into a morass.
The problem here is that the maintainer(s) would then need to understand
how each and every distribution is installed and how it would be
installed on each and every distribution. It would distill the worse of
all the problems we have now in the templates into one great big dung
pile. It would rapidly become unmaintainable. The "extremely large" is
the red letter warning that it will become unmaintainable as fewer and
fewer people understand what this great big huge blob does.
I tend to agree with this.

What I do think could be done is to use template APIs. Either by having
a script "library" with helper functions (for things like:
generate_mac_address etc) or let the template scripts be "plugins" that
must provide a set of pre-defined functions (eg. install_distro,
configure_distro, copy_configuration etc) or maybe a combination of
those two.

-nc
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20130909/c8f4111b/attachment.pgp>
Michael H. Warfield
2013-09-09 11:28:43 UTC
Permalink
Post by Natanael Copa
On Sun, 08 Sep 2013 20:33:16 -0400
Post by Michael H. Warfield
With all due respect...
Post by Tony Su
After putting some thought into this,
IMO LXC badly needs a universal tool with the following features
- A single script should be used to run on <any> HostOS, creating
<any> supported Container OS. Although this would make the script
extremely large, IMO it would actually be easier to maintain in the
long run.
Actually, no. From my experience (30+ years in software development),
it would turn into a morass.
The problem here is that the maintainer(s) would then need to understand
how each and every distribution is installed and how it would be
installed on each and every distribution. It would distill the worse of
all the problems we have now in the templates into one great big dung
pile. It would rapidly become unmaintainable. The "extremely large" is
the red letter warning that it will become unmaintainable as fewer and
fewer people understand what this great big huge blob does.
I tend to agree with this.
What I do think could be done is to use template APIs. Either by having
generate_mac_address etc) or let the template scripts be "plugins" that
must provide a set of pre-defined functions (eg. install_distro,
configure_distro, copy_configuration etc) or maybe a combination of
those two.
I like this idea, it's just that, in the short term, we have to get past
this one gotcha.

In the git-stage current Fedora template, the entire problem is embodied
in the "download_fedora" function starting around line 201... The
gotcha's are three commands around line 272 after we've identified and
downloaded the initial release rpm. We have this:

mkdir -p $INSTALL_ROOT/var/lib/rpm
rpm --root $INSTALL_ROOT --initdb
rpm --root $INSTALL_ROOT -ivh ${INSTALL_ROOT}/${RELEASE_RPM}
$YUM install $PKG_LIST

Ok... Those are running in the host local run time environment.
Obviously, if the host does not have a (compatible) version of rpm
and/or yum in the host local environment, you're screwed. That's the
catch-22 situation and it's the same situation with zypper in the
OpenSuse template. That short space of code has to be recreated in a
totally distro-agnostic manner so it runs on any distro to create our
initial rootfs. After that, we can do what ever distro (Fedora)
specific commands we want by chrooting into the target container first
(including rebuilding the rpm database to the target version). That's
only even needed if you don't already have a cached rootfs for that
distro and version.

I was SOOOO close last week working on this while on vacation. Recent
revs of Fedora have this downloadable LiveOS core that runs on the
netinst CD and others. That's the 200MB - 300MB blob I was referring
to. You just download it, mount it (requires squashfs) to a directory
and then mount the ext3 image it contains on another directory. Then
create and bind mount a few rw directories (etc var run), mount proc in
the image, and bind mount your rootfs to run/install in the image. Then
chroot into the image and voila, instant RTE. Yum and rpm are capable
of installing other versions of Fedora, so I wouldn't (shouldn't) even
need a version specific instantiation of the RTE, just one that can
install every version we might be interested in.

Except... It didn't work. Sigh... $#@$#@#@

Fedora came so close and then face planted for what I wanted to do.
Sure rpm is in there. But rpmdb is not. So, no --initdb and no
--rebuilddb. They've got yum in there, but no yummain.py so yum hurls
chunks immediately upon execution trying to import yummain. What the
hell good does it do to have yum in there but no yummain?!?! But,
something, fixed up, like that from all the distros would be a perfect
answer (albeit somewhat less than high performance thanks to that
download, but it's a single download that could be cached). The only
potential gotcha I see in there is requiring squashfs available for
mounting the image. Anyone have heartburn over that?

That squashfs image has to be able to do an install or it would be
useless on the netinst CD. I'm not sure if they still have anaconda on
there or not but it has to be able to do a kickstart install, so all I
need is a minimal.ks kickstart file to perform essentially an unattended
install of a minimal build into the target rootfs. That's where I'm at
now, trying to get that to work.

But it's all that work just to replace those few lines of code above
where you can not perform a host native install of the rootfs. That's
all that's standing in the way and it's frustratingly close for the
Fedora template.

We've got the following distro templates in the project, currently:

lxc-alpine
lxc-alt
lxc-arch
lxc-busybox (is this really a distro template - I'm not sure)
lxc-debian
lxc-fedora
lxc-opensuse
lxc-oracle (really?)
lxc-ubuntu / lxc-ubuntu-cloud

Who are the owners? I don't "own" the Fedora template but I work on it.
Each template owner has to solve this problem in their own way and it
all boils down to the same problem. Building that initial rootfs in the
host native environment.

After that, the idea of generalizing the functions into an API form
sounds like a great idea. It could be that this catch-22 could be
buried in a "create_rootfs" function which would create a cacheable
rootfs.

I would love to see templates for CentOS, Scientific (SL), Slackware and
maybe even RedHat EL. NST (Network Security Toolkit) and Kali (fka
BackTrack) would also be nice and I might work on them (which can I say,
I'm a security fanatic). But I gotta solve this problem first in a way
I can use for the ones I might want to write down the road.

But, in the meant time, we have "the recipe for rabbit stew"... First
you catch your rabbit. First we have to have a way to build the initial
rootfs.

<aside>

This entire process reminds me of when we (I) had to bootstrap gcc into
environments (HPUX and AIX years ago) which had no C compiler installed.
First you cross compiled a compiler (on Solaris in my case) for your
target arch and transported it there. Then you used that "stage 1"
compiler to compile the gcc compiler on the native platform (stage 2).
Then you used the stage 2 compiler to compile a full stage 3 compiler
where you could then confirm the outputs from the stage 2 and stage 3
compilers to confirm they matched (which never seems to happen the first
time you try it). When you were done and finally confirmed working, you
really needed a beer. This is the same sort of bootstrap process.

</aside>
Post by Natanael Copa
-nc
Regards,
Mike
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20130909/a1b60d22/attachment.pgp>
Michael H. Warfield
2013-09-09 12:10:48 UTC
Permalink
Post by Michael H. Warfield
In the git-stage current Fedora template, the entire problem is embodied
in the "download_fedora" function starting around line 201... The
gotcha's are three commands around line 272 after we've identified and
mkdir -p $INSTALL_ROOT/var/lib/rpm
rpm --root $INSTALL_ROOT --initdb
rpm --root $INSTALL_ROOT -ivh ${INSTALL_ROOT}/${RELEASE_RPM}
$YUM install $PKG_LIST
Ok... Those are running in the host local run time environment.
Obviously, if the host does not have a (compatible) version of rpm
and/or yum in the host local environment, you're screwed. That's the
catch-22 situation and it's the same situation with zypper in the
OpenSuse template. That short space of code has to be recreated in a
totally distro-agnostic manner so it runs on any distro to create our
initial rootfs. After that, we can do what ever distro (Fedora)
specific commands we want by chrooting into the target container first
(including rebuilding the rpm database to the target version). That's
only even needed if you don't already have a cached rootfs for that
distro and version.
Another approach could be to popularize the container downloads by the distro.
If each distro. could add a .tar.gz for a working container of a given
release, one could just download and configure them, no?
then the lxc project upstream, could just list those links or include them in
the a separate tool, that just downloads and untars the same?
That would completely sidestep the bootstrapping one-distro.-on-another
problem.
True, and that's been mentioned before in several different contexts.
The problem is that we have to get the cooperation of ALL the distros we
wish to support, which seems to be a bit of an intractable problem. In
Fedora, it would require someone to take that on as a project and be the
maintainer. It would also have to be architected into the build system
so it would be automatically build when a release it cut.

Since they already have the squashfs LiveOS system (which is 95% of what
we need), I don't think it would be a major leap for them to add a
parallel build to build an LXC LiveOS to live, say, in the same download
directory. In fact, if they fixed some of the deficiencies in the
LiveOS image, we could work directly from the squashfs image that's
already there.

The problem is in getting someone interested (I'm not a Fedora
maintainer) and getting them to do it. It would probably have to be
filled as an enhancement request and go through the months long vetting
process at best. We might have a better shot (in this case) of filing a
bug report in bugzilla for the busted components of the LiveOS image and
getting them to fix it. Even there, though, it's likely impossible to
get them to retroactively fix any of the previous images.

I agree it would work, I just don't think we can depend on getting
everyone else to agree to it and implement it in their distros.

Considering the direction Fedora has been taking in recent decisions
(removing sendmail / any MTA as well as the syslog server from the base
install) makes me seriously question if they would even care.
--
Regards
Shridhar
Regards,
Mike
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20130909/e05a956f/attachment.pgp>
Tony Su
2013-09-09 13:43:09 UTC
Permalink
Hello Michael,

Yes, I would really appreciate your comments on what I'm attempting to do.

Here is my thought process...

In general, I've found that the problems you describe are all too
common exactly because no one seems to have sat down and taken a close
look at the flow involved in creating a Container, installing the OS
from scratch like any other install instead of using pre-built
bootstrap file systems.

The problem has been exaccerbated by using templates which are
literally spaghetti code with little organization. When code like that
works, no one can modify the code because no one can reasonably see
what the consequences are and things will break without seeming rhyme
or reason.

This is why after looking at each template script, I'm choosing to
build on the Fedora template. Although it's broken several ways, I
admire its modularity and organization.so it should be a good
foundation.

I had considered something similar to what you're describing you are
building but decided against it because as I described eariler... If
you require a whole new set of tools (even cross platform) then it's
something else that needs to be maintained which I'd think should be
unnecessary. But, if someone decided to go down this path, then I
strongly considered using a standardized, cross platform packaging
system like Ruby.

But, as I said eariler, given a preference I would rather not require
installing new applications which largely duplicate what is already
available and likely installed in each distro when the problem is
mainly one of <configuration>, not missing functionality.



Without the bits of early code, this(following, below) is the
organization and structure of my proposed script...
I'm also still looking at the best cross-platform/cross-packaging way
to download the initial bootstrap packages, the fedora template
invokes pycurl. Still investigating whether it will work everywhere
and if something else (like wget if http is supported everywhere) is
more appropriate. But, after the bootstrap packages are downloaded,
the ContainerOS' native packaging manager can be used without
installing into the HostOS.

Note that putting everything in one large script also emphasizes the
re-usability of a number of modules across all distros, both HostOS
and Container creation.


<Global Variables>
Variables which apply to any and all Containers regardless of HostOS
or ContainerOS
</Global Variables>

<Default Distro Container Parameters>
Within this section there will be a section for each supported distro
Probably initially this may also be the main place to specify a
desired distro version for a Container but eventually making the
script interactive could be an eventual objective
These items will largely be determined by the content of the
<lxc-create> section
</Default Distro Container Parameters>


<HostOS "pre-Container" >
All the code in this section utilizes packages from the HostOS repos

<HostOS Identification>
This is the first part that is specific to the HostOS, so unless you
want to distribute different versions of this script for each distro,
this is where the HostOS must be identified, likely by reading
/etc/os-release
</HostOS Identification>

<Container creation>
<lxc-create>
Note that executing lxc-create is still in the context of the
HostOS, <using the HostOS packages> which means at this stage you
can't be for instance be using package managers not supported by the
HostOS. From what I can see lxc-create most importantly sets up the
chroot.

This section also most importantly configures and downloads the
required packages into

<Container Install>
Execute a standard network install for the target distro or
something similar
Any package managers specific to the target distro should run only
within this chroot environment
</Container Install>

</lxc-create>
</Container creation>
Post by Michael H. Warfield
Post by Michael H. Warfield
In the git-stage current Fedora template, the entire problem is embodied
in the "download_fedora" function starting around line 201... The
gotcha's are three commands around line 272 after we've identified and
mkdir -p $INSTALL_ROOT/var/lib/rpm
rpm --root $INSTALL_ROOT --initdb
rpm --root $INSTALL_ROOT -ivh ${INSTALL_ROOT}/${RELEASE_RPM}
$YUM install $PKG_LIST
Ok... Those are running in the host local run time environment.
Obviously, if the host does not have a (compatible) version of rpm
and/or yum in the host local environment, you're screwed. That's the
catch-22 situation and it's the same situation with zypper in the
OpenSuse template. That short space of code has to be recreated in a
totally distro-agnostic manner so it runs on any distro to create our
initial rootfs. After that, we can do what ever distro (Fedora)
specific commands we want by chrooting into the target container first
(including rebuilding the rpm database to the target version). That's
only even needed if you don't already have a cached rootfs for that
distro and version.
Another approach could be to popularize the container downloads by the distro.
If each distro. could add a .tar.gz for a working container of a given
release, one could just download and configure them, no?
then the lxc project upstream, could just list those links or include them in
the a separate tool, that just downloads and untars the same?
That would completely sidestep the bootstrapping one-distro.-on-another
problem.
True, and that's been mentioned before in several different contexts.
The problem is that we have to get the cooperation of ALL the distros we
wish to support, which seems to be a bit of an intractable problem. In
Fedora, it would require someone to take that on as a project and be the
maintainer. It would also have to be architected into the build system
so it would be automatically build when a release it cut.
Since they already have the squashfs LiveOS system (which is 95% of what
we need), I don't think it would be a major leap for them to add a
parallel build to build an LXC LiveOS to live, say, in the same download
directory. In fact, if they fixed some of the deficiencies in the
LiveOS image, we could work directly from the squashfs image that's
already there.
The problem is in getting someone interested (I'm not a Fedora
maintainer) and getting them to do it. It would probably have to be
filled as an enhancement request and go through the months long vetting
process at best. We might have a better shot (in this case) of filing a
bug report in bugzilla for the busted components of the LiveOS image and
getting them to fix it. Even there, though, it's likely impossible to
get them to retroactively fix any of the previous images.
I agree it would work, I just don't think we can depend on getting
everyone else to agree to it and implement it in their distros.
Considering the direction Fedora has been taking in recent decisions
(removing sendmail / any MTA as well as the syslog server from the base
install) makes me seriously question if they would even care.
--
Regards
Shridhar
Regards,
Mike
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
------------------------------------------------------------------------------
Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more!
Discover the easy way to master current and previous Microsoft technologies
and advance your career. Get an incredible 1,500+ hours of step-by-step
tutorial videos with LearnDevNow. Subscribe today and save!
http://pubads.g.doubleclick.net/gampad/clk?id=58041391&iu=/4140/ostg.clktrk
_______________________________________________
Lxc-devel mailing list
Lxc-devel at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lxc-devel
Shridhar Daithankar
2013-09-09 11:53:40 UTC
Permalink
Post by Michael H. Warfield
In the git-stage current Fedora template, the entire problem is embodied
in the "download_fedora" function starting around line 201... The
gotcha's are three commands around line 272 after we've identified and
mkdir -p $INSTALL_ROOT/var/lib/rpm
rpm --root $INSTALL_ROOT --initdb
rpm --root $INSTALL_ROOT -ivh ${INSTALL_ROOT}/${RELEASE_RPM}
$YUM install $PKG_LIST
Ok... Those are running in the host local run time environment.
Obviously, if the host does not have a (compatible) version of rpm
and/or yum in the host local environment, you're screwed. That's the
catch-22 situation and it's the same situation with zypper in the
OpenSuse template. That short space of code has to be recreated in a
totally distro-agnostic manner so it runs on any distro to create our
initial rootfs. After that, we can do what ever distro (Fedora)
specific commands we want by chrooting into the target container first
(including rebuilding the rpm database to the target version). That's
only even needed if you don't already have a cached rootfs for that
distro and version.
Another approach could be to popularize the container downloads by the distro.
If each distro. could add a .tar.gz for a working container of a given
release, one could just download and configure them, no?

then the lxc project upstream, could just list those links or include them in
the a separate tool, that just downloads and untars the same?

That would completely sidestep the bootstrapping one-distro.-on-another
problem.
--
Regards
Shridhar
Michael H. Warfield
2013-09-12 19:23:02 UTC
Permalink
All - Especially Tony Su,

Couple of people where I work thought you couldn't do what I was trying
to do, that it was "impossible". Oh well. Looks like they were
wrong. :-P It may not be "efficient" but it can be made to work.

Way down below, in-line...
Post by Michael H. Warfield
Post by Natanael Copa
On Sun, 08 Sep 2013 20:33:16 -0400
Post by Michael H. Warfield
With all due respect...
Post by Tony Su
After putting some thought into this,
IMO LXC badly needs a universal tool with the following features
- A single script should be used to run on <any> HostOS, creating
<any> supported Container OS. Although this would make the script
extremely large, IMO it would actually be easier to maintain in the
long run.
Actually, no. From my experience (30+ years in software development),
it would turn into a morass.
The problem here is that the maintainer(s) would then need to understand
how each and every distribution is installed and how it would be
installed on each and every distribution. It would distill the worse of
all the problems we have now in the templates into one great big dung
pile. It would rapidly become unmaintainable. The "extremely large" is
the red letter warning that it will become unmaintainable as fewer and
fewer people understand what this great big huge blob does.
I tend to agree with this.
What I do think could be done is to use template APIs. Either by having
generate_mac_address etc) or let the template scripts be "plugins" that
must provide a set of pre-defined functions (eg. install_distro,
configure_distro, copy_configuration etc) or maybe a combination of
those two.
I like this idea, it's just that, in the short term, we have to get past
this one gotcha.
In the git-stage current Fedora template, the entire problem is embodied
in the "download_fedora" function starting around line 201... The
gotcha's are three commands around line 272 after we've identified and
mkdir -p $INSTALL_ROOT/var/lib/rpm
rpm --root $INSTALL_ROOT --initdb
rpm --root $INSTALL_ROOT -ivh ${INSTALL_ROOT}/${RELEASE_RPM}
$YUM install $PKG_LIST
Ok... Those are running in the host local run time environment.
Obviously, if the host does not have a (compatible) version of rpm
and/or yum in the host local environment, you're screwed. That's the
catch-22 situation and it's the same situation with zypper in the
OpenSuse template. That short space of code has to be recreated in a
totally distro-agnostic manner so it runs on any distro to create our
initial rootfs. After that, we can do what ever distro (Fedora)
specific commands we want by chrooting into the target container first
(including rebuilding the rpm database to the target version). That's
only even needed if you don't already have a cached rootfs for that
distro and version.
I was SOOOO close last week working on this while on vacation. Recent
revs of Fedora have this downloadable LiveOS core that runs on the
netinst CD and others. That's the 200MB - 300MB blob I was referring
to. You just download it, mount it (requires squashfs) to a directory
and then mount the ext3 image it contains on another directory. Then
create and bind mount a few rw directories (etc var run), mount proc in
the image, and bind mount your rootfs to run/install in the image. Then
chroot into the image and voila, instant RTE. Yum and rpm are capable
of installing other versions of Fedora, so I wouldn't (shouldn't) even
need a version specific instantiation of the RTE, just one that can
install every version we might be interested in.
Fedora came so close and then face planted for what I wanted to do.
Sure rpm is in there. But rpmdb is not. So, no --initdb and no
--rebuilddb. They've got yum in there, but no yummain.py so yum hurls
chunks immediately upon execution trying to import yummain. What the
hell good does it do to have yum in there but no yummain?!?! But,
something, fixed up, like that from all the distros would be a perfect
answer (albeit somewhat less than high performance thanks to that
download, but it's a single download that could be cached). The only
potential gotcha I see in there is requiring squashfs available for
mounting the image. Anyone have heartburn over that?
That squashfs image has to be able to do an install or it would be
useless on the netinst CD. I'm not sure if they still have anaconda on
there or not but it has to be able to do a kickstart install, so all I
need is a minimal.ks kickstart file to perform essentially an unattended
install of a minimal build into the target rootfs. That's where I'm at
now, trying to get that to work.
I got this to work with just one (err, minor, gag, couch) butt ugly
hack / dirty trick. I seems, in this case, the problem with the Fedora
LiveOS squashfs image is that it has rpm (but no rpmdb) and yum (but no
yummain.py) partially installed and damn near everything else
(dependeices) are but it's not build with an rpm database in place.

The answer was to stage this with the original LiveOS fs being the stage
0 and copying that to a read-write directory and then downloading and
force (--nodeps) an install of rpm and yum installing the full packages
in place for a stage 1 run time environment. That enabled me to create
a stage 2 uber-minimal run time environment with rpm and yum installed
along with all their dependencies. This I could then cache and is
capable of installing arbitrary versions of Fedora (so it only needs to
be done once for any given arch).

I've integrated it into the lxc-fedora template but it has undergone
minimal testing (fedora on fedora). It has a "white listing" facility
in it to run a native install on distros (and also checks if the distro
has yum and rpm available) that will support it but that's disabled on
this, attached, template so it can get more testing. White listing is
based on the CPE OS identifier. What will go into the patch to the
project will have fedora, centos, and maybe a few others whitelisted
that are known to work.

Attached to this message is my trial balloon lxc-fedora template. This
is NOT final. This is just a test for comments and feedback. Please
try it out. Other template owners, PLEASE look at the code and see if
something similar makes sense for your templates! I would love to
install your distros on my Fedora hosts!

The code is commented moderately heavy (and somewhat sarcasticly) but
should explain what I was trying to do (if not - tell me!). The bulk of
the changes are a block of code and comments starting around line 201
plus a few modifications just to the "download_fedora()" routine, now at
line 543 (yes it was a big insert in front of it).

I can not emphasize this enough... READ THE COMMENTS! If I'm full of
shit doing it this way, then tell me so!

Once it builds that RTE the first time, it will use it for all future
Fedora container builds.

Tony: You are on OpenSuse. You, more than anyone else, I really need
to test this code by. I'll answer your other questions about WHY
systemd is incompatible with 0.8.0 and earlier in another message. But
this template should run and build a container. If it doesn't, I want
to fix that before I submit a patch. I could use some people on
AltLinux and ArchLinux to also test this out on a host and give me
feedback.

Maybe I'll see a few of you at the LinuxPlumbers conference in a few
days in New Orleans.

Regards,
Mike
Post by Michael H. Warfield
But it's all that work just to replace those few lines of code above
where you can not perform a host native install of the rootfs. That's
all that's standing in the way and it's frustratingly close for the
Fedora template.
lxc-alpine
lxc-alt
lxc-arch
lxc-busybox (is this really a distro template - I'm not sure)
lxc-debian
lxc-fedora
lxc-opensuse
lxc-oracle (really?)
lxc-ubuntu / lxc-ubuntu-cloud
Who are the owners? I don't "own" the Fedora template but I work on it.
Each template owner has to solve this problem in their own way and it
all boils down to the same problem. Building that initial rootfs in the
host native environment.
After that, the idea of generalizing the functions into an API form
sounds like a great idea. It could be that this catch-22 could be
buried in a "create_rootfs" function which would create a cacheable
rootfs.
I would love to see templates for CentOS, Scientific (SL), Slackware and
maybe even RedHat EL. NST (Network Security Toolkit) and Kali (fka
BackTrack) would also be nice and I might work on them (which can I say,
I'm a security fanatic). But I gotta solve this problem first in a way
I can use for the ones I might want to write down the road.
But, in the meant time, we have "the recipe for rabbit stew"... First
you catch your rabbit. First we have to have a way to build the initial
rootfs.
<aside>
This entire process reminds me of when we (I) had to bootstrap gcc into
environments (HPUX and AIX years ago) which had no C compiler installed.
First you cross compiled a compiler (on Solaris in my case) for your
target arch and transported it there. Then you used that "stage 1"
compiler to compile the gcc compiler on the native platform (stage 2).
Then you used the stage 2 compiler to compile a full stage 3 compiler
where you could then confirm the outputs from the stage 2 and stage 3
compilers to confirm they matched (which never seems to happen the first
time you try it). When you were done and finally confirmed working, you
really needed a beer. This is the same sort of bootstrap process.
</aside>
Post by Natanael Copa
-nc
Regards,
Mike
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: lxc-fedora
Type: application/x-shellscript
Size: 27544 bytes
Desc: not available
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20130912/81b26597/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20130912/81b26597/attachment.pgp>
Michael H. Warfield
2013-09-13 04:20:14 UTC
Permalink
Post by Michael H. Warfield
All - Especially Tony Su,
Couple of people where I work thought you couldn't do what I was trying
to do, that it was "impossible". Oh well. Looks like they were
wrong. :-P It may not be "efficient" but it can be made to work.
Never fails, I swear, it never fails. Not minutes after I posted this
original message, one of my test runs blew an error that pointed out a
typo that pointed out six other spots where I had environment variables
out of sync and, after fixing them, I realized I had missed several
update paths. And then there were the initialization checks I missed.
Would work fine on fedora on fedora but not what my target was.

Ok... So... I fixed all that. The attached template should actually
work if you're not on a Fedora host (I hope).

It does dawn on me that all this effort really is a "one trick pony"
and, once it has built the very first container, is no longer needed
after the first run. Reason? Each container built can be a run time
environment to build new containers.

So... Before I submit an actual patch back to the upstream project,
I'll probably scope out the logic to scan the cached containers for the
most appropriate container and use it for the RTE and never use the
cached install RTE again.

This may also be appropriate for other templates. Once you've
bootstrapped the template process, you may not need the bootstrap again.
Just build and cache your best shot and use it in the future.

I have tested this template and built containers from F14 through F19.
F15 failed but, I already know it would not have run as a container
anyways because of the horribly broken and incompatible systemd in that
distro. Prior to F14, some versions will build and some won't.
Shouldn't be a problem with Fedora on Fedora and anything prior to F18
is past EOL anyways, so why should I care?

IAC... Please discard the previous template. Here is my current
template. And please test it.

Thanks!

Regards,
Mike
Post by Michael H. Warfield
Way down below, in-line...
Post by Michael H. Warfield
Post by Natanael Copa
On Sun, 08 Sep 2013 20:33:16 -0400
Post by Michael H. Warfield
With all due respect...
Post by Tony Su
After putting some thought into this,
IMO LXC badly needs a universal tool with the following features
- A single script should be used to run on <any> HostOS, creating
<any> supported Container OS. Although this would make the script
extremely large, IMO it would actually be easier to maintain in the
long run.
Actually, no. From my experience (30+ years in software development),
it would turn into a morass.
The problem here is that the maintainer(s) would then need to understand
how each and every distribution is installed and how it would be
installed on each and every distribution. It would distill the worse of
all the problems we have now in the templates into one great big dung
pile. It would rapidly become unmaintainable. The "extremely large" is
the red letter warning that it will become unmaintainable as fewer and
fewer people understand what this great big huge blob does.
I tend to agree with this.
What I do think could be done is to use template APIs. Either by having
generate_mac_address etc) or let the template scripts be "plugins" that
must provide a set of pre-defined functions (eg. install_distro,
configure_distro, copy_configuration etc) or maybe a combination of
those two.
I like this idea, it's just that, in the short term, we have to get past
this one gotcha.
In the git-stage current Fedora template, the entire problem is embodied
in the "download_fedora" function starting around line 201... The
gotcha's are three commands around line 272 after we've identified and
mkdir -p $INSTALL_ROOT/var/lib/rpm
rpm --root $INSTALL_ROOT --initdb
rpm --root $INSTALL_ROOT -ivh ${INSTALL_ROOT}/${RELEASE_RPM}
$YUM install $PKG_LIST
Ok... Those are running in the host local run time environment.
Obviously, if the host does not have a (compatible) version of rpm
and/or yum in the host local environment, you're screwed. That's the
catch-22 situation and it's the same situation with zypper in the
OpenSuse template. That short space of code has to be recreated in a
totally distro-agnostic manner so it runs on any distro to create our
initial rootfs. After that, we can do what ever distro (Fedora)
specific commands we want by chrooting into the target container first
(including rebuilding the rpm database to the target version). That's
only even needed if you don't already have a cached rootfs for that
distro and version.
I was SOOOO close last week working on this while on vacation. Recent
revs of Fedora have this downloadable LiveOS core that runs on the
netinst CD and others. That's the 200MB - 300MB blob I was referring
to. You just download it, mount it (requires squashfs) to a directory
and then mount the ext3 image it contains on another directory. Then
create and bind mount a few rw directories (etc var run), mount proc in
the image, and bind mount your rootfs to run/install in the image. Then
chroot into the image and voila, instant RTE. Yum and rpm are capable
of installing other versions of Fedora, so I wouldn't (shouldn't) even
need a version specific instantiation of the RTE, just one that can
install every version we might be interested in.
Fedora came so close and then face planted for what I wanted to do.
Sure rpm is in there. But rpmdb is not. So, no --initdb and no
--rebuilddb. They've got yum in there, but no yummain.py so yum hurls
chunks immediately upon execution trying to import yummain. What the
hell good does it do to have yum in there but no yummain?!?! But,
something, fixed up, like that from all the distros would be a perfect
answer (albeit somewhat less than high performance thanks to that
download, but it's a single download that could be cached). The only
potential gotcha I see in there is requiring squashfs available for
mounting the image. Anyone have heartburn over that?
That squashfs image has to be able to do an install or it would be
useless on the netinst CD. I'm not sure if they still have anaconda on
there or not but it has to be able to do a kickstart install, so all I
need is a minimal.ks kickstart file to perform essentially an unattended
install of a minimal build into the target rootfs. That's where I'm at
now, trying to get that to work.
I got this to work with just one (err, minor, gag, couch) butt ugly
hack / dirty trick. I seems, in this case, the problem with the Fedora
LiveOS squashfs image is that it has rpm (but no rpmdb) and yum (but no
yummain.py) partially installed and damn near everything else
(dependeices) are but it's not build with an rpm database in place.
The answer was to stage this with the original LiveOS fs being the stage
0 and copying that to a read-write directory and then downloading and
force (--nodeps) an install of rpm and yum installing the full packages
in place for a stage 1 run time environment. That enabled me to create
a stage 2 uber-minimal run time environment with rpm and yum installed
along with all their dependencies. This I could then cache and is
capable of installing arbitrary versions of Fedora (so it only needs to
be done once for any given arch).
I've integrated it into the lxc-fedora template but it has undergone
minimal testing (fedora on fedora). It has a "white listing" facility
in it to run a native install on distros (and also checks if the distro
has yum and rpm available) that will support it but that's disabled on
this, attached, template so it can get more testing. White listing is
based on the CPE OS identifier. What will go into the patch to the
project will have fedora, centos, and maybe a few others whitelisted
that are known to work.
Attached to this message is my trial balloon lxc-fedora template. This
is NOT final. This is just a test for comments and feedback. Please
try it out. Other template owners, PLEASE look at the code and see if
something similar makes sense for your templates! I would love to
install your distros on my Fedora hosts!
The code is commented moderately heavy (and somewhat sarcasticly) but
should explain what I was trying to do (if not - tell me!). The bulk of
the changes are a block of code and comments starting around line 201
plus a few modifications just to the "download_fedora()" routine, now at
line 543 (yes it was a big insert in front of it).
I can not emphasize this enough... READ THE COMMENTS! If I'm full of
shit doing it this way, then tell me so!
Once it builds that RTE the first time, it will use it for all future
Fedora container builds.
Tony: You are on OpenSuse. You, more than anyone else, I really need
to test this code by. I'll answer your other questions about WHY
systemd is incompatible with 0.8.0 and earlier in another message. But
this template should run and build a container. If it doesn't, I want
to fix that before I submit a patch. I could use some people on
AltLinux and ArchLinux to also test this out on a host and give me
feedback.
Maybe I'll see a few of you at the LinuxPlumbers conference in a few
days in New Orleans.
Regards,
Mike
Post by Michael H. Warfield
But it's all that work just to replace those few lines of code above
where you can not perform a host native install of the rootfs. That's
all that's standing in the way and it's frustratingly close for the
Fedora template.
lxc-alpine
lxc-alt
lxc-arch
lxc-busybox (is this really a distro template - I'm not sure)
lxc-debian
lxc-fedora
lxc-opensuse
lxc-oracle (really?)
lxc-ubuntu / lxc-ubuntu-cloud
Who are the owners? I don't "own" the Fedora template but I work on it.
Each template owner has to solve this problem in their own way and it
all boils down to the same problem. Building that initial rootfs in the
host native environment.
After that, the idea of generalizing the functions into an API form
sounds like a great idea. It could be that this catch-22 could be
buried in a "create_rootfs" function which would create a cacheable
rootfs.
I would love to see templates for CentOS, Scientific (SL), Slackware and
maybe even RedHat EL. NST (Network Security Toolkit) and Kali (fka
BackTrack) would also be nice and I might work on them (which can I say,
I'm a security fanatic). But I gotta solve this problem first in a way
I can use for the ones I might want to write down the road.
But, in the meant time, we have "the recipe for rabbit stew"... First
you catch your rabbit. First we have to have a way to build the initial
rootfs.
<aside>
This entire process reminds me of when we (I) had to bootstrap gcc into
environments (HPUX and AIX years ago) which had no C compiler installed.
First you cross compiled a compiler (on Solaris in my case) for your
target arch and transported it there. Then you used that "stage 1"
compiler to compile the gcc compiler on the native platform (stage 2).
Then you used the stage 2 compiler to compile a full stage 3 compiler
where you could then confirm the outputs from the stage 2 and stage 3
compilers to confirm they matched (which never seems to happen the first
time you try it). When you were done and finally confirmed working, you
really needed a beer. This is the same sort of bootstrap process.
</aside>
Post by Natanael Copa
-nc
Regards,
Mike
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: lxc-fedora
Type: application/x-shellscript
Size: 28003 bytes
Desc: not available
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20130913/a499bbcd/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20130913/a499bbcd/attachment.pgp>
Michael H. Warfield
2013-09-13 20:06:56 UTC
Permalink
Post by Michael H. Warfield
Post by Michael H. Warfield
All - Especially Tony Su,
Couple of people where I work thought you couldn't do what I was trying
to do, that it was "impossible". Oh well. Looks like they were
wrong. :-P It may not be "efficient" but it can be made to work.
Never fails, I swear, it never fails. Not minutes after I posted this
original message, one of my test runs blew an error that pointed out a
typo that pointed out six other spots where I had environment variables
out of sync and, after fixing them, I realized I had missed several
update paths. And then there were the initialization checks I missed.
Would work fine on fedora on fedora but not what my target was.
Ok... So... I fixed all that. The attached template should actually
work if you're not on a Fedora host (I hope).
Ok... This morning, I wrapped up some testing of an OpenClient Fedora
19 (IBM internal distribution layer) on a spare Dell GX 620 which I
could then burn that puppy down and retask it. I built it back up as an
OpenSUSE-1203 build and installed lxc on it. Generic (no mods) tried
running lxc-create -t fedora on that. Yeah, that caught fire and burned
real quick. How did you get as far as you did??? No matter. It was
destined to fail anyways.

IAC... I copied my new template in place and it happily built Fedora
19, 18, 17, and 16 containers (I have run the F19 container and I
haven't gotten that far with playing with LXC on OpenSUSE). So this
template should work for you, if you have the other pieces of the puzzle
(generic LXC setup) in place. There IS a problem with versions prior to
F16 and I strongly encourage that NOBODY build containers of these
unsupported version (and F16 is guaranteed to NOT RUN - DON'T TRY)! I'd
like this script to be as generic as possible but there's going to have
to be some limits. F15 has a STUPID dependency on fedora-rawhide but I
can work around that for testing. WTH???

Regards,
Mike
Post by Michael H. Warfield
It does dawn on me that all this effort really is a "one trick pony"
and, once it has built the very first container, is no longer needed
after the first run. Reason? Each container built can be a run time
environment to build new containers.
So... Before I submit an actual patch back to the upstream project,
I'll probably scope out the logic to scan the cached containers for the
most appropriate container and use it for the RTE and never use the
cached install RTE again.
This may also be appropriate for other templates. Once you've
bootstrapped the template process, you may not need the bootstrap again.
Just build and cache your best shot and use it in the future.
I have tested this template and built containers from F14 through F19.
F15 failed but, I already know it would not have run as a container
anyways because of the horribly broken and incompatible systemd in that
distro. Prior to F14, some versions will build and some won't.
Shouldn't be a problem with Fedora on Fedora and anything prior to F18
is past EOL anyways, so why should I care?
IAC... Please discard the previous template. Here is my current
template. And please test it.
Thanks!
Regards,
Mike
Post by Michael H. Warfield
Way down below, in-line...
Post by Michael H. Warfield
Post by Natanael Copa
On Sun, 08 Sep 2013 20:33:16 -0400
Post by Michael H. Warfield
With all due respect...
Post by Tony Su
After putting some thought into this,
IMO LXC badly needs a universal tool with the following features
- A single script should be used to run on <any> HostOS, creating
<any> supported Container OS. Although this would make the script
extremely large, IMO it would actually be easier to maintain in the
long run.
Actually, no. From my experience (30+ years in software development),
it would turn into a morass.
The problem here is that the maintainer(s) would then need to understand
how each and every distribution is installed and how it would be
installed on each and every distribution. It would distill the worse of
all the problems we have now in the templates into one great big dung
pile. It would rapidly become unmaintainable. The "extremely large" is
the red letter warning that it will become unmaintainable as fewer and
fewer people understand what this great big huge blob does.
I tend to agree with this.
What I do think could be done is to use template APIs. Either by having
generate_mac_address etc) or let the template scripts be "plugins" that
must provide a set of pre-defined functions (eg. install_distro,
configure_distro, copy_configuration etc) or maybe a combination of
those two.
I like this idea, it's just that, in the short term, we have to get past
this one gotcha.
In the git-stage current Fedora template, the entire problem is embodied
in the "download_fedora" function starting around line 201... The
gotcha's are three commands around line 272 after we've identified and
mkdir -p $INSTALL_ROOT/var/lib/rpm
rpm --root $INSTALL_ROOT --initdb
rpm --root $INSTALL_ROOT -ivh ${INSTALL_ROOT}/${RELEASE_RPM}
$YUM install $PKG_LIST
Ok... Those are running in the host local run time environment.
Obviously, if the host does not have a (compatible) version of rpm
and/or yum in the host local environment, you're screwed. That's the
catch-22 situation and it's the same situation with zypper in the
OpenSuse template. That short space of code has to be recreated in a
totally distro-agnostic manner so it runs on any distro to create our
initial rootfs. After that, we can do what ever distro (Fedora)
specific commands we want by chrooting into the target container first
(including rebuilding the rpm database to the target version). That's
only even needed if you don't already have a cached rootfs for that
distro and version.
I was SOOOO close last week working on this while on vacation. Recent
revs of Fedora have this downloadable LiveOS core that runs on the
netinst CD and others. That's the 200MB - 300MB blob I was referring
to. You just download it, mount it (requires squashfs) to a directory
and then mount the ext3 image it contains on another directory. Then
create and bind mount a few rw directories (etc var run), mount proc in
the image, and bind mount your rootfs to run/install in the image. Then
chroot into the image and voila, instant RTE. Yum and rpm are capable
of installing other versions of Fedora, so I wouldn't (shouldn't) even
need a version specific instantiation of the RTE, just one that can
install every version we might be interested in.
Fedora came so close and then face planted for what I wanted to do.
Sure rpm is in there. But rpmdb is not. So, no --initdb and no
--rebuilddb. They've got yum in there, but no yummain.py so yum hurls
chunks immediately upon execution trying to import yummain. What the
hell good does it do to have yum in there but no yummain?!?! But,
something, fixed up, like that from all the distros would be a perfect
answer (albeit somewhat less than high performance thanks to that
download, but it's a single download that could be cached). The only
potential gotcha I see in there is requiring squashfs available for
mounting the image. Anyone have heartburn over that?
That squashfs image has to be able to do an install or it would be
useless on the netinst CD. I'm not sure if they still have anaconda on
there or not but it has to be able to do a kickstart install, so all I
need is a minimal.ks kickstart file to perform essentially an unattended
install of a minimal build into the target rootfs. That's where I'm at
now, trying to get that to work.
I got this to work with just one (err, minor, gag, couch) butt ugly
hack / dirty trick. I seems, in this case, the problem with the Fedora
LiveOS squashfs image is that it has rpm (but no rpmdb) and yum (but no
yummain.py) partially installed and damn near everything else
(dependeices) are but it's not build with an rpm database in place.
The answer was to stage this with the original LiveOS fs being the stage
0 and copying that to a read-write directory and then downloading and
force (--nodeps) an install of rpm and yum installing the full packages
in place for a stage 1 run time environment. That enabled me to create
a stage 2 uber-minimal run time environment with rpm and yum installed
along with all their dependencies. This I could then cache and is
capable of installing arbitrary versions of Fedora (so it only needs to
be done once for any given arch).
I've integrated it into the lxc-fedora template but it has undergone
minimal testing (fedora on fedora). It has a "white listing" facility
in it to run a native install on distros (and also checks if the distro
has yum and rpm available) that will support it but that's disabled on
this, attached, template so it can get more testing. White listing is
based on the CPE OS identifier. What will go into the patch to the
project will have fedora, centos, and maybe a few others whitelisted
that are known to work.
Attached to this message is my trial balloon lxc-fedora template. This
is NOT final. This is just a test for comments and feedback. Please
try it out. Other template owners, PLEASE look at the code and see if
something similar makes sense for your templates! I would love to
install your distros on my Fedora hosts!
The code is commented moderately heavy (and somewhat sarcasticly) but
should explain what I was trying to do (if not - tell me!). The bulk of
the changes are a block of code and comments starting around line 201
plus a few modifications just to the "download_fedora()" routine, now at
line 543 (yes it was a big insert in front of it).
I can not emphasize this enough... READ THE COMMENTS! If I'm full of
shit doing it this way, then tell me so!
Once it builds that RTE the first time, it will use it for all future
Fedora container builds.
Tony: You are on OpenSuse. You, more than anyone else, I really need
to test this code by. I'll answer your other questions about WHY
systemd is incompatible with 0.8.0 and earlier in another message. But
this template should run and build a container. If it doesn't, I want
to fix that before I submit a patch. I could use some people on
AltLinux and ArchLinux to also test this out on a host and give me
feedback.
Maybe I'll see a few of you at the LinuxPlumbers conference in a few
days in New Orleans.
Regards,
Mike
Post by Michael H. Warfield
But it's all that work just to replace those few lines of code above
where you can not perform a host native install of the rootfs. That's
all that's standing in the way and it's frustratingly close for the
Fedora template.
lxc-alpine
lxc-alt
lxc-arch
lxc-busybox (is this really a distro template - I'm not sure)
lxc-debian
lxc-fedora
lxc-opensuse
lxc-oracle (really?)
lxc-ubuntu / lxc-ubuntu-cloud
Who are the owners? I don't "own" the Fedora template but I work on it.
Each template owner has to solve this problem in their own way and it
all boils down to the same problem. Building that initial rootfs in the
host native environment.
After that, the idea of generalizing the functions into an API form
sounds like a great idea. It could be that this catch-22 could be
buried in a "create_rootfs" function which would create a cacheable
rootfs.
I would love to see templates for CentOS, Scientific (SL), Slackware and
maybe even RedHat EL. NST (Network Security Toolkit) and Kali (fka
BackTrack) would also be nice and I might work on them (which can I say,
I'm a security fanatic). But I gotta solve this problem first in a way
I can use for the ones I might want to write down the road.
But, in the meant time, we have "the recipe for rabbit stew"... First
you catch your rabbit. First we have to have a way to build the initial
rootfs.
<aside>
This entire process reminds me of when we (I) had to bootstrap gcc into
environments (HPUX and AIX years ago) which had no C compiler installed.
First you cross compiled a compiler (on Solaris in my case) for your
target arch and transported it there. Then you used that "stage 1"
compiler to compile the gcc compiler on the native platform (stage 2).
Then you used the stage 2 compiler to compile a full stage 3 compiler
where you could then confirm the outputs from the stage 2 and stage 3
compilers to confirm they matched (which never seems to happen the first
time you try it). When you were done and finally confirmed working, you
really needed a beer. This is the same sort of bootstrap process.
</aside>
Post by Natanael Copa
-nc
Regards,
Mike
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20130913/83972cb0/attachment.pgp>
Tony Su
2013-09-14 22:35:25 UTC
Permalink
Cool.

I'll block some significant time to look at what you built over the next 3
days.

Tony
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20130914/8a8573c9/attachment.html>
Tony Su
2013-09-15 23:17:04 UTC
Permalink
Hello Michael,

First a comment on problems with systemd you descrbe.
I probably have run into many of the things you itemized, but since my
time is usually focused on something I'm trying to use LXC and not LXC
itself, I usually just drop any further attempts and move on to find a
workaround(eg consoles) or use a different technology(x server issue).

Regarding many of the issues you describe though, I wonder if they
couldn't be addressed with more strict enforcement of using namespaces
(and less often cgroups). I've read how namespaces are supposed to be
an extremely powerful means of isolating processes and yet I don't see
any obvious indications it's being done consistently... by either
prepending to standard process or service names (if the goal is to
easily identify the namespace) or using a random string (if the goal
is better security so exploits can't anticipate commonly used
namespaces).

In fact, I think I see this namespace issue in various parts of the
template you created. If I understand what is happening, there are
numerous places where you create special nodes on the HostOS instead
of
(a) using the existing HostOS nodes but using namespaces to isolate
Container processes
(b) creating nodes entirely within the Container which would make the
Container entirely portable but lose the benefit perhaps of the better
ways nodes are created and mounted today(eg tmpfs in RAM).

Diving more into your template code, I applaud your effort, it's
significant and no minor effort.

As of this moment, I've mainly been perusing what I might call "HostOS
Container Pre-Install," the part which precedes the actual
installation and relies on components running in the HostOS only. This
would be your script approx lines 0-410.

1. I like your method of identifying whether the OS is Fedora, and
additionally whether is ARM or not.

2. It looks like you're configuring networking binding directly to
eth0. I would recommend instead supporting the use of Linux Bridge
devices, make declaration of a bridge device name as one of the early
Global Parameters, then if exists to bind to that device by name. Your
code to bind to the physical interface is less flexible but can be a
default option if no bridge device is specified.

3. Interesting that you include an option for "nm controlled" yet at
least initially I don't see where your code might rely on this
setting.

4. mknod I'll have to take a closer look whether and why you appear to
be setting up various consoles, some /dev/ nodes, an explicit console
path, more. I've generally been under the impression that a full
install automatically creates these. Peeking a bit ahead of where I've
been reading your script, I notice your install method uses a
pre-built squashfs image, perhaps these are a special requirement
because your chosen squashfs image doesn't include these by default or
requires those nodes to already exist?

5. Your use of yum will work in the RH family plus various others like
openSUSE but I don't think it's native to distros like the Debian
family. IMO there is no special benefit to using a package manager
specific to the HostOS to download bootstrap images and packages, they
aren't too relevant to the overall apps running in the HostOS and I
think we should avoid installing non-native packages in any OS. For
that reason, I've been looking at pycurl, curl and wget which are
generic apps common to all distros which can accomplish the simple
task of retrieving the bootstrap objects. (See the template I included
as an attachment which uses pycurl and finds fedora repos rather than
installing a pre-built img)

A small FYI -
Although the Fedora template distributed with openSUSE which I've
included as an attachment to this message <does not work as is> it
might be useful to see a different way of obtaining and installing
Fedora bootstrap packages so I've included it as an attachment to this
message.

I've made the two following modifications
1. line 33 - added "release=18"
The comments in this script describe passing the release number as an
option to "lxc-create" but is not supported in openSUSE. Despite
unable to pass as a command line option, passing within the template
with this line works. BTW - If no release is specified, Fedora
defaults to earliest release in the repos(which is 14) rather than
latest.
2. Line 153 - The template hardcodes the RELEASE_URL string which is
created by appending a hardcoded string to the MIRROR_URL string, but
it appears that Fedora restructured their repos since this template
was created. Now, an "/f/" has to be inserted into the RELEASE_URL
(initial letter of the word "fedora").

Additional - Especially when connecting to repos of a Distro different
than the HostOS, GPG authentication keys are not yet installed. Have
been investigating whether it's possible to simply download ahead of
time and install into the default Key Ring or something more is
required. If this approach is feasable, then this needs to be added
early to the template script, but maybe a better method is for the
User to be prompted for an interactive answer to confirm key
installation.

Tony
Post by Tony Su
Cool.
I'll block some significant time to look at what you built over the next 3
days.
Tony
-------------- next part --------------
A non-text attachment was scrubbed...
Name: lxc-fedora18
Type: application/octet-stream
Size: 11411 bytes
Desc: not available
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20130915/37c8d7f4/attachment.obj>
Tony Su
2013-09-16 17:28:17 UTC
Permalink
Michael,
Considering your substantial existing work, if you'd post on github,
I'd we willing to fork and contribute in that manner (of course, also
opens up ways for others to contribute).

I'd create the original repo myself except that your work should be
credited by you (and contributions managed by you as well).

Tony
After some additional thought,
If the goal is a template which will execute with minimal issues on
non-Fedora HostOS,
Besides my observations in my previous post to this,
IMO the vaarious lines which "touch /etc/fstab" also would need to be
modified since IMO the mount points of current, later versions of
Fedora can be sometimes downright weird and wouldn't produce the
desired result when executing that command on other distros.
Tony
Post by Tony Su
Hello Michael,
First a comment on problems with systemd you descrbe.
I probably have run into many of the things you itemized, but since my
time is usually focused on something I'm trying to use LXC and not LXC
itself, I usually just drop any further attempts and move on to find a
workaround(eg consoles) or use a different technology(x server issue).
Regarding many of the issues you describe though, I wonder if they
couldn't be addressed with more strict enforcement of using namespaces
(and less often cgroups). I've read how namespaces are supposed to be
an extremely powerful means of isolating processes and yet I don't see
any obvious indications it's being done consistently... by either
prepending to standard process or service names (if the goal is to
easily identify the namespace) or using a random string (if the goal
is better security so exploits can't anticipate commonly used
namespaces).
In fact, I think I see this namespace issue in various parts of the
template you created. If I understand what is happening, there are
numerous places where you create special nodes on the HostOS instead
of
(a) using the existing HostOS nodes but using namespaces to isolate
Container processes
(b) creating nodes entirely within the Container which would make the
Container entirely portable but lose the benefit perhaps of the better
ways nodes are created and mounted today(eg tmpfs in RAM).
Diving more into your template code, I applaud your effort, it's
significant and no minor effort.
As of this moment, I've mainly been perusing what I might call "HostOS
Container Pre-Install," the part which precedes the actual
installation and relies on components running in the HostOS only. This
would be your script approx lines 0-410.
1. I like your method of identifying whether the OS is Fedora, and
additionally whether is ARM or not.
2. It looks like you're configuring networking binding directly to
eth0. I would recommend instead supporting the use of Linux Bridge
devices, make declaration of a bridge device name as one of the early
Global Parameters, then if exists to bind to that device by name. Your
code to bind to the physical interface is less flexible but can be a
default option if no bridge device is specified.
3. Interesting that you include an option for "nm controlled" yet at
least initially I don't see where your code might rely on this
setting.
4. mknod I'll have to take a closer look whether and why you appear to
be setting up various consoles, some /dev/ nodes, an explicit console
path, more. I've generally been under the impression that a full
install automatically creates these. Peeking a bit ahead of where I've
been reading your script, I notice your install method uses a
pre-built squashfs image, perhaps these are a special requirement
because your chosen squashfs image doesn't include these by default or
requires those nodes to already exist?
5. Your use of yum will work in the RH family plus various others like
openSUSE but I don't think it's native to distros like the Debian
family. IMO there is no special benefit to using a package manager
specific to the HostOS to download bootstrap images and packages, they
aren't too relevant to the overall apps running in the HostOS and I
think we should avoid installing non-native packages in any OS. For
that reason, I've been looking at pycurl, curl and wget which are
generic apps common to all distros which can accomplish the simple
task of retrieving the bootstrap objects. (See the template I included
as an attachment which uses pycurl and finds fedora repos rather than
installing a pre-built img)
A small FYI -
Although the Fedora template distributed with openSUSE which I've
included as an attachment to this message <does not work as is> it
might be useful to see a different way of obtaining and installing
Fedora bootstrap packages so I've included it as an attachment to this
message.
I've made the two following modifications
1. line 33 - added "release=18"
The comments in this script describe passing the release number as an
option to "lxc-create" but is not supported in openSUSE. Despite
unable to pass as a command line option, passing within the template
with this line works. BTW - If no release is specified, Fedora
defaults to earliest release in the repos(which is 14) rather than
latest.
2. Line 153 - The template hardcodes the RELEASE_URL string which is
created by appending a hardcoded string to the MIRROR_URL string, but
it appears that Fedora restructured their repos since this template
was created. Now, an "/f/" has to be inserted into the RELEASE_URL
(initial letter of the word "fedora").
Additional - Especially when connecting to repos of a Distro different
than the HostOS, GPG authentication keys are not yet installed. Have
been investigating whether it's possible to simply download ahead of
time and install into the default Key Ring or something more is
required. If this approach is feasable, then this needs to be added
early to the template script, but maybe a better method is for the
User to be prompted for an interactive answer to confirm key
installation.
Tony
Post by Tony Su
Cool.
I'll block some significant time to look at what you built over the next 3
days.
Tony
Michael H. Warfield
2013-09-17 13:34:18 UTC
Permalink
Post by Tony Su
Hello Michael,
First a comment on problems with systemd you descrbe.
I probably have run into many of the things you itemized, but since my
time is usually focused on something I'm trying to use LXC and not LXC
itself, I usually just drop any further attempts and move on to find a
workaround(eg consoles) or use a different technology(x server issue).
Regarding many of the issues you describe though, I wonder if they
couldn't be addressed with more strict enforcement of using namespaces
(and less often cgroups). I've read how namespaces are supposed to be
an extremely powerful means of isolating processes and yet I don't see
any obvious indications it's being done consistently... by either
prepending to standard process or service names (if the goal is to
easily identify the namespace) or using a random string (if the goal
is better security so exploits can't anticipate commonly used
namespaces).
In fact, I think I see this namespace issue in various parts of the
template you created. If I understand what is happening, there are
numerous places where you create special nodes on the HostOS instead
of
(a) using the existing HostOS nodes but using namespaces to isolate
Container processes
(b) creating nodes entirely within the Container which would make the
Container entirely portable but lose the benefit perhaps of the better
ways nodes are created and mounted today(eg tmpfs in RAM).
Diving more into your template code, I applaud your effort, it's
significant and no minor effort.
As of this moment, I've mainly been perusing what I might call "HostOS
Container Pre-Install," the part which precedes the actual
installation and relies on components running in the HostOS only. This
would be your script approx lines 0-410.
1. I like your method of identifying whether the OS is Fedora, and
additionally whether is ARM or not.
That was an effort working on my Raspberry Pi's.
Post by Tony Su
2. It looks like you're configuring networking binding directly to
eth0. I would recommend instead supporting the use of Linux Bridge
devices, make declaration of a bridge device name as one of the early
Global Parameters, then if exists to bind to that device by name. Your
code to bind to the physical interface is less flexible but can be a
default option if no bridge device is specified.
Original code. I haven't really looked it it. As far as bridges and
all go, much of that is in the greater LXC setup and the default
settings in /etc/lxc/default.conf and outside of our control.
Post by Tony Su
3. Interesting that you include an option for "nm controlled" yet at
least initially I don't see where your code might rely on this
setting.
My code does not. In the back of my mind, the setting for
NetworkMangler control is something that is used internally to the
container and we're merely setting a default value.
Post by Tony Su
4. mknod I'll have to take a closer look whether and why you appear to
be setting up various consoles, some /dev/ nodes, an explicit console
path, more. I've generally been under the impression that a full
install automatically creates these. Peeking a bit ahead of where I've
been reading your script, I notice your install method uses a
pre-built squashfs image, perhaps these are a special requirement
because your chosen squashfs image doesn't include these by default or
requires those nodes to already exist?
5. Your use of yum will work in the RH family plus various others like
openSUSE but I don't think it's native to distros like the Debian
family. IMO there is no special benefit to using a package manager
specific to the HostOS to download bootstrap images and packages, they
aren't too relevant to the overall apps running in the HostOS and I
think we should avoid installing non-native packages in any OS. For
that reason, I've been looking at pycurl, curl and wget which are
generic apps common to all distros which can accomplish the simple
task of retrieving the bootstrap objects. (See the template I included
as an attachment which uses pycurl and finds fedora repos rather than
installing a pre-built img)
Because yum (and rpm) are no ONLY and EVER executed in a chrooted
environment, they are executing in a Fedora context and independent of
the host environment. That's the whole point, it never execute the
distro-dependent components in the host environment and only execute
distro-agnostic (independent) components in the host environment.
Post by Tony Su
A small FYI -
Although the Fedora template distributed with openSUSE which I've
included as an attachment to this message <does not work as is> it
might be useful to see a different way of obtaining and installing
Fedora bootstrap packages so I've included it as an attachment to this
message.
I've said this before. Take it to the bank. The template that is
distributed with OpenSuse is OUT OF DATE AND WILL NOT WORK! In fact,
the entire LXC package that is distributed with OpenSuse is out of date,
but that is beyond our control.

The template I sent out is also very preliminary and will be heavily
revised before being submitted as a patch (now, after the LinuxPlumbers
conference). It may not even get into the 1.0.0 release at this point
as we're already seeing our first Alpha tags and pulls.
Post by Tony Su
I've made the two following modifications
1. line 33 - added "release=18"
The comments in this script describe passing the release number as an
option to "lxc-create" but is not supported in openSUSE. Despite
unable to pass as a command line option, passing within the template
with this line works. BTW - If no release is specified, Fedora
defaults to earliest release in the repos(which is 14) rather than
latest.
2. Line 153 - The template hardcodes the RELEASE_URL string which is
created by appending a hardcoded string to the MIRROR_URL string, but
it appears that Fedora restructured their repos since this template
was created. Now, an "/f/" has to be inserted into the RELEASE_URL
(initial letter of the word "fedora").
Fedora, after a certain release, changed their "Packages" directory to
include a first letter of the package as a subdirectory. So
"fedora-release" is in "Packages/f/fedora-release-*" and yum is in
"Packages/y/yum-*". That's the Fedora convention now as of a certain
rev of Fedora. There's an if check in there for it.
Post by Tony Su
Additional - Especially when connecting to repos of a Distro different
than the HostOS, GPG authentication keys are not yet installed. Have
been investigating whether it's possible to simply download ahead of
time and install into the default Key Ring or something more is
required. If this approach is feasable, then this needs to be added
early to the template script, but maybe a better method is for the
User to be prompted for an interactive answer to confirm key
installation.
Actually, this should now be largely resolved by using the LiveOS image
and minimizing the use of --nogpgcheck to yum, but I'll verify that.
There is a catch-22 of verifying the LiveOS image but I'm not sure I
have a solution to that one.
Post by Tony Su
Tony
Post by Tony Su
Cool.
I'll block some significant time to look at what you built over the next 3
days.
Tony
Later!

Regards,
Mike
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20130917/aa1a6998/attachment.pgp>
Tony Su
2013-09-17 17:26:08 UTC
Permalink
Regarding

LXC and the use of Linux Bridge devices for configured networking,
At least on openSUSE, LXC is not configured with any /etc/lxc/* by
default, and possibly because I also have libvirt configured to
support LXC (although that is not working on my system).
From what I've seen though, I cannot see why configuring networking
should be in a general lxc configuration file, or even if it should
exist I see it as reaasonable to configure within a template(why not
point the build to a pre-existing virtual network complete with its
own configuration?).

In any case as I noted before, pointing the Container interface to a
pre-existing virtual Linux Bridge device instead of a physical
interface in the template should be nearly trivial. If you wanted to
support bridge creation and configuration, that'd be a big project
outside the scope of what I'm describing.

********************************************
yum and chroot. Hmmmm. From what I can see, at the point you invoke
yum, nothing has yet been downloaded so any functionality you invoke
must exist in the HostOS. Despite running in the chroot environment, I
can't see how yum can work unless it's also present in the HostOS.
Maybe if I test this on a distro that doesn't support yum natively
I'll find different, but based purely on perusing the code I can't see
how it would work otherwise.

line 155
chroot ${rootfs_path} yum --releasever=${release} -y install fedora-release

********************************************
Requirement for GPG keys
Again, based at this point purely on perusing and not actually testing
the code on an appropriately set up system, unless the repo is
configured without keys or the retrieval utility is configured not to
require key verification, I don't see how you've avoided this
requirement, particularly in this case you're using "yum" which is a
repo client utility. This is aside from verifying the downloaded image
whose integrity could be verified easily by invoking and comparing
checksum values if desired. I'd recommend no verification, though
because it would add maintenance issues. A superior solution might be
a file transfer transport that automatically does checksum (like
torrent but that opens up potential issues since some ISPs block
torrent indiscriminitately).

Regards,
Tony
Post by Tony Su
Hello Michael,
First a comment on problems with systemd you descrbe.
I probably have run into many of the things you itemized, but since my
time is usually focused on something I'm trying to use LXC and not LXC
itself, I usually just drop any further attempts and move on to find a
workaround(eg consoles) or use a different technology(x server issue).
Regarding many of the issues you describe though, I wonder if they
couldn't be addressed with more strict enforcement of using namespaces
(and less often cgroups). I've read how namespaces are supposed to be
an extremely powerful means of isolating processes and yet I don't see
any obvious indications it's being done consistently... by either
prepending to standard process or service names (if the goal is to
easily identify the namespace) or using a random string (if the goal
is better security so exploits can't anticipate commonly used
namespaces).
In fact, I think I see this namespace issue in various parts of the
template you created. If I understand what is happening, there are
numerous places where you create special nodes on the HostOS instead
of
(a) using the existing HostOS nodes but using namespaces to isolate
Container processes
(b) creating nodes entirely within the Container which would make the
Container entirely portable but lose the benefit perhaps of the better
ways nodes are created and mounted today(eg tmpfs in RAM).
Diving more into your template code, I applaud your effort, it's
significant and no minor effort.
As of this moment, I've mainly been perusing what I might call "HostOS
Container Pre-Install," the part which precedes the actual
installation and relies on components running in the HostOS only. This
would be your script approx lines 0-410.
1. I like your method of identifying whether the OS is Fedora, and
additionally whether is ARM or not.
That was an effort working on my Raspberry Pi's.
Post by Tony Su
2. It looks like you're configuring networking binding directly to
eth0. I would recommend instead supporting the use of Linux Bridge
devices, make declaration of a bridge device name as one of the early
Global Parameters, then if exists to bind to that device by name. Your
code to bind to the physical interface is less flexible but can be a
default option if no bridge device is specified.
Original code. I haven't really looked it it. As far as bridges and
all go, much of that is in the greater LXC setup and the default
settings in /etc/lxc/default.conf and outside of our control.
Post by Tony Su
3. Interesting that you include an option for "nm controlled" yet at
least initially I don't see where your code might rely on this
setting.
My code does not. In the back of my mind, the setting for
NetworkMangler control is something that is used internally to the
container and we're merely setting a default value.
Post by Tony Su
4. mknod I'll have to take a closer look whether and why you appear to
be setting up various consoles, some /dev/ nodes, an explicit console
path, more. I've generally been under the impression that a full
install automatically creates these. Peeking a bit ahead of where I've
been reading your script, I notice your install method uses a
pre-built squashfs image, perhaps these are a special requirement
because your chosen squashfs image doesn't include these by default or
requires those nodes to already exist?
5. Your use of yum will work in the RH family plus various others like
openSUSE but I don't think it's native to distros like the Debian
family. IMO there is no special benefit to using a package manager
specific to the HostOS to download bootstrap images and packages, they
aren't too relevant to the overall apps running in the HostOS and I
think we should avoid installing non-native packages in any OS. For
that reason, I've been looking at pycurl, curl and wget which are
generic apps common to all distros which can accomplish the simple
task of retrieving the bootstrap objects. (See the template I included
as an attachment which uses pycurl and finds fedora repos rather than
installing a pre-built img)
Because yum (and rpm) are no ONLY and EVER executed in a chrooted
environment, they are executing in a Fedora context and independent of
the host environment. That's the whole point, it never execute the
distro-dependent components in the host environment and only execute
distro-agnostic (independent) components in the host environment.
Post by Tony Su
A small FYI -
Although the Fedora template distributed with openSUSE which I've
included as an attachment to this message <does not work as is> it
might be useful to see a different way of obtaining and installing
Fedora bootstrap packages so I've included it as an attachment to this
message.
I've said this before. Take it to the bank. The template that is
distributed with OpenSuse is OUT OF DATE AND WILL NOT WORK! In fact,
the entire LXC package that is distributed with OpenSuse is out of date,
but that is beyond our control.
The template I sent out is also very preliminary and will be heavily
revised before being submitted as a patch (now, after the LinuxPlumbers
conference). It may not even get into the 1.0.0 release at this point
as we're already seeing our first Alpha tags and pulls.
Post by Tony Su
I've made the two following modifications
1. line 33 - added "release=18"
The comments in this script describe passing the release number as an
option to "lxc-create" but is not supported in openSUSE. Despite
unable to pass as a command line option, passing within the template
with this line works. BTW - If no release is specified, Fedora
defaults to earliest release in the repos(which is 14) rather than
latest.
2. Line 153 - The template hardcodes the RELEASE_URL string which is
created by appending a hardcoded string to the MIRROR_URL string, but
it appears that Fedora restructured their repos since this template
was created. Now, an "/f/" has to be inserted into the RELEASE_URL
(initial letter of the word "fedora").
Fedora, after a certain release, changed their "Packages" directory to
include a first letter of the package as a subdirectory. So
"fedora-release" is in "Packages/f/fedora-release-*" and yum is in
"Packages/y/yum-*". That's the Fedora convention now as of a certain
rev of Fedora. There's an if check in there for it.
Post by Tony Su
Additional - Especially when connecting to repos of a Distro different
than the HostOS, GPG authentication keys are not yet installed. Have
been investigating whether it's possible to simply download ahead of
time and install into the default Key Ring or something more is
required. If this approach is feasable, then this needs to be added
early to the template script, but maybe a better method is for the
User to be prompted for an interactive answer to confirm key
installation.
Actually, this should now be largely resolved by using the LiveOS image
and minimizing the use of --nogpgcheck to yum, but I'll verify that.
There is a catch-22 of verifying the LiveOS image but I'm not sure I
have a solution to that one.
Post by Tony Su
Tony
Post by Tony Su
Cool.
I'll block some significant time to look at what you built over the next 3
days.
Tony
Later!
Regards,
Mike
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
Michael H. Warfield
2013-09-20 21:34:51 UTC
Permalink
Post by Tony Su
Regarding
LXC and the use of Linux Bridge devices for configured networking,
At least on openSUSE, LXC is not configured with any /etc/lxc/* by
default, and possibly because I also have libvirt configured to
support LXC (although that is not working on my system).
From what I've seen though, I cannot see why configuring networking
should be in a general lxc configuration file, or even if it should
exist I see it as reaasonable to configure within a template(why not
point the build to a pre-existing virtual network complete with its
own configuration?).
In any case as I noted before, pointing the Container interface to a
pre-existing virtual Linux Bridge device instead of a physical
interface in the template should be nearly trivial. If you wanted to
support bridge creation and configuration, that'd be a big project
outside the scope of what I'm describing.
********************************************
yum and chroot. Hmmmm. From what I can see, at the point you invoke
yum, nothing has yet been downloaded so any functionality you invoke
must exist in the HostOS. Despite running in the chroot environment, I
can't see how yum can work unless it's also present in the HostOS.
Maybe if I test this on a distro that doesn't support yum natively
I'll find different, but based purely on perusing the code I can't see
how it would work otherwise.
line 155
chroot ${rootfs_path} yum --releasever=${release} -y install fedora-release
That line is in "config_fedora, called at line 926 after the call to
install_fedora at line 920 where the run time install is downloaded and
run. It should be installed at the time in question or something
earlier has failed. Flow of control definitely needs to be cleaned up.

Regards,
Mike
Post by Tony Su
********************************************
Requirement for GPG keys
Again, based at this point purely on perusing and not actually testing
the code on an appropriately set up system, unless the repo is
configured without keys or the retrieval utility is configured not to
require key verification, I don't see how you've avoided this
requirement, particularly in this case you're using "yum" which is a
repo client utility. This is aside from verifying the downloaded image
whose integrity could be verified easily by invoking and comparing
checksum values if desired. I'd recommend no verification, though
because it would add maintenance issues. A superior solution might be
a file transfer transport that automatically does checksum (like
torrent but that opens up potential issues since some ISPs block
torrent indiscriminitately).
Regards,
Tony
Post by Michael H. Warfield
Post by Tony Su
Hello Michael,
First a comment on problems with systemd you descrbe.
I probably have run into many of the things you itemized, but since my
time is usually focused on something I'm trying to use LXC and not LXC
itself, I usually just drop any further attempts and move on to find a
workaround(eg consoles) or use a different technology(x server issue).
Regarding many of the issues you describe though, I wonder if they
couldn't be addressed with more strict enforcement of using namespaces
(and less often cgroups). I've read how namespaces are supposed to be
an extremely powerful means of isolating processes and yet I don't see
any obvious indications it's being done consistently... by either
prepending to standard process or service names (if the goal is to
easily identify the namespace) or using a random string (if the goal
is better security so exploits can't anticipate commonly used
namespaces).
In fact, I think I see this namespace issue in various parts of the
template you created. If I understand what is happening, there are
numerous places where you create special nodes on the HostOS instead
of
(a) using the existing HostOS nodes but using namespaces to isolate
Container processes
(b) creating nodes entirely within the Container which would make the
Container entirely portable but lose the benefit perhaps of the better
ways nodes are created and mounted today(eg tmpfs in RAM).
Diving more into your template code, I applaud your effort, it's
significant and no minor effort.
As of this moment, I've mainly been perusing what I might call "HostOS
Container Pre-Install," the part which precedes the actual
installation and relies on components running in the HostOS only. This
would be your script approx lines 0-410.
1. I like your method of identifying whether the OS is Fedora, and
additionally whether is ARM or not.
That was an effort working on my Raspberry Pi's.
Post by Tony Su
2. It looks like you're configuring networking binding directly to
eth0. I would recommend instead supporting the use of Linux Bridge
devices, make declaration of a bridge device name as one of the early
Global Parameters, then if exists to bind to that device by name. Your
code to bind to the physical interface is less flexible but can be a
default option if no bridge device is specified.
Original code. I haven't really looked it it. As far as bridges and
all go, much of that is in the greater LXC setup and the default
settings in /etc/lxc/default.conf and outside of our control.
Post by Tony Su
3. Interesting that you include an option for "nm controlled" yet at
least initially I don't see where your code might rely on this
setting.
My code does not. In the back of my mind, the setting for
NetworkMangler control is something that is used internally to the
container and we're merely setting a default value.
Post by Tony Su
4. mknod I'll have to take a closer look whether and why you appear to
be setting up various consoles, some /dev/ nodes, an explicit console
path, more. I've generally been under the impression that a full
install automatically creates these. Peeking a bit ahead of where I've
been reading your script, I notice your install method uses a
pre-built squashfs image, perhaps these are a special requirement
because your chosen squashfs image doesn't include these by default or
requires those nodes to already exist?
5. Your use of yum will work in the RH family plus various others like
openSUSE but I don't think it's native to distros like the Debian
family. IMO there is no special benefit to using a package manager
specific to the HostOS to download bootstrap images and packages, they
aren't too relevant to the overall apps running in the HostOS and I
think we should avoid installing non-native packages in any OS. For
that reason, I've been looking at pycurl, curl and wget which are
generic apps common to all distros which can accomplish the simple
task of retrieving the bootstrap objects. (See the template I included
as an attachment which uses pycurl and finds fedora repos rather than
installing a pre-built img)
Because yum (and rpm) are no ONLY and EVER executed in a chrooted
environment, they are executing in a Fedora context and independent of
the host environment. That's the whole point, it never execute the
distro-dependent components in the host environment and only execute
distro-agnostic (independent) components in the host environment.
Post by Tony Su
A small FYI -
Although the Fedora template distributed with openSUSE which I've
included as an attachment to this message <does not work as is> it
might be useful to see a different way of obtaining and installing
Fedora bootstrap packages so I've included it as an attachment to this
message.
I've said this before. Take it to the bank. The template that is
distributed with OpenSuse is OUT OF DATE AND WILL NOT WORK! In fact,
the entire LXC package that is distributed with OpenSuse is out of date,
but that is beyond our control.
The template I sent out is also very preliminary and will be heavily
revised before being submitted as a patch (now, after the LinuxPlumbers
conference). It may not even get into the 1.0.0 release at this point
as we're already seeing our first Alpha tags and pulls.
Post by Tony Su
I've made the two following modifications
1. line 33 - added "release=18"
The comments in this script describe passing the release number as an
option to "lxc-create" but is not supported in openSUSE. Despite
unable to pass as a command line option, passing within the template
with this line works. BTW - If no release is specified, Fedora
defaults to earliest release in the repos(which is 14) rather than
latest.
2. Line 153 - The template hardcodes the RELEASE_URL string which is
created by appending a hardcoded string to the MIRROR_URL string, but
it appears that Fedora restructured their repos since this template
was created. Now, an "/f/" has to be inserted into the RELEASE_URL
(initial letter of the word "fedora").
Fedora, after a certain release, changed their "Packages" directory to
include a first letter of the package as a subdirectory. So
"fedora-release" is in "Packages/f/fedora-release-*" and yum is in
"Packages/y/yum-*". That's the Fedora convention now as of a certain
rev of Fedora. There's an if check in there for it.
Post by Tony Su
Additional - Especially when connecting to repos of a Distro different
than the HostOS, GPG authentication keys are not yet installed. Have
been investigating whether it's possible to simply download ahead of
time and install into the default Key Ring or something more is
required. If this approach is feasable, then this needs to be added
early to the template script, but maybe a better method is for the
User to be prompted for an interactive answer to confirm key
installation.
Actually, this should now be largely resolved by using the LiveOS image
and minimizing the use of --nogpgcheck to yum, but I'll verify that.
There is a catch-22 of verifying the LiveOS image but I'm not sure I
have a solution to that one.
Post by Tony Su
Tony
Post by Tony Su
Cool.
I'll block some significant time to look at what you built over the next 3
days.
Tony
Later!
Regards,
Mike
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20130920/17bdf1a9/attachment.pgp>
Serge Hallyn
2013-09-20 14:57:12 UTC
Permalink
Hey Michael,

tried this out on a saucy vm, and it looked good until it died with
receiving incremental file list
fedora-release-19-2.noarch.rpm
sent 47 bytes received 33329 bytes 9536.00 bytes/sec
total size is 32472 speedup is 0.97
warning: fedora-release-19-2.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID fb4b18e6: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:fedora-release-19-2 ################################# [100%]
Loaded plugins: fastestmirror, langpacks
Error: Cannot retrieve metalink for repository: fedora/19/x86_64. Please verify its path and try again
mount: mount point proc does not exist
chroot: failed to run command ?yum?: No such file or directory
Build of Installation RTE failed. Temp directory
not removed so it can be investigated.
Fedora Run Time Environment setup failed
Failed to download 'fedora base'
failed to install fedora
lxc-create: container creation template for f1 failed
lxc-create: Error creating container f1
Looks like unpacking didn't go right? <shrug>

-serge
Tony Su
2013-09-20 17:49:49 UTC
Permalink
After perusing the entire script now,
It seems to me that Michael's major contribution is all <after> the
parts described in my previous post.

In fact, it looks like Michael grafted his good work on to a Fedora
setup script.
If I'm right, then IMO substantial parts of what I posted about
earlier may be removed.
There are some good ideas in the later code which I think may be
re-organized slightly to allow for modularization to possibly
incorporate code that would run on other distros.

It's a big read and analysis...Will post again when I've tested some
modifications and will likely include some recommendations for
restructuring slightly.

In fact, I suspect that the error Serge just posted has to do with
this "pre-Container" code and isn't really a result of Michael's main
body of code so when that is removed to allow Michael's main script to
run, it might be successful.

Tony
Post by Serge Hallyn
Hey Michael,
tried this out on a saucy vm, and it looked good until it died with
receiving incremental file list
fedora-release-19-2.noarch.rpm
sent 47 bytes received 33329 bytes 9536.00 bytes/sec
total size is 32472 speedup is 0.97
warning: fedora-release-19-2.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID fb4b18e6: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:fedora-release-19-2 ################################# [100%]
Loaded plugins: fastestmirror, langpacks
Error: Cannot retrieve metalink for repository: fedora/19/x86_64. Please verify its path and try again
mount: mount point proc does not exist
chroot: failed to run command ?yum?: No such file or directory
Build of Installation RTE failed. Temp directory
not removed so it can be investigated.
Fedora Run Time Environment setup failed
Failed to download 'fedora base'
failed to install fedora
lxc-create: container creation template for f1 failed
lxc-create: Error creating container f1
Looks like unpacking didn't go right? <shrug>
-serge
------------------------------------------------------------------------------
LIMITED TIME SALE - Full Year of Microsoft Training For Just $49.99!
1,500+ hours of tutorials including VisualStudio 2012, Windows 8, SharePoint
2013, SQL 2012, MVC 4, more. BEST VALUE: New Multi-Library Power Pack includes
Mobile, Cloud, Java, and UX Design. Lowest price ever! Ends 9/20/13.
http://pubads.g.doubleclick.net/gampad/clk?id=58041151&iu=/4140/ostg.clktrk
_______________________________________________
Lxc-devel mailing list
Lxc-devel at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lxc-devel
Michael H. Warfield
2013-09-20 21:36:24 UTC
Permalink
Hey Serge,
Post by Serge Hallyn
Hey Michael,
tried this out on a saucy vm, and it looked good until it died with
receiving incremental file list
fedora-release-19-2.noarch.rpm
sent 47 bytes received 33329 bytes 9536.00 bytes/sec
total size is 32472 speedup is 0.97
warning: fedora-release-19-2.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID fb4b18e6: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:fedora-release-19-2 ################################# [100%]
Loaded plugins: fastestmirror, langpacks
Error: Cannot retrieve metalink for repository: fedora/19/x86_64. Please verify its path and try again
mount: mount point proc does not exist
chroot: failed to run command ?yum?: No such file or directory
Build of Installation RTE failed. Temp directory
not removed so it can be investigated.
Fedora Run Time Environment setup failed
Failed to download 'fedora base'
failed to install fedora
lxc-create: container creation template for f1 failed
lxc-create: Error creating container f1
Looks like unpacking didn't go right? <shrug>
That looks like the first one I sent out. I caught a couple of typos
immediately after that and recent it with an "opps". I'll double check
it though.
Post by Serge Hallyn
-serge
Regards,
Mike
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20130920/f5d6b2ec/attachment.pgp>
Tony Su
2013-09-04 18:14:33 UTC
Permalink
---------- Forwarded message ----------
From: Tony Su <tonysu at su-networking.com>
Date: Wed, Sep 4, 2013 at 10:23 AM
Subject: Re: [Lxc-users] Working LXC templates?
To: mhw at wittsend.com
Cc: Frederic Crozat <fcrozat at suse.com>, lxc-users
<lxc-users at lists.sourceforge.net>, lxc-devel at lists.sourceforge.net


Thx for the thoughts, Michael...

Would like to know specifically what you believe is incompatible
running 0.8.0 with systemd.

I have not noticed any issues running openSUSE 12.2 (systemd v44?) and
openSUSE 12.3 (systemd v195) on an openSUSE Host.

I've been able to run systemd commands (typically systemctl and
journalctl) from within the Containers without an issue most of the
time without issue and in the rare cases something didn't work I've
typically guessed it was just a potential security issue.

I'm guessing that running systemd both fundamentally and using its
various associated commands should not be an issue by simply applying
namespaces (personally have been speculating on the use of simple
random strings instead of user-readable text strings for security
reasons).

Based on the various templates I've tried, this issue is likely
related to how complete and self-contained the container template is,
the less it bind mounts or otherwise re-uses the Host system beyond
/proc I consider the template more agnostic. Binding Host resources is
what causes agnostic problems despite its various benefits.

The reason why a specific repo URL wouldn't be useful is because
interestingly Fedora's installation only looks up a Host which based
on the install's world-wide location and type of proposed installation
returns a recommended URL pointing to some repo mirror. This means
that there is no point testing against any of the mirror URLs, they
should all operate the same way. The URL I provided should be the only
relevant and critical URL the install template needs to use correctly.

As for why the authentication (GPG?) check isn't working, I haven't
dived into exactly what package is being used... But IMO it would make
sense in this pre-install environment, the Host's function is being
used and openSUSE does require User verification (unless the "silent"
switch is invoked which now that I'm thinking about it could be the
answer to my specific situation but may not address other distros).
Hmmmm... so, maybe I have something to experiment with...

Thx,
Tony
Post by Michael H. Warfield
This issue really belongs on -devel since it's a template development
issue that really impacts all the template writers.
Thx all for the replies,
- lxc-version returns 0.8.0. Looking around, there might be a more current
unstable, but AFAIK it's the most recently released stable.
This is not going to work. 0.8.0 will not support systemd in a
container, which all recent supported versions of Fedora are going to
require. 0.8.0 may be the more recently released stable "FROM OpenSuse"
but it is not the more recently release stable from LXC. The most
recently released stable from LXC is 0.9.0 and even that doesn't have
some of the necessary patches to the Fedora Template. Your best bet
would be to built from the stage branch in git.
You may need to wait until we release 1.0.0 and I'll take some of your
thoughts into consideration for the Fedora template but I have no way to
test them on OpenSuse at this time. Once we release 1.0.0, you'll still
have the problem with what OpenSuse releases as their stable. We have
no control over what they decide and do.
- I'd have to re-run to get the SSL error again but I think I've described
its error accurately, no further explanation except that the identity of
the remote server cannot be authenticated. This would lead me to guess that
the server is not registered properly with a public CA (eg using a CA root
that isn't in a bootstrap Fedora) so guessing that perhaps an option should
be offered that allows over-riding authentication? SSL encryption of course
should still be implemented for security.
Well, I can give you an argument if the error was described accurately
enough. I didn't see any site names I could test to see what the root
CA is. Without that, I can't tell you why you're seeing that error. I
understand fully what the error is (having my own private CA for private
activities) but I can't determine the origin without knowing the source.
<WAG - Wild Ass Guess>
I suspect that their certs are properly signed by a CA particular to the
Fedora Project and properly contained in the Fedora rpm root store, so
it may really be yet another cross distribution issue that depends on
the distribution peculiar packages and configurations.
Since there's probably no need (I see no need) for the Fedora
Repositories to be "registered properly with a public CA" (and pay the
extra expense), I would say the term "properly" is missued in this case.
The rpms are all signed by the Fedora Project and their gpg key so,
integrity shouldn't be an example unless someone intercepts the original
rootfs build and provides a trojaned package with the gpg keys.
</WAG>
Since this is an inter-distribution issue, I'm not sure what the proper
solution would be (assuming my WAG is true) or what LXC can do to
address it. I also don't know why Ubuntu / Debian is not experiencing
this problem either. But, without some example names of specific sites
exhibiting this problem (I don't run OpenSuse) I have no way to
investigate further.
Yeah, we could probably add an option to the template to ignore the SSL
check or to use and alternate rootca store (if we can avoid the
catch-22) but it may be better to investigate a more generic,
distribution agnostic, solution to these types of problems.
I do think it is an issue with the whole "distribution agnostic
template" problem that may require some help from the distros or some
innovative ideas of how we can bootstrap distros using distro agnostic
tools (like stone knives and bear skins style install of the rootfs
using nothing more than tar, gzip, gpg, and curl or wget).
- The lxc-create issue is definitely there. At first, I encountered it
using the openSUSE YAST LXC container applet. but then when I invoked
lxc-create from a console, but the "help" verifies it supports few options
and not these. But, as I described if the template requires parameters,
it's also possible to simply provide them in the script instead of at
runtime as command switches (but might not be apparent at first to someone
reading your script).
Tony
Regards,
Mike
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
Michael H. Warfield
2013-09-13 14:53:48 UTC
Permalink
Hey Tony!

I've kept meaning to revisit this and answer your question...
Post by Tony Su
Thx for the thoughts, Michael...
Would like to know specifically what you believe is incompatible
running 0.8.0 with systemd.
Ok... Here are the things we know about systemd...

1) There were radical changes in the behavior of systemd at various
times wrt (with regards to) the use of cgroups, devtmpfs, udev, and file
system mount options. The behaviors amounted to "magic cookies". If
you knew the magic cookie you might be able to get it to work, or you
might not. Different things changed with different versions creating a
morass of incompatibility and unpredictability from systemd depending on
a matrix of versions. We thought we had things fixed in 0.8.0 and then
they went and busted some more stuff we had to fix in 0.9.0 and the beat
goes on...

2) WRT cgroups, I think that was all resolved in the 0.8.0 release
framework by supporting and adapting to a variety of hierarchical mount
points for cgroups. That case is closed.

3) WRT devtmpfs, this was the NASTY one. At one point (on particular
systemd rev), the changed systemd to mount devtmpfs on /dev effectively
bypassing our mknod control over the file system and breaking the use of
pty pipes mounted to ttys and consoles. This caused all sorts of random
acts of terrorism like crashing X in the host system, hijacking the host
console device, and rebooting the host by issuing init 6 in the
container. This was misery incarnate. After some (cough) heated
discussions with some systemd proponents, we learned that we needed to
mount a ramfs device on /dev (magic cookie) in the container to prevent
systemd from mounting devtmpfs there and that we then needed to
autopopulate that /dev/ directory. There was also something to do with
environment variables in there as well (another magic cookie). This all
gave birth to the "lxc.autodev" option in the configuration files (and
some code changes by Serge and myself to the lxc-start C files) to tell
lxc to perform those operations (mount ramfs on /dev and populate).
That might be in 0.8.0, I'm not sure. I think it is. I'm pretty sure
it is.

4) WRT the device mount options. At a different, later, systemd rev,
they change one of the mount point options for the host file systems (I
think it was some shared mount propagation (MS_SHARED, MS_SLAVE???)
causing a failure in the pivot_root when attempting to start a container
when you have systemd running in the host (yet another magic cookie to
get the mount operations right). That one, I'm pretty much certain is
NOT in 0.8.0 and you have to be at least running 0.9.0.

5) Various setup issues (such as lxc.autodev) for systemd based
containers were not being automatically performed in the lxc-fedora
template scripts and required manual intervention after initial
container creation. That's where my coding has come in and much of that
is NOT even in 0.9.0 but you have to go to the stage branch of git for
those.

6) Systemd has had a history of changes in functionality and side
effects for a long time. I was never able to get F15 or F16 to reliably
run as containers due to incompatibility with the versions of systemd
included. Both involved changes in the way systemd behaved wrt udev. I
was almost able to get F15 to work but it was a butt ugly hack that had
warts all over it. It did run, for some value of "run" but I wouldn't
use it or trust it. F16 was a totally dead-on-arrival non-starter due
to a horribly incompatible udev handler in systemd that was in F16.
Never did figure out how they managed to break things THAT BADLY in that
version of systemd. Neither of those are supported any longer so, quite
frankly, I couldn't give a damn any more.

So, all of this has led Serge to list me on the roster for the
LinuxPlumbers conference as the LXC "systemd expert". I'll get even
with him later next week for that one...
Post by Tony Su
I have not noticed any issues running openSUSE 12.2 (systemd v44?) and
openSUSE 12.3 (systemd v195) on an openSUSE Host.
I've been able to run systemd commands (typically systemctl and
journalctl) from within the Containers without an issue most of the
time without issue and in the rare cases something didn't work I've
typically guessed it was just a potential security issue.
I'm guessing that running systemd both fundamentally and using its
various associated commands should not be an issue by simply applying
namespaces (personally have been speculating on the use of simple
random strings instead of user-readable text strings for security
reasons).
Devices namespaces should solve many of the problems systemd wantonly
introduced with devtmpfs. But that will then introduce yet another
change we'll have to adapt too. ITMT, namespaces in general should not
be a problem.
Post by Tony Su
Based on the various templates I've tried, this issue is likely
related to how complete and self-contained the container template is,
the less it bind mounts or otherwise re-uses the Host system
beyond /proc I consider the template more agnostic. Binding Host
resources is what causes agnostic problems despite its various
benefits.
Not really true. Bind mounting actually provides more insulation and
control. There are far more issues than just bind mounting. Bind
mounts must be managed properly and are the ideal solution for the
appropriate shared resources but should not be used for things like
shared memory, dev, run, etc, etc...
Post by Tony Su
The reason why a specific repo URL wouldn't be useful is because
interestingly Fedora's installation only looks up a Host which based
on the install's world-wide location and type of proposed installation
returns a recommended URL pointing to some repo mirror. This means
that there is no point testing against any of the mirror URLs, they
should all operate the same way. The URL I provided should be the only
relevant and critical URL the install template needs to use correctly.
Actually, my proposed template I posted earlier specifically uses the
mirrors.kernel.org mirror specifically because not all of the mirrors
are identical. Some don't carry certain non-package files (LiveOS
images or isos) and not all support rsync. So, that one particular
mirror is a good reference point for certain, very specific, operations
(most especially rsync access to the LiveOS file systems).
Post by Tony Su
As for why the authentication (GPG?) check isn't working, I haven't
dived into exactly what package is being used... But IMO it would make
sense in this pre-install environment, the Host's function is being
used and openSUSE does require User verification (unless the "silent"
switch is invoked which now that I'm thinking about it could be the
answer to my specific situation but may not address other distros).
Hmmmm... so, maybe I have something to experiment with...
Thx,
Tony
Regards, Mike
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | mhw at WittsEnd.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20130913/13282f52/attachment.pgp>
Serge Hallyn
2013-09-16 18:25:21 UTC
Permalink
Post by Michael H. Warfield
So, all of this has led Serge to list me on the roster for the
LinuxPlumbers conference as the LXC "systemd expert". I'll get even
with him later next week for that one...
Lol! Buy you a beer thu night? :)

-serge
Loading...