Let's enable AppArmor by default (why not?)

[intrigeri]

Hi!

tl;dr: I hereby propose we enable AppArmor by default in testing/sid,
and decide one year later if we want to keep it this way in the
Buster release.

My goals when initiating this discussion are:

- Get a rough idea of what amount of effort the Debian project is
happy (and able) to invest into such proactive security measures.

- Learn about any relevant social & technical concerns I am not
aware of.

I don't expect we'll make a decision based on this very proposal:
I expect the proposal will need to be refined, or abandoned, to take
into account what we will learn during this preliminary discussion.

Why do we need AppArmor?
========================

AppArmor is a Mandatory Access Control framework implemented as
a Linux Security Module (LSM), user space utilities, and a quite
simple language to define policy.

AppArmor confines programs according to a set of rules that specify
what operations a given program can access, e.g. it can prevent
exploited server software from accessing data it does not own and
executing arbitrary code. This proactive approach helps protect the
system against some known and unknown vulnerabilities.

Various actors are actively exploiting software. Random users are
victimized every day, and specific populations are specifically
targeted, e.g. government opponents, human rights defenders, system
administrators, software developers and distributors, as revealed by
the Snowden leaks.

Every month we learn about many new attack vectors made possible by
programming errors. We fix them after the fact, which is great but
a bit too late: users may already have been exploited. Most operating
systems have adopted proactive approaches to mitigate the impact of
such problems.

In Debian, great efforts are in progress: hardening binaries makes it
harder to write successful exploits, and making our packages build
reproducibly will make it harder to introduce vulnerabilities at the
binary level.

Still, Debian is far from being best in class on this front: we have
no widespread mechanism for sandboxing desktop applications. We can
surely do better. The great news is that there is one low-hanging
fruit waiting to be picked, and it's what this proposal is about :)

A proposal
==========

1. Enable AppArmor by default in testing/sid as soon as feasible in
the Buster cycle.

I can think of several possible ways to do it but for now I'd
rather focus on the "do we want to do it at all" conversation.

If curious some options are listed on the wiki:
https://wiki.debian.org/AppArmor/Progress#Enabling_AppArmor_by_default.3F
Feel free to discuss them on <pkg-apparmor-***@lists.alioth.debian.org>.

And anyway, you can already opt-in for AppArmor on your system today:
https://wiki.debian.org/AppArmor/HowToUse :)


2. During a year, watch out for AppArmor related issues and address
them in a prompt manner.

Note that the best way to address them quickly enough is sometimes
to simply disable the problematic AppArmor profile: it's cheap,
doesn't require advanced AppArmor skills, and IMO a smaller
AppArmor policy enabled by default is more useful than a broader
but less robust one that only a couple thousand users benefit from.

3. A year after AppArmor was enabled by default: evaluate how it went
and decide if Buster should be shipped with AppArmor enabled by
default or not.

I commit to do an analysis using BTS data to help make this
decision. If we need formal success criteria and a clearly defined
team who'll make the call, I'm happy to think about it. But here
again I'd rather focus on the general idea than on implementation
details at this point.

Questions and Answers
=====================

Table of contents:

- What's the benefit, exactly?
- What do other distributions do?
- What's the history of AppArmor in Debian?
- How popular is AppArmor in Debian?
- What's the cost for Debian users?
- What's the cost for package maintainers?
- Is the Debian AppArmor team strong enough to support this change?
- Why AppArmor and not SELinux?
- Why AppArmor and not sandboxing based on XYZ?
- Will this prevent users from using another Linux Security Module?
- What does upstream look like?
- How much will we depend on Canonical's business priorities?
- No thanks: I've tried AppArmor and it broke stuff too often
- Doesn't AppArmor need out-of-tree kernel patches?
- How can I help?
- Credits

What's the benefit, exactly?
----------------------------

Before we even bother looking at the cost of enabling AppArmor by
default, let's look closer at the expected benefit. In other words:
what kind of attacks does AppArmor really mitigate or prevent in the
real world?

tl;dr: big benefit for server software, and for desktop applications
limited to less sophisticated, non-targeted attacks (but it'll get
better).

AppArmor is well suited to protect against remote exploitation of
security issues in server software and non-GUI programs often run with
elevated privileges (think of dhclient, ping, tcpdump). I'm sure one
could identify a few serious issues that would have been mitigated or
prevented by our current AppArmor policy, by looking at a list of
DSA/CVE. Also, one gets interesting security properties when software
is tuned for AppArmor: e.g. a given libvirt/QEMU virtual machine can
only access its assigned storage volumes, and not other VMs'; this is
useful against QEMU security issues that allow guests to escape the
virtualization layer.

On the desktop, to be honest things look pretty bad *currently*:
AppArmor is not enough, and we need new concepts and technologies to
fix serious limitations on the desktop sandboxing front.
Thankfully this is being actively worked on and the future of desktop
sandboxing on GNU/Linux looks bright and shiny! Some of the future
options rely on AppArmor, some others don't. See the "Why AppArmor and
not sandboxing based on XYZ?" section below.

Still, confining desktop apps with AppArmor has benefit against
scripted or non-targeted attacks: it will mitigate some attacks and
others will get through. So while it's probably not worth starting to
write lots of new policy for GUI applications now, I think that
leveraging the existing one will already serve our users.

What do other distributions do?
-------------------------------

AppArmor has been enabled by default in several other GNU/Linux
distributions and Debian derivatives for a while:

* in openSUSE + SLES since 2006
* in Ubuntu since 2008, with a growing policy:
https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/AppArmorProfiles
* in Tails, since 2014 for a few important services (CUPS, Tor) and
a few desktop applications (e.g. Totem, Evince, Pidgin, Tor
Browser, Thunderbird)
* in a few other Debian derivatives (Whonix, Subgraph OS) for at
least a couple years.

What's the history of AppArmor in Debian?
-----------------------------------------

AppArmor has been available (opt-in) in Debian since 2011. In 2014
a Debian AppArmor packaging team was created, that has been taking
care of the AppArmor packages and policy since then.

In the last 3 years the AppArmor policy shipped in Debian was extended
substantially and its coverage is now on par with Ubuntu's. It's still
rather small due to the strategy we chose: we wanted to avoid
traumatizing early adopters and to avoid creating a culture of
"AppArmor always breaks stuff, let's get used to disabling it".
So like Ubuntu and openSUSE, we're shipping a rather small and mature
AppArmor policy. I believe this strategy has been successful so far,
and even some non-trivial pieces of software like Thunderbird now ship
an AppArmor policy; but of course it has one drawback: most software,
including web browsers, is not confined with AppArmor whatsoever.
Surely with more people contributing to our AppArmor policy we could
have it cover other important pieces of software; time will tell.

A number of maintainers accepted shipping AppArmor policy in their own
package. If you're one of them, please consider providing feedback
about how it went for you.

How popular is AppArmor in Debian?
----------------------------------

tl;dr: AppArmor has steadily become more and more popular in Debian in
the last few years. I think the user base has reached a critical mass
that proves it works OK.

Here's what popcon says ("Vote" count) for the apparmor binary
package, that's needed to use AppArmor:

* 2015-01: ~400
* 2016-01: ~700 (+75% in a year)
* 2017-01: ~1300 (+85% in a year)
* today: 1870 (+44% in 7 months)

But we have no way to tell whether a user who has AppArmor packages
installed actually enabled the AppArmor LSM, so the data for
apparmor-profiles-extra might be more useful here: I expect that only
users who really want to use AppArmor with an extended policy would
bother installing it. This one has 435 registered installations
("Vote" has always been 0 for some reason that I did not investigate);
it was introduced in October 2014, and since then its popcon stats
have been steadily increasing.

What's the cost for Debian users?
---------------------------------

AppArmor unavoidably breaks functionality from time to time: e.g.
new versions of software we package (or of their dependencies)
regularly start needing access to new file locations.

And then users see broken applications from time to time, after
upgrading their testing/sid system. This is to be taken seriously, but
not a big concern IMO:

- Apparently Ubuntu users have been coping with AppArmor enforced
by default for 9 years. I see no reason why Debian users would not.

- I've counted 14 bugs bugs reported in the Debian BTS during the
Stretch development cycle against our supported AppArmor policy.
Among those, 11 were closed (106 days after being reported on
average); all the important ones were closed within 2 months;
larger delays were due to users developing fixes and/or upstream
taking some time to review merge requests. About the 3 bugs still
open: one is waiting for input from other package maintainers since
2 years, another one had a patch waiting to be applied, and the
last one needs to be fixed in libvirt upstream.

- Serious breakage is less likely to happen once AppArmor is enabled
by default, as there are greater chances that the maintainer would
have noticed it before uploading.

- Workarounds are regularly suggested to the bug reporter on the BTS,
and in many cases the bug reporter documents in the bug report the
workaround they have *already* applied.

Implementing a suggested workarounds requires being able to edit
a text file and running one command as root, which should be doable
by the vast majority of testing/sid users.

What's the cost for package maintainers?
----------------------------------------

For most of them: none at all. As said earlier, our AppArmor policy
does not cover that much software yet.

But maintainers of software confined by AppArmor will have to deal
with a new kind of bug reports, whose number is likely to grow
significantly once AppArmor is enabled by default. This means they
have to:

1. identify if a bug report can possibly be related to AppArmor;

2. either learn how to debug AppArmor issues themselves, or ask the
pkg-apparmor team for help (thanks to usertags:
https://wiki.debian.org/AppArmor/Reportbug#Usertags).

I expect that initially pkg-apparmor will need to provide help in many
cases, but over time the affected maintainers will slowly learn just
enough about AppArmor to handle at least the simplest cases
themselves, just like it happened in Ubuntu years ago.

Is the Debian AppArmor team strong enough to support this change?
-----------------------------------------------------------------

This is a valid concern, as I have been doing the greatest part of the
work on this team.

So far I've found my AppArmor-related workload totally sustainable:
it took me just a few hours here and there, and I would be doing this
work for Tails anyway, so better do it directly in Debian. Still,
primarily relying on one single person is concerning.

Thankfully, a number of other people have been contributing in various
ways. A few Debian users and contributors got used to reporting bugs
and contribute improvements to our AppArmor policy upstream.
Another team member uploaded src:apparmor once. Ulrike Uhlig wrote
documentation about AppArmor for Debian users and
contributors during an Outreachy project whose outcome was posted to
debian-devel-announce in March, 2015.

Also, just like any such distro-wide change, I expect the amount of
work required to support the broader project:

- will be large initially; I'm confident that the current state of
our team is good enough to support the project during the first
stage of the transition;

- will only decrease over time, as Debian people get used to it and
learn the small bits they need to know about the new technology,
and eventually the cases when our AppArmor team has to give a hand
will become rare;

- will be done by AppArmor people from other distributions as well:
a few of them actively participate on the pkg-apparmor mailing list
and help on issues reported in the Debian BTS.

So I think it's totally reasonable to give it a try.

Why AppArmor and not SELinux?
-----------------------------

SELinux is another LSM that tackles similar problems.

Disclaimer: I've picked AppArmor years ago and didn't look much at
SELinux recently, so some of what follows may be totally wrong or
outdated. Sorry! Debian SELinux people, if you don't mind please help
me get the basic facts right :)

Pros:

* Allows mediating more kernel objects / interfaces than AppArmor, so
policy can be made stricter and safer given sufficient expertise
and available time for maintenance.

* Enabled by default in RHEL so in theory a great number of sysadmins
are at ease with it (see below why reality may not match this).

* A quick look at popcon suggests that SELinux might be more popular
in Debian than AppArmor, but I'm not sure I am interpreting the
numbers right (and I suspect that just like AppArmor, the popcon
won't tell us if users who have installed the relevant support
packages actually run their system with the corresponding LSM
enabled & enforced).

Cons:

* Writing, maintaining, auditing and debugging SELinux policy
requires grasping a complex conceptual model; I am told this is not
as easy as doing the same with AppArmor.

* As far as I could understand when chatting with sysadmins of Red
Hat systems, this has resulted in a culture where many users got
used to disable SELinux entirely on their systems, instead of
trying to fix the buggy policy. I've seen the opposite happen with
AppArmor, which is good: for example, pretty often bug reporters to
the Debian BTS document themselves how they could workaround the
problem locally *without* turning AppArmor off. Looking at open
bugs in the BTS against src:refpolicy, this seems to happen very
rarely for SELinux, so I wonder if it would be realistic to ship
Debian with SELinux enforced by default and have our community
support it.

* https://wiki.debian.org/SELinux/Issues says "Graphical/Desktop
installs of Debian are not heavily tested with selinux, so you
might run into quite some issues".

* I'm not aware of any Debian derivative shipping with SELinux
enabled by default. If that's correct, then it means that we would
have to deal with quite some policy compatibility issues ourselves.

To me, the complexity of SELinux is a deal breaker: it seems that we
would need to get lots more expertise and energy to enforce SELinux by
default than doing the same with AppArmor.

Now, if for some reason the project prefers to ship with SELinux
enforced instead of AppArmor, fine by me: I would strongly prefer this
option to nothing at all.

Why AppArmor and not sandboxing based on XYZ?
---------------------------------------------

(Replace "XYZ" with Flatpak, Ubuntu's Snappy, Subgraph OS' oz,
Firejail, Subuser, or you preferred other promising option.)

We need both!

AppArmor covers server software well, and on the desktop it currently
protects against not-too-sophisticated, non-targeted attacks.

In a nutshell, the GNU/Linux desktop really wasn't designed for
applications to be siloed. To fix that we need new concepts and
technologies, such as Wayland, portals, and fine-grained D-Bus
mediation. Next generation desktop sandboxing technologies will fix
this and improve UX at the same time, and it will be amazing. But they
are not ready for prime-time yet. A Debian user cannot benefit from
them *today* much; this might change in time for Buster, but really
we're comparing a well-established, polished solution with a bunch of
other ones whose integration with Debian is being brainstormed.

Anyway, this is not an either/or situation: even though there are
currently compatibility issues, one can perfectly well develop/adapt
such tools in a way that makes them work fine with AppArmor enabled.

Let's enable AppArmor so we cover at least the server use case and the
low-hanging fruits of the desktop one, and figure out later where we
should put our efforts for securing the desktop, once the dust has
settled and next generation solutions have matured and been integrated
in Debian.

Will this prevent users from using another Linux Security Module?
-----------------------------------------------------------------

Some "minor" Linux Security Modules, such as Yama, live perfectly well
with others.

But currently it is not possible to enable several of the major
security modules. There's been (slow) work in progress to fix this for
a while, but it has picked up recently and there is a lot of interest
to have, say, AppArmor and SELinux stackable:

https://lwn.net/Articles/719731/

Now, every user will still be able to opt-out from AppArmor and
instead enable their preferred LSM.

What does upstream look like?
-----------------------------

The upstream project is almost 20 years old, very mature and
cooperative with Debian. E.g. the upstream release schedule has been
adjusted both for Jessie and Stretch to accommodate Debian's
schedule nicely.

Regarding who does the work:

- Canonical employees do most of the kernel work. They also maintain
the library and other C code, e.g. the policy parser.

- The Python utilities are primarily maintained by openSUSE's
Christian Boltz.

- Maintaining AppArmor policy is a cross-distro team effort, mostly
done by Debian, Ubuntu and openSUSE people.

How much will we depend on Canonical's business priorities?
-----------------------------------------------------------

Given Canonical employees do the greatest part of the work upstream:
indeed, we will. I see two main concerns about this:

Long-term reliability: this funding could run out some day.
I personally am not overly concerned, as Canonical has been investing
a lot into products (Snappy, LXC/LXD) that strongly depend on AppArmor
in the last few years.

Power imbalance: the company that does so much of the work has great
power over the priorities of the upstream project. This is the case
for a large amount of critical software we ship, so like it or not, it
is something we are living with already. AppArmor developers employed
by Canonical have shown great willingness in cooperating with Debian
in the last few years, so I'm confident that our contributions will be
welcome for the foreseeable future, whenever we need to adapt the
software to our needs. But of course management/business decisions can
change this at any time.

No thanks: I've tried AppArmor and it broke stuff too often
-----------------------------------------------------------

Sorry about that. I think this has had three main causes so far, that
all share one single root cause i.e. "AppArmor is not enabled by
default" (chicken'n'egg!):

1. Most package maintainers don't test packages with AppArmor before
uploading, so users notice breakage that could easily be avoided.

2. The huge majority of our users are not affected by breakage caused
by AppArmor, so we handle such breakage in a way that saves
maintainers' time: e.g. in many cases I've personally preferred to
wait for my fixes to AppArmor profiles to be approved and merged
upstream before applying them in Debian.

Once AppArmor is enabled by default, as far as I'm concerned
I don't plan to wait for upstream review before fixing regressions
in testing/sid.

3. The huge majority of our users are not affected by breakage caused
by AppArmor, so such breakage was kinda acceptable and thus we
*sometimes* preferred to give a specific AppArmor profile more
exposure to testers, even if it had a couple known issues, in order
to identify problems and help stabilize it (e.g. Tor, libvirt).

I think we will need to be more conservative once AppArmor is
enabled by default, i.e. profiles that break functionality too much
or too often should not be enabled by default.

Doesn't AppArmor need out-of-tree kernel patches?
-------------------------------------------------

Yes and no.

Historically, the mainline Linux kernel has supported a rather small
subset of the AppArmor mediation made possible by the out-of-tree
kernel patch. This made the value of enabling AppArmor smaller than it
could be (e.g. LXC is not well confined in Debian: #750106), and
smaller than it is in distros that apply the out-of-tree kernel patch
(such as Ubuntu).

Still, even with the set of features available in mainline Linux
*today*, IMO enabling AppArmor already has a good cost/benefit ratio
for Debian and our users. I'm not proposing we apply out-of-tree
AppArmor kernel patches.

Thankfully, the AppArmor kernel developers recently changed how they
proceed: new features are now added to Linux mainline before they
reach Ubuntu, so I'm confident that this situation will get better and
better in the future, and Buster's kernel will support tons of new
AppArmor mediation types compared to Stretch.

How can I help?
---------------

* Enable AppArmor on your Debian systems:
https://wiki.debian.org/AppArmor/HowToUse

* If you maintain a package for which we ship AppArmor policy in
Debian: test it with AppArmor enabled before uploading.

Ask for help if needed:
https://wiki.debian.org/AppArmor/Reportbug#Usertags

* Join the team:
https://wiki.debian.org/AppArmor/Contribute

* Talk to us:
<pkg-apparmor-***@lists.alioth.debian.org>

Credits
-------

A huge thank you to the people who reviewed this text, provided
valuable input that I took into account & integrated, and helped me
change my mind here and there: Christian Boltz, gregoa, Guido Günther,
Jamie Strandboge, John Johansen, Sebastien Delafond, Simon McVittie
and Solveig! Sorry to those I forgot. I shamelessly stole bits of text
they wrote. I absolutely do *not* imply they endorse this proposal.

Thanks a lot to my pkg-apparmor team-mates, to Kees Cook who
introduced AppArmor in Debian in the first place, and to all AppArmor
contributors upstream and in other distros :)

Cheers,

[Niels Thykier]

intrigeri:

Hi,

Overall, this sounds like an interesting proposal and personally, I
agree that I think the Debian Linux ports would be better off with an
LSM enabled by default.

Post by intrigeri
What's the cost for Debian users?
---------------------------------
AppArmor unavoidably breaks functionality from time to time: e.g.
new versions of software we package (or of their dependencies)
regularly start needing access to new file locations.

s/AppArmor/Any LSM/

Can we integrate these LSM policies into our testing frameworks (e.g.
autopkgtests), so we can start having automated tests of even basic
functionality. Or will that happen "out of the box" if we enable it by
default (and, possibly, enable it on our test hosts)?

Thanks,
~Niels

[Simon McVittie]

Post by Niels Thykier
Can we integrate these LSM policies into our testing frameworks (e.g.
autopkgtests), so we can start having automated tests of even basic
functionality. Or will that happen "out of the box" if we enable it by
default (and, possibly, enable it on our test hosts)?

In practice probably only if our test hosts change from being lxc
containers to being full virtual machines, which would be good for
other aspects of test coverage too (flatpak/bubblewrap currently skip
many of their autopkgtests, and so does debootstrap).

AppArmor has some amount of support for being used inside a privileged
lxc container, but it's very new and I'm not sure it's going to be a
particularly accurate representation of what would happen on real
hardware. (One of my colleagues looked at lxc + apparmor recently and
was able to make it work, but not trivially.)

S

[Antonio Terceiro]

Post by Simon McVittie

Post by Niels Thykier
Can we integrate these LSM policies into our testing frameworks (e.g.
autopkgtests), so we can start having automated tests of even basic
functionality. Or will that happen "out of the box" if we enable it by
default (and, possibly, enable it on our test hosts)?

In practice probably only if our test hosts change from being lxc
containers to being full virtual machines, which would be good for
other aspects of test coverage too (flatpak/bubblewrap currently skip
many of their autopkgtests, and so does debootstrap).

FWIW, my last upload of debci has a prototype qemu backend. I want to
switch to that sooner rather than later.

[Christoph Biedl]

intrigeri wrote...

Post by intrigeri
tl;dr: I hereby propose we enable AppArmor by default in testing/sid,
and decide one year later if we want to keep it this way in the
Buster release.

I really appreciate your approach of trying this out while being
prepared this might turn out to be a bad idea. Or: Promoting an idea
without being pushy about it.

So while adding another security layer is certainly something to
consider, I'm as well interested in whether this is feasible for a
generic-purpose distribution like Debian. The worst thing that could
happen was people will have to do the counterpart of chmod 777. Then it
was a bad idea, but we (as in Debian) have substantiation for such a
claim.

Christoph

[Dr. Bas Wijnen]

Post by Christoph Biedl
intrigeri wrote...

Post by intrigeri
tl;dr: I hereby propose we enable AppArmor by default in testing/sid,
and decide one year later if we want to keep it this way in the
Buster release.

[...] while adding another security layer is certainly something to
consider, I'm as well interested in whether this is feasible for a
generic-purpose distribution like Debian.

Enabling it by default doesn't mean it can't be switched off, right? I think
it makes a lot of sense to enable something like this by default, and in fact I
can't think of a situation where you would not want it, but indeed users should
be able to set their system up without it if they so wish.

Post by Christoph Biedl
The worst thing that could happen was people will have to do the counterpart
of chmod 777. Then it was a bad idea, but we (as in Debian) have
substantiation for such a claim.

Yes, we should certainly avoid that; if it looks like that is happening, we
should abort the operation. But from the well written plan, it sounds to me
like this is unlikely to be the case.

So just to be clear: Yes, please enable AppArmor by default.

Thanks,
Bas

[Andrey Rahmatullin]

I can't think of a situation where you would not want it

The "I don't want yet another thing that can cause subtle breakages and
doesn't give me anything" situation (see disabling selinux after install
on RH systems).

[Wouter Verhelst]

Post by Andrey Rahmatullin

I can't think of a situation where you would not want it

The "I don't want yet another thing that can cause subtle breakages and
doesn't give me anything" situation (see disabling selinux after install
on RH systems).

It actually does give you anything, and debugging SELinux breakages is
fairly simple, with the fix usually being rather small.

For a while after RedHat started enabling SELinux by default on their
systems, a number of security researchers yelled "I've found an
exploitable bug in RedHat" for which the first step to exploit was
always "first, disable SELinux". I'm not saying that that's not a
problem, but it *does* show that using SELinux or AppArmor has benefits.

I think enabling an LSM by default is a good idea, and we should do it
if we can. The "subtle breakages" you mention are annoying for you, but
they can be a showstopper for an attacker, and that's a *good* thing.
Yes, obviously when we enable an LSM we should also make it easy for
users to understand that something is blocked by the LSM and explain to
them how they can unblock it if they want to. But just saying "it causes
issues, let's not" is the same as saying "permissions on the file system
causes issues, let's install to FAT32 everywhere", and that's just not a
good idea.

[intrigeri]

Post by Dr. Bas Wijnen
Enabling it by default doesn't mean it can't be switched off, right?

Yes, passing apparmor=0 on the kernel command line will turn it off.

Cheers,

[Holger Levsen]

Post by intrigeri
tl;dr: I hereby propose we enable AppArmor by default in testing/sid,
and decide one year later if we want to keep it this way in the
Buster release.

loving it, please go ahead! ;-)

[Moritz Mühlenhoff]

Post by intrigeri
Still, even with the set of features available in mainline Linux
*today*, IMO enabling AppArmor already has a good cost/benefit ratio
for Debian and our users. I'm not proposing we apply out-of-tree
AppArmor kernel patches.

I'd expect that a lot of the AppArmor profiles currently in use are
coming from Ubuntu or OpenSUSE. If one of those profiles relies
on features which are not upstreamed on the kernel end, how's
that handled?

Cheers,
Moritz

[intrigeri]

Hi,

Post by Moritz Mühlenhoff
I'd expect that a lot of the AppArmor profiles currently in use are
coming from Ubuntu or OpenSUSE.

Yes, historically (most of them are now maintained via a cross-distro
collaborative effort that a few Debian people participate in).

Post by Moritz Mühlenhoff
If one of those profiles relies on features which are not upstreamed
on the kernel end, how's that handled?

Rules that are not supported by the running kernel are silently
ignored, i.e. the operation is allowed.

For example, the profile for ping(8) specifies:

network inet raw,
network inet6 raw,

In Ubuntu, and in Debian once network socket mediation support lands
in Linux mainline, this means "no socket(2) based operation is allowed
except inet and inet6 raw sockets". In current Debian, no network
mediation by AppArmor is available so all socket(2) based operations
are allowed.

This way:

* the same profile can be shared between distros, regardless of
whether they apply not-upstreamed-yet AppArmor kernel patches;

* once new AppArmor features land in Linux mainline, we automatically
benefit from stronger confinement that's already implemented in our
AppArmor policy.

Cheers,

[Christian Seiler]

Post by intrigeri

Post by Moritz Mühlenhoff
If one of those profiles relies on features which are not upstreamed
on the kernel end, how's that handled?

Rules that are not supported by the running kernel are silently
ignored, i.e. the operation is allowed.

Is there at least a warning during the load of the profile? Consider a
local sysadmin that creates an own profile for a specific service they
run - and assume that AppArmor will confine it. But because the
kernel doesn't support a specific thing yet the confinement will be
incomplete. Which is probably better than not having AppArmor, and is
probably also OK for profiles shipped with the distribution and / or
upstream software - but not necessarily a good idea for something an
admin touches themselves.

Or, conversely, is there a possibility to add a flag to the AppArmor
profile to say "fail to load it if something is not understood"? In
that case all profiles shipped by Debian would not include that (for
interoperability reasons) but it could be documented that as a best
practice for admins they should use that flag so that they can be
sure that all protections they specified are actually affected.

Another thing to consider: if a profile is too restrictive, but the
part that is too restrictive isn't in the upstream kernel yet, then
things could break if you upgrade the kernel to a newer version from
e.g. backports later on. How would you deal with that kind of
breakage during the lifetime of a stable release?

Regards,
Christian

[Moritz Mühlenhoff]

Post by Christian Seiler
Another thing to consider: if a profile is too restrictive, but the
part that is too restrictive isn't in the upstream kernel yet, then
things could break if you upgrade the kernel to a newer version from
e.g. backports later on. How would you deal with that kind of
breakage during the lifetime of a stable release?

Agreed, that was pretty much my concern. Ideally the feature set
used would also be controlled by the apparmor userspace side.

Also, I'm wondering about the status of kernel support which is
currently not upstreamed: intrigeri mention that new features are
now added to Linux mainline. Was there ever an attempt to upstream
those existing patches (e.g. for network socket mediation); was it
NACKed by upstream for conceptual problems or was it simply never
attempted due to time/resource constraints?

Cheers,
Moritz

[intrigeri]

Hi,

[John, there's a question for you at the bottom, but you probably have
useful input about the first part of the discussion below too.]

Post by Moritz Mühlenhoff

Post by Christian Seiler
Another thing to consider: if a profile is too restrictive, but the
part that is too restrictive isn't in the upstream kernel yet, then
things could break if you upgrade the kernel to a newer version from
e.g. backports later on. How would you deal with that kind of
breakage during the lifetime of a stable release?

Agreed, that was pretty much my concern.

Thank you so much for highlighting problems I had missed! :)

A simple, but not entirely satisfying answer is:

1. Gather info about how real this problem has been in practice for
Ubuntu: they frequently update their kernel for various already
released distros with the latest AppArmor bits. I think they
occasionally have to update other packages accordingly to adjust
AppArmor policy. I don't know how often this happens. I'll ask them
to compile a list of such stable updates.

2. Evaluate for a year how it goes for Stretch + Linux from backports.

Desktop: I'm in a good place to provide data points, as Tails
generally ships this combination and we exercise the vast majority
of the desktop AppArmor stuff that's in Debian.

Server: sorry, I use the stable kernel except on bare-metal
virtualization hosts. But I think (1) will give us enough data on
this front.

3. Depending on what (1) and (2) tell us, decide whether "we might
have to update AppArmor policy from time to time in stable
point-releases or backports" is good enough… keeping in mind that
other distros are already dealing with the exact same problem, so

Post by Moritz Mühlenhoff
Ideally the feature set used would also be controlled by the
apparmor userspace side.

If we need to go this far: apparmor_parser has a --features-file
option that we could leverage to tie the feature set being used to
something else than the version of the running kernel, e.g.
with a file shipped in a new package built from src:linux with
appropriate versioned dependencies.

Post by Moritz Mühlenhoff
Also, I'm wondering about the status of kernel support which is
currently not upstreamed: intrigeri mention that new features are
now added to Linux mainline. Was there ever an attempt to upstream
those existing patches (e.g. for network socket mediation); was it
NACKed by upstream for conceptual problems or was it simply never
attempted due to time/resource constraints?

I'm not sure, so I'll let the main AppArmor kernel developer (John,
Cc'ed) answer this.

Cheers,

[John Johansen]

Post by intrigeri
Hi,
[John, there's a question for you at the bottom, but you probably have
useful input about the first part of the discussion below too.]

Post by Moritz Mühlenhoff

Post by Christian Seiler
Another thing to consider: if a profile is too restrictive, but the
part that is too restrictive isn't in the upstream kernel yet, then
things could break if you upgrade the kernel to a newer version from
e.g. backports later on. How would you deal with that kind of
breakage during the lifetime of a stable release?

Agreed, that was pretty much my concern.

Thank you so much for highlighting problems I had missed! :)
1. Gather info about how real this problem has been in practice for
Ubuntu: they frequently update their kernel for various already
released distros with the latest AppArmor bits. I think they
occasionally have to update other packages accordingly to adjust
AppArmor policy. I don't know how often this happens. I'll ask them
to compile a list of such stable updates.
2. Evaluate for a year how it goes for Stretch + Linux from backports.
Desktop: I'm in a good place to provide data points, as Tails
generally ships this combination and we exercise the vast majority
of the desktop AppArmor stuff that's in Debian.
Server: sorry, I use the stable kernel except on bare-metal
virtualization hosts. But I think (1) will give us enough data on
this front.
3. Depending on what (1) and (2) tell us, decide whether "we might
have to update AppArmor policy from time to time in stable
point-releases or backports" is good enough… keeping in mind that
other distros are already dealing with the exact same problem, so

There are several distinct issues when dealing with stable release
support. There are basically 3 different potentially moving components
to consider

kernel - changes may result in a change in the supported feature
set.

policy - a package may backport/drop in policy that was developed on
a different feature set.

user space - changes/updates may be required to support new features
in a kernel or policy.

The question specifically asks about, an updated kernel with a policy
that was developed under a different feature set, suddenly breaking
when a new kernel is run on an older system.

The kernel portion is designed to support multiple versions of
userspace policy abis, and while we have dropped some of the older
abis, it is done slowly. The current kernel code supports the abi from
2009. So newer kernels will support older policy and userspaces.

Similarly the userspace is designed to support multiple kernel
versions and abis, so a newer userspace (if it is SRUed for some
reason) can support older kernels.

This leaves us dealing with policy. As long as the policy has not been
changed, it is possible to force userspace to build policy for a
certain kernel feature set by specifying the feature file.

This can be done in the apparmor/subdomain.conf file

with this specified, policy should remain to be compiled as for the
older kernel, and the new kernel should support and enforce it under
that abi.

There is however a caveat that sometimes the kernel changes abi
behavior in ways that apparmor can not adapt to without policy
modification.

An example of this would be
commit 9f834ec18defc369d73ccf9e87a2790bfa05bf46
binfmt_elf: switch to new creds when switching to new mm

which resulted in a change of which cred the PROT_EXEC request was
made against. With the way the lsm hook is designed only the single
cred is available so it is not even possible for apparmor to emulate
the old behavior.

In this case older policy needed to be patched to support the newer
behavior if it was to be used on a newer kernel, and a flag was
added to the apparmor kernel features interface making it possible
for userspace to detect that the kernel contained the change.

Thankfully this type of change is has not been very common.

Post by intrigeri

Post by Moritz Mühlenhoff
Ideally the feature set used would also be controlled by the
apparmor userspace side.

If we need to go this far: apparmor_parser has a --features-file
option that we could leverage to tie the feature set being used to
something else than the version of the running kernel, e.g.
with a file shipped in a new package built from src:linux with
appropriate versioned dependencies.

the feature file can indeed be specified on the command line using
--feature-file, but from a support pov I think specifying it in the
config file

apparmor/subdomain.conf

would be better as then you don't have to mess with initscripts, unit
files, etc.

Post by intrigeri

Post by Moritz Mühlenhoff
Also, I'm wondering about the status of kernel support which is
currently not upstreamed: intrigeri mention that new features are
now added to Linux mainline. Was there ever an attempt to upstream
those existing patches (e.g. for network socket mediation); was it
NACKed by upstream for conceptual problems or was it simply never
attempted due to time/resource constraints?

I'm not sure, so I'll let the main AppArmor kernel developer (John,
Cc'ed) answer this.

So current status of kernel support that hasn't been upstreamed is

4.13 - has most everything. It has the virtualized apparmorfs and
namespacing interfaces, the core rework, stacking, etc. It is
missing some key features, specifically signals, mount and
network mediation

4.14 - isn't fully decided yet, but it should pickup everything except
maybe the extended unix socket mediation

As for why upstreaming has taken so long, there have been several
reasons.

Time constraints have been the major issue. Certain new feature
development was certainly prioritized over getting existing out of
tree features upstream. There is recognition that this was the wrong
approach and there is now an upstream first policy.

In addition some of the delay has also been just a matter of letting
code bake and shaking out bugs. Some of the code has had several
revisions before we were happy with it and thought it ready to be
upstreamed.

Eg. the extension to securityfs to allow apparmor to virtualize its
policy dir has seen several revisions. And even changed approach
before the final version which was upstreamed in 4.13. It just took
some time for the code to bake and for the correct approach to settle
out. Ubuntu was willing to carry the dev code to get certain features
early, at an increased cost in support.

The dbus code went through several revisions as well. While the dbus
code doesn't require a lot from the kernel, it did have some influence
on the kernel support interfaces.

Another reason many of these features were delayed in their
upstreaming, is the apparmor core was being rewritten during their
development, which unfortunately resulted in there not being a clean
separation between the different parts of development. So while some
features could have been upstreamed sooner they had to wait for the
core changes to bake, or be rewritten to work with the existing
upstream code (time and resources weren't available for the rewrite).

None of the current development has been NAKed upstream. At times in
the past certain approaches have been NAKed, and required reworking.
This was largely around how apparmor interacted with the system and
modifications to the LSM hooks. That is not to say that some of the
existing out of tree features might not get naked or need some
reworking before they land upstream.

We have been trying to drive the diff down to 0, and we are now close.

[Simon McVittie]

Post by John Johansen
The dbus code went through several revisions as well. While the dbus
code doesn't require a lot from the kernel, it did have some influence
on the kernel support interfaces.

I'm curious: is the kernel side of D-Bus mediation specific to D-Bus,
or is it general-purpose support for mediating user-space activities
that look roughly the same shape as D-Bus?

S

[John Johansen]

Post by Simon McVittie

Post by John Johansen
The dbus code went through several revisions as well. While the dbus
code doesn't require a lot from the kernel, it did have some influence
on the kernel support interfaces.

I'm curious: is the kernel side of D-Bus mediation specific to D-Bus,
or is it general-purpose support for mediating user-space activities
that look roughly the same shape as D-Bus?

The kernel side of D-Bus is generic except one flag.

It provides

- label/context info to the proc/<pid>/attr/current and SO_PEER_SEC
which are wrapped by the libapparmor api fns

- the query interface which allows user space to query policy that
is loaded into the kernel. The dbus code was the first consumer
so it helped shape what the interface looks like and how the
queries are constructed.

- a flag in the features advertising that dbus mediation support
is available. This last one currently is set by the kernel
but ideally would be enabled by the dbus code advising the
kernel module it is mediating.

For dbus mediation we assigned dbus its own class with in the
policydb. Queries to the dbus class need have the dbus query
structure, queries to other classes must follow those classes.
There is certainly some similarity between the classes, but
some of them are quite different from dbus.

[Simon McVittie]

Post by John Johansen
but ideally would be enabled by the dbus code advising the
kernel module it is mediating

"The" dbus code? There can be several parallel instances of dbus-daemon,
possibly different versions of the executable, certainly differently
configured, which can result in any combination of them having
AppArmor mediation enabled or disabled. For example a typical GNOME
laptop will have a system bus, a session bus for the system user
that runs the gdm greeter, and a session bus for the logged-in user
account.

It is meaningful to ask whether a specific dbus-daemon instance is
applying AppArmor mediation, and the latest development branches
advertise this by putting "apparmor" in the bus driver's Features
property. In general it isn't necessarily meaningful to say
"the dbus-daemons running on this kernel are applying AppArmor
mediation" because some of them might be an executable that doesn't
support it, and some of them might support it but have it disabled
in configuration.

So I think this is something that should be queried by asking each
dbus-daemon whether it is mediating, rather than by asking the kernel.

S

[John Johansen]

Post by Simon McVittie

Post by John Johansen
but ideally would be enabled by the dbus code advising the
kernel module it is mediating

"The" dbus code? There can be several parallel instances of dbus-daemon,
possibly different versions of the executable, certainly differently
configured, which can result in any combination of them having
AppArmor mediation enabled or disabled. For example a typical GNOME
laptop will have a system bus, a session bus for the system user
that runs the gdm greeter, and a session bus for the logged-in user
account.
It is meaningful to ask whether a specific dbus-daemon instance is
applying AppArmor mediation, and the latest development branches
advertise this by putting "apparmor" in the bus driver's Features
property. In general it isn't necessarily meaningful to say
"the dbus-daemons running on this kernel are applying AppArmor
mediation" because some of them might be an executable that doesn't
support it, and some of them might support it but have it disabled
in configuration.
So I think this is something that should be queried by asking each
dbus-daemon whether it is mediating, rather than by asking the kernel.

yep having a way to detect/ask individual deamons is the way to go.

I was merely commenting on that the current kernel flag not being
reflective of actual mediation. Which the dbus daemon is providing, and
it (they) should be what is setting the support status, whether in
kernel or by a different means.

Regardless we will be keeping the kernel flag for several years to
provide backwards compat for newer kernels on earlier releases.

[intrigeri]

Hi John et al,

Post by John Johansen

Post by intrigeri

Post by Moritz Mühlenhoff

Post by Christian Seiler
Another thing to consider: if a profile is too restrictive, but the
part that is too restrictive isn't in the upstream kernel yet, then
things could break if you upgrade the kernel to a newer version from
e.g. backports later on. How would you deal with that kind of
breakage during the lifetime of a stable release?

Agreed, that was pretty much my concern.

Thank you so much for highlighting problems I had missed! :)
1. Gather info about how real this problem has been in practice for
Ubuntu: they frequently update their kernel for various already
released distros with the latest AppArmor bits. I think they
occasionally have to update other packages accordingly to adjust
AppArmor policy. I don't know how often this happens. I'll ask them
to compile a list of such stable updates.

[...]
The question specifically asks about, an updated kernel with a policy
that was developed under a different feature set, suddenly breaking
when a new kernel is run on an older system.

Right.

Below you elaborate about ABI compatibility between the kernel,
userspace and policy. Thanks, I've learned a lot!

But even more specifically, the question was about policy updates
mandated to avoid breaking *confined applications* when upgrading to
a kernel that mediates more interfaces than the one in Debian stable.

Christian Seiler put it clearly (quoted above) but here's a more
practical example: say 1. D-Bus mediation lands in Linux
4.15 (totally made up, but this would be nice!); 2. I run Debian
Stretch; 3. I have to run Linux 4.15+ from stretch-backports (e.g.
on a shiny laptop that needs recent drivers). Then any AppArmor
profile shipped in Debian Stretch that was developed without D-Bus
mediation in mind can potentially start breaking the application
it confines.

So our questions to Ubuntu & other distros are:
How have you been dealing with such problems in the past few years?
How often did you have to update packages in a stable release in
order to fix them?

Now, simply enabling AppArmor by default during the Buster development
cycle will give us some of the answers: given many AppArmor features
will land in Linux in the next months/years, we *will* notice if our
policy is outdated :)

Post by John Johansen

Post by intrigeri

Post by Moritz Mühlenhoff
Ideally the feature set used would also be controlled by the
apparmor userspace side.

If we need to go this far: apparmor_parser has a --features-file
option that we could leverage to tie the feature set being used to
something else than the version of the running kernel, e.g.
with a file shipped in a new package built from src:linux with
appropriate versioned dependencies.

the feature file can indeed be specified on the command line using
--feature-file, but from a support pov I think specifying it in the
config file
apparmor/subdomain.conf

Do you mean /etc/apparmor/parser.conf?
I can't find anything related in subdomain.conf(5).

Post by John Johansen
would be better as then you don't have to mess with initscripts, unit
files, etc.

Absolutely. I guess we would want a package built from src:apparmor to
ship that conffile containing "features-file XYZ", where XYZ encodes
the feature set supported by the policy in the version of Debian this
src:apparmor was built for. Which raises a number of technical and
policy questions, not all of them trivial, so I want to first check
whether we really need to go that far (see above).

Post by John Johansen
4.14 - isn't fully decided yet, but it should pickup everything except
maybe the extended unix socket mediation

Just curious: does this "everything except" include D-Bus mediation?

Post by John Johansen
There is recognition that this was the wrong approach and there is
now an upstream first policy.

This, along with the vivid collaboration I see between the GNOME and
Ubuntu projects these days, is very good news :)

Cheers,

[John Johansen]

Post by intrigeri
Hi John et al,

Post by John Johansen

Post by intrigeri

Post by Moritz Mühlenhoff

Post by Christian Seiler
Another thing to consider: if a profile is too restrictive, but the
part that is too restrictive isn't in the upstream kernel yet, then
things could break if you upgrade the kernel to a newer version from
e.g. backports later on. How would you deal with that kind of
breakage during the lifetime of a stable release?

Agreed, that was pretty much my concern.

Thank you so much for highlighting problems I had missed! :)
1. Gather info about how real this problem has been in practice for
Ubuntu: they frequently update their kernel for various already
released distros with the latest AppArmor bits. I think they
occasionally have to update other packages accordingly to adjust
AppArmor policy. I don't know how often this happens. I'll ask them
to compile a list of such stable updates.

[...]
The question specifically asks about, an updated kernel with a policy
that was developed under a different feature set, suddenly breaking
when a new kernel is run on an older system.

Right.
Below you elaborate about ABI compatibility between the kernel,
userspace and policy. Thanks, I've learned a lot!
But even more specifically, the question was about policy updates
mandated to avoid breaking *confined applications* when upgrading to
a kernel that mediates more interfaces than the one in Debian stable.

haha, I had a broader answer dealing with some of this and upon review
had decided the question was about a newer kernel on an older release,
and it would be best to have concise answers around that :)

Post by intrigeri
Christian Seiler put it clearly (quoted above) but here's a more
practical example: say 1. D-Bus mediation lands in Linux
4.15 (totally made up, but this would be nice!); 2. I run Debian
Stretch; 3. I have to run Linux 4.15+ from stretch-backports (e.g.
on a shiny laptop that needs recent drivers). Then any AppArmor
profile shipped in Debian Stretch that was developed without D-Bus
mediation in mind can potentially start breaking the application
it confines.

This is true, hence the suggestion to pin the feature set by
setting the features file in parser.conf This would prevent policy
from enforcing the dbus feature, and prevent the application
from breaking.

I will admit this is not ideal because it applies to all policy loaded
in the namespace (a container could have its own parser and flags)
unless policy is manually loaded with a flag to override it. Which
prevents policy that has been developed with the new feature from
taking advantage of it in this scenario.

There is some work to expose the feature set to policy which will let
policy conditionally choose which features it supports but I can't
promise when that work will land.

Post by intrigeri
How have you been dealing with such problems in the past few years?
How often did you have to update packages in a stable release in
order to fix them?
Now, simply enabling AppArmor by default during the Buster development
cycle will give us some of the answers: given many AppArmor features
will land in Linux in the next months/years, we *will* notice if our
policy is outdated :)

So there are four separate components (kernel, userspace, policy,
application) to discuss here and different potential problems,
depending on the combination.

- kernel: If the kernel is backported and the feature set is pinned
there is a low likely hood of problems. As I addressed previously
there is the potential for a kernel to make changes beyond
apparmor's control that change how/what permission requests reach
apparmor and this can cause problems. Thankfully in practice this
has not happened often.

- apparmor userspace: Baring bugs, new userspaces should just work
with old kernels. Even if the feature set is not pinned, the
userspace will use the old kernel's feature set, so it is equivalent
to pinning.

- applications: When newer versions of applications are backported
they can break under old policy because they provide new features
that old policy wasn't designed for. Policy must be tested and
updated as part of an application backport/SRU.

- policy: The backporting of policy is the most problematic. New
policy shouldn't be dropped onto older applications without testing.

Also new policy may make use of features that are not supported by
an older userspace. In this case policy should be adjusted or a
newer apparmor userspace can be used. If the feature set is pinned
the apparmor userspace can gracefully downgrade unsupported features
so that new policy can work on older feature sets (you can also
configure it to warn or abort).

The single biggest problem is applications that would like to share
a single policy across multiple releases. Ie. they only want to
maintain a single policy for Stretch, and Buster but Stretch
apparmor userspace doesn't support some feature required in Buster.

Currently this requires backporting a newer apparmor userspace and
pinning of the feature set if the kernel changes.

There is work in progress to allow older parsers to recognize and
downgrade new features without directly supporting them but I can
not say when this will land.


Speaking from experience with Ubuntu, the kernel backports are seldom
problematic. We have seen the most issues around application
backports; either policy needing to be updated if it is not shipped as
part of the application, or the application policy requiring features
not supported by the apparmor userspace. This can be dealt with by
either editing the policy or backporting a newer apparmor userspace.

These issues should be and generally are caught during the SRU
process.

Post by intrigeri

Post by John Johansen

Post by intrigeri

Post by Moritz Mühlenhoff
Ideally the feature set used would also be controlled by the
apparmor userspace side.

If we need to go this far: apparmor_parser has a --features-file
option that we could leverage to tie the feature set being used to
something else than the version of the running kernel, e.g.
with a file shipped in a new package built from src:linux with
appropriate versioned dependencies.

the feature file can indeed be specified on the command line using
--feature-file, but from a support pov I think specifying it in the
config file
apparmor/subdomain.conf

Do you mean /etc/apparmor/parser.conf?
I can't find anything related in subdomain.conf(5).

ah yep, oops there used to be a subdomain.conf, back before Novell
renamed the project to AppArmor.

Post by intrigeri

Post by John Johansen
would be better as then you don't have to mess with initscripts, unit
files, etc.

Absolutely. I guess we would want a package built from src:apparmor to
ship that conffile containing "features-file XYZ", where XYZ encodes
the feature set supported by the policy in the version of Debian this
src:apparmor was built for. Which raises a number of technical and
policy questions, not all of them trivial, so I want to first check
whether we really need to go that far (see above).

Post by John Johansen
4.14 - isn't fully decided yet, but it should pickup everything except
maybe the extended unix socket mediation

Just curious: does this "everything except" include D-Bus mediation?

No D-Bus mediation depends on the extended unix domain socket
mediation.

And I can now say that the extended unix socket mediation didn't make
it into 4.14 but everything else did. So you can use basic af socket
rules.

Post by intrigeri

Post by John Johansen
There is recognition that this was the wrong approach and there is
now an upstream first policy.

This, along with the vivid collaboration I see between the GNOME and
Ubuntu projects these days, is very good news :)
Cheers,

[Guido Günther]

Hi John,

very interesting read!

Post by John Johansen

Post by intrigeri
Hi John et al,

Post by John Johansen

Post by intrigeri

Post by Moritz Mühlenhoff

Post by Christian Seiler
Another thing to consider: if a profile is too restrictive, but the
part that is too restrictive isn't in the upstream kernel yet, then
things could break if you upgrade the kernel to a newer version from
e.g. backports later on. How would you deal with that kind of
breakage during the lifetime of a stable release?

Agreed, that was pretty much my concern.

Thank you so much for highlighting problems I had missed! :)
1. Gather info about how real this problem has been in practice for
Ubuntu: they frequently update their kernel for various already
released distros with the latest AppArmor bits. I think they
occasionally have to update other packages accordingly to adjust
AppArmor policy. I don't know how often this happens. I'll ask them
to compile a list of such stable updates.

[...]
The question specifically asks about, an updated kernel with a policy
that was developed under a different feature set, suddenly breaking
when a new kernel is run on an older system.

Right.
Below you elaborate about ABI compatibility between the kernel,
userspace and policy. Thanks, I've learned a lot!
But even more specifically, the question was about policy updates
mandated to avoid breaking *confined applications* when upgrading to
a kernel that mediates more interfaces than the one in Debian stable.

haha, I had a broader answer dealing with some of this and upon review
had decided the question was about a newer kernel on an older release,
and it would be best to have concise answers around that :)

Post by intrigeri
Christian Seiler put it clearly (quoted above) but here's a more
practical example: say 1. D-Bus mediation lands in Linux
4.15 (totally made up, but this would be nice!); 2. I run Debian
Stretch; 3. I have to run Linux 4.15+ from stretch-backports (e.g.
on a shiny laptop that needs recent drivers). Then any AppArmor
profile shipped in Debian Stretch that was developed without D-Bus
mediation in mind can potentially start breaking the application
it confines.

This is true, hence the suggestion to pin the feature set by
setting the features file in parser.conf This would prevent policy
from enforcing the dbus feature, and prevent the application
from breaking.
I will admit this is not ideal because it applies to all policy loaded
in the namespace (a container could have its own parser and flags)
unless policy is manually loaded with a flag to override it. Which
prevents policy that has been developed with the new feature from
taking advantage of it in this scenario.
There is some work to expose the feature set to policy which will let
policy conditionally choose which features it supports but I can't
promise when that work will land.

Post by intrigeri
How have you been dealing with such problems in the past few years?
How often did you have to update packages in a stable release in
order to fix them?
Now, simply enabling AppArmor by default during the Buster development
cycle will give us some of the answers: given many AppArmor features
will land in Linux in the next months/years, we *will* notice if our
policy is outdated :)

So there are four separate components (kernel, userspace, policy,
application) to discuss here and different potential problems,
depending on the combination.
- kernel: If the kernel is backported and the feature set is pinned
there is a low likely hood of problems. As I addressed previously
there is the potential for a kernel to make changes beyond
apparmor's control that change how/what permission requests reach
apparmor and this can cause problems. Thankfully in practice this
has not happened often.
- apparmor userspace: Baring bugs, new userspaces should just work
with old kernels. Even if the feature set is not pinned, the
userspace will use the old kernel's feature set, so it is equivalent
to pinning.
- applications: When newer versions of applications are backported
they can break under old policy because they provide new features
that old policy wasn't designed for. Policy must be tested and
updated as part of an application backport/SRU.

This (and somewhat related the next point) is the reason why policy
should be shipped by the application (that is the Debian package
containing the application), not in an apparmor-profiles package. This
makes sure the profile matches the application. Everything else calls
out for trouble.

Cheers,
-- Guido

Post by John Johansen
- policy: The backporting of policy is the most problematic. New
policy shouldn't be dropped onto older applications without testing.
Also new policy may make use of features that are not supported by
an older userspace. In this case policy should be adjusted or a
newer apparmor userspace can be used. If the feature set is pinned
the apparmor userspace can gracefully downgrade unsupported features
so that new policy can work on older feature sets (you can also
configure it to warn or abort).
The single biggest problem is applications that would like to share
a single policy across multiple releases. Ie. they only want to
maintain a single policy for Stretch, and Buster but Stretch
apparmor userspace doesn't support some feature required in Buster.
Currently this requires backporting a newer apparmor userspace and
pinning of the feature set if the kernel changes.
There is work in progress to allow older parsers to recognize and
downgrade new features without directly supporting them but I can
not say when this will land.
Speaking from experience with Ubuntu, the kernel backports are seldom
problematic. We have seen the most issues around application
backports; either policy needing to be updated if it is not shipped as
part of the application, or the application policy requiring features
not supported by the apparmor userspace. This can be dealt with by
either editing the policy or backporting a newer apparmor userspace.
These issues should be and generally are caught during the SRU
process.

Post by intrigeri

Post by John Johansen

Post by intrigeri

Post by Moritz Mühlenhoff
Ideally the feature set used would also be controlled by the
apparmor userspace side.

If we need to go this far: apparmor_parser has a --features-file
option that we could leverage to tie the feature set being used to
something else than the version of the running kernel, e.g.
with a file shipped in a new package built from src:linux with
appropriate versioned dependencies.

the feature file can indeed be specified on the command line using
--feature-file, but from a support pov I think specifying it in the
config file
apparmor/subdomain.conf

Do you mean /etc/apparmor/parser.conf?
I can't find anything related in subdomain.conf(5).

ah yep, oops there used to be a subdomain.conf, back before Novell
renamed the project to AppArmor.

Post by intrigeri

Post by John Johansen
would be better as then you don't have to mess with initscripts, unit
files, etc.

Absolutely. I guess we would want a package built from src:apparmor to
ship that conffile containing "features-file XYZ", where XYZ encodes
the feature set supported by the policy in the version of Debian this
src:apparmor was built for. Which raises a number of technical and
policy questions, not all of them trivial, so I want to first check
whether we really need to go that far (see above).

Post by John Johansen
4.14 - isn't fully decided yet, but it should pickup everything except
maybe the extended unix socket mediation

Just curious: does this "everything except" include D-Bus mediation?

No D-Bus mediation depends on the extended unix domain socket
mediation.
And I can now say that the extended unix socket mediation didn't make
it into 4.14 but everything else did. So you can use basic af socket
rules.

Post by intrigeri

Post by John Johansen
There is recognition that this was the wrong approach and there is
now an upstream first policy.

This, along with the vivid collaboration I see between the GNOME and
Ubuntu projects these days, is very good news :)
Cheers,

[intrigeri]

Post by Guido Günther

Post by John Johansen
- applications: When newer versions of applications are backported
they can break under old policy because they provide new features
that old policy wasn't designed for. Policy must be tested and
updated as part of an application backport/SRU.

This (and somewhat related the next point) is the reason why policy
should be shipped by the application (that is the Debian package
containing the application), not in an apparmor-profiles package. This
makes sure the profile matches the application.

Absolutely. Here's the status / progress update on this front:

- In the last 1.5 year three profiles (Evince, ntp and tcpdump) were
moved from apparmor-profiles-extra to the corresponding package.

- apparmor-profiles ships only a few profiles currently, all of them
in complain mode; there's discussion (#830502) about what should be
done there.

- The only other apparmor* package that ships policy for other
software is apparmor-profiles-extra, which currently enforces
profiles for apt-cacher-ng, Pidgin and Totem.

I want to move them to the corresponding package during the Buster
cycle. If you want to help, I recommend starting with
usr.bin.pidgin and usr.sbin.apt-cacher-ng: both are pretty stable
and have needed very few updates in the last few years. Totem is
more complicated; it's also the one that would benefit the most
from being shipped in src:totem, provided a good workflow is set up
so users, Totem maintainers and AppArmor people are all happy.

Cheers,

[intrigeri]

Hi,

Post by John Johansen

Post by intrigeri
Christian Seiler put it clearly (quoted above) but here's a more
practical example: say 1. D-Bus mediation lands in Linux
4.15 (totally made up, but this would be nice!); 2. I run Debian
Stretch; 3. I have to run Linux 4.15+ from stretch-backports (e.g.
on a shiny laptop that needs recent drivers). Then any AppArmor
profile shipped in Debian Stretch that was developed without D-Bus
mediation in mind can potentially start breaking the application
it confines.

This is now happening: the side-effect of many new AppArmor features
having landed in Linux mainline (good news!) — before our policy was
updated accordingly — is that AppArmor broke a couple things when
Linux 4.13 landed in sid (sorry, it's the first time we face this
situation and we're learning along the way!), and breaks *lots* of
things when running Linux 4.14. This situation is exceptional in that
years of kernel development are being upstreamed at once, but it's
bound to happen again at a smaller scale, and we do need to deal with
it to support the use case described above anyway.

My plan is:

1. In testing/sid, ship a conffile (in a package built from
src:apparmor) that pins the most recent feature set fully supported
by our policy, i.e. Linux 4.12's or 4.13's (depending on whether
we've fixed all the regressions brought by 4.13 yet); this is now
tracked by #879584. This should make the transition to Linux 4.14
smooth;

2. Once our policy has been updated to work well with Linux 4.14's
AppArmor features (#877581), bump the pinned feature set to 4.14's.

3. Rinse & repeat for Linux 4.15 etc.

And wrt. Stretch + Linux from backports, I'll propose a stable update
of src:apparmor that ships a similar conffile that pins the feature
set to Linux 4.9's (#879585).

I've mentioned a few issues with this plan on the corresponding bug
reports. I propose we discuss them there to avoid overloading this
thread with too many fine details :)

Post by John Johansen
- kernel: If the kernel is backported and the feature set is pinned
there is a low likely hood of problems.

… so we should be good with the above plan. Thanks!

Cheers,

[intrigeri]

Post by Christian Seiler

Post by intrigeri
Rules that are not supported by the running kernel are silently
ignored, i.e. the operation is allowed.

Is there at least a warning during the load of the profile?

There used to be a warning, but it was causing lots of confusion in
Debian: users were wondering if *any* of the AppArmor profile that
caused warning was applied at all. So in Jessie we decided to hide
these warnings.

Post by Christian Seiler
Or, conversely, is there a possibility to add a flag to the AppArmor
profile to say "fail to load it if something is not understood"? In
that case all profiles shipped by Debian would not include that (for
interoperability reasons) but it could be documented that as a best
practice for admins they should use that flag so that they can be
sure that all protections they specified are actually affected.

If we're fine with relying purely on documentation to address this
problem for sysadmins writing their own profiles, then we can suggest
they use the existing apparmor_parser options about this:

alias apparmor_parser='apparmor_parser --warn=rules-not-enforced --warn=rule-downgraded'

… and then no new code needs to be written :)

Would that be good enough in your opinion?

Thanks for all this useful input!

Cheers,

[Christian Seiler]

Post by intrigeri

Post by Christian Seiler

Post by intrigeri
Rules that are not supported by the running kernel are silently
ignored, i.e. the operation is allowed.

Is there at least a warning during the load of the profile?

There used to be a warning, but it was causing lots of confusion in
Debian: users were wondering if *any* of the AppArmor profile that
caused warning was applied at all. So in Jessie we decided to hide
these warnings.

Good to know. And since this kind of warning is more likely to lead
people to disable AppArmor altogether because they want to get rid
of the message and are scared by it, you're probably right in
removing that warning, because limited AppArmor protections are
probably still better as a defense-in-depth strategy than none at
all.

Post by intrigeri

Post by Christian Seiler
Or, conversely, is there a possibility to add a flag to the AppArmor
profile to say "fail to load it if something is not understood"? In
that case all profiles shipped by Debian would not include that (for
interoperability reasons) but it could be documented that as a best
practice for admins they should use that flag so that they can be
sure that all protections they specified are actually affected.

If we're fine with relying purely on documentation to address this
problem for sysadmins writing their own profiles, then we can suggest
alias apparmor_parser='apparmor_parser --warn=rules-not-enforced --warn=rule-downgraded'
… and then no new code needs to be written :)
Would that be good enough in your opinion?

If that documentation is easy enough to find: sure, yes.

Speaking of: are there any good introductions for AppArmor? Not
necessarily for me personally (I've had some experience with
SELinux, so I recon I'll figure AppArmor out easily enough), but
for beginners? Something I can point friends of mine to? Ideally
something that doesn't just describe the syntax, but also best
practices? When I first ran into SELinux I found a couple of
tutorials which did things where I thought, well this doesn't
seem quite well enough thought out, I can think of a couple of
corner cases where this would fainot do what it was advertising,
only to later find that my impression was true and people with
even more knowledge of SELinux talked about not doing the things
I had seen in these tutorials, and suggesting more sensible
things as best practices. Unfortunately the writings by the real
experts were not in tutorial form. For buster, if AppArmor is
enabled by default (which btw. I'm in favor of, in case that was
not clear), I think we should take care to nudge people towards
the resources that describe best practices.

Regards,
Christian

[Raphael Hertzog]

Hi,

Post by Christian Seiler
Speaking of: are there any good introductions for AppArmor? Not
necessarily for me personally (I've had some experience with
SELinux, so I recon I'll figure AppArmor out easily enough), but
for beginners? Something I can point friends of mine to? Ideally
something that doesn't just describe the syntax, but also best
practices? When I first ran into SELinux I found a couple of

https://debian-handbook.info/browse/stable/sect.apparmor.html

Feedback welcome as bug reports against the debian-handbook package.
Note that this (still) describes the status of the package
in jessie, I have not yet finished to update the book for stretch.

Cheers,

[intrigeri]

Post by Raphael Hertzog
https://debian-handbook.info/browse/stable/sect.apparmor.html

Thanks, added to https://wiki.debian.org/AppArmor#External_links :)

Cheers,

[intrigeri]

Hi,

Post by Christian Seiler

Post by intrigeri

Post by Christian Seiler
Or, conversely, is there a possibility to add a flag to the AppArmor
profile to say "fail to load it if something is not understood"? In
that case all profiles shipped by Debian would not include that (for
interoperability reasons) but it could be documented that as a best
practice for admins they should use that flag so that they can be
sure that all protections they specified are actually affected.

If we're fine with relying purely on documentation to address this
problem for sysadmins writing their own profiles, then we can suggest
alias apparmor_parser='apparmor_parser --warn=rules-not-enforced --warn=rule-downgraded'
… and then no new code needs to be written :)
Would that be good enough in your opinion?

If that documentation is easy enough to find: sure, yes.

For now, I've added this to
https://wiki.debian.org/AppArmor/Contribute#Create_new_profiles

Once we've made progress on the documentation front (see below)
we can ensure that whatever resource we recommend documents this.

Post by Christian Seiler
Speaking of: are there any good introductions for AppArmor?
[…]
For buster, if AppArmor is enabled by default (which btw. I'm in
favor of, in case that was not clear), I think we should take care
to nudge people towards the resources that describe best practices.

There are a number of AppArmor tutorials aimed at beginners. We link
to a few of them on https://wiki.debian.org/AppArmor#External_links.
but the mere fact that there are 7 of them, with greatly overlapping
content and no indication of who the target audience is, suggests we
don't have a good answer to your question yet.

So I've filed #874873 (severity: important) to keep this on our radar.

Thanks for caring,
cheers!

[Clint Byrum]

Post by intrigeri
- Apparently Ubuntu users have been coping with AppArmor enforced
by default for 9 years. I see no reason why Debian users would not.

This is really important. A few packages in Ubuntu largely differ from
Debian because they have an AppArmor profile that hasn't yet been
incorporated into Debian. That's a profile that is already tested and
used by many Ubuntu users, and should work well with a Debian system.

I found your proposal thorough and reasonable. I hope it happens.

[Ritesh Raj Sarraf]

Post by intrigeri
tl;dr: I hereby propose we enable AppArmor by default in testing/sid,
and decide one year later if we want to keep it this way in the
Buster release.

On most systems, people tend to disable LSM first. Because many a times
an inadequate policy hinders the use of the tool. And on the desktop
machine this becomes more common an issue.

On the SELinux side, there used to be a nice reporting tool for the
desktop, setroubleshoot. It used to alert (graphical and console) any
policy violations.

A DE agnostic alert tool would be necessary for wide adoption of any
LSM implementation.

In my opinion, we should start with an alert tool (to report policy
violations), and a handful of server packages.

[Jeremy Bicha]

Post by Ritesh Raj Sarraf
On most systems, people tend to disable LSM first. Because many a times
an inadequate policy hinders the use of the tool. And on the desktop
machine this becomes more common an issue.

While there are plenty of people who recommend disabling SELinux by
default on Fedora/RHEL (at least in the past), it's far, far less
common to hear about anyone recommending disabling AppArmor. And
that's one reason AppArmor is being proposed here and not SELinux.

Thanks,
Jeremy Bicha

[Ritesh Raj Sarraf]

Hello Jeremy,

Post by Jeremy Bicha
While there are plenty of people who recommend disabling SELinux by
default on Fedora/RHEL (at least in the past), it's far, far less
common to hear about anyone recommending disabling AppArmor. And
that's one reason AppArmor is being proposed here and not SELinux.

They both serve the same purpose; to confine a running process, by the
mandated policy. The core problem is the policy, which may/will not fit
all users irrespective of the LSM implementation they use.

The previous adopters were all targeted distributions and it still took
some time for users to get accustomed to it.

RHEL confined most server packages. For the rest of the system, I
believe it ran unconfined. And they used to have a decent reporting
tool for policy violations.

SUSE used to have a robust LSM integration into YaST.

For a general purpose distribution like Debian, intri's proposal on the
server side is good. We could definitely leverage the policy files from
Ubuntu and SUSE.

But for desktop users, I worry this would cause more pain.
But I see there's an apparmor-notify package. Maybe that is the answer.
I'll check that out some time soon.


Thanks,
Ritesh

[Jeremy Bicha]

Post by Ritesh Raj Sarraf
But for desktop users, I worry this would cause more pain.

My point is that it hasn't so far and Ubuntu and SUSE has had millions
of people using it for a decade, more or less.

Thanks,
Jeremy Bicha

[Garrett R.]

I have Ubuntu running on another box. I haven't had any trouble (to my knowledge) that has been caused from apparmor. Ubuntu being perceived as an entry OS for linux, I would think Canonical wouldn't have included it if it would introduce pain to desktop users. What sort of "pain" might apparmor cause?

I assumed apparmor was also on debian. If it isn't, doesn't this make Ubuntu significantly more secure than a debian installation?

----- Original Message -----
From: "Jeremy Bicha" <***@ubuntu.com>
To: "Ritesh Raj Sarraf" <***@debian.org>
Cc: debian-***@lists.debian.org
Sent: Monday, August 7, 2017 9:11:32 AM
Subject: Re: Let's enable AppArmor by default (why not?)

Post by Ritesh Raj Sarraf
But for desktop users, I worry this would cause more pain.

My point is that it hasn't so far and Ubuntu and SUSE has had millions
of people using it for a decade, more or less.

Thanks,
Jeremy Bicha

[intrigeri]

Hi,

Post by Ritesh Raj Sarraf
But I see there's an apparmor-notify package.

Sadly it's not well integrated in Debian currently.

Root cause of the problem:
https://bugs.launchpad.net/apparmor/+bug/1597671

Short term workaround: https://bugs.debian.org/759604

Post by Ritesh Raj Sarraf
Maybe that is the answer.

I suspect the expected benefits are not worth the effort. I'd rather
see us put efforts in automated testing (to identify & fix bugs before
they affect less tech-savvy users) and in fixing actual UX
stumbling blocks.

(apparmor-notify is merely a glorified log reader displaying low-level
technical details in a GUI. So the info it gives people who are not
tech-savvy is essentially "the problem I'm experiencing might have
something to do with AppArmor". I suspect they will have a harder time
interacting with our BTS than copy'n'pasting the corresponding
AppArmor logs, and apparmor-notify won't change that.)

Cheers,

[Chris Lamb]

Hi intrigeri,

Post by intrigeri
tl;dr: I hereby propose we enable AppArmor by default in testing/sid,
and decide one year later if we want to keep it this way in the
Buster release.

Thanks for such a comprehensive and compelling write-up :)

Post by intrigeri
https://wiki.debian.org/AppArmor/HowToUse

$ sudo aa-status | head -n2
apparmor module is loaded.
49 profiles are loaded.

(Well, I should take more risks, right…?)

Post by intrigeri
* If you maintain a package for which we ship AppArmor policy in
Debian: test it with AppArmor enabled before uploading.

Related to this, most of my packages are 'server'-ish and it feels
like some of the hardening features are also/already covered by my
systemd .service files.

Should/could I be also reimplementing these in AppArmor for defense
in depth or any comments in this general area?


Best wishes,

[intrigeri]

Hi,

Post by Chris Lamb
Related to this, most of my packages are 'server'-ish and it feels
like some of the hardening features are also/already covered by my
systemd .service files.

Quite possibly :)

Post by Chris Lamb
Should/could I be also reimplementing these in AppArmor for defense
in depth or any comments in this general area?

You surely could, but reimplementing exactly the same protections is
probably not the best use of your time.

Now, each of systemd and AppArmor can do hardening stuff that the
other doesn't support (at all or nicely), so sometimes it's good to do
a little bit of both.

What I would do these days:

1. Use the simplest of systemd's hardening features (e.g.
Protect{Home,System}=, Private{Devices,Tmp,Network}=,
CapabilityBoundingSet=) to their full extend.

Not many unit files we ship do that yet. Generally these
improvements can be implemented upstream and benefit users of
systemd on other distros :)

2. For finer-grained mediation of filesystem access, I will use
AppArmor that I find much nicer to implement, use and debug:

* To limit write access, with systemd one must set
ReadOnlyDirectories=/ and then a bunch of ReadWriteDirectories=
directives. It works fine once you're done, but I found it hard
to implement and debug this setup: with AppArmor I have clear
logs about what access the service was denied, and with systemd
I have no such thing. This probably explains why only one
systemd unit installed on my system bothers doing so.

* To limit read access, AFAIK with systemd one can only list
forbidden places and everything else is allowed by default.
No wonder InaccessiblePaths= is not used by any unit installed
on my system.

Cheers,

[Russ Allbery]

Post by intrigeri
You surely could, but reimplementing exactly the same protections is
probably not the best use of your time.
Now, each of systemd and AppArmor can do hardening stuff that the
other doesn't support (at all or nicely), so sometimes it's good to do
a little bit of both.
1. Use the simplest of systemd's hardening features (e.g.
Protect{Home,System}=, Private{Devices,Tmp,Network}=,
CapabilityBoundingSet=) to their full extend.
Not many unit files we ship do that yet. Generally these
improvements can be implemented upstream and benefit users of
systemd on other distros :)

To this, I would add SystemCallFilters, which is immensely powerful and
much easier to use now than it was originally. I think a lot of the
benefit of hardening for a lot of network daemons would come from tested
use of SystemCallFilters (probably only using the @group syntax). This
blocks so many local privilege escalation bugs in random Linux syscalls,
making an RCE in a daemon running as a non-root user much less scary.

(For those not familiar, this is seccomp filtering, except about as easy
to use as seccomp can be. It's not as good as a carefully crafted full
seccomp policy, but you can get quite a bit of mileage out of some fairly
simple rules.)

Note that SystemCallFilters was very difficult to use in jessie since the
syscall groups weren't implemented yet. I would not recommend trying to
use it if targeting anything older than stretch.

Post by intrigeri
2. For finer-grained mediation of filesystem access, I will use

Agreed -- AppArmor seems to offer much more granular control over file
system access in a way that's quite a bit easier to configure.
ProtectSystem is great, but the bits that list specific files or paths are
somewhat harder to use and less useful.

[Chris Lamb]

Hey intri,

Post by intrigeri
1. Use the simplest of systemd's hardening features (e.g.
Protect{Home,System}=, Private{Devices,Tmp,Network}=,
CapabilityBoundingSet=) to their full extend.
Not many unit files we ship do that yet. Generally these
improvements can be implemented upstream and benefit users of
systemd on other distros :)

Indeed! :) For example, here I'm merging upstream's rather more locked
down unit file into the Debian one:

https://bugs.debian.org/871610


Regards,

[Lisandro Damián Nicanor Pérez Meyer]

Post by intrigeri
Hi!
tl;dr: I hereby propose we enable AppArmor by default in testing/sid,
and decide one year later if we want to keep it this way in the
Buster release.
- Get a rough idea of what amount of effort the Debian project is
happy (and able) to invest into such proactive security measures.
- Learn about any relevant social & technical concerns I am not
aware of.
I expect the proposal will need to be refined, or abandoned, to take
into account what we will learn during this preliminary discussion.

From the way you redacted this you made me want to try apparmor. That's
already a big compliment :-)

For what I read in the proposal and in the thread I do believe we need to at
least try this for a year.

So my +1 for going ahead.

[Chris Lamb]

Hi intri,

Post by intrigeri
tl;dr: I hereby propose we enable AppArmor by default in testing/sid,
and decide one year later if we want to keep it this way in the
Buster release.

So… in the spirit of taking (reversible!) risks, can you briefly outline
what's blocking us enabling this today? :)


Best wishes,

[intrigeri]

Hi,

Post by Chris Lamb
So… in the spirit of taking (reversible!) risks, can you briefly outline
what's blocking us enabling this today? :)

Thanks for asking!

I've scheduled time on October 23-27 to:

1. identify what still prevents us from starting the proposed
experiment

2. fix all the problems identified in #1

3. document on the BTS what must be fixed between the time we start
the experiment and the time we decide what to do for the Buster
release

After this we should be in a very good position to go ahead and press
the big red button :)

The feedback I've received off-list and on the BTS so far has been
extremely encouraging: a few DDs enabled AppArmor on their systems,
reported bugs that were promptly fixed, and some of them successfully
fixed the bug themselves locally despite having no previous experience
with AppArmor… which is exactly how it should be for the LSM enabled
by default in Debian IMO. In at least one case we realized only after
I had fixed the bug and submitted a fix upstream that their own, local
workaround was identical to my own fix, which I find enlightening wrt.
the AppArmor learning curve.

Cheers,

[intrigeri]

Hi,

Post by intrigeri

Post by Chris Lamb
So… in the spirit of taking (reversible!) risks, can you briefly outline
what's blocking us enabling this today? :)

Thanks for asking!

We made good progress. Thanks a lot to Vincas Dargis and the Ubuntu
and openSUSE folks who helped quite a bit :)

Post by intrigeri
1. identify what still prevents us from starting the proposed
experiment

Done, see below for the remaining ones.

Post by intrigeri
2. fix all the problems identified in #1

We're almost there! Remaining blockers:

- deal with Linux 4.14 bringing in new mediation features and having
a bug (until -rc6 at least) precisely in the way it handles the
obvious mitigation I've applied (feature set pinning): tracked by
#877581, likely 4.14-rc7 will fix it; worst case, if Linux 4.14
reaches sid with this bug not fixed yet, I'll revert the feature
set pinning and we'll deal with whatever bits of policy need
updates (the most important ones all have patches submitted
upstream + to the BTS already so I'm confident)

- enable AppArmor by default in our Linux kernel: I'll file a bug
about it once the above issue is resolved

- decide how to pull the AppArmor policy + userspace: tracked by
#879590, started discussion on this list and with debian-boot@

- anything else?

Post by intrigeri
3. document on the BTS what must be fixed between the time we start
the experiment and the time we decide what to do for the Buster
release

I'll come back to this later. Not a blocker to start the experiment IMO.

Cheers,

[Ben Hutchings]

On Fri, 2017-10-27 at 11:53 +0200, intrigeri wrote:
[...]

Post by intrigeri

Post by intrigeri
2. fix all the problems identified in #1

- deal with Linux 4.14 bringing in new mediation features and having
a bug (until -rc6 at least) precisely in the way it handles the
obvious mitigation I've applied (feature set pinning): tracked by
#877581, likely 4.14-rc7 will fix it;

It seems to have been fixed - kind of - by a revert:

commit 80c094a47dd4ea63375e3f60b5e076064f16e857
Author: Linus Torvalds <***@linux-foundation.org>
Date: Thu Oct 26 19:35:35 2017 +0200

Revert "apparmor: add base infastructure for socket mediation"

Let's hope socket mediation will be enabled again in a compatible way
for 4.15.

Post by intrigeri
worst case, if Linux 4.14
reaches sid with this bug not fixed yet, I'll revert the feature
set pinning and we'll deal with whatever bits of policy need
updates (the most important ones all have patches submitted
upstream + to the BTS already so I'm confident)
- enable AppArmor by default in our Linux kernel: I'll file a bug
about it once the above issue is resolved

[...]

Already did it with today's uploads. :-)

Ben.

[Neal Gompa]

Hello,

I was recently pointed to the thread going on debian-devel about
enabling AppArmor LSM in Debian, and as I read through your proposal,
I felt it should be warranted to point out a few things in regards to

Post by intrigeri
Why AppArmor and not SELinux?
-----------------------------
SELinux is another LSM that tackles similar problems.
Disclaimer: I've picked AppArmor years ago and didn't look much at
SELinux recently, so some of what follows may be totally wrong or
outdated. Sorry! Debian SELinux people, if you don't mind please help
me get the basic facts right :)
* Allows mediating more kernel objects / interfaces than AppArmor, so
policy can be made stricter and safer given sufficient expertise
and available time for maintenance.
* Enabled by default in RHEL so in theory a great number of sysadmins
are at ease with it (see below why reality may not match this).

It's also important to note that it is also enabled by default in
Fedora, which is the upstream for RHEL.

Post by intrigeri
* A quick look at popcon suggests that SELinux might be more popular
in Debian than AppArmor, but I'm not sure I am interpreting the
numbers right (and I suspect that just like AppArmor, the popcon
won't tell us if users who have installed the relevant support
packages actually run their system with the corresponding LSM
enabled & enforced).

I do know of users of SELinux in Debian and Ubuntu, though they often
fork from refpolicy or fedora-selinux the bits they want to use and
install it on top of the current refpolicy offered in Debian.

Post by intrigeri
* Writing, maintaining, auditing and debugging SELinux policy
requires grasping a complex conceptual model; I am told this is not
as easy as doing the same with AppArmor.

This is not really true. While it is true that the conceptual model is
more complex, the tooling for doing all the regular work with SELinux
is great. In many cases, the tools can analyze what's happened and
suggest a course of action about how to fix it. If it looks like a
bug, they suggest filing one with the vendor (in my case, when weird
things happen with the SELinux policy in Fedora, bugs get filed on
selinux-policy with the information from setroubleshoot so that things
can get fixed).

As for the complexity of making policies and policy modules, I've
written a few policy modules, and they're not that bad. You can make
some pretty simple policies if you don't want to expose any
administrative tunables. That said, even with the tunables, it's not
that bad.

For example, the container-selinux policy module is pretty easy to
understand: https://github.com/projectatomic/container-selinux

The refpolicy documentation is pretty comprehensive too:
http://oss.tresys.com/docs/refpolicy/api/

One of the members of the Red Hat/Fedora SELinux team maintains a nice
blog describing improvements as they come:
https://lvrabec-selinux.rhcloud.com/

There's a few videos done by Red Hat folks on SELinux that are worth
watching on YouTube:

* Are you listening to what SELinux is telling you?:

* Security-enhanced Linux for mere mortals - 2015 Red Hat Summit:

* SELinux by Paul Moore:

There's also a great video by Noah Chelliah about SELinux:

* Security Enhanced Linux | Ask Noah 9:
http://www.jupiterbroadcasting.com/115151/security-enhanced-linux-ask-noah-9/

Post by intrigeri
* As far as I could understand when chatting with sysadmins of Red
Hat systems, this has resulted in a culture where many users got
used to disable SELinux entirely on their systems, instead of
trying to fix the buggy policy. I've seen the opposite happen with
AppArmor, which is good: for example, pretty often bug reporters to
the Debian BTS document themselves how they could workaround the
problem locally *without* turning AppArmor off. Looking at open
bugs in the BTS against src:refpolicy, this seems to happen very
rarely for SELinux, so I wonder if it would be realistic to ship
Debian with SELinux enforced by default and have our community
support it.

Back in the RHEL 5 days, this is definitely true. And if many of of
the Red Hat sysadmins you've talked to primarily maintain RHEL 5
systems (which is not unlikely), then it makes sense. Back in the RHEL
5 days (circa 2007), the tooling was very primitive, and for the most
part, the troubleshooting tools didn't exist.

Today in Fedora and RHEL 7, the tooling is very advanced, and in
almost every case where I've hit AVC denials in SELinux,
setroubleshoot has given me very useful information including
suggested course of actions to actually fix it locally. It would
probably not be all that difficult to also support the case of
submitting bugreports to the BTS (it currently can for Bugzilla based
systems, such as Red Hat Bugzilla and SUSE Bugzilla).

Most of my fellow RHEL/CentOS users on RHEL/CentOS 7 have been pretty
happy with leaving it on in enforcing mode with the default targeted
policy. It's well-tuned and stays out of the user's way in most cases.

I won't say it's perfect, and there certainly have been issues. But
when they do happen, they do get filed in the Red Hat Bugzilla for the
SELinux team to know about it and fix.

You can see some of the bug reports that roll in the Red Hat Bugzilla
here: https://bugzilla.redhat.com/buglist.cgi?bug_status=NEW&bug_status=ASSIGNED&bug_status=POST&bug_status=MODIFIED&bug_status=ON_QA&bug_status=VERIFIED&bug_status=CLOSED&classification=Fedora&component=selinux-policy&list_id=7918855&product=Fedora&query_format=advanced

Post by intrigeri
* https://wiki.debian.org/SELinux/Issues says "Graphical/Desktop
installs of Debian are not heavily tested with selinux, so you
might run into quite some issues".

This is true for both AppArmor and SELinux. With the exception of the
Snap case, neither MAC has been optimized for handling desktop issues
that much. There *is* some work in Fedora for confining browsers, in
part due to the usage of plugins, but modern browsers ship their own
seccomp-based confinement mechanisms, so SELinux is usually configured
to stay out of the way and just make sure the process doesn't access
anything a browser isn't supposed to access.

And I'm hopeful that eventually even the Snap case, SELinux-based
confinement will come soon. I've already written a basic SELinux
policy for snapd itself (no such profile exists for AppArmor, IIRC), and I
hope to continue to extend and improve the state of the SELinux policy
for snapd so that we know what it does and the daemon itself doesn't
do things it's not supposed to.

Post by intrigeri
* I'm not aware of any Debian derivative shipping with SELinux
enabled by default. If that's correct, then it means that we would
have to deal with quite some policy compatibility issues ourselves.

To be fair, refpolicy was RC'd from Debian in 8, due to failing to
build and no one fixed it quickly enough. It was reintroduced in
Debian 9, though. I have not personally tested the SELinux support in
Debian 9, but I've heard from a few friends that it does work.

Post by intrigeri
To me, the complexity of SELinux is a deal breaker: it seems that we
would need to get lots more expertise and energy to enforce SELinux by
default than doing the same with AppArmor.

The unfortunate thing is that more comprehensive security models do
lead to more complexity.

Post by intrigeri
Now, if for some reason the project prefers to ship with SELinux
enforced instead of AppArmor, fine by me: I would strongly prefer this
option to nothing at all.

I personally would like to see Debian ship SELinux by default, but as
I'm not a part of Debian, my opinion doesn't matter. ;)

But I definitely don't want people to think that SELinux is some crazy
mountainous path full of terrible unknowns.

If you have any other questions or would like to know more, feel free
to ask, and I'll do my best to answer. :)


Best regards,

[intrigeri]

Hi Neal & others,

Post by Neal Gompa
I was recently pointed to the thread going on debian-devel about
enabling AppArmor LSM in Debian, and as I read through your proposal,
I felt it should be warranted to point out a few things in regards to

Thanks a lot for your carefully worded and extremely well sourced
email! I've already learned quite a few interesting things.

Post by Neal Gompa

Post by intrigeri
Why AppArmor and not SELinux?
-----------------------------
SELinux is another LSM that tackles similar problems.

[...]

Post by Neal Gompa

Post by intrigeri
* Enabled by default in RHEL so in theory a great number of sysadmins
are at ease with it (see below why reality may not match this).

It's also important to note that it is also enabled by default in
Fedora, which is the upstream for RHEL.

Sure. I didn't mention it because I don't see this as very relevant in
the context of this discussion: it's a fact that many sysadmins active
in Debian have to use RHEL/CentOS at work, but I doubt many Debian
people are this much exposed to Fedora, so I don't think it's a good
source of pre-existing SELinux expertise in Debian.

Post by Neal Gompa
I do know of users of SELinux in Debian and Ubuntu, though they often
fork from refpolicy or fedora-selinux the bits they want to use and
install it on top of the current refpolicy offered in Debian.

Interesting. It's good to know there are such options to use SELinux
on Debian :) It also says something that I'm inclined to interpret as
"the SELinux policy in Debian is not ready for prime-time". I'd be
glad to be wrong though!

Post by Neal Gompa

Post by intrigeri
* Writing, maintaining, auditing and debugging SELinux policy
requires grasping a complex conceptual model; I am told this is not
as easy as doing the same with AppArmor.

This is not really true. While it is true that the conceptual model is
more complex, the tooling for doing all the regular work with SELinux
is great. In many cases, the tools can analyze what's happened and
suggest a course of action about how to fix it. If it looks like a
bug, they suggest filing one with the vendor (in my case, when weird
things happen with the SELinux policy in Fedora, bugs get filed on
selinux-policy with the information from setroubleshoot so that things
can get fixed).

This sounds great UX; it makes me wish to try it out and draw
inspiration from it to improve AppArmor's UX too. Thanks for sharing.

Post by Neal Gompa
As for the complexity of making policies and policy modules, I've
written a few policy modules, and they're not that bad. You can make
some pretty simple policies if you don't want to expose any
administrative tunables. That said, even with the tunables, it's not
that bad.
For example, the container-selinux policy module is pretty easy to
understand: https://github.com/projectatomic/container-selinux
http://oss.tresys.com/docs/refpolicy/api/

I had a quick look and I agree: it's not that bad. Still feels much
scarier than AppArmor policy to me, but I'm clearly not the right
person to judge these days :)

Post by Neal Gompa

Post by intrigeri
* As far as I could understand when chatting with sysadmins of Red
Hat systems, this has resulted in a culture where many users got
used to disable SELinux entirely on their systems, instead of
trying to fix the buggy policy.

Back in the RHEL 5 days, this is definitely true. And if many of of
the Red Hat sysadmins you've talked to primarily maintain RHEL 5
systems (which is not unlikely), then it makes sense. Back in the RHEL
5 days (circa 2007), the tooling was very primitive, and for the most
part, the troubleshooting tools didn't exist.
Today in Fedora and RHEL 7, the tooling is very advanced, and in
almost every case where I've hit AVC denials in SELinux,
setroubleshoot has given me very useful information including
suggested course of actions to actually fix it locally.

OK, this does explain things. It's sad that this culture has been
created in the first place — changing users' habits is hard: the
sysadmins I'm talking about kept "disable SELinux" in their
post-installation checklist and have no clue that RHEL 7 solved all
these problems. I suspect they'll need many more years to realize they
could change their habits. I'll tell them about it!

Now, my point is not very relevant in the context of the Debian
discussion: hopefully not many Debian users are affected by this
"always disable SELinux" culture.

Post by Neal Gompa

Post by intrigeri
I've seen the opposite happen with
AppArmor, which is good: for example, pretty often bug reporters to
the Debian BTS document themselves how they could workaround the
problem locally *without* turning AppArmor off. Looking at open
bugs in the BTS against src:refpolicy, this seems to happen very
rarely for SELinux, so I wonder if it would be realistic to ship
Debian with SELinux enforced by default and have our community
support it.

I think this was not contested.

Post by Neal Gompa

Post by intrigeri
* https://wiki.debian.org/SELinux/Issues says "Graphical/Desktop
installs of Debian are not heavily tested with selinux, so you
might run into quite some issues".

This is true for both AppArmor and SELinux. With the exception of the
Snap case, neither MAC has been optimized for handling desktop issues
that much.

Right, they were not optimized for such use cases. But something very
close to Graphical/Desktop installs of Debian *are* heavily tested
with AppArmor thanks to Debian derivatives (Ubuntu, Tails) that enable
AppArmor. So I can't really agree with "This is true for both AppArmor
and SELinux".

Post by Neal Gompa

Post by intrigeri
* I'm not aware of any Debian derivative shipping with SELinux
enabled by default. If that's correct, then it means that we would
have to deal with quite some policy compatibility issues ourselves.

To be fair, refpolicy was RC'd from Debian in 8, due to failing to
build and no one fixed it quickly enough. It was reintroduced in
Debian 9, though. I have not personally tested the SELinux support in
Debian 9, but I've heard from a few friends that it does work.

Good to know :)

Post by Neal Gompa

Post by intrigeri
To me, the complexity of SELinux is a deal breaker: it seems that we
would need to get lots more expertise and energy to enforce SELinux by
default than doing the same with AppArmor.

The unfortunate thing is that more comprehensive security models do
lead to more complexity.

ACK. If we had a strong team of people dedicated to supporting SELinux
in Debian, e.g. as part of their paid job, it may be an option; my
understanding is that it's basically how distros that ship with
SELinux can handle it. But we have no such thing in Debian, and
AppArmor seems more suited to the more distributed model Debian is
based on: many developers will need to learn _a little bit_ about
AppArmor, and with the existing expertise we have in the project that
can provide advice & help when needed, we should be good without
having to hire ninjas (and FWIW Ubuntu handles it similarly).

Post by Neal Gompa
But I definitely don't want people to think that SELinux is some crazy
mountainous path full of terrible unknowns.

Thanks a lot for clarifying!

Best regards,

[Neal Gompa]

Post by intrigeri
Hi Neal & others,

Post by Neal Gompa
I was recently pointed to the thread going on debian-devel about
enabling AppArmor LSM in Debian, and as I read through your proposal,
I felt it should be warranted to point out a few things in regards to

Thanks a lot for your carefully worded and extremely well sourced
email! I've already learned quite a few interesting things.

Post by Neal Gompa

Post by intrigeri
Why AppArmor and not SELinux?
-----------------------------
SELinux is another LSM that tackles similar problems.

[...]

Post by Neal Gompa

Post by intrigeri
* Enabled by default in RHEL so in theory a great number of sysadmins
are at ease with it (see below why reality may not match this).

It's also important to note that it is also enabled by default in
Fedora, which is the upstream for RHEL.

Sure. I didn't mention it because I don't see this as very relevant in
the context of this discussion: it's a fact that many sysadmins active
in Debian have to use RHEL/CentOS at work, but I doubt many Debian
people are this much exposed to Fedora, so I don't think it's a good
source of pre-existing SELinux expertise in Debian.

Post by Neal Gompa
I do know of users of SELinux in Debian and Ubuntu, though they often
fork from refpolicy or fedora-selinux the bits they want to use and
install it on top of the current refpolicy offered in Debian.

Interesting. It's good to know there are such options to use SELinux
on Debian :) It also says something that I'm inclined to interpret as
"the SELinux policy in Debian is not ready for prime-time". I'd be
glad to be wrong though!

I'm not sure that's actually the case. I can't really speak for it, as
I generally don't use Debian (I primarily use Fedora, openSUSE,
Mageia, and CentOS). One thing I have observed is that there are no
guidelines or policy documentation from Debian on how to install
policy modules. That's a very annoying gap for anyone who wants to
leverage the modular nature of SELinux policies.

Many, many, many common services and applications ship SELinux policy
modules, and they are not packaged in Debian because no one is sure
how to do it.

I've more or less fallen back to telling people to use the Makefile to
build and install the module and hope that Debian does The Right
Thing(TM). But of course, I don't know if this is true.

Post by intrigeri

Post by Neal Gompa

Post by intrigeri
* Writing, maintaining, auditing and debugging SELinux policy
requires grasping a complex conceptual model; I am told this is not
as easy as doing the same with AppArmor.

This is not really true. While it is true that the conceptual model is
more complex, the tooling for doing all the regular work with SELinux
is great. In many cases, the tools can analyze what's happened and
suggest a course of action about how to fix it. If it looks like a
bug, they suggest filing one with the vendor (in my case, when weird
things happen with the SELinux policy in Fedora, bugs get filed on
selinux-policy with the information from setroubleshoot so that things
can get fixed).

This sounds great UX; it makes me wish to try it out and draw
inspiration from it to improve AppArmor's UX too. Thanks for sharing.

Post by Neal Gompa
As for the complexity of making policies and policy modules, I've
written a few policy modules, and they're not that bad. You can make
some pretty simple policies if you don't want to expose any
administrative tunables. That said, even with the tunables, it's not
that bad.
For example, the container-selinux policy module is pretty easy to
understand: https://github.com/projectatomic/container-selinux
http://oss.tresys.com/docs/refpolicy/api/

I had a quick look and I agree: it's not that bad. Still feels much
scarier than AppArmor policy to me, but I'm clearly not the right
person to judge these days :)

Post by Neal Gompa

Post by intrigeri
* As far as I could understand when chatting with sysadmins of Red
Hat systems, this has resulted in a culture where many users got
used to disable SELinux entirely on their systems, instead of
trying to fix the buggy policy.

Back in the RHEL 5 days, this is definitely true. And if many of of
the Red Hat sysadmins you've talked to primarily maintain RHEL 5
systems (which is not unlikely), then it makes sense. Back in the RHEL
5 days (circa 2007), the tooling was very primitive, and for the most
part, the troubleshooting tools didn't exist.
Today in Fedora and RHEL 7, the tooling is very advanced, and in
almost every case where I've hit AVC denials in SELinux,
setroubleshoot has given me very useful information including
suggested course of actions to actually fix it locally.

OK, this does explain things. It's sad that this culture has been
created in the first place — changing users' habits is hard: the
sysadmins I'm talking about kept "disable SELinux" in their
post-installation checklist and have no clue that RHEL 7 solved all
these problems. I suspect they'll need many more years to realize they
could change their habits. I'll tell them about it!
Now, my point is not very relevant in the context of the Debian
discussion: hopefully not many Debian users are affected by this
"always disable SELinux" culture.

Post by Neal Gompa

Post by intrigeri
I've seen the opposite happen with
AppArmor, which is good: for example, pretty often bug reporters to
the Debian BTS document themselves how they could workaround the
problem locally *without* turning AppArmor off. Looking at open
bugs in the BTS against src:refpolicy, this seems to happen very
rarely for SELinux, so I wonder if it would be realistic to ship
Debian with SELinux enforced by default and have our community
support it.

I think this was not contested.

I can't really say anything about this because the Debian BTS UX is so
awful that I can't ever figure out how to figure out anything in it.
That's a completely different topic, though. :P

Post by intrigeri

Post by Neal Gompa

Post by intrigeri
* https://wiki.debian.org/SELinux/Issues says "Graphical/Desktop
installs of Debian are not heavily tested with selinux, so you
might run into quite some issues".

This is true for both AppArmor and SELinux. With the exception of the
Snap case, neither MAC has been optimized for handling desktop issues
that much.

Right, they were not optimized for such use cases. But something very
close to Graphical/Desktop installs of Debian *are* heavily tested
with AppArmor thanks to Debian derivatives (Ubuntu, Tails) that enable
AppArmor. So I can't really agree with "This is true for both AppArmor
and SELinux".

Post by Neal Gompa

Post by intrigeri
* I'm not aware of any Debian derivative shipping with SELinux
enabled by default. If that's correct, then it means that we would
have to deal with quite some policy compatibility issues ourselves.

To be fair, refpolicy was RC'd from Debian in 8, due to failing to
build and no one fixed it quickly enough. It was reintroduced in
Debian 9, though. I have not personally tested the SELinux support in
Debian 9, but I've heard from a few friends that it does work.

Good to know :)

Post by Neal Gompa

Post by intrigeri
To me, the complexity of SELinux is a deal breaker: it seems that we
would need to get lots more expertise and energy to enforce SELinux by
default than doing the same with AppArmor.

The unfortunate thing is that more comprehensive security models do
lead to more complexity.

ACK. If we had a strong team of people dedicated to supporting SELinux
in Debian, e.g. as part of their paid job, it may be an option; my
understanding is that it's basically how distros that ship with
SELinux can handle it. But we have no such thing in Debian, and
AppArmor seems more suited to the more distributed model Debian is
based on: many developers will need to learn _a little bit_ about
AppArmor, and with the existing expertise we have in the project that
can provide advice & help when needed, we should be good without
having to hire ninjas (and FWIW Ubuntu handles it similarly).

I don't think it is quite so bad. Several other distribution families
have Hardened variants that leverage the work in Fedora. For example,
Gentoo and Arch Hardened leverage our work in Fedora and they work
with us in developing and improving stuff for everyone. Generally
speaking, I don't think anyone in the Fedora SELinux project is
opposed to Debian folks coming in and contributing to improve things
for Debian. It's certainly already happening with Arch and Gentoo
folks working with us in Fedora from time to time.

The SUSE SELinux policy is derived from ours in Fedora, for example.

Just because we're on different sides of the divide doesn't mean we
can't come together and do great things. :)

Post by intrigeri

Post by Neal Gompa
But I definitely don't want people to think that SELinux is some crazy
mountainous path full of terrible unknowns.

Thanks a lot for clarifying!

It's my pleasure!

[intrigeri]

Hi,

Post by intrigeri
tl;dr: I hereby propose we enable AppArmor by default in testing/sid,
and decide one year later if we want to keep it this way in the
Buster release.

Thanks a lot to everyone who participated on this thread, tried
AppArmor and reported bugs, or expressed support/interest privately!

Summary of the discussion: no strong objection was raised; quite a few
potential issues were mentioned; the most serious ones were either
resolved already, or in good way to be resolved in the next 2 weeks.
So, my understanding is that we have a broad consensus and can start
the proposed experiment.

I need advice from you folks on one specific matter, see below.

Post by intrigeri
1. Enable AppArmor by default in testing/sid as soon as feasible in
the Buster cycle.
I can think of several possible ways to do it but for now I'd
rather focus on the "do we want to do it at all" conversation.

It's now time to discuss the "how" aspect.

Enabling AppArmor by default requires two changes:

1. enabling the LSM in Linux: problem solved, Ben Hutchings agreed
we should do this in src:linux, at least for the time being;

2. installing the apparmor package by default: it ships the bits of
code that load AppArmor policy during boot and some shared bits of
policy that most other AppArmor profiles rely upon.

This email is about (2). There are two aspects to it.

For new installations, it seems that making the apparmor package
"Priority: standard" is the way to go. I've asked debian-boot@'s
opinion about it [priority:standard?] but the rest of our developers
community is of course welcome to comment as well.

For upgrades it seems much more complicated. Ideally I would like the
apparmor package to be installed automatically:

- on testing/sid upgrades, during the Buster dev cycle: this would
greatly increase the value of the "enable AppArmor by default for
a while" experiment as we would get lots more data to reason about
when the time comes;

- during Stretch to Buster upgrades: this seems necessary so every
user gets the AppArmor benefits, regardless of when they installed
their system initially.

I also want to provide easy means for users to opt-out from
the experiment.

I've requested advice on this topic from a few fellow Debian people
and the conclusion seems to be:

- I was told essentially "we generally don't do that in Debian" by
a few people who suggested me asking this mailing list.

I don't understand the rationale though — during system upgrades we
do change the distro behavior in many ways: we add new features, we
enable new security measures, we switch init systems, we switch
from MySQL to MariaDB and all sort of things — so it's not obvious
to me why doing the same to enable a security system like AppArmor
would be a Bad Thing™.

Is the concern specifically about doing so by pulling a new
package in?

Or is it specifically about enabling a LSM that was previously
disabled? (Any such big change brings a risk of introducing
regressions, so the underlying questions seem to be "is the risk
worth it? is the risk well managed?")

- We have no better option to achieve that than having another
package, that's already installed by default, add a "Recommends:
apparmor". This feels artificial and rather ugly, but it might be
the only option. I don't know which other package would be the most
suitable to add this dependency. Any suggestion? Any other idea?

I'd love to read your thoughts about this. Let's discuss it.

[priority:standard?] https://bugs.debian.org/879590#25

Cheers,

[Anthony DeRobertis]

I think the only two ways to get a new package installed upon stretch →
buster are:

1. Suggest the admin do it in the release notes. (It should be
documented in the release notes no matter which option we pick, of course.)

2. Suggest the admin do it in a NEWS.Debian entry (but it needs to be an
upgraded package, not a new one, else it won't be displayed. So the
linux-image-4.* packages won't work, but e.g., linux-image-amd64 would).

3. Have a Recommends or Depends on it from another package that is
installed. (Presumably that'd be a Recommends from the linux-image-*
packages, and would be dropped down to a Suggests for buster+1).

4. Suggest the admin do it in a debconf note. Highly discouraged nowadays.

All of those except #1 also work for folks following testing or unstable.

Personally, I don't have a preference between #1 and #3, as long as we
find some reasonable way to opt-out if we go with #3 (and document it in
the release notes).

Post by intrigeri
Hi,

Post by intrigeri
tl;dr: I hereby propose we enable AppArmor by default in testing/sid,
and decide one year later if we want to keep it this way in the
Buster release.

Thanks a lot to everyone who participated on this thread, tried
AppArmor and reported bugs, or expressed support/interest privately!
Summary of the discussion: no strong objection was raised; quite a few
potential issues were mentioned; the most serious ones were either
resolved already, or in good way to be resolved in the next 2 weeks.
So, my understanding is that we have a broad consensus and can start
the proposed experiment.
I need advice from you folks on one specific matter, see below.

Post by intrigeri
1. Enable AppArmor by default in testing/sid as soon as feasible in
the Buster cycle.
I can think of several possible ways to do it but for now I'd
rather focus on the "do we want to do it at all" conversation.

It's now time to discuss the "how" aspect.
1. enabling the LSM in Linux: problem solved, Ben Hutchings agreed
we should do this in src:linux, at least for the time being;
2. installing the apparmor package by default: it ships the bits of
code that load AppArmor policy during boot and some shared bits of
policy that most other AppArmor profiles rely upon.
This email is about (2). There are two aspects to it.
For new installations, it seems that making the apparmor package
opinion about it [priority:standard?] but the rest of our developers
community is of course welcome to comment as well.
For upgrades it seems much more complicated. Ideally I would like the
- on testing/sid upgrades, during the Buster dev cycle: this would
greatly increase the value of the "enable AppArmor by default for
a while" experiment as we would get lots more data to reason about
when the time comes;
- during Stretch to Buster upgrades: this seems necessary so every
user gets the AppArmor benefits, regardless of when they installed
their system initially.
I also want to provide easy means for users to opt-out from
the experiment.
I've requested advice on this topic from a few fellow Debian people
- I was told essentially "we generally don't do that in Debian" by
a few people who suggested me asking this mailing list.
I don't understand the rationale though — during system upgrades we
do change the distro behavior in many ways: we add new features, we
enable new security measures, we switch init systems, we switch
from MySQL to MariaDB and all sort of things — so it's not obvious
to me why doing the same to enable a security system like AppArmor
would be a Bad Thing™.
Is the concern specifically about doing so by pulling a new
package in?
Or is it specifically about enabling a LSM that was previously
disabled? (Any such big change brings a risk of introducing
regressions, so the underlying questions seem to be "is the risk
worth it? is the risk well managed?")
- We have no better option to achieve that than having another
apparmor". This feels artificial and rather ugly, but it might be
the only option. I don't know which other package would be the most
suitable to add this dependency. Any suggestion? Any other idea?
I'd love to read your thoughts about this. Let's discuss it.
[priority:standard?] https://bugs.debian.org/879590#25
Cheers,

[Mathieu Parent]

Hi,

Post by Anthony DeRobertis
I think the only two ways to get a new package installed upon stretch →
1. Suggest the admin do it in the release notes. (It should be documented in
the release notes no matter which option we pick, of course.)
2. Suggest the admin do it in a NEWS.Debian entry (but it needs to be an
upgraded package, not a new one, else it won't be displayed. So the
linux-image-4.* packages won't work, but e.g., linux-image-amd64 would).
3. Have a Recommends or Depends on it from another package that is
installed. (Presumably that'd be a Recommends from the linux-image-*
packages, and would be dropped down to a Suggests for buster+1).
4. Suggest the admin do it in a debconf note. Highly discouraged nowadays.

Could'nt we:

5. Make linux-image-$abi-$arch Depends on apparmor | selinux-basics |
tomoyo-tools | linux-no-lsm

With linux-no-lsm being a new empty package, and all of apparmor,
selinux-basics, tomoyo-tools enable the corresponding LSM.

Regards

[intrigeri]

Post by Mathieu Parent
5. Make linux-image-$abi-$arch Depends on apparmor | selinux-basics |
tomoyo-tools | linux-no-lsm
With linux-no-lsm being a new empty package, and all of apparmor,
selinux-basics, tomoyo-tools enable the corresponding LSM.

This would be ideal on the long term and a lot of thought has been put
into it (#702030), but it requires quite some work in various places
and AFAIK nobody looked into how it could work for non-GRUB
bootloaders; I suspect some of the major bootloaders we support don't
offer the same flexibility as GRUB wrt. random packages injecting
parameters on the kernel command line.

As Ben Hutchings wrote on https://bugs.debian.org/879590#10, "We
really should have a common way to append things to the kernel command
line […] but this shouldn't have to wait for that" with which
I couldn't agree more.

Thankfully we already have another, cheap solution to address the "how
to enable the AppArmor LSM in the kernel" problem :) So now I'd rather
focus on the other, remaining problem, i.e. "how to pull in the
AppArmor policy + userspace tools".

Cheers,

[Anthony DeRobertis]

Post by Mathieu Parent
5. Make linux-image-$abi-$arch Depends on apparmor | selinux-basics |
tomoyo-tools | linux-no-lsm
With linux-no-lsm being a new empty package, and all of apparmor,
selinux-basics, tomoyo-tools enable the corresponding LSM.

I don't think there is a good way to guarantee which alternative there
apt picks. It could pick to install linux-no-lsm for example (and who
knows, maybe due to fewer dependencies, or a conflict, or whatever, it
will). Even if it works today, that seems fragile...

Also, a Depends: with one of the alternatives being "don't install
anything" strikes me as a hack to work around not having Recommends or
Suggests — but we do have those, so I'm not sure why we wouldn't use
them instead.

[Jeremy Bicha]

On Thu, Oct 26, 2017 at 11:29 PM, Anthony DeRobertis

Post by Anthony DeRobertis
3. Have a Recommends or Depends on it from another package that is
installed. (Presumably that'd be a Recommends from the linux-image-*
packages, and would be dropped down to a Suggests for buster+1).

Why shouldn't it stay a Recommends for Bullseye (buster+1) too?

Thanks,
Jeremy Bicha

[Holger Levsen]

Post by Jeremy Bicha

Post by Anthony DeRobertis
3. Have a Recommends or Depends on it from another package that is
installed. (Presumably that'd be a Recommends from the linux-image-*
packages, and would be dropped down to a Suggests for buster+1).

Why shouldn't it stay a Recommends for Bullseye (buster+1) too?

recommends wont work, they arent installed on upgrades


[Anthony DeRobertis]

recommends wont work, they arent installed on upgrades…

I haven't tested it, but at least according to apt's changelog new
recommends are installed on upgrade as of 0.7.0 as log as
APT::Install-Recommends is true, which has been the default since
0.7.17. So unless the changelog lies, recommends have been installed on
upgrade since 05 Nov 2008.

[Holger Levsen]

Post by Anthony DeRobertis

Post by Holger Levsen
recommends wont work, they arent installed on upgrades


I haven't tested it, but at least according to apt's changelog new
recommends are installed on upgrade as of 0.7.0 as log as
APT::Install-Recommends is true, which has been the default since
0.7.17. So unless the changelog lies, recommends have been installed on
upgrade since 05 Nov 2008.

oh wow, I happily stand corrected then.

[Anthony DeRobertis]

Post by Jeremy Bicha
On Thu, Oct 26, 2017 at 11:29 PM, Anthony DeRobertis

Post by Anthony DeRobertis
3. Have a Recommends or Depends on it from another package that is
installed. (Presumably that'd be a Recommends from the linux-image-*
packages, and would be dropped down to a Suggests for buster+1).

Why shouldn't it stay a Recommends for Bullseye (buster+1) too?

Because for Bullseye, it'll already be pulled in (during the Buster
upgrade) or during the install (as both the Buster and Bullseye
installers will install it). So there is no longer any reason to inflate
the importance — the kernel runs just fine w/o and doesn't lose any
major functionality.

Normally it'd only be a Suggests (if anything), but Suggests won't pull
the package in on upgrade.

[Jeremy Bicha]

On Fri, Oct 27, 2017 at 11:06 AM, Anthony DeRobertis

Post by Anthony DeRobertis
the kernel runs just fine w/o and doesn't lose any
major functionality.

I think the whole point of this thread is that AppArmor is major
functionality that we want in default Debian systems. Therefore,
demoting it to Suggests in Bullseye seems counter-productive.

Thanks,
Jeremy Bicha

[intrigeri]

Hi,

[...]
3. Have a Recommends or Depends on it from another package that is installed.
(Presumably that'd be a Recommends from the linux-image-*
packages, […]

The next upload of the linux-image packages will "Recommends: apparmor".

Cheers,

[intrigeri]

Hi,

Post by intrigeri
The next upload of the linux-image packages will "Recommends: apparmor".

Done ⇒ AppArmor is now enabled by default in sid.
Let the experiment begin!

Now is time to report and fix bugs. To make sure they are on the radar
of the AppArmor team, please apply the relevant usertag:

https://wiki.debian.org/AppArmor/Reportbug

Thanks in advance, and sorry for any inconvenience it may cause (e.g.
the AppArmor policy for Thunderbird has various issues in sid; all of
those I'm aware of are fixed in experimental already).

Cheers,

[Ben Caradoc-Davies]

Post by intrigeri
Thanks in advance, and sorry for any inconvenience it may cause (e.g.
the AppArmor policy for Thunderbird has various issues in sid; all of
those I'm aware of are fixed in experimental already).

Where "various issues" means no thunderbird external helpers work under
xfce. Not a single one, as far as I can tell. And there goes another
one: what happened to my .signature? I have filed as many bugs as I can
given the time available. I will file one more for the missing
.signature, and then I am disabling apparmor.

aa-complain thunderbird does not work (see #882047), so I used
"systemctl mask apparmor.service" followed by a reboot. Ah, that is much
better (#882048):

[intrigeri]

Hi,

Post by intrigeri
Thanks in advance, and sorry for any inconvenience it may cause (e.g.
the AppArmor policy for Thunderbird has various issues in sid; all of
those I'm aware of are fixed in experimental already).

Where "various issues" means no thunderbird external helpers work under xfce.
Not a single one, as far as I can tell. And there goes another one: what happened
to my .signature? I have filed as many bugs as I can given the time available.
I will file one more for the missing .signature, and then I am disabling apparmor.

Thanks.

The last 9 days have taught us that it's too hard to confine
Thunderbird with AppArmor meaningfully without breaking at least some
common use cases. We've fixed a bunch of bugs but some use cases
really can't be fixed without making the AppArmor policy essentially
useless. Not a big surprise, it was probably worth trying, but let's
be realistic and move on: Thunderbird will ship its AppArmor profile
disabled by default soon (#882672); users may choose to opt-in.

Cheers,

[John Johansen]

Post by intrigeri
Thanks in advance, and sorry for any inconvenience it may cause (e.g.
the AppArmor policy for Thunderbird has various issues in sid; all of
those I'm aware of are fixed in experimental already).

Where "various issues" means no thunderbird external helpers work under xfce. Not a single one, as far as I can tell. And there goes another one: what happened to my .signature? I have filed as many bugs as I can given the time available. I will file one more for the missing .signature, and then I am disabling apparmor.

thank you for taking time to file bugs and provide a report here to help make the apparmor experience better. You have several options for disabling parts of apparmor policy enforcement or its enforcement entirely.

You can disable individual profiles without editing them and messing up the packaging by using aa-disable

sudo aa-disable /etc/apparmor.d/usr.bin.thunderbird

or by manually by manually removing the profile and dropping a symlink in

/etc/apparmor.d/disable/

so for example to disable the thunderbird profile you can do
sudo apparmor_parser -R /etc/apparmor.d/usr.bin.thunderbird
sudo ln -s /etc/apparmor.d/usr.bin.thunderbird /etc/apparmor.d/disbale/thunderbird

it is important to do the removal before adding the symlink, and as in the example above the symlink does not have to be the same name as that of the profile file.
you can reverse the above by using
sudo aa-enable /etc/apparmor.d/usr.bin.thunderbird

or manually by removing the symlink and loading the profile
sudo rm /etc/apparmor.d/disable/thunderbird
audo apparmor_parser -r /etc/apparmor.d/usr.bin.thunderbird


You can disable the apparmor service at the systemd level with

sudo systemctl disable apparmor

You can remove the apparmor package

sudo apt-get remove apparmor
or
sudo dpkg --remove apparmor

and you can also set the kernel boot parameter
apparmor=0

to disable apparmor on a particular boot, or set it as part of your grub config to permanently disable it without touching the packaging


* for the above examples I have used /etc/apparmor.d/ for the profile location but it could be configured to other locations like /var/lib/apparmor/ etc, it depends on the distro and sometimes the package eg. ubuntu has profiles configured to different locations depending on whether they are system profiles, snap profiles, etc.

[Marvin Renich]

Post by John Johansen
You can disable individual profiles without editing them and messing up the packaging by using aa-disable

[some really good beginner stuff snipped]

John, many thanks for these tidbits. Can they be put in a text file in
/usr/share/doc/apparmor, with a NEWS.Debian entry pointing to it, so
that when the package is pulled in, the user has some idea where to
start? Since Thunderbird seems to be one of the problem packages,
having it in a text file on the local system seems like a good idea.

Actually, a short beginner's guide as a text file in
/usr/share/doc/apparmor, which has more than just "how to disable a
profile" would be extremely helpful. I don't have the apparmor
knowledge to write it, though.

Thanks...Marvin

[John Johansen]

Post by Marvin Renich

Post by John Johansen
You can disable individual profiles without editing them and messing up the packaging by using aa-disable

[some really good beginner stuff snipped]
John, many thanks for these tidbits. Can they be put in a text file in
/usr/share/doc/apparmor, with a NEWS.Debian entry pointing to it, so
that when the package is pulled in, the user has some idea where to
start? Since Thunderbird seems to be one of the problem packages,
having it in a text file on the local system seems like a good idea.

yes we can certainly create the text file, its a good idea. I'll leave it
up to the debian maintainer to decide on the NEWS.Debian entry but it
certainly sound like a good idea to me as well.

Post by Marvin Renich
Actually, a short beginner's guide as a text file in
/usr/share/doc/apparmor, which has more than just "how to disable a
profile" would be extremely helpful. I don't have the apparmor
knowledge to write it, though.

yeah, I will start working on the doc. Make sure it has links to
more comprehensive info (the wiki, ml, some man pages, etc.)

[Wouter Verhelst]

Post by John Johansen

Post by Marvin Renich

Post by John Johansen
You can disable individual profiles without editing them and messing up the packaging by using aa-disable

[some really good beginner stuff snipped]
John, many thanks for these tidbits. Can they be put in a text file in
/usr/share/doc/apparmor, with a NEWS.Debian entry pointing to it, so
that when the package is pulled in, the user has some idea where to
start? Since Thunderbird seems to be one of the problem packages,
having it in a text file on the local system seems like a good idea.

yes we can certainly create the text file, its a good idea. I'll leave it
up to the debian maintainer to decide on the NEWS.Debian entry but it
certainly sound like a good idea to me as well.

Post by Marvin Renich
Actually, a short beginner's guide as a text file in
/usr/share/doc/apparmor, which has more than just "how to disable a
profile" would be extremely helpful. I don't have the apparmor
knowledge to write it, though.

yeah, I will start working on the doc. Make sure it has links to
more comprehensive info (the wiki, ml, some man pages, etc.)

Thanks for doing that.

It would be awesome if you could also include some documentation in the
style "I'm a Debian package maintainer and the apparmor profile for some
of the binaries in one of my packages is buggy, how can I fix it?" or
"I'm a Debian package maintainer and I'd like to write an apparmor
profile for one of the binaries in my package, where do I start".

IMO, if apparmor is enabled by default, then making sure it works well
with particular programs is the responsibility of the maintainer of the
program in question.

Thanks again,

[intrigeri]

Post by Wouter Verhelst
It would be awesome if you could also include some documentation in the
style "I'm a Debian package maintainer and the apparmor profile for some
of the binaries in one of my packages is buggy, how can I fix it?"
or "I'm a Debian package maintainer and I'd like to write an
apparmor profile for one of the binaries in my package, where do
I start".

Some of this doc has been written by Ulrike Uhlig a few years ago:
https://wiki.debian.org/AppArmor/Contribute#Ship_an_AppArmor_profile_in_.22your.22_package

Cheers,

[Wouter Verhelst]

Post by intrigeri

Post by Wouter Verhelst
It would be awesome if you could also include some documentation in the
style "I'm a Debian package maintainer and the apparmor profile for some
of the binaries in one of my packages is buggy, how can I fix it?"
or "I'm a Debian package maintainer and I'd like to write an
apparmor profile for one of the binaries in my package, where do
I start".

https://wiki.debian.org/AppArmor/Contribute#Ship_an_AppArmor_profile_in_.22your.22_package

Oh, great, didn't know that. Thanks.

(minor nitpick would be that perhaps it might make sense to link to that
page from somewhere inside /usr/share/doc or thereabouts, but that's
just a detail)

[Marvin Renich]

[added d-dev back]

Post by Marvin Renich
Actually, a short beginner's guide as a text file in
/usr/share/doc/apparmor, which has more than just "how to disable a
profile" would be extremely helpful. I don't have the apparmor
knowledge to write it, though.

FYI the most useful bits were added to
https://wiki.debian.org/AppArmor/HowToUse
which is linked from /usr/share/doc/apparmor/README.Debian :)
It's only a start and there's lots of room for improvement,
but it's a start.

Thanks for this pointer!

Adding these two links [1], [2] on that page might be helpful. I found
them by following links to [3].

As a side note, my laptop runs testing, and I allowed apparmor to be
enabled when that change hit testing. The only issue I have noticed so
far is that smbd would not have access to some (intentionally public,
not in /home) shares if it were in enforce mode, rather than complain
mode. If I were not aware of apparmor, and if smbd were in enforce
mode, I would have had a difficult time tracking this down.

Is there a way that an app (e.g. smbd) whose file access requirements
change dynamically through admin and user configuration can at least
inspect its own apparmor profile and give the user a clue that the admin
must update the profile? For Samba, perhaps at least a comment in
/etc/samba/smb.conf at "Share Definitions" giving a reminder that if any
LSM is enabled, the LSM config may need to be updated to reflect changes
to shares.

(Samba maintainers added to CC; please remove them for replies not
pertaining to samba.)

...Marvin

[1] Creating and modifying AppArmor policy with the tools
https://gitlab.com/apparmor/apparmor/wikis/Profiling_with_tools
[2] Creating and modifying AppArmor policy by hand
https://gitlab.com/apparmor/apparmor/wikis/Profiling_by_hand
[3] https://gitlab.com/apparmor/apparmor/wikis/Documentation

[Mathieu Parent]

Hi,

Samba maintainer here ...

2018-03-19 15:10 GMT+01:00 Marvin Renich <***@renich.org>:
[...]

Post by Marvin Renich
As a side note, my laptop runs testing, and I allowed apparmor to be
enabled when that change hit testing. The only issue I have noticed so
far is that smbd would not have access to some (intentionally public,
not in /home) shares if it were in enforce mode, rather than complain
mode. If I were not aware of apparmor, and if smbd were in enforce
mode, I would have had a difficult time tracking this down.
Is there a way that an app (e.g. smbd) whose file access requirements
change dynamically through admin and user configuration can at least
inspect its own apparmor profile and give the user a clue that the admin
must update the profile? For Samba, perhaps at least a comment in
/etc/samba/smb.conf at "Share Definitions" giving a reminder that if any
LSM is enabled, the LSM config may need to be updated to reflect changes
to shares.

I'm balanced about this as AppArmor logs denied access.

Merge request [1] welcome, either for debian/smb.conf or debian/README.Debian.

[1] https://salsa.debian.org/samba-team/samba/merge_requests


Regards

[Seth Arnold]

Post by Marvin Renich
Is there a way that an app (e.g. smbd) whose file access requirements
change dynamically through admin and user configuration can at least
inspect its own apparmor profile and give the user a clue that the admin
must update the profile?

Our friends at SUSE have a script that automatically generates portions of
an AppArmor profile for Samba based on the Samba configuration:
https://bugzilla.novell.com/show_bug.cgi?id=688040

I'm not entirely sold on the idea, as a hand-authored security policy
can serve as belt-and-suspenders against misconfiguration or a broken
management system that allows unauthenticated users to create too-wide
shares.

The usability gain is undeniable.

Thanks

[Christian Boltz]

Hello,

Post by Seth Arnold

Post by Marvin Renich
Is there a way that an app (e.g. smbd) whose file access
requirements
change dynamically through admin and user configuration can at least
inspect its own apparmor profile and give the user a clue that the
admin must update the profile?

Our friends at SUSE have a script that automatically generates
portions of an AppArmor profile for Samba based on the Samba
configuration: https://bugzilla.novell.com/show_bug.cgi?id=688040
I'm not entirely sold on the idea, as a hand-authored security policy
can serve as belt-and-suspenders against misconfiguration or a broken
management system that allows unauthenticated users to create too-wide
shares.
The usability gain is undeniable.

As the author of that script, I can tell you that it made *lots of*
users happy ;-) Before we had that script, we[1] got a bugreport each month
about AppArmor denials in Samba because of shares outside of /home.
Since the script is in use, that number went down to zero :-)

Yes, there is a risk that a samba misconfiguration results in too wide
permissions, but the script has a few safety checks and won't auto-add
- paths with variables (anything containing a % sign)
- "/" - because sharing your complete filesystem is insane
to reduce that risk.

The big advantage of the script is that we can ship the samba profile
in enforce mode without annoying users ;-) - and that's much better
than having to disable the profile by default because it breaks Samba
with non-default configuration/shares.
Oh, and the smb profile helped to prevent exploiting SambaCry :-)

I'll attach the latest version of the script to this mail. [2]

You'll need to call it in smb.service as:
ExecStartPre=/usr/share/samba/update-apparmor-samba-profile

You'll also need to apply
https://build.opensuse.org/package/view_file/openSUSE:Factory/apparmor/apparmor-samba-include-permissions-for-shares.diff?expand=1
to the smb AppArmor profile to include the autogenerated sniplet. [3]


Regards,

Christian Boltz

[1] Just in case it isn't obvious on Debian mailinglists - "we" means
"openSUSE" ;-)

[2] directly taken from the package:
https://build.opensuse.org/package/show/openSUSE:Factory/samba
(it's in the vendor-files-*.tar.bz2 tarball)

[3] Actually it should now be possible to push this patch upstream
using "#include if exists" ;-)

[Philipp Kern]

Post by intrigeri
What's the cost for package maintainers?
----------------------------------------
For most of them: none at all. As said earlier, our AppArmor policy
does not cover that much software yet.

So how will bug reports work? For instance I turned it on and now I see
a bunch of warnings[1] from Thunderbird and a bunch of actual failures
when opening links (which is completely broken), because Thunderbird
cannot exec google-chrome-beta. What about integration issues where a
browser should be able to register itself as a browser and hence be
available from applications that try to open links?

Right now thunderbird's profile is owned by thunderbird. Is
thunderbird's maintainer expected to deal with all of these issues?
Should there be some kind of tool where the apparmor team could
aggregate the updates? (I.e. routinely review denies?)

Kind regards
Philipp Kern

[1] e.g.
[ 3459.624852] audit: type=1400 audit(1509283082.571:59):
apparmor="DENIED" operation="file_inherit" profile="thunderbird//gpg"
name="/usr/share/thunderbird/omni.ja" pid=24720 comm="gpg2"
requested_mask="r" denied_mask="r" fsuid=1000 ouid=0

[2] e.g.
[ 3795.153239] audit: type=1400 audit(1509283418.100:64):
apparmor="DENIED" operation="exec" profile="thunderbird"
name="/opt/google/chrome-beta/google-chrome-beta" pid=31896
comm="thunderbird" requested_mask="x" denied_mask="x" fsuid=1000 ouid=0

[Carsten Schoenert]

Hello Philip,

Post by Philipp Kern

Post by intrigeri
What's the cost for package maintainers?
----------------------------------------
For most of them: none at all. As said earlier, our AppArmor policy
does not cover that much software yet.

So how will bug reports work? For instance I turned it on and now I see
a bunch of warnings[1] from Thunderbird and a bunch of actual failures
when opening links (which is completely broken), because Thunderbird
cannot exec google-chrome-beta. What about integration issues where a
browser should be able to register itself as a browser and hence be
available from applications that try to open links?
Right now thunderbird's profile is owned by thunderbird. Is
thunderbird's maintainer expected to deal with all of these issues?
Should there be some kind of tool where the apparmor team could
aggregate the updates? (I.e. routinely review denies?)

in the near past I've forwarded bug reports about apparmor suggestions
and issues to Simon Diezel (CC'd) and also to intrigeri. This works
quite well now due a good responsive behavior of both and I'm really
thankful for this!

Right after the beginning of the apparmor profile for
Icedove/Thunderbird I was a bit skeptic if the shipping of the profile
within the ID/TB packaging will work and is maintainable as I haven't
use apparmor ever before and due this have quite zero experience with
that. I got the impression that the profile would be better under the
hood of the apparmor team as there is much more knowledge about the
working model.
Starting with this thread and by some talking to various people I
changed my mind about this. For better flexibility and customizing,
thinking about various releases (like *-security, *-backports e.g.) that
need to be supported, I believe apparmor profiles for the applications
should stay in the belonging source packages in most cases.
Ubuntu is doing the opposite as far as I know [1], the time will show
which way is batter.

But yes, the maintainers of such packages need the help of the apparmor
folks and also vice versa.

For Thunderbird intrigeri and myself came to the conclusion that
especially for the apparmor profile someone from the apparmor team
should be able to contribute changes to the profile directly to the git
tree. So intrigeri has become a member of the pkg-mozilla group to be
able to push changes by himself. I trust intrigeri enough that he will
do good contributions. For now it's the best we can do. This at all is
for sure improvable and we should talk about this on upcoming Debian
events or directly via email.

...

Post by Philipp Kern
[1] e.g.
apparmor="DENIED" operation="file_inherit" profile="thunderbird//gpg"
name="/usr/share/thunderbird/omni.ja" pid=24720 comm="gpg2"
requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
[2] e.g.
apparmor="DENIED" operation="exec" profile="thunderbird"
name="/opt/google/chrome-beta/google-chrome-beta" pid=31896
comm="thunderbird" requested_mask="x" denied_mask="x" fsuid=1000 ouid=0

I suggest to open a bug report for each of such issues against
thunderbird with a description what was done and what was expected.

[1] https://git.launchpad.net/apparmor-profiles/tree/ubuntu/17.10

[Philipp Kern]

Hi Carsten,

thanks for your reply!

Post by Carsten Schoenert
For Thunderbird intrigeri and myself came to the conclusion that
especially for the apparmor profile someone from the apparmor team
should be able to contribute changes to the profile directly to the git
tree. So intrigeri has become a member of the pkg-mozilla group to be
able to push changes by himself. I trust intrigeri enough that he will
do good contributions. For now it's the best we can do. This at all is
for sure improvable and we should talk about this on upcoming Debian
events or directly via email.

Okay, filed the bugs, lets see where they go. :) I was especially
concerned about the browser part.

Post by Carsten Schoenert
...

Post by Philipp Kern
[1] e.g.
apparmor="DENIED" operation="file_inherit" profile="thunderbird//gpg"
name="/usr/share/thunderbird/omni.ja" pid=24720 comm="gpg2"
requested_mask="r" denied_mask="r" fsuid=1000 ouid=0

Filed as #880425[1].

Post by Carsten Schoenert

Post by Philipp Kern
[2] e.g.
apparmor="DENIED" operation="exec" profile="thunderbird"
name="/opt/google/chrome-beta/google-chrome-beta" pid=31896
comm="thunderbird" requested_mask="x" denied_mask="x" fsuid=1000 ouid=0

Filed as #880424[0]. I think there is a deeper question here as to how
to handle the browser abstraction for AppArmor in general.

Post by Carsten Schoenert
I suggest to open a bug report for each of such issues against
thunderbird with a description what was done and what was expected.

As above. :)

Kind regards and thanks
Philipp Kern

[0] https://bugs.debian.org/880424
[1] https://bugs.debian.org/880425

[Guido Günther]

Hi,

Post by Philipp Kern
Hi Carsten,
thanks for your reply!

Post by Carsten Schoenert
For Thunderbird intrigeri and myself came to the conclusion that
especially for the apparmor profile someone from the apparmor team
should be able to contribute changes to the profile directly to the git
tree. So intrigeri has become a member of the pkg-mozilla group to be
able to push changes by himself. I trust intrigeri enough that he will
do good contributions. For now it's the best we can do. This at all is
for sure improvable and we should talk about this on upcoming Debian
events or directly via email.

Okay, filed the bugs, lets see where they go. :) I was especially
concerned about the browser part.

Post by Carsten Schoenert
...

Post by Philipp Kern
[1] e.g.
apparmor="DENIED" operation="file_inherit" profile="thunderbird//gpg"
name="/usr/share/thunderbird/omni.ja" pid=24720 comm="gpg2"
requested_mask="r" denied_mask="r" fsuid=1000 ouid=0

Filed as #880425[1].

Post by Carsten Schoenert

Post by Philipp Kern
[2] e.g.
apparmor="DENIED" operation="exec" profile="thunderbird"
name="/opt/google/chrome-beta/google-chrome-beta" pid=31896
comm="thunderbird" requested_mask="x" denied_mask="x" fsuid=1000 ouid=0

Filed as #880424[0]. I think there is a deeper question here as to how
to handle the browser abstraction for AppArmor in general.

There is /etc/apparmor.d/abstractions/ubuntu-browsers. The name isn't
very nice it's a start if we rename it to
/etc/apparmor.d/abstractions/browsers.

Cheers,
-- Guido

Post by Philipp Kern

Post by Carsten Schoenert
I suggest to open a bug report for each of such issues against
thunderbird with a description what was done and what was expected.

As above. :)
Kind regards and thanks
Philipp Kern
[0] https://bugs.debian.org/880424
[1] https://bugs.debian.org/880425

[Christian Seiler]

Hi there,

Post by Philipp Kern

Post by Philipp Kern
[2] e.g.
apparmor="DENIED" operation="exec" profile="thunderbird"
name="/opt/google/chrome-beta/google-chrome-beta" pid=31896
comm="thunderbird" requested_mask="x" denied_mask="x" fsuid=1000 ouid=0

Filed as #880424[0]. I think there is a deeper question here as to how
to handle the browser abstraction for AppArmor in general.

Isn't this a really generic problem for any AppArmor/SELinux-like LSM?

Because this isn't just restricted to links, this is restricted to any
kind of file association (links are handled like file associations with
the MIME type x-scheme-handler/$PROTOCOL in the XDG specs).

Because in the end you have the following conflicting requirements:

- On the one hand, you want the user to be able to open arbitrary
files and links from many programs (especially e.g. email programs),
for example opening PDF attachments with your favorite PDF reader
or external links in your browser.

This means that the email program must be able to execute arbitrary
executables, because the user may have assigned arbitrary
executables (for example also wrapper scripts in their home
directory) to different file associations.

- The user might also want to open a specific file with another
program that is also associated with the file but not the default.
For example, JPEG attachments might most commonly be opened by
the favorite image viewer, but sometimes users may want to open
the JPEG file in an image editor such as GIMP, and many programs
offer the user a choice to choose between the installed programs
that are associated with the type.

- On the other hand one of the key features of AppArmor is to mitigate
exploits so that an attacker can't just get the program to call the
syscall execve("/bin/sh", {"/bin/sh", "-c", "wget payload | sh"});

I think this is a more general problem. It appears to me that there are
currently two possibilities:

- Either one allows the execution of arbitrary executables by all
desktop applications (because we don't know in advance what file
types will be associated with what program, and the user may have
their own wrapper scripts) - but that leaves a gaping hole in the
possible mitigations AppArmor may provide.

- Or one whitelists certain applications. This will have the
unfortunate side-effect that any time the user installs a piece of
software that isn't on that whitelist (or wants to use their own
wrapper script) it won't work (because of AppArmor) - and
unfortunately many users will then simply resort to disabling
AppArmor in that case instead of actually creating a locally
adapted policy. (Yes, sysadmins might not, but simple desktop
users will - I know way too many people who simply don't even want
to use group ownership and instead are happy to just do a
chmod 0777 - and groups are mentally a lot simpler than AppArmor.)

I don't know what the best short-term compromise is here, but in the
long term the only real solution is to somehow abstract this away from
applications to ensure that the application started in these cases is
actually what the user wanted. (I'm thinking towards something like
the 'portals' concept in Flatpak.) Because if the default policy does
not cover these kinds of customization needs out of the box at least a
lot of desktop users will simply revert AppArmor and complain about
it.

Regards,
Christian

[Philipp Kern]

Post by Christian Seiler
I don't know what the best short-term compromise is here, but in the
long term the only real solution is to somehow abstract this away from
applications to ensure that the application started in these cases is
actually what the user wanted. (I'm thinking towards something like
the 'portals' concept in Flatpak.) Because if the default policy does
not cover these kinds of customization needs out of the box at least a
lot of desktop users will simply revert AppArmor and complain about
it.

Sadly I have so far been ignorant about Flatpak, which was probably a
mistake. I think the abstractions I seek here is that the file is passed
with its type to a different arbiter of defaults that is then
responsible for opening the right program. And there needs to be a
context transition in which the arbiter can then actually execute what
it wants. At the same time it needs to be a very limited interface that
does not allow customization. On the other hand Thunderbird parses the
file associations by itself today and raises an application picker. This
would again need to be isolated away[1].

I'm not sure if I missed some kind of alert tool like the selinux
troubleshooting bits, but in my case it just silently failed: you click
on links and nothing happens. I suppose such failures that the kind of
failures we'd want to avoid because users just keep being confused about
how to re-mediate the issue and then turn off security features. (Not
unsimilar to how people deal with SELinux, except that some might pipe
the denials through audit2allow?)

Kind regards and thanks
Philipp Kern

[1] And even then I know that a bunch of uncommon file types raise a
bunch of red flags in terms of auto-detection and exploitation.
CVE-2017-1000083 comes to mind. Which at least could have been mitigated
by AppArmor on evince, in theory. Except that evince also needs to be
able to handle links in some fashion and back to square one.

[gregor herrmann]

Post by Philipp Kern
I'm not sure if I missed some kind of alert tool like the selinux

In case you don't know it, apparmor-notify has been helpful for me.


Cheers,
gregor

[Philipp Kern]

Post by gregor herrmann

Post by Philipp Kern
I'm not sure if I missed some kind of alert tool like the selinux

In case you don't know it, apparmor-notify has been helpful for me.

Thanks for the pointer! I indeed didn't know about it yet.

With the omni.js denial it's a tad noisy and the user has to be in group
adm to read /var/log/kern.log (because dmesg is restricted to root these
days). But at least it's immediate when a denial happens, so that's good. :)

Kind regards
Philipp Kern

[Simon McVittie]

Post by Christian Seiler
I don't know what the best short-term compromise is here, but in the
long term the only real solution is to somehow abstract this away from
applications to ensure that the application started in these cases is
actually what the user wanted. (I'm thinking towards something like
the 'portals' concept in Flatpak.)

A couple of the key things that the Flatpak developers hope will make it
work better than previous approaches to a similar problem-space are:

* Accepting that expecting an unmodified app designed to be used in a
non-sandboxed context to be sandbox-friendly does not lead to a
usefully-strict sandbox, because of factors like those you mentioned;
* Being able to change the libraries that apps use so that some aspects
of the app can transparently become more sandbox-friendly

Accordingly, various APIs in GLib/GTK+ have been modified to detect when
they are operating in a sandbox and call out to portals instead of doing
the work themselves. These APIs are already sufficiently high-level that
the application doesn't need to see a difference. Of course, this only
works for applications that use a higher-level library API rather than
implementing it themselves, so that probably rules out the Mozilla
stack...

GLib's helper executables like `gio open` (as used by recent versions
of xdg-open when running on GNOME) use those same APIs, so they will
also do the right thing in a Flatpak sandbox.

I don't have an overview of what's happening in this direction outside
GNOME, but I hope that other "platform" libraries like Qt have done
similarly or will do so in future.

smcv

[intrigeri]

Hi,

thanks Christian and Simon for summing up the problem and pointing
to promising work.

As mentioned in my introductory email I don't think it's worth putting
too much effort into AppArmor for the GUI apps use case, and one
should not expect too much security from it. I suggest anyone
interested in major redesigns and solutions for the bright future to
look into the "security by designation" concept & the corresponding
implementations in Flatpak and friends instead.

This is why my preferred strategy here is to focus on the low-hanging
fruits i.e. ship policy that works well already, or needs only small
amounts of polishing.

For more complex cases for which AppArmor is not well suited, let's
either make the profile wide enough to avoid breaking stuff (at the
cost of making it provide little security) or simply disable it by
default. Thunderbird is definitely one of these complex cases so let's
keep an eye on it: if AppArmor is too disruptive there then we will
disable it by default for Thunderbird.

Cheers,

[Ian Jackson]

Post by Christian Seiler
- Or one whitelists certain applications. This will have the
unfortunate side-effect that any time the user installs a piece of
software that isn't on that whitelist (or wants to use their own
wrapper script) it won't work (because of AppArmor) - and
unfortunately many users will then simply resort to disabling
AppArmor in that case instead of actually creating a locally
adapted policy. (Yes, sysadmins might not, but simple desktop
users will - I know way too many people who simply don't even want
to use group ownership and instead are happy to just do a
chmod 0777 - and groups are mentally a lot simpler than AppArmor.)

I think this whitelisting approach is best. The reason is that most
of our system has not been hardened and audited against malicious
files. The need for an entry on the whitelist means that some
developer somewhere has decided that yes, this specific program should
be automaticaly exposed to potentially hostile data.

The lack of a useable exception mechanism, with a sensible UI, is a
big problem though. Ideally you would ask the user something like

This { email attachment | web download } is a DESCRIPTION. The
program for this is NAME but it has not been audited for security.
If you're not expecting this kind of file then you probably didn't
want to open it - it might be malware.

[[ Don't open ]]
[ Open this once ]
[ Always open these kinds of files without asking ]

You can change your mind about "always" by going to "settings"
/ "security" in the { Thunderbird | Firefox } menu.

Ian.

[Philipp Kern]

Post by Ian Jackson
The lack of a useable exception mechanism, with a sensible UI, is a
big problem though. Ideally you would ask the user something like
This { email attachment | web download } is a DESCRIPTION. The
program for this is NAME but it has not been audited for security.
If you're not expecting this kind of file then you probably didn't
want to open it - it might be malware.
[[ Don't open ]]
[ Open this once ]
[ Always open these kinds of files without asking ]
You can change your mind about "always" by going to "settings"
/ "security" in the { Thunderbird | Firefox } menu.

Designing UIs that are not just ignored by the users are hard, though,
while sandboxes give you some defense in depth. For instance opening a
.pdf file with Chrom(e|ium) gives you a sandboxed PDF renderer (PDFium).

Plus it's not like you can present a super useful description. Go by
file extension? Go by included MIME type? (Hello,
application/octet-stream.) Go by whatever the target application sniffs
it to be[1]? And of course never try to obtain a locally generated
thumbnail of untrusted content.

In the end you'd just show that warning half of the time and users are
trained to just click the button without reading the box because users
are bad at reading.

Kind regards
Philipp Kern

[1]
https://scarybeastsecurity.blogspot.de/2016/11/0day-exploit-compromising-linux-desktop.html

[intrigeri]

Hi,

Post by Carsten Schoenert
Starting with this thread and by some talking to various people I
changed my mind about this. For better flexibility and customizing,
thinking about various releases (like *-security, *-backports e.g.) that
need to be supported, I believe apparmor profiles for the applications
should stay in the belonging source packages in most cases.

Fully agreed.

Post by Carsten Schoenert
Ubuntu is doing the opposite as far as I know [1], the time will show
which way is batter.

FYI we're doing just like Ubuntu. I understand why you've been mislead
— you're definitely not the first one — and Ulrike has documented
where things are:

https://wiki.debian.org/AppArmor/Contribute/Upstream#Debian_.2F_Upstream_relationship

Cheers,

[Clément Hermann]

Hi,


A year ago, intrigeri proposed an experiment:  let's enable AppArmor by
default in testing/sid.

Here is an extract from the proposal (you can find the full proposal and

Post by intrigeri
tl;dr: I hereby propose we enable AppArmor by default in testing/sid,
and decide one year later if we want to keep it this way in the
Buster release.
- Get a rough idea of what amount of effort the Debian project is
happy (and able) to invest into such proactive security measures.
- Learn about any relevant social & technical concerns I am not
aware of.
I expect the proposal will need to be refined, or abandoned, to take
into account what we will learn during this preliminary discussion.
Why do we need AppArmor?
========================
AppArmor is a Mandatory Access Control framework implemented as
a Linux Security Module (LSM), user space utilities, and a quite
simple language to define policy.
AppArmor confines programs according to a set of rules that specify
what operations a given program can access, e.g. it can prevent
exploited server software from accessing data it does not own and
executing arbitrary code. This proactive approach helps protect the
system against some known and unknown vulnerabilities.
Various actors are actively exploiting software. Random users are
victimized every day, and specific populations are specifically
targeted, e.g. government opponents, human rights defenders, system
administrators, software developers and distributors, as revealed by
the Snowden leaks.
Every month we learn about many new attack vectors made possible by
programming errors. We fix them after the fact, which is great but
a bit too late: users may already have been exploited. Most operating
systems have adopted proactive approaches to mitigate the impact of
such problems.
In Debian, great efforts are in progress: hardening binaries makes it
harder to write successful exploits, and making our packages build
reproducibly will make it harder to introduce vulnerabilities at the
binary level.
Still, Debian is far from being best in class on this front: we have
no widespread mechanism for sandboxing desktop applications. We can
surely do better. The great news is that there is one low-hanging
fruit waiting to be picked, and it's what this proposal is about :)
A proposal
==========
1. Enable AppArmor by default in testing/sid as soon as feasible in
the Buster cycle.
I can think of several possible ways to do it but for now I'd
rather focus on the "do we want to do it at all" conversation.
https://wiki.debian.org/AppArmor/Progress#Enabling_AppArmor_by_default.3F
https://wiki.debian.org/AppArmor/HowToUse :)
2. During a year, watch out for AppArmor related issues and address
them in a prompt manner.
Note that the best way to address them quickly enough is sometimes
to simply disable the problematic AppArmor profile: it's cheap,
doesn't require advanced AppArmor skills, and IMO a smaller
AppArmor policy enabled by default is more useful than a broader
but less robust one that only a couple thousand users benefit from.
3. A year after AppArmor was enabled by default: evaluate how it went
and decide if Buster should be shipped with AppArmor enabled by
default or not.
I commit to do an analysis using BTS data to help make this
decision. If we need formal success criteria and a clearly defined
team who'll make the call, I'm happy to think about it. But here
again I'd rather focus on the general idea than on implementation
details at this point.

So, this is just a heads up for people attending Debconf18 in Hsinchu:
there is a BoF this afternoon [1] were we would like *you* to share your
feelings about it, be they positive or negative, and share skills, tips
and tricks.


Please, come and join us in room Xueshan (雪山) at 16:00 if you're
interested: we're very excited about it!


[0] https://lists.debian.org/debian-devel/2017/08/msg00090.html

[1]
https://debconf18.debconf.org/talks/32-apparmor-in-debian-lets-share-feelings-technical-feedback-and-skills/


Cheers,

[Clément Hermann]

Post by Clément Hermann
there is a BoF this afternoon [1] were we would like *you* to share your
feelings about it, be they positive or negative, and share skills, tips
and tricks.
Please, come and join us in room Xueshan (雪山) at 16:00 if you're
interested: we're very excited about it!

Aaaaand I've been bitten by the slightly moving schedule (it's alive!):
it's actually at 16:30:
https://debconf18.debconf.org/talks/32-apparmor-in-debian-lets-share-feelings-technical-feedback-and-skills/


Cheers!

[intrigeri]

Hi,

Post by intrigeri
tl;dr: I hereby propose we enable AppArmor by default in testing/sid,
and decide one year later if we want to keep it this way in the
Buster release.

Here are some data points relevant to this decision making process.
I think we're in a good place to make a decision in November 2018
as planned.

Post by intrigeri
A proposal
==========
1. Enable AppArmor by default in testing/sid as soon as feasible in
the Buster cycle. […]

AppArmor was enabled by default in November 2017 in our kernel in sid
and the change quickly propagated to testing.

Post by intrigeri
2. During a year, watch out for AppArmor related issues and address
them in a prompt manner. […]

This happened. See latency stats below.

Post by intrigeri
3. A year after AppArmor was enabled by default: evaluate how it went
and decide if Buster should be shipped with AppArmor enabled by
default or not.

As part of this evaluation process, see the report of the BoF we've
hosted on this topic at DebConf18:
https://people.debian.org/~intrigeri/blog/posts/Report:_AppArmor_BoF_at_DebConf18/

Next step for the AppArmor team: compile the list of remaining issues
we consider as blockers and ensure they're fixed before Buster
is frozen.

Post by intrigeri
I commit to do an analysis using BTS data to help make this
decision.

AFAICT the BTS currently has very few bugs opened that are caused by
AppArmor and have no patch attached. If the ones you care are not
on our radar, see https://wiki.debian.org/AppArmor/Reportbug#Usertags.

I've analyzed how the AppArmor team has performed once a given bug
appeared on our radar (with the documented usertags) between
2017-08-01 and 2018-08-04, inclusive. tl;dr:

- 47 bugs were tagged since 2017-08-01 (not counting duplicates);
I suspect quite a few more issues were not usertagged and are
therefore not accounted for in my stats; I hope this means the
maintainer of the affected packages could solve the problem
themselves somehow;

- in the vast majority of the cases, the AppArmor team sent a first
reply on the very day the bug popped up on our radar;

- in 75% of the cases, a solution was proposed within 8 days once
the bug was brought to our attention.

If you're into things like standard deviation and want details, see
the statistics section at the bottom of this email :)

reportbug 7.1.8+ reports the enabled LSM so we could compute the
ratio of testing/sid users who opted-out from AppArmor. I doubt
that spending time on it would be the best use of my Debian time
but if you folks feel it's needed to make a decision, I'll do it.

Post by intrigeri
Here's what popcon says ("Vote" count) for the apparmor binary
* 2015-01: ~400
* 2016-01: ~700 (+75% in a year)
* 2017-01: ~1300 (+85% in a year)
* today: 1870 (+44% in 7 months)

^^ i.e. 2017-08-04

Update:

* 2018-08-04 12986 (+594% in a year)
… presumably because linux-image-* now "Recommends: apparmor".

Statistics
==========

First reply (days)
------------------

- count: 47
- variance: 3.22201665124884
- standard deviation: 1.79499767444107
- mean: 0.319148936170213
- per-quartiles:
* min: 0
* 25th percentile: 0
* 50th percentile (median): 0
* 75th percentile: 0
* max: 12

Fix available (days)
--------------------

That is, either a fix is available and could be applied by the
maintainer, or for very rare corner case, a workaround was provided to
the bug reporter.

- count: 46
- variance: 1219.94975845411
- standard deviation: 34.927779180104
- mean: 14.695652173913
- per-quartiles:
* min: 0
* 25th percentile: 0
* 50th percentile (median): 1
* 75th percentile: 8
* max: 167

Cheers,