[Distutils] Migrating Hashes from MD5 to SHA256

[Donald Stufft]

PyPI has historically used MD5 in order to verify the downloads. However MD5 is severely broken and is generally regarded as something that should be migrated away from ASAP. From speaking with a number of cryptographers they've more or less said that the major reason they believe that MD5 hasn't had a published pre-image attack is just because it's so broken that most researchers have moved on to newer hashes.

Since versions 1.2 pip has supported md5, sha1, and any of the sha2 family. Additionally it has only supported SSL verification since 1.3. This means there is no version of pip which both verifies SSL and only allows MD5.

Since version 0.9 setuptools has supported md5, sha1, and any of the sha2 family and it has only supported SSL verification since 0.7.

I propose we switch PyPI from using MD5 to using SHA256. There is no security lost from using a hash that pip prior to version 1.2 doesn't understand as it didn't verify SSL so an attacker could simply modify the hashes if they wanted. Additionally there is no security list from setuptools versions earlier than 0.7.

However setuptools versions 0.7-0.8 will lose their hashes. I believe this is a ok thing to happen because the uptake on 0.7-0.8 is fairly low. Most people will use the setuptools bundled with virtualenv which has only ever bundled 0.6 or 0.9.

So essentially moving from MD5 to SHA256 will only negatively affect the security of a handful of users while positively impacting the security of everyone else.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[PJ Eby]

Post by Donald Stufft
Additionally there is no security list from setuptools versions earlier than 0.7.

Not true, actually. Setuptools 0.6 dev releases supported SSL
verification since mid-May, but don't support any hashes besides MD5.
Anybody who updated their setuptools between then and the release of
0.7 would have that version. Unfortunately, it's hard to tell how
many people that is, though I could try and dig through my server logs
to find out.

There's also another issue with jumping to SHA256: Python prior to 2.5
didn't support it.

Which brings up another point: the setuptools 0.6 series is the only
setuptools available for Python 2.3. That's one of the reasons it's
still available for download. If you want SSL verification on 2.3,
it's the only thing available. (Meanwhile, a lot of people are still
downloading 0.6c11; probably I should package up an 0.6c12 so those
folks pick it up instead of 0.6c11.)

Anyway, this is all somewhat moot since the hashes only matter when
the download is hosted somewhere besides PyPI, since SSL verification
is available for the PyPI part. Even so, I'd suggest that moving to
SHA1 might be a good intermediate step: it's available on Python 2.3,
so I could backport the relevant support to the 0.6 branch. (IIUC,
Python 2.3 is still the default version for many Linux distros that
have not reached end-of-life support.)

[Alex Gaynor]

Post by PJ Eby
There's also another issue with jumping to SHA256: Python prior to 2.5
didn't support it.

I'm not sure this is a particularly relevant concern. Python's prior to 2.5 are
no longer supported by the people who wrote them, or almost any major packages
(SQLAlchemy, Django, pip, virtualenv, the list goes on and on). I'm not sure
why we should expend any effort to support them either.

Alex

[Donald Stufft]

Post by PJ Eby

Post by Donald Stufft
Additionally there is no security list from setuptools versions earlier than 0.7.

Not true, actually. Setuptools 0.6 dev releases supported SSL
verification since mid-May, but don't support any hashes besides MD5.
Anybody who updated their setuptools between then and the release of
0.7 would have that version. Unfortunately, it's hard to tell how
many people that is, though I could try and dig through my server logs
to find out.
There's also another issue with jumping to SHA256: Python prior to 2.5
didn't support it.
Which brings up another point: the setuptools 0.6 series is the only
setuptools available for Python 2.3. That's one of the reasons it's
still available for download. If you want SSL verification on 2.3,
it's the only thing available. (Meanwhile, a lot of people are still
downloading 0.6c11; probably I should package up an 0.6c12 so those
folks pick it up instead of 0.6c11.)
Anyway, this is all somewhat moot since the hashes only matter when
the download is hosted somewhere besides PyPI, since SSL verification
is available for the PyPI part. Even so, I'd suggest that moving to
SHA1 might be a good intermediate step: it's available on Python 2.3,
so I could backport the relevant support to the 0.6 branch. (IIUC,
Python 2.3 is still the default version for many Linux distros that
have not reached end-of-life support.)

I don't have a Python 2.3 available to attempt to test. To be honest I've
never even used Python 2.3. Does the hashlib backport I added to
setuptools 0.9 for Python 2.4 work on 2.3? It's a pure python
implementation of hashlib.

Sha1 is better but it already has weaknesses so if at all possible it
would be much preferred and significantly better to switch to sha256.

Setuptools doesn't appear to include the python version in it's user
agent so I can't get any sort of information about Python 2.3 usage.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[PJ Eby]

Post by Donald Stufft
Does the hashlib backport I added to
setuptools 0.9 for Python 2.4 work on 2.3? It's a pure python
implementation of hashlib.

Ah, didn't know about that! I can't imagine what problems there would
be; not much changed in 2.4 that can't be emulated in 2.3.

Anyway, I'll have a look. Thanks!

Post by Donald Stufft
I don't have a Python 2.3 available to attempt to test. To be honest I've
never even used Python 2.3.

Heh. Noob. ;-) (j/k)

2.3 is basically 2.4 minus decorators, generator expressions, various
builtins and stdlib features. Unless you used set types, reversed(),
or various other odds and ends, I should be able to backport it.

Post by Donald Stufft
[stuff about RHEL support]

If there's a 2.4 hashlib backport, that addresses my concerns just
fine. If I need to, I'll backport it to setuptools 0.6. Thanks.

[Donald Stufft]

Post by PJ Eby

Post by Donald Stufft
Does the hashlib backport I added to
setuptools 0.9 for Python 2.4 work on 2.3? It's a pure python
implementation of hashlib.

Ah, didn't know about that! I can't imagine what problems there would
be; not much changed in 2.4 that can't be emulated in 2.3.
Anyway, I'll have a look. Thanks!

Here's the relevant commits in the new setuptools stuff:

https://bitbucket.org/pypa/setuptools/commits/330b628f38c9380c95a818e65fd56812cbc854c4
https://bitbucket.org/pypa/setuptools/commits/b1d4e48beebdcc3cf7cb06fae4c4005a85dfc9bc
https://bitbucket.org/pypa/setuptools/commits/12dd4b89148a225856a060cbee1137fc4cf79736

The implementations are taken from PyPy so they are made
to work on Python 2.7, but they worked just fine on 2.4 after
removing a single b"".

Post by PJ Eby

Post by Donald Stufft
I don't have a Python 2.3 available to attempt to test. To be honest I've
never even used Python 2.3.

Heh. Noob. ;-) (j/k)

:) I was still in high school when the last 2.3 was released :/

Post by PJ Eby
2.3 is basically 2.4 minus decorators, generator expressions, various
builtins and stdlib features. Unless you used set types, reversed(),
or various other odds and ends, I should be able to backport it.

Great!

Post by PJ Eby

Post by Donald Stufft
[stuff about RHEL support]

If there's a 2.4 hashlib backport, that addresses my concerns just
fine. If I need to, I'll backport it to setuptools 0.6. Thanks.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Nick Coghlan]

Post by PJ Eby

Post by Donald Stufft
Does the hashlib backport I added to
setuptools 0.9 for Python 2.4 work on 2.3? It's a pure python
implementation of hashlib.

Ah, didn't know about that! I can't imagine what problems there would
be; not much changed in 2.4 that can't be emulated in 2.3.
Anyway, I'll have a look. Thanks!

FWIW, I expect the intersection of "running RHEL 4" and "downloading
software directly from PyPI" to be a vanishingly small subset of
humanity - anybody conservative enough to be running a version of RHEL
that old is going to have *very* strict rules about how software gets
onto their production servers and are also likely to be using
something far more recent to talk to the outside world.

Cheers,
Nick.

[Donald Stufft]

Post by PJ Eby
Anyway, this is all somewhat moot since the hashes only matter when
the download is hosted somewhere besides PyPI, since SSL verification
is available for the PyPI part. Even so, I'd suggest that moving to
SHA1 might be a good intermediate step: it's available on Python 2.3,
so I could backport the relevant support to the 0.6 branch. (IIUC,
Python 2.3 is still the default version for many Linux distros that
have not reached end-of-life support.)

I think RHEL is the one that will support things the longest. As far as I
can tell Python 2.3 was default in RHEL4 and RHEL5 used Python 2.4

ELS support for RHEL4 ends Feb of 2015, so that's roughly a year and
a half till not even Red Hat supports Python 2.3 anymore that I can tell?
It also appears that support for new installations ended roughly a year
and a half ago.

Many (most?) projects no longer support Python 2.3 on PyPI, and I
seriously doubt that there is a significant number of people who both
want the stability provided by RHEL and is willing to continue using a
release that is this close to being EOL'd while simultaneously wanting
to download things from PyPI. CPython hasn't supported Python 2.3
in years.

Basically If RHEL customers want the security updates they should
bother Red Hat for them, that's part of why they pay for RHEL instead
of going with a free system. I don't think it's appropriate to allow a
handful of people who might still be on a version of python first
released 10 years ago and last released 8 years ago to negatively
impact everyone else.

Note: I don't use RHEL so my understanding of it's life cycle is from
reading their support page.


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Christian Heimes]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Post by Donald Stufft
PyPI has historically used MD5 in order to verify the downloads.
However MD5 is severely broken and is generally regarded as
something that should be migrated away from ASAP.

[Donald Stufft]

A couple of months ago I suggested a schema that includes MD5, SHA-2
file.tar.gz#MD5=1234&SHA-256=abcd&filesize=5023
That should work for old versions of setuptool and can easily be
supported in new versions of pip and setuptools.

It won't work for old versions, it explicitly includes the end of line terminator and the #.

A new hash sum scheme must include the possibility to add multiple and
new hash algorithms. A download tool shall check the hash sum for all
supported algorithms, too. I also like to see the file size in the
scheme. It's useful to know the file size in preparation of the
download. The file size validation mitigates some attack possibilities.

Right now that would break too much. I agree this is where we need to
get too but It'll likely need to wait for the new API in Warehouse.

Christian

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[zooko]

Post by Donald Stufft
PyPI has historically used MD5 in order to verify the downloads. However MD5 is severely broken and is generally regarded as something that should be migrated away from ASAP.

[Donald Stufft]

Post by Donald Stufft
PyPI has historically used MD5 in order to verify the downloads. However MD5 is severely broken and is generally regarded as something that should be migrated away from ASAP. From speaking with a number of cryptographers they've more or less said that the major reason they believe that MD5 hasn't had a published pre-image attack is just because it's so broken that most researchers have moved on to newer hashes.

Who said that? That contradicts my beliefs.

It's possible I misunderstood the exact implications of what they were
saying. I am not a cryptographer and it was a month or two ago we
spoke. It was stressed to me that PyPI should be moving off of MD5.

I do believe however that we don't know for sure if MD5 is going to be
have a practical pre-image attack tomorrow, or if it will last another 10
years. Given that all security systems are not infallible and are generally
designed so that you have margins of security so there is time to migrate.

The safety margins on MD5 have long since gone so by continuing to use
it we are ignoring prudence (especially at a fairly ideal time where we are
at a transitioning from unverified HTTPS/HTTP to HTTPS so we do not
need to regard backwards compatibility as highly). As far as I am aware
these attacks tend to come all of a sudden and without warning. I would
much rather have already migrated to something that still has it's safety
margins than be caught with our proverbial pants down and need to
scramble *if* an attack is discovered.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Antoine Pitrou]

Post by Donald Stufft
Most people will use the setuptools bundled with virtualenv which has
only ever bundled 0.6 or 0.9.

Why do you think that? According to PyPI:
- virtualenv:
4664 downloads in the last day
34512 downloads in the last week
- setuptools:
23901 downloads in the last day
197351 downloads in the last week

Certainly setuptools use seems much more widespread than virtualenv use.

Regards

Antoine.

[Donald Stufft]

Post by Antoine Pitrou

Post by Donald Stufft
Most people will use the setuptools bundled with virtualenv which has
only ever bundled 0.6 or 0.9.

4664 downloads in the last day
34512 downloads in the last week
23901 downloads in the last day
197351 downloads in the last week
Certainly setuptools use seems much more widespread than virtualenv use.
Regards
Antoine.
_______________________________________________
http://mail.python.org/mailman/listinfo/distutils-sig

Comparing download numbers isn't generally useful. People
almost never install virtualenv more than once per machine. There
are a myriad of reasons why they might directly install setuptools.

It's impossible to know for sure how it'll be gotten but my gut is
that most people use whatever is default inside of virtualenv
because that's how almost everyone i've seen who uses virtualenv
does it unless they have special needs.

However the 0.9 line have almost as many downloads alone as the
0.7 and 0.8 lines combined.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Antoine Pitrou]

Post by Donald Stufft

Post by Antoine Pitrou

Post by Donald Stufft
Most people will use the setuptools bundled with virtualenv which has
only ever bundled 0.6 or 0.9.

4664 downloads in the last day
34512 downloads in the last week
23901 downloads in the last day
197351 downloads in the last week
Certainly setuptools use seems much more widespread than virtualenv use.
Regards
Antoine.

Comparing download numbers isn't generally useful. People
almost never install virtualenv more than once per machine. There
are a myriad of reasons why they might directly install setuptools.

If your assertion were true ("Most people will use the setuptools
bundled with virtualenv"), then the setuptools download numbers
would be minuscule. The actual numbers show it to be untrue.
Whether or not they are directly comparable isn't important: the
orders are magnitude are sufficiently eloquent.

Post by Donald Stufft
It's impossible to know for sure how it'll be gotten but my gut is
that most people use whatever is default inside of virtualenv
because that's how almost everyone i've seen who uses virtualenv
does it unless they have special needs.

Perhaps you don't realize that many people don't use virtualenv at all,
so they simply cannot use virtualenv's setuptools, either. Which is
perfectly compatible with those download numbers, unlike your original
assertion.

Regards

Antoine.

[Donald Stufft]

Post by Antoine Pitrou
If your assertion were true ("Most people will use the setuptools
bundled with virtualenv"), then the setuptools download numbers
would be minuscule. The actual numbers show it to be untrue.
Whether or not they are directly comparable isn't important: the
orders are magnitude are sufficiently eloquent.

Post by Donald Stufft
It's impossible to know for sure how it'll be gotten but my gut is
that most people use whatever is default inside of virtualenv
because that's how almost everyone i've seen who uses virtualenv
does it unless they have special needs.

Perhaps you don't realize that many people don't use virtualenv at all,
so they simply cannot use virtualenv's setuptools, either. Which is
perfectly compatible with those download numbers, unlike your original
assertion.

I don't think any claim can be made about the relative use between the
two tools by looking at the download counts because their typical use is
generally very different. But sure you're right whatever does that make
you feel better?

Are you trying to claim we shouldn't move to a stronger hash?

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Antoine Pitrou]

Post by Donald Stufft
I don't think any claim can be made about the relative use between the
two tools by looking at the download counts because their typical use is
generally very different.

I'll try to phrase it more clearly then: I am not *comparing* their
relative use. I am simply pointing out that an extremely large number
of people install setuptools separately. Whether or not they also use
virtualenv is completely irrelevent (but, of course, chances are
they don't: otherwise, as you say, they'd use the bundled versions).

Post by Donald Stufft
But sure you're right whatever does that make
you feel better?

Now, please calm down...

Post by Donald Stufft
Are you trying to claim we shouldn't move to a stronger hash?

No, I'm just saying the possibility of regressions isn't as small
as you think based on a misinterpretation of how people actually
get setuptools installed (many of them get it directly from PyPI).

But, yes, we should of course move to something better than md5,
and ideally make the format flexible enough to avoid further
breakage when switching hashes again.

Regards

Antoine.

[Donald Stufft]

Ok so given that:

- There's a readably available solution for Python 2.4+ with the likelihood
being that most users are either using it or using an older version which
doesn't support SSL.
- The number of folks likely to be on Python 2.3 and wanting to install things
from PyPI is likely to be very small.
- There's possibly a future solution for Python 2.3
- The safety margins for MD5 are gone and cryptographers heavily suggest
moving away from it.
- A revised scheme will break backwards compatibility with the versions of
the tooling that do support a stronger hash.

I'm going to go ahead and make this change unless someone comes out and
contests moving PyPI to SHA256. I'll give it a bit to make sure no one does
have an issue with the move.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Nick Coghlan]

Post by Donald Stufft
- There's a readably available solution for Python 2.4+ with the likelihood
being that most users are either using it or using an older version which
doesn't support SSL.
- The number of folks likely to be on Python 2.3 and wanting to install things
from PyPI is likely to be very small.
- There's possibly a future solution for Python 2.3
- The safety margins for MD5 are gone and cryptographers heavily suggest
moving away from it.
- A revised scheme will break backwards compatibility with the versions of
the tooling that do support a stronger hash.
I'm going to go ahead and make this change unless someone comes out and
contests moving PyPI to SHA256. I'll give it a bit to make sure no one does
have an issue with the move.

+1, this sounds like a good way forward for the existing PyPI interfaces.

We can do something better once the focus shifts from "make the status
quo not broken" to making the next generation interfaces a reality
(PEP 426 et al).

Cheers,
Nick.

[holger krekel]

Hi Nick, Donald, all,

Post by Nick Coghlan

Post by Donald Stufft
- There's a readably available solution for Python 2.4+ with the likelihood
being that most users are either using it or using an older version which
doesn't support SSL.
- The number of folks likely to be on Python 2.3 and wanting to install things
from PyPI is likely to be very small.
- There's possibly a future solution for Python 2.3
- The safety margins for MD5 are gone and cryptographers heavily suggest
moving away from it.

Please detail the actual attack scenario wrt PyPI/installer processes.

Post by Nick Coghlan

Post by Donald Stufft
- A revised scheme will break backwards compatibility with the versions of
the tooling that do support a stronger hash.
I'm going to go ahead and make this change unless someone comes out and
contests moving PyPI to SHA256. I'll give it a bit to make sure no one does
have an issue with the move.

Actually, i strongly object further backward-incompatible changes.

Please (generally) find a way to introduce improvements without breaking
existing installation processes at the same time.

For example, in this case pip/easy_install could indicate to PYPI what
kind of hashes it accepts (through a header or query param or whatever)
and PyPI could serve it but we'd default to MD5 for now if nothing else
was requested. Please also consider the PEP438 vetted registration of
externals+hashses in this context. Once things and tools are working
nicely we can switch to serving a non-MD5 hash as default after a
sufficient grace period.

Post by Nick Coghlan
+1, this sounds like a good way forward for the existing PyPI interfaces.
We can do something better once the focus shifts from "make the status
quo not broken" to making the next generation interfaces a reality
(PEP 426 et al).

Just switching the hashes is likely to break things. Do you want to take
a bet on that? Just switching to SSL broke things. Just switching
pip-1.4 broke things. Just switching to newer setuptools broke things.
For me the fall-out of all the well-intentioned changes has been
frustrating lately.

For example, one of my projects documented how to generate a
supervisor-controled deployment and suddenly "pip install supervisor"
does not work anymore because pip-1.4 doesn't consider any existing
supervisor distributions as non-dev. You can argue that's supervisor's
fault but in the end it doesn't matter: What used to work and was
documented is now broken, users complaining or simply turning away, with
a re-inforced view of "python packaging sucks" or "this tool sucks, the
docs are wrong".

There are >1000 times more people using python packaging tools than
there are ones following the recent distutils-sig/Donald/Nick verdicts
or tool release notes. And heck, even for me being quite involved in all
this stuff, having written a related PEP, it's really hard to see what's
coming and to prevent breakage for my tools's/projects's users.
Therefore, to be honest, at this point i am almost afraid of what is
released/deployed next in PyPI/Installer land. I'd rather like to
get into a "can't wait to get the next release" mode.

best,
holger

[Nick Coghlan]

Post by holger krekel
Hi Nick, Donald, all,

Post by Donald Stufft
- There's a readably available solution for Python 2.4+ with the likelihood
being that most users are either using it or using an older version which
doesn't support SSL.
- The number of folks likely to be on Python 2.3 and wanting to install things
from PyPI is likely to be very small.
- There's possibly a future solution for Python 2.3
- The safety margins for MD5 are gone and cryptographers heavily suggest
moving away from it.

Please detail the actual attack scenario wrt PyPI/installer processes.

Post by Donald Stufft
- A revised scheme will break backwards compatibility with the versions of
the tooling that do support a stronger hash.
I'm going to go ahead and make this change unless someone comes out and
contests moving PyPI to SHA256. I'll give it a bit to make sure no one does
have an issue with the move.

Actually, i strongly object further backward-incompatible changes.
Please (generally) find a way to introduce improvements without breaking
existing installation processes at the same time.
For example, in this case pip/easy_install could indicate to PYPI what
kind of hashes it accepts (through a header or query param or whatever)
and PyPI could serve it but we'd default to MD5 for now if nothing else
was requested. Please also consider the PEP438 vetted registration of
externals+hashses in this context. Once things and tools are working
nicely we can switch to serving a non-MD5 hash as default after a
sufficient grace period.

Having the improved hashes be opt-in (by the client) strikes me as a
reasonable request.

Yes, this means nothing will actually happen until easy_install/pip
are updated to request those improved hashes and those versions see
significant uptake, but as Holger says, we need to ensure we put
sufficient effort into smoothing out the roller coaster ride that has
been the recent experience of packaging system users.

Cheers,
Nick.

[Donald Stufft]

Post by Nick Coghlan

Post by holger krekel
Actually, i strongly object further backward-incompatible changes.
Please (generally) find a way to introduce improvements without breaking
existing installation processes at the same time.
For example, in this case pip/easy_install could indicate to PYPI what
kind of hashes it accepts (through a header or query param or whatever)
and PyPI could serve it but we'd default to MD5 for now if nothing else
was requested. Please also consider the PEP438 vetted registration of
externals+hashses in this context. Once things and tools are working
nicely we can switch to serving a non-MD5 hash as default after a
sufficient grace period.

Having the improved hashes be opt-in (by the client) strikes me as a
reasonable request.
Yes, this means nothing will actually happen until easy_install/pip
are updated to request those improved hashes and those versions see
significant uptake, but as Holger says, we need to ensure we put
sufficient effort into smoothing out the roller coaster ride that has
been the recent experience of packaging system users.

There's basically zero way for this to fail closed in any of the current installers. The
failure mode is unverified packages not uninstallable packages. I am not aware of
a single installer that mandates the use of a hash. Crate.io has never used md5
hashes and has always used sha256 and I've never received a single report of
an installer being unable to install because of it, which is exactly what I expect.

Indicating via Header or query param pretty much destroys the effectiveness of the
CDN's cache in order to fix a problem with a theoretical (as far as I am aware)
installer that requires a md5 hash (and thus has never worked for any of the externally
hosted packages. Additionally it doesn't account for external urls which need to be
registered *with* a hash.

As far as available attacks, *today* an author could upload a package that has been
created so as to have a sister package with malicious code that has the same hash
allowing them to have a malicious package they can substitute at will without the hashes
changing at all. In the future it's possible that a pre-image attack on MD5 will be found
and then we'll be dealing with this problem then when we've lost all verification on
external urls instead of now when we have time to get external urls to switch.

So by all means I will not migrate us if that's what you want. Old versions of the
installation clients stick around far to long for the opt in mechanism to be much
use. The point of switching was to cover the existing clients as well to narrow
the gap until a new API is developed. Hopefully no one is relying on these
hashes to prevent an author from maliciously injecting a sister package and
hopefully the strength of MD5 holds and no new research is found that blows
it's pre-image attack residence to pieces.

As far as not breaking things goes backwards compatibility has been an important
concern however progress forward *requires* breakage. It is required because there
is a vast array of available ways to have your package and/or hosting configured
many of them horrible practices which need to be killed. Killing them requires breaking
backwards compatibility. You cite SSL, yes SSL has caused a number of errors for
people mostly related to older versions of OpenSSL being unable to use a SSL
certificate but downloading code you're going to execute over plaintext isn't just
bad, it's downright negligent on the part of the toolchain. So that was a required breakage.

You also mention the pip 1.4 *not* installing pre-releases by default. Yes that
broke a handful of packages Supervisor and pytz being the major ones that I've
seen anyone complain about. It was also known ahead of time that this was a
backwards incompatible change (and it was noted as such in the release notes). It
wasn't a surprising outcome. The pip developers "drew a line in the sand" to quote
Paul Moore and I expect pip 1.5 where PEP438 becomes default to break even more
packages from people who just haven't bothered to change their practices until
it's forced on them.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[holger krekel]

Post by Donald Stufft

Post by Nick Coghlan

Post by holger krekel
Actually, i strongly object further backward-incompatible changes.
Please (generally) find a way to introduce improvements without breaking
existing installation processes at the same time.
For example, in this case pip/easy_install could indicate to PYPI what
kind of hashes it accepts (through a header or query param or whatever)
and PyPI could serve it but we'd default to MD5 for now if nothing else
was requested. Please also consider the PEP438 vetted registration of
externals+hashses in this context. Once things and tools are working
nicely we can switch to serving a non-MD5 hash as default after a
sufficient grace period.

Having the improved hashes be opt-in (by the client) strikes me as a
reasonable request.
Yes, this means nothing will actually happen until easy_install/pip
are updated to request those improved hashes and those versions see
significant uptake, but as Holger says, we need to ensure we put
sufficient effort into smoothing out the roller coaster ride that has
been the recent experience of packaging system users.

There's basically zero way for this to fail closed in any of the
current installers. The failure mode is unverified packages not
uninstallable packages. I am not aware of a single installer that
mandates the use of a hash. Crate.io has never used md5 hashes and has
always used sha256 and I've never received a single report of an
installer being unable to install because of it, which is exactly what
I expect.

So you think the worst case for forcing SHA256 hashes is that installers
who don't yet support sha256 hashes would just ignore it (and thus wouldn't
do hash verification)?

Post by Donald Stufft
Indicating via Header or query param pretty much destroys the
effectiveness of the CDN's cache in order to fix a problem with a
theoretical (as far as I am aware) installer that requires a md5 hash
(and thus has never worked for any of the externally hosted packages.
Additionally it doesn't account for external urls which need to be
registered *with* a hash.

Currently there is no hash-type enforcement on registered externals, is there?

Post by Donald Stufft
As far as available attacks, *today* an author could upload a package
that has been created so as to have a sister package with malicious
code that has the same hash allowing them to have a malicious package
they can substitute at will without the hashes changing at all. In the
future it's possible that a pre-image attack on MD5 will be found and
then we'll be dealing with this problem then when we've lost all
verification on external urls instead of now when we have time to get
external urls to switch.

So the attack is a malicious author or someone else modifying an external
release file (either directly on the server or via MITM) while maintaining
the pre-registered MD5 hash, right?

I am currently merely trying to understand more exactly what
you are worried about.

best,
holger

Post by Donald Stufft
So by all means I will not migrate us if that's what you want. Old
versions of the installation clients stick around far to long for the
opt in mechanism to be much use. The point of switching was to cover
the existing clients as well to narrow the gap until a new API is
developed.
Hopefully no one is relying on these hashes to prevent an
author from maliciously injecting a sister package and hopefully the
strength of MD5 holds and no new research is found that blows it's
pre-image attack residence to pieces.
As far as not breaking things goes backwards compatibility has been an
important concern however progress forward *requires* breakage. It is
required because there is a vast array of available ways to have your
package and/or hosting configured many of them horrible practices
which need to be killed. Killing them requires breaking backwards
compatibility. You cite SSL, yes SSL has caused a number of errors for
people mostly related to older versions of OpenSSL being unable to use
a SSL certificate but downloading code you're going to execute over
plaintext isn't just bad, it's downright negligent on the part of the
toolchain. So that was a required breakage.
You also mention the pip 1.4 *not* installing pre-releases by default.
Yes that broke a handful of packages Supervisor and pytz being the
major ones that I've seen anyone complain about. It was also known
ahead of time that this was a backwards incompatible change (and it
was noted as such in the release notes). It wasn't a surprising
outcome. The pip developers "drew a line in the sand" to quote Paul
Moore and I expect pip 1.5 where PEP438 becomes default to break even
more packages from people who just haven't bothered to change their
practices until it's forced on them.
-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Donald Stufft]

Post by holger krekel

Post by Donald Stufft

Post by Nick Coghlan

Post by holger krekel
Actually, i strongly object further backward-incompatible changes.
Please (generally) find a way to introduce improvements without breaking
existing installation processes at the same time.
For example, in this case pip/easy_install could indicate to PYPI what
kind of hashes it accepts (through a header or query param or whatever)
and PyPI could serve it but we'd default to MD5 for now if nothing else
was requested. Please also consider the PEP438 vetted registration of
externals+hashses in this context. Once things and tools are working
nicely we can switch to serving a non-MD5 hash as default after a
sufficient grace period.

Having the improved hashes be opt-in (by the client) strikes me as a
reasonable request.
Yes, this means nothing will actually happen until easy_install/pip
are updated to request those improved hashes and those versions see
significant uptake, but as Holger says, we need to ensure we put
sufficient effort into smoothing out the roller coaster ride that has
been the recent experience of packaging system users.

There's basically zero way for this to fail closed in any of the
current installers. The failure mode is unverified packages not
uninstallable packages. I am not aware of a single installer that
mandates the use of a hash. Crate.io has never used md5 hashes and has
always used sha256 and I've never received a single report of an
installer being unable to install because of it, which is exactly what
I expect.

So you think the worst case for forcing SHA256 hashes is that installers
who don't yet support sha256 hashes would just ignore it (and thus wouldn't
do hash verification)?

Yes. I've been using sha256 on simple.crate.io for over a year and zero people have
ever stated it didn't work for them. This also fits in with my knowledge of how
setuptools and pip works. I know zc.buildout less well but to my knowledge
they simple allow setuptools to handle the downloading.

Post by holger krekel

Post by Donald Stufft
Indicating via Header or query param pretty much destroys the
effectiveness of the CDN's cache in order to fix a problem with a
theoretical (as far as I am aware) installer that requires a md5 hash
(and thus has never worked for any of the externally hosted packages.
Additionally it doesn't account for external urls which need to be
registered *with* a hash.

Currently there is no hash-type enforcement on registered externals, is there?

Registered externals must register with a md5 hash, scraped links and download
urls etc do not require it because they are indirectly added. There is no
verification by PyPI that the given hash matches the package at the
end of the url.

Post by holger krekel

Post by Donald Stufft
As far as available attacks, *today* an author could upload a package
that has been created so as to have a sister package with malicious
code that has the same hash allowing them to have a malicious package
they can substitute at will without the hashes changing at all. In the
future it's possible that a pre-image attack on MD5 will be found and
then we'll be dealing with this problem then when we've lost all
verification on external urls instead of now when we have time to get
external urls to switch.

So the attack is a malicious author or someone else modifying an external
release file (either directly on the server or via MITM) while maintaining
the pre-registered MD5 hash, right?
I am currently merely trying to understand more exactly what
you are worried about.

For any hash function there are two major types of attacks you worry about. The
first is a collision attack which is the ability to generate two arbitrary inputs that
hash to the same thing. The second is a pre-image attack (either first or second
pre-image) which essentially means given an already existing input generate
another input that hashes to the same thing. So basically the difference between
the two attacks are wether you have a hash you're trying to match or if you
just need two inputs that hash to the same thing.

MD5 is currently broken for collision resistance. This means that an author can
generate two packages that hash to the same thing. Once package might be
benign and one might be malicious. Given those two packages people using
the md5 hashes will not be able to differentiate between the benign and the
malicous package.

MD5 is currently *not* broken for pre image resistance. This means that as of
right now someone can not take an already existing package on PyPI and generate
a second package that hashes to the same thing (besides via brute forcing).

So right now, collision attacks possible == yes, pre image attacks possible == no.

However designing secure systems is a practice of building in safety margins. If
someone, for instance, can break 5 rounds of a function you use 15 rounds. With
cryptographic hashes collision attacks are easier than pre-image attacks, so if you
have two functions, one that has a collision attack and one that doesn't you can
generally assume that the one without a collision attack is stronger and has a
longer shelf life.

So the problem with MD5 (ignoring for a second the fact that a collision attack can
be bad on it's own) is that there are no more safety nets. If it gets broken for a pre-image
then there's not likely to be any warning (we've already *had* the warning). It will
just be broken and we will be scrambling to update things then (and hopefully nobody
gets attacked in the meantime).

And I do say *if* because as zooko pointed out, it's not a guarantee that MD5 will
ever lose it's pre-image resistance (which just means that brute forcing is the quickest
way to generate a hash).

Post by holger krekel
best,
holger

Post by Donald Stufft
So by all means I will not migrate us if that's what you want. Old
versions of the installation clients stick around far to long for the
opt in mechanism to be much use. The point of switching was to cover
the existing clients as well to narrow the gap until a new API is
developed.
Hopefully no one is relying on these hashes to prevent an
author from maliciously injecting a sister package and hopefully the
strength of MD5 holds and no new research is found that blows it's
pre-image attack residence to pieces.
As far as not breaking things goes backwards compatibility has been an
important concern however progress forward *requires* breakage. It is
required because there is a vast array of available ways to have your
package and/or hosting configured many of them horrible practices
which need to be killed. Killing them requires breaking backwards
compatibility. You cite SSL, yes SSL has caused a number of errors for
people mostly related to older versions of OpenSSL being unable to use
a SSL certificate but downloading code you're going to execute over
plaintext isn't just bad, it's downright negligent on the part of the
toolchain. So that was a required breakage.
You also mention the pip 1.4 *not* installing pre-releases by default.
Yes that broke a handful of packages Supervisor and pytz being the
major ones that I've seen anyone complain about. It was also known
ahead of time that this was a backwards incompatible change (and it
was noted as such in the release notes). It wasn't a surprising
outcome. The pip developers "drew a line in the sand" to quote Paul
Moore and I expect pip 1.5 where PEP438 becomes default to break even
more packages from people who just haven't bothered to change their
practices until it's forced on them.
-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[holger krekel]

Post by Donald Stufft

Post by holger krekel

Post by Donald Stufft

Post by Nick Coghlan

Post by holger krekel
Actually, i strongly object further backward-incompatible changes.
Please (generally) find a way to introduce improvements without breaking
existing installation processes at the same time.
For example, in this case pip/easy_install could indicate to PYPI what
kind of hashes it accepts (through a header or query param or whatever)
and PyPI could serve it but we'd default to MD5 for now if nothing else
was requested. Please also consider the PEP438 vetted registration of
externals+hashses in this context. Once things and tools are working
nicely we can switch to serving a non-MD5 hash as default after a
sufficient grace period.

Having the improved hashes be opt-in (by the client) strikes me as a
reasonable request.
Yes, this means nothing will actually happen until easy_install/pip
are updated to request those improved hashes and those versions see
significant uptake, but as Holger says, we need to ensure we put
sufficient effort into smoothing out the roller coaster ride that has
been the recent experience of packaging system users.

There's basically zero way for this to fail closed in any of the
current installers. The failure mode is unverified packages not
uninstallable packages. I am not aware of a single installer that
mandates the use of a hash. Crate.io has never used md5 hashes and has
always used sha256 and I've never received a single report of an
installer being unable to install because of it, which is exactly what
I expect.

So you think the worst case for forcing SHA256 hashes is that installers
who don't yet support sha256 hashes would just ignore it (and thus wouldn't
do hash verification)?

Yes. I've been using sha256 on simple.crate.io for over a year and
zero people have ever stated it didn't work for them. This also fits
in with my knowledge of how setuptools and pip works. I know
zc.buildout less well but to my knowledge they simple allow setuptools
to handle the downloading.

Sounds good. Maybe "secondary" tools could get problems, though.
I know for sure that devpi-server might stumble but i can fix that.
Also i remember there were tools that memorized MD5 hashes in requirements
files etc.

Post by Donald Stufft

Post by holger krekel

Post by Donald Stufft
Indicating via Header or query param pretty much destroys the
effectiveness of the CDN's cache in order to fix a problem with a
theoretical (as far as I am aware) installer that requires a md5 hash
(and thus has never worked for any of the externally hosted packages.
Additionally it doesn't account for external urls which need to be
registered *with* a hash.

Currently there is no hash-type enforcement on registered externals, is there?

Registered externals must register with a md5 hash, scraped links and
download urls etc do not require it because they are indirectly added.
There is no verification by PyPI that the given hash matches the
package at the end of the url.

Hum, can we allow submitting multiple hashes? Are there tools already
that help with registering externals?

Post by Donald Stufft

Post by holger krekel

Post by Donald Stufft
As far as available attacks, *today* an author could upload a package
that has been created so as to have a sister package with malicious
code that has the same hash allowing them to have a malicious package
they can substitute at will without the hashes changing at all. In the
future it's possible that a pre-image attack on MD5 will be found and
then we'll be dealing with this problem then when we've lost all
verification on external urls instead of now when we have time to get
external urls to switch.

So the attack is a malicious author or someone else modifying an external
release file (either directly on the server or via MITM) while maintaining
the pre-registered MD5 hash, right?
I am currently merely trying to understand more exactly what
you are worried about.

(...)
MD5 is currently broken for collision resistance. This means that an
author can generate two packages that hash to the same thing. Once
package might be benign and one might be malicious. Given those two
packages people using the md5 hashes will not be able to differentiate
between the benign and the malicous package.

I think we should not pretend that PyPI has (by itself) any safety belts
against malicious authors. There are numerous ways for malicious authors
to do evil if they choose to. The potential ability to fake a package
using a collision attack merely adds another way.

Do you know, btw, if TUF is going to help with any of what we are discussing
here? (I am again a bit lost as to the roadmap wrt to TUF - is there
something?)

cheers,
holger

Post by Donald Stufft
MD5 is currently *not* broken for pre image resistance. This means that as of
right now someone can not take an already existing package on PyPI and generate
a second package that hashes to the same thing (besides via brute forcing).
So right now, collision attacks possible == yes, pre image attacks possible == no.
However designing secure systems is a practice of building in safety margins. If
someone, for instance, can break 5 rounds of a function you use 15 rounds. With
cryptographic hashes collision attacks are easier than pre-image attacks, so if you
have two functions, one that has a collision attack and one that doesn't you can
generally assume that the one without a collision attack is stronger and has a
longer shelf life.
So the problem with MD5 (ignoring for a second the fact that a collision attack can
be bad on it's own) is that there are no more safety nets. If it gets broken for a pre-image
then there's not likely to be any warning (we've already *had* the warning). It will
just be broken and we will be scrambling to update things then (and hopefully nobody
gets attacked in the meantime).
And I do say *if* because as zooko pointed out, it's not a guarantee that MD5 will
ever lose it's pre-image resistance (which just means that brute forcing is the quickest
way to generate a hash).

Post by holger krekel
best,
holger

Post by Donald Stufft
So by all means I will not migrate us if that's what you want. Old
versions of the installation clients stick around far to long for the
opt in mechanism to be much use. The point of switching was to cover
the existing clients as well to narrow the gap until a new API is
developed.
Hopefully no one is relying on these hashes to prevent an
author from maliciously injecting a sister package and hopefully the
strength of MD5 holds and no new research is found that blows it's
pre-image attack residence to pieces.
As far as not breaking things goes backwards compatibility has been an
important concern however progress forward *requires* breakage. It is
required because there is a vast array of available ways to have your
package and/or hosting configured many of them horrible practices
which need to be killed. Killing them requires breaking backwards
compatibility. You cite SSL, yes SSL has caused a number of errors for
people mostly related to older versions of OpenSSL being unable to use
a SSL certificate but downloading code you're going to execute over
plaintext isn't just bad, it's downright negligent on the part of the
toolchain. So that was a required breakage.
You also mention the pip 1.4 *not* installing pre-releases by default.
Yes that broke a handful of packages Supervisor and pytz being the
major ones that I've seen anyone complain about. It was also known
ahead of time that this was a backwards incompatible change (and it
was noted as such in the release notes). It wasn't a surprising
outcome. The pip developers "drew a line in the sand" to quote Paul
Moore and I expect pip 1.5 where PEP438 becomes default to break even
more packages from people who just haven't bothered to change their
practices until it's forced on them.
-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Donald Stufft]

Post by holger krekel

Post by Donald Stufft
Yes. I've been using sha256 on simple.crate.io for over a year and
zero people have ever stated it didn't work for them. This also fits
in with my knowledge of how setuptools and pip works. I know
zc.buildout less well but to my knowledge they simple allow setuptools
to handle the downloading.

Sounds good. Maybe "secondary" tools could get problems, though.
I know for sure that devpi-server might stumble but i can fix that.
Also i remember there were tools that memorized MD5 hashes in requirements
files etc.

The change was nixed but it wasn't about removing the ability for pip and
such to use MD5s. Merely what PyPI serves.

Post by holger krekel

Post by Donald Stufft
Registered externals must register with a md5 hash, scraped links and
download urls etc do not require it because they are indirectly added.
There is no verification by PyPI that the given hash matches the
package at the end of the url.

Hum, can we allow submitting multiple hashes? Are there tools already
that help with registering externals?

Not easily and in a backwards compatible way.

Post by holger krekel

Post by Donald Stufft
MD5 is currently broken for collision resistance. This means that an
author can generate two packages that hash to the same thing. Once
package might be benign and one might be malicious. Given those two
packages people using the md5 hashes will not be able to differentiate
between the benign and the malicous package.

I think we should not pretend that PyPI has (by itself) any safety belts
against malicious authors. There are numerous ways for malicious authors
to do evil if they choose to. The potential ability to fake a package
using a collision attack merely adds another way.

Correct, which is why I tried to be very specific about the types of attacks :)

Post by holger krekel
Do you know, btw, if TUF is going to help with any of what we are discussing
here? (I am again a bit lost as to the roadmap wrt to TUF - is there
something?)

TUF would provide protections against a pre-image attack yes. However it has
it's own problems and is still likely a ways out (if we use it).

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Donald Stufft]

Post by holger krekel
There are >1000 times more people using python packaging tools than
there are ones following the recent distutils-sig/Donald/Nick verdicts
or tool release notes. And heck, even for me being quite involved in all
this stuff, having written a related PEP, it's really hard to see what's
coming and to prevent breakage for my tools's/projects's users.
Therefore, to be honest, at this point i am almost afraid of what is
released/deployed next in PyPI/Installer land. I'd rather like to
get into a "can't wait to get the next release" mode.

I'd also like to point out this. I think you got bitten by one particular backward
incompatibility and are "Once bitten twice shy". I had a number people asking
me near every day how long until pip 1.4 was coming out because they were
excited for the new changes including the improved pre-release handling.

I've personally gotten or seen more complaints over the naming of a variable
in the config file then I have over any changes we've made. The runner up
to that is the fallout from switching to requiring verified SSL.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Tres Seaver]

Post by Donald Stufft
I've personally gotten or seen more complaints over the naming of a
variable in the config file then I have over any changes we've made.
The runner up to that is the fallout from switching to requiring
verified SSL.

The past few months have generated a *lot* of teeth-gnashing /
hair-pulling, especially among "downstream" developers (those unlikely to
be reading this SIG):

- - HTTPS-only PyPY

- - Distribute / setuptools merge, e.g. cratering folks who use a
distro-managed 'python-distribute' package.

- - Pip's new backward-imcompatible "final releases by default".

I think we are going to be in a much better place for all that, but let's
not deny the pain involved for *everybody* in getting there.


Tres.
- --
===================================================================
Tres Seaver [phone] ***@palladion.com
Palladion Software "Excellence by Design" http://palladion.com

[Paul Moore]

Post by Tres Seaver
I think we are going to be in a much better place for all that, but let's
not deny the pain involved for *everybody* in getting there.

Agreed. I think the goal is valid, and the approach is fine. But we need to
do a better job in managing people's expectations. I'd like to see a
roadmap of the various changes planned, as well as some sort of explanation
of how each of the changes contributes towards the end goal.

Personally, none of the changes have detrimentally affected me, so my
opinion is largely theoretical. But even I am getting a little frustrated
by the constant claims that "what we have now is insecure and broken, and
must be fixed ASAP". The reality is that everything's more or less OK -
there's a risk, certainly, and it could be severe, but many, many people
are routinely using PyPI all the time without issues. And telling them that
they are wrong to do so, or that they are being extremely naive over
security, isn't helping.

Paul

[Marcus Smith]

Post by Paul Moore

Post by Tres Seaver
I think we are going to be in a much better place for all that, but let's
not deny the pain involved for *everybody* in getting there.

Agreed. I think the goal is valid, and the approach is fine. But we need
to do a better job in managing people's expectations. I'd like to see a
roadmap of the various changes planned, as well as some sort of explanation
of how each of the changes contributes towards the end goal.

Nick said he's planning a roadmap on the new User Guide (in August I think
after vacation), so that's something.

[Donald Stufft]

But even I am getting a little frustrated by the constant claims that "what we have now is insecure and broken, and must be fixed ASAP". The reality is that everything's more or less OK - there's a risk, certainly, and it could be severe, but many, many people are routinely using PyPI all the time without issues. And telling them that they are wrong to do so, or that they are being extremely naive over security, isn't helping.

This shows a fundamental misunderstanding of how security issues present themselves. Of course things just work for people because security issues are not like regular bugs. They don't negatively affect you until someone attempts to use them to attack you. Keep your front door unlocked on your house and your valuables will remain inside _until_ someone decides to try and rob you. If you wait until people are affected by a security vulnerability then the horse has already fled the pasture and you're just attempting to close the gate after the fact.

I'm pushing hard on doing what we can to secure the infrastructure because this shit matters. Everything is more or less OK, only because no one has decided that people installing from PyPI are not a valuable enough target to go after. Prior to this push that was basically the only thing prevent someone from attacking people, that they had never decided to bother too. We are better, it's somewhat harder now, but in many areas that's still the only thing keeping people safe.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Jim Fulton]

Post by Donald Stufft

But even I am getting a little frustrated by the constant claims that "what
we have now is insecure and broken, and must be fixed ASAP". The reality is
that everything's more or less OK - there's a risk, certainly, and it could
be severe, but many, many people are routinely using PyPI all the time
without issues. And telling them that they are wrong to do so, or that they
are being extremely naive over security, isn't helping.

This shows a fundamental misunderstanding of how security issues present
themselves. Of course things just work for people because security issues
are not like regular bugs. They don't negatively affect you until someone
attempts to use them to attack you. Keep your front door unlocked on your
house and your valuables will remain inside _until_ someone decides to try
and rob you. If you wait until people are affected by a security
vulnerability then the horse has already fled the pasture and you're just
attempting to close the gate after the fact.
I'm pushing hard on doing what we can to secure the infrastructure because
this shit matters. Everything is more or less OK, only because no one has
decided that people installing from PyPI are not a valuable enough target to
go after. Prior to this push that was basically the only thing prevent
someone from attacking people, that they had never decided to bother too. We
are better, it's somewhat harder now, but in many areas that's still the
only thing keeping people safe.

Well said.

Security is a pain, but I'm really glad and appreciate that you and others are
paying attention to it.

Jim

[zooko]

Folks:

dstufft already correctly explained that relying on MD5 allows for
"doppelganger" packages -- two (or more) packages which are engineered at birth
to have the same hash as each other. It isn't clear to me that this can be used
for evil, but it isn't obvious that it *can't* be used for evil, either. So it
would certainly be helpful to upgrade the hash function so that we don't have
to think about that anymore, but in my opinion it is not an emergency.

I'd like to push back on the other risk, that someone might figure out how to
make MD5 second-pre-images. I don't think this is a risk that we need to
urgently address, and I've written a short note explaining why. This note is
incomplete, badly edited, has not been peer-reviewed, and is not ready for
publication, but I thought it might help folks evaluate how urgent it is to
upgrade from MD5, so here it is.

Regards,

Zooko

[Donald Stufft]

Post by zooko
I'd like to push back on the other risk, that someone might figure out how to
make MD5 second-pre-images. I don't think this is a risk that we need to
urgently address, and I've written a short note explaining why. This note is
incomplete, badly edited, has not been peer-reviewed, and is not ready for
publication, but I thought it might help folks evaluate how urgent it is to
upgrade from MD5, so here it is.

I don't think it's urgent to fix it, but I think it's a good security hardening effort
with very little downside and very little chance of regression. However, as I
said if Holger, or anyone else, has a concern about the affects of adding this
bit of security hardening to give us a safety net again then I simply won't do
it in the simple API.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Donald Stufft]

Post by Donald Stufft

Post by zooko
I'd like to push back on the other risk, that someone might figure out how to
make MD5 second-pre-images. I don't think this is a risk that we need to
urgently address, and I've written a short note explaining why. This note is
incomplete, badly edited, has not been peer-reviewed, and is not ready for
publication, but I thought it might help folks evaluate how urgent it is to
upgrade from MD5, so here it is.

I don't think it's urgent to fix it, but I think it's a good security hardening effort
with very little downside and very little chance of regression. However, as I
said if Holger, or anyone else, has a concern about the affects of adding this
bit of security hardening to give us a safety net again then I simply won't do
it in the simple API.
-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________
http://mail.python.org/mailman/listinfo/distutils-sig

Somewhat relevant to the question at hand: http://valerieaurora.org/hash.html

(Yes it lists sha-2 as weakened, which it is. However sha-3 isn't widespread enough for us :( )

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[zooko]

Post by Donald Stufft
Somewhat relevant to the question at hand: http://valerieaurora.org/hash.html

Heh heh. That page is cited in my note. My note is kind of a response to that
page, showing that the history of pre-image attacks is completely different
than the history of collision attacks.

Post by Donald Stufft
(Yes it lists sha-2 as weakened, which it is. However sha-3 isn't widespread
enough for us :( )

There's no reason to worry about SHA-2. In my opinion, there's no particular
reason to think that it will be made vulnerable to collisions within the next
decade!

By the way, I'm a co-author of a secure hash function -- BLAKE2:

https://blake2.net/

The intent of BLAKE2 is to be as secure as SHA-3 but as fast as MD5. Not only
is it as fast as MD5, but it also has an optional parallel mode that can go 4
or 8 times as fast as MD5 by using 4 or 8 CPU cores!

It is currently being adopted for uses like data deduplication, archiving, and
distributed filesystems, where the data can be large (terabytes or more), and
the performance of the hash function is a bottleneck.

I don't think Python packaging has such needs, and BLAKE2 is not a standard
like SHA-2 and SHA-3, so I'm not pushing to add support for it.

Regards,

Zooko

[Ronald Oussoren]

Post by Donald Stufft

Post by Donald Stufft

Post by zooko
I'd like to push back on the other risk, that someone might figure out how to
make MD5 second-pre-images. I don't think this is a risk that we need to
urgently address, and I've written a short note explaining why. This note is
incomplete, badly edited, has not been peer-reviewed, and is not ready for
publication, but I thought it might help folks evaluate how urgent it is to
upgrade from MD5, so here it is.

I don't think it's urgent to fix it, but I think it's a good security hardening effort
with very little downside and very little chance of regression. However, as I
said if Holger, or anyone else, has a concern about the affects of adding this
bit of security hardening to give us a safety net again then I simply won't do
it in the simple API.
-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________
http://mail.python.org/mailman/listinfo/distutils-sig

Somewhat relevant to the question at hand: http://valerieaurora.org/hash.html
(Yes it lists sha-2 as weakened, which it is. However sha-3 isn't widespread enough for us :( )

That SHA-3 isn't widespread yet is not a surprise, AFAIK it isn't even a standard yet :-). According to <http://csrc.nist.gov/groups/ST/hash/sha-3/timeline_fips.html> the standard will be finalized in Q2 2014.

BTW. I agree that the MD5 checksums on PyPI will have to go some time, and it would be nice if the replacement scheme had a way to use multiple hashes to make it easier to switch to a hash in future. I know to little of the setuptools and pip implementations to have anything useful to add to the discussion about the timing for this.

Ronald

Post by Donald Stufft
-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________
http://mail.python.org/mailman/listinfo/distutils-sig

[Nick Coghlan]

Post by Donald Stufft

Post by zooko
I'd like to push back on the other risk, that someone might figure out how to
make MD5 second-pre-images. I don't think this is a risk that we need to
urgently address, and I've written a short note explaining why. This note is
incomplete, badly edited, has not been peer-reviewed, and is not ready for
publication, but I thought it might help folks evaluate how urgent it is to
upgrade from MD5, so here it is.

I don't think it's urgent to fix it, but I think it's a good security hardening effort
with very little downside and very little chance of regression. However, as I
said if Holger, or anyone else, has a concern about the affects of adding this
bit of security hardening to give us a safety net again then I simply won't do
it in the simple API.

I'm thinking that may be the way to go - treat verified SSL as our final
stop-gap for the simple API and focus on hardening the next generation APIs.

This is more for social reasons than strictly technical ones. I think
you're right this particular change is unlikely to break anything, but
there are also enough genuinely essential changes needed that we should
avoid unnecessary flux in other areas.

In this case, I think the need for a pre-image attack that still produces a
working download and an old installer that isn't using verified SSL but can
check SHA256 hashes reduces the attack window to a point where I'm prepared
to live with the use of MD5 as a known risk.

Cheers,
Nick.

Post by Donald Stufft
-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________
http://mail.python.org/mailman/listinfo/distutils-sig

[Antoine Pitrou]

Post by Paul Moore
Personally, none of the changes have detrimentally affected me, so my
opinion is largely theoretical. But even I am getting a little frustrated
by the constant claims that "what we have now is insecure and broken, and
must be fixed ASAP".

FWIW, +1. You may be paranoid, but not everyone has to be (or suffer the
consequences of it). Security issues should be fixed without breaking things
in a hassle (which is the policy we followed e.g. for the ssl module, or hash
randomization).

The whole python.org infrastructure is built on an OS kernel written by someone
who thinks security issues are normal bugs. AFAIK there is no plan to switch to
OpenBSD.

Regards

Antoine.

[Noah Kantrowitz]

Post by Antoine Pitrou

Post by Paul Moore
Personally, none of the changes have detrimentally affected me, so my
opinion is largely theoretical. But even I am getting a little frustrated
by the constant claims that "what we have now is insecure and broken, and
must be fixed ASAP".

FWIW, +1. You may be paranoid, but not everyone has to be (or suffer the
consequences of it). Security issues should be fixed without breaking things
in a hassle (which is the policy we followed e.g. for the ssl module, or hash
randomization).

You missed a key word "… when possible". If there is a problem we will fix it, when we can do that in a way that minimizes breakages we will do that. Its all just about cost-benefit, and when you are talking about "executing code downloaded from the internet" it becomes quite easy to see benefits outweighing costs even with pretty major UX changes. Not something we do lightly, but status quo does not win here, sorry.

Post by Antoine Pitrou
The whole python.org infrastructure is built on an OS kernel written by someone
who thinks security issues are normal bugs. AFAIK there is no plan to switch to
OpenBSD.

This is news to me, we specifically run Ubuntu LTS because Canonical's security response team has a proven track record of handling issues. If you mean that Linus doesn't handle security issues well, then it is fortunate indeed that we don't actually use his software.

--Noah

[Donald Stufft]

Post by Noah Kantrowitz

Post by Antoine Pitrou

Post by Paul Moore
Personally, none of the changes have detrimentally affected me, so my
opinion is largely theoretical. But even I am getting a little frustrated
by the constant claims that "what we have now is insecure and broken, and
must be fixed ASAP".

FWIW, +1. You may be paranoid, but not everyone has to be (or suffer the
consequences of it). Security issues should be fixed without breaking things
in a hassle (which is the policy we followed e.g. for the ssl module, or hash
randomization).

You missed a key word "… when possible". If there is a problem we will fix it, when we can do that in a way that minimizes breakages we will do that. Its all just about cost-benefit, and when you are talking about "executing code downloaded from the internet" it becomes quite easy to see benefits outweighing costs even with pretty major UX changes. Not something we do lightly, but status quo does not win here, sorry.

Basically said it better than I could :)

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Antoine Pitrou]

Post by Noah Kantrowitz

Post by Antoine Pitrou
The whole python.org infrastructure is built on an OS kernel written by someone
who thinks security issues are normal bugs. AFAIK there is no plan to switch to
OpenBSD.

This is news to me, we specifically run Ubuntu LTS because Canonical's

security response team has a proven

Post by Noah Kantrowitz
track record of handling issues. If you mean that Linus doesn't handle

security issues well, then it is

Post by Noah Kantrowitz
fortunate indeed that we don't actually use his software.

Did you already forget what the discussion is about?
Security/bugfix Ubuntu LTS updates don't break compatibility for the sake of
hardening
things, which is the whole point.

(As for the idea that using "Canonical's kernel" amounts to not using "Linus'
software", that's a rather unorthodox notion of authorship. It's very likely
Canonical
doesn't change more than 1% LOC in the kernel, so you're still bound to Linus'
decisions for at least 99% of the code - and even probably for the remaining 1%,
since Canonical's version won't be massively divergent.)

Regards

Antoine.

[Noah Kantrowitz]

Post by Donald Stufft

Post by Noah Kantrowitz

Post by Antoine Pitrou
The whole python.org infrastructure is built on an OS kernel written by

someone

Post by Noah Kantrowitz

Post by Antoine Pitrou
who thinks security issues are normal bugs. AFAIK there is no plan to

switch to

Post by Noah Kantrowitz

Post by Antoine Pitrou
OpenBSD.

This is news to me, we specifically run Ubuntu LTS because Canonical's

security response team has a proven

Post by Noah Kantrowitz
track record of handling issues. If you mean that Linus doesn't handle

security issues well, then it is

Post by Noah Kantrowitz
fortunate indeed that we don't actually use his software.

Did you already forget what the discussion is about?
Security/bugfix Ubuntu LTS updates don't break compatibility for the sake of
hardening
things, which is the whole point.

Again, speaking as the guy that has to clean up the mess when they do break compat, I promise you they do. Same deal, they only break compat when keeping compat would present a threat to users, which is quite often the case with security bugs. They are fortunately a bit further ahead of us on the long tail of finding problems, so this is far less frequent than it was in years past. We will get there too, but like I said, status quo is not a defense here, just strap in and hang on.

--Noah

[Donald Stufft]

Post by Donald Stufft

Post by Noah Kantrowitz

Post by Antoine Pitrou
The whole python.org infrastructure is built on an OS kernel written by

someone

Post by Noah Kantrowitz

Post by Antoine Pitrou
who thinks security issues are normal bugs. AFAIK there is no plan to

switch to

Post by Noah Kantrowitz

Post by Antoine Pitrou
OpenBSD.

This is news to me, we specifically run Ubuntu LTS because Canonical's

security response team has a proven

Post by Noah Kantrowitz
track record of handling issues. If you mean that Linus doesn't handle

security issues well, then it is

Post by Noah Kantrowitz
fortunate indeed that we don't actually use his software.

Did you already forget what the discussion is about?
Security/bugfix Ubuntu LTS updates don't break compatibility for the sake of
hardening
things, which is the whole point.

Well for one PyPI doesn't have releases so there is no LTS or not LTS, it's just
what's being served so there's no simple place to break backwards compatibility.

As far as forgetting what's being discussed here then it sounds like you've apparently
missed the fact I already conceded the change to MD5 and further more this thread
was explicitly split off from the MD5 request because, as far as I can tell, Holger
wanted to discuss the broader topic of compatibility in general and not just specific
to this particular issue.

Post by Donald Stufft
(As for the idea that using "Canonical's kernel" amounts to not using "Linus'
software", that's a rather unorthodox notion of authorship. It's very likely
Canonical
doesn't change more than 1% LOC in the kernel, so you're still bound to Linus'
decisions for at least 99% of the code - and even probably for the remaining 1%,
since Canonical's version won't be massively divergent.)
Regards
Antoine.
_______________________________________________
http://mail.python.org/mailman/listinfo/distutils-sig

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Donald Stufft]

Post by Antoine Pitrou

Post by Paul Moore
Personally, none of the changes have detrimentally affected me, so my
opinion is largely theoretical. But even I am getting a little frustrated
by the constant claims that "what we have now is insecure and broken, and
must be fixed ASAP".

FWIW, +1. You may be paranoid, but not everyone has to be (or suffer the
consequences of it). Security issues should be fixed without breaking things
in a hassle (which is the policy we followed e.g. for the ssl module, or hash
randomization).

People are generally not paranoid until they've been successfully attacked. I
*will* advocate and push for breaking things where security is concerned because
regardless of if you care or not, a lot of people *do* care and the nature of the
beast is that you're only as strong as the weakest link. This particular change
wasn't an immediate vulnerability that I felt was urgent, hence why I've backed
off on it when people were concerned about the backwards compat implications. I
will not back off when it comes to issues that *do* have an immediate or near
term issue, regardless of if some people don't care or not.

Post by Antoine Pitrou
The whole python.org infrastructure is built on an OS kernel written by someone
who thinks security issues are normal bugs. AFAIK there is no plan to switch to
OpenBSD.

So classifying bugs as security vs "normal" is supposed to make it easier on people.
The thought is that creating new releases and applying updates is a time consuming
process and often times requires things such as reboots or service restarts so by
dividing issues up into security vs not security the amount of disruption can be
minimized for only "important" updates. There's actually pretty strong evidence that
shows the process of classifying bugs as security bugs is a harmful process and that
all updates should be treated the same because it's often times not immediately
obvious what the security implications are, even to security experts[1].

I'm sure your dig at the OS is supposed to be some sort of masterstroke about how
we're not being as secure as possible anyways however I would contest that
OpenBSD is actually more secure. It's major claim to fame is that they haven't had
a vulnerably in the OpenBSD base system in "a heck of a long time". The problem
is the OpenBSD base system is terribly small and that claim cannot be made
once you include their packages. Further more at the last I checked OpenBSD
does not provide (although this may have changed) and abilities to do MAC
which means you're relying entirely on an attackers ability to *not* get in versus
providing fail safes to contain an attack once it's happened. Infrastructure is not
using MAC currently but I would love to get us to that point as well.


[1] citeseerx.ist.psu.edu/viewdoc/download;jsessionid=7B6E224144709E99B7FAEBFC497621A1?doi=10.1.1.148.9757&rep=rep1&type=pdf

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Antoine Pitrou]

I *will* advocate and push for breaking things where security is concerned

because

regardless of if you care or not, a lot of people *do* care and the nature of the
beast is that you're only as strong as the weakest link.

That's nice, but you're not alone here, so whatever you want to "push for"
needn't
always happen.

There's actually pretty strong evidence that
shows the process of classifying bugs as security bugs is a harmful process and that
all updates should be treated the same because it's often times not immediately
obvious what the security implications are, even to security experts[1].

Doesn't it contradict your own stance on the subject?

("This shows a fundamental misunderstanding of how security issues present
themselves. Of course things just work for people because security issues
are not
like regular bugs" - which is a flawed argument btw. Many bugs have random or
rare occurrences - not just security issues)

I'm sure your dig at the OS is supposed to be some sort of masterstroke about how
we're not being as secure as possible anyways however I would contest that
OpenBSD is actually more secure.

WTF are you talking about? No it's not. I'm simply pointing out that, for
some strange reason, you decided to trust an OS whose author has very
different views on how to fix
security issues than you have.

Regards

Antoine.

[Donald Stufft]

Post by Donald Stufft

I *will* advocate and push for breaking things where security is concerned

because

regardless of if you care or not, a lot of people *do* care and the nature

of the

beast is that you're only as strong as the weakest link.

That's nice, but you're not alone here, so whatever you want to "push for"
needn't
always happen.

I have zero qualms about releasing a full disclosure along with working exploits
into the wild for a security vulnerability that people block me on. If I'm unable
to rectify the problem I will make sure that everyone *knows* about the problem.

Post by Donald Stufft

There's actually pretty strong evidence that
shows the process of classifying bugs as security bugs is a harmful

process and that

all updates should be treated the same because it's often times not

immediately

obvious what the security implications are, even to security experts[1].

Doesn't it contradict your own stance on the subject?
("This shows a fundamental misunderstanding of how security issues present
themselves. Of course things just work for people because security issues
are not
like regular bugs" - which is a flawed argument btw. Many bugs have random or
rare occurrences - not just security issues)

No? How you treat a bug and how they present themselves are not the same thing?

Even a random occurrence will break for some percentage of people using
the software some percentage of the time. If it didn't then it's unlikely anyone
would notice it. Security vulnerabilities typically won't break until someone actively
tries to break them.

Post by Donald Stufft

I'm sure your dig at the OS is supposed to be some sort of masterstroke

about how

we're not being as secure as possible anyways however I would contest that
OpenBSD is actually more secure.

WTF are you talking about? No it's not. I'm simply pointing out that, for
some strange reason, you decided to trust an OS whose author has very
different views on how to fix
security issues than you have.

Well the Kernel isn't the OS, it's part of the OS and we run an OS whose
authors actually care a whole lot about security.

Post by Donald Stufft
Regards
Antoine.
_______________________________________________
http://mail.python.org/mailman/listinfo/distutils-sig

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Antoine Pitrou]

Post by Donald Stufft
I have zero qualms about releasing a full disclosure along with working exploits
into the wild for a security vulnerability that people block me on. If I'm unable
to rectify the problem I will make sure that everyone *knows* about the problem.

I don't know what I'm supposed to infer from such a statement, except that I
probably don't want to trust you. You might think that "publish[ing] working
exploits into the wild" is some kind of heroic, altruistic act, but I think few
people would agree.

Post by Donald Stufft
Even a random occurrence will break for some percentage of people using
the software some percentage of the time. If it didn't then it's unlikely anyone
would notice it. Security vulnerabilities typically won't break until someone actively
tries to break them.

You're mistaken. Bugs can sometimes be fixed preemptively, even before
they're noticed
in the wild (by means of perusing the code and noticing an issue, for example).
Which also includes, of course, security issues (which often get fixed
before they
ever get exploited).

Regards

Antoine.

[Noah Kantrowitz]

Post by Donald Stufft

Post by Donald Stufft
I have zero qualms about releasing a full disclosure along with working

exploits

Post by Donald Stufft
into the wild for a security vulnerability that people block me on. If I'm

unable

Post by Donald Stufft
to rectify the problem I will make sure that everyone *knows* about the

problem.
I don't know what I'm supposed to infer from such a statement, except that I
probably don't want to trust you. You might think that "publish[ing] working
exploits into the wild" is some kind of heroic, altruistic act, but I think few
people would agree.

No, this is the standard for security researchers. If the vendor ignores the reported exploit for long enough, they go public and try to make sure users understand the risks and how to mitigate them in the time it takes the vendor to fix it.

--Noah

[Donald Stufft]

Post by Antoine Pitrou
I don't know what I'm supposed to infer from such a statement, except that I
probably don't want to trust you. You might think that "publish[ing] working
exploits into the wild" is some kind of heroic, altruistic act, but I think few
people would agree.

Full Disclosure is a common practice amongst security professionals when
the upstream project is unwilling to rectify the problem. So yes I do think
the practice of Full Disclosure is an altruistic act and often times the only
thing that gets people who don't care to pull their head out of the sand
and actually care.

If you don't believe my words on it here's an essay by Bruce Schneier
one of the foremost experts on security and a well respected and well
trusted member of the security community.

https://www.schneier.com/essay-146.html

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Antoine Pitrou]

Post by Antoine Pitrou
I don't know what I'm supposed to infer from such a statement, except that

Iprobably don't want to trust you. You might think that "publish[ing]
workingexploits into the wild" is some kind of heroic, altruistic act, but I
think fewpeople would agree.

Post by Antoine Pitrou
Full Disclosure is a common practice amongst security professionals
whenthe upstream project is unwilling to rectify the problem. So yes I do

think

Post by Antoine Pitrou
the practice of Full Disclosure is an altruistic act and often times the only
thing that gets people who don't care to pull their head out of the sand
and actually care.

You don't happen to be a random security professional, you are actually part
of that upstream project and you have access to non-public (possibly
confidential)
data about its infrastructure, which gives you responsibilities towards your
peers.

I don't think I would be the only one to be angry if an infrastructure member
starting publishing working exploits for unfixed vulnerabilities in the pdo
infrastructure. It is a completely irresponsible way to act when you are part
of a project or community.

Regards

Antoine.

[Donald Stufft]

Post by Antoine Pitrou

Post by Antoine Pitrou
I don't know what I'm supposed to infer from such a statement, except that

Iprobably don't want to trust you. You might think that "publish[ing]
workingexploits into the wild" is some kind of heroic, altruistic act, but I
think fewpeople would agree.

Post by Antoine Pitrou
Full Disclosure is a common practice amongst security professionals
whenthe upstream project is unwilling to rectify the problem. So yes I do

think

Post by Antoine Pitrou
the practice of Full Disclosure is an altruistic act and often times the only
thing that gets people who don't care to pull their head out of the sand
and actually care.

You don't happen to be a random security professional, you are actually part
of that upstream project and you have access to non-public (possibly
confidential)
data about its infrastructure, which gives you responsibilities towards your
peers.
I don't think I would be the only one to be angry if an infrastructure member
starting publishing working exploits for unfixed vulnerabilities in the pdo
infrastructure. It is a completely irresponsible way to act when you are part
of a project or community.

I don't really care if you'd be angry. The point of Full Disclosure (and it's cousin
Responsible Disclosure) is to A) Inform everyone involved that they are taking
a huge risk by using a particular thing and B) Provide incentive to people to
fix their shit. If others are preventing fixes from landing then both reasons still
apply wether the reporter is involved with the project or not.

If I can find a vulnerability then so can someone else. Someone who won't
inform people and will use it to maliciously attack people. If you feel I'd be
overstepping my bounds then complain to my superiors, Richard/Nick on the
packaging side of things and Noah on the Infrastructure team side of things.

I'll continue to do what I feel best serves the community for as long as I have
the ability to do so. Which I believe is work on improving these issues, fight
and advocate for the important ones, accept defeat on less important ones,
and, if necessary, issue a Full Disclosure.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Antoine Pitrou]

Post by Donald Stufft

Post by Antoine Pitrou
You don't happen to be a random security professional, you are actually part
of that upstream project and you have access to non-public (possibly
confidential)
data about its infrastructure, which gives you responsibilities towards your
peers.
I don't think I would be the only one to be angry if an infrastructure member
starting publishing working exploits for unfixed vulnerabilities in the pdo
infrastructure. It is a completely irresponsible way to act when you are part
of a project or community.

I don't really care if you'd be angry.

Great to hear. This mindset is typical of many "security specialists":
you're ready to tell everyone to go f*** themselves (I don't know how to
voice this differently) if you think you have a higher mission to
denounce some vulnerability.

Post by Donald Stufft
The point of Full Disclosure (and it's cousin
Responsible Disclosure) is to A) Inform everyone involved that they are taking
a huge risk by using a particular thing and B) Provide incentive to people to
fix their shit.

This does not necessarily involve publishing working exploits. By giving out
code that can immediately attack and bring down the pdo infrastructure, you
would be doing something else than merely "informing the public".

(neither Bruce Schneier nor Wikipedia states that Full Disclosure implies
publishing working exploits, btw. I suppose it means there is at the
minimum some contention, rather than consensus, over the issue.)

Post by Donald Stufft
If I can find a vulnerability then so can someone else.

You may (and probably do) have domain knowledge and inside knowledge that
others don't.

Post by Donald Stufft
If you feel I'd be
overstepping my bounds then complain to my superiors, Richard/Nick on the
packaging side of things and Noah on the Infrastructure team side of things.

"Superiors"? Are you making things up, or do you have an org chart to back that
up? :-)
(regardless, I would be surprised if any of those ordered *you* to publish an
exploit, rather than take the responsibility of doing it themselves - or,
rather, not doing it at all)

Regards

Antoine.

[Donald Stufft]

Post by Donald Stufft

Post by Donald Stufft

Post by Antoine Pitrou
You don't happen to be a random security professional, you are actually part
of that upstream project and you have access to non-public (possibly
confidential)
data about its infrastructure, which gives you responsibilities towards your
peers.
I don't think I would be the only one to be angry if an infrastructure

member

Post by Donald Stufft

Post by Antoine Pitrou
starting publishing working exploits for unfixed vulnerabilities in the pdo
infrastructure. It is a completely irresponsible way to act when you are

part

Post by Donald Stufft

Post by Antoine Pitrou
of a project or community.

I don't really care if you'd be angry.

you're ready to tell everyone to go f*** themselves (I don't know how to
voice this differently) if you think you have a higher mission to
denounce some vulnerability.

Keeping an issue secret doesn't make people more secure, it only prevents
them from making an informed decision about the things they use.

Post by Donald Stufft

Post by Donald Stufft
The point of Full Disclosure (and it's cousin
Responsible Disclosure) is to A) Inform everyone involved that they are taking
a huge risk by using a particular thing and B) Provide incentive to people to
fix their shit.

This does not necessarily involve publishing working exploits. By giving out
code that can immediately attack and bring down the pdo infrastructure, you
would be doing something else than merely "informing the public".
(neither Bruce Schneier nor Wikipedia states that Full Disclosure implies
publishing working exploits, btw. I suppose it means there is at the
minimum some contention, rather than consensus, over the issue.)

Partial disclosure is typically publishing enough information to allow people
to make an informed choice but (hopefully) not enough information to allow
others to attack it.

Full Disclosure implies laying out the full details of the vulnerability. It doesn't
necessarily mean wrapping it up in ready to run code but they almost always
provide enough information so that anyone with a cursory understanding of
programming (or the area it affects depending on how hard it is to replicate)
can implement the attack and reproduce it.

The problem with Partial Disclosure is that often times vendors/upstream/whatever
will simply call the problem theoretical and still attempt to not fix things.

Post by Donald Stufft

Post by Donald Stufft
If I can find a vulnerability then so can someone else.

You may (and probably do) have domain knowledge and inside knowledge that
others don't.

There's not really any inside knowledge. Everything is OSS.

Domain knowledge? Sure, but I really hope we aren't relying on people not
knowing enough about how the tooling works to exploit it.

Post by Donald Stufft

Post by Donald Stufft
If you feel I'd be
overstepping my bounds then complain to my superiors, Richard/Nick on the
packaging side of things and Noah on the Infrastructure team side of things.

"Superiors"? Are you making things up, or do you have an org chart to back that
up? :-)
(regardless, I would be surprised if any of those ordered *you* to publish an
exploit, rather than take the responsibility of doing it themselves - or,
rather, not doing it at all)

Nick is the BDFL delegate for packaging tooling and Richard is the same for PyPI
and I generally consider my involvement as a PyPI admin to be under him.

Noah is in charge of the Infra team of which I am a member.

Post by Donald Stufft
Regards
Antoine.
_______________________________________________
http://mail.python.org/mailman/listinfo/distutils-sig

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Nick Coghlan]

Post by Donald Stufft

Post by Donald Stufft

Post by Antoine Pitrou
You don't happen to be a random security professional, you are actually part
of that upstream project and you have access to non-public (possibly
confidential)
data about its infrastructure, which gives you responsibilities towards your
peers.
I don't think I would be the only one to be angry if an infrastructure

member

Post by Donald Stufft

Post by Antoine Pitrou
starting publishing working exploits for unfixed vulnerabilities in the pdo
infrastructure. It is a completely irresponsible way to act when you

are

Post by Donald Stufft
part

Post by Donald Stufft

Post by Antoine Pitrou
of a project or community.

I don't really care if you'd be angry.

you're ready to tell everyone to go f*** themselves (I don't know how to
voice this differently) if you think you have a higher mission to
denounce some vulnerability.

Post by Donald Stufft
The point of Full Disclosure (and it's cousin
Responsible Disclosure) is to A) Inform everyone involved that they are taking
a huge risk by using a particular thing and B) Provide incentive to people to
fix their shit.

This does not necessarily involve publishing working exploits. By giving out
code that can immediately attack and bring down the pdo infrastructure, you
would be doing something else than merely "informing the public".
(neither Bruce Schneier nor Wikipedia states that Full Disclosure implies
publishing working exploits, btw. I suppose it means there is at the
minimum some contention, rather than consensus, over the issue.)

Post by Donald Stufft
If I can find a vulnerability then so can someone else.

You may (and probably do) have domain knowledge and inside knowledge that
others don't.

Post by Donald Stufft
If you feel I'd be
overstepping my bounds then complain to my superiors, Richard/Nick on the
packaging side of things and Noah on the Infrastructure team side of things.

"Superiors"? Are you making things up, or do you have an org chart to back that
up? :-)

Effectively he does, yes. Richard is responsible for approving PyPI API
changes (including PyPI specific PEPs), I'm BDFL delegate for other
packaging PEPs and Noah has final say over the operation of the
python.orginfrastructure.

One or more of us are the ones that need to say yes on potentially
controversial changes, so the responsibility for any mistakes ultimately
lies with us, rather than Donald (and I'm greatly appreciative of the huge
amount of work he is putting into improving the PyPI security story).

Post by Donald Stufft
(regardless, I would be surprised if any of those ordered *you* to publish an
exploit, rather than take the responsibility of doing it themselves - or,
rather, not doing it at all)

If Donald informed us of a vulnerability and we refused to allow him (or
anyone else) to take the necessary steps to close it, then he would be
*completely* justified in publishing full details of the vulnerability, up
to and including working exploit code.

It won't come to that though, because we're taking this seriously and
closing security holes as quickly as is feasible while still ensuring a
reasonable level of backwards compatibility :)

Cheers,
Nick.

Post by Donald Stufft
Regards
Antoine.
_______________________________________________
http://mail.python.org/mailman/listinfo/distutils-sig

[Donald Stufft]

If Donald informed us of a vulnerability and we refused to allow him (or anyone else) to take the necessary steps to close it, then he would be *completely* justified in publishing full details of the vulnerability, up to and including working exploit code.
It won't come to that though, because we're taking this seriously and closing security holes as quickly as is feasible while still ensuring a reasonable level of backwards compatibility :)

This basically.

Maybe I'm not being clear because I have a headache and I'm reading too
much into things because I'm sensitive to being shutdown on efforts to fix these
things*. I don't expect with Nick, Richard, and Noah to ever need to do a Full
Disclosure. I was only trying to be clear about what I consider my escalation path
to be if a current, or near future vulnerability is forced to remain open.


* I started trying to push for this ~2 years ago and got repeatedly shut down,
for one reason or another. Which lead to to create Crate.io. It's only been
relatively recently that I've been given permission to actually fix things.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Lennart Regebro]

Post by Donald Stufft
I have zero qualms about releasing a full disclosure along with working exploits
into the wild for a security vulnerability that people block me on.

This is a moot point, as nobody is going to block anyone. The
discussion is silly.

//Lennart

[Donald Stufft]

Post by Lennart Regebro

Post by Donald Stufft
I have zero qualms about releasing a full disclosure along with working exploits
into the wild for a security vulnerability that people block me on.

This is a moot point, as nobody is going to block anyone. The
discussion is silly.
//Lennart

*Shrug*

If someone says people who don't care shouldn't have to suffer the consequences i
assume that means they are advocating for no breakages in API (else the hypothetical
person would be suffering). If you can't break fundamentally insecure things then that's
a blockade on fixing the system. If that's not what the comment was proposing then I
have no idea what it means.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Paul Moore]

Post by Paul Moore

Post by Antoine Pitrou

Post by Paul Moore
Personally, none of the changes have detrimentally affected me, so my
opinion is largely theoretical. But even I am getting a little

frustrated

Post by Antoine Pitrou

Post by Paul Moore
by the constant claims that "what we have now is insecure and broken,

and

Post by Antoine Pitrou

Post by Paul Moore
must be fixed ASAP".

FWIW, +1. You may be paranoid, but not everyone has to be (or suffer the
consequences of it). Security issues should be fixed without breaking

things

Post by Antoine Pitrou
in a hassle (which is the policy we followed e.g. for the ssl module, or

hash

Post by Antoine Pitrou
randomization).

People are generally not paranoid until they've been successfully attacked. I
*will* advocate and push for breaking things where security is concerned because
regardless of if you care or not, a lot of people *do* care and the nature of the
beast is that you're only as strong as the weakest link. This particular change
wasn't an immediate vulnerability that I felt was urgent, hence why I've backed
off on it when people were concerned about the backwards compat implications. I
will not back off when it comes to issues that *do* have an immediate or near
term issue, regardless of if some people don't care or not.

And in case it's not obvious, I think this is important. We need to have
this sort of debate, certainly, but it won't happen without someone
advocating (and implementing!) the changes, so many thanks for being that
person and putting up with the flak.

Paul

[Nick Coghlan]

Post by Paul Moore
And in case it's not obvious, I think this is important. We need to have
this sort of debate, certainly, but it won't happen without someone
advocating (and implementing!) the changes, so many thanks for being that
person and putting up with the flak.

Part of the idea of having myself and Richard in the approval chain
for packaging ecosystem and PyPI changes is to encourage people to get
angry at *us* for saying "Yes, let's do that" when things break,
rather than getting mad at the people actually volunteering to do the
work needed to improve the Python community's software distribution
infrastructure.

That's the plan, anyway, even if it isn't always all that effective in
practice :(

Cheers,
Nick.

[Donald Stufft]

Post by Nick Coghlan

Post by Paul Moore
And in case it's not obvious, I think this is important. We need to have
this sort of debate, certainly, but it won't happen without someone
advocating (and implementing!) the changes, so many thanks for being that
person and putting up with the flak.

Part of the idea of having myself and Richard in the approval chain
for packaging ecosystem and PyPI changes is to encourage people to get
angry at *us* for saying "Yes, let's do that" when things break,
rather than getting mad at the people actually volunteering to do the
work needed to improve the Python community's software distribution
infrastructure.
That's the plan, anyway, even if it isn't always all that effective in
practice :(

Heh, I'm pretty good at getting yelled at :)

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[PJ Eby]

Post by Donald Stufft
Heh, I'm pretty good at getting yelled at :)

Nick is also pretty good at making people feel like he both knows and
*cares* about their breakage, and isn't just dismissing their concerns
as trivial or unimportant. Breakage isn't trivial or unimportant to
the person who's yelling, so this is an important
community-maintenance skill. It builds trust, and reduces the total
amount of yelling.

[Donald Stufft]

Post by PJ Eby

Post by Donald Stufft
Heh, I'm pretty good at getting yelled at :)

Nick is also pretty good at making people feel like he both knows and
*cares* about their breakage, and isn't just dismissing their concerns
as trivial or unimportant. Breakage isn't trivial or unimportant to
the person who's yelling, so this is an important
community-maintenance skill. It builds trust, and reduces the total
amount of yelling.

*shrug*, If I didn't care I would have made this change as soon as Nick
said it was ok. Instead I declared I was going to and waited to make sure
nobody else had any concerns. And once Holger said he did I said
ok I won't do it. Maybe my mannerisms give the impression I don't but
that's actually pretty far from the truth. For this particular change I originally
created the pip commit that allowed it, and then again I created the setuptools
commit, backporting hashlib into setuptools to support Python 2.4. I put
a decent amount of effort into trying to make sure that nothing broke
but in the end there were still concerns :)

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Antoine Pitrou]

Post by Donald Stufft
*shrug*, If I didn't care I would have made this change as soon as Nick
said it was ok. Instead I declared I was going to and waited to make sure
nobody else had any concerns. And once Holger said he did I said
ok I won't do it. Maybe my mannerisms give the impression I don't but
that's actually pretty far from the truth. For this particular change I originally
created the pip commit that allowed it, and then again I created the setuptools
commit, backporting hashlib into setuptools to support Python 2.4. I put
a decent amount of effort into trying to make sure that nothing broke
but in the end there were still concerns :)

You're right. I've been a bit harsh, but you're definitely doing some
very useful things here.

Regards

Antoine.

[PJ Eby]

Post by Donald Stufft

Post by PJ Eby

Post by Donald Stufft
Heh, I'm pretty good at getting yelled at :)

Nick is also pretty good at making people feel like he both knows and
*cares* about their breakage, and isn't just dismissing their concerns
as trivial or unimportant. Breakage isn't trivial or unimportant to
the person who's yelling, so this is an important
community-maintenance skill. It builds trust, and reduces the total
amount of yelling.

*shrug*, If I didn't care I would have made this change as soon as Nick
said it was ok. Instead I declared I was going to and waited to make sure
nobody else had any concerns. And once Holger said he did I said
ok I won't do it. Maybe my mannerisms give the impression I don't but
that's actually pretty far from the truth.

I did say "feel like". ;-)

Nick usually gives more of an impression that he's thought about
concerns raised before rejecting them; your responses often sound
like, "Who cares about that?" Asking for suggestions, for example,
would be good. Nick also rarely seems irritated by people's concerns
or problems, whereas you sometimes seem in a big hurry to fix
something today or this week that's been broken for years, without
giving folks a while to get used to the idea. Often your proposals
seem less like proposals than, "I've decided to do this, so deal with
it".

I'm not saying all this because I want to complain or yell at you; I'm
saying it because I think you do care enough to know how you're coming
across, at least to me. Our discussions have gotten heated in the
past because my impression of your reaction to the concerns I raise
often seems like, "I don't care about supporting [group of people
affected], so neither should you."

Perhaps the issue is just one of confusion. When I raise an issue
about, say, Python 2.3 users (who are still downloading setuptools 0.6
releases, and presumably also using them), it's not because I expect
*you* to change your plans to support them, but because I need to know
how *I* can, if the issue arises. So I don't actually expect you to
care about Python 2.3 users (again, as an example), but I do expect
you to care about *me* supporting them.

In the most recent situation, you did in fact point me to your awesome
hashlib port, so I do know you *do* care to at least that extent. But
the rhetoric that you sent both before and after the helpful bit
seemed on the inflammatory side, as though I were crazy to be thinking
of Python 2.3. Whether or not this is true ;-) -- it's not especially
*helpful* to the discussion.

If I may offer a suggestion, asking questions in response to
objections is generally more useful than immediately arguing the
relevance of the objection. First, it tells the objector that you're
interested in what they have to say, and second, it may well help the
objector understand that there isn't actually any real problem, and
gives them an easier path to backing down and/or compromising, whereas
a frontal assault tends to focus people on responding to you instead
of reconsidering their objection.

On the hashlib issue, for example, it actually occurred to me later
that it's completely a non-issue because the actual hash scenario I
was most concerned about *can't actually happen*. (MD5 hashes in code
or dependency_links, used e.g. by setuptools itself to secure its own
downloads. Changing PyPI won't affect these, duh.) It might've
occurred to me sooner, though, if you'd actually asked what scenario I
was worried about, instead of arguing about the relevance.

This isn't to say that you're responsible for what I do or don't
figure out; my point is simply that asking questions and inviting
suggestions in response to people's objections will generally get you
more thoughtful responses and more trust, and resolve issues sooner,
with less arguing.

[Donald Stufft]

Post by PJ Eby

Post by Donald Stufft

Post by PJ Eby

Post by Donald Stufft
Heh, I'm pretty good at getting yelled at :)

Nick is also pretty good at making people feel like he both knows and
*cares* about their breakage, and isn't just dismissing their concerns
as trivial or unimportant. Breakage isn't trivial or unimportant to
the person who's yelling, so this is an important
community-maintenance skill. It builds trust, and reduces the total
amount of yelling.

*shrug*, If I didn't care I would have made this change as soon as Nick
said it was ok. Instead I declared I was going to and waited to make sure
nobody else had any concerns. And once Holger said he did I said
ok I won't do it. Maybe my mannerisms give the impression I don't but
that's actually pretty far from the truth.

I did say "feel like". ;-)
Nick usually gives more of an impression that he's thought about
concerns raised before rejecting them; your responses often sound
like, "Who cares about that?" Asking for suggestions, for example,
would be good. Nick also rarely seems irritated by people's concerns
or problems, whereas you sometimes seem in a big hurry to fix
something today or this week that's been broken for years, without
giving folks a while to get used to the idea. Often your proposals
seem less like proposals than, "I've decided to do this, so deal with
it".
I'm not saying all this because I want to complain or yell at you; I'm
saying it because I think you do care enough to know how you're coming
across, at least to me. Our discussions have gotten heated in the
past because my impression of your reaction to the concerns I raise
often seems like, "I don't care about supporting [group of people
affected], so neither should you."
Perhaps the issue is just one of confusion. When I raise an issue
about, say, Python 2.3 users (who are still downloading setuptools 0.6
releases, and presumably also using them), it's not because I expect
*you* to change your plans to support them, but because I need to know
how *I* can, if the issue arises. So I don't actually expect you to
care about Python 2.3 users (again, as an example), but I do expect
you to care about *me* supporting them.
In the most recent situation, you did in fact point me to your awesome
hashlib port, so I do know you *do* care to at least that extent. But
the rhetoric that you sent both before and after the helpful bit
seemed on the inflammatory side, as though I were crazy to be thinking
of Python 2.3. Whether or not this is true ;-) -- it's not especially
*helpful* to the discussion.
If I may offer a suggestion, asking questions in response to
objections is generally more useful than immediately arguing the
relevance of the objection. First, it tells the objector that you're
interested in what they have to say, and second, it may well help the
objector understand that there isn't actually any real problem, and
gives them an easier path to backing down and/or compromising, whereas
a frontal assault tends to focus people on responding to you instead
of reconsidering their objection.
On the hashlib issue, for example, it actually occurred to me later
that it's completely a non-issue because the actual hash scenario I
was most concerned about *can't actually happen*. (MD5 hashes in code
or dependency_links, used e.g. by setuptools itself to secure its own
downloads. Changing PyPI won't affect these, duh.) It might've
occurred to me sooner, though, if you'd actually asked what scenario I
was worried about, instead of arguing about the relevance.
This isn't to say that you're responsible for what I do or don't
figure out; my point is simply that asking questions and inviting
suggestions in response to people's objections will generally get you
more thoughtful responses and more trust, and resolve issues sooner,
with less arguing.

Hrm.

So I hear what you're saying and part of the problem is likely due to the history
of where I tried to get a change through and then felt like all I was getting was
stop energy and people wanting to keep the status quo which ultimately
ended up preventing changes has lead me to view distutils-sig in more of an
adversarial light then is probably appropriate for the distutils-sig of 2013 (versus
the distutils-sig of 2011/2012). This is probably reflected in my tone and likely
has others, such as yourself, respond similarly, pushing us further down that
path. My thought process has become "Ok here's something that needs to
happen, now how do I get distutils-sig not to prevent it from happening".

This was again reflected in the Python 2.3 discussion because my immediate
reaction and impression was that you were attempting to block the move
from MD5 due to Python 2.3, which I felt 2.3 wasn't worth blocking enhancements
to PyPI. The "snark" in my statements primarily came from that feeling of
someone was trying to "shut down" an enhancement.

I'd have to look to see when I first starting trying to advocate for change on
catalog-sig but it was sometime in 2011 because I had finally gotten frustrated
and went to "take my ball and go home" and start a competitor to PyPI with
Crate.io on Dec 25th 2011. So there was a good 1-2 years there where the
various mailing lists *were* an adversary towards getting fixes put into place.
Now that I am able to get changes put into place I'm trying to go back and solve
the various issues that I've found and I do have a feeling of urgency because I
get concerned that this willingness to move forward and fix things is temporary
and I want to fit in as many fixes as I can as quickly as I can before my window
of opportunity closes. I've gone so far as begun sleeping less (not that i ever
slept anything that one could call a normal sleep schedule) in order to buy
more hours to work on these issues to get them fixed faster.

So I do get frustrated and/or irritated and a lot of that comes from the history there
because prior to Nick/Richard becoming BDFL Delegates there was never
anyone willing to say "Yes, we can do that". I'm also generally a person who gets
angry quickly (but gets un-angry quickly as well) which leads to it being harder
for me not to snark or go on the offensive. There is a reason my personal domain
is https://caremad.io/ ;)

As far as how to fix it I don't have a particularly magic answer. I will try to be more
mindful of my tone and that distutils-sig is likely not my adversary anymore as well
as try to ask questions instead of arguing the relevance immediately.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Paul Moore]

Post by Donald Stufft
As far as how to fix it I don't have a particularly magic answer. I will try to be more
mindful of my tone and that distutils-sig is likely not my adversary anymore as well
as try to ask questions instead of arguing the relevance immediately.

Thank you - both for the thoughtful response and the explanation of what is
going on in your thinking.

I certainly tend to think about possible issues rather than benefits, maybe
more than I should as I don't personally have any particular problems with
a rapid pace of change (except in some specific areas where I'm in so much
of a minority that I don't expect my concerns to be a big driver in the
grand scheme of things). I think I'm doing this to make sure people haven't
missed potential issues I'm aware of, but I can easily imagine that this
comes across as negative "stop energy". I'll try to stick to issues where I
have genuine concerns, and not theoretical ones in future.

And yes, it does feel like distutils-sig of 2013 is a much nicer place than
it used to be. Thanks to everyone for working to keep that the case.

Paul

[Donald Stufft]

Post by Donald Stufft
As far as how to fix it I don't have a particularly magic answer. I will try to be more
mindful of my tone and that distutils-sig is likely not my adversary anymore as well
as try to ask questions instead of arguing the relevance immediately.
Thank you - both for the thoughtful response and the explanation of what is going on in your thinking.
I certainly tend to think about possible issues rather than benefits, maybe more than I should as I don't personally have any particular problems with a rapid pace of change (except in some specific areas where I'm in so much of a minority that I don't expect my concerns to be a big driver in the grand scheme of things). I think I'm doing this to make sure people haven't missed potential issues I'm aware of, but I can easily imagine that this comes across as negative "stop energy". I'll try to stick to issues where I have genuine concerns, and not theoretical ones in future.

I don't actually mind even theoretical problems nor do I want people to feel like they need to coddle me because of my history with distutils-sig. That history and how It affects me is *my* problem. I think as long as we, including myself, approach theoretical problems as "Let's figure out if this theoretical problem is actually a problem, and if it is do we care about it" and not "here's a bunch of possible problems, we can't do this" then there will be no issues.

It's true that theoretical problems can make me feel more like someones applying stop energy than concrete problems, but that's not a problem of the person bringing up that problem and more just it triggering old feelings from a time where I couldn't even get a SSL certificate trusted by most major browsers by default deployed ;). Not allowing those feelings to poison current efforts is on me.

Post by Donald Stufft
And yes, it does feel like distutils-sig of 2013 is a much nicer place than it used to be. Thanks to everyone for working to keep that the case.
Paul

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Vinay Sajip]

Post by Donald Stufft
to PyPI. The "snark" in my statements primarily came from that feeling of
someone was trying to "shut down" an enhancement.

To be fair, the tone you've taken (which can rub people up the wrong way)
hasn't just been over long-standing security issues. I've also felt
uncomfortable with some of your responses relating to the PEP 426
discussion around the JSON schema for requirements, where there is not the
same level of urgency, nor valid reason to believe that you're being thwarted
somehow.

If someone's approach comes off as unduly dismissive (even if not meant that
way), it makes it harder to engage in constructive discussion. I recognise
that it's hard communicating with people remotely, many of whom one might
never have met. It's easy to miss nuance and misconstrue turns of phrase. It's
best to tread gingerly, else too much time is spent smoothing ruffled feathers
at the expense of getting real work done.

Regards,

Vinay Sajip

[Donald Stufft]

Post by Vinay Sajip

Post by Donald Stufft
to PyPI. The "snark" in my statements primarily came from that feeling of
someone was trying to "shut down" an enhancement.

To be fair, the tone you've taken (which can rub people up the wrong way)
hasn't just been over long-standing security issues. I've also felt
uncomfortable with some of your responses relating to the PEP 426
discussion around the JSON schema for requirements, where there is not the
same level of urgency, nor valid reason to believe that you're being thwarted
somehow.

I never meant to claim it was simply over security issues. I'm certainly more
"energetic" about advocating for security issues and they frame the backdrop
for a lot of my feelings with distutils-sig as an adversary rather than an ally . The
simple fact is that every disagreement, or even simple suggestion, is framed
against that feeling leading towards the Me vs Them tone.

Post by Vinay Sajip
If someone's approach comes off as unduly dismissive (even if not meant that
way), it makes it harder to engage in constructive discussion. I recognise
that it's hard communicating with people remotely, many of whom one might
never have met. It's easy to miss nuance and misconstrue turns of phrase. It's
best to tread gingerly, else too much time is spent smoothing ruffled feathers
at the expense of getting real work done.
Regards,
Vinay Sajip
_______________________________________________
http://mail.python.org/mailman/listinfo/distutils-sig

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[PJ Eby]

Post by Donald Stufft
Hrm.
So I hear what you're saying and part of the problem is likely due to the history
of where I tried to get a change through and then felt like all I was getting was
stop energy and people wanting to keep the status quo which ultimately
ended up preventing changes has lead me to view distutils-sig in more of an
adversarial light then is probably appropriate for the distutils-sig of 2013 (versus
the distutils-sig of 2011/2012). This is probably reflected in my tone and likely
has others, such as yourself, respond similarly, pushing us further down that
path. My thought process has become "Ok here's something that needs to
happen, now how do I get distutils-sig not to prevent it from happening".

Thanks for the thoughtful response. I appreciate it.

I also want to just throw in one extra piece of information for you
and anybody else reading this: 99% of "stop energy" doesn't happen
because people actively want to prevent progress or frustrate other
people. It simply happens when people notice a problem but don't have
as much personal stake in your goal as they do in not experiencing the
problem they will experience (or perceive they will), from the
proposed change.

When you look at it from this perspective, it's easier to understand
that the way to fix this is with more engagement on their part, which
can only be gotten by engagement on your part.

When I first proposed WSGI, the initial reaction of Web-SIG was pretty
negative. "Stop energy" if you will. Things only moved forward once
I was able to channel the energy of objections into offering
solutions. It's helpful to remember that asking, "okay, so how you
would recommend I do it?" *doesn't* obligate you to actually follow
all of the recommendations you get. (Especially since some of them
will be mutually contradictory!)

Anyway, I guess what I'm saying is that people lacking enthusiasm for
your goals is not really them trying to stop you. In fact, objections
are a positive thing: it means you got somebody's attention. The next
step is to leverage that attention into actually getting help, or at
least more constructive input. ;-)

It's true that some individuals will never provide really helpful
input. In the WSGI effort, there were people whose advice I never
took, because their goals were directed entirely opposite to where I
wanted to go. But I remained engaged until it was mutually clear (or
at least I thought it was) that our goals were different, and didn't
try to persuade them to go in the same direction. Such attempts at
persuasion are pretty much a waste of time, and a big energy drain.
Consensus-building is something that you do with people who have at
least some goals in common, so it's best to focus on finding out what
you *do* agree on.

Post by Donald Stufft
This was again reflected in the Python 2.3 discussion because my immediate
reaction and impression was that you were attempting to block the move
from MD5 due to Python 2.3, which I felt 2.3 wasn't worth blocking enhancements
to PyPI. The "snark" in my statements primarily came from that feeling of
someone was trying to "shut down" an enhancement.

Right. In such a case, a question you could ask is, "Do you agree in
general that we should move to a better hash at some point in the
future?", because then the disagreement can be narrowed down to
timeframe, migration or deprecation process, etc. The truth is, I had
no intention of "blocking the move", I had concerns I wanted addressed
about the impact, timing and process. (Actually, I originally just
noticed a couple of errors in what you'd laid out as the current state
of things, and wanted to make sure they were included in the
assessment.)

The point is, if somebody doesn't have *any* common ground with you,
it's unlikely they're even talking to you. At the very least, if
they're talking with you about PyPI, they must care about PyPI, even
if they care about different things than you, or with different
relative priorities. ;-)

Post by Donald Stufft
As far as how to fix it I don't have a particularly magic answer. I will try to be more
mindful of my tone and that distutils-sig is likely not my adversary anymore as well
as try to ask questions instead of arguing the relevance immediately.

Again, thank you. And hopefully, remember that probably nobody was
intentionally being your adversary before, either. As the old adage
says, the best way to destroy your enemies is to make friends with
them. ;-) And we do that by focusing on common ground, and inviting
participation.

(This is again not to say that I've been 100% Mr. Wonderful myself; I
know I haven't. But the community's best consensus-building happens
when somebody is doing the tough work of engaging with all parties.
Sometimes this doesn't happen, alas; back when I was developing
setuptools there just weren't enough people interested in the problems
available on Distutils-SIG to build any sort of consensus on the
solutions, so I *had* to go run with the ball myself. I'm really
happy that there is now BOTH a quorum of interested parties who
understand the problems, AND a few leaders able to drive the
consensus-building and actual development. If only we had done this
ten years ago, setuptools might not have been necessary. Well,
actually, that's probably not true: without *something* existing
first, that quorum of people who understand the problem wouldn't have
existed back then. But you know what I mean.)

[Donald Stufft]

Post by PJ Eby
I also want to just throw in one extra piece of information for you
and anybody else reading this: 99% of "stop energy" doesn't happen
because people actively want to prevent progress or frustrate other
people. It simply happens when people notice a problem but don't have
as much personal stake in your goal as they do in not experiencing the
problem they will experience (or perceive they will), from the
proposed change.

One of the biggest things I viewed as stop energy was how impossible
it seemed to get a SSL certificate that was trusted in most major browsers
to be used on PyPI. There was never any reason given that we couldn't
move to one, just simply "we already have a certificate" without addressing
the reasons I was trying to get a different one.

Beyond that it was other random proposals that it felt like people put in
the minimal amount of effort in order to find a reason to say no to it. Then
if I addressed that particular concern, they'd again seem to put in the minimal
amount of effort to find a reason to say no. It felt like the goal posts were just
constantly being moved to slightly further away each time.

That sort of behavior is mostly what I mean by "stop energy". It's entirely
frustrating and it feels like it's entire purpose is to dissuade someone from
changing the status quo, wether that was the intention or not.

Everything else you said has been noted :)


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[holger krekel]

Hi Donald,

Post by Donald Stufft

Post by PJ Eby

Post by Donald Stufft
Heh, I'm pretty good at getting yelled at :)

Nick is also pretty good at making people feel like he both knows and
*cares* about their breakage, and isn't just dismissing their concerns
as trivial or unimportant. Breakage isn't trivial or unimportant to
the person who's yelling, so this is an important
community-maintenance skill. It builds trust, and reduces the total
amount of yelling.

*shrug*, If I didn't care I would have made this change as soon as
Nick said it was ok. Instead I declared I was going to and waited to
make sure nobody else had any concerns. And once Holger said he did I
said ok I won't do it. Maybe my mannerisms give the impression I don't
but that's actually pretty far from the truth. For this particular
change I originally created the pip commit that allowed it, and then
again I created the setuptools commit, backporting hashlib into
setuptools to support Python 2.4. I put a decent amount of effort into
trying to make sure that nothing broke but in the end there were still
concerns :)

For the record, i am all for putting generic hash support into the
installers and maybe prepare for an eventual change to make PyPI serve
sha256 hashes. However, to me it's not clear if such a move may become
obsolete through the potential advent of TUF.

My original objection reason was tied to generally pushing for more focus
on backward-compatibility. I am grateful that several people including you,
Nick and Jannis acknowledged the point.

best,
holger

Post by Donald Stufft
-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________
http://mail.python.org/mailman/listinfo/distutils-sig

[Donald Stufft]

Post by holger krekel
Hi Donald,

Post by Donald Stufft

Post by PJ Eby

Post by Donald Stufft
Heh, I'm pretty good at getting yelled at :)

Nick is also pretty good at making people feel like he both knows and
*cares* about their breakage, and isn't just dismissing their concerns
as trivial or unimportant. Breakage isn't trivial or unimportant to
the person who's yelling, so this is an important
community-maintenance skill. It builds trust, and reduces the total
amount of yelling.

*shrug*, If I didn't care I would have made this change as soon as
Nick said it was ok. Instead I declared I was going to and waited to
make sure nobody else had any concerns. And once Holger said he did I
said ok I won't do it. Maybe my mannerisms give the impression I don't
but that's actually pretty far from the truth. For this particular
change I originally created the pip commit that allowed it, and then
again I created the setuptools commit, backporting hashlib into
setuptools to support Python 2.4. I put a decent amount of effort into
trying to make sure that nothing broke but in the end there were still
concerns :)

For the record, i am all for putting generic hash support into the
installers and maybe prepare for an eventual change to make PyPI serve
sha256 hashes. However, to me it's not clear if such a move may become
obsolete through the potential advent of TUF.

pip has had generic hash support since 1.2, setuptools since 0.9, and since
I believe zc.buildout just lets setuptools do the downloads so it'd be when
using 0.9 for it.

The idea was this was an easy win security hardening. Something like TUF
would more or less obsolete it but TUF isn't on the immediate radar.

Post by holger krekel
My original objection reason was tied to generally pushing for more focus
on backward-compatibility. I am grateful that several people including you,
Nick and Jannis acknowledged the point.
best,
holger

Post by Donald Stufft
-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________
http://mail.python.org/mailman/listinfo/distutils-sig

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Donald Stufft]

Signed PGP part

Post by Donald Stufft
I've personally gotten or seen more complaints over the naming of a
variable in the config file then I have over any changes we've made.
The runner up to that is the fallout from switching to requiring
verified SSL.

The past few months have generated a *lot* of teeth-gnashing /
hair-pulling, especially among "downstream" developers (those unlikely to
- - HTTPS-only PyPY

I've seen very few people complain about this. It's probably the largest issue in either of the projects I'm involved with.

- - Distribute / setuptools merge, e.g. cratering folks who use a
distro-managed 'python-distribute' package.

This is the biggest issue. I wasn't involved and it could have been handled better sure.

- - Pip's new backward-imcompatible "final releases by default".

Again this negatively affects very few packages and it positively affects a number of people who were bit by the fact that pip does this. SQLAlchemy was one of the ones that were offhand. People don't realize the tools *didn't* do this and often times caused pain inadvertently for the down stream developers.

I think we are going to be in a much better place for all that, but let's
not deny the pain involved for *everybody* in getting there.
Tres.
- --
===================================================================
Palladion Software "Excellence by Design" http://palladion.com
_______________________________________________
http://mail.python.org/mailman/listinfo/distutils-sig

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Marcus Smith]

Post by Tres Seaver
- - Distribute / setuptools merge, e.g. cratering folks who use a
distro-managed 'python-distribute' package.
This is the biggest issue. I wasn't involved and it could have been handled better sure.

what issue are we talking about exactly?

I'm aware of the "Import setuptools" problem during upgrades from
distribute to setuptools: https://github.com/pypa/pip/issues/1064
pip-1.4 does prevent that now (released last week)

but you specifically mentioned the linux distro name, "python-distribute"?
I'm guessing maybe you're concerned with the scenario of how upgrading some
project (that depends on setuptools/distribute) forces an unintended
upgrade of distribute on a system where distribute is system-managed.
Is there a link to a tracker issue for that somewhere?

3 solutions to ponder:
1) unreleasing the distribute-0.7.3 wrapper, and making any upgrades from
distribute to setuptools manual.
2) changing pip's -U logic to not recursively upgrade satisfied
dependencies, but that's a *big* change (it's a long story, but there are
tentative plans in 1.5 for changing it)
3) in a patch release, include a special case for distribute to *not*
upgrade if satisfied (when distribute is a dependency in the requirement
set).

Marcus

[Marcus Smith]

Post by Marcus Smith

Post by Tres Seaver
- - Distribute / setuptools merge, e.g. cratering folks who use a
distro-managed 'python-distribute' package.
This is the biggest issue. I wasn't involved and it could have been handled better sure.

what issue are we talking about exactly?
I'm aware of the "Import setuptools" problem during upgrades from
distribute to setuptools: https://github.com/pypa/pip/issues/1064
pip-1.4 does prevent that now (released last week)
but you specifically mentioned the linux distro name, "python-distribute"?
I'm guessing maybe you're concerned with the scenario of how upgrading
some project (that depends on setuptools/distribute) forces an unintended
upgrade of distribute on a system where distribute is system-managed.
Is there a link to a tracker issue for that somewhere?

but on 2nd thought, this is not a new problem. this would always been
happening, just not involving a transition to setuptools.
so I imagine, your concern was just the "Import" error?

[Lennart Regebro]

Post by Donald Stufft

Post by Tres Seaver
- - Distribute / setuptools merge, e.g. cratering folks who use a
distro-managed 'python-distribute' package.

This is the biggest issue. I wasn't involved and it could have been handled better sure.

I'm not convinced. The use-cases and ways these things can be
installed in conjunction with things like system packages, virtualenv
and buildout in various configurations and various python versions are
mindboggling. Some of them were not initially tested, probably mostly
because nobody realized the finer details of what would happen in
these sometimes fairly obscure cases.

The problems have, afaik, now been fixed, and this was done reasonably
quickly IMO. OK, of course it *could* have been handled better, but I
don't think it was handled badly.

The current situation is a huge mess. Untangling it *will* be tricky.

//Lennart

[Vinay Sajip]

Post by Donald Stufft
I'm going to go ahead and make this change unless someone comes out and
contests moving PyPI to SHA256. I'll give it a bit to make sure no one does
have an issue with the move.

Your proposal is a little light on specification, unless I've missed it. For
example:

* How exactly will download URLs change? One would assume they'd have a
fragment of 'sha256=...', where they currently have 'md5=...', but can you
confirm this?

* PyPI's XML-RPC API provides MD5 hashes in result dictionaries using a key
'md5_digest'. How will these result dictionaries change under your
proposal?

* PyPI's web interface has actions such as 'show_md5', will these stop
working? (By actions, I mean query strings such as ':action=show_md5'.)
Will new actions be added?

I'm not familiar with the change process for PyPI - what is the workflow?
For example, are patches posted for review?

Regards,

Vinay Sajip

[Donald Stufft]

Post by Vinay Sajip

Post by Donald Stufft
I'm going to go ahead and make this change unless someone comes out and
contests moving PyPI to SHA256. I'll give it a bit to make sure no one does
have an issue with the move.

Your proposal is a little light on specification, unless I've missed it. For
* How exactly will download URLs change? One would assume they'd have a
fragment of 'sha256=...', where they currently have 'md5=...', but can you
confirm this?

Yes they will change to have #sha256=…. instead of #md5=...

Post by Vinay Sajip
* PyPI's XML-RPC API provides MD5 hashes in result dictionaries using a key
'md5_digest'. How will these result dictionaries change under your
proposal?

Here we are a little more flexible. I can leave the md5_digest key there and
simply add a sha256_digest key.

Post by Vinay Sajip
* PyPI's web interface has actions such as 'show_md5', will these stop
working? (By actions, I mean query strings such as ':action=show_md5'.)
Will new actions be added?

Again more flexible. I can simply add a show_sha256 action.

Post by Vinay Sajip
I'm not familiar with the change process for PyPI - what is the workflow?
For example, are patches posted for review?

Typically it's left up to us. We often just work and deploy changes without
any review process but we can (and I have) get reviews before hand. The
biggest problem with Reviews is PyPI is a very messy codebase with very
few people who understand it so the pool of developers qualified to review
the code is very small.

On the warehouse side of things I don't develop directly on master everything
comes through pull requests and while there's no formal review process
A number of folks have been checking my PR's and making comments as
they deemed fit.

Post by Vinay Sajip
Regards,
Vinay Sajip
_______________________________________________
http://mail.python.org/mailman/listinfo/distutils-sig

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Vinay Sajip]

Yes they will change to have #sha256=…. instead of #md5=...

[snip]

Here we are a little more flexible. I can leave the md5_digest key there and
simply add a sha256_digest key.

[snip]

Again more flexible. I can simply add a show_sha256 action.

[snip]

Typically it's left up to us. We often just work and deploy changes without

[snip]

Thanks for the update, it'll help me to update distlib when the time comes.

Regards,

Vinay Sajip

[Donald Stufft]

Post by Vinay Sajip

Post by Donald Stufft
Yes they will change to have #sha256=…. instead of #md5=...

[snip]

Post by Donald Stufft
Here we are a little more flexible. I can leave the md5_digest key there and
simply add a sha256_digest key.

[snip]

Post by Donald Stufft
Again more flexible. I can simply add a show_sha256 action.

[snip]

Post by Donald Stufft
Typically it's left up to us. We often just work and deploy changes without

[snip]
Thanks for the update, it'll help me to update distlib when the time comes.
Regards,
Vinay Sajip

I sugggest distlib take the same approach as pip and setuptools. Allow any
#hashname=hash so that it will function with whatever hash people use.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA