Please do not drop Python 2 modules

[Adrian Bunk]

Hi,

first two facts:

1. Upstream EOL for Python 2 is 2020

2. Debian will fully (security) support Python 2 in buster
until the EOL of buster (ETA: mid-2022)


Python 2 is obsolete, no doubt about that.

But in many cases a Linux distribution is just a platform for running
own applications, and for various good and bad reasons many of our
users will have to support custom and/or 3rd party Python 2 code on
systems running buster.

We are only making it unnecessarily harder for our users when
Python 2 modules are dropped before buster.

The tip of the iceberg are some recent cases where Python 2 modules
were dropped that still had reverse dependencies in unstable, but
also for those without there will in many cases be users who will
still need these modules in buster.


All of the above applies especially in cases where continuing to
provide a Python 2 module does not cause problems or extra work
(in several cases Python 2 modules were dropped in a new Debian
revision of a package without any real reason).

There are of course cases (e.g. OpenStack) where providing Python 2
modules in buster is not possible with reasonable effort.


Thanks
Adrian

[Chris Lamb]

Post by Adrian Bunk
The tip of the iceberg are some recent cases where Python 2 modules
were dropped that still had reverse dependencies in unstable

I suspect developers may be reading too much into Lintian output,
reading them as "Please remove your Python 2.x module".

The motivating behind these tags were to prevent new Python 2.x
packages being added to the archive (due to habit if anything
else!) unless they were needed or requested, of course.

Suggestions for rewording gratefully received.


Regards,

[olivier sallou]

Post by Chris Lamb

Post by Adrian Bunk
The tip of the iceberg are some recent cases where Python 2 modules
were dropped that still had reverse dependencies in unstable

I suspect developers may be reading too much into Lintian output,
reading them as "Please remove your Python 2.x module".
The motivating behind these tags were to prevent new Python 2.x
packages being added to the archive (due to habit if anything
else!) unless they were needed or requested, of course.

Indeed, python 2 modules are not forbidden, but NEW packages are *expected*
to provide python 3 plus optionally python 2 support, not python 2 only.
But good reasons would allow package to be accepted anyway.
I think that warning is really to hint users to go to python 3 which should
be default now....

Olivier

Post by Chris Lamb
Suggestions for rewording gratefully received.
Regards,
--
,''`.
: :' : Chris Lamb
`-

[Julien Muchembled]

Post by Chris Lamb

Post by Adrian Bunk
The tip of the iceberg are some recent cases where Python 2 modules
were dropped that still had reverse dependencies in unstable

I suspect developers may be reading too much into Lintian output,
reading them as "Please remove your Python 2.x module".
The motivating behind these tags were to prevent new Python 2.x
packages being added to the archive (due to habit if anything
else!) unless they were needed or requested, of course.

A lintian warning is even a reason for REJECT. "I" (my mentor) uploaded a new source package "zodbpickle" 5 weeks ago and I wonder if it's stuck because of this. I found strange to put an override for this so I didn't.

The ITP contains a link to an email where I explain why it is needed:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=842870
(to sum up: required dependency in order to package a new version of ZODB with support for Python 3)

But I don't want to drop the Python 2 module of ZODB. That's what I only use for the moment.

(Actually, I read all the recent discussions about NEW closely: it's quite frustrating that all the work for #783377 was roughly done 2 years ago, and after difficulties at getting sponsored, I may now be blocked by this warning.)

Julien

[Luke W Faraone]

Post by Julien Muchembled
A lintian warning is even a reason for REJECT.

Technically yes, Lintian warnings and errors are a thing that ftpteam
looks at when processing new packages.

But if all Lintian warnings without an override were cause for reject,
we'd just configure dak to do that work for us, no need to spend time
reading the package. In fact we do this for a set of Lintian
warnings/errors with minimal false-positives[1].

We're human, we're going to think about whether it makes sense, whether
it's a particularly egregious screwup that will make the archive worse,
whether it's a false-positive or otherwise ignorable, etc.

Post by Julien Muchembled
"I" (my mentor) uploaded a new source package "zodbpickle" 5 weeks ago and I wonder if it's stuck because of this.

If that were the case, you should have gotten mail about it. (either a
prod for more information, or a REJECT)

Looking at the current state of the queue[1], there are a number of
(non-node) packages ahead of your package. As far as I can tell, it just
hasn't been reviewed yet.

Post by Julien Muchembled
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=842870
(to sum up: required dependency in order to package a new version of ZODB with support for Python 3)
But I don't want to drop the Python 2 module of ZODB. That's what I only use for the moment.

Great! Then don't! From the Lintian warning text:

Python 2.x modules should not be packaged unless strictly necessary
(such as being explicitly requested by an end-user or required as part
of a dependency chain) as the 2.x series of Python is due for
deprecation and will not be maintained past 2020.

Post by Julien Muchembled
I found strange to put an override for this so I didn't.

While not required, including an override is a sign to other Debian
developers that "yes, I thought about this and it is not applicable for
this reason". So I think it would have been good to include one, but not
absolutely required.

Cheers,
Luke W Faraone

[Scott Kitterman]

Post by Luke W Faraone

Post by Julien Muchembled
A lintian warning is even a reason for REJECT.

Technically yes, Lintian warnings and errors are a thing that ftpteam
looks at when processing new packages.
But if all Lintian warnings without an override were cause for reject,
we'd just configure dak to do that work for us, no need to spend time
reading the package. In fact we do this for a set of Lintian
warnings/errors with minimal false-positives[1].
We're human, we're going to think about whether it makes sense, whether
it's a particularly egregious screwup that will make the archive worse,
whether it's a false-positive or otherwise ignorable, etc.

Post by Julien Muchembled
"I" (my mentor) uploaded a new source package "zodbpickle" 5 weeks

ago and I wonder if it's stuck because of this.
If that were the case, you should have gotten mail about it. (either a
prod for more information, or a REJECT)
Looking at the current state of the queue[1], there are a number of
(non-node) packages ahead of your package. As far as I can tell, it just
hasn't been reviewed yet.

Post by Julien Muchembled
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=842870
(to sum up: required dependency in order to package a new version of

ZODB with support for Python 3)

Post by Julien Muchembled
But I don't want to drop the Python 2 module of ZODB. That's what I

only use for the moment.
Python 2.x modules should not be packaged unless strictly necessary
(such as being explicitly requested by an end-user or required as part
of a dependency chain) as the 2.x series of Python is due for
deprecation and will not be maintained past 2020.

Post by Julien Muchembled
I found strange to put an override for this so I didn't.

While not required, including an override is a sign to other Debian
developers that "yes, I thought about this and it is not applicable for
this reason". So I think it would have been good to include one, but not
absolutely required.

Fundamentally not a lintian warnings are created equal. Some have solid foundation in Debian project consensus and policy. Others are nothing more than the opinions of the lintian maintainers. This is one of the latter.

Scott K

[Holger Levsen]

Post by Scott Kitterman
Fundamentally not a lintian warnings are created equal. Some have solid
foundation in Debian project consensus and policy. Others are nothing
more than the opinions of the lintian maintainers. This is one of the latter.

you make it sound like the lintian maintainers are a bunch of lunatics,
but according to src:piuparts/debian/copyright, that's us, the piuparts
maintainers. the lintian maintainers (and uploaders) are a bunch of
(ex- and current) people from the release team, ftp team, policy editors
and others.

and, afaik, they react to bug reports. maybe for now this python2 warning
should be downgraded to 'info'? what would be the best way to tell them?

[Luke W Faraone]

Hi Holger,

Post by Holger Levsen

Post by Scott Kitterman
Fundamentally not a lintian warnings are created equal. Some have solid
foundation in Debian project consensus and policy. Others are nothing
more than the opinions of the lintian maintainers. This is one of the latter.

you make it sound like the lintian maintainers are a bunch of lunatics,
but according to src:piuparts/debian/copyright, that's us, the piuparts
maintainers. the lintian maintainers (and uploaders) are a bunch of
(ex- and current) people from the release team, ftp team, policy editors
and others.

I can understand that the above is one reading of Scott's mail, but I
personally didn't take anything super negatively w.r.t. "nothing more
than the opinions of the lintian maintainers".

But again, in the context of my mail, I was (quite verbosely) outlining
that Lintian's findings may sometimes be, on their own, sufficient
justifications for a REJECT, and sometimes not.

Even a Lintian warning may still result in a REJECT if it was clear it
was 100% apt, and represents something we don't want in the archive.

Post by Holger Levsen
and, afaik, they react to bug reports. maybe for now this python2 warning
should be downgraded to 'info'? what would be the best way to tell them

I agree, people who feel strongly this issue is misclassified could best
instigate action by taking this discussion to a bug (perhaps spilling
over to debian-devel if it is in fact of interest to the wider project
for discussion).

Cheers,
Luke W Faraone

[Holger Levsen]

Post by Luke W Faraone

Post by Holger Levsen
you make it sound like the lintian maintainers are a bunch of lunatics,
but according to src:piuparts/debian/copyright, that's us, the piuparts
maintainers. the lintian maintainers (and uploaders) are a bunch of
(ex- and current) people from the release team, ftp team, policy editors
and others.

I can understand that the above is one reading of Scott's mail, but I
personally didn't take anything super negatively w.r.t. "nothing more
than the opinions of the lintian maintainers".

me neither. I mostly replied that way because just a few hours before I
had read piuparts's debian/copyright again and found it a nice match :)

Post by Luke W Faraone
But again, in the context of my mail, I was (quite verbosely) outlining
that Lintian's findings may sometimes be, on their own, sufficient
justifications for a REJECT, and sometimes not.

sure!

Post by Luke W Faraone
I agree, people who feel strongly this issue is misclassified could best
instigate action by taking this discussion to a bug

:)

[Scott Kitterman]

Post by Scott Kitterman

Post by Scott Kitterman
Fundamentally not a lintian warnings are created equal. Some have

solid

Post by Scott Kitterman
foundation in Debian project consensus and policy. Others are

nothing

Post by Scott Kitterman
more than the opinions of the lintian maintainers. This is one of

the latter.
you make it sound like the lintian maintainers are a bunch of lunatics,
but according to src:piuparts/debian/copyright, that's us, the piuparts
maintainers. the lintian maintainers (and uploaders) are a bunch of
(ex- and current) people from the release team, ftp team, policy editors
and others.

That certainly wasn't my intention. My only point is that there's no need for any project wide discussion or approval for lintian checks. Some of them are better than others and that's okay, but people shouldn't treat lintian as a source of revealed gospel.

Post by Scott Kitterman
and, afaik, they react to bug reports. maybe for now this python2 warning
should be downgraded to 'info'? what would be the best way to tell them?

Personally, I don't think it should exist at all, but I lost that discussion.

Scott K

[Chris Lamb]

Hey Scott,

(Somehow this got wedged in my 'Drafts' folder. Please don't read
anything into the delay in replying...)

Post by Scott Kitterman
Fundamentally not a lintian warnings are created equal. Some have
solid foundation in Debian project consensus and policy. Others are
nothing more than the opinions of the lintian maintainers.

True, but one would hope the Lintian maintainers were amenable to
reason and logic as well as being open to the idea that they might
mistakes in error and judgement in the relative importance of
tags. :)

You seem to have specific warnings in mind.. File bugs?


Best wishes,

[Scott Kitterman]

Post by Chris Lamb
Hey Scott,
(Somehow this got wedged in my 'Drafts' folder. Please don't read
anything into the delay in replying...)

Post by Scott Kitterman
Fundamentally not a lintian warnings are created equal. Some have
solid foundation in Debian project consensus and policy. Others are
nothing more than the opinions of the lintian maintainers.

True, but one would hope the Lintian maintainers were amenable to
reason and logic as well as being open to the idea that they might
mistakes in error and judgement in the relative importance of
tags. :)
You seem to have specific warnings in mind.. File bugs?

OK. #897213.

Scott K

[Andrey Rahmatullin]

Post by Julien Muchembled
A lintian warning is even a reason for REJECT. "I" (my mentor) uploaded
a new source package "zodbpickle" 5 weeks ago and I wonder if it's stuck
because of this. I found strange to put an override for this so I
didn't.

"Stuck" and "REJECTed" are mutually exclusive states.

[Chris Lamb]

Hi Julien,

Post by Julien Muchembled
I found strange to put an override for this so I didn't.

I'm afraid I'm struggling to see Lintian could be any clearer
here:

N: If upstream have not moved or have no intention to move to Python 3,
N: please be certain that Debian would benefit from the inclusion,
N: continued maintenance burden and (eventual) removal of this package
N: before you upload.
[…]
N: Please do not override this warning; rather, add a justification to your
N: changelog entry; Lintian looks in this version's changelog entry for the
N: specified package name or the phrase "Python 2 version" or similar.

This is not asking anyone to remove anything from the archive,
merely to double-check whether the addition of new Python 2.x
packages is required.

(If they are, so be it; add the rationale to the changelog and
upload away.)


Regards,

[Ian Jackson]

Post by Chris Lamb
Hi Julien,

Post by Julien Muchembled
I found strange to put an override for this so I didn't.

I'm afraid I'm struggling to see Lintian could be any clearer
N: If upstream have not moved or have no intention to move to Python 3,
N: please be certain that Debian would benefit from the inclusion,
N: continued maintenance burden and (eventual) removal of this package
N: before you upload.
[…]
N: Please do not override this warning; rather, add a justification to your
N: changelog entry; Lintian looks in this version's changelog entry for the
N: specified package name or the phrase "Python 2 version" or similar.
This is not asking anyone to remove anything from the archive,
merely to double-check whether the addition of new Python 2.x
packages is required.
(If they are, so be it; add the rationale to the changelog and
upload away.)

Given that Python 2 will be fully supported in buster, I think even
this is too strong.

Can lintian tell whether there is a Python 3 module too ? If so then
I think a better criterion for warning would be "there is no Python 3
module".

Ian.

[Stuart Prescott]

Post by Ian Jackson
Can lintian tell whether there is a Python 3 module too ? If so then
I think a better criterion for warning would be "there is no Python 3
module".

$ lintian-info -t python-foo-but-no-python3-foo
W: python-foo-but-no-python3-foo
N:
N: This source package appears to generate the specified Python 2 package
N: without creating a variant for Python 3.
N:
N: The 2.x series of Python is due for deprecation and will not be
N: maintained past 2020.
N:
N: If upstream have not moved or have no intention to move to Python 3,
N: please be certain that Debian would benefit from the continued
N: inclusion of this package and, if not, consider removing it.
N:
N: Alternatively, ensure that the corresponding package specifies the
N: ${python3:Depends} substvar in its binary dependencies.
N:
N: Severity: normal, Certainty: certain
N:
N: Check: python, Type: source, binary
N:

[Ian Jackson]

Post by Stuart Prescott

Post by Ian Jackson
Can lintian tell whether there is a Python 3 module too ? If so then
I think a better criterion for warning would be "there is no Python 3
module".

$ lintian-info -t python-foo-but-no-python3-foo
W: python-foo-but-no-python3-foo
N: This source package appears to generate the specified Python 2 package
N: without creating a variant for Python 3.
N: The 2.x series of Python is due for deprecation and will not be
N: maintained past 2020.
N: If upstream have not moved or have no intention to move to Python 3,
N: please be certain that Debian would benefit from the continued
N: inclusion of this package and, if not, consider removing it.
N: Alternatively, ensure that the corresponding package specifies the
N: ${python3:Depends} substvar in its binary dependencies.
N: Severity: normal, Certainty: certain
N: Check: python, Type: source, binary

Oh, great. Then I think the other warning should go away for now.

Ian.

[Holger Levsen]

Post by Ian Jackson

Post by Chris Lamb
N: If upstream have not moved or have no intention to move to Python 3,
N: please be certain that Debian would benefit from the inclusion,
N: continued maintenance burden and (eventual) removal of this package
N: before you upload.
[
]
N: Please do not override this warning; rather, add a justification to your
N: changelog entry; Lintian looks in this version's changelog entry for the
N: specified package name or the phrase "Python 2 version" or similar.

Given that Python 2 will be fully supported in buster, I think even
this is too strong.

I think I agree.

I'm also struggling with how to rephrase this, though I'm thinking that
maybe the 2nd paragraph should simply go away, as overriding these lintian
warning at this point in time seems like the right thing to do.

(Which then in conclusion makes me think that this whole warning
shouldnt be a warning but rather a pedantic info right now, because this
will save affected maintainers from adding the override now and removing
it once Buster has been released.)

oh, and btw, piuparts has a python3 branch in git, I would be delighted
if someone could finish it ;)

[Chris Lamb]

Dear -devel,

Post by Chris Lamb
N: Please do not override this warning; rather, add a justification to your
N: changelog entry; Lintian looks in this version's changelog entry for the
N: specified package name or the phrase "Python 2 version" or similar.
I'm also struggling with how to rephrase this, though I'm thinking that
maybe the 2nd paragraph should simply go away, as overriding these lintian
warning at this point in time seems like the right thing to do.
(Which then in conclusion makes me think that this whole warning
shouldnt be a warning but rather a pedantic info right now, because this
will save affected maintainers from adding the override now and removing
it once Buster has been released.)

There are a number of Lintian tags around Python 2.x support and I
believe we are conflating them in this discussion.

For example, I think Holger is interpreting this particular tag as
"this source package is shipping a Python 2.x" module. This is not
the case.

Rather, Lintian detects that this is the *initial* upload of the
source package and, if so, asks the maintainer just to double-check
that there is any requirement for it.

If there is such a need, the maintainer just needs to add a short,
quick justification in the initial changelog entry and Lintian will
then be silent on the matter.

It is thus not asking maintainers to drop Python 2 support…

As there can only be one initial upload by definition, adding an
override here is not only a little silly given that it will trigger
an "unused override" warning on the next upload, it can simply be
avoided in the changelog entry as previously discussed, which
implicitly serves as some documentation too.

(This talk of overrides was why I believe Holger is accidentally
confusing the tag.)


Regards,

[Thomas Goirand]

Post by Ian Jackson

Post by Chris Lamb
Hi Julien,

Post by Julien Muchembled
I found strange to put an override for this so I didn't.

I'm afraid I'm struggling to see Lintian could be any clearer
N: If upstream have not moved or have no intention to move to Python 3,
N: please be certain that Debian would benefit from the inclusion,
N: continued maintenance burden and (eventual) removal of this package
N: before you upload.
[…]
N: Please do not override this warning; rather, add a justification to your
N: changelog entry; Lintian looks in this version's changelog entry for the
N: specified package name or the phrase "Python 2 version" or similar.
This is not asking anyone to remove anything from the archive,
merely to double-check whether the addition of new Python 2.x
packages is required.
(If they are, so be it; add the rationale to the changelog and
upload away.)

Given that Python 2 will be fully supported in buster, I think even
this is too strong.
Can lintian tell whether there is a Python 3 module too ? If so then
I think a better criterion for warning would be "there is no Python 3
module".

IMO, we're here thinking one release too late. This was ok to think this
way before releasing Stretch. I still remember that when we discussed
the state of Py3 for Stretch, we decided that it'd be ok to keep Python
2 for it. But that for Buster, we'd get rid of it. The more time passes,
the more I see we're heading directly for the wall.

This cannot go on, and on, and on, and on... We have to send a clear
message on the right direction, which is Python 2 removal. Yes, removal!
Why are we even discussing this? Isn't it obvious?

Python 3 was released the 3rd of December 2008. 2020 is 12 years later.
How many more years does one think we need until we send the message
that yes, we should port our app/module to Python 3? Sorry, but legacy
*must* die, as it doesn't have upstream support.

To everyone that is vouching for more Python 2, are you volunteering for
helping maintaining the Python 2 interpreter in Debian? It's not going
to be trivial to maintain it 5 years after upstream stops...

Cheers,

Thomas Goirand (zigo)

[Adrian Bunk]

Post by Thomas Goirand
...
This cannot go on, and on, and on, and on... We have to send a clear
message on the right direction, which is Python 2 removal. Yes, removal!
Why are we even discussing this? Isn't it obvious?

It is not for us to decide what tools our users should use,
we should support them no matter what tools they have to use.

Post by Thomas Goirand
Python 3 was released the 3rd of December 2008. 2020 is 12 years later.
How many more years does one think we need until we send the message
that yes, we should port our app/module to Python 3? Sorry, but legacy
*must* die, as it doesn't have upstream support.

Why does legacy "*must*" die?

Upstream support is mainly relevant for security support.

In real life many of our users have to support huge amounts of code
written in Python 2.

There are big codebases written in languages like Tcl around,
so still using Python 2 doesn't strike me as exceptionally weird.

Post by Thomas Goirand
To everyone that is vouching for more Python 2, are you volunteering for
helping maintaining the Python 2 interpreter in Debian? It's not going
to be trivial to maintain it 5 years after upstream stops...

Maintaining the Python 2 interpreter is actually reasonably trivial.

Three years after the release of jessie there has been one security
update for python2.7 in jessie, this is not a problematic package,
and there are other distributions who will create security fixes
for the Python 2 interpreter for many years to come).

Post by Thomas Goirand
Cheers,
Thomas Goirand (zigo)

cu
Adrian

[Andrej Shadura]

Hi,

Post by Adrian Bunk
There are big codebases written in languages like Tcl around,
so still using Python 2 doesn't strike me as exceptionally weird.

Not really disagreeing with you on anything, but I felt I have to be a
shameless plug and note Tcl is actually quite a nice language, as long
as you’re not using it the way it was done in the 90s.

[Ian Jackson]

Post by Andrej Shadura

Post by Adrian Bunk
There are big codebases written in languages like Tcl around,
so still using Python 2 doesn't strike me as exceptionally weird.

Not really disagreeing with you on anything, but I felt I have to be a
shameless plug and note Tcl is actually quite a nice language, as long
as you’re not using it the way it was done in the 90s.

I love Tcl. I am still writing lots of new Tcl.

But of course Tcl is not deprecated upstream.

Having said that, the question is how much work is it to keep having
python2. Helmut gave an example of a situation where it _is_ work:
having python2 gets in the way of crossbuilding and porting.

I think that there comes a point where it is reasonable to ask people
who want to keep something going, in the absence of upstream support,
to do that kind of work.

Adrian: are you volunteering to write patches to solve Helmut's cross
building problem ?

Ian.

[Adrian Bunk]

Post by Ian Jackson
...
Adrian: are you volunteering to write patches to solve Helmut's cross
building problem ?

I am willing to stop for several weeks/months to monitor RC bugs and
report FTBFS if you can make the case that it is more beneficial for
Debian to spend this time instead on making ~ 100 additional packages
cross-buildable.

Post by Ian Jackson
Ian.

cu
Adrian

[Andreas Tille]

Hi Adrian,

Post by Adrian Bunk
I am willing to stop for several weeks/months to monitor RC bugs and
report FTBFS

so that is really you who is so terribly fast in spotting and reporting
bugs (sometimes even with patches). I recently came to the conclusion
that this must be a bot (wondering how the bot would be even able to
create patches). So its really a human beeing. Than thanks for all the
very helpful QA work.

Kind regards

Andreas.

[Ian Jackson]

Post by Adrian Bunk

Post by Ian Jackson
Adrian: are you volunteering to write patches to solve Helmut's cross
building problem ?

I am willing to stop for several weeks/months to monitor RC bugs and
report FTBFS if you can make the case that it is more beneficial for
Debian to spend this time instead on making ~ 100 additional packages
cross-buildable.

I'll take that as a "no". What you are saying is that this other work
is more important for you than continued support for python2.

That's fine, of course, but then you cannot criticise others for
making similar judgements, and for the result that python2 is dropped.

Ian.

[Adrian Bunk]

Post by Ian Jackson

Post by Adrian Bunk

Post by Ian Jackson
Adrian: are you volunteering to write patches to solve Helmut's cross
building problem ?

I am willing to stop for several weeks/months to monitor RC bugs and
report FTBFS if you can make the case that it is more beneficial for
Debian to spend this time instead on making ~ 100 additional packages
cross-buildable.

I'll take that as a "no". What you are saying is that this other work
is more important for you than continued support for python2.
...

No, this is not what I am saying.

Helmut has the ambition to make the whole archive cross-buildable.
If this will ever happen it will be after buster.

It is clear that the problem at hand is not trivial,
otherwise Helmut would have submitted a patch long ago.

If spending a not so small amount of my time on making ~ 100 additional
packages cross-buildable in buster is considered a very high priority
for Debian, please make your case why it is.

Trying to bully me into doing this work with very aggressive wording
"is more important for you than continued support for python2" and
zero explanation why doing this specific work would be important
for Debian is pretty rude.

Post by Ian Jackson
Ian.

cu
Adrian

[Thomas Goirand]

Post by Adrian Bunk

Post by Thomas Goirand
...
This cannot go on, and on, and on, and on... We have to send a clear
message on the right direction, which is Python 2 removal. Yes, removal!
Why are we even discussing this? Isn't it obvious?

It is not for us to decide what tools our users should use,
we should support them no matter what tools they have to use.

I'd really love if it was the case. Reality is otherwise.

What if our users want to use Python 2.4? Will we do it? Of course not,
at some point, we may say it's not doable reasonably, because of the
lost of upstream support and the work it involves to keep it alive. So
yes, it's up to us to decide what we're able or what we want to do for
our users, and our users cannot dictate what tool we must maintain. It
has to be the other way around, like it or not.

Post by Adrian Bunk

Post by Thomas Goirand
Python 3 was released the 3rd of December 2008. 2020 is 12 years later.
How many more years does one think we need until we send the message
that yes, we should port our app/module to Python 3? Sorry, but legacy
*must* die, as it doesn't have upstream support.

Why does legacy "*must*" die?

Any code that isn't updated to be able to run with the newest of
everything it uses (library or interpreter) is in danger of becoming not
maintainable in the long run. That's exactly what we're doing everywhere
in Debian: each time a new version of something is out, all the reverse
(build-)dependencies must be updated to support it.

So, legacy code must be either updated to support newer environment, or
just die if nobody can support it...

Post by Adrian Bunk
In real life many of our users have to support huge amounts of code
written in Python 2.

Of course! And it's the same for old versions of PHP, Java, etc. This
isn't a valid point of argumentation to tell there's legacy code that we
need to support. The only thing we can discuss is for how long we can
continue to support it without too much effort.

Post by Adrian Bunk
There are big codebases written in languages like Tcl around,
so still using Python 2 doesn't strike me as exceptionally weird.

There's still companies offering hosting with php 4... It doesn't mean
it's reasonable to continue to support it.

Post by Adrian Bunk
Maintaining the Python 2 interpreter is actually reasonably trivial.

That's not the question I was asking. I was asking if someone is
volunteering for the next 5 years (ie: Buster + LTS support, that's
quite a huge commitment).

Post by Adrian Bunk
there are other distributions who will create security fixes
for the Python 2 interpreter for many years to come).

There's no such thing (yet?) as a cross-distro security team, unfortunately.

Cheers,

Thomas Goirand (zigo)

[Adrian Bunk]

Post by Thomas Goirand

Post by Adrian Bunk

Post by Thomas Goirand
...
This cannot go on, and on, and on, and on... We have to send a clear
message on the right direction, which is Python 2 removal. Yes, removal!
Why are we even discussing this? Isn't it obvious?

It is not for us to decide what tools our users should use,
we should support them no matter what tools they have to use.

I'd really love if it was the case. Reality is otherwise.
What if our users want to use Python 2.4? Will we do it? Of course not,
at some point, we may say it's not doable reasonably, because of the
lost of upstream support and the work it involves to keep it alive. So
yes, it's up to us to decide what we're able or what we want to do for
our users, and our users cannot dictate what tool we must maintain. It
has to be the other way around, like it or not.

We (Debian) have decided to support Python 2.7 in buster, like it or not.

At that point it is not up to individual maintainers to sabotage
Python 2.7 support in buster by dropping Python 2 packages without
a valid technical reason.

"I want Python 2 to die" is not a valid technical reason here.

Post by Thomas Goirand

Post by Adrian Bunk

Post by Thomas Goirand
Python 3 was released the 3rd of December 2008. 2020 is 12 years later.
How many more years does one think we need until we send the message
that yes, we should port our app/module to Python 3? Sorry, but legacy
*must* die, as it doesn't have upstream support.

Why does legacy "*must*" die?

Any code that isn't updated to be able to run with the newest of
everything it uses (library or interpreter) is in danger of becoming not
maintainable in the long run. That's exactly what we're doing everywhere
in Debian: each time a new version of something is out, all the reverse
(build-)dependencies must be updated to support it.

We are not doing this everywhere in Debian.

For software like Python or Tcl we are shipping and security-supporting
two different versions in stable.

Post by Thomas Goirand
...

Post by Adrian Bunk
Maintaining the Python 2 interpreter is actually reasonably trivial.

That's not the question I was asking. I was asking if someone is
volunteering for the next 5 years (ie: Buster + LTS support, that's
quite a huge commitment).

LTS is an effort by a 3rd party external company,
what and how they support in LTS is not our problem.

Our python2.7 maintainer has already stated in this thread:
otoh, I would be fine to have a very reduced set of python2 apps in
buster+1, if that's needed.

Post by Thomas Goirand

Post by Adrian Bunk
there are other distributions who will create security fixes
for the Python 2 interpreter for many years to come).

There's no such thing (yet?) as a cross-distro security team, unfortunately.

An advantage of code that is dead upstream is that everyone ships the
same sources.

This makes porting fixes from one distribution to another trivial.

Post by Thomas Goirand
Cheers,
Thomas Goirand (zigo)

cu
Adrian

[Don Armstrong]

Post by Adrian Bunk
We (Debian) have decided to support Python 2.7 in buster, like it or not.
At that point it is not up to individual maintainers to sabotage
Python 2.7 support in buster by dropping Python 2 packages without a
valid technical reason.
"I want Python 2 to die" is not a valid technical reason here.

Some package maintainers do not wish to spend the time and effort
necessary to support python 2 indefinitely in their packages. That's a
reasonable personal assessment, and I applaud package maintainers for
saying so early, clearly, and publicly.

The question is what (if anything) do any of us want to do to provide
resources to provide that support.

We do not have unlimited maintainer time; triaging is sad, but it
produces better outcomes.

[Adrian Bunk]

Post by Don Armstrong

Post by Adrian Bunk
We (Debian) have decided to support Python 2.7 in buster, like it or not.
At that point it is not up to individual maintainers to sabotage
Python 2.7 support in buster by dropping Python 2 packages without a
valid technical reason.
"I want Python 2 to die" is not a valid technical reason here.

Some package maintainers do not wish to spend the time and effort
necessary to support python 2 indefinitely in their packages. That's a
reasonable personal assessment, and I applaud package maintainers for
saying so early, clearly, and publicly.
The question is what (if anything) do any of us want to do to provide
resources to provide that support.
We do not have unlimited maintainer time; triaging is sad, but it
produces better outcomes.

This discussion is not about triaging, it is about maintainers dropping
Python 2 modules for no technical reason at all.

We already had several cases of a changelog for a new Debian revision
that looked approximately:
* move vcs to salsa
* Standards-Version: 4.1.3
* remove python2 package

Soon afterwards there was an RC bug because this removed package still
had a two digit number of reverse dependencies.

This is the tip of the iceberg I mentioned.

Triaging would imply a valid technical reason like problems with the
Python 2 module, not blind dropping out of a desire to kill Python 2.

cu
Adrian

[Ondrej Novy]

Hi,

Post by Adrian Bunk
Triaging would imply a valid technical reason like problems with the
Python 2 module, not blind dropping out of a desire to kill Python 2.

I completely agree with you Adrian, thanks!

[Ben Hutchings]

[...]

Post by Adrian Bunk

Post by Thomas Goirand
...

Post by Adrian Bunk
Maintaining the Python 2 interpreter is actually reasonably trivial.

That's not the question I was asking. I was asking if someone is
volunteering for the next 5 years (ie: Buster + LTS support, that's
quite a huge commitment).

LTS is an effort by a 3rd party external company,
what and how they support in LTS is not our problem.

[...]

LTS is a Debian project running on Debian infrastructure. It's true
that most of the news you see about it is related to Freexian, but
there are other contributors quietly working on it.

But you are right that packages can be excluded from support - and in
fact this can be done during the regular stable period, at the
beginning of LTS, or later. Specific exclusions are recorded in the
debian-security-support package.

Ben.

[Chris Lamb]

Adrian,

Post by Adrian Bunk
We (Debian) have decided to support Python 2.7 in buster, like it or not.
At that point it is not up to individual maintainers to sabotage
Python 2.7 support in buster by dropping Python 2 packages without
a valid technical reason.

Most maintainers are likely dropping Python 2 modules due to a
misunderstanding or otherwise mistaken misinterpretation of the
situation surrounding Python 2.x.

Any such removals are fairly easily and cordially reversed if
their error is politely pointed out.

To therefore jump to assuming this is "sabotage" seems to be
premature and unnecessarily antagonistic.


Best wishes,

[Thomas Goirand]

Post by Adrian Bunk

Post by Thomas Goirand

Post by Adrian Bunk

Post by Thomas Goirand
...
This cannot go on, and on, and on, and on... We have to send a clear
message on the right direction, which is Python 2 removal. Yes, removal!
Why are we even discussing this? Isn't it obvious?

It is not for us to decide what tools our users should use,
we should support them no matter what tools they have to use.

I'd really love if it was the case. Reality is otherwise.
What if our users want to use Python 2.4? Will we do it? Of course not,
at some point, we may say it's not doable reasonably, because of the
lost of upstream support and the work it involves to keep it alive. So
yes, it's up to us to decide what we're able or what we want to do for
our users, and our users cannot dictate what tool we must maintain. It
has to be the other way around, like it or not.

We (Debian) have decided to support Python 2.7 in buster, like it or not.

Adrian,

You've made general comments about supporting our users, and I reacted
to that and wrote that we can't always please our users because of real
world difficulties to do that.

Now, if you're back to supporting Python 2.7, if the Debian project is
committed to that, of course, I support the decision if it's a team
decision!

Post by Adrian Bunk

Post by Thomas Goirand
That's not the question I was asking. I was asking if someone is
volunteering for the next 5 years (ie: Buster + LTS support, that's
quite a huge commitment).

LTS is an effort by a 3rd party external company,
what and how they support in LTS is not our problem.

Wow, hopefully not ! LTS is an effort by the Debian project. What the
external company does is an effort to *FUND* individual to work on it.
Currently, only Freexian does this sponsor gathering and redistribution
work, but it's my understanding that it would be perfectly valid (but
IMO probably not desirable at this point) that another company competes
with this funding effort.

That's a huge difference. If it was only the way you wrote, IE only a
private company stuff, then the LTS uploads wouldn't be allowed on the
Debian infrastructure. If here, I'm mistaking, then please, correct me,
and let's discuss again about LTS and how Debian supports it.

So yes, LTS is fully part of the Debian project, and how Python 2.7 is
supported should IMO very much be our concern. Now, we think that we
should only support Python 2.7 for more than until Buster is EOL and
becomes an LTS, I support this restriction. Though it'd be nice if we
had this consensus before Buster is frozen and have the discussion
closed early.

Post by Adrian Bunk
We already had several cases of a changelog for a new Debian revision
* move vcs to salsa
* Standards-Version: 4.1.3
* remove python2 package
Soon afterwards there was an RC bug because this removed package still
had a two digit number of reverse dependencies.

That is very wrong, I very much agree with what you wrote just above.

But what about softly removing Python 2 support when there's no reverse
(build-)dependency in the archive, like I already started to do here and
there? Do you think that's acceptable?

If you do think it's acceptable, then why not starting that effort as
soon as possible, since it takes a much much longer time to do things
this way, which is also the way that is the least disrupting for our
users? (/me just earned a godwin point for invoking social contract #4:)

Cheers,

Thomas Goirand (zigo)

[Adrian Bunk]

Post by Ondrej Novy

Post by Adrian Bunk

Post by Thomas Goirand

Post by Adrian Bunk

Post by Thomas Goirand
...
This cannot go on, and on, and on, and on... We have to send a clear
message on the right direction, which is Python 2 removal. Yes, removal!
Why are we even discussing this? Isn't it obvious?

It is not for us to decide what tools our users should use,
we should support them no matter what tools they have to use.

I'd really love if it was the case. Reality is otherwise.
What if our users want to use Python 2.4? Will we do it? Of course not,
at some point, we may say it's not doable reasonably, because of the
lost of upstream support and the work it involves to keep it alive. So
yes, it's up to us to decide what we're able or what we want to do for
our users, and our users cannot dictate what tool we must maintain. It
has to be the other way around, like it or not.

We (Debian) have decided to support Python 2.7 in buster, like it or not.

Adrian,
You've made general comments about supporting our users, and I reacted
to that and wrote that we can't always please our users because of real
world difficulties to do that.

My point is exactly about the difference between "We have to send a
clear message on the right direction, which is Python 2 removal."
and "real world difficulties".

There are no real world difficulties with supporting the Python 2.7
interpreter and most Pythin 2 modules in buster.

It is actually more relevant what is supportable by us than whether or
not upstream still supports it.

Rarity of CVEs makes security support easy, while many security issues
with fast-paced upstream development can be a security support nightmare
(e.g. webkit).

There are of difficult cases like "upstream dropped Python 2 support",
these are a separate topic.

Post by Ondrej Novy
...

Post by Adrian Bunk

Post by Thomas Goirand
That's not the question I was asking. I was asking if someone is
volunteering for the next 5 years (ie: Buster + LTS support, that's
quite a huge commitment).

LTS is an effort by a 3rd party external company,
what and how they support in LTS is not our problem.

Wow, hopefully not ! LTS is an effort by the Debian project. What the
external company does is an effort to *FUND* individual to work on it.
Currently, only Freexian does this sponsor gathering and redistribution
work, but it's my understanding that it would be perfectly valid (but
IMO probably not desirable at this point) that another company competes
with this funding effort.

I don't think it would ever be a good idea to have companies competing
on that.

No matter whether it's the Debian release team for stable or Freexian
for LTS, you need one place where the decisions are made.

Post by Ondrej Novy
That's a huge difference. If it was only the way you wrote, IE only a
private company stuff, then the LTS uploads wouldn't be allowed on the
Debian infrastructure. If here, I'm mistaking, then please, correct me,
and let's discuss again about LTS and how Debian supports it.
So yes, LTS is fully part of the Debian project, and how Python 2.7 is
supported should IMO very much be our concern. Now, we think that we
should only support Python 2.7 for more than until Buster is EOL and
becomes an LTS, I support this restriction. Though it'd be nice if we
had this consensus before Buster is frozen and have the discussion
closed early.

Such decisions are made by Freexian and their sponsors.

Post by Ondrej Novy

Post by Adrian Bunk
We already had several cases of a changelog for a new Debian revision
* move vcs to salsa
* Standards-Version: 4.1.3
* remove python2 package
Soon afterwards there was an RC bug because this removed package still
had a two digit number of reverse dependencies.

That is very wrong, I very much agree with what you wrote just above.
But what about softly removing Python 2 support when there's no reverse
(build-)dependency in the archive, like I already started to do here and
there? Do you think that's acceptable?
...

Unfortunately this would be the problem below the tip of the iceberg:

Python modules are not provided only for usage in other packages.

Users who are using our packaged python with the packaged modules
might suddenly lose modules they need.

Post by Ondrej Novy
Cheers,
Thomas Goirand (zigo)

cu
Adrian

[Holger Levsen]

Post by Adrian Bunk

Post by Thomas Goirand
Wow, hopefully not ! LTS is an effort by the Debian project. What the
external company does is an effort to *FUND* individual to work on it.
Currently, only Freexian does this sponsor gathering and redistribution
work, but it's my understanding that it would be perfectly valid (but
IMO probably not desirable at this point) that another company competes
with this funding effort.

I don't think it would ever be a good idea to have companies competing
on that.
No matter whether it's the Debian release team for stable or Freexian
for LTS, you need one place where the decisions are made.

Thomas pointed out that the (current) LTS design allows for several
companies competing about *funding sources*, not about the work.

Eg there could be a company collecting money to exclusivly fund DDs from
Latin America or only women or... anyone.

Post by Adrian Bunk

Post by Thomas Goirand
So yes, LTS is fully part of the Debian project, and how Python 2.7 is
supported should IMO very much be our concern.

Such decisions are made by Freexian and their sponsors.

No, those decisions are made by those doing the work. (And eg I work for
myself, not for Freexian.)

[Raphael Hertzog]

Hi,

Post by Adrian Bunk

Post by Thomas Goirand
So yes, LTS is fully part of the Debian project, and how Python 2.7 is
supported should IMO very much be our concern. Now, we think that we
should only support Python 2.7 for more than until Buster is EOL and
becomes an LTS, I support this restriction. Though it'd be nice if we
had this consensus before Buster is frozen and have the discussion
closed early.

Such decisions are made by Freexian and their sponsors.

In fact, such decisions are made collectively on
debian-***@lists.debian.org based on the inputs of all stakeholders.

Freexian does not make such decisions. Freexian handles the funding
and relations with the sponsors, but it is not the decision-maker on
what gets supported or not.

Cheers,

[Jeremy Stanley]

On 2018-04-21 20:57:55 +0300 (+0300), Adrian Bunk wrote:
[...]

Post by Adrian Bunk
There are of course cases (e.g. OpenStack) where providing Python 2
modules in buster is not possible with reasonable effort.

Consider me mildly curious as to why that is. The OpenStack
community rigorously tests all its current projects (both libraries
and services) against Python 2.7, and is only just now starting to
plan for some lengthy discussions about how to evaluate when it will
be safe to officially stop claiming support for that or whether it
will even accept new projects which lack 2.7 support. It's currently
got a mandate to support latest RHEL, which still doesn't officially
provide Python 3.x (and no, "software collections" doesn't count).

[Jeremy Bicha]

Post by Adrian Bunk
The tip of the iceberg are some recent cases where Python 2 modules
were dropped that still had reverse dependencies in unstable, but
also for those without there will in many cases be users who will
still need these modules in buster.

Once a Python2 library has no reverse depends, I see no reason in
general why the Maintainer can't remove the Python2 library. Further,
I think we ought to encourage Maintainers to remove the Python2
libraries.

You and I seem to be clashing a bit often on the issue of when it is
appropriate to remove obsolete packages from Debian. It seems like
some of your resistance is a bit theoretical. It sounds to me like
you're saying don't remove any Python2 libraries because somebody
somewhere might find it inconvenient to need to port some third-party
app to Python3 before 2022. But I think if we had that philosophy, we
wouldn't ever remove anything until identified security concerns force
it out.

Jeremy Bicha

[Scott Kitterman]

Post by Jeremy Bicha

Post by Adrian Bunk
The tip of the iceberg are some recent cases where Python 2 modules
were dropped that still had reverse dependencies in unstable, but
also for those without there will in many cases be users who will
still need these modules in buster.

Once a Python2 library has no reverse depends, I see no reason in
general why the Maintainer can't remove the Python2 library. Further,
I think we ought to encourage Maintainers to remove the Python2
libraries.
You and I seem to be clashing a bit often on the issue of when it is
appropriate to remove obsolete packages from Debian. It seems like
some of your resistance is a bit theoretical. It sounds to me like
you're saying don't remove any Python2 libraries because somebody
somewhere might find it inconvenient to need to port some third-party
app to Python3 before 2022. But I think if we had that philosophy, we
wouldn't ever remove anything until identified security concerns force
it out.

Since we are supporting Python2 in the next release, there is no value in dumping python-* packages now. Unlike many areas of the archive, Python packages are actively used by third-party code that isn't and won't be in Debian.

I've personally invested time and political capital in external projects I've worked on to get people to use Debian packages as a stable base and not just download semi-random code from wherever. Please don't make this harder.

I'm generally in favor of getting rid of old stuff, but python2 isn't there yet. It's much newer and better maintained than things like Qt4 that are getting to the better part of a decade without upstream support.

Please wait until after Buster and then feel free to resume the charge.

Scott K

[Luke Faraone]

Post by Scott Kitterman

Post by Jeremy Bicha
You and I seem to be clashing a bit often on the issue of when it is
appropriate to remove obsolete packages from Debian. It seems like
some of your resistance is a bit theoretical. It sounds to me like
you're saying don't remove any Python2 libraries because somebody
somewhere might find it inconvenient to need to port some third-party
app to Python3 before 2022. But I think if we had that philosophy, we
wouldn't ever remove anything until identified security concerns force
it out.

Since we are supporting Python2 in the next release, there is no value in dumping python-* packages now. Unlike many areas of the archive, Python packages are actively used by third-party code that isn't and won't be in Debian.
I've personally invested time and political capital in external projects I've worked on to get people to use Debian packages as a stable base and not just download semi-random code from wherever. Please don't make this harder.

+100 to this. Even at a previous life in a fast-moving startup, I
convinced the rest of engineering that "if you want to use it, and it
isn't in Debian, you get to package it first".

(aside, also had the advantage of convincing people to use more
commonly-used packages, rather than some random fork of a fork they
found one day…)

Cheers,
Luke Faraone

[Thomas Goirand]

Since we are supporting Python2 in the next release, there is no value> in dumping python-* packages now. Unlike many areas of the archive,
Python packages are actively used by third-party code that isn't and
won't be in Debian.

There's always value to start de-crufting legacy stuff early. That being
said, I hear your message: this is breaking 3rd party code, and we need
to keep this in mind.

I'm generally in favor of getting rid of old stuff, but python2 isn't
there yet.

Right. But I do believe we need to be very careful to not send a wrong
message to our users. Debian deprecating Python 2 is good. A strong,
bold deprecation message is needed, even if we want to continue
supporting Python 2 for a bit more.

Please wait until after Buster and then feel free to resume the charge.

IMO, it's already too late to get rid of all Python 2 for Buster. It's
just too much work, unfortunately. Though could we get into the
agreement that we *MUST* get rid of all traces of Python 2 for bullseye?
Can we agree that we wont change our minds?

We need to discuss what this means. Maybe adding a hard lintian *error*
when there's Python 2 just right after Buster is released? Or a little
bit after that, to leave enough time for maintainers to do the removal?

Looking at other distros is interesting. If I understand well, they will
never have Python 2 and 3 interpreters in the distro, and will
completely switch from 2 to 3 at once. I very much prefer the Debian
way. Though at some point, we need to take the hard decision, and if we
keep Python 2 forever and never act, the our way is pointless.

Cheers,

Thomas Goirand (zigo)

[Niels Thykier]

Post by Thomas Goirand
[...]

Post by Scott Kitterman
I'm generally in favor of getting rid of old stuff, but python2 isn't
there yet.

Right. But I do believe we need to be very careful to not send a wrong
message to our users. Debian deprecating Python 2 is good. A strong,
bold deprecation message is needed, even if we want to continue
supporting Python 2 for a bit more.

Reminder: The release-notes does have a section for "deprecations" (Note
as I recall, it was empty for stretch, so it is probably not visible there).

If there is consensus that buster is the last release to support
python2, then we can document it in the release-notes.

Thanks,
~Niels

[Matthias Klose]

Post by Niels Thykier

Post by Thomas Goirand
[...]

Post by Scott Kitterman
I'm generally in favor of getting rid of old stuff, but python2 isn't
there yet.

Right. But I do believe we need to be very careful to not send a wrong
message to our users. Debian deprecating Python 2 is good. A strong,
bold deprecation message is needed, even if we want to continue
supporting Python 2 for a bit more.

Reminder: The release-notes does have a section for "deprecations" (Note
as I recall, it was empty for stretch, so it is probably not visible there).
If there is consensus that buster is the last release to support
python2, then we can document it in the release-notes.

if that's possible, fine. otoh, I would be fine to have a very reduced set of
python2 apps in buster+1, if that's needed. Sure we can remove mercurial and
OpenStack if they are not ready for Python3, but I'd like to avoid that. It
doesn't mean that we should any old, upstream unmaintained Python2 dependent
package.

Matthias

[Jeremy Stanley]

On 2018-04-24 07:45:16 +0200 (+0200), Matthias Klose wrote:
[...]

Sure we can remove mercurial and OpenStack if they are not ready
for Python3, but I'd like to avoid that. It doesn't mean that we
should any old, upstream unmaintained Python2 dependent package.

To be clear, OpenStack has been thoroughly testing its upstream
releases against Python 3 for a while. The reason it was brought up
earlier in this thread is because Thomas has recently dropped Python
2.7 builds for it and switched to only building Python 3.x packages.

[Thomas Goirand]

Post by Jeremy Stanley
The reason it was brought up
earlier in this thread is because Thomas has recently dropped Python
2.7 builds for it and switched to only building Python 3.x packages.

I have *not* removed Python 2 support. Only services are switched to
Python 3 by default. It was a hard switch as they couldn't reasonably be
dual-python (too much packaging work for no real benefits). For example,
there's still Python 2 modules for clients.

I would like to start dropping Python 2 as early as possible though. The
only question that remains is: how many people still have Python 2 only
code using clients.

Cheers,

Thomas Goirand (zigo)

[Ondrej Novy]

Hi,

Post by Thomas Goirand
I would like to start dropping Python 2 as early as possible though. The
only question that remains is: how many people still have Python 2 only
code using clients.

/me

And because:

1. OS clients are already packaged for Py2 now
2. upstreams official supports Py2 now (and probably will support in OS
release targeted for Buster)
3. Python 2 will be in Buster

I didn't see any real benefits dropping them before Buster release. Let's
drop them just after Buster release.

[Thomas Goirand]

Post by Thomas Goirand
Hi,
I would like to start dropping Python 2 as early as possible though. The
only question that remains is: how many people still have Python 2 only
code using clients.
/me
1. OS clients are already packaged for Py2 now
2. upstreams official supports Py2 now (and probably will support in OS
release targeted for Buster)
3. Python 2 will be in Buster
I didn't see any real benefits dropping them before Buster release.
Let's drop them just after Buster release.

I would have preferred if we had this discussion in our own list,
however since you've started it here, let's do it. I'm adding the
OpenStack list just for the archive.

I'd love to get the removal of Python 2 prepared for the next OpenStack
release, Rocky, due for next August, and which will be the OpenStack
release for Buster. I see a lot of real benefits. Here's a few of them
which are popping now, but I don't think it's exhaustive:

- faster build time (no need to test with Python 2, so less chances of
build failure).

- no chance to have any Python 2 packages installed, so we're sure we're
on a full Python 3 stack (in my current setup, unfortunately some Python
2 packages are still pulled). This may go on for another 3 years if we
don't remove Python 2 now, with the added issue that it will pull
*older* version of clients if selecting Python 2 and if we still have
them in Buster (ie: case of OpenStack backports on top of Stable).

- packaging and decrufting will take a long time, so we'd better start
early. Especially if we want to do it the proper way, without breaking
any reverse (build-)dependency that are outside of OpenStack.

- at some point, even upstream will move to Python 3, and Python 2 will
be the legacy old thing. Do we really want to be the only distribution
that will hit these Python 2 bugs? In such case, we'll be alone writing
these Python 2 bugfix patches. :/

- Other distros (RHEL and Ubuntu) will move to Python 3 only in the next
OpenStack development cycle, meaning we'll be alone with Python 2
support at some point.

The other thing is, it's ready! I was able to do functional testing on
top of the Python 3 release of OpenStack Debian packages. So why should
we hold any longer?

The only benefit we get with keeping Python 2 support is for thirdparty
software that is still using Python 2. We don't have this in Debian at
the moment. What's the status in your company? Do you still need Python
2 for your internal use? What's the ETA for switching to Python 3?

Cheers,

Thomas Goirand (zigo)

[Ondrej Novy]

Hi,

Post by Thomas Goirand
- faster build time (no need to test with Python 2, so less chances of
build failure).

build is done once, customer happiness is for years (buster lifetime).

- no chance to have any Python 2 packages installed, so we're sure we're

Post by Thomas Goirand
on a full Python 3 stack (in my current setup, unfortunately some Python
2 packages are still pulled). This may go on for another 3 years if we
don't remove Python 2 now, with the added issue that it will pull
*older* version of clients if selecting Python 2 and if we still have
them in Buster (ie: case of OpenStack backports on top of Stable).

so let's fix packages to not pull Py2 if it's not needed.

Post by Thomas Goirand
- packaging and decrufting will take a long time, so we'd better start
early. Especially if we want to do it the proper way, without breaking
any reverse (build-)dependency that are outside of OpenStack.

more than one release cycle? I'm sure we can do it during Bullseye. And we
will do it for non-OS Python packages too. Better is to do removing in
earlier phase of development cycle.

- at some point, even upstream will move to Python 3, and Python 2 will

Post by Thomas Goirand
be the legacy old thing. Do we really want to be the only distribution
that will hit these Python 2 bugs? In such case, we'll be alone writing
these Python 2 bugfix patches. :/

that point is after Rocky, so it's doesn't matter for Buster.

- Other distros (RHEL and Ubuntu) will move to Python 3 only in the next

Post by Thomas Goirand
OpenStack development cycle, meaning we'll be alone with Python 2
support at some point.

any links where Ubuntu said they will not support Py2 for Rocky? I'm sure
we will not be alone.

The other thing is, it's ready! I was able to do functional testing on

Post by Thomas Goirand
top of the Python 3 release of OpenStack Debian packages. So why should
we hold any longer?

What is ready? Py3 support, or Py2 dropping? If foremost, let's do it.
Let's have good Py3-only services, let's have default Py3 clients, but
support Py2 clients. It cost us nothing, it's already done this way.

Post by Thomas Goirand
The only benefit we get with keeping Python 2 support is for thirdparty
software that is still using Python 2. We don't have this in Debian at
the moment. What's the status in your company? Do you still need Python
2 for your internal use? What's the ETA for switching to Python 3?

yep, that's important benefit. Because our priority is our users.

"My" company is not important, important is our users and my company is
just one of them.

[Thomas Goirand]

Post by Thomas Goirand
- faster build time (no need to test with Python 2, so less chances of
build failure).
build is done once, customer happiness is for years (buster lifetime).

More work ...

Post by Thomas Goirand
- no chance to have any Python 2 packages installed, so we're sure we're
on a full Python 3 stack (in my current setup, unfortunately some Python
2 packages are still pulled). This may go on for another 3 years if we
don't remove Python 2 now, with the added issue that it will pull
*older* version of clients if selecting Python 2 and if we still have
them in Buster (ie: case of OpenStack backports on top of Stable).
so let's fix packages to not pull Py2 if it's not needed.

More work...

Post by Thomas Goirand
- packaging and decrufting will take a long time, so we'd better start
early. Especially if we want to do it the proper way, without breaking
any reverse (build-)dependency that are outside of OpenStack.
more than one release cycle? I'm sure we can do it during Bullseye. And
we will do it for non-OS Python packages too. Better is to do removing
in earlier phase of development cycle.

And even more work...

If you're only points of argumentation are "well, it's easy, we just
need to do this or that", then it's not acceptable. There's no way you
can force anyone in Debian into doing one thing or another, if you don't
*significantly* contribute. Optimizing the way I can do OpenStack
packaging means I can do more with that limited time, and I really think
that's the best way to serve our users.

We've played that game last summer at Debconf, with lots of promises
from potential contributors. I accepted the rules, and lost the game
with more work for me, doing the way the team was supposed to do, but
finally being nearly the only one doing all. I wont do the same mistake
again unless I see things have changed *already* (ie: not just
promises). Here, I'm talking about at least 3 or 4 uploads per week (I
was tempted to write one per day...) of non-cosmetic changes.

There's btw a bunch of RC bugs in the BTS that I would love to get
fixed. Anyone up for contributing that?

Post by Thomas Goirand
- at some point, even upstream will move to Python 3, and Python 2 will
be the legacy old thing. Do we really want to be the only distribution
that will hit these Python 2 bugs? In such case, we'll be alone writing
these Python 2 bugfix patches. :/
that point is after Rocky, so it's doesn't matter for Buster.
- Other distros (RHEL and Ubuntu) will move to Python 3 only in the next
OpenStack development cycle, meaning we'll be alone with Python 2
support at some point.
any links where Ubuntu said they will not support Py2 for Rocky? I'm
sure we will not be alone.

This was part of a non-formal discussion with Ubuntu people, I'm not
sure it would be right to say with who, but I'm sure you can guess. We
can discuss with that person again and see what he says now, as maybe
things have changed since that last conversation. So no, I don't have
any formal announcement/links to support that it will happen in Ubuntu
for Rocky.

Post by Thomas Goirand
The other thing is, it's ready! I was able to do functional testing on
top of the Python 3 release of OpenStack Debian packages. So why should
we hold any longer?
What is ready? Py3 support, or Py2 dropping? If foremost, let's do it.
Let's have good Py3-only services, let's have default Py3 clients, but
support Py2 clients. It cost us nothing, it's already done this way.

That's were I don't agree (with both you and Adrian Bunk). Supporting a
dual-python everything but services, and keeping the legacy in place
isn't costless. It is a significant overhead. The question is just if
it's worth the effort or not.

Post by Thomas Goirand
The only benefit we get with keeping Python 2 support is for thirdparty
software that is still using Python 2. We don't have this in Debian at
the moment. What's the status in your company? Do you still need Python
2 for your internal use? What's the ETA for switching to Python 3?
yep, that's important benefit. Because our priority is our users.

LOL ! Writing "our priority is our users" in a thread is nearly equal to
winning a godwin point, do you know? Just like it, it's supposed to end
the discussion with no argumentation possible.

Jokes aside (yes, the above is only a joke...), to serve our users the
best way possible, we need to set priorities the right way, and be
efficient. Because being efficient in the way we do the work is the best
way to get more time for things that matters. Add a bit of polish to
everything, so that users are happy. There's the need for polishing many
rough edges of OpenStack in Debian, I'm sure you know it.

Now, what needs to be accounted, is how many people use Py2 clients (and
of course, by that, I mean the Python modules, not the cli). To answer
that question, we have unfortunately no data except your own case.

Post by Thomas Goirand
"My" company is not important, important is our users and my company is
just one of them.

But is the whole set made of just one? Or more? Impossible to say...
Which is why I asked you how much time it would take for fixing the
situation in your own company, as it could be a good example. How many
months do you need? 6 months? A year? It'd be super nice if you could
explain your use case in more details, and tell what your code base is
about. Though maybe you can't do it publicly? Or even, maybe you can't
tell me? I would understand in both cases. But if you can explain, it
would help understanding at least your use case, which may be similar to
others.

Cheers,

Thomas Goirand (zigo)

[Ondrej Novy]

Hi,

Post by Thomas Goirand

Post by Thomas Goirand
- faster build time (no need to test with Python 2, so less chances

of

Post by Thomas Goirand
build failure).
build is done once, customer happiness is for years (buster lifetime).

More work ...

more work for machines (build time). I never seen build failure when Py3
tests was fine and Py2 wasn't. Because OpenStack upstream officially
support Py2.

Post by Thomas Goirand

Post by Thomas Goirand
- no chance to have any Python 2 packages installed, so we're sure

we're

Post by Thomas Goirand
on a full Python 3 stack (in my current setup, unfortunately some

Python

Post by Thomas Goirand
2 packages are still pulled). This may go on for another 3 years if

we

Post by Thomas Goirand
don't remove Python 2 now, with the added issue that it will pull
*older* version of clients if selecting Python 2 and if we still have
them in Buster (ie: case of OpenStack backports on top of Stable).
so let's fix packages to not pull Py2 if it's not needed.

More work...

we need to fix it in both cases. So it's same work.

Post by Thomas Goirand
- packaging and decrufting will take a long time, so we'd better start

Post by Thomas Goirand
early. Especially if we want to do it the proper way, without

breaking

Post by Thomas Goirand
any reverse (build-)dependency that are outside of OpenStack.
more than one release cycle? I'm sure we can do it during Bullseye. And
we will do it for non-OS Python packages too. Better is to do removing
in earlier phase of development cycle.

And even more work...

huh? Doing now or doing later. Same work.

Post by Thomas Goirand
There's btw a bunch of RC bugs in the BTS that I would love to get
fixed. Anyone up for contributing that?

problem is that nobody want's to cooperate with you, that's all. All other
arguments are useless. I already explained to you many times what is
problem.

Post by Thomas Goirand
This was part of a non-formal discussion with Ubuntu people, I'm not
sure it would be right to say with who, but I'm sure you can guess. We
can discuss with that person again and see what he says now, as maybe
things have changed since that last conversation. So no, I don't have
any formal announcement/links to support that it will happen in Ubuntu
for Rocky.

so it's just your opinion.

Now, what needs to be accounted, is how many people use Py2 clients (and

Post by Thomas Goirand
of course, by that, I mean the Python modules, not the cli). To answer
that question, we have unfortunately no data except your own case.

we have data:
https://qa.debian.org/popcon.php?package=python-openstackclient

Not ideal I know. But better than nothing.

But is the whole set made of just one? Or more? Impossible to say...

Post by Thomas Goirand
Which is why I asked you how much time it would take for fixing the
situation in your own company, as it could be a good example. How many
months do you need? 6 months? A year? It'd be super nice if you could
explain your use case in more details, and tell what your code base is
about. Though maybe you can't do it publicly? Or even, maybe you can't
tell me? I would understand in both cases. But if you can explain, it
would help understanding at least your use case, which may be similar to
others.

we need Buster stable period for Py2->Py3 migration. We are going to be
ready for Py3-only for Bullseye. Thousands of servers, millions lines of
code.

[Scott Kitterman]

On Thursday, April 26, 2018 10:40:49 AM Ondrej Novy wrote:
...

Post by Ondrej Novy
we need Buster stable period for Py2->Py3 migration. We are going to be
ready for Py3-only for Bullseye. Thousands of servers, millions lines of
code.

...

I know very little about the details of OpenStack, but in case a somewhat
parallel example is useful, that's approximately what Django will do.
Bullseye will be Django 2.0, which is Python 3 only. Buster is the pivot
release where the third party elements of the Django ecosystem almost all
support Python 3, so transition is possible to make ready for an all Python 3
future. (AIUI anyway, I'm not a Django maintainer)

Scott K

[Ondrej Novy]

Hi,

Post by Scott Kitterman
I know very little about the details of OpenStack, but in case a somewhat
parallel example is useful, that's approximately what Django will do.
Bullseye will be Django 2.0, which is Python 3 only. Buster is the pivot
release where the third party elements of the Django ecosystem almost all
support Python 3, so transition is possible to make ready for an all Python 3
future. (AIUI anyway, I'm not a Django maintainer)

which is perfect for seamless migration. +1

[Chris Lamb]

Hi Scott,

Post by Scott Kitterman
Bullseye will be Django 2.0, which is Python 3 only.

(Indeed. The only complication is that as we rejected having a
seperate source package for Django 2.x can't then upload this
version to unstable. It is, however, available in experimental.)


Regards,

[Thomas Goirand]

Post by Ondrej Novy
problem is that nobody want's to cooperate with you, that's all. All
other arguments are useless. I already explained to you many times what
is problem.

Excuse my words, but that's plain bullshit. A year ago, I did nothing
for the OpenStack Ocata release, and officially announced that I would
stop doing the work. Then nobody uploaded the release to Debian.

Please stop finger pointing at me anywhere public.

Cheers,

Thomas Goirand (zigo)

[Thomas Goirand]

Post by Matthias Klose
if that's possible, fine. otoh, I would be fine to have a very reduced set of
python2 apps in buster+1, if that's needed. Sure we can remove mercurial and
OpenStack if they are not ready for Python3

OpenStack in Sid/Buster is already fully switched to Python 3. It was
painful on the puppet-openstack part that needed to know about it, but
that is almost already fully addressed.

The only question that remains is: when do I start to aggressively
remove Python 2 support in OpenStack clients (and all other packages
therefore).

Cheers,

Thomas Goirand (zigo)

[Andrea Bolognani]

Post by Thomas Goirand
Looking at other distros is interesting. If I understand well, they will
never have Python 2 and 3 interpreters in the distro, and will
completely switch from 2 to 3 at once.

Unless I'm misunderstanding, I don't think you're correct.

To give a concrete example, Fedora switched to using Python 3
as the default several releases ago[1]; despite that, Python 2
is still available in the archive, and will get pulled in when
installing software that (regrettably) hasn't been ported yet.

The same is true for FreeBSD and, I believe, Ubuntu. I'm not
familiar with the approach other distributions and OS are
taking, but I would expect it to be fairly similar.

[1] https://fedoraproject.org/wiki/Changes/Python_3_as_Default

[Jeremy Stanley]

Post by Andrea Bolognani

Post by Thomas Goirand
Looking at other distros is interesting. If I understand well, they will
never have Python 2 and 3 interpreters in the distro, and will
completely switch from 2 to 3 at once.

Unless I'm misunderstanding, I don't think you're correct.
To give a concrete example, Fedora switched to using Python 3
as the default several releases ago[1]; despite that, Python 2
is still available in the archive, and will get pulled in when
installing software that (regrettably) hasn't been ported yet.
The same is true for FreeBSD and, I believe, Ubuntu. I'm not
familiar with the approach other distributions and OS are
taking, but I would expect it to be fairly similar.

[...]

Rumor has it that RHEL 8 will be dropping Python 2 while (finally!)
adding Python 3. Much of that is fueled by the Deprecated
Functionality[*] section of the RHEL 7.5 Release Notes wherein it
states, "Python 2 will be replaced with Python 3 in the next Red Hat
Enterprise Linux (RHEL) major release."

[*] <URL: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/7.5_release_notes/chap-red_hat_enterprise_linux-7.5_release_notes-deprecated_functionality >

[Andrea Bolognani]

Post by Jeremy Stanley

Post by Andrea Bolognani
To give a concrete example, Fedora switched to using Python 3
as the default several releases ago[1]; despite that, Python 2
is still available in the archive, and will get pulled in when
installing software that (regrettably) hasn't been ported yet.
The same is true for FreeBSD and, I believe, Ubuntu. I'm not
familiar with the approach other distributions and OS are
taking, but I would expect it to be fairly similar.

[...]
Rumor has it that RHEL 8 will be dropping Python 2 while (finally!)
adding Python 3. Much of that is fueled by the Deprecated
Functionality[*] section of the RHEL 7.5 Release Notes wherein it
states, "Python 2 will be replaced with Python 3 in the next Red Hat
Enterprise Linux (RHEL) major release."
Note that Python 3 is available to RHEL customers, and supported
on RHEL, as a part of Red Hat Software Collections.

So you could say that RHEL is taking the approach described above -
having a transitional period where both versions are available side
by side - with the only difference being that Python 3 is currently
not delivered through the same channel as Python 2.

[Jeremy Stanley]

On 2018-04-25 01:05:59 +0200 (+0200), Andrea Bolognani wrote:
[...]

Post by Andrea Bolognani
So you could say that RHEL is taking the approach described above -
having a transitional period where both versions are available side
by side - with the only difference being that Python 3 is currently
not delivered through the same channel as Python 2.

Given that "software collections" provides a containerized Python 3
build and basically none of the rest of the Python ecosystem
modules outside the stdlib (which would all require manual
rebuilding against it), this is nowhere close to the same as
providing an optional Python interpreter within the global system
context as Debian has done. At least the projects I work on don't
see RHEL software collections Python 3 as remotely supportable.

[Andrea Bolognani]

Post by Jeremy Stanley
[...]

Post by Andrea Bolognani
So you could say that RHEL is taking the approach described above -
having a transitional period where both versions are available side
by side - with the only difference being that Python 3 is currently
not delivered through the same channel as Python 2.

Given that "software collections" provides a containerized Python 3
build and basically none of the rest of the Python ecosystem
modules outside the stdlib (which would all require manual
rebuilding against it), this is nowhere close to the same as
providing an optional Python interpreter within the global system
context as Debian has done. At least the projects I work on don't
see RHEL software collections Python 3 as remotely supportable.

Fair enough; the point about distribution with lifecycles closer to
Debian's keeping Python 2 around for a while after switching their
default to Python 3 still stands.

[Scott Kitterman]

Post by Andrea Bolognani

Post by Jeremy Stanley
[...]

Post by Andrea Bolognani
So you could say that RHEL is taking the approach described above -
having a transitional period where both versions are available side
by side - with the only difference being that Python 3 is currently
not delivered through the same channel as Python 2.

Given that "software collections" provides a containerized Python 3
build and basically none of the rest of the Python ecosystem
modules outside the stdlib (which would all require manual
rebuilding against it), this is nowhere close to the same as
providing an optional Python interpreter within the global system
context as Debian has done. At least the projects I work on don't
see RHEL software collections Python 3 as remotely supportable.

Fair enough; the point about distribution with lifecycles closer to
Debian's keeping Python 2 around for a while after switching their
default to Python 3 still stands.

In Debian there's no such thing as a 'default' python. There's none in a minimal install. All that ends up on a system is what is pulled in by dependency.

Scott K

[The Wanderer]

Post by Scott Kitterman

Post by Andrea Bolognani

Post by Jeremy Stanley
Given that "software collections" provides a containerized Python
3 build and basically none of the rest of the Python ecosystem
modules outside the stdlib (which would all require manual
rebuilding against it), this is nowhere close to the same as
providing an optional Python interpreter within the global
system context as Debian has done. At least the projects I work
on don't see RHEL software collections Python 3 as remotely
supportable.

Fair enough; the point about distribution with lifecycles closer
to Debian's keeping Python 2 around for a while after switching
their default to Python 3 still stands.

In Debian there's no such thing as a 'default' python. There's none
in a minimal install. All that ends up on a system is what is pulled
in by dependency.

The word "default" means different things in different contexts and to
different people, and the one you cite (which I parse as being roughly
"installed automatically as part of the distribution") is only one of
them.

It is true that there is no such thing as "the version of Python which
will be present in a (default / minimal) install of Debian". However,
there is such a thing as "the version of Python which will be present in
a default install of Python on Debian" - meaning, an install of Python
on Debian with no version explicitly specified - and it is reasonable to
refer to that latter as "Debian's default version of Python".

The simple, obvious means of installing Python in Debian - either
manually, or as a dependency of another package - is via the package
named 'python'. At present, in current testing, doing this will pull in
python2.7 and will not (as far as I can see) pull in anything named
python3*.

That is enough to qualify Python 2 as "the Python which will be present
in a default install of Python on Debian", and therefore as "Debian's
default version of Python".

[Stephan Seitz]

Post by The Wanderer
The simple, obvious means of installing Python in Debian - either
manually, or as a dependency of another package - is via the package
named 'python'. At present, in current testing, doing this will pull in

I don’t think you can see it this way. For now /usr/bin/python is a link
to python2 and, as was mentioned on this list, always will be until
a time comes when version 2 will be so long dead that no one remembers
it.

If you want version 3 you have to install python3.

[***@osgiliath]: dpkg -s python
Package: python
This package is a dependency package, which depends on Debian's default
Python version (currently v2.7).

[***@osgiliath]: dpkg -s python3
Package: python3
This package is a dependency package, which depends on Debian's default
Python 3 version (currently v3.6).

As you can see both packages pull in their default version of python.

Of course someone can make a transition so that python will be renamed to
python2 and python3 will be renamed to python.

Shade and sweet water!

Stephan

[Scott Kitterman]

Post by The Wanderer

Post by Scott Kitterman

Post by Andrea Bolognani

Post by Jeremy Stanley
Given that "software collections" provides a containerized Python
3 build and basically none of the rest of the Python ecosystem
modules outside the stdlib (which would all require manual
rebuilding against it), this is nowhere close to the same as
providing an optional Python interpreter within the global
system context as Debian has done. At least the projects I work
on don't see RHEL software collections Python 3 as remotely
supportable.

Fair enough; the point about distribution with lifecycles closer
to Debian's keeping Python 2 around for a while after switching
their default to Python 3 still stands.

In Debian there's no such thing as a 'default' python. There's none
in a minimal install. All that ends up on a system is what is pulled
in by dependency.

The word "default" means different things in different contexts and to
different people, and the one you cite (which I parse as being roughly
"installed automatically as part of the distribution") is only one of
them.
It is true that there is no such thing as "the version of Python which
will be present in a (default / minimal) install of Debian". However,
there is such a thing as "the version of Python which will be present in
a default install of Python on Debian" - meaning, an install of Python
on Debian with no version explicitly specified - and it is reasonable to
refer to that latter as "Debian's default version of Python".
The simple, obvious means of installing Python in Debian - either
manually, or as a dependency of another package - is via the package
named 'python'. At present, in current testing, doing this will pull in
python2.7 and will not (as far as I can see) pull in anything named
python3*.
That is enough to qualify Python 2 as "the Python which will be present
in a default install of Python on Debian", and therefore as "Debian's
default version of Python".

That's true and will likely remain so. Personally, I expect both the package name python and the usr/bin/python symlink will be retired when python2.7 is removed.

Even after Debian removes python2.7, we expect local usage to remain for the foreseeable future and it would not be doing our users any favors to pollute the python2 namespace with python3.

Scott K

[Thomas Goirand]

Post by The Wanderer
The simple, obvious means of installing Python in Debian - either
manually, or as a dependency of another package - is via the package
named 'python'. At present, in current testing, doing this will pull in
python2.7 and will not (as far as I can see) pull in anything named
python3*.

It just happen to be the case that the Python 2 package is named
"python", when really, it should have been called "python2".

Just like python-foo is the Python 2 module package for the "foo"
module, and probably it would have been better called "python2-foo".

That's probably unfortunate naming, but never the less, there's still no
such thing as the "default python interpreter" package in Debian, our
users still have to manually choose between 2 and 3.

Post by The Wanderer
That is enough to qualify Python 2 as "the Python which will be present
in a default install of Python on Debian", and therefore as "Debian's
default version of Python".

No ! See above.

Cheers,

Thomas Goirand (zigo)

[Nicholas D Steeves]

Post by Thomas Goirand

Since we are supporting Python2 in the next release, there is no value> in dumping python-* packages now. Unlike many areas of the archive,
Python packages are actively used by third-party code that isn't and
won't be in Debian.

There's always value to start de-crufting legacy stuff early. That being
said, I hear your message: this is breaking 3rd party code, and we need
to keep this in mind.

I'm generally in favor of getting rid of old stuff, but python2 isn't
there yet.

Right. But I do believe we need to be very careful to not send a wrong
message to our users. Debian deprecating Python 2 is good. A strong,
bold deprecation message is needed, even if we want to continue
supporting Python 2 for a bit more.

At what stage should Python IDEs uploaded to NEW disable or not
install Python 2 support? Now?

Should Python 2-specific IDEs and/or Python 2-specific educational
materials be removed from the archive? If so, then when?

Andreas, I've CCed you because we've been in touch in the past, and
because I have friends who do graduate work in bioinformatics on
Debian systems. With the upcoming end of Python 2 in Debian, what can
we do to avoid losing them to CentOS? Also, if appropriate and if it
wouldn't be too much work, does Debian have a role in encouraging
new masters and PhD candidates to choose Python 3?

Sincerely,
Nicholas

[olivier sallou]

Post by Thomas Goirand

Post by Scott Kitterman
Since we are supporting Python2 in the next release, there is no

value> in dumping python-* packages now. Unlike many areas of the archive,

Post by Thomas Goirand

Post by Scott Kitterman
Python packages are actively used by third-party code that isn't and
won't be in Debian.

There's always value to start de-crufting legacy stuff early. That being
said, I hear your message: this is breaking 3rd party code, and we need
to keep this in mind.

Post by Scott Kitterman
I'm generally in favor of getting rid of old stuff, but python2 isn't
there yet.

Right. But I do believe we need to be very careful to not send a wrong
message to our users. Debian deprecating Python 2 is good. A strong,
bold deprecation message is needed, even if we want to continue
supporting Python 2 for a bit more.

At what stage should Python IDEs uploaded to NEW disable or not
install Python 2 support? Now?
Should Python 2-specific IDEs and/or Python 2-specific educational
materials be removed from the archive? If so, then when?
Andreas, I've CCed you because we've been in touch in the past, and
because I have friends who do graduate work in bioinformatics on
Debian systems. With the upcoming end of Python 2 in Debian, what can
we do to avoid losing them to CentOS? Also, if appropriate and if it
wouldn't be too much work, does Debian have a role in encouraging
new masters and PhD candidates to choose Python 3?

I do not see your point here. Regarding education, using python 2 or 3
should be transparent. The matter is only to have the expected
programs/libraries available in python3. The only issue would be for
students not to have their favorite program available.
Anyway, py2 will not be supported anymore at some time...
Olivier

Sincerely,
Nicholas

[Thomas Goirand]

Post by olivier sallou
Regarding education, using python 2 or 3
should be transparent.

At this point, new comers should learn Python 3 only.

Cheers,

Thomas Goirand (zigo)

[Andreas Tille]

Hi Nicholas,

Post by Nicholas D Steeves
At what stage should Python IDEs uploaded to NEW disable or not
install Python 2 support? Now?
Should Python 2-specific IDEs and/or Python 2-specific educational
materials be removed from the archive? If so, then when?
Andreas, I've CCed you because we've been in touch in the past, and
because I have friends who do graduate work in bioinformatics on
Debian systems.

Thanks for the trust you have in me specifically but it is wrong to
assume that I have a more competent opinion in this topic than other
developers who raised their opinion in this thread (rather the contrary
since I'm not very deeply involved in Python).

Since you asked about bioinformatics I can simply confirm that there is
some code around that is used and definitely useful (may be partly
without any replacement). That's a pity for the users who might need
that code. The only chance we have is to ask for helping hands to port
**and** **test** the Python2 code to Python3. Since two GSoC/Outreachy
periods I'm mentoring students to write autopkgtests - the next
promising outreachy student was just accepted yesterday. So at some
point in time we will hopefully be able to at least reproduce the
results Python2 code has created in the test cases in a potential
Python3 port. Helping hands are always welcome - feel free to send this
message to the friends you have.

Post by Nicholas D Steeves
With the upcoming end of Python 2 in Debian, what can
we do to avoid losing them to CentOS?

Everybody chooses the distribution that has advantages for the task that
needs to be done. I do not know about CentOS, but Debian is organised
as Do-O-cracy. Everybody who realises a problem is free to fix it
inside Debian. For instance in 2001 I realised that Debian is not
specifically helpful for biologists and medicine. I'm on a good way
to fix this. At that time Debian was the only distribution that enabled
this way of "fix".

Using the menace to switch the distribution as a club to push some
Debian developer does not work. I did not interpreted your sentence
that way - I just want to express in some more drastical words that you
can not push somebody elses agenda on some Debian developer by this kind
of argument. The best way to reach something in Debian is to start
yourself doing what you need and if you have no success ask for help.

Post by Nicholas D Steeves
Also, if appropriate and if it
wouldn't be too much work, does Debian have a role in encouraging
new masters and PhD candidates to choose Python 3?

I do not think that Debian has such a role. I'd consider it common
sense to not use outdated development tools and that is a known fact
that was made pretty clear by Python developers. The role of any
distribution is to provide existing software and care for a secure and
reliable installation process, provide security updates, etc. It is not
the role to hand-hold random users to pick their tools sensibly - or
rather the only way to implement this hand-holding is if we do not
provide anything that should not be used any more. So also drastically
spoken: The only possibly answer I could imagine to your question is
that the people who are currently asking for the removal of Python2 are
fullfilling this role. May be a bit more drastical than you might wish
(and my personal opinion is to make the transition as smooth as possible
- but see in the beginning about my personal competence in this
question) but otherwise we need to trust the common sense of our users.

Kind regards

Andreas.

[1] https://communitywiki.org/wiki/DoOcracy

[Steve Robbins]

Post by Jeremy Bicha
But I think if we had that philosophy, we
wouldn't ever remove anything until identified security concerns force
it out.

I don't see anything wrong with that philosophy.

Assuming someone is willing to maintain a package and it isn't causing harm
("security concerns"), why would one want to remove it?

-S

[Ole Streicher]

Post by Steve Robbins

Post by Jeremy Bicha
But I think if we had that philosophy, we
wouldn't ever remove anything until identified security concerns force
it out.

I don't see anything wrong with that philosophy.
Assuming someone is willing to maintain a package and it isn't causing harm
("security concerns"), why would one want to remove it?

Sure, *if* someone is willing to maintain. But, as a maintainer of a
number of Python packages, I want to get rid of Python 2 support ASAP:

* Many upstreams already drop Python 2 support from their newest
releases. Keeping Python 2 would either mean to stay with the older
version completely, or to split the source package into a Python 2
and a Python 3 version. Both is ugly.

* Many sources, especially when they provide Python packages only as a
secondary goal, are not supporting to build Python 2 and Python 3 at
the same time. Aside from non-pythonic build systems (cmake for
example), there are f.e. complications with tools built on top of the
Python package.

* The EOL of Python 2 was announced long ago, but many people didn't
switch just because it is still there and supported (experience from
my surrounding). Keeping Python 2 supported only shifts the problem
for them by another two years

So, while I would not *encourage* people to drop Python 2 support in
their packages, it is perfectly OK if a maintainer does so. If someone
else wants to jump in, he still can upload his own legacy Python 2
source package. And I will (recursively) remove Python 2 support from
the packages I maintain whenever I feel that support becomes difficult,
for any of the reasons stated above (with astropy as temporary
exception, due to the importance of the package).

Best regards

Ole

[Ian Jackson]

Post by Jeremy Bicha

Post by Adrian Bunk
The tip of the iceberg are some recent cases where Python 2 modules
were dropped that still had reverse dependencies in unstable, but
also for those without there will in many cases be users who will
still need these modules in buster.

Once a Python2 library has no reverse depends, I see no reason in
general why the Maintainer can't remove the Python2 library.

People use Debian for things other than QAing Debian packages.

Ian.

[Helmut Grohne]

Post by Adrian Bunk
The tip of the iceberg are some recent cases where Python 2 modules
were dropped that still had reverse dependencies in unstable, but
also for those without there will in many cases be users who will
still need these modules in buster.

We had this discussion on IRC recently and there consensus of a
significant portion of participants was in favour of keeping Python 2
modules. The major theme was leaving the choice to users and use cases
that depend on other software not packaged for Debian.

Actually removing Python 2 modules is a rather easy task in most cases:
You drop the binary package. I'm confident that we'll easily be able to
drop all Python 2 modules after buster with little problems. The problem
is with rdeps. What is much harder is switching applications from Python
2 to Python 3. Also moving applications from Python 2 to Python 3 is
something we can and should do in time for buster as much as possible.
In some cases (e.g. gimp) it can be difficult or impossible, but it is
something that should be worked on now or removing Python 2 after buster
will be painful. Switching applications also means that many
installations will be able to do without a (potentially) poorly
supported Python release. In my opinion, removing Python 2 from the most
common default installations would be useful.

Thus let me propose adding a new lintian tag for non-module packages
that depend on Python 2 and kindly ask them to switch to Python 3. Such
a tag can very well be a warning today. It would hit around 800 binary
packages at present. I'd love to see that number go down to around 100
when buster is released.

Do people agree that such a tag would be useful?

Other than that, I think the discussion is quite similar to the one
about dropping init scripts after stretch. We resolved, yes keep them
unless you know that they don't work at all. Can we simply do the same
for Python 2 modules?

Post by Adrian Bunk
All of the above applies especially in cases where continuing to
provide a Python 2 module does not cause problems or extra work
(in several cases Python 2 modules were dropped in a new Debian
revision of a package without any real reason).

I actually face the issue you are trying to exclude here. You likely
know that I work on making packages cross-buildable and I do not stop at
python extensions. As it happens, cross building extensions tends to
work for Python 3, but not for Python 2 these days. Now I am faced with
a choice:
* Ask for removing Python 2 extensions to make packages cross
buildable.
* Produce patches for the deprecated Python 2 and ask the Python
maintainer to take them while knowing that they cannot be upstreamed.
* Insert a new nopython2 build profile to allow building without Python
2.
* Wait for the problem to solve itself after buster.

I tend to use the last option, but the pile is ever increasing. What do
you suggest here?

So yes, not dropping Python 2 extensions makes my work harder and yes, I
am in favour of keeping them for buster anyway.

Helmut

[Ian Jackson]

Post by Helmut Grohne
I actually face the issue you are trying to exclude here. You likely
know that I work on making packages cross-buildable and I do not stop at
python extensions. As it happens, cross building extensions tends to
work for Python 3, but not for Python 2 these days. Now I am faced with
* Ask for removing Python 2 extensions to make packages cross
buildable.
* Produce patches for the deprecated Python 2 and ask the Python
maintainer to take them while knowing that they cannot be upstreamed.
* Insert a new nopython2 build profile to allow building without Python
2.
* Wait for the problem to solve itself after buster.
I tend to use the last option, but the pile is ever increasing. What do
you suggest here?

I would do (2), assuming the patches are small. The effort of
carrying a patch is much reduced if there is no upstream who are
constantly releasing new versions for us to rebase our patch onto.

But I appreciate that (4) looks attractive.

Post by Helmut Grohne
So yes, not dropping Python 2 extensions makes my work harder and yes, I
am in favour of keeping them for buster anyway.

Thanks for this excellent attitude. I do appreciate you doing this
work to benefit others, even though it is an inconvenience to your own
projects.

Regards,
Ian.

[Adrian Bunk]

Post by Helmut Grohne

Post by Adrian Bunk
The tip of the iceberg are some recent cases where Python 2 modules
were dropped that still had reverse dependencies in unstable, but
also for those without there will in many cases be users who will
still need these modules in buster.

We had this discussion on IRC recently and there consensus of a
significant portion of participants was in favour of keeping Python 2
modules. The major theme was leaving the choice to users and use cases
that depend on other software not packaged for Debian.
You drop the binary package. I'm confident that we'll easily be able to
drop all Python 2 modules after buster with little problems. The problem
is with rdeps. What is much harder is switching applications from Python
2 to Python 3. Also moving applications from Python 2 to Python 3 is
something we can and should do in time for buster as much as possible.
...

For native packages, yes.

For non-native packages it depends on upstream.

In many cases this will be solved automatically by updating to the
latest upstream version after the release of buster, while doing
anything now ourselves would only produce additional work.

Post by Helmut Grohne
Thus let me propose adding a new lintian tag for non-module packages
that depend on Python 2 and kindly ask them to switch to Python 3. Such
a tag can very well be a warning today. It would hit around 800 binary
packages at present. I'd love to see that number go down to around 100
when buster is released.
Do people agree that such a tag would be useful?
...

That would be
https://lintian.debian.org/tags/dependency-on-python-version-marked-for-end-of-life.html

Post by Helmut Grohne
Helmut

cu
Adrian

[Holger Levsen]

package: lintian
severity: wishlist

Post by Chris Lamb
For example, I think Holger is interpreting this particular tag as
"this source package is shipping a Python 2.x" module. This is not
the case.
Rather, Lintian detects that this is the *initial* upload of the
source package and, if so, asks the maintainer just to double-check
that there is any requirement for it.
If there is such a need, the maintainer just needs to add a short,
quick justification in the initial changelog entry and Lintian will
then be silent on the matter.
It is thus not asking maintainers to drop Python 2 support


Then the lintian message should be appended to say this only happens on
the first upload.

Thanks.

Post by Chris Lamb
As there can only be one initial upload by definition, adding an
override here is not only a little silly given that it will trigger
an "unused override" warning on the next upload, it can simply be
avoided in the changelog entry as previously discussed, which
implicitly serves as some documentation too.

Maybe it's also worth pointing this out.

Post by Chris Lamb
(This talk of overrides was why I believe Holger is accidentally
confusing the tag.)

Yes. And because the tag description needs improving. ;)

[Chris Lamb]

Hi Holger,

Post by Holger Levsen

Post by Chris Lamb
For example, I think Holger is interpreting this particular tag as
"this source package is shipping a Python 2.x" module. This is not
the case.

Then the lintian message should be appended to say this only happens on
the first upload.

It does, no? The current text is:

Info: This package appears to be the initial packaging of a new upstream
software package (ie. it contains a single changelog entry).

:)


Best wishes,

[Holger Levsen]

ah, thanks for clarifying. I indeed filed the bug only by judging what
was said and quoted on -***@. Sorry for the noise!

[Thomas Goirand]

Post by Adrian Bunk
Hi,
1. Upstream EOL for Python 2 is 2020
2. Debian will fully (security) support Python 2 in buster
until the EOL of buster (ETA: mid-2022)
Python 2 is obsolete, no doubt about that.
But in many cases a Linux distribution is just a platform for running
own applications, and for various good and bad reasons many of our
users will have to support custom and/or 3rd party Python 2 code on
systems running buster.
We are only making it unnecessarily harder for our users when
Python 2 modules are dropped before buster.
The tip of the iceberg are some recent cases where Python 2 modules
were dropped that still had reverse dependencies in unstable, but
also for those without there will in many cases be users who will
still need these modules in buster.
All of the above applies especially in cases where continuing to
provide a Python 2 module does not cause problems or extra work
(in several cases Python 2 modules were dropped in a new Debian
revision of a package without any real reason).
There are of course cases (e.g. OpenStack) where providing Python 2
modules in buster is not possible with reasonable effort.

OpenStack isn't any different from other Python app/modules. I simply
happened to flip the switch, and made every service run on top of Python
3. And right now, it bites hard, because I have to fix all puppet
modules to run using Python 3 instead of 2. It's far from trivial,
unfortunately.

So, as I wrote, it's not any different. Once you've switched, why
keeping the legacy? Isn't 10 years of Python 3 enough time for a
migration? What's the incentive? Bit rotting legacy code? This code is
already not maintainable, and that's the issue that should be addressed.

Right now, the only reason I'm keeping Python 2 support within a number
of packages in OpenStack, is because one of our team member wrote he's
using Python 2 in his company, and that we never finished the
conversation as to when we can remove that support. So we're doing here
a favor, but I'm not sure that's a favor to Debian. To the contrary,
this means we're keeping the legacy, and prevent clean-ups.

Also, I have noticed that when removing Python 2 legacy packages, a lot
of cruft remains in the archive. This isn't trivial to track and clean.
I'd love to have the opinion of the FTP master team about this. How can
we file sensible bugs about it? How can I track what hasn't been un-crufted?

Last: it is my opinion here that you're sending a very wrong message in
the wrong direction that opposes to the general consensus within the
Python team. Maybe it would have been wiser to discuss with that team
first, before raising the thread in debian-devel.

Anyway, thanks for your tireless bug reports. Their value cannot be
overstated.

Cheers,

Thomas Goirand (zigo)

[Vincent Bernat]

Isn't 10 years of Python 3 enough time for a migration?

Python 3.3, the first release people could reasonably start migrating
to, is from 2012. Before that, it was very difficult to have a codebase
compatible with Python 2 and Python 3. Debian Wheezy didn't have
it. Python 3.4 was released with Debian Jessie (3 years ago). So, people
didn't have 10 years to migrate.

[Thomas Goirand]

Post by Vincent Bernat

Isn't 10 years of Python 3 enough time for a migration?

Python 3.3, the first release people could reasonably start migrating
to, is from 2012. Before that, it was very difficult to have a codebase
compatible with Python 2 and Python 3. Debian Wheezy didn't have
it. Python 3.4 was released with Debian Jessie (3 years ago). So, people
didn't have 10 years to migrate.

I do remember having done some work to have babel working on Python 3.2.
It is my opinion that it was already possible to port to 3.2 when it was
there, and that's one release earlier in Debian (ie: wheezy).

Cheers,

Thomas Goirand (zigo)

[Wouter Verhelst]

Post by Thomas Goirand

Post by Vincent Bernat

Isn't 10 years of Python 3 enough time for a migration?

Python 3.3, the first release people could reasonably start migrating
to, is from 2012. Before that, it was very difficult to have a codebase
compatible with Python 2 and Python 3. Debian Wheezy didn't have
it. Python 3.4 was released with Debian Jessie (3 years ago). So, people
didn't have 10 years to migrate.

I do remember having done some work to have babel working on Python 3.2.
It is my opinion that it was already possible to port to 3.2 when it was
there, and that's one release earlier in Debian (ie: wheezy).

Still, wheezy is from may 2013; that's (soon to be) 5, not 10 years.

[Emilio Pozuelo Monfort]

Post by Thomas Goirand
Also, I have noticed that when removing Python 2 legacy packages, a lot
of cruft remains in the archive. This isn't trivial to track and clean.
I'd love to have the opinion of the FTP master team about this. How can
we file sensible bugs about it? How can I track what hasn't been un-crufted?

If the package that you removed has no reverse dependencies, then it will be
automatically removed (decrufted), and the removal will be mentioned in

https://ftp-master.debian.org/removals.txt

If it has rdeps and can't be auto-removed, then it will appear in the cruft report:

https://ftp-master.debian.org/cruft-report-daily.txt

But please do not remove packages that have rdeps if you don't have a transition
plan. Otherwise you're just going to cause your package to get stuck in unstable
with no chance to migrate to testing, see e.g.:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=894560

By all means start getting your rdeps / python2 apps ported, so that when the
time comes, dropping Python 2 support won't be such a daunting task.

Cheers,
Emilio

[Adrian Bunk]

Post by Thomas Goirand
...
Right now, the only reason I'm keeping Python 2 support within a number
of packages in OpenStack, is because one of our team member wrote he's
using Python 2 in his company, and that we never finished the
conversation as to when we can remove that support. So we're doing here
a favor, but I'm not sure that's a favor to Debian.
...

I thought there was a technical reason why OpenStack packages
had to drop Python 2 support.

The existence of this team member is actually a good sign that there
will be many more users whom you would cause problems without these
Python 2 modules.

In practice "problems" might often imply "you cannot upgrade to buster"
or "you cannot use Debian".

Post by Thomas Goirand
Cheers,
Thomas Goirand (zigo)

cu
Adrian

[Holger Levsen]

package: release-notes
severity: wishlist

Post by Niels Thykier
Reminder: The release-notes does have a section for "deprecations" (Note
as I recall, it was empty for stretch, so it is probably not visible there).

thanks for that reminder.

Post by Niels Thykier
If there is consensus that buster is the last release to support
python2, then we can document it in the release-notes.

i'm not sure we have consensus that buster is the last release to
support python2, but it's definitly time to have a note there saying
that python2 will be EOL upstream by 2020.