[openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible

[Victor Stinner]

Hi,

During the last OpenStack Summit at Paris, we discussed how we can port OpenStack to Python 3, because eventlet was not compatible with Python 3. There are multiple approaches: port eventlet to Python 3, replace eventlet with asyncio, replace eventlet with threads, etc. We decided to not take a decision and instead investigate all options.

I fixed 4 issues with monkey-patching in Python 3 (importlib, os.open(), threading.RLock, threading.Thread). Good news: the just released eventlet 0.17.3 includes these fixes and it is now fully compatible with Python 3! For example, the Oslo Messaging test suite now pass with this eventlet version! Currently, eventlet is disabled in Oslo Messaging on Python 3 (eventlet tests are skipped).

I just sent a patch for requirements and Oslo Messaging to bump to eventlet 0.17.3, but it will have to wait until everyone has master as Liberty.

https://review.openstack.org/#/c/172132/
https://review.openstack.org/#/c/172135/

It becomes possible to port more projects depending on eventlet to Python 3!

Liberty cycle will be a good opportunity to port more OpenStack components to Python 3. Most OpenStack clients and Common Libraries are *already* Python 3 compatible, see the wiki page:

https://wiki.openstack.org/wiki/Python3

--

To replace eventlet, I wrote a spec to replace it with asyncio:

https://review.openstack.org/#/c/153298/

Joshua Harlow wrote a spec to replace eventlet with threads:

https://review.openstack.org/#/c/156711/

But then he wrote a single spec "Replace eventlet + monkey-patching with ??" which covers threads and asyncio:

https://review.openstack.org/#/c/164035/

Victor

[Victor Sergeyev]

Thanks for your work on this! :)

Post by Victor Stinner
Hi,
During the last OpenStack Summit at Paris, we discussed how we can port
OpenStack to Python 3, because eventlet was not compatible with Python 3.
There are multiple approaches: port eventlet to Python 3, replace eventlet
with asyncio, replace eventlet with threads, etc. We decided to not take a
decision and instead investigate all options.
I fixed 4 issues with monkey-patching in Python 3 (importlib, os.open(),
threading.RLock, threading.Thread). Good news: the just released eventlet
0.17.3 includes these fixes and it is now fully compatible with Python 3!
For example, the Oslo Messaging test suite now pass with this eventlet
version! Currently, eventlet is disabled in Oslo Messaging on Python 3
(eventlet tests are skipped).
I just sent a patch for requirements and Oslo Messaging to bump to
eventlet 0.17.3, but it will have to wait until everyone has master as
Liberty.
https://review.openstack.org/#/c/172132/
https://review.openstack.org/#/c/172135/
It becomes possible to port more projects depending on eventlet to Python 3!
Liberty cycle will be a good opportunity to port more OpenStack components
to Python 3. Most OpenStack clients and Common Libraries are *already*
https://wiki.openstack.org/wiki/Python3
--
https://review.openstack.org/#/c/153298/
https://review.openstack.org/#/c/156711/
But then he wrote a single spec "Replace eventlet + monkey-patching with
https://review.openstack.org/#/c/164035/
Victor
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[Joe Gordon]

Post by Victor Stinner
Hi,
During the last OpenStack Summit at Paris, we discussed how we can port
OpenStack to Python 3, because eventlet was not compatible with Python 3.
There are multiple approaches: port eventlet to Python 3, replace eventlet
with asyncio, replace eventlet with threads, etc. We decided to not take a
decision and instead investigate all options.
I fixed 4 issues with monkey-patching in Python 3 (importlib, os.open(),
threading.RLock, threading.Thread). Good news: the just released eventlet
0.17.3 includes these fixes and it is now fully compatible with Python 3!
For example, the Oslo Messaging test suite now pass with this eventlet
version! Currently, eventlet is disabled in Oslo Messaging on Python 3
(eventlet tests are skipped).
I just sent a patch for requirements and Oslo Messaging to bump to
eventlet 0.17.3, but it will have to wait until everyone has master as
Liberty.
https://review.openstack.org/#/c/172132/
https://review.openstack.org/#/c/172135/
It becomes possible to port more projects depending on eventlet to Python 3!

Awesome!

Post by Victor Stinner
Liberty cycle will be a good opportunity to port more OpenStack components
to Python 3. Most OpenStack clients and Common Libraries are *already*
https://wiki.openstack.org/wiki/Python3

https://wiki.openstack.org/wiki/Python3#Dependencies appears to be fairly
out of date. For example hacking works under python34 as does oslo
messaging as per this email etc.

Also what is the status of all the dependencies in
https://github.com/openstack/nova/blob/master/requirements.txt and more
generally
https://github.com/openstack/requirements/blob/master/global-requirements.txt

It would be nice to get a better sense of what the remaining libraries to
port over are before the summit so we can start planning how to do the
python34 migration.

Post by Victor Stinner
--
https://review.openstack.org/#/c/153298/
https://review.openstack.org/#/c/156711/
But then he wrote a single spec "Replace eventlet + monkey-patching with
https://review.openstack.org/#/c/164035/
Victor
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[Victor Stinner]

https://wiki.openstack.org/wiki/Python3#Dependencies appears to be fairly out of date.

You're right. I updated this wiki page. In practice, much more OpenStack clients, Common Libraries and Development Tools are already Python 3 compatible. I added the link to my pull request for Oslo Messaging.

It would be nice to get a better sense of what the remaining libraries to port over are before the summit so we can start planning how to do the python34 migration.

I checked quickly. There are small libraries like pyEClib required by Swift, but the major blocker libraries are: MySQL-Python, suds, Paste. For oslo.db, it's already Python 3 compatible no?


* MySQL-Python

MySQL-Python doesn't look to be active (last commit in january 2014). There are multiple Python 3 pending pull requests: https://github.com/farcepest/MySQLdb1/pulls

Mike Bayer is evaluating PyMySQL which is Python 3 compatible:
https://wiki.openstack.org/wiki/PyMySQL_evaluation

See also https://github.com/farcepest/moist (is it alive? is it Python 3 compatible?)


* suds

There is https://bitbucket.org/jurko/suds : a fork compatible with Python 3. Global requirements contain this comment:

# NOTE(dims): suds is not python 3.x compatible, suds-jurko is a fork that
# works with py3x. oslo.vmware would convert to suds-jurko first then nova
# and cinder would follow. suds should be remove immediately once those
# projects move to suds-jurko for all jobs.


* Paste

I already fixed Python 3 compatibility issues and my changes were merged, but there is no release including my fixes yet :-(

I heard that Paste is completly outdated and should be replaced. Ok, but in practice it's still used and not Python 3 compatible.

Workaround: use the development (git) version of Paste.


For the full list, see the wiki page:
https://wiki.openstack.org/wiki/Python3#Core_OpenStack_projects

Victor

[Jeremy Stanley]

On 2015-04-10 07:01:44 -0400 (-0400), Victor Stinner wrote:
[...]

Post by Victor Stinner
https://wiki.openstack.org/wiki/PyMySQL_evaluation

[...]

Worth noting we've already switched to using PyMySQL in nodepool,
storyboard and some of the subunit2sql tooling. It's been working
out great so far.

[Victor Stinner]

Post by Jeremy Stanley
Worth noting we've already switched to using PyMySQL in nodepool,
storyboard and some of the subunit2sql tooling. It's been working
out great so far.

Great. Did you notice a performance regression? Mike wrote that PyMySQL is much slower than MySQL-Python.

Victor

[Jeremy Stanley]

Post by Victor Stinner
Great. Did you notice a performance regression?

Nope. Worth noting, we implemented it primarily for its lack of
compiled extensions, and to a lesser because it supports Python 3.x.
I suspect if we do later run into any unexpected performance
issues... well, it's pure Python. We have lots of people who can
help.

Post by Victor Stinner
Mike wrote that PyMySQL is much slower than MySQL-Python.

I don't recall him saying that specifically. Also last I recall he
admitted he hadn't actually tested under the sorts of load we would
drive in a production OpenStack service--merely performed some
fairly artificial benchmarks looping known-expensive operations that
may not ultimately reflect places in our codebase where introducing
any sort of slowdown would be noticeable compared to other
operations being performed.

Chances are the Project Infrastructure systems will continue
incrementally switching to PyMySQL mainly because it's easier to
install and works on a broader variety of platforms.

[Mike Bayer]

Post by Jeremy Stanley

Post by Victor Stinner
Great. Did you notice a performance regression?

Nope. Worth noting, we implemented it primarily for its lack of
compiled extensions, and to a lesser because it supports Python 3.x.
I suspect if we do later run into any unexpected performance
issues... well, it's pure Python. We have lots of people who can
help.

Post by Victor Stinner
Mike wrote that PyMySQL is much slower than MySQL-Python.

right so it's all a question of proportionality. First off, some
major performance improvements have been merged into PyMySQL since I
wrote my wiki entry on it. Secondly, one thing I didn't do in that
wiki (the one here https://wiki.openstack.org/wiki/PyMySQL_evaluation)
do a performance test of MySQL-Python and PyMySQL talking to a
non-localhost database; as you can see in the RunSnake profiles I have
at
http://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/
the proportion of network takes up a lot more over the wire, and when
you compare MySQL-Python to PyMySQL in that sense the differences in
Python overhead vs. not become much less a proportion of the overhead
already.

Take both drivers and put them way deep into the very elaborate
operations we see Openstack DB backends doing and it's quite difficult
to observe differences qualitatively; unless you really did speed tests
for many thousands of identical operations on both. It's usually in
the "we tested how long it would take to do 10K or 100K rows /
operations / etc" space that differences like these add up. Which is
why, as is in my blog post, I take the stance that using things like
eventlet / gevent / async for database code in the first place is not
really worth the extra headaches at all, even if they did prove to be
faster than traditional threads, which also I show that they are in fact
not, for CRUD-style database code as we do in Openstack apps.

Post by Jeremy Stanley
I don't recall him saying that specifically. Also last I recall he
admitted he hadn't actually tested under the sorts of load we would
drive in a production OpenStack service--merely performed some
fairly artificial benchmarks looping known-expensive operations that
may not ultimately reflect places in our codebase where introducing
any sort of slowdown would be noticeable compared to other
operations being performed.
Chances are the Project Infrastructure systems will continue
incrementally switching to PyMySQL mainly because it's easier to
install and works on a broader variety of platforms.

[Joe Gordon]

Post by Joe Gordon
https://wiki.openstack.org/wiki/Python3#Dependencies appears to be

fairly out of date.
You're right. I updated this wiki page. In practice, much more OpenStack
clients, Common Libraries and Development Tools are already Python 3
compatible. I added the link to my pull request for Oslo Messaging.

Post by Joe Gordon
It would be nice to get a better sense of what the remaining libraries

to port over are before the summit so we can start planning how to do the
python34 migration.
I checked quickly. There are small libraries like pyEClib required by
Swift, but the major blocker libraries are: MySQL-Python, suds, Paste. For
oslo.db, it's already Python 3 compatible no?
* MySQL-Python
MySQL-Python doesn't look to be active (last commit in january 2014).
https://github.com/farcepest/MySQLdb1/pulls
https://wiki.openstack.org/wiki/PyMySQL_evaluation
See also https://github.com/farcepest/moist (is it alive? is it Python 3 compatible?)
* suds
There is https://bitbucket.org/jurko/suds : a fork compatible with Python
# NOTE(dims): suds is not python 3.x compatible, suds-jurko is a fork that
# works with py3x. oslo.vmware would convert to suds-jurko first then nova
# and cinder would follow. suds should be remove immediately once those
# projects move to suds-jurko for all jobs.
* Paste
I already fixed Python 3 compatibility issues and my changes were merged,
but there is no release including my fixes yet :-(
I heard that Paste is completly outdated and should be replaced. Ok, but
in practice it's still used and not Python 3 compatible.
Workaround: use the development (git) version of Paste.
https://wiki.openstack.org/wiki/Python3#Core_OpenStack_projects

Thanks for updating the wiki page that is a very useful list.

From the looks of things, it seems like nova getting Python3 support in
Liberty is not going to happen. But we can make good progress in
dependencies sorted out. By fixing the dependencies and switching a few out
for better ones.

What are your thoughts on how to tackle sqlalchemy-migrate? It looks like
that is a blocker for several projects. And something I think we have
wanted to move off of for some time now.

Victor
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[Clint Byrum]

Post by Joe Gordon

Post by Joe Gordon
https://wiki.openstack.org/wiki/Python3#Dependencies appears to be

fairly out of date.
You're right. I updated this wiki page. In practice, much more OpenStack
clients, Common Libraries and Development Tools are already Python 3
compatible. I added the link to my pull request for Oslo Messaging.

Post by Joe Gordon
It would be nice to get a better sense of what the remaining libraries

to port over are before the summit so we can start planning how to do the
python34 migration.
I checked quickly. There are small libraries like pyEClib required by
Swift, but the major blocker libraries are: MySQL-Python, suds, Paste. For
oslo.db, it's already Python 3 compatible no?
* MySQL-Python
MySQL-Python doesn't look to be active (last commit in january 2014).
https://github.com/farcepest/MySQLdb1/pulls
https://wiki.openstack.org/wiki/PyMySQL_evaluation
See also https://github.com/farcepest/moist (is it alive? is it Python 3 compatible?)
* suds
There is https://bitbucket.org/jurko/suds : a fork compatible with Python
# NOTE(dims): suds is not python 3.x compatible, suds-jurko is a fork that
# works with py3x. oslo.vmware would convert to suds-jurko first then nova
# and cinder would follow. suds should be remove immediately once those
# projects move to suds-jurko for all jobs.
* Paste
I already fixed Python 3 compatibility issues and my changes were merged,
but there is no release including my fixes yet :-(
I heard that Paste is completly outdated and should be replaced. Ok, but
in practice it's still used and not Python 3 compatible.
Workaround: use the development (git) version of Paste.
https://wiki.openstack.org/wiki/Python3#Core_OpenStack_projects

Thanks for updating the wiki page that is a very useful list.
From the looks of things, it seems like nova getting Python3 support in
Liberty is not going to happen. But we can make good progress in
dependencies sorted out. By fixing the dependencies and switching a few out
for better ones.
What are your thoughts on how to tackle sqlalchemy-migrate? It looks like
that is a blocker for several projects. And something I think we have
wanted to move off of for some time now.

IMHO it is quite a bit easier to port something to python 3 than to
move off of it entirely. I'd say it's worth it for forward progress to
try and port sqlalchemy-migrate, even if that means the effort becomes
a sunk cost in a year.

[Ian Cordasco]

Post by Clint Byrum

Post by Victor Stinner

Post by Joe Gordon
https://wiki.openstack.org/wiki/Python3#Dependencies appears to be

fairly out of date.
You're right. I updated this wiki page. In practice, much more

OpenStack

clients, Common Libraries and Development Tools are already Python 3
compatible. I added the link to my pull request for Oslo Messaging.

Post by Joe Gordon
It would be nice to get a better sense of what the remaining

libraries

to port over are before the summit so we can start planning how to do

the

python34 migration.
I checked quickly. There are small libraries like pyEClib required by
Swift, but the major blocker libraries are: MySQL-Python, suds,

Paste. For

oslo.db, it's already Python 3 compatible no?
* MySQL-Python
MySQL-Python doesn't look to be active (last commit in january 2014).
https://github.com/farcepest/MySQLdb1/pulls
https://wiki.openstack.org/wiki/PyMySQL_evaluation
See also https://github.com/farcepest/moist (is it alive? is it

Python 3

compatible?)
* suds
There is https://bitbucket.org/jurko/suds : a fork compatible with

Python

# NOTE(dims): suds is not python 3.x compatible, suds-jurko is a fork

that

# works with py3x. oslo.vmware would convert to suds-jurko first then

nova

# and cinder would follow. suds should be remove immediately once

those

# projects move to suds-jurko for all jobs.
* Paste
I already fixed Python 3 compatibility issues and my changes were

merged,

but there is no release including my fixes yet :-(
I heard that Paste is completly outdated and should be replaced. Ok,

but

in practice it's still used and not Python 3 compatible.
Workaround: use the development (git) version of Paste.
https://wiki.openstack.org/wiki/Python3#Core_OpenStack_projects

Thanks for updating the wiki page that is a very useful list.
From the looks of things, it seems like nova getting Python3 support in
Liberty is not going to happen. But we can make good progress in
dependencies sorted out. By fixing the dependencies and switching a few out
for better ones.
What are your thoughts on how to tackle sqlalchemy-migrate? It looks like
that is a blocker for several projects. And something I think we have
wanted to move off of for some time now.

IMHO it is quite a bit easier to port something to python 3 than to
move off of it entirely. I'd say it's worth it for forward progress to
try and port sqlalchemy-migrate, even if that means the effort becomes
a sunk cost in a year.

Also, isn’t sqlalchemy-migrate something we currently maintain (or a group
of OpenStack developers do it for OpenStack. Can’t we work with them to
add support for Python 3?

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-***@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listin

[Joe Gordon]

Post by Clint Byrum

Post by Victor Stinner

Post by Joe Gordon
https://wiki.openstack.org/wiki/Python3#Dependencies appears to be

fairly out of date.
You're right. I updated this wiki page. In practice, much more

OpenStack

clients, Common Libraries and Development Tools are already Python 3
compatible. I added the link to my pull request for Oslo Messaging.

Post by Joe Gordon
It would be nice to get a better sense of what the remaining

libraries

to port over are before the summit so we can start planning how to do

the

python34 migration.
I checked quickly. There are small libraries like pyEClib required by
Swift, but the major blocker libraries are: MySQL-Python, suds,

Paste. For

oslo.db, it's already Python 3 compatible no?
* MySQL-Python
MySQL-Python doesn't look to be active (last commit in january 2014).
https://github.com/farcepest/MySQLdb1/pulls
https://wiki.openstack.org/wiki/PyMySQL_evaluation
See also https://github.com/farcepest/moist (is it alive? is it

Python 3

compatible?)
* suds
There is https://bitbucket.org/jurko/suds : a fork compatible with

Python

# NOTE(dims): suds is not python 3.x compatible, suds-jurko is a fork

that

# works with py3x. oslo.vmware would convert to suds-jurko first then

nova

# and cinder would follow. suds should be remove immediately once

those

# projects move to suds-jurko for all jobs.
* Paste
I already fixed Python 3 compatibility issues and my changes were

merged,

but there is no release including my fixes yet :-(
I heard that Paste is completly outdated and should be replaced. Ok,

but

in practice it's still used and not Python 3 compatible.
Workaround: use the development (git) version of Paste.
https://wiki.openstack.org/wiki/Python3#Core_OpenStack_projects

Thanks for updating the wiki page that is a very useful list.
From the looks of things, it seems like nova getting Python3 support in
Liberty is not going to happen. But we can make good progress in
dependencies sorted out. By fixing the dependencies and switching a few out
for better ones.
What are your thoughts on how to tackle sqlalchemy-migrate? It looks like
that is a blocker for several projects. And something I think we have
wanted to move off of for some time now.

IMHO it is quite a bit easier to port something to python 3 than to
move off of it entirely. I'd say it's worth it for forward progress to
try and port sqlalchemy-migrate, even if that means the effort becomes
a sunk cost in a year.

Also, isn’t sqlalchemy-migrate something we currently maintain (or a group
of OpenStack developers do it for OpenStack. Can’t we work with them to
add support for Python 3?

yup https://github.com/stackforge/sqlalchemy-migrate. The better question
is: 'is it worth adding support in versus moving over to alembic, since we
want to do that anyway?' I don't personally have an answer for that.

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[Pádraig Brady]

Post by Ian Cordasco
Also, isn’t sqlalchemy-migrate something we currently maintain (or a group
of OpenStack developers do it for OpenStack. Can’t we work with them to
add support for Python 3?

There seems to have been some work on that already:
https://github.com/stackforge/sqlalchemy-migrate/commit/a03b141a95

Pádraig.

[Victor Stinner]

Hi,

sqlachemy-migrate already works on Python3. I sent a patch to add missing Python 3 classifiers, so the "caniusepython3" automated tool will stop reporting false alarm on this library.
https://review.openstack.org/174738

Or is someone aware of issues with Python 3?

Again, check the wiki page for up to date information:
https://wiki.openstack.org/wiki/Python3#Dependencies

Post by Pádraig Brady
https://github.com/stackforge/sqlalchemy-migrate/commit/a03b141a95

Yeah, Cyril made great work! I ported a lot of OpenStack libraries.

Victor

[Victor Stinner]

Post by Joe Gordon

Post by Victor Stinner
https://wiki.openstack.org/wiki/Python3#Core_OpenStack_projects

Thanks for updating the wiki page that is a very useful list.
From the looks of things, it seems like nova getting Python3 support in Liberty is not going to happen.

Why? I plan to work on porting nova to Python 3. I proposed a nova session on Python 3 at the next OpenStack Summit at Vancouver. I plan to write a spec too.

I'm not aware of any real blocker for nova.

Post by Joe Gordon
What are your thoughts on how to tackle sqlalchemy-migrate? It looks like that is a blocker for several projects. And something I think we have wanted to move off of for some time now.

I just checked sqlachemy-migrate. The README and the documentation are completly outdated, but the project is very active: latest commit one month ago and latest release (0.9.6) one month ago. There are py33 and py34 environments and tests pass on Python 3.3 and Python 3.4! I didn't check yet, but I guess that sqlachemy-migrate 0.9.6 already works on Python 3. Python 3 classifiers are just missing in setup.cfg.

I sent patches to update the doc, to add Python 3 classifiers and to upgrade requirements. The project moved to stackforge, reviews are at review.openstack.org:

https://review.openstack.org/#/q/status:open+project:stackforge/sqlalchemy-migrate,n,z

The wiki page said that scripttest and ibm-db-sa were not Python 3 compatible. It's no more true: scripttest is compatible Python 3, and there is ibm-db-sa-py3 which is Python 3 compatible.

I updated the wiki page for sqlachemy-migrate.

Victor

[Joe Gordon]

Post by Joe Gordon

Post by Joe Gordon

Post by Victor Stinner
https://wiki.openstack.org/wiki/Python3#Core_OpenStack_projects

Thanks for updating the wiki page that is a very useful list.
From the looks of things, it seems like nova getting Python3 support in

Liberty is not going to happen.

I based this on the wiki, but maybe I am wrong.

remaining libraries for nova:
oslo.db -- looks like it is almost there
oslo.messaging -- same
paste -- almost there
sqlalchemy-migrate -- almost there
suds -- with the suds fork shouldn't be too hard
websockify -- unknown
libvirt-python -- unknown
mysql-python -- alternitive looks viable.

Based on there being two unknowns, and a lot of dependencies that are just
almost there, and a few we may want to migrate off of, I was assuming
addressing those issues would make it hard for us to make nova python3
compatible for Liberty.

Post by Joe Gordon
Why? I plan to work on porting nova to Python 3. I proposed a nova session
on Python 3 at the next OpenStack Summit at Vancouver. I plan to write a
spec too.
I'm not aware of any real blocker for nova.

Post by Joe Gordon
What are your thoughts on how to tackle sqlalchemy-migrate? It looks

like that is a blocker for several projects. And something I think we have
wanted to move off of for some time now.
I just checked sqlachemy-migrate. The README and the documentation are
completly outdated, but the project is very active: latest commit one month
ago and latest release (0.9.6) one month ago. There are py33 and py34
environments and tests pass on Python 3.3 and Python 3.4! I didn't check
yet, but I guess that sqlachemy-migrate 0.9.6 already works on Python 3.
Python 3 classifiers are just missing in setup.cfg.
I sent patches to update the doc, to add Python 3 classifiers and to
upgrade requirements. The project moved to stackforge, reviews are at
https://review.openstack.org/#/q/status:open+project:stackforge/sqlalchemy-migrate,n,z
The wiki page said that scripttest and ibm-db-sa were not Python 3
compatible. It's no more true: scripttest is compatible Python 3, and there
is ibm-db-sa-py3 which is Python 3 compatible.
I updated the wiki page for sqlachemy-migrate.
Victor
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[Victor Stinner]

Hi,

It's moving fast. I'm currently working on porting remaining libraries to prepare my spec for nova.

Post by Joe Gordon
oslo.db -- looks like it is almost there

I don't know the status of oslo.db support of Python 3. I guess that it already works on Python 3, it's just a matter of running tests with MySQL (MySQL-Python blocks again here).

Post by Joe Gordon
oslo.messaging -- same

Two changes of my Python 3 changes were already merged last week. Three remaining Python 3 changes are almost there, they are mostly blocked by the freeze on requirements, but changes are already approved. The blocker is the bump of eventlet to 0.17.3:
https://review.openstack.org/#/c/172132/

Post by Joe Gordon
paste -- almost there

I fixed that last night (!) with the release of Paste 2.0. It's the first Paste release since 2010. Paste 2.0 has an experimental support of Python 3. It should be enough for nova, according to my quick tests in my local nova repository where I fixed most obvious Python 3 issues. Ian Bicking allowed me to publish new versions of Paste.

Post by Joe Gordon
sqlalchemy-migrate -- almost there

It already supports Python 3, so it doesn't block.

Post by Joe Gordon
suds -- with the suds fork shouldn't be too hard

You should just switch to the fork. I contacted the author of suds-jurko: he plans to rename its component from "suds-jurko" to "suds". So his fork would be simple drop-in which would not require any change in OpenStack except of requirements.

Post by Joe Gordon
websockify -- unknown

It announces Python 3.3 and 3.4 support and has a py34 env in tox.ini. I didn't check yet if it supports Python 3, but since the project is active (last commit 12 hours ago), it's shouldn't be hard to fix them quickly.

Post by Joe Gordon
libvirt-python -- unknown

Oh, I missed this dependency. It's my next target :-)

Post by Joe Gordon
mysql-python -- alternitive looks viable.

Yes, it's possible to replace it with PyMySQL. Does anyone know the status of this switch?

Post by Joe Gordon
Based on there being two unknowns, and a lot of dependencies that are just almost there, and a few we may want to migrate off of, I was assuming addressing those issues would make it hard for us to make nova python3 compatible for Liberty.

My plan for Liberty is not to support fully Python 3 in nova, but only start the port (well, continue to be exact). The idea is to fix syntax errors and obvious issues like dict.iteritems() => six.iteritems(dict), then more complex changes. Maybe it's possible to finish the port in a cycle, but it really depends on the time taken to merge patches.

I started to port nova in my local repository. Some unit tests already pass.

nova already uses six in many places, so it's not like we really start from scratch, the port is already an ingoing process.

Victor

[Victor Sergeyev]

Hi, All,

My 2c are:

- yes, oslo.db supports python 3 (unittests passes, at least :) )
- MySQL-python still default MySQL DB driver in OpenStack, but at the
moment the only DB driver for MySQL in python3 environment is PyMySQL, so I
think, it's ok to use it with python 3.

Post by Victor Stinner
Hi,
It's moving fast. I'm currently working on porting remaining libraries to
prepare my spec for nova.

Post by Joe Gordon
oslo.db -- looks like it is almost there

I don't know the status of oslo.db support of Python 3. I guess that it
already works on Python 3, it's just a matter of running tests with MySQL
(MySQL-Python blocks again here).

Post by Joe Gordon
oslo.messaging -- same

Two changes of my Python 3 changes were already merged last week. Three
remaining Python 3 changes are almost there, they are mostly blocked by the
freeze on requirements, but changes are already approved. The blocker is
https://review.openstack.org/#/c/172132/

Post by Joe Gordon
paste -- almost there

I fixed that last night (!) with the release of Paste 2.0. It's the first
Paste release since 2010. Paste 2.0 has an experimental support of Python
3. It should be enough for nova, according to my quick tests in my local
nova repository where I fixed most obvious Python 3 issues. Ian Bicking
allowed me to publish new versions of Paste.

Post by Joe Gordon
sqlalchemy-migrate -- almost there

It already supports Python 3, so it doesn't block.

Post by Joe Gordon
suds -- with the suds fork shouldn't be too hard

he plans to rename its component from "suds-jurko" to "suds". So his fork
would be simple drop-in which would not require any change in OpenStack
except of requirements.

Post by Joe Gordon
websockify -- unknown

It announces Python 3.3 and 3.4 support and has a py34 env in tox.ini. I
didn't check yet if it supports Python 3, but since the project is active
(last commit 12 hours ago), it's shouldn't be hard to fix them quickly.

Post by Joe Gordon
libvirt-python -- unknown

Oh, I missed this dependency. It's my next target :-)

Post by Joe Gordon
mysql-python -- alternitive looks viable.

Yes, it's possible to replace it with PyMySQL. Does anyone know the status of this switch?

Post by Joe Gordon
Based on there being two unknowns, and a lot of dependencies that are

just almost there, and a few we may want to migrate off of, I was assuming
addressing those issues would make it hard for us to make nova python3
compatible for Liberty.
My plan for Liberty is not to support fully Python 3 in nova, but only
start the port (well, continue to be exact). The idea is to fix syntax
errors and obvious issues like dict.iteritems() => six.iteritems(dict),
then more complex changes. Maybe it's possible to finish the port in a
cycle, but it really depends on the time taken to merge patches.
I started to port nova in my local repository. Some unit tests already pass.
nova already uses six in many places, so it's not like we really start
from scratch, the port is already an ingoing process.
Victor
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[Mike Bayer]

Post by Victor Sergeyev
Hi, All,
- yes, oslo.db supports python 3 (unittests passes, at least :) )
- MySQL-python still default MySQL DB driver in OpenStack, but at the
moment the only DB driver for MySQL in python3 environment is PyMySQL,
so I think, it's ok to use it with python 3.

the same maintainer of PyMySQL has also ported MySQL-Python to Python 3,
and this driver works pretty well also:
https://github.com/PyMySQL/mysqlclient-python

[Joe Gordon]

Post by Victor Stinner
Hi,
It's moving fast. I'm currently working on porting remaining libraries to
prepare my spec for nova.

Great, I did't realize how close all the dependencies were.

Post by Victor Stinner

Post by Joe Gordon
oslo.db -- looks like it is almost there

I don't know the status of oslo.db support of Python 3. I guess that it
already works on Python 3, it's just a matter of running tests with MySQL
(MySQL-Python blocks again here).

Post by Joe Gordon
oslo.messaging -- same

Two changes of my Python 3 changes were already merged last week. Three
remaining Python 3 changes are almost there, they are mostly blocked by the
freeze on requirements, but changes are already approved. The blocker is
https://review.openstack.org/#/c/172132/

Post by Joe Gordon
paste -- almost there

I fixed that last night (!) with the release of Paste 2.0. It's the first
Paste release since 2010. Paste 2.0 has an experimental support of Python
3. It should be enough for nova, according to my quick tests in my local
nova repository where I fixed most obvious Python 3 issues. Ian Bicking
allowed me to publish new versions of Paste.

Post by Joe Gordon
sqlalchemy-migrate -- almost there

It already supports Python 3, so it doesn't block.

Post by Joe Gordon
suds -- with the suds fork shouldn't be too hard

he plans to rename its component from "suds-jurko" to "suds". So his fork
would be simple drop-in which would not require any change in OpenStack
except of requirements.

Post by Joe Gordon
websockify -- unknown

It announces Python 3.3 and 3.4 support and has a py34 env in tox.ini. I
didn't check yet if it supports Python 3, but since the project is active
(last commit 12 hours ago), it's shouldn't be hard to fix them quickly.

Post by Joe Gordon
libvirt-python -- unknown

Oh, I missed this dependency. It's my next target :-)

Post by Joe Gordon
mysql-python -- alternitive looks viable.

Yes, it's possible to replace it with PyMySQL. Does anyone know the status of this switch?

Post by Joe Gordon
Based on there being two unknowns, and a lot of dependencies that are

just almost there, and a few we may want to migrate off of, I was assuming
addressing those issues would make it hard for us to make nova python3
compatible for Liberty.
My plan for Liberty is not to support fully Python 3 in nova, but only
start the port (well, continue to be exact). The idea is to fix syntax
errors and obvious issues like dict.iteritems() => six.iteritems(dict),
then more complex changes. Maybe it's possible to finish the port in a
cycle, but it really depends on the time taken to merge patches.

How invasive would the port to python3 be? Would it be easier to have a
python3 migration sprint and rip the band aid off instead of dragging it
out and having partial python3 support for a whole cycle?

In general what is the least painful way to add python3 support for a very
large python project?

Post by Victor Stinner
I started to port nova in my local repository. Some unit tests already pass.
nova already uses six in many places, so it's not like we really start
from scratch, the port is already an ingoing process.
Victor
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[Victor Stinner]

Hi,

Post by Joe Gordon
How invasive would the port to python3 be?

I squashed all my commits into a single commit of my draft port and I pushed it at:
https://github.com/haypo/nova/commit/bad54bc2b278c7c7cb7fa6cc73d03c70138bd89d

As announced, changes are boring, just obvious Python2/Python3 issues:

- strip L from long integer literals: 123L => 123
- replace dict.iteritems() with six.iteritems(dict)
- replace list.sort(cmp_func) with list.sort(key=key_func)
- replace "raise exc_info[0], exc_info[1], exc_info[2]" with six.reraise(*exc_info)
- moved functions / modules

* get http.client, urllib.request and other stdlib modules from six.moves since they moved between Python 2 and Python 3
* get itertools.izip/zip_longest from six.moves
* replace unichr() with six.unichr()

- replace filter(func, data) with [item for item in data if func(item)]
- replace unicode() with six.text_type
- replace (int, long) with six.integer_types
- replace file() with open()
- replace StringIO() with six.StringIO()

These changes are not enough to port nova to Python 3. But they are required to be able to find next Python 3 bugs.

Reminder: Python 2 compatibility is kept! There is not reason right now to drop Python 2 support, it would be a pain to upgrade!

Post by Joe Gordon
Would it be easier to have a python3 migration sprint and rip the band aid off instead of dragging it out and having partial python3 support for a whole cycle?

Do you mean a physical meeting? Or focus on Python 3 during a short period (review/merge all Python 3 patches)?

A single week may not be enough to fix all Python 3 issues. I prefer to submit changes smoothly during the Liberty cycle.

Post by Joe Gordon
In general what is the least painful way to add python3 support for a very large python project?

Send patches and make incremental changes, as any other change made in OpenStack.

Victor

[Angus Lees]

Post by Victor Stinner
- strip L from long integer literals: 123L => 123
- replace dict.iteritems() with six.iteritems(dict)
- replace list.sort(cmp_func) with list.sort(key=key_func)
- replace "raise exc_info[0], exc_info[1], exc_info[2]" with
six.reraise(*exc_info)
- moved functions / modules
* get http.client, urllib.request and other stdlib modules from
six.moves since they moved between Python 2 and Python 3
* get itertools.izip/zip_longest from six.moves
* replace unichr() with six.unichr()
- replace filter(func, data) with [item for item in data if func(item)]
- replace unicode() with six.text_type
- replace (int, long) with six.integer_types
- replace file() with open()
- replace StringIO() with six.StringIO()
These changes are not enough to port nova to Python 3. But they are
required to be able to find next Python 3 bugs.

Is there already a static analysis tool that helps find these things?
(Would a pylint check for the above be useful? Some of them would be hard
to find reliably, but a bunch of the above would be trivial)

- Gus

[Victor Stinner]

Hi,

Post by Victor Stinner
These changes are not enough to port nova to Python 3. But they are required to be able to find next Python 3 bugs.

Is there already a static analysis tool that helps find these things? (Would a pylint check for the above be useful? Some of them would be hard to find reliably, but a bunch of the above would be trivial)

I read that hacking has some checks. It's quite easy to run 2to3 to see modified lines.

Personally, I prefer to just run tests and fix issues one by one. It's faster to fix failing tests without having the modify the whole project at one.

Sometimes, I'm also using regular expressions (in vim or grep). For example, I used "[0-9]+L" to find 123L. You can also use regex to modify code inplace.

2to3 tool drops Python 2 compatibility, so it's not great. But it helps to find code that needs to be modified.

I now prefer 2to6 which uses six and so keeps Python 2 compatibility. I modified it a little bit to reduce changes:
https://github.com/haypo/2to6/

Victor

[Chmouel Boudjnah]

Post by Victor Stinner

Is there already a static analysis tool that helps find these things? (Would a
pylint check for the above be useful? Some of them would be hard to find
reliably, but a bunch of the above would be trivial)

I read that hacking has some checks. It's quite easy to run 2to3 to see modified lines.

In Swift, Alex has installed this tox.ini target to detect those :

https://github.com/openstack/swift/blob/master/tox.ini#L38

may be a good idea to generalize.

Cheers,
Chmouel

[Joe Gordon]

Post by Victor Stinner
Hi,

Post by Joe Gordon
How invasive would the port to python3 be?

https://github.com/haypo/nova/commit/bad54bc2b278c7c7cb7fa6cc73d03c70138bd89d

I like how the sha1 starts with 'bad'

Overall that is a pretty small patch.

Post by Victor Stinner
- strip L from long integer literals: 123L => 123
- replace dict.iteritems() with six.iteritems(dict)
- replace list.sort(cmp_func) with list.sort(key=key_func)
- replace "raise exc_info[0], exc_info[1], exc_info[2]" with
six.reraise(*exc_info)
- moved functions / modules
* get http.client, urllib.request and other stdlib modules from
six.moves since they moved between Python 2 and Python 3
* get itertools.izip/zip_longest from six.moves
* replace unichr() with six.unichr()
- replace filter(func, data) with [item for item in data if func(item)]
- replace unicode() with six.text_type
- replace (int, long) with six.integer_types
- replace file() with open()
- replace StringIO() with six.StringIO()
These changes are not enough to port nova to Python 3. But they are
required to be able to find next Python 3 bugs.
Reminder: Python 2 compatibility is kept! There is not reason right now to
drop Python 2 support, it would be a pain to upgrade!

Post by Joe Gordon
Would it be easier to have a python3 migration sprint and rip the band

aid off instead of dragging it out and having partial python3 support for a
whole cycle?
Do you mean a physical meeting? Or focus on Python 3 during a short period
(review/merge all Python 3 patches)?

Focus on Python 3 during a short period.

Post by Victor Stinner
A single week may not be enough to fix all Python 3 issues. I prefer to
submit changes smoothly during the Liberty cycle.

My general concern, is having to manually review new code for python3
compliance.

If this will take more then a week or two to make a big project python3
compat (during a virtual sprint), it would be good to have some tooling in
place to make sure we don't add a lot more burden on reviewers to make sure
new patches are python3 compatible by hand.

Post by Victor Stinner
In general what is the least painful way to add python3 support for a
very large python project?
Send patches and make incremental changes, as any other change made in OpenStack.
Victor
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[Victor Stinner]

Hi,

I wrote my spec to Port Nova to Python 3:
https://review.openstack.org/#/c/176868/

Post by Joe Gordon

Post by Victor Stinner
https://github.com/haypo/nova/commit/bad54bc2b278c7c7cb7fa6cc73d03c70138bd89d

I like how the sha1 starts with 'bad'

Ah ah, I didn't notice that. I would prefer "python" prefix, but it's not possible.

Post by Joe Gordon
Overall that is a pretty small patch.

Cool.

Post by Joe Gordon
My general concern, is having to manually review new code for python3 compliance.

Do you prefer a single very large patch (as the one I posted above) or multiple small patches fixing similar issues?

Post by Joe Gordon
If this will take more then a week or two to make a big project python3 compat (during a virtual sprint), it would be good to have some tooling in place to make sure we don't add a lot more burden on reviewers to make sure new patches are python3 compatible by hand.

I tried to write a detailed plan in my spec. Until "tox -e py34" pass, I would prefer to not touch nova gates nor any add Python 3 checks.

Victor

[Joe Gordon]

Post by Victor Stinner
Hi,
https://review.openstack.org/#/c/176868/

Post by Joe Gordon

Post by Victor Stinner
I squashed all my commits into a single commit of my draft port and I

https://github.com/haypo/nova/commit/bad54bc2b278c7c7cb7fa6cc73d03c70138bd89d

Post by Joe Gordon
I like how the sha1 starts with 'bad'

Ah ah, I didn't notice that. I would prefer "python" prefix, but it's not possible.

Post by Joe Gordon
Overall that is a pretty small patch.

Cool.

Post by Joe Gordon
My general concern, is having to manually review new code for python3

compliance.
Do you prefer a single very large patch (as the one I posted above) or
multiple small patches fixing similar issues?

Post by Joe Gordon
If this will take more then a week or two to make a big project python3

compat (during a virtual sprint), it would be good to have some tooling in
place to make sure we don't add a lot more burden on reviewers to make sure
new patches are python3 compatible by hand.
I tried to write a detailed plan in my spec. Until "tox -e py34" pass, I
would prefer to not touch nova gates nor any add Python 3 checks.

Great, I'll follow up on that spec.

Post by Victor Stinner
Victor
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[Kashyap Chamarthy]

Post by Joe Gordon

Post by Joe Gordon

Post by Joe Gordon

Post by Victor Stinner
https://wiki.openstack.org/wiki/Python3#Core_OpenStack_projects

Thanks for updating the wiki page that is a very useful list.
From the looks of things, it seems like nova getting Python3 support in

Liberty is not going to happen.

I based this on the wiki, but maybe I am wrong.
oslo.db -- looks like it is almost there
oslo.messaging -- same
paste -- almost there
sqlalchemy-migrate -- almost there
suds -- with the suds fork shouldn't be too hard
websockify -- unknown
libvirt-python -- unknown

I notice some work from Dan Berrange in late 2013 about this:

https://www.redhat.com/archives/libvir-list/2013-December/msg00171.html
-- [PATCH python 00/15] Initial work porting to python3

And, I see several commits in libvirt-python git repo in related to
Python 3.

Maybe Dan can comment when he notices this.

Post by Joe Gordon
mysql-python -- alternitive looks viable.
Based on there being two unknowns, and a lot of dependencies that are just
almost there, and a few we may want to migrate off of, I was assuming
addressing those issues would make it hard for us to make nova python3
compatible for Liberty.

[. . .]

[Daniel P. Berrange]

Post by Kashyap Chamarthy

Post by Joe Gordon

Post by Joe Gordon

Post by Joe Gordon

Post by Victor Stinner
https://wiki.openstack.org/wiki/Python3#Core_OpenStack_projects

Thanks for updating the wiki page that is a very useful list.
From the looks of things, it seems like nova getting Python3 support in

Liberty is not going to happen.

I based this on the wiki, but maybe I am wrong.
oslo.db -- looks like it is almost there
oslo.messaging -- same
paste -- almost there
sqlalchemy-migrate -- almost there
suds -- with the suds fork shouldn't be too hard
websockify -- unknown
libvirt-python -- unknown

https://www.redhat.com/archives/libvir-list/2013-December/msg00171.html
-- [PATCH python 00/15] Initial work porting to python3
And, I see several commits in libvirt-python git repo in related to
Python 3.
Maybe Dan can comment when he notices this.

Libvirt fully supports Python 3 ever since we split libvirt-python
off from main libvirt and hosted it on pypi.

Regards,
Daniel

[Thierry Carrez]

Post by Victor Stinner
I fixed 4 issues with monkey-patching in Python 3 (importlib, os.open(), threading.RLock, threading.Thread). Good news: the just released eventlet 0.17.3 includes these fixes and it is now fully compatible with Python 3! For example, the Oslo Messaging test suite now pass with this eventlet version! Currently, eventlet is disabled in Oslo Messaging on Python 3 (eventlet tests are skipped).

Great news ! That makes the "port to Python 3" question independent of
the "Moving off eventlet" question, which should facilitate immediate
progress on the former.

On the latter, do you plan to file a Concurrency models cross-project
session ? That sounds like a good topic to discuss face to face...

See
http://lists.openstack.org/pipermail/openstack-dev/2015-April/061070.html for
details on how to file there.

[Joshua Harlow]

Post by Thierry Carrez

Post by Victor Stinner
I fixed 4 issues with monkey-patching in Python 3 (importlib, os.open(), threading.RLock, threading.Thread). Good news: the just released eventlet 0.17.3 includes these fixes and it is now fully compatible with Python 3! For example, the Oslo Messaging test suite now pass with this eventlet version! Currently, eventlet is disabled in Oslo Messaging on Python 3 (eventlet tests are skipped).

Great news ! That makes the "port to Python 3" question independent of
the "Moving off eventlet" question, which should facilitate immediate
progress on the former.
On the latter, do you plan to file a Concurrency models cross-project
session ? That sounds like a good topic to discuss face to face...

That sounds like a neat idea; although very high-level and such (so that
means it could just turn into a useless session IMHO), but might be
fun/interesting anyway :-P

Post by Thierry Carrez
See
http://lists.openstack.org/pipermail/openstack-dev/2015-April/061070.html for
details on how to file there.

[Thomas Goirand]

Post by Thierry Carrez

Post by Victor Stinner
I fixed 4 issues with monkey-patching in Python 3 (importlib, os.open(), threading.RLock, threading.Thread). Good news: the just released eventlet 0.17.3 includes these fixes and it is now fully compatible with Python 3! For example, the Oslo Messaging test suite now pass with this eventlet version! Currently, eventlet is disabled in Oslo Messaging on Python 3 (eventlet tests are skipped).

Great news ! That makes the "port to Python 3" question independent of
the "Moving off eventlet" question, which should facilitate immediate
progress on the former.
On the latter, do you plan to file a Concurrency models cross-project
session ? That sounds like a good topic to discuss face to face...
See
http://lists.openstack.org/pipermail/openstack-dev/2015-April/061070.html for
details on how to file there.

Also, on the Python 3 topic, there's still a big issue with memcached
(aka: python-memcache). It's blocking me from adding Python3 support to
keystoneclient, and as a consequence, to almost all of OpenStack.

BTW, the Eventlet module for 0.17.3 is available from here:
http://kilo-jessie.pkgs.mirantis.com/debian/pool/jessie-kilo-backports-nochange/main/p/python-eventlet/

and I will upload this to Experimental as soon as Jessie is released
(that's in 3 days now...).

Cheers,

Thomas Goirand (zigo)

[Victor Stinner]

Post by Thomas Goirand
Also, on the Python 3 topic, there's still a big issue with memcached
(aka: python-memcache).

Oh, thanks Thomas for the reminder. I just sent a pull request to port python-memcached to Python 3:

https://github.com/linsomniac/python-memcached/pull/67

I don't understand. I saw a lot of "Port to Python 3" fixes merged in 2014, and these changes are now part of the release 1.54. Problem: running python-memcached tests with tox fail on obvious Python 3 errors. Anyway, all tests now pass with my pull request. The good news is that python-memcached looks to be actively developed.


Julien Danjou ported pymemcache to Python 3, another memcached client. He suggests to use this one instead of python-memcached client.

Post by Thomas Goirand
It's blocking me from adding Python3 support to
keystoneclient, and as a consequence, to almost all of OpenStack.

python-keystoneclient announces a full Python 3 support with a voting Python 3 gate. I just checked locally, "tox -e py34" pass.

The problem is that python-memcached miss in test dependencies and so middleware tests using memcache are never run!

Victor

[Clint Byrum]

Post by Victor Stinner

Post by Thomas Goirand
Also, on the Python 3 topic, there's still a big issue with memcached
(aka: python-memcache).

https://github.com/linsomniac/python-memcached/pull/67
I don't understand. I saw a lot of "Port to Python 3" fixes merged in 2014, and these changes are now part of the release 1.54. Problem: running python-memcached tests with tox fail on obvious Python 3 errors. Anyway, all tests now pass with my pull request. The good news is that python-memcached looks to be actively developed.
Julien Danjou ported pymemcache to Python 3, another memcached client. He suggests to use this one instead of python-memcached client.

Post by Thomas Goirand
It's blocking me from adding Python3 support to
keystoneclient, and as a consequence, to almost all of OpenStack.

python-keystoneclient announces a full Python 3 support with a voting Python 3 gate. I just checked locally, "tox -e py34" pass.
The problem is that python-memcached miss in test dependencies and so middleware tests using memcache are never run!

This is a bug worth fixing. Those skips should be removed, as memcached
is quite popular as a token store. So I opened this bug to track it:

https://bugs.launchpad.net/python-keystoneclient/+bug/1447731

[Joshua Harlow]

Post by Victor Stinner

Post by Thomas Goirand
Also, on the Python 3 topic, there's still a big issue with memcached
(aka: python-memcache).

https://github.com/linsomniac/python-memcached/pull/67
I don't understand. I saw a lot of "Port to Python 3" fixes merged in 2014, and these changes are now part of the release 1.54. Problem: running python-memcached tests with tox fail on obvious Python 3 errors. Anyway, all tests now pass with my pull request. The good news is that python-memcached looks to be actively developed.
Julien Danjou ported pymemcache to Python 3, another memcached client. He suggests to use this one instead of python-memcached client.

Yes please to using pymemcache (I'm also an active contributor[1] there,
so feel free to ask me questions to...), although what else in openstack
uses it? From my current understand the nova usage of it is suboptimal
(may not even be fully working/functional) due to:
https://review.openstack.org/#/c/138607/ which has exposed some
interesting details...

[1] https://github.com/pinterest/pymemcache/graphs/contributors

Post by Victor Stinner

Post by Thomas Goirand
It's blocking me from adding Python3 support to
keystoneclient, and as a consequence, to almost all of OpenStack.

python-keystoneclient announces a full Python 3 support with a voting Python 3 gate. I just checked locally, "tox -e py34" pass.
The problem is that python-memcached miss in test dependencies and so middleware tests using memcache are never run!
Victor
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[Thomas Goirand]

Post by Victor Stinner

Post by Thomas Goirand
Also, on the Python 3 topic, there's still a big issue with memcached
(aka: python-memcache).

https://github.com/linsomniac/python-memcached/pull/67

Cool! I'll try it, apply the patch to the version 1.54, and upload to
Debian, if this works well.

Post by Victor Stinner
I don't understand. I saw a lot of "Port to Python 3" fixes merged in 2014, and these changes are now part of the release 1.54. Problem: running python-memcached tests with tox fail on obvious Python 3 errors. Anyway, all tests now pass with my pull request. The good news is that python-memcached looks to be actively developed.
Julien Danjou ported pymemcache to Python 3, another memcached client. He suggests to use this one instead of python-memcached client.

pymemcache is much better in many ways, and switching to it shouldn't be
motivated only because of Python 3 compat.

Post by Victor Stinner

Post by Thomas Goirand
It's blocking me from adding Python3 support to
keystoneclient, and as a consequence, to almost all of OpenStack.

python-keystoneclient announces a full Python 3 support with a voting Python 3 gate. I just checked locally, "tox -e py34" pass.
The problem is that python-memcached miss in test dependencies and so middleware tests using memcache are never run!
Victor

Yeah, memcached (aka: python-memcache in Debian) is said to be
"optional", but in reality, we really need it for the full Python 3
compatibility, otherwise that's bullshit (unless we're giving up on
using memcache completely?).

Cheers,

Thomas Goirand (zigo)

[Thomas Goirand]

Post by Thomas Goirand

Post by Victor Stinner

Post by Thomas Goirand
Also, on the Python 3 topic, there's still a big issue with memcached
(aka: python-memcache).

Oh, thanks Thomas for the reminder. I just sent a pull request to port
https://github.com/linsomniac/python-memcached/pull/67

Cool! I'll try it, apply the patch to the version 1.54, and upload to
Debian, if this works well.

Yesterday, I uploaded the latest commit of python-memcache + your patch
to Debian Experimental.

Post by Thomas Goirand

Post by Victor Stinner

Post by Thomas Goirand
It's blocking me from adding Python3 support to
keystoneclient, and as a consequence, to almost all of OpenStack.

python-keystoneclient announces a full Python 3 support with a voting
Python 3 gate. I just checked locally, "tox -e py34" pass.
The problem is that python-memcached miss in test dependencies and so
middleware tests using memcache are never run!
Victor

Yeah, memcached (aka: python-memcache in Debian) is said to be
"optional", but in reality, we really need it for the full Python 3
compatibility, otherwise that's bullshit (unless we're giving up on
using memcache completely?).

Now that python-memcache is fixed for Python 3, the other issue is
pysaml2 which also only supports Python 2. That's also a blocker for
keystoneclient.

Cheers,

Thomas Goirand (zigo)

[Brant Knudson]

Post by Thomas Goirand

Post by Thierry Carrez

Post by Victor Stinner
I fixed 4 issues with monkey-patching in Python 3 (importlib, os.open(),
threading.RLock, threading.Thread). Good news: the just released eventlet
0.17.3 includes these fixes and it is now fully compatible with Python 3!
For example, the Oslo Messaging test suite now pass with this eventlet
version! Currently, eventlet is disabled in Oslo Messaging on Python 3
(eventlet tests are skipped).

Great news ! That makes the "port to Python 3" question independent of
the "Moving off eventlet" question, which should facilitate immediate
progress on the former.
On the latter, do you plan to file a Concurrency models cross-project
session ? That sounds like a good topic to discuss face to face...
See
http://lists.openstack.org/pipermail/openstack-dev/2015-April/061070.html for
details on how to file there.

Also, on the Python 3 topic, there's still a big issue with memcached
(aka: python-memcache). It's blocking me from adding Python3 support to
keystoneclient, and as a consequence, to almost all of OpenStack.
http://kilo-jessie.pkgs.mirantis.com/debian/pool/jessie-kilo-backports-nochange/main/p/python-eventlet/
and I will upload this to Experimental as soon as Jessie is released
(that's in 3 days now...).
Cheers,
Thomas Goirand (zigo)

The part of keystoneclient that uses the memcached client was deprecated in
Juno (as it was moved to the keystonemiddleware repo), so I think we can
remove it now. You might want to patch it out of your keystoneclient
package if you know everything's using the auth_token middleware from
keystonemiddleware.

- Brant

Post by Thomas Goirand
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[Victor Stinner]

Hi,

The part of keystoneclient that uses the memcached client was deprecated in Juno (as it was moved to the keystonemiddleware repo),

Oh, I was not aware of the keystonemiddleware project. I see that Nova uses it for example.

so I think we can remove it now.

Do someone know if the middleware of keystoneclient is still used?

You might want to patch it out of your keystoneclient package if you know everything's using the auth_token middleware from keystonemiddleware.

What should we do with the following bug?

"memcache tests are skipped on python 3"
https://bugs.launchpad.net/python-keystoneclient/+bug/1447731

I also wrote a patch for keystoneclient:

"Enable test_auth_token_middleware() on Python 2"
https://review.openstack.org/#/c/176778/

If the keystoneclient middleware is removed, we can simply close the bug and abandon the change.

Victor

[Julien Danjou]

Post by Victor Stinner
What should we do with the following bug?
"memcache tests are skipped on python 3"
https://bugs.launchpad.net/python-keystoneclient/+bug/1447731

Check if it applies to keystonemiddleware and if so move it there
otherwise close it?

[Morgan Fainberg]

Keystone client version of the middleware is deprecated and only receiving minimal security updates. This code is unlikely to see any real changes due it its deprecation and frozen state.

We are evaluating how to remove it from the client lib.

Sent via mobile

Post by Victor Stinner
Hi,

The part of keystoneclient that uses the memcached client was deprecated in Juno (as it was moved to the keystonemiddleware repo),

Oh, I was not aware of the keystonemiddleware project. I see that Nova uses it for example.

so I think we can remove it now.

Do someone know if the middleware of keystoneclient is still used?

You might want to patch it out of your keystoneclient package if you know everything's using the auth_token middleware from keystonemiddleware.

What should we do with the following bug?
"memcache tests are skipped on python 3"
https://bugs.launchpad.net/python-keystoneclient/+bug/1447731
"Enable test_auth_token_middleware() on Python 2"
https://review.openstack.org/#/c/176778/
If the keystoneclient middleware is removed, we can simply close the bug and abandon the change.
Victor
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[Victor Stinner]

Hi,

Post by Morgan Fainberg
Keystone client version of the middleware is deprecated and only receiving minimal security updates. This code is unlikely to see any real changes due it its deprecation and frozen state.
We are evaluating how to remove it from the client lib.

I wrote a patch to remove keystoneclient.middleware:

https://review.openstack.org/#/c/177694/

We may discuss the removal there. Are you aware of an application still relying on keystoneclient.middleware.

Victor

[INADA Naoki]

Post by Victor Sergeyev
Hi, All,
- yes, oslo.db supports python 3 (unittests passes, at least :) )
- MySQL-python still default MySQL DB driver in OpenStack, but at the
moment the only DB driver for MySQL in python3 environment is PyMySQL, so I
think, it's ok to use it with python 3.

​Hi.​

I'm maintainer of PyMySQL and mysqlclient.

mysqlclient is fork of MySQL-python. It uses libmysqlclient.so.
It fixes some bugs, build issues and it support Python 3. For example:

* Support MariaDB's libmysqlclient.so
* Support microsecond in TIME column

I recommend to use mysqlclient instead of MySQL-python even on Python 2.

https://pypi.python.org/pypi/mysqlclient
https://github.com/PyMySQL/mysqlclient-python

[Ihar Hrachyshka]

Post by Victor Sergeyev
Hi, All,
- yes, oslo.db supports python 3 (unittests passes, at least :) ) -
MySQL-python still default MySQL DB driver in OpenStack, but at
the moment the only DB driver for MySQL in python3 environment is
PyMySQL, so I think, it's ok to use it with python 3.
?Hi.?
I'm maintainer of PyMySQL and mysqlclient.
mysqlclient is fork of MySQL-python. It uses libmysqlclient.so. It
fixes some bugs, build issues and it support Python 3. For
* Support MariaDB's libmysqlclient.so * Support microsecond in TIME
column
I recommend to use mysqlclient instead of MySQL-python even on
Python 2.
https://pypi.python.org/pypi/mysqlclient
https://github.com/PyMySQL/mysqlclient-python

Is it packaged in popular distributions? RHEL? Fedora? SuSe? Ubuntu?
Debian? Gentoo?

/Ihar

[Victor Stinner]

Post by Ihar Hrachyshka

Post by INADA Naoki
I recommend to use mysqlclient instead of MySQL-python even on Python 2.
https://pypi.python.org/pypi/mysqlclient
https://github.com/PyMySQL/mysqlclient-python

Is it packaged in popular distributions? RHEL? Fedora? SuSe? Ubuntu?
Debian? Gentoo?

If this library solves real bugs and provides Python 3 compatibility, I think it's worth to replace MySQL-python with this new library. Linux distro can package it, they can probably copy the packaging of MySQL-Python since it's a fork.

Note: mysqlclient-python conflicts with MySQL-Python, both libraries use the same "MySQLdb" Python module. The good news is that mysqlclient-python is a drop-in library for MySQL-Python: no need to change any line of code, only modify requirements.

@Naoki: Did you try to contact MySQL-Python authors to take over their project instead of forking?

Victor