[Distutils] The problem with Setuptools on Python 3.

[Lennart Regebro]

Executive summary:
I have catch 22 problems in finishing the setuptools port to Python 3,
and unless somebody can come up with alternatives, I may rip out
setuptools dependency on itself completely, and only depend on
distutils.


I have as you know ported setuptools to Python 3. I'm in the process
of trying to fix and clean up that porting so that it can be merged
into trunk, but there is one obstacle left, and that's the question of
installing.

Currently there are several ways to install setuptools:

1) Run ez_install.py
2) Download / check-out source and run python setup.py install.
3) Download the correct egg and install that (which always confuses
me, I usually fail, and end up doing 2) instead).

Setuptools also helps with running tests by

4) python setup.py test

Currently for the Python 3 version, I've made a script that copies the
source tree and runs 2to3 on it. Then you can use that new tree to run
tests, install, make eggs and releases etc. The problems with this is:

a) The script is made for my computer only. It wouldn't work on
Windows, and it requires you to have lib2to3 checked out in a
particular place (although I think that's only if you use 3.0.0, and
not 3.0.1, but I haven't tested).

b) It's slow, as it copies all files, even those who hasn't changed
when you fixed a bug. Making a script that only copies the changed
files and runs 2to3 on them is possible, but I'd like not too, it
takes time and is likely to break. It seems like a waste.

Obviously, this is exactly the things distutils are meant to solve.
And indeed, distutils in Python 3 has a command called build_2to3
which solves exactly these problems. And indeed, this is the technique
I've used for the Python 3 branch of zope.interfaces. There I just run
setup.py install, and if I'm doing that under Python 3, it will first
run 2to3 on the code before installing. Same thing with running
setup.py test. It will sync changes to build/, run 2to3 on changed
files and then run the tests in build. It makes porting to Python 3
much easier, and it makes installing from source painless.

So why don't I use that for setuptools? Well, because:

c) The setup of setuptools requires setuptools. So to be able to do
the 2to3 conversion in the setup, I need to first convert the source
with 2to3. Yes, catch 22.

I've tried to get around this problem, but failed. Solutions I tried was:

I) Fall back to distutils if setuptools can't be imported. This means
that I can with some effort make a setup that will work under Python 3
and install setuptools under Python 3. Problem solved? No. Because
running setup.py again will just mean that it tries to import
setuptools from the local Python2 location, and it will fail. So in
this case I can't run tests of build eggs for setuptool.

II) Moving the source into a src directory. This means that when
setup.py runs, it will use the version of setuptools installed under
I), and building eggs etc is possible (I think, I haven't tested that
the eggs are correct). Problem solved? No, because you still can't
test it. Which leads us to problem d:

d) When running the tests, the setuptools module will already be
imported. But it will have imported the one in site-packages, *not*
the one in src, and it will therefore test the one in site-packages.
And that one doesn't have the api_tests.txt copied in, so that will
fail, and

e) even if api_tests.txt was copied, this would mean you had to
install setuptools before you test it, which is daft.


I don't have a good solution to this, unless we can drop setuptools
dependency on setuptools completely, and just use plain distutils for
installing and testing setuptools. This will mean no more setuptools
source eggs, but then I don't see the purpose of those, really.
Arguments for not doing this and alternative solutions are highly
welcome.

[Paul Moore]

Post by Lennart Regebro
I don't have a good solution to this, unless we can drop setuptools
dependency on setuptools completely, and just use plain distutils for
installing and testing setuptools. This will mean no more setuptools
source eggs, but then I don't see the purpose of those, really.
Arguments for not doing this and alternative solutions are highly
welcome.

Personally, this type of circular dependency seems totally wrong. I'd
argue that making setuptools not depend on itself is the right thing
to do, unless someone can provide a compelling reason why it is
necessary (and such a reason would hopefully then provide more
concrete avenues for looking for other possible alternative
approaches).

Paul.

[Ben Finney]

Post by Paul Moore

Post by Lennart Regebro
I don't have a good solution to this, unless we can drop setuptools
dependency on setuptools completely, and just use plain distutils
for installing and testing setuptools.

Personally, this type of circular dependency seems totally wrong. I'd
argue that making setuptools not depend on itself is the right thing
to do, unless someone can provide a compelling reason why it is
necessary (and such a reason would hopefully then provide more
concrete avenues for looking for other possible alternative
approaches).

Yes, that seems crazy-making to me and I wasn't aware it was the current
situation.

+1 for building setuptools on a base of distutils only, especially if
it's already been achieved.

[Lennart Regebro]

Post by Ben Finney
+1 for building setuptools on a base of distutils only, especially if
it's already been achieved.

No, we are going to have to make special custom extensions, at least
for running tests. It's not that much work, but it is code
duplication. I don't have the full overview of what features we lose
either. But it does seem to me that most of these features, like the
source eggs, cause more trouble than they solve.

We'll probably need a separate Python 3 version of easy_install and
the virtual-python scripts as well, but I already did those. But I
think getting easy_install to download the source tgz and run setup.py
install on it are going to simplify the code and remove the need for
the eggs.

Setuptools seems to have a lot of clever code to fix non-problems.
This may very well be because I don't know what the real problems are,
and these problems are not documented. I guess we need Phillip's input
on that.

[Paul Moore]

Post by Lennart Regebro

Post by Ben Finney
+1 for building setuptools on a base of distutils only, especially if
it's already been achieved.

No, we are going to have to make special custom extensions, at least
for running tests. It's not that much work, but it is code
duplication. I don't have the full overview of what features we lose
either. But it does seem to me that most of these features, like the
source eggs, cause more trouble than they solve.

Are you saying that you need to use setuptools (or at least the
feaures of setuptools) to develop setuptools? That's crazy. To run the
setuptools tests, just run the test.py (or whatever) script. The
setuptools ability to type python setup.py test, while convenient,
simply isn't available while you're developing setuptools. The same
logic applies to *any* setuptools feature that is used in the
development of setuptools itself. Trying to make it available adds
lots of complexity for the benefit of very few people (ie, people
writing the setuptools code).

Bootstrapping like this should be reserved for people writing C
compilers in C, and other equally major-league projects.

Just my view, and I'm not a setuptools fan, so you can probably
disregard it. (But this type of complexity for its own sake is the
reason I dislike the whole setuptools philosophy, so maybe it's worth
making explicit...)

Paul.

[Lennart Regebro]

Post by Paul Moore
Are you saying that you need to use setuptools (or at least the
feaures of setuptools) to develop setuptools?

Currently, yes. The testsuite is run with the testrunner of
setuptools. You can run them separately too, I'm sure, but that's a
pain. There is no test.py to run. Maybe you can use nose or something
to run the tests, but then again you would need to port *nose* to
Python 3... :-/

Post by Paul Moore
That's crazy. To run the
setuptools tests, just run the test.py (or whatever) script.

There isn't any. And to do that you need to convert it to Python 3
first, and we are back to the "run a porting script, then run a test
command", which is what we want to avoid in the first place. distutils
have code exactly to help this kind of situation, so I think we should
depend on that.

Post by Paul Moore
setuptools ability to type python setup.py test, while convenient,
simply isn't available while you're developing setuptools.

Sure it is. Maybe it *shouldn't*. But it *is*. Except under Python 3,
of course, which is the problem.

Post by Paul Moore
logic applies to *any* setuptools feature that is used in the
development of setuptools itself. Trying to make it available adds
lots of complexity for the benefit of very few people (ie, people
writing the setuptools code).

It's not so much "trying to". Since the structure of the package is
such that setuptools end up directly on the path, it *is* available.
So the question is if we should make sure it *isn't* available, by
moving it all down into a src directory, and then modify the
setup-script so that it no longer depends on setuptools.

Post by Paul Moore
Bootstrapping like this should be reserved for people writing C
compilers in C, and other equally major-league projects.

Are you saying setuptools isn't major-league? ;-)

[David Cournapeau]

Post by Paul Moore
Are you saying that you need to use setuptools (or at least the
feaures of setuptools) to develop setuptools? That's crazy.To run the
setuptools tests, just run the test.py (or whatever) script. The
setuptools ability to type python setup.py test, while convenient,
simply isn't available while you're developing setuptools. The same
logic applies to *any* setuptools feature that is used in the
development of setuptools itself. Trying to make it available adds
lots of complexity for the benefit of very few people (ie, people
writing the setuptools code).
Bootstrapping like this should be reserved for people writing C
compilers in C, and other equally major-league projects.

I don't know the details of setuptools, but it is generally quite
tempting to develop a new build/distribution tool using the new build
tool, with some bootstrap process. That's how most build tools I know
work, actually. FWIW, we use this as well in numpy for numpy.distutils
extensions.

And as much as you can count me in the "not a setuptools fan camp", I
think it is easy to say setuptools code is bad - that's the natural
reaction, really, and I would be surprised if P.J.E would not agree. But
I also think anyone who had to deal with distutils extensions will tell
you the same story - that's inherent to how distutils was conceived (10
years ago) and implemented. The distutils codebase is pretty horrible -
I find m4 macro and 100000 lines of shell code in autoconf easier to
deal with, really. You can deal with it, but it will certainly never be
pretty.

David

[Tarek Ziadé]

On Mon, Apr 20, 2009 at 3:23 PM, David Cournapeau

Post by David Cournapeau

Post by Paul Moore
Are you saying that you need to use setuptools (or at least the
feaures of setuptools) to develop setuptools? That's crazy.To run the
setuptools tests, just run the test.py (or whatever) script. The
setuptools ability to type python setup.py test, while convenient,
simply isn't available while you're developing setuptools. The same
logic applies to *any* setuptools feature that is used in the
development of setuptools itself. Trying to make it available adds
lots of complexity for the benefit of very few people (ie, people
writing the setuptools code).
Bootstrapping like this should be reserved for people writing C
compilers in C, and other equally major-league projects.

I don't know the details of setuptools, but it is generally quite
tempting to develop a new build/distribution tool using the new build
tool, with some bootstrap process. That's how most build tools I know
work, actually. FWIW, we use this as well in numpy for numpy.distutils
extensions.
And as much as you can count me in the "not a setuptools fan camp", I
think it is easy to say  setuptools code is bad - that's the natural
reaction, really, and I would be surprised if P.J.E would not agree. But
I also think anyone who had to deal with distutils extensions will tell
you the same story - that's inherent to how distutils was conceived (10
years ago) and implemented. The distutils codebase is pretty horrible -
I find m4 macro and 100000 lines of shell code in autoconf easier to
deal with, really. You can deal with it, but it will certainly never be
pretty.

this can change. I am working on it. I need feedback though.

let me know how my extension code proposal fits your needs.

http://wiki.python.org/moin/Distutils/ManifestPluginSystem/Draft

Post by David Cournapeau
David
_______________________________________________
http://mail.python.org/mailman/listinfo/distutils-sig

[David Cournapeau]

Hi Tarek,

Post by Tarek Ziadé
this can change. I am working on it. I need feedback though.
let me know how my extension code proposal fits your needs.
http://wiki.python.org/moin/Distutils/ManifestPluginSystem/Draft

I will fill in the examples (this pages does not concern manifest only,
then ?) from my own experience.

IMHO, adding plugins systems will not change the fundamental
deficiencies of distutils, though. You may be able to mitigate some
problems, but I don't think you will be able to solve the fundamental
issues, because they are fundamental design issues. The only way to fix
them would induce serious breakage, and in that case, given the quality
of distutils code, starting from scratch would be easier - I have yet
encountered a single major distutils abstraction which was not poorly
conceived and implemented (my own experience is related to build issues,
for which numpy has requirements which go much beyond the usual python
package, though). Commands classes, compiler classes, distributions, all
this is very wrong for someone who wants more than compiling a couple of
C files and tight control.

David

[Tarek Ziadé]

On Mon, Apr 20, 2009 at 4:00 PM, David Cournapeau

Post by David Cournapeau
Hi Tarek,

Post by Tarek Ziadé
this can change. I am working on it. I need feedback though.
let me know how my extension code proposal fits your needs.
http://wiki.python.org/moin/Distutils/ManifestPluginSystem/Draft

I will fill in the examples (this pages does not concern manifest only,
then ?) from my own experience.

no it doesn't. it's a generic tool to extend a command using plugins .

Post by David Cournapeau
IMHO, adding plugins systems will not change the fundamental
deficiencies of distutils, though. You may be able to mitigate some
problems, but I don't think you will be able to solve the fundamental
issues, because they are fundamental design issues. The only way to fix
them would induce serious breakage, and in that case, given the quality
of distutils code, starting from scratch would be easier -

Sorry, it's too easy to say "it sucks, scratch it and re-do it" ;)

1/ I am on Guido side here. I'd rather refactor the existing code.
2/ It's there, it works for a lot of people. let's improve it slowly
step by step
3/ the quality is increasing, the test coverage too. compare the trunk
with Python 2.5 one..

my proposal:

Step 1 - check the extension mechanism and see if it fits your brain
Step 2 - see where you would like to refactor distutils in order to
make it configurable/extensible

--

Tarek

[David Cournapeau]

no it doesn't. it's a generic tool to extend a command using plugins.

OK.

Sorry, it's too easy to say "it sucks, scratch it and re-do it" ;)

Yes, it is easy to say it sucks, but I am not saying that out of thin
air. Refactoring works if the general API is ok - but that's not the
case with distutils. As a data point, several people have tried to add
features to distutils in numpy extensions, and I went much further than
anyone else by bypassing distutils entirely, replacing the build_*
commands by scons. The core codebase is ten times smaller than the
original code it replaces (numpy.distutils is around 10 000 LOC, like
distutils itself).

I agree that in the short term, distutils should be improved, but in the
long term, I don't see anything which would be both a significant
improvement while staying backward compatible with distutils, if only
because so many setup.py files depend on implementation details of
distutils.

cheers,

Davud

[Tarek Ziadé]

On Mon, Apr 20, 2009 at 4:36 PM, David Cournapeau

Post by David Cournapeau

no it doesn't.  it's a generic tool to extend a command using plugins.

OK.

Sorry, it's too easy to say "it sucks, scratch it and re-do it" ;)

Yes, it is easy to say it sucks, but I am not saying that out of thin
air. Refactoring works if the general API is ok - but that's not the
case with distutils. As a data point, several people have tried to add
features to distutils in numpy extensions, and I went much further than
anyone else by bypassing distutils entirely, replacing the build_*
commands by scons. The core codebase is ten times smaller than the
original code it replaces (numpy.distutils is around 10 000 LOC, like
distutils itself).
I agree that in the short term, distutils should be improved, but in the
long term, I don't see anything which would be both a significant
improvement while staying backward compatible with distutils, if only
because so many setup.py files depend on implementation details of
distutils.

In the short term, Distutils needs to be a better citizen when people
wants to extend it, so they can do it without re-writing everything.

You have this experience of "pain", share it through real use cases.

In the long term Distutils needs to be *smaller* and to provide
a playground for other tools. in particular for all the tools that need to
build a binary package on a given platform.

(We can't keep those platform specific tools in the scope of distutils)

And the work done on various PEPs (all links at the top of
http://wiki.python.org/moin/Distutils)
are aiming to this goal.

What about organizing a sprint soon btw, so we don't loose Pycon efforts ?

[David Cournapeau]

Post by Tarek Ziadé
In the short term, Distutils needs to be a better citizen when people
wants to extend it, so they can do it without re-writing everything.
You have this experience of "pain", share it through real use cases.

I have added 5 usecases in the manifest plugin page, in order of
difficulty/pain. The first ones should be solvable in distutils with
your proposal, I am not sure about the last ones.

David

[Lennart Regebro]

On Mon, Apr 20, 2009 at 16:36, David Cournapeau

Post by David Cournapeau
I agree that in the short term, distutils should be improved, but in the
long term, I don't see anything which would be both a significant
improvement while staying backward compatible with distutils, if only
because so many setup.py files depend on implementation details of
distutils.

I think the answer to this can be only one: Since this is open source,
you can't expect somebody to do it. It will only be done if you do it
yourself, "you" being the people who experience the pain, in this
case, well, you. :-D

I sure as heck do not have the knowledge to do it. So the answer I
think, is the cold and brutal "Yes! Great Idea! When are you done?"
:-)


What I'm worried about with distutils is both what you mention:
Missing flexibility internally, and also that the plans are to make it
*smaller*, which inevitably will break backwards compatibility, which
will cause pains. And I still haven't gotten any answer on how to make
a backport of a new distutils and have that automatically installed
when you install a package that uses the new features. (Although maybe
just failing with an error asking you to install distutils vX.Y might
be acceptable).

It's quite possible that it's better to make something that at least
has another name, which totally avoids all the above problems. That
doesn't mean the code has to be all new and shiny. So I sure wouldn't
mind something new, that avoids the distutil problems. But wanting it
is something else than doing it. I can't. Can you? And it would be
needed rather quickly too. :-D

At least somebody could maybe make some sort of rough plan of what
this golden distutils should contain and how it should work, and we
can get a feeling if it's even remotely feasible.

[Lennart Regebro]

Post by Lennart Regebro
At least somebody could maybe make some sort of rough plan of what
this golden distutils should contain and how it should work, and we
can get a feeling if it's even remotely feasible.

And, as I frequently run into walls that make me thing setuptools
should be completely ignored, and then after fiddling about quite a
bit, find a way around it, and then run into the next wall, etc, etc,
etc. And these walls are getting more and more frequent... I'm
beginning to think that indeed rewrites are the only way.

We need a plan. Somebody please make a plan, maybe I can help. But
probably not. But I can try.

[Antonio Cavallo]

Post by Lennart Regebro
And, as I frequently run into walls that make me thing setuptools
should be completely ignored, and then after fiddling about quite a
bit, find a way around it, and then run into the next wall, etc, etc,
etc. And these walls are getting more and more frequent... I'm
beginning to think that indeed rewrites are the only way.
We need a plan. Somebody please make a plan, maybe I can help. But
probably not. But I can try.

I second that especially because I think setuptools looks like an over
engineering excercise: I have many doubts about about setuptools,
especially from a system administrative point of view.

Why a build system (and setup.py is) should have a scm support in it?
Why the plugin entry points should be part of such a system?
Why the dependecied should be part of it?

The scm support is part of a "development" stage, possibly useful to
the developers only: that'd be better covered by a python script not
tighted to the "building system".

The plugin entry points (belonging to a "deployment" stage) are a bit
of a problem because the paths might not be in their "final" form
(think of DESTDIR temporary staging area while building a rpm): this
mean the onlyreliable way to define entry points would be relative to
modulename.__file__ variable.

The dependencies part (belonging to a "deployment" stage) is a bit of
a system wide problem.
Rpm defines the same fields and then this introduce a (not so) hidden
duplication: will the package respect the python dependencies or the
rpm ones?

I would recomend having a look at what rpm (and especially the spec
files) does because it had embedded a lot of knoweledge about building
packages across the years: and it still has its own problems....

Moving forward, itwould be beneficial a flow chart of what distutils
should do (and shouldn't) and a possibly a usage cases study.
Regards,
Antonio

[Tres Seaver]

Post by Antonio Cavallo

Post by Lennart Regebro
And, as I frequently run into walls that make me thing setuptools
should be completely ignored, and then after fiddling about quite a
bit, find a way around it, and then run into the next wall, etc, etc,
etc. And these walls are getting more and more frequent... I'm
beginning to think that indeed rewrites are the only way.
We need a plan. Somebody please make a plan, maybe I can help. But
probably not. But I can try.

I second that especially because I think setuptools looks like an over
engineering excercise: I have many doubts about about setuptools,
especially from a system administrative point of view.
Why a build system (and setup.py is) should have a scm support in it?

Setuptools builds on distutils, which is primarily aimed at distributing
software; "building" is a minor part of the exercise (a huge
supermajority of distutils packages require no real "build" effort at all).

Post by Antonio Cavallo
Why the plugin entry points should be part of such a system?

Distributing plugins for Chandler was PJE's driving use case in the
original development of setuptools.

Post by Antonio Cavallo
Why the dependecied should be part of it?

Because distributing interrelated software sanely requires it.

Post by Antonio Cavallo
The scm support is part of a "development" stage, possibly useful to
the developers only: that'd be better covered by a python script not
tighted to the "building system".

I don't know where you get the idea that setuptools / distutils is
primarily about "building" software.


Tres.
- --
===================================================================
Tres Seaver [phone] ***@palladion.com
Palladion Software "Excellence by Design" http://palladion.com

[David Cournapeau]

Post by Tres Seaver
I don't know where you get the idea that setuptools / distutils is
primarily about "building" software.

The main documentation of distutils says "building and installing python
modules". A large fraction of distutils code is related to building.

David

[Greg Ewing]

Post by Lennart Regebro
We need a plan. Somebody please make a plan

Whatever plan you come up with, I'd like to see the
basic architecture built around a dependency graph,
at least for the parts concerned with building
extension modules from sources.

My use case for handling Pyrex sources. I'd like
to be able to plug in a module that knows how
to turn .pyx files into .c files, without requiring
arcane internal knowledge and without interfering
with any other extensions I might want to use at
the same time.

[Tarek Ziadé]

Please Greg, add your use case here with David's one, at the end of the page,
since we are trying to work on a solution.

http://wiki.python.org/moin/Distutils/ManifestPluginSystem/Draft

Post by Greg Ewing

Post by Lennart Regebro
We need a plan. Somebody please make a plan

Whatever plan you come up with, I'd like to see the
basic architecture built around a dependency graph,
at least for the parts concerned with building
extension modules from sources.
My use case for handling Pyrex sources. I'd like
to be able to plug in a module that knows how
to turn .pyx files into .c files, without requiring
arcane internal knowledge and without interfering
with any other extensions I might want to use at
the same time.
--
Greg
_______________________________________________
http://mail.python.org/mailman/listinfo/distutils-sig

[Lennart Regebro]

Post by Tarek Ziadé
Please Greg, add your use case here with David's one, at the end of the page,
since we are trying to work on a solution.
http://wiki.python.org/moin/Distutils/ManifestPluginSystem/Draft

I realized another usecase, which may or may not be covered already,
should I add it to the page?


== Optional extensions ==

Some modules, such as zope.interface, have C-extensions which are only
performance enhancing and have pure-python fallbacks. It should be
easy to write an a command that if it fails doesn't stop the rest of
the build, and also includes the right files if it succeeds, but
doesn't fail when these files can't be built.

[David Lyon]

Post by Tarek Ziadé
Please Greg, add your use case here with David's one, at the end of the

page,

Post by Tarek Ziadé
since we are trying to work on a solution.
http://wiki.python.org/moin/Distutils/ManifestPluginSystem/Draft

I can identify three major areas of concern:

1) Platform testing/builds (via cloud computing)
2) External build tools
3) too many versions of python

One thing you guys should also consider is automated building/testing
of packages. On different platforms.

Let me give you a practical example...

package win32...

latest version.. no longer works on windows 2000...

only works on xp... or vista...

Bad for me.. takes me time to figure out whats wrong...

Only solution.. go back and find an older release.. yes.. i know..
I can do that... that works...

With cloud computing... you can get test environments for every platform

Amazon web services as one example offer this

Perl does nightly "builds" and tests....

their automated test system "tests everything"

Another big issue i dislike with many packages... they depend on
having msvc installed to be able to build them.

In many cases because I don't have MSC installed... this prevents
me from building some packages. I don't like that...

I don't believe building "everything" on the 5-10 major platforms
we have (or however many there are) is too much.

Plus now, we have all these different versions of python.. and
instead of getting better it is only seeming to get more worse...

David

[Tarek Ziadé]

Post by David Lyon
One thing you guys should also consider is automated building/testing
of packages. On different platforms.
Let me give you a practical example...
package win32...
latest version.. no longer works on windows 2000...
only works on xp... or vista...
Bad for me.. takes me time to figure out whats wrong...
Only solution.. go back and find an older release.. yes.. i know..
I can do that... that works...
With cloud computing... you can get test environments for every platform
Amazon web services as one example offer this
Perl does nightly "builds" and tests....
their automated test system "tests everything"
Another big issue i dislike with many packages... they depend on
having msvc installed to be able to build them.
In many cases because I don't have MSC installed... this prevents
me from building some packages. I don't like that...
I don't believe building "everything" on the 5-10 major platforms
we have (or however many there are) is too much.
Plus now, we have all these different versions of python.. and
instead of getting better it is only seeming to get more worse...

I had this idea of running a test server that would listen to PyPI and
get every new package to build it and test it on various platform/python.

I think this could be done right now, with how distutils works,

It requires creating virtual machines and destroying them every time
for security reasons.

But it's quite a work...

Cheers
Tarek

[David Lyon]

Post by Tarek Ziadé
I had this idea of running a test server that would listen to PyPI and
get every new package to build it and test it on various platform/python.
I think this could be done right now, with how distutils works,
It requires creating virtual machines and destroying them every time
for security reasons.
But it's quite a work...

I think it is very important to put this software infrastructure into
place.

or at least include it in some sort of plan going forward....

have the server farm report build bugs back to the maintainers...

I am prepared to assist with this... and perphaps cover some of the costs.

Amazon web services provide the very same virtual machines that you are
describing at extremely low cost. Otherwise it is just two much work.

Google also offer a similar thing.


David

[Tarek Ziadé]

Post by David Lyon

Post by Tarek Ziadé
But it's quite a work...

I think it is very important to put this software infrastructure into
place.
or at least include it in some sort of plan going forward....
have the server farm report build bugs back to the maintainers...
I am prepared to assist with this... and perphaps cover some of the costs.
Amazon web services provide the very same virtual machines that you are
describing at extremely low cost. Otherwise it is just two much work.
Google also offer a similar thing.

Wonderful ! Money-wise, I think the PSF would probably cover these
expenses if we come up
with a good, well-defined project. I have tried to make it a GSoc
topic but I didn't find any student
picking it up because I didn't find the time to describe it deeply enough.

I have created a page here :
http://wiki.python.org/moin/Distutils/TestingInfrastructure

If you want to start there on the topic, that could be great.

Tarek

Post by David Lyon
David

[Lennart Regebro]

Maybe we should stop talking about how to improve distutils under the
topic of what the problem is with setuptools under Python 3? It's
pretty unrelated. ;-)

[Antonio Cavallo]

SuSE runs a build service for free and support automatic rebuild of
packages from sources:


https://build.opensuse.org


For anyone interested you can find the lates svn python snapshot under:

http://download.opensuse.org/repositories/home:/cavallo71:/python-opt/

Each subdirectory (CentOS_5, RHEL_5 etc) contains a yum repository
to download the packages opt-python-XXXX

These python packages install under their own separate directory tree
(/opt/opt-python-2.7a0 is the latest snapshot).

In order to use one of them all it is needed is:
$> . /opt/opt-python-2.7a0/opt-python-env.sh
$> python

Python 2.7a0 (trunk, Apr 20 2009, 12:45:44)
[GCC 4.3.1 20080507 (prerelease) [gcc-4_3-branch revision 135036]] on linux2
Type "help", "copyright", "credits" or "license" for more information.

Post by Tarek Ziadé

Post by David Lyon

import os; print os.__file__

/opt/opt-python-2.7a0/lib/python2.7/os.pyc


I hope this helps in testing,
Regards,
Antonio

Post by Tarek Ziadé

Post by David Lyon

But it's quite a work...

I think it is very important to put this software infrastructure into
place.
or at least include it in some sort of plan going forward....
have the server farm report build bugs back to the maintainers...
I am prepared to assist with this... and perphaps cover some of the costs.
Amazon web services provide the very same virtual machines that you are
describing at extremely low cost. Otherwise it is just two much work.
Google also offer a similar thing.

Wonderful ! Money-wise, I think the PSF would probably cover these
expenses if we come up
with a good, well-defined project. I have tried to make it a GSoc
topic but I didn't find any student
picking it up because I didn't find the time to describe it deeply enough.
http://wiki.python.org/moin/Distutils/TestingInfrastructure
If you want to start there on the topic, that could be great.
Tarek

Post by David Lyon
David

--
Tarek Ziadé | http://ziade.org
_______________________________________________
http://mail.python.org/mailman/listinfo/distutils-sig

[Eric Smith]

Post by Tarek Ziadé
Please Greg, add your use case here with David's one, at the end of the page,
since we are trying to work on a solution.
http://wiki.python.org/moin/Distutils/ManifestPluginSystem/Draft

Could the people adding the use cases put their names by them, so if we
have questions later we know who to ask?

[David Cournapeau]

Post by Eric Smith

Post by Tarek Ziadé
Please Greg, add your use case here with David's one, at the end of the page,
since we are trying to work on a solution.
http://wiki.python.org/moin/Distutils/ManifestPluginSystem/Draft

Could the people adding the use cases put their names by them, so if
we have questions later we know who to ask?

Good idea, I have added my names to 'my' examples,

David

[Greg Ewing]

Post by Tarek Ziadé
Please Greg, add your use case here with David's one, at the end of the page,
since we are trying to work on a solution.
http://wiki.python.org/moin/Distutils/ManifestPluginSystem/Draft

Is this really the best place for it? My use case
doesn't have anything to do with manifests.

We need a page like this concerning overall distutils
reorganisation.

[David Cournapeau]

Post by Greg Ewing

Post by Tarek Ziadé
Please Greg, add your use case here with David's one, at the end of the page,
since we are trying to work on a solution.
http://wiki.python.org/moin/Distutils/ManifestPluginSystem/Draft

Is this really the best place for it? My use case
doesn't have anything to do with manifests.

Mine neither. I agree the title is confusing, but OTOH, moving things in
wiki is easy, whereas waiting for things to be at the right place
increases the risk of losing the information. That's how wiki work.

We need more people who care about the building issues (vs the
distribution issues) if we want to be taken in account for distutils
improvements :)

David

[David Lyon]

I really second this...

after all.. we're only talking about a file copying program..

Surely after so many years... we must have a clear handle on what
files need to be copied where with what permissions....

On Mon, 20 Apr 2009 23:00:39 +0900, David Cournapeau

Post by David Cournapeau
IMHO, adding plugins systems will not change the fundamental
deficiencies of distutils, though. You may be able to mitigate some
problems, but I don't think you will be able to solve the fundamental
issues, because they are fundamental design issues. The only way to fix
them would induce serious breakage, and in that case, given the quality
of distutils code, starting from scratch would be easier - I have yet
encountered a single major distutils abstraction which was not poorly
conceived and implemented (my own experience is related to build issues,
for which numpy has requirements which go much beyond the usual python
package, though). Commands classes, compiler classes, distributions, all
this is very wrong for someone who wants more than compiling a couple of
C files and tight control.

[P.J. Eby]

Post by Lennart Regebro
I don't have a good solution to this, unless we can drop setuptools
dependency on setuptools completely, and just use plain distutils for
installing and testing setuptools.

I still don't understand why you can't have a setup3.py that uses
only distutils, in order to run the build2to3 on. Am I missing something?

i.e., ISTM that you should be able to do something like:

python3 setup3.py build_2to3
cd wherevertheoutputgoes
python3 setup.py test (or whatever)

[Lennart Regebro]

I still don't understand why you can't have a setup3.py that uses only
distutils, in order to run the build2to3 on.  Am I missing something?

Well, the question then becomes why not just depends on distutils all
the way? With a setup3.py that does this, most of the work for
removing this self-dependency is done.

 python3 setup3.py build_2to3
 cd wherevertheoutputgoes
 python3 setup.py test  (or whatever)

Well, first of all the distutils build command doesn't include the
files needed, such as setup.py, and also it doesn't copy the txt files
that is the doctests. So it would need an extended version of
build_py_2to3 that does copy these files. And instead of doing "cd
wherever, python2 setup.py test", we can just as well just do a custom
command that runs the custom build_py_2to3 and runs the tests from
there too.

So in fact, what you are suggesting here is almost exactly what I'm
suggesting, except that I don't see the point in confusing and
complicating things by keeping the old setup.py.

[P.J. Eby]

Post by Lennart Regebro

I still don't understand why you can't have a setup3.py that uses only
distutils, in order to run the build2to3 on. Am I missing something?

Well, the question then becomes why not just depends on distutils all
the way? With a setup3.py that does this, most of the work for
removing this self-dependency is done.

python3 setup3.py build_2to3
cd wherevertheoutputgoes
python3 setup.py test (or whatever)

Well, first of all the distutils build command doesn't include the
files needed, such as setup.py, and also it doesn't copy the txt files
that is the doctests. So it would need an extended version of
build_py_2to3 that does copy these files. And instead of doing "cd
wherever, python2 setup.py test", we can just as well just do a custom
command that runs the custom build_py_2to3 and runs the tests from
there too.
So in fact, what you are suggesting here is almost exactly what I'm
suggesting, except that I don't see the point in confusing and
complicating things by keeping the old setup.py.

Because that's the one that generates the metadata setuptools needs
to run, test itself, etc.

[Lennart Regebro]

Because that's the one that generates the metadata setuptools needs to run,
test itself, etc.

You really need to stop assuming everybody else here understands the
issues as well as you do, because we don't. Or well, I don't, anyway.
I don't understand a word in your answer, it makes no sense to me
whatsoever, Try answering like I'm a moron, because I probably am.

Because that's the one that generates the metadata setuptools needs to run,
test itself, etc.

No, setup3.py does that. Why do I need setup.py?

[Lennart Regebro]

Because that's the one that generates the metadata setuptools needs to run,
test itself, etc.

Why do I need setuptools to do that? Why is not distutils enough?

[P.J. Eby]

Post by Lennart Regebro

Because that's the one that generates the metadata setuptools needs to run,
test itself, etc.

Why do I need setuptools to do that? Why is not distutils enough?

Because distutils doesn't have an egg_info command, or SVN-based
manifest building (for making sdists).

[Antonio Cavallo]

Post by Lennart Regebro

Because that's the one that generates the metadata setuptools needs to run,
test itself, etc.

Why do I need setuptools to do that? Why is not distutils enough?

Because distutils doesn't have an egg_info command, or SVN-based manifest
building (for making sdists).

Why the SVN handling should be part of setup.py anyway?
Regards,
Antonio

[P.J. Eby]

Post by Lennart Regebro

Because that's the one that generates the metadata setuptools needs to run,
test itself, etc.

No, setup3.py does that.

I thought you couldn't import setuptools in setup3.py, for all the
reasons you just described?

Post by Lennart Regebro
Why do I need setup.py?

To define the core entry points, basically. These are really
bootstrapped from the existing entry_points.txt (which is why it's in
SVN), but they need to be regenerated at egg_info/bdist_egg/install
time to match the current Python version.

Apart from that, I believe setuptools only uses itself for finding
its own packages (including data) and manifest building (i.e. finding
files that are under SVN control).

So I suppose that if you dropped the use of find_packages(), you
could make a setup.py that would let you run distutils commands on
the code base, I just don't see how you'd get any setuptools-level
commands to work properly without setuptools being imported first.

i.e., you could probably manage having one setup.py, or two
setup.py's whose only difference is whether they import setuptools
and use find_packages; like a setup3.py that does distutils, and a
regular setup.py that imports setuptools and then execfiles setup3.py.

None of this makes setuptools any less reliant on itself for testing
and building, though, since the core entry points have to exist in
order for commands to be found, options to be verified, etc.

[Lennart Regebro]

I thought you couldn't import setuptools in setup3.py, for all the reasons
you just described?

Yes? And then you draw some sort of conclusion of this that totally escapes me.

Post by Lennart Regebro
Why do I need setup.py?

To define the core entry points, basically.  These are really bootstrapped
from the existing entry_points.txt (which is why it's in SVN), but they need
to be regenerated at egg_info/bdist_egg/install time to match the current
Python version.

OK. And why are they needed to install and test setuptools?

Apart from that, I believe setuptools only uses itself for finding its own
packages (including data) and manifest building (i.e. finding files that are
under SVN control).

Sure, but that's easily avoided.

So I suppose that if you dropped the use of find_packages(), you could make
a setup.py that would let you run distutils commands on the code base, I
just don't see how you'd get any setuptools-level commands to work properly
without setuptools being imported first.

They won't work. The question is why they need to work.

None of this makes setuptools any less reliant on itself for testing and
building, though, since the core entry points have to exist in order for
commands to be found, options to be verified, etc.

Which still doesn't really answer the question: Why setuptools need to
rely on setuptools.

Because distutils doesn't have an egg_info command

I've already mentioned that getting rid of the setuptools dependency
on itself means you can't build setuptools eggs. But I've also pointed
out that I don't understand why they would be needed. For all other
packages it's recommended that only source distributions are done
unless you have c-extensions. Why is setuptools an exception?

or SVN-based manifest building (for making sdists).

Well, we can survive without that, as far as I can see.

[P.J. Eby]

Post by Lennart Regebro
Which still doesn't really answer the question: Why setuptools need to
rely on setuptools.

Because there's less duplication and chances of error that
way. (Earlier versions of setuptools relied on manually-created text
files, instead of using egg_info to generate them, and
creating/maintaining the files was a pain.)

Post by Lennart Regebro

Post by P.J. Eby
Because distutils doesn't have an egg_info command

I've already mentioned that getting rid of the setuptools dependency
on itself means you can't build setuptools eggs. But I've also pointed
out that I don't understand why they would be needed.

Because the egg-info files (irrespective of whether you build an
sdist, an .egg, an .exe, or even install a "flat" package) contain
information like entry points and requirements.

In the case of setuptools itself, the entry points are critical
because setuptools finds its own commands, options, etc., almost
exclusively by using entry points. This is so that it's on a level
playing field with other packages wanting to extend those features,
and serves as an example of how to extend itself.

Look, if you want to fork setuptools to not depend on itself, knock
yourself out. I sure can't stop you.

It's simply that, from my POV, I'd rather not complicate the main
branches with duplication to support Python 3, when ISTM that the
Python-3-support bits could be isolated instead. Also, I think that
setuptools self-bootstrapping is both a non-trivial integration test
that shouldn't be dropped, AND a non-trivial example of how to extend
setuptools (since so much of it is implemented using independent
entry points rather than hardwired behavior).

All that having been said, I'm not the one doing the port, so do what
you think best.

Post by Lennart Regebro
For all other
packages it's recommended that only source distributions are done
unless you have c-extensions.

Recommended by whom, for what purpose?

Post by Lennart Regebro
Why is setuptools an exception?

For end-user convenience. A large number of people installing
setuptools are not installing it because they are personally
interested in it, but because something else they want uses it.

[Lennart Regebro]

Look, if you want to fork setuptools to not depend on itself, knock yourself
out.  I sure can't stop you.

Well, unless somebody can come up with an alternative... But I have to
say I'm extremely reluctant to do so. And I think that my action
instead will be to abandon all further efforts of porting things to
Python 3. This is too much pain, takes to much time and doesn't lead
anywhere. I have since July wanted to port the zope.component
architecture to Python 3. I've put down weeks of work on it. Almost
*all* that work has been related to setuptools.

It's simply that, from my POV, I'd rather not complicate the main branches
with duplication to support Python 3, when ISTM that the Python-3-support
bits could be isolated instead.

"Isolated"? What do you mean? If you have suggestions on how to solve
the problems I'm describing, can you please come forward with them,
I've asked several times already. All you do is say "no, no, no, no,
no", and "becasue". It isn't helpful.

All that having been said, I'm not the one doing the port, so do what you
think best.

No, because although you aren't the one doing the port, you still are
the main setuptools maintainer, and this will need to merged. Soon. As
I mentioned above, the fact that setuptools doesn't have good Python 3
support is a major hindrance for Python 3 acceptance.

 For all other
packages it's recommended that only source distributions are done
unless you have c-extensions.

Recommended by whom, for what purpose?

Everybody, for all purposes.

For end-user convenience.  A large number of people installing setuptools
are not installing it because they are personally interested in it, but
because something else they want uses it.

Yes? And again you don't explain how this leads to you conclusion.
Eggs are not easier to install, on the contrary, I have tried and
failed a couple of times, and ended up using the source install
instead. It's easy to install. You download, unpack, run python
setup.py install, like any other package.

[P.J. Eby]

Post by Lennart Regebro
"Isolated"? What do you mean?

Making a separate setup script for Python 3, at least for setuptools
itself, if not having a general convention for that, since other
packages may want to ship 2+3 stuff in the same package.

Or, in the alternative, using version testing in setup.py to run an
alternate script for Python 3.

Post by Lennart Regebro
If you have suggestions on how to solve
the problems I'm describing, can you please come forward with them,
I've asked several times already. All you do is say "no, no, no, no,
no", and "becasue". It isn't helpful.

I'm sorry you feel that way, as I've been *trying* to help. I just
still don't get what the problem is. If I were porting setuptools to
Python 3, I would *want* it to be circular, even if I had to hack on
it a little at first. So I have a hard time understanding why you don't.

But I'm not you -- I understand the code base and I'm not trying to
port it. Maybe if I were trying to port it, I would get what problem
you're having, or maybe I would just keep right on going and not
notice. I don't know. I've been trying to find out what exactly is
stopping you and just can't seem to wrap my brain around it, any more
than you've been able to about the reverse.

Post by Lennart Regebro

For end-user convenience. A large number of people installing setuptools
are not installing it because they are personally interested in it, but
because something else they want uses it.

Yes? And again you don't explain how this leads to you conclusion.

Mostly because your questions aren't pinpointing what you want to
know. You ask general questions, so I give general answers. We seem
to both be suffering from a surplus of assumptions of what should be

Post by Lennart Regebro
Eggs are not easier to install, on the contrary, I have tried and
failed a couple of times, and ended up using the source install
instead.

...seems to indicate that your question was actually about eggs, not
other-than-source distributions in general. I was mostly talking
about the wininst installers and source RPM. The eggs are there, on
the other hand, for ez_setup.py to download. (Not to mention
buildout's bootstrap script, and other tools that depend on
setuptools and want to have an automated overall install process.)

[Lennart Regebro]

Making a separate setup script for Python 3, at least for setuptools itself,
if not having a general convention for that, since other packages may want
to ship 2+3 stuff in the same package.

The typical setup script will look exactly the same under python 2 and
python 3. There is no need for separate scripts in the general
usecase. If you want to run 2to3 automatically, all you need to do is
set up build_py_2to3 instead of build_2to3, that's the only
difference, and that's easily fixed with a importexception.

This goes pretty much for setuptools also. The setup3.py script will
more or less work under python2 as well.

Or, in the alternative, using version testing in setup.py to run an
alternate script for Python 3.

You don't need alternative scripts.

setuptools is an exception, because it depends on itself, providing a
catch22 situation.

I'm sorry you feel that way, as I've been *trying* to help.  I just still
don't get what the problem is.  If I were porting setuptools to Python 3, I
would *want* it to be circular, even if I had to hack on it a little at
first.  So I have a hard time understanding why you don't.

But it CAN NOT be circular under Python 3.

 Maybe if I were trying to port it, I would get what problem you're having,
or maybe I would just keep right on going and not notice.  I don't know.

You will have it and I explained in the mail I sent as a start of this
discussion. If I was unclear, please tell me what you didn't
understand.

 I've been trying to find out what exactly is stopping you and just can't
seem to wrap my brain around it, any more than you've been able to about the
reverse.
Mostly because your questions aren't pinpointing what you want to know.

It is pinpointing them exactly. I want to know why setuptools need to
depends on setuptools. Your answer, as I can understand it is for
convenience, and so that it serves as a test and example of it's own
features.

The fact that is serves as a test of it's own features is another
pain. That was a big reason for the difficulty of porting, as even
when testcases all passed, not all features worked. So I have to say
that although it sounds reasonable, I think it's misguided.

Post by Lennart Regebro
Eggs are not easier to install, on the contrary, I have tried and
failed a couple of times, and ended up using the source install
instead.

...seems to indicate that your question was actually about eggs, not
other-than-source distributions in general.

Which probably is why I said eggs...

 The eggs are there, on the other hand,
for ez_setup.py to download. (Not to mention buildout's bootstrap script,
and other tools that depend on setuptools and want to have an automated
overall install process.)

OK, I wonder if there is a way around that. If not, then as far as I
can see, there is no way to install or develop with setuptools
smoothly in Python 3.

[Ronald Oussoren]

Post by Lennart Regebro

Post by P.J. Eby
I'm sorry you feel that way, as I've been *trying* to help. I just
still
don't get what the problem is. If I were porting setuptools to
Python 3, I
would *want* it to be circular, even if I had to hack on it a
little at
first. So I have a hard time understanding why you don't.

But it CAN NOT be circular under Python 3.

I don't understand why not, doing it may be not entirely trivial but
it should be possible with some trickery. As PJE noted one way to do
this is to explicitly convert setuptools to python 3.x syntax before
actually running setup.py (e.g. his setup3.py file). With some care
this could even be done in setup.py itself.

Ronald

[Lennart Regebro]

I don't understand why not, doing it may be not entirely trivial but it
should be possible with some trickery. As PJE noted one way to do this is
to explicitly convert setuptools to python 3.x syntax before actually
running setup.py (e.g. his setup3.py file). With some care this could even
be done in setup.py itself.

I'm sorry, I don't think I'm able to explain it more in detail than
what I did in the first post in this discussion, unless you are more
explicit about what I need to explain.

Here it is again, for convenience:

"Currently for the Python 3 version, I've made a script that copies the
source tree and runs 2to3 on it. Then you can use that new tree to run
tests, install, make eggs and releases etc. The problems with this is:

a) The script is made for my computer only. It wouldn't work on
Windows, and it requires you to have lib2to3 checked out in a
particular place (although I think that's only if you use 3.0.0, and
not 3.0.1, but I haven't tested).

b) It's slow, as it copies all files, even those who hasn't changed
when you fixed a bug. Making a script that only copies the changed
files and runs 2to3 on them is possible, but I'd like not too, it
takes time and is likely to break. It seems like a waste.

Obviously, this is exactly the things distutils are meant to solve.
And indeed, distutils in Python 3 has a command called build_2to3
which solves exactly these problems. And indeed, this is the technique
I've used for the Python 3 branch of zope.interfaces. There I just run
setup.py install, and if I'm doing that under Python 3, it will first
run 2to3 on the code before installing. Same thing with running
setup.py test. It will sync changes to build/, run 2to3 on changed
files and then run the tests in build. It makes porting to Python 3
much easier, and it makes installing from source painless.

So why don't I use that for setuptools? Well, because:

c) The setup of setuptools requires setuptools. So to be able to do
the 2to3 conversion in the setup, I need to first convert the source
with 2to3. Yes, catch 22.

I've tried to get around this problem, but failed. Solutions I tried was:

I) Fall back to distutils if setuptools can't be imported. This means
that I can with some effort make a setup that will work under Python 3
and install setuptools under Python 3. Problem solved? No. Because
running setup.py again will just mean that it tries to import
setuptools from the local Python2 location, and it will fail. So in
this case I can't run tests of build eggs for setuptool.

II) Moving the source into a src directory. This means that when
setup.py runs, it will use the version of setuptools installed under
I), and building eggs etc is possible (I think, I haven't tested that
the eggs are correct). Problem solved? No, because you still can't
test it. Which leads us to problem d:

d) When running the tests, the setuptools module will already be
imported. But it will have imported the one in site-packages, *not*
the one in src, and it will therefore test the one in site-packages.
And that one doesn't have the api_tests.txt copied in, so that will
fail, and

e) even if api_tests.txt was copied, this would mean you had to
install setuptools before you test it"

[P.J. Eby]

Post by Lennart Regebro
c) The setup of setuptools requires setuptools. So to be able to do
the 2to3 conversion in the setup, I need to first convert the source
with 2to3. Yes, catch 22.

What I still don't get is why it can't work in a 2-stage process,
running one setup.py with distutils to do the build_2to3, and then
running a different setup.py to do the tests.

I imagine that, if I were trying to support Python 3, what I would do
first is make a Python 2 setuptools command that ran 2to3 on a
setuptools-based Python 2 project, and generated a new source tree --
with all sdist-targeted content copied over, and all .py files
converted (including setup.py itself)... and then ran whatever extra
commands you gave, running Python 3 on the resulting setup.py, such that:

python2 setup.py 2to3 test

would automatically do the equivalent of

cd build/2to3; python3 setup.py test

after creating a converted distribution in build/2to3. That way, you
could also do things like:

python2 setup.py test 2to3 test

to run the tests in Python 2 before converting and running them in Python 3.

If somebody wants to create this command, perhaps that would be a
good idea. It can of course be implemented as a plugin, so a change
to setuptools itself is not required.

In the simplest case, the command could just derive from sdist and
build an sdist tree in build/2to3 each time, and then run 2to3 in
place. Or it could reuse sdist and unpack the sdist into the build
tree. This would be slower, but easier to code. A more advanced
version could check for changes in SOURCES.txt between the original
and the build/2to3 directory in order to find files to add/remove,
and only run 2to3 on changed .py files.

Something like:

if not os.path.exists(target_sources_txt):
# wipe build tree, build sdist and unpack

target_manifest = load_manifest(target_sources_txt)
for filename in target_manifest:
if filename not in original_manifest:
# delete file
continue
if is_older_or_nonexistent(filename):
# copy or run 2to3
for filename in original_manifest:
if filename not in target_manifest:
# copy or run 2to3

...but with a lot more os.path.join operations and a lot less
handwaving. ;-) And the initial version of this could just always
do the wipe-and-unpack step, although it'd still need to loop over
the files to run 2to3 anyway, I suppose.

Anyway, I know this is a fair amount of work; It just seems to me
that it has more uses than converting setuptools; i.e., it'd be a
useful rig for anybody doing 2-to-3 porting work.

[Lennart Regebro]

 python2 setup.py 2to3 test

Well, yes, but it should be

python 3 setup.py 2to3 test

Otherwise it can't reasonably have any idea of which python to use.
And when you run it with python3, the 2to3 command isn't needed, as
the 2to3 step can be automatically inserted through version detection.
In fact, this is exactly what I'm doing for zope.interface, as
explained. For zope.interface, you run
python2.5 setup.py test
and it will test it with python2.5 You run
python3.0 setup.py test
and it will do exactly what you say: copy the files to build, run
2to3, and then run the test on that in-place.
python3.0 setup.py install
will copy the files to build, run 2to3 and then install it.

Anyway, I know this is a fair amount of work; It just seems to me that it
has more uses than converting setuptools; i.e., it'd be a useful rig for
anybody doing 2-to-3 porting work.

Yes. That's is exactly what I'm trying to achieve. And I aim to extend
setuptools with build_2to3 and test_2to3 commands to do exactly that.
But to do that I need a good setuptools distribution that supports
Python3. And to do that, I need to make setuptools support Python 3 in
a way that is acceptable to somebody who can merge it to trunk and
make an official release. As far as I know that's you. If I can't find
a solution that is both acceptable to you, and makes Python 3 support
relatively painless, then my efforts have been in vain.

Your comment about setuptools using itself as a good test case did
make me think that this is the case. It explains why the test cases
could run even though setuptools still couldn't install itself. I
thought it was badly tested, now I know it's by design. Removing the
self-dependency then means we have to write a whole bunch of new test
cases.

I do not have the energy to do this. As mentioned I have already put
down a lot of effort into this, and I'm expac...experc... desperate
and have given up. Even writing this email makes me all physically
tired, besides taking a lot of time I should use to make money. I'm
getting too old for this. Somebody else can do it.

Over and out.

[P.J. Eby]

Post by Lennart Regebro

Post by P.J. Eby
python2 setup.py 2to3 test

Well, yes, but it should be
python 3 setup.py 2to3 test
Otherwise it can't reasonably have any idea of which python to use.

Why not? The 2to3 command could simply take an option for the
python3 executable, and be set from the standard config files (e.g. setup.cfg).

Post by Lennart Regebro
And when you run it with python3, the 2to3 command isn't needed, as
the 2to3 step can be automatically inserted through version detection.

Um... I'm still not following you. You sound like you're describing
something very different from what I outlined.

Post by Lennart Regebro
In fact, this is exactly what I'm doing for zope.interface, as
explained. For zope.interface, you run
python2.5 setup.py test
and it will test it with python2.5 You run
python3.0 setup.py test
and it will do exactly what you say: copy the files to build, run
2to3, and then run the test on that in-place.
python3.0 setup.py install
will copy the files to build, run 2to3 and then install it.

Right -- but this is totally *not* what I'm saying. Because the
strategy that I outlined doesn't require setuptools to be ported to
python 3 first.

Post by Lennart Regebro
Yes. That's is exactly what I'm trying to achieve. And I aim to extend
setuptools with build_2to3 and test_2to3 commands to do exactly that.
But to do that I need a good setuptools distribution that supports
Python3.

No, you don't. You only need one command, implemented as a plugin
for the Python2 version of setuptools, and you drive everything from python2.

Once you get to the point where setuptools is ported enough to be
able to have the Python3 version actually run its tests without
crashing, you can go self-hosted. But before that point, you need
Python2 as the bootstrap/host environment for the build process.

Post by Lennart Regebro
It explains why the test cases
could run even though setuptools still couldn't install itself. I
thought it was badly tested, now I know it's by design.

No, it's also badly tested; that's kind of inherited from the
distutils. And by the time I added the 'sandbox' virtualization
tool, there was already tons of stuff that wasn't tested, such that
it didn't seem worth it to do something else.

[P.J. Eby]

Post by Lennart Regebro

Post by P.J. Eby
python2 setup.py 2to3 test

Well, yes, but it should be
python 3 setup.py 2to3 test
Otherwise it can't reasonably have any idea of which python to use.

Why not? The 2to3 command could simply take an option for the python3
executable, and be set from the standard config files (e.g. setup.cfg).

Because that would mean that Python 2 needs to be installed to use
Python 3. It also means all programs that do any sort of installing
need to either know the position of the python 3 executable to use
when installing, and be run with Python 2, or they need to be run with
Python 3 and know the position of a Python 2 interpreter to run
setup.py with.

Er, no. It only means that you need Python 2 to be installed *while
porting a package* to Python 3.

[Lennart Regebro]

Er, no.  It only means that you need Python 2 to be installed *while porting
a package* to Python 3.

No. It means it needs to be installed when installing the package from
a source distribution. Which is the normal way of distributing
modules.

[P.J. Eby]

Post by P.J. Eby
Er, no. It only means that you need Python 2 to be installed

*while porting

Post by P.J. Eby
a package* to Python 3.

No. It means it needs to be installed when installing the package from
a source distribution. Which is the normal way of distributing
modules.

I don't understand you. Here is what I understand:

1. Setuptools requires setuptools
2. Setuptools doesn't run on Python 3 (yet)
3. There needs to be a way to build a Py3 version of setuptools in
order to fix #2

Therefore, adding a new setuptools comand to do #3, that runs under
either Py2 or Py3, fixes #1 in the context of #2.

However, once setuptools *does* run on Python 3, then there is no
longer a need for the build process to run exclusively under...

Aha! Now I (finally) get what you're talking about! In order for
this to work, there'd have to be a separate Py3 source distro for
setuptools, or else setup.py would need to have a (non-setuptools
depending) way to build its own Python3 version.

Okay, now that I actually understand the problem, I will give it some
more thought. I see now that what I was proposing works only for the
porting process and for non-self-dependent packages, but not for
distribution of self-dependent packages like setuptools. Either the
sdist would need to ship with a Python3 version already included (or
have a distinct Py3 sdist), or there'd need to be a
non-setuptools-dependent bootstrap process.

I'll have to think about this one a bit more.

[David Lyon]

Hi All,

Once a package is installed on a system... is there any way to find out
where it was installed to?

I am looking for both metadata and a directory name/path.

In my Package Manager GUI, I want to be able to "reveal" all the
hidden "goodies" that seem to dissapear into the subsystem.

Either "Documentation" or "Demos"

Here is an example....

C:\Python25\Lib\site-packages\win32\Demos

or

C:\Python25\Lib\site-packages\win32com\HTML

Under Linux, it would be just as valuable. If not more.

Regards

David

[Tarek Ziadé]

Post by David Lyon
Hi All,
Once a package is installed on a system... is there any way to find out
where it was installed to?

Not at the moment. PEP 376 though, proposes a new API to be able to reach
the metadata of an installed package, browsing the directories from the PATH
(that's how setuptools does)

Post by David Lyon
I am looking for both metadata and a directory name/path.
In my Package Manager GUI, I want to be able to "reveal" all the
hidden "goodies" that seem to dissapear into the subsystem.
Either "Documentation" or "Demos"
Here is an example....
C:\Python25\Lib\site-packages\win32\Demos
or
C:\Python25\Lib\site-packages\win32com\HTML
Under Linux, it would be just as valuable. If not more.

There's no explicit way at distutils level to point things like documentation,
maybe that could be a metadata.

Maybe you can propose this, by commenting ths page :

http://wiki.python.org/moin/Distutils/Metadata

Regards
TArek

Post by David Lyon
Regards
David
_______________________________________________
http://mail.python.org/mailman/listinfo/distutils-sig

[David Lyon]

Post by Tarek Ziadé
There's no explicit way at distutils level to point things like documentation,

Well I thank you for the answer. At least I have an answer from an expert..

Post by Tarek Ziadé
maybe that could be a metadata.
http://wiki.python.org/moin/Distutils/Metadata

I would like to help but I'm not an expert in that area.

A simple answer is that it would be good if the developer of a package
could:

- Register their documentation file ie index.html etc...

- Register their examples directory

- Register program executeables - (an example - ez_install.exe)
otherwise it's easy for them to get lost.

All other information seems to be available on Pypi in the DOAP record.

Regards David

[Eric Smith]

Post by Lennart Regebro

Post by P.J. Eby
I'm sorry you feel that way, as I've been *trying* to help. I just
still
don't get what the problem is. If I were porting setuptools to
Python 3, I
would *want* it to be circular, even if I had to hack on it a little at
first. So I have a hard time understanding why you don't.

But it CAN NOT be circular under Python 3.

I don't understand why not, doing it may be not entirely trivial but it
should be possible with some trickery. As PJE noted one way to do this
is to explicitly convert setuptools to python 3.x syntax before actually
running setup.py (e.g. his setup3.py file). With some care this could
even be done in setup.py itself.

But even if it's possible to make it circular, is that really a good
design? I think not.

Eric.

[Ronald Oussoren]

Post by Eric Smith

Post by Ronald Oussoren

Post by Lennart Regebro

Post by P.J. Eby
I'm sorry you feel that way, as I've been *trying* to help. I
just still
don't get what the problem is. If I were porting setuptools to
Python 3, I
would *want* it to be circular, even if I had to hack on it a little at
first. So I have a hard time understanding why you don't.

But it CAN NOT be circular under Python 3.

I don't understand why not, doing it may be not entirely trivial
but it should be possible with some trickery. As PJE noted one way
to do this is to explicitly convert setuptools to python 3.x syntax
before actually running setup.py (e.g. his setup3.py file). With
some care this could even be done in setup.py itself.

But even if it's possible to make it circular, is that really a good
design? I think not.

Given what setuptools tries to do I don't think the circular design is
a problem. A standalone distutils would have had the same problem.

In the long run the useful features of setuptools should incorperated
into distutils, instead of monkeypatching distutils. That is, of
course, an entirely different issue and one that's already being
worked on.

Ronald

Post by Eric Smith
Eric.

[David Lyon]

Hi Lennart,

Sorry for my late posting...

I'm working on a gui python package manager on sourceforge...

Post by Lennart Regebro
I have catch 22 problems in finishing the setuptools port to Python 3,
and unless somebody can come up with alternatives, I may rip out
setuptools dependency on itself completely, and only depend on
distutils.

...

It's not easy I agree...

I am a GUI junky myself... :-) I don't use chainsaw (code) unless i need to

I'm trying out all the installers on one machine...

einstaller won't install because I already installed setuptools....

ez_installer won't de-install setuptools...

pip doesn't want to go on either...

I will figure it out... I am sympathising saying this package stuff
is hard going....

Post by Lennart Regebro
I don't have a good solution to this, unless we can drop setuptools
dependency on setuptools completely, and just use plain distutils for
installing and testing setuptools. This will mean no more setuptools
source eggs, but then I don't see the purpose of those, really.
Arguments for not doing this and alternative solutions are highly
welcome.

:-)

me too