Discussion:
Unified DEPENDENCIES concept
Ciaran McCreesh
2012-09-07 11:45:59 UTC
Permalink
Since DEPENDENCIES hasn't been written up in a Gentoo-friendly manner,
and since the Exherbo documentation doesn't seem to suffice to explain
the idea here, here's some more details on the DEPENDENCIES proposal.

We observe that a typical package will have something like this:

DEPEND="
cat/first
cat/second
cat/third
cat/fourth
foo? (
foo/one
=foo/two-2.34[monkeys]
)
bar? (
bar/one
)
baz? (
baz/one
baz/two
)
fnord? (
fnord/one
)"

RDEPEND="
cat/second
cat/third
cat/fourth
cat/fifth
foo? (
=foo/two-2.34[monkeys]
foo/three
)
bar? (
bar/one
bar/two
bar/three
bar/four
)"

Note how many dependencies that we think of as being "the same" are
listed in two places, and are not in any way "the same".

Note also how the foo-related things, the bar-related things etc cannot
be grouped together by their fooness or barness, but are rather grouped
by their DEPENDness and RDEPENDness.

Right now we've just got three variables. In a not too distant EAPI, it
will be considerably more than three. We also have duplication; with
HDEPEND, this will sometimes become triplication.

Here's the important bit, which I shall prefix with some stars:

*** The point of DEPENDENCIES is not to replace n variables with one
*** variable.

*** The point of DEPENDENCIES is not to provide syntactic sugar over a
*** group of n variables.

*** The point of DEPENDENCIES is to allow a package's dependencies,
*** which can be very complicated, to be expressed accurately and
*** simply, in a way friendly to how developers deal with dependencies
*** in practice.

So here's what DEPENDENCIES solves:

Firstly, it allows developers to group together foo-related dependencies
and bar-related dependencies by their fooness and barness, not by their
role.

Secondly, it reduces error-prone duplication.

Thirdly, it avoids some fairly obtuse names (BADEPEND? LDEPEND?) in
favour of something a tad more readable.

Here's the other important bit, with more stars:

*** It does it by replacing the concept of "a package has build
*** dependencies, run dependencies, etc" with "a package has
*** dependencies, and each dependency is applicable at one or more of
*** build time, run tme, etc".

We could just mark each dependency individually by when it's
applicable. But that turns out to be quite verbose, since packages have
fairly long lists of dependencies of the same kind. So here's a much
more concise syntax:

DEPENDENCIES="
build:
cat/first
build+run:
cat/second
cat/third
cat/fourth
run:
cat/fifth

foo? (
build:
foo/one
=foo/two-2.34[monkeys]
run:
foo/three
)

bar? (
build+run:
bar/one
run:
bar/two
bar/three
bar/four
)

build:
baz? (
baz/one
baz/two
)

fnord? (
fnord/one
)"

Here are the rules for working out which labels apply to a given spec:

We have a "labels stack" when parsing. Initially, the value "build+run"
is on the top. Whenever we encounter a block ("( )", "use? ( )" etc),
we duplicate the top value of the stack, and when we leave the block we
pop the value off. Whenever we encounter labels, we replace the top
value on the stack with those labels. Whenever we encounter a spec, its
associated labels are those on the top of the stack.

To avoid weirdness, so long as we're inside a || block, use of labels
is forbidden. Note that this is a reflection of the current ambiguity
as to what DEPEND="|| ( a b )" RDEPEND="|| ( a b )" means (can you
build against a, and run against b?), and if we're wanting to tidy that
up, that should be an independent proposal.

These rules are mostly for flexibility: if developers sometimes prefer
to group things by role rather than by fooness, then that's possible
too.

The rules for eclass merging need changing too, to add a ( ) around
values rather than merely appending. This is a technicality, and isn't
developer visible.
--
Ciaran McCreesh
Michał Górny
2012-09-07 12:29:41 UTC
Permalink
On Fri, 7 Sep 2012 12:45:59 +0100
Post by Ciaran McCreesh
Since DEPENDENCIES hasn't been written up in a Gentoo-friendly manner,
and since the Exherbo documentation doesn't seem to suffice to explain
the idea here, here's some more details on the DEPENDENCIES proposal.
DEPEND="
cat/first
cat/second
cat/third
cat/fourth
foo? (
foo/one
=foo/two-2.34[monkeys]
)
bar? (
bar/one
)
baz? (
baz/one
baz/two
)
fnord? (
fnord/one
)"
RDEPEND="
cat/second
cat/third
cat/fourth
cat/fifth
foo? (
=foo/two-2.34[monkeys]
foo/three
)
bar? (
bar/one
bar/two
bar/three
bar/four
)"
Now please express that using bash variable references instead of
the overly verbose expansions. Also, real examples are much more useful
than counting and monkeys. This is not mathematics.
Post by Ciaran McCreesh
Note how many dependencies that we think of as being "the same" are
listed in two places, and are not in any way "the same".
Note also how the foo-related things, the bar-related things etc
cannot be grouped together by their fooness or barness, but are
rather grouped by their DEPENDness and RDEPENDness.
What is fooness and barness? Because I got lost near the four
non-existent word.
Post by Ciaran McCreesh
Right now we've just got three variables. In a not too distant EAPI,
it will be considerably more than three. We also have duplication;
with HDEPEND, this will sometimes become triplication.
Sometimes != always. Sometimes HDEPEND will actually simplify
the current state. By moving a few things from DEPEND to HDEPEND, we
may be actually able to remove COMMON_DEPEND. Sometimes.
Post by Ciaran McCreesh
*** The point of DEPENDENCIES is not to replace n variables with one
*** variable.
Yes, it is.
Post by Ciaran McCreesh
*** The point of DEPENDENCIES is not to provide syntactic sugar over a
*** group of n variables.
Yes, it is.
Post by Ciaran McCreesh
*** The point of DEPENDENCIES is to allow a package's dependencies,
*** which can be very complicated, to be expressed accurately and
*** simply, in a way friendly to how developers deal with dependencies
*** in practice.
Didn't we already prove that the syntactic sugar DEPENDENCIES allows
can make the dependency specifications very unclear and not anywhere
close to simple?
Post by Ciaran McCreesh
Firstly, it allows developers to group together foo-related
dependencies and bar-related dependencies by their fooness and
barness, not by their role.
Secondly, it reduces error-prone duplication.
No, it doesn't. It just provides an alternate solution over how we do
that now. Does variable substitution ring a bell?
Post by Ciaran McCreesh
Thirdly, it avoids some fairly obtuse names (BADEPEND? LDEPEND?) in
favour of something a tad more readable.
That is a side effect. DEPENDENCIES doesn't disallow us from choosing
to use labels like 'd:', 'r:'. Similarly, bash variables don't anyhow
stand in the way of using more verbose names.
Post by Ciaran McCreesh
*** It does it by replacing the concept of "a package has build
*** dependencies, run dependencies, etc" with "a package has
*** dependencies, and each dependency is applicable at one or more of
*** build time, run tme, etc".
Really?

I think it actually gives concept of 'a package has a few random groups
of dependencies, where each group is applicable at one or more of build
time, run time, etc and consists of one or more dependencies or groups
of dependencies, where each group is applicable at one or more of build
time, run time, etc (which may or may not be the same as parent group)
and consists of one or more dependencies or groups of dependencies,
where each group is applicable at one or more of build time, run time,
etc (which may or may not be the same as parent group) and consists of
one or more dependencies or groups of dependencies, where each group is
applicable at one or more of build time, run time, etc (which may or
may not be the same as parent group) and consists of one or more
dependencies or groups of dependencies, where each group is applicable
at one or more of build time, run time, etc (which may or may not be
the same as parent group) and consists of one or more dependencies or
groups of dependencies, where each group is applicable at one or more
of build time, run time, etc (which may or may not be the same as
parent group) and consists ...'
Post by Ciaran McCreesh
We could just mark each dependency individually by when it's
applicable. But that turns out to be quite verbose, since packages
have fairly long lists of dependencies of the same kind. So here's a
DEPENDENCIES="
cat/first
cat/second
cat/third
cat/fourth
cat/fifth
foo? (
foo/one
=foo/two-2.34[monkeys]
foo/three
)
bar? (
bar/one
bar/two
bar/three
bar/four
)
baz? (
baz/one
baz/two
)
fnord? (
fnord/one
)"
We have a "labels stack" when parsing. Initially, the value
"build+run" is on the top. Whenever we encounter a block ("( )",
"use? ( )" etc), we duplicate the top value of the stack, and when we
leave the block we pop the value off. Whenever we encounter labels,
we replace the top value on the stack with those labels. Whenever we
encounter a spec, its associated labels are those on the top of the
stack.
These are the rules for a machine. People don't actually read
dependencies sequentially. Provide a good algorithm which works from
any position.
Post by Ciaran McCreesh
To avoid weirdness, so long as we're inside a || block, use of labels
is forbidden. Note that this is a reflection of the current ambiguity
as to what DEPEND="|| ( a b )" RDEPEND="|| ( a b )" means (can you
build against a, and run against b?), and if we're wanting to tidy
that up, that should be an independent proposal.
Yes, that's something you could work on instead of throwing a lot of
work on us just to reinvent the wheel your way because you believe that
your bicycle is much better when it has square wheels.
Post by Ciaran McCreesh
These rules are mostly for flexibility: if developers sometimes prefer
to group things by role rather than by fooness, then that's possible
too.
The rules for eclass merging need changing too, to add a ( ) around
values rather than merely appending. This is a technicality, and isn't
developer visible.
The algorithm you presented above was technicality as well.

So, now that your forked the thread shall we repeat all
the disadvantages which you love to forgot to mention?
--
Best regards,
Michał Górny
Ciaran McCreesh
2012-09-07 12:36:05 UTC
Permalink
On Fri, 7 Sep 2012 14:29:41 +0200
Post by Michał Górny
Post by Ciaran McCreesh
*** The point of DEPENDENCIES is not to replace n variables with one
*** variable.
Yes, it is.
You've clearly either completely missed the point of all of this,
despite it being pointed out to you, or are just trying to cause
trouble. Whilst you may be thinking of a DEPENDENCIES proposal that is
about replacing n variables with one variable (and I've never seen such
a proposal being made), what we're discussing here is not that. Kindly
stay out of this discussion until you understand what it is we're
discussing.
Post by Michał Górny
These are the rules for a machine. People don't actually read
dependencies sequentially. Provide a good algorithm which works from
any position.
Read backwards from the current position until you find a label. It's
the same algorithm you use to find out what the current section is when
you're reading an article.
Post by Michał Górny
The algorithm you presented above was technicality as well.
Agreed, but it was explicitly requested by Brian, so I provided it.
Post by Michał Górny
So, now that your forked the thread
...as requested by Brian...
Post by Michał Górny
shall we repeat all the disadvantages which you love to forgot to
mention?
No, please stay out of this until you understand the original proposal,
and in particular the parts marked with ***. So long as you don't
understand what we're discussing, I'm afraid any contributions you make
will simply be shouting and waving.
--
Ciaran McCreesh
Michał Górny
2012-09-07 14:23:16 UTC
Permalink
On Fri, 7 Sep 2012 13:36:05 +0100
Post by Ciaran McCreesh
Post by Michał Górny
These are the rules for a machine. People don't actually read
dependencies sequentially. Provide a good algorithm which works from
any position.
Read backwards from the current position until you find a label. It's
the same algorithm you use to find out what the current section is
when you're reading an article.
No, it doesn't work that way.

See:

foo? (
label:
dev-foo/bar
)
dev-bar/foo <-

Reading backwards, I find 'label'.
Post by Ciaran McCreesh
Post by Michał Górny
shall we repeat all the disadvantages which you love to forgot to
mention?
No, please stay out of this until you understand the original
proposal, and in particular the parts marked with ***. So long as you
don't understand what we're discussing, I'm afraid any contributions
you make will simply be shouting and waving.
What have parts marked with '***' to do with the disadvantages we have
already proved and which you didn't even bother answering?
--
Best regards,
Michał Górny
Ciaran McCreesh
2012-09-07 14:53:50 UTC
Permalink
On Fri, 7 Sep 2012 16:23:16 +0200
Post by Michał Górny
On Fri, 7 Sep 2012 13:36:05 +0100
Post by Ciaran McCreesh
Post by Michał Górny
These are the rules for a machine. People don't actually read
dependencies sequentially. Provide a good algorithm which works
from any position.
Read backwards from the current position until you find a label.
It's the same algorithm you use to find out what the current
section is when you're reading an article.
No, it doesn't work that way.
foo? (
dev-foo/bar
)
dev-bar/foo <-
Reading backwards, I find 'label'.
...skipping subblocks. Although the case you've given is unlikely to
be used in practice, since experience shows that given this feature,
developers use it in an easily understood manner.
Post by Michał Górny
Post by Ciaran McCreesh
Post by Michał Górny
shall we repeat all the disadvantages which you love to forgot to
mention?
No, please stay out of this until you understand the original
proposal, and in particular the parts marked with ***. So long as
you don't understand what we're discussing, I'm afraid any
contributions you make will simply be shouting and waving.
What have parts marked with '***' to do with the disadvantages we have
already proved and which you didn't even bother answering?
The problem is that you're arguing against a proposal that doesn't
exist except in your head. If you'd like to read and understand the
proposal being made, which starts with understanding the bits marked
clearly with stars, and then once you've understood it, rethink and
present any issues you find with that proposal then we might have
something to discuss.
--
Ciaran McCreesh
Michał Górny
2012-09-07 15:02:57 UTC
Permalink
On Fri, 7 Sep 2012 15:53:50 +0100
Post by Ciaran McCreesh
On Fri, 7 Sep 2012 16:23:16 +0200
Post by Michał Górny
On Fri, 7 Sep 2012 13:36:05 +0100
Post by Ciaran McCreesh
Post by Michał Górny
These are the rules for a machine. People don't actually read
dependencies sequentially. Provide a good algorithm which works
from any position.
Read backwards from the current position until you find a label.
It's the same algorithm you use to find out what the current
section is when you're reading an article.
No, it doesn't work that way.
foo? (
dev-foo/bar
)
dev-bar/foo <-
Reading backwards, I find 'label'.
...skipping subblocks. Although the case you've given is unlikely to
be used in practice, since experience shows that given this feature,
developers use it in an easily understood manner.
Post by Michał Górny
Post by Ciaran McCreesh
Post by Michał Górny
shall we repeat all the disadvantages which you love to forgot
to mention?
No, please stay out of this until you understand the original
proposal, and in particular the parts marked with ***. So long as
you don't understand what we're discussing, I'm afraid any
contributions you make will simply be shouting and waving.
What have parts marked with '***' to do with the disadvantages we
have already proved and which you didn't even bother answering?
The problem is that you're arguing against a proposal that doesn't
exist except in your head. If you'd like to read and understand the
proposal being made, which starts with understanding the bits marked
clearly with stars, and then once you've understood it, rethink and
present any issues you find with that proposal then we might have
something to discuss.
Renaming and/or reordering something doesn't change its implications.
It's just 'main disadvantage' vs 'side disadvantage'.
--
Best regards,
Michał Górny
Ciaran McCreesh
2012-09-07 15:07:54 UTC
Permalink
On Fri, 7 Sep 2012 17:02:57 +0200
Post by Michał Górny
Post by Ciaran McCreesh
The problem is that you're arguing against a proposal that doesn't
exist except in your head. If you'd like to read and understand the
proposal being made, which starts with understanding the bits marked
clearly with stars, and then once you've understood it, rethink and
present any issues you find with that proposal then we might have
something to discuss.
Renaming and/or reordering something doesn't change its implications.
It's just 'main disadvantage' vs 'side disadvantage'.
Well no. You've been arguing against some mysterious proposal that
doesn't exist that you think is about replacing lots of variables with
one variable. That's not what DEPENDENCIES is, and now that you've got
a proper write-up, you can try reading it, understanding it and then
seeing if you have any objections to what's actually being proposed.
--
Ciaran McCreesh
Michał Górny
2012-09-07 15:16:21 UTC
Permalink
On Fri, 7 Sep 2012 16:07:54 +0100
Post by Ciaran McCreesh
On Fri, 7 Sep 2012 17:02:57 +0200
Post by Michał Górny
Post by Ciaran McCreesh
The problem is that you're arguing against a proposal that doesn't
exist except in your head. If you'd like to read and understand
the proposal being made, which starts with understanding the bits
marked clearly with stars, and then once you've understood it,
rethink and present any issues you find with that proposal then
we might have something to discuss.
Renaming and/or reordering something doesn't change its
implications. It's just 'main disadvantage' vs 'side disadvantage'.
Well no. You've been arguing against some mysterious proposal that
doesn't exist that you think is about replacing lots of variables with
one variable. That's not what DEPENDENCIES is, and now that you've got
a proper write-up, you can try reading it, understanding it and then
seeing if you have any objections to what's actually being proposed.
Ah, I forgot how the goals change *everything*. Because it's good to
kill hundreds of people for the good reasons.
--
Best regards,
Michał Górny
Wulf C. Krueger
2012-09-07 15:25:55 UTC
Permalink
(Just for the record, I don't care about the merits or demerits of
*DEPEND or DEPENDENCIES.)
Post by Michał Górny
Ah, I forgot how the goals change *everything*. Because it's good
to kill hundreds of people for the good reasons.
You might want to take a short break to consider if you're taking
things a bit too seriously or are at least phrasing them a little
strongly here.
Ian Stakenvicius
2012-09-07 14:50:40 UTC
Permalink
[ Snip! ] Note also how the foo-related things, the bar-related
things etc cannot be grouped together by their fooness or barness,
but are rather grouped by their DEPENDness and RDEPENDness.
[ Snip! ]
Firstly, it allows developers to group together foo-related
dependencies and bar-related dependencies by their fooness and
barness, not by their role. [ Snip! ] *** It does it by replacing
the concept of "a package has build *** dependencies, run
dependencies, etc" with "a package has *** dependencies, and each
dependency is applicable at one or more of *** build time, run tme,
etc".
And this is the specific point that I don't like about DEPENDENCIES
versus *DEPEND. As a developer, I personally find it much more
straight-forward to fill in the deps needed for each role, rather than
specifying the role(s) that each dep will play a part in.

Although I realize that technically I could still do that (have the
dep list be role-centric rather than dep-centric), given that the
point of this change is (as stated above) to organize deps the other
way, I can't really get behind the idea.
Ciaran McCreesh
2012-09-07 14:58:43 UTC
Permalink
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Fri, 07 Sep 2012 10:50:40 -0400
Post by Ian Stakenvicius
[ Snip! ] Note also how the foo-related things, the bar-related
things etc cannot be grouped together by their fooness or barness,
but are rather grouped by their DEPENDness and RDEPENDness.
[ Snip! ]
Firstly, it allows developers to group together foo-related
dependencies and bar-related dependencies by their fooness and
barness, not by their role. [ Snip! ] *** It does it by replacing
the concept of "a package has build *** dependencies, run
dependencies, etc" with "a package has *** dependencies, and each
dependency is applicable at one or more of *** build time, run tme,
etc".
And this is the specific point that I don't like about DEPENDENCIES
versus *DEPEND. As a developer, I personally find it much more
straight-forward to fill in the deps needed for each role, rather than
specifying the role(s) that each dep will play a part in.
Have you tried doing both? You may find you're just arguing from
familiarity, and that after having worked the other way for a few
packages, that the advantages become clearer.

The wide-spread use of hacks like COMMON_DEPEND are a pretty strong
indication that people *do* think in something closer to a
DEPENDENCIES-like fashion. In particular, I find it hard to believe
that you think "ok, so I've got a build dependency upon
Post by Ian Stakenvicius
=cat/pkg-2.3[foo]" and then independently work out "ok, I've got a run
dependency upon >=cat/pkg-2.3[foo]".
- --
Ciaran McCreesh
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iEYEARECAAYFAlBKC6cACgkQ96zL6DUtXhG94gCgmXP0C+nAItSnCTIMoeJHzqzK
AyYAniXFZR5mJr
Alexis Ballier
2012-09-07 15:46:41 UTC
Permalink
I actually do like the concept but I'm not sure we can reach consensus
about '*DEPEND vs DEPENDENCIES'; a possibility to get people used to it
could be to have two parallel EAPIs, like 6 and 6-dependencies, where
the former will keep the old style and the latter use DEPENDENCIES.

After some time has passed, it could be decided to kill the less useful
one, say in EAPI 8, and get only one 'latest' EAPI again. This decision
doesn't need to be left only to the council, but since it affects
everyone it could be a vote from all the dev community.

There is also the possibility that a consensus will never be reached
and that the two styles will have to live forever, but after all, the
EAPI concept is made for this.

Given this possibility, a debate 'I perfer *DEPEND' vs 'DEPENDENCIES are
more flexible' doesn't make much sense now. What should be discussed is
the concept itself.

For example, what is the HDEPEND equivalent for DEPENDENCIES ? exherbo
documentation doesn't seem to mention an equivalent label.

Alexis.
Michał Górny
2012-09-07 16:03:51 UTC
Permalink
On Fri, 7 Sep 2012 12:46:41 -0300
Post by Alexis Ballier
I actually do like the concept but I'm not sure we can reach consensus
about '*DEPEND vs DEPENDENCIES'; a possibility to get people used to
it could be to have two parallel EAPIs, like 6 and 6-dependencies,
where the former will keep the old style and the latter use
DEPENDENCIES.
With eclasses supporting both of them? That's more than crazy.
Post by Alexis Ballier
After some time has passed, it could be decided to kill the less
useful one, say in EAPI 8, and get only one 'latest' EAPI again. This
decision doesn't need to be left only to the council, but since it
affects everyone it could be a vote from all the dev community.
Why the dev community only? We have many active contributors who aren't
devs and who work hard with ebuilds. It's *their* time which will be
wasted on rewriting dependencies into new form, not yours.
Post by Alexis Ballier
There is also the possibility that a consensus will never be reached
and that the two styles will have to live forever, but after all, the
EAPI concept is made for this.
I believe the correct concept is 'fork'. And that's what Exherbo did.
Post by Alexis Ballier
Given this possibility, a debate 'I perfer *DEPEND' vs 'DEPENDENCIES
are more flexible' doesn't make much sense now. What should be
discussed is the concept itself.
For example, what is the HDEPEND equivalent for DEPENDENCIES ? exherbo
documentation doesn't seem to mention an equivalent label.
But Ciaran has just said that this isn't about the labels! It's about
a philosophy in life.

Seriously saying, this isn't about equivalent types. We can choose any
labels we like. It's all about the form in which we will do it.
--
Best regards,
Michał Górny
Ian Stakenvicius
2012-09-07 16:11:27 UTC
Permalink
On Fri, 7 Sep 2012 12:46:41 -0300 Alexis Ballier
Post by Alexis Ballier
I actually do like the concept but I'm not sure we can reach
consensus about '*DEPEND vs DEPENDENCIES'; a possibility to get
people used to it could be to have two parallel EAPIs, like 6 and
6-dependencies, where the former will keep the old style and the
latter use DEPENDENCIES.
With eclasses supporting both of them? That's more than crazy.
By the time EAPI=6* would happen we should have a git tree so we'd
just fork the tree for EAPI=6-dependencies. :)

A change like this would *NEED* a long-term test phase with a lot of
developer participation, so if we were to consider it we'd need to
fork the tree and implement/maintain it in parallel to main tree
maintenance, imo. A translation script could probably also be used to
convert *DEPEND into DEPENDENCIES automatically for any packages that
dev's aren't manually managing...

...But I digress; we're nowhere near doing this yet.
Michael Mol
2012-09-07 16:28:31 UTC
Permalink
Post by Ciaran McCreesh
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
On Fri, 7 Sep 2012 12:46:41 -0300 Alexis Ballier
Post by Alexis Ballier
I actually do like the concept but I'm not sure we can reach
consensus about '*DEPEND vs DEPENDENCIES'; a possibility to get
people used to it could be to have two parallel EAPIs, like 6 and
6-dependencies, where the former will keep the old style and the
latter use DEPENDENCIES.
With eclasses supporting both of them? That's more than crazy.
By the time EAPI=6* would happen we should have a git tree so we'd
just fork the tree for EAPI=6-dependencies. :)
A change like this would *NEED* a long-term test phase with a lot of
developer participation, so if we were to consider it we'd need to
fork the tree and implement/maintain it in parallel to main tree
maintenance, imo. A translation script could probably also be used to
convert *DEPEND into DEPENDENCIES automatically for any packages that
dev's aren't manually managing...
...But I digress; we're nowhere near doing this yet.
An intermediate form of that might be useful for auditing the tree and
finding packages which aren't expressing, e.g. RDEPENDS, but probably
should.
--
:wq
Ciaran McCreesh
2012-09-07 16:34:47 UTC
Permalink
On Fri, 7 Sep 2012 12:28:31 -0400
Post by Michael Mol
An intermediate form of that might be useful for auditing the tree and
finding packages which aren't expressing, e.g. RDEPENDS, but probably
should.
RDEPEND=DEPEND was removed in EAPI 4, if that's what you mean.
--
Ciaran McCreesh
Paweł Hajdan, Jr.
2012-09-07 16:40:47 UTC
Permalink
Post by Michał Górny
Why the dev community only? We have many active contributors who aren't
devs and who work hard with ebuilds. It's *their* time which will be
wasted on rewriting dependencies into new form, not yours.
Should those contributors also vote? Do they have any formal status? Do
we have established procedures for that case?

By the way, for everyone else, there is a forums thread about this topic
here:
<http://forums.gentoo.org/viewtopic-t-935812-start-0-postdays-0-postorder-asc-highlight-.html>

I guess the contributors follow gentoo-dev, so they at least can voice
their concerns, if any.

Also, requiring a rewrite of all existing ebuilds doesn't sound like a
good idea. I think this should be designed not to require a rewrite, and
then the concern about wasted time disappears.
Ciaran McCreesh
2012-09-07 16:47:31 UTC
Permalink
On Fri, 07 Sep 2012 18:40:47 +0200
Post by Paweł Hajdan, Jr.
Also, requiring a rewrite of all existing ebuilds doesn't sound like a
good idea. I think this should be designed not to require a rewrite,
and then the concern about wasted time disappears.
Uh, there is no "rewriting all existing ebuilds" anywhere. I've no idea
where you got that from...
--
Ciaran McCreesh
Alexis Ballier
2012-09-07 17:40:25 UTC
Permalink
On Fri, 7 Sep 2012 18:03:51 +0200
Post by Michał Górny
On Fri, 7 Sep 2012 12:46:41 -0300
Post by Alexis Ballier
I actually do like the concept but I'm not sure we can reach
consensus about '*DEPEND vs DEPENDENCIES'; a possibility to get
people used to it could be to have two parallel EAPIs, like 6 and
6-dependencies, where the former will keep the old style and the
latter use DEPENDENCIES.
With eclasses supporting both of them? That's more than crazy.
depstr=cat/foo

case $EAPI in
*-dependencies) DEPENDENCIES="build+run: $depstr";;
*) DEPEND="$depstr"
RDEPEND="$depstr";;
esac


Yes, eclasses supporting more than one EAPI is crazy and we should
create a new eclass for every EAPI :)
Post by Michał Górny
Post by Alexis Ballier
After some time has passed, it could be decided to kill the less
useful one, say in EAPI 8, and get only one 'latest' EAPI again.
This decision doesn't need to be left only to the council, but
since it affects everyone it could be a vote from all the dev
community.
Why the dev community only? We have many active contributors who
aren't devs and who work hard with ebuilds. It's *their* time which
will be wasted on rewriting dependencies into new form, not yours.
It seems we have a different definition of 'dev community'. That's true
we have well established voting procedures for gentoo devs or foundation
members, but feel free to propose one for the rest of contributors.
Post by Michał Górny
Post by Alexis Ballier
There is also the possibility that a consensus will never be reached
and that the two styles will have to live forever, but after all,
the EAPI concept is made for this.
I believe the correct concept is 'fork'. And that's what Exherbo did.
An EAPI is a fork of the ebuild API already. Exherbo is not a fork as
far as I know, or at least not more than Gentoo is a Redhat fork
because it can process rpm's.

A.
Michał Górny
2012-09-07 18:21:03 UTC
Permalink
On Fri, 7 Sep 2012 14:40:25 -0300
Post by Alexis Ballier
On Fri, 7 Sep 2012 18:03:51 +0200
Post by Michał Górny
On Fri, 7 Sep 2012 12:46:41 -0300
Post by Alexis Ballier
I actually do like the concept but I'm not sure we can reach
consensus about '*DEPEND vs DEPENDENCIES'; a possibility to get
people used to it could be to have two parallel EAPIs, like 6 and
6-dependencies, where the former will keep the old style and the
latter use DEPENDENCIES.
With eclasses supporting both of them? That's more than crazy.
depstr=cat/foo
case $EAPI in
*-dependencies) DEPENDENCIES="build+run: $depstr";;
*) DEPEND="$depstr"
RDEPEND="$depstr";;
esac
Yes, we have many eclasses where this is actually the only expected
result. Maybe start with python.eclass, that should be quite an extreme
example.
--
Best regards,
Michał Górny
Alexis Ballier
2012-09-07 19:59:48 UTC
Permalink
On Fri, 7 Sep 2012 20:21:03 +0200
Post by Michał Górny
On Fri, 7 Sep 2012 14:40:25 -0300
Post by Alexis Ballier
On Fri, 7 Sep 2012 18:03:51 +0200
Post by Michał Górny
On Fri, 7 Sep 2012 12:46:41 -0300
Post by Alexis Ballier
I actually do like the concept but I'm not sure we can reach
consensus about '*DEPEND vs DEPENDENCIES'; a possibility to get
people used to it could be to have two parallel EAPIs, like 6
and 6-dependencies, where the former will keep the old style
and the latter use DEPENDENCIES.
With eclasses supporting both of them? That's more than crazy.
depstr=cat/foo
case $EAPI in
*-dependencies) DEPENDENCIES="build+run: $depstr";;
*) DEPEND="$depstr"
RDEPEND="$depstr";;
esac
Yes, we have many eclasses where this is actually the only expected
result. Maybe start with python.eclass, that should be quite an
extreme example.
Reference needed. You probably didn't even think more than 2 seconds
before making this claim about python.eclass, because it is not
particularly hard.
Michał Górny
2012-09-07 20:10:51 UTC
Permalink
On Fri, 7 Sep 2012 16:59:48 -0300
Post by Alexis Ballier
On Fri, 7 Sep 2012 20:21:03 +0200
Post by Michał Górny
On Fri, 7 Sep 2012 14:40:25 -0300
Post by Alexis Ballier
On Fri, 7 Sep 2012 18:03:51 +0200
Post by Michał Górny
On Fri, 7 Sep 2012 12:46:41 -0300
Post by Alexis Ballier
I actually do like the concept but I'm not sure we can reach
consensus about '*DEPEND vs DEPENDENCIES'; a possibility to
get people used to it could be to have two parallel EAPIs,
like 6 and 6-dependencies, where the former will keep the old
style and the latter use DEPENDENCIES.
With eclasses supporting both of them? That's more than crazy.
depstr=cat/foo
case $EAPI in
*-dependencies) DEPENDENCIES="build+run: $depstr";;
*) DEPEND="$depstr"
RDEPEND="$depstr";;
esac
Yes, we have many eclasses where this is actually the only expected
result. Maybe start with python.eclass, that should be quite an
extreme example.
Reference needed. You probably didn't even think more than 2 seconds
before making this claim about python.eclass, because it is not
particularly hard.
Hmm, didn't it used to support having python as DEPEND only?

In any case, I'm thinking more of that line. Eclasses which sometimes
add RDEP+DEP, sometimes DEP only, and sometimes do even crazier things.
--
Best regards,
Michał Górny
Ian Stakenvicius
2012-09-07 20:14:17 UTC
Permalink
On Fri, 7 Sep 2012 16:59:48 -0300 Alexis Ballier
On Fri, 7 Sep 2012 20:21:03 +0200 Michał Górny
On Fri, 7 Sep 2012 14:40:25 -0300 Alexis Ballier
On Fri, 7 Sep 2012 18:03:51 +0200 Michał Górny
On Fri, 7 Sep 2012 12:46:41 -0300 Alexis Ballier
Post by Alexis Ballier
I actually do like the concept but I'm not sure we can
reach consensus about '*DEPEND vs DEPENDENCIES'; a
possibility to get people used to it could be to have two
parallel EAPIs, like 6 and 6-dependencies, where the
former will keep the old style and the latter use
DEPENDENCIES.
With eclasses supporting both of them? That's more than
crazy.
depstr=cat/foo
$depstr";; *) DEPEND="$depstr" RDEPEND="$depstr";; esac
Yes, we have many eclasses where this is actually the only
expected result. Maybe start with python.eclass, that should be
quite an extreme example.
Reference needed. You probably didn't even think more than 2
seconds before making this claim about python.eclass, because it
is not particularly hard.
Hmm, didn't it used to support having python as DEPEND only?
In any case, I'm thinking more of that line. Eclasses which
sometimes add RDEP+DEP, sometimes DEP only, and sometimes do even
crazier things.
Is there anything in particular in the spec/proposal for DEPENDENCIES
that would exclude the addition of individual "build: app-cat/myatom"
"run: app-cat/myatom" deps by an eclass or eclasses? I know the
"goal" here is to make things atom-centric, but I can't see an
implementation ever working of this that wouldn't permit the "pile-on"
of additional entries of different (or even the same) roles on
identical or near-identical atoms.
Brian Harring
2012-09-11 02:16:17 UTC
Permalink
Post by Ian Stakenvicius
Is there anything in particular in the spec/proposal for DEPENDENCIES
that would exclude the addition of individual "build: app-cat/myatom"
"run: app-cat/myatom" deps by an eclass or eclasses? I know the
"goal" here is to make things atom-centric, but I can't see an
implementation ever working of this that wouldn't permit the "pile-on"
of additional entries of different (or even the same) roles on
identical or near-identical atoms.
They could be piled on; it would require each eclass to reset the
label for safety reasons though; same goes for ebuilds frankly (or the
PM would have to reset the context to build+run: each time through).

Pardon if addressed elsewhere; this thread is a fucking mess...
~harring
Kent Fredric
2012-09-13 19:18:54 UTC
Permalink
Post by Brian Harring
Post by Ian Stakenvicius
Is there anything in particular in the spec/proposal for DEPENDENCIES
that would exclude the addition of individual "build: app-cat/myatom"
"run: app-cat/myatom" deps by an eclass or eclasses? I know the
"goal" here is to make things atom-centric, but I can't see an
implementation ever working of this that wouldn't permit the "pile-on"
of additional entries of different (or even the same) roles on
identical or near-identical atoms.
They could be piled on; it would require each eclass to reset the
label for safety reasons though; same goes for ebuilds frankly (or the
PM would have to reset the context to build+run: each time through).
Pardon if addressed elsewhere; this thread is a fucking mess...
~harring
Correct me if I'm wrong, but couldn't the entire proposition could be
implemented in an eclass, not needing the EAPI development cycle to be
tied up with it.

All you need is something in bash that can parse DEPENDENCIES and
populate *DEPEND , and the underlying guts could be done in
practically any language without requiring PM specific
implementations.

[[[

inherit polydep;

DEPENDENCIES="
Stuff Here.
";

]]]

The only thing I know of that is limiting the above from being
implemented that way is the lack of post-source eclass hooks, that is:
currently you'd have to do either

[[[

DEPENDENCIES="..."
inherit polydep;

]]]

or use a callback

[[[

inherit polydep;
DEPENDENCIES=" ... "

polydeps;

]]]
--
Kent

perl -e "print substr( \"edrgmaM SPA NOcomil.ic\\@tfrken\", \$_ * 3,
3 ) for ( 9,8,0,7,1,6,5,4,3,2 );"

http://kent-fredric.fox.geek.nz
Brian Harring
2012-09-13 22:17:32 UTC
Permalink
Post by Kent Fredric
Post by Brian Harring
Post by Ian Stakenvicius
Is there anything in particular in the spec/proposal for DEPENDENCIES
that would exclude the addition of individual "build: app-cat/myatom"
"run: app-cat/myatom" deps by an eclass or eclasses? I know the
"goal" here is to make things atom-centric, but I can't see an
implementation ever working of this that wouldn't permit the "pile-on"
of additional entries of different (or even the same) roles on
identical or near-identical atoms.
They could be piled on; it would require each eclass to reset the
label for safety reasons though; same goes for ebuilds frankly (or the
PM would have to reset the context to build+run: each time through).
Pardon if addressed elsewhere; this thread is a fucking mess...
~harring
Correct me if I'm wrong, but couldn't the entire proposition could be
implemented in an eclass, not needing the EAPI development cycle to be
tied up with it.
All you need is something in bash that can parse DEPENDENCIES and
populate *DEPEND , and the underlying guts could be done in
practically any language without requiring PM specific
implementations.
You've got it inverted; if any autopopulation is occuring, *DEPEND ->
DEPENDENCIES is the sane form.

While it definitely *is* possible to render DEPENDENCIES down into
depend/rdepend (after all, the PM has to do exactly this for
resolution), that does /not/ mean doing it in bash is a good idea.

I'd really not want to try that using labels; using use conditionals
('dep:run,build? ( targets )') is frankly a bit easier imo, but still;
why do so unless one likes pain? It doesn't actually gain us
anything via missing the point of DEPENDENCIES.

The point of unified DEPENDENCIES var (regardless of the form) is
thus:
1) ability to specify common deps once, w/out having to use
intermediate vars/copy-pasting/etc. Think COMMON_DEPEND, and this
should make sense.

2) To shift to a form where adding new dependency targets is easy-
whether it be sdepend, fdepend, tdepend, or hdepend (or
ONE-RING-DEPEND to rule them all). This actually is rather important;
for the average 95% case, devs won't actually have to pay much
attention to those vars; but for those of us a bit further out (cross
compilation, heavy parallelization, etc) those depend forms are
becoming increasingly painful in their absense.


Basically, having devs specify DEPENDENCIES in ebuilds, which then an
eclass chunks out into DEPEND/RDEPEND misses the point of this; it's
doable, it's just not particularly sane imo.

The other way around, having *DEPEND automatically be collapsed into
DEPENDENCIES, however is very sane- it makes transition/compatibilty
for devs bloody simple, while structuring it so we can do further
enhancements.

~harring
Kent Fredric
2012-09-15 11:06:01 UTC
Permalink
Post by Brian Harring
Post by Kent Fredric
All you need is something in bash that can parse DEPENDENCIES and
populate *DEPEND , and the underlying guts could be done in
practically any language without requiring PM specific
implementations.
You've got it inverted; if any autopopulation is occuring, *DEPEND ->
DEPENDENCIES is the sane form.
While it definitely *is* possible to render DEPENDENCIES down into
depend/rdepend (after all, the PM has to do exactly this for
resolution), that does /not/ mean doing it in bash is a good idea.
I'd really not want to try that using labels; using use conditionals
('dep:run,build? ( targets )') is frankly a bit easier imo, but still;
why do so unless one likes pain? It doesn't actually gain us
anything via missing the point of DEPENDENCIES.
The point of unified DEPENDENCIES var (regardless of the form) is
1) ability to specify common deps once, w/out having to use
intermediate vars/copy-pasting/etc. Think COMMON_DEPEND, and this
should make sense.
2) To shift to a form where adding new dependency targets is easy-
whether it be sdepend, fdepend, tdepend, or hdepend (or
ONE-RING-DEPEND to rule them all). This actually is rather important;
for the average 95% case, devs won't actually have to pay much
attention to those vars; but for those of us a bit further out (cross
compilation, heavy parallelization, etc) those depend forms are
becoming increasingly painful in their absense.
Basically, having devs specify DEPENDENCIES in ebuilds, which then an
eclass chunks out into DEPEND/RDEPEND misses the point of this; it's
doable, it's just not particularly sane imo.
The other way around, having *DEPEND automatically be collapsed into
DEPENDENCIES, however is very sane- it makes transition/compatibilty
for devs bloody simple, while structuring it so we can do further
enhancements.
~harring
Sure, but at least this makes it a viable proof-of-concept without
needing all the different PM's to implement the new spec first, and
due to not being EAPI bound when done this way, means you can just do
it and have it work both now and in the future.

And because of this "experimental" nature, you don't have to do *ALL*
the parsing in bash, you could make the eclass use some external code
to parse it and spit it out, and simply have the eclass depend on that
external program regardless.

I agree that long term, a Unified DEPENDENCIES implementation is the
way forward, but if you want to convince people, having something
which has been demonstrated and tested in a real world setting goes a
long way.
--
Kent

perl -e "print substr( \"edrgmaM SPA NOcomil.ic\\@tfrken\", \$_ * 3,
3 ) for ( 9,8,0,7,1,6,5,4,3,2 );"

http://kent-fredric.fox.geek.nz
Brian Harring
2012-09-15 20:33:18 UTC
Permalink
Post by Kent Fredric
Post by Brian Harring
Post by Kent Fredric
All you need is something in bash that can parse DEPENDENCIES and
populate *DEPEND , and the underlying guts could be done in
practically any language without requiring PM specific
implementations.
You've got it inverted; if any autopopulation is occuring, *DEPEND ->
DEPENDENCIES is the sane form.
While it definitely *is* possible to render DEPENDENCIES down into
depend/rdepend (after all, the PM has to do exactly this for
resolution), that does /not/ mean doing it in bash is a good idea.
I'd really not want to try that using labels; using use conditionals
('dep:run,build? ( targets )') is frankly a bit easier imo, but still;
why do so unless one likes pain? It doesn't actually gain us
anything via missing the point of DEPENDENCIES.
The point of unified DEPENDENCIES var (regardless of the form) is
1) ability to specify common deps once, w/out having to use
intermediate vars/copy-pasting/etc. Think COMMON_DEPEND, and this
should make sense.
2) To shift to a form where adding new dependency targets is easy-
whether it be sdepend, fdepend, tdepend, or hdepend (or
ONE-RING-DEPEND to rule them all). This actually is rather important;
for the average 95% case, devs won't actually have to pay much
attention to those vars; but for those of us a bit further out (cross
compilation, heavy parallelization, etc) those depend forms are
becoming increasingly painful in their absense.
Basically, having devs specify DEPENDENCIES in ebuilds, which then an
eclass chunks out into DEPEND/RDEPEND misses the point of this; it's
doable, it's just not particularly sane imo.
The other way around, having *DEPEND automatically be collapsed into
DEPENDENCIES, however is very sane- it makes transition/compatibilty
for devs bloody simple, while structuring it so we can do further
enhancements.
~harring
Sure, but at least this makes it a viable proof-of-concept without
needing all the different PM's to implement the new spec first, and
due to not being EAPI bound when done this way, means you can just do
it and have it work both now and in the future.
And because of this "experimental" nature, you don't have to do *ALL*
the parsing in bash, you could make the eclass use some external code
to parse it and spit it out, and simply have the eclass depend on that
external program regardless.
I agree that long term, a Unified DEPENDENCIES implementation is the
way forward, but if you want to convince people, having something
which has been demonstrated and tested in a real world setting goes a
long way.
Honestly, QA would be well within their rights to kick anyone who did
this, *hard* in the shins.

I understand your notion- specifically proof of concept, show the
data, etc; I just think you've still got inverted, too focused on
trying to do it in bash.

To demonstrate the gain of this, we basically take the existing tree's
deps, and re-render it into a unified DEPENDENCIES form.

As for adding support to a PM, if we use the use conditional proposal
of mine, it's bloody simple- the PM already supports it, we just need
some minor USE_EXPAND adjustments.

~harring
Michał Górny
2012-09-15 22:03:36 UTC
Permalink
On Sat, 15 Sep 2012 13:33:18 -0700
Post by Brian Harring
Post by Kent Fredric
Post by Brian Harring
Post by Kent Fredric
All you need is something in bash that can parse DEPENDENCIES and
populate *DEPEND , and the underlying guts could be done in
practically any language without requiring PM specific
implementations.
You've got it inverted; if any autopopulation is occuring,
*DEPEND -> DEPENDENCIES is the sane form.
While it definitely *is* possible to render DEPENDENCIES down into
depend/rdepend (after all, the PM has to do exactly this for
resolution), that does /not/ mean doing it in bash is a good idea.
I'd really not want to try that using labels; using use
conditionals ('dep:run,build? ( targets )') is frankly a bit
easier imo, but still; why do so unless one likes pain? It
doesn't actually gain us anything via missing the point of
DEPENDENCIES.
The point of unified DEPENDENCIES var (regardless of the form) is
1) ability to specify common deps once, w/out having to use
intermediate vars/copy-pasting/etc. Think COMMON_DEPEND, and this
should make sense.
2) To shift to a form where adding new dependency targets is easy-
whether it be sdepend, fdepend, tdepend, or hdepend (or
ONE-RING-DEPEND to rule them all). This actually is rather
important; for the average 95% case, devs won't actually have to
pay much attention to those vars; but for those of us a bit
further out (cross compilation, heavy parallelization, etc) those
depend forms are becoming increasingly painful in their absense.
Basically, having devs specify DEPENDENCIES in ebuilds, which
then an eclass chunks out into DEPEND/RDEPEND misses the point of
this; it's doable, it's just not particularly sane imo.
The other way around, having *DEPEND automatically be collapsed
into DEPENDENCIES, however is very sane- it makes
transition/compatibilty for devs bloody simple, while structuring
it so we can do further enhancements.
~harring
Sure, but at least this makes it a viable proof-of-concept without
needing all the different PM's to implement the new spec first, and
due to not being EAPI bound when done this way, means you can just
do it and have it work both now and in the future.
And because of this "experimental" nature, you don't have to do
*ALL* the parsing in bash, you could make the eclass use some
external code to parse it and spit it out, and simply have the
eclass depend on that external program regardless.
I agree that long term, a Unified DEPENDENCIES implementation is the
way forward, but if you want to convince people, having something
which has been demonstrated and tested in a real world setting goes
a long way.
Honestly, QA would be well within their rights to kick anyone who did
this, *hard* in the shins.
I understand your notion- specifically proof of concept, show the
data, etc; I just think you've still got inverted, too focused on
trying to do it in bash.
To demonstrate the gain of this, we basically take the existing
tree's deps, and re-render it into a unified DEPENDENCIES form.
But in order to do this, we first have to decide exactly what kind
of dependencies do we want to have. Then convert the tree to
a separate-variable form with new dependencies. Then we can compare
it with the DEPENDENCIES form and decide which one is better.
--
Best regards,
Michał Górny
Brian Harring
2012-09-16 01:20:26 UTC
Permalink
Post by Michał Górny
On Sat, 15 Sep 2012 13:33:18 -0700
Post by Brian Harring
To demonstrate the gain of this, we basically take the existing
tree's deps, and re-render it into a unified DEPENDENCIES form.
But in order to do this, we first have to decide exactly what kind
of dependencies do we want to have. Then convert the tree to
a separate-variable form with new dependencies. Then we can compare
it with the DEPENDENCIES form and decide which one is better.
Funny you mentioned that, I just finished tweaking pquery to generate
real world example unified dependencies; these *are* accurate, just to
be clear.

Dumps are at
http://dev.gentoo.org/~ferringb/unified-dependencies-example/ .

Herds, if you want to see what your pkgs would look like, look at
http://dev.gentoo.org/~ferringb/unified-dependencies-example/herds/ .

If you'd like to see an *example effect* it has on what gets displayed
to the user (aka, after all major use conditionals are stripped), look
at
http://dev.gentoo.org/~ferringb/unified-dependencies-example/user-visible.txt
; warning, that's a 55MB file. The syntax in use there isn't great,
but as said, it's an example.

Total cache savings from doing this for a full tree conversion, for
our existing md5-cache format is 2.73MB (90 byes per cache entry).
Calculating the savings from the ebuild/eclass standpoint is dependent
on how the deps are built up, so I skipped that.

The algorithim used is fairly stupid, but reasonably effectively;
essentially it intersects the top level of each individual type of
dep, breaking out common groupings.

In other words, it won't pick up this:
DEPEND="x? ( dev-util/diffball dev-util/bsdiff )"
RDEPEND="x? ( dev-util/diffball )"

and convert it into thus
DEPENDENCIES="
dep:build,run? (
x? (
dev-util/diffball
dep:run? (
dev-util/diffball
)
)
)"

Additionally, the form used here makes *no assumption about default
context*; in any final solution we use, a default context would be
wise- say build,run. Again, an example of what I mean.

If we said "in the absense of a context, the default is dep:build,run"
the following:

DEPEND="dev-util/diffball dev-util/bsdiff"
RDEPEND="dev-util/diffball de-vutil/bsdiff x? ( sys-apps/pkgcore )"
PDEPEND="dev-python/snakeoil"

would be:
DEPENDENCIES="
dev-util/diffball
dev-util/bsdiff
dep:run? ( x? ( sys-apps/pkgcore ) )
dep:post? ( dev-python/snakeoil )
"

The quicky algo I used assumes no default context, thus it writes
this:
DEPENDENCIES="
dep:build,run? (
dev-util/diffball
dev-util/bsdiff
)
dep:run? ( x? ( sys-apps/pkgcore ) )
dep:post? ( dev-python/snakeoil )
"

Etc.

~harring
Diego Elio Pettenò
2012-09-16 02:39:19 UTC
Permalink
Post by Brian Harring
Herds, if you want to see what your pkgs would look like, look at
http://dev.gentoo.org/~ferringb/unified-dependencies-example/herds/ .
Ruby team could make use of a dep:test and automatic conversion of that :P
--
Diego Elio Pettenò — Flameeyes
***@flameeyes.eu — http://blog.flameeyes.eu/
Ben de Groot
2012-09-16 07:39:22 UTC
Permalink
Post by Brian Harring
Post by Michał Górny
On Sat, 15 Sep 2012 13:33:18 -0700
Post by Brian Harring
To demonstrate the gain of this, we basically take the existing
tree's deps, and re-render it into a unified DEPENDENCIES form.
But in order to do this, we first have to decide exactly what kind
of dependencies do we want to have. Then convert the tree to
a separate-variable form with new dependencies. Then we can compare
it with the DEPENDENCIES form and decide which one is better.
Funny you mentioned that, I just finished tweaking pquery to generate
real world example unified dependencies; these *are* accurate, just to
be clear.
Dumps are at
http://dev.gentoo.org/~ferringb/unified-dependencies-example/ .
Herds, if you want to see what your pkgs would look like, look at
http://dev.gentoo.org/~ferringb/unified-dependencies-example/herds/ .
If you'd like to see an *example effect* it has on what gets displayed
to the user (aka, after all major use conditionals are stripped), look
at
http://dev.gentoo.org/~ferringb/unified-dependencies-example/user-visible.txt
; warning, that's a 55MB file. The syntax in use there isn't great,
but as said, it's an example.
Total cache savings from doing this for a full tree conversion, for
our existing md5-cache format is 2.73MB (90 byes per cache entry).
Calculating the savings from the ebuild/eclass standpoint is dependent
on how the deps are built up, so I skipped that.
The algorithim used is fairly stupid, but reasonably effectively;
essentially it intersects the top level of each individual type of
dep, breaking out common groupings.
DEPEND="x? ( dev-util/diffball dev-util/bsdiff )"
RDEPEND="x? ( dev-util/diffball )"
and convert it into thus
DEPENDENCIES="
dep:build,run? (
x? (
dev-util/diffball
dep:run? (
dev-util/diffball
)
)
)"
Additionally, the form used here makes *no assumption about default
context*; in any final solution we use, a default context would be
wise- say build,run. Again, an example of what I mean.
If we said "in the absense of a context, the default is dep:build,run"
DEPEND="dev-util/diffball dev-util/bsdiff"
RDEPEND="dev-util/diffball de-vutil/bsdiff x? ( sys-apps/pkgcore )"
PDEPEND="dev-python/snakeoil"
DEPENDENCIES="
dev-util/diffball
dev-util/bsdiff
dep:run? ( x? ( sys-apps/pkgcore ) )
dep:post? ( dev-python/snakeoil )
"
The quicky algo I used assumes no default context, thus it writes
DEPENDENCIES="
dep:build,run? (
dev-util/diffball
dev-util/bsdiff
)
dep:run? ( x? ( sys-apps/pkgcore ) )
dep:post? ( dev-python/snakeoil )
"
Etc.
~harring
Thanks. I have given it a quick overview for the qt herd. I still
don't see what using DEPENDENCIES adds to what we do now with separate
*DEPEND variables. I see no convincing reason to change what we do.

As I've said before on IRC, we need a good costs/benefits overview.
Right now I only see costs (migrating ebuilds and eclasses) and no
benefits.
--
Cheers,

Ben | yngwin
Gentoo developer
Gentoo Qt project lead, Gentoo Wiki admin
Brian Harring
2012-09-16 13:15:01 UTC
Permalink
Post by Ben de Groot
Post by Brian Harring
Dumps are at
http://dev.gentoo.org/~ferringb/unified-dependencies-example/ .
Herds, if you want to see what your pkgs would look like, look at
http://dev.gentoo.org/~ferringb/unified-dependencies-example/herds/ .
If you'd like to see an *example effect* it has on what gets displayed
to the user (aka, after all major use conditionals are stripped), look
at
http://dev.gentoo.org/~ferringb/unified-dependencies-example/user-visible.txt
; warning, that's a 55MB file. The syntax in use there isn't great,
but as said, it's an example.
...
Additionally, the form used here makes *no assumption about default
context*; in any final solution we use, a default context would be
wise- say build,run. Again, an example of what I mean.
The dumps were regenerated; a default context of 'build,run' was
added. Basically in the absense of an explicit dep: targetting,
dep:build,run is assumed.

Essentially it makes the deps cleaner to read for the common case,
while also reducing the footprint at the cache, and vdb level; see
http://dev.gentoo.org/~ferringb/unified-dependencies-example/vdb-effect.txt
Post by Ben de Groot
Thanks. I have given it a quick overview for the qt herd. I still
don't see what using DEPENDENCIES adds to what we do now with separate
*DEPEND variables. I see no convincing reason to change what we do.
Ok, so here's some stats: in the tree, we have 31360 ebuilds, and 194
eclasses; grand total of 31554 sources of metadata content (I say
metadata since vapier has eblits in use which are just phase
functions- I'm not scanning those intentionally).

Doing some simple scans of the tree, here's some stats; note these
stats are duplicated in the glep (they're nice selling points, thus
might as well):

1) 746 hits in the tree for COMMON_DEPEND; that's 2%, and the usages
I'm aware of have been for literally, what it sounds like- depends
that are both DEPEND and RDEPEND.

2) scanning for assignments of RDEPEND=.*|$\{?DEPEND\}? gets a hit on
5343 unique sources. Searching for the inverse gets a hit on 10008
unique sources. Meaning that right there, ~48.6% of the tree is
duplicating deps between the two forms. This puts us to 16083 unique
sources in the tree that would benefit in some form (~51%).

3) What's interesting about that 51% is the eapi groupings; in EAPI4,
the autosetting of RDEPEND="${RDEPEND:-${DEPEND}}" was discontinued.
Roughly 50% of the initial 51% match is EAPI4; the rest are eapi0-3.

4) Again, keep in mind that the grep's in use above are single line
matches- I'm definitely missing some ebuilds, and for complex
dependencies that are appended/set by the eclass, likely missing a lot
of that too.

So... basically, people are already doing this manually with their own
intermediate vars.

Just a rough but mildly entertaining stat there's basically 8.38MB
worth of normalized, literal fricking dependency; using a crappy algo
(rather than a human doing the deps who can do it better), 37.4%
(3.1MB) of that is removed via going to dependencies. It goes
without saying that this would be a helluva lot less torturous on a
proper PM implementation that parses the tree once, and renders- let
alone repoman being able to avoid repeat scans of things it already
has examined.

Mind you, portage doesn't do that, but this would be good incentive to
do a proper tree. :)
Post by Ben de Groot
As I've said before on IRC, we need a good costs/benefits overview.
Right now I only see costs (migrating ebuilds and eclasses) and no
benefits.
Offhand... I wouldn't be pushing for this if I didn't think this would
be a boon over all- both for devs, and PMs.

I think the actual 'cost' probably got lost in the noise of the
various flames; what I'm proposing is basically zero cost for devs, it
shoves the work to the PM, leaving the option for devs to do a more
fine grained form if they can.

I'm starting a seperate thread w/ a glep for this; I think you should
take a look at the exact details of the specification including how
DEPEND/RDEPEND/PDEPEND are handled in parallel to DEPENDENCIES (short
version: if existent, they're automatically folded into DEPENDENCIES
in my proposal); the end result is basically zero pain transition,
while enabling us to start getting gains.

Cheers-
~brian
Matt Turner
2012-09-18 22:51:21 UTC
Permalink
Post by Brian Harring
1) 746 hits in the tree for COMMON_DEPEND; that's 2%, and the usages
I'm aware of have been for literally, what it sounds like- depends
that are both DEPEND and RDEPEND.
CDEPEND is pretty common as well. I could 466 files with CDEPEND.
Ben de Groot
2012-09-19 04:22:06 UTC
Permalink
Post by Brian Harring
Post by Ben de Groot
Thanks. I have given it a quick overview for the qt herd. I still
don't see what using DEPENDENCIES adds to what we do now with separate
*DEPEND variables. I see no convincing reason to change what we do.
Ok, so here's some stats: in the tree, we have 31360 ebuilds, and 194
eclasses; grand total of 31554 sources of metadata content (I say
metadata since vapier has eblits in use which are just phase
functions- I'm not scanning those intentionally).
Doing some simple scans of the tree, here's some stats; note these
stats are duplicated in the glep (they're nice selling points, thus
1) 746 hits in the tree for COMMON_DEPEND; that's 2%, and the usages
I'm aware of have been for literally, what it sounds like- depends
that are both DEPEND and RDEPEND.
2) scanning for assignments of RDEPEND=.*|$\{?DEPEND\}? gets a hit on
5343 unique sources. Searching for the inverse gets a hit on 10008
unique sources. Meaning that right there, ~48.6% of the tree is
duplicating deps between the two forms. This puts us to 16083 unique
sources in the tree that would benefit in some form (~51%).
3) What's interesting about that 51% is the eapi groupings; in EAPI4,
the autosetting of RDEPEND="${RDEPEND:-${DEPEND}}" was discontinued.
Roughly 50% of the initial 51% match is EAPI4; the rest are eapi0-3.
4) Again, keep in mind that the grep's in use above are single line
matches- I'm definitely missing some ebuilds, and for complex
dependencies that are appended/set by the eclass, likely missing a lot
of that too.
So... basically, people are already doing this manually with their own
intermediate vars.
And this works fine, so it doesn't warrant a cosmetic change.
Post by Brian Harring
Just a rough but mildly entertaining stat there's basically 8.38MB
worth of normalized, literal fricking dependency; using a crappy algo
(rather than a human doing the deps who can do it better), 37.4%
(3.1MB) of that is removed via going to dependencies. It goes
without saying that this would be a helluva lot less torturous on a
proper PM implementation that parses the tree once, and renders- let
alone repoman being able to avoid repeat scans of things it already
has examined.
Mind you, portage doesn't do that, but this would be good incentive to
do a proper tree. :)
PM internals are not really relevant to us developers. Just make it work... ;-)

The current *DEPEND system works. So again, I don't see a convincing
reason for change.
Post by Brian Harring
Post by Ben de Groot
As I've said before on IRC, we need a good costs/benefits overview.
Right now I only see costs (migrating ebuilds and eclasses) and no
benefits.
Offhand... I wouldn't be pushing for this if I didn't think this would
be a boon over all- both for devs, and PMs.
I think the actual 'cost' probably got lost in the noise of the
various flames; what I'm proposing is basically zero cost for devs, it
shoves the work to the PM, leaving the option for devs to do a more
fine grained form if they can.
As soon as you expose this to devs (ebuild/eclass developers and
maintainers), you can no longer talk about zero cost.

We either have the enormous cost of migrating the whole tree to the
new system — which I understand you are not advocating because of the
cost; or we have the (lower, but not zero) cost of maintaining two
systems in the tree, which developers will have to know and understand
in order to be able to deal with them correctly. The latter will also
make eclasses needlessly more complex.

So in my opinion it is folly either way, because the costs are high,
and the benefits only cosmetic.
Post by Brian Harring
I'm starting a seperate thread w/ a glep for this; I think you should
take a look at the exact details of the specification including how
DEPEND/RDEPEND/PDEPEND are handled in parallel to DEPENDENCIES (short
version: if existent, they're automatically folded into DEPENDENCIES
in my proposal); the end result is basically zero pain transition,
while enabling us to start getting gains.
Which gains? The glep only talks about cosmetics...
--
Cheers,

Ben | yngwin
Gentoo developer
Gentoo Qt project lead, Gentoo Wiki admin
Duncan
2012-09-19 10:59:58 UTC
Permalink
Post by Ben de Groot
Post by Brian Harring
So... basically, people are already doing this manually with their own
intermediate vars.
And this works fine, so it doesn't warrant a cosmetic change.
@ferringb:

yngwin has a point that I've not seen addressed.

What /is/ wrong with the whole CDEPEND intermediate var idea? It seems
to work and /I/ don't know of any problems with it (and it would appear,
neither does yngwin), yet you talk about it as if there's something wrong
with it.

And while we're at it, do DEPEND="$RDEPEND ..." style solutions have the
same problems (or lack thereof)?

FWIW I personally like the whole single-var idea, and CERTAINLY
appreciate the various statistical cache savings, etc. If we were
starting from scratch now, I'd definitely favor the single var approach.
But the combined developer mental cost of having to learn the new method
and then maintain a working understanding of both over some longer period
is nothing to sneeze at, and I'm not entirely convinced that it's worth
that cost, even assuming a doubling of the number of dependency types
with a lot of commonality between them, and the added benefit a single
deps var would have in that case.

And the case for a single deps var isn't being helped by the implication
that there's something wrong with both the intermediate var and copying
var methods, without ever saying what that "wrong" might be, in the face
of the experience of many that those existing methods "just work". So if
there's something wrong with them, let's get it out there where people
can see it. And if there isn't, please eliminate the noise of that
implication from the argument.

Thanks. =:^)
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
Michael Orlitzky
2012-09-19 13:09:25 UTC
Permalink
Post by Duncan
Post by Ben de Groot
Post by Brian Harring
So... basically, people are already doing this manually with their own
intermediate vars.
And this works fine, so it doesn't warrant a cosmetic change.
yngwin has a point that I've not seen addressed.
What /is/ wrong with the whole CDEPEND intermediate var idea? It seems
to work and /I/ don't know of any problems with it (and it would appear,
neither does yngwin), yet you talk about it as if there's something wrong
with it.
And while we're at it, do DEPEND="$RDEPEND ..." style solutions have the
same problems (or lack thereof)?
The problem appears as we introduce more DEPEND variables (which is what
prompted the proposal, IIRC). If we have ADEPEND, BDEPEND, CDEPEND, and
DDEPEND, and there's only some (i.e. not total) sharing going on then
the COMMON_DEPEND pattern starts to fall apart. You potentially need,

AB_DEPEND
AC_DEPEND
AD_DEPEND
BC_DEPEND
BD_DEPEND
CD_DEPEND
ABC_DEPEND
ABD_DEPEND
ACD_DEPEND
BCD_DEPEND
ABCD_DEPEND (COMMON_DEPEND)

This obviously gets worse as more DEPEND vars are introduced.
Ian Stakenvicius
2012-09-19 13:16:02 UTC
Permalink
Post by Michael Orlitzky
Ben de Groot posted on Wed, 19 Sep 2012 12:22:06 +0800 as
Post by Ben de Groot
Post by Brian Harring
So... basically, people are already doing this manually with
their own intermediate vars.
And this works fine, so it doesn't warrant a cosmetic change.
yngwin has a point that I've not seen addressed.
What /is/ wrong with the whole CDEPEND intermediate var idea? It
seems to work and /I/ don't know of any problems with it (and it
would appear, neither does yngwin), yet you talk about it as if
there's something wrong with it.
And while we're at it, do DEPEND="$RDEPEND ..." style solutions
have the same problems (or lack thereof)?
The problem appears as we introduce more DEPEND variables (which is
what prompted the proposal, IIRC). If we have ADEPEND, BDEPEND,
CDEPEND, and DDEPEND, and there's only some (i.e. not total)
sharing going on then the COMMON_DEPEND pattern starts to fall
apart. You potentially need,
AB_DEPEND AC_DEPEND AD_DEPEND BC_DEPEND BD_DEPEND CD_DEPEND
ABC_DEPEND ABD_DEPEND ACD_DEPEND BCD_DEPEND ABCD_DEPEND
(COMMON_DEPEND)
This obviously gets worse as more DEPEND vars are introduced.
Well not really, no -- the additional *DEPENDs that are being proposed
(or at least mentioned) for new EAPI will either remove atoms from
COMMON_DEPEND/DEPEND/RDEPEND or will be used so tersely that a
COMMON_DEPEND or other intermediate variable won't really be necessary
for them.

Besides, this isn't actually a -problem- as there's nothing which
really requires one to use such helpers; ebuild writers just, well,
can. :)
Brian Harring
2012-09-30 22:15:18 UTC
Permalink
Pardon the belated response; responding to emails that are quick where
possible, but lagging on -dev. Missed this one however...
Post by Ciaran McCreesh
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Post by Michael Orlitzky
Ben de Groot posted on Wed, 19 Sep 2012 12:22:06 +0800 as
Post by Ben de Groot
Post by Brian Harring
So... basically, people are already doing this manually with
their own intermediate vars.
And this works fine, so it doesn't warrant a cosmetic change.
yngwin has a point that I've not seen addressed.
What /is/ wrong with the whole CDEPEND intermediate var idea? It
seems to work and /I/ don't know of any problems with it (and it
would appear, neither does yngwin), yet you talk about it as if
there's something wrong with it.
And while we're at it, do DEPEND="$RDEPEND ..." style solutions
have the same problems (or lack thereof)?
The problem appears as we introduce more DEPEND variables (which is
what prompted the proposal, IIRC). If we have ADEPEND, BDEPEND,
CDEPEND, and DDEPEND, and there's only some (i.e. not total)
sharing going on then the COMMON_DEPEND pattern starts to fall
apart. You potentially need,
AB_DEPEND AC_DEPEND AD_DEPEND BC_DEPEND BD_DEPEND CD_DEPEND
ABC_DEPEND ABD_DEPEND ACD_DEPEND BCD_DEPEND ABCD_DEPEND
(COMMON_DEPEND)
This obviously gets worse as more DEPEND vars are introduced.
Well not really, no -- the additional *DEPENDs that are being proposed
(or at least mentioned) for new EAPI will either remove atoms from
COMMON_DEPEND/DEPEND/RDEPEND or will be used so tersely that a
COMMON_DEPEND or other intermediate variable won't really be necessary
for them.
It depends on the dep type actually, and how we're viewing those deps-
if they must be complete or not.

Consider test depends for example. Either we can specify it as "in
addition to depend"- which sucks, because it's entirely possible for a
DEPEND target to not be required for TDEPEND.

While that's a corner case, it's actually easy to address; just
require TDEPEND to be comprehensive for the test phase.

Thus DEPEND and TDEPEND suddenly share a lot, and TDEPEND shares part
of RDEPEND.

Moving on to a real world example... PDEPEND. Currently a full graph
resolution requires pulling RDEPEND and PDEPEND, collapsing them, and
ensuring they're fullfilled for anything that is already installed
(ie, PDEPEND is required after the transaction is completed).

In our current separated var setup, this as said, this requires the PM
to track/handle it separately. However, were PDEPEND to be
complete/full- that would mean the PM could just render for dep:post
and know "this is what is necessary outside of the transactional block
of building/installing it".

So RDEPEND and PDEPEND actually share a *shitton*- they're the same
contents in 97% of the tree. Meaning dep:run,post? for 97% of the
tree, w/ 3% needing to have an addition dep:post? section.

Offhand, comprehensive deps make things easier for devs- it gives them
the ability to be crystal-freaking-clear as to what's needed at each
stage; via them doing so, it means the resolver has a greater space to
dig itself out of fucked up situations if necessary.

Additionally, it actually makes life easier for PM authors. If we had
TDEPEND (test), this is how things would go; note this is written for
just an install, not a replacement (replace is similar, just noisier
deflecting from my point):

1) ensure DEPEND is satisfied
2) run phases setup -> compile
3) ensure TDEPEND is satisfied; Implicitly, DEPEND in the process (if
there is a cycle induced by TDEPEND + DEPEND being required, we have
no way out of it).
4) run install phase
5) ensure RDEPEND is satisfied. No longer care about TDEPEND/DEPEND.
6) run pkg_preinst, merge, pkg_postinst.
7) a transaction is opened up, that must be completed before the PM
exits; before that exit, RDEPEND must still be satisfied, as must
PDEPEND.

With what I'm suggesting.
1) ensure dep:build is satisfied
2) run phases setup -> compile
3) Ensure dep:test is satisfied. If cycle breaking is necessary,
anything in dep:build but not dep:test can be pulled.
4) install phase
5) ensure dep:run is satisfied. Again, if cycle breaking is
necessary, punt whatever isn't in dep:run as necessary to break that
cycle.
6) pkg_preinst, merge, pkg_postinst.
7) transactional block; ensure dep:post is satisfied before exiting
said transaction.

That's for install; for the PM considering a pre-installed pkg, it
converts from;
1) ensure RDEPEND and PDEPEND are satisfied

to
1) ensure dep:post is satisfied.

Tbh, I'm not sure I'm communicating these gains as well as I could be;
at first glance, I'm pretty sure people will say "what's the
difference?" because the gains/difference there are subtle.
Post by Ciaran McCreesh
From a PM level, I can tell you this shift to complete deps per stage
*does* make things easier via simplifying the rules at each step
(namely, render for this form, that's what's required for this point
on), while giving full control to devs to be dead on accurate in their
deps specified (which the PM can then exploit for better cycle
breaking).

The point I'm trying to make here is that each dep phase
should be authorative; in doing so, you start getting
a lot of potential subsets (DEPEND is a subset of TDEPEND, TDEPEND
isn't completely, but mostly a subset of RDEPEND as RDEPEND is a
likely a superset of DEPEND; PDEPEND is a superset of RDEPEND).

So... you could do COMMON_DEPEND, COMMON_TDEPEND, COMMON_RDEPEND in
the ebuild. Or you could just use a syntax form that allows you to
directly inline that up front, rather than having to muck around w/
intermediate vars.
Post by Ciaran McCreesh
Besides, this isn't actually a -problem- as there's nothing which
really requires one to use such helpers; ebuild writers just, well,
can. :)
Getting to this point; yes, people could hack around it manually.
Pretty sure that hasn't been in doubt.

The question is what do we *gain* from making them do so. No one here
has stated what we gain from COMMON_DEPENDS in comparison to
DEPENDENCIES (where they can just inline it).

And as I'm trying to lay out, where dependencies *should* be going
towards, this issue will get worse if we're doing it out of band from
DEPENDENCIES.

~harring
Duncan
2012-10-01 00:23:52 UTC
Permalink
The point I'm trying to make here is that each dep phase should be
authorative; in doing so, you start getting a lot of potential subsets
(DEPEND is a subset of TDEPEND, TDEPEND isn't completely, but mostly a
subset of RDEPEND as RDEPEND is a likely a superset of DEPEND; PDEPEND
is a superset of RDEPEND).
So... you could do COMMON_DEPEND, COMMON_TDEPEND, COMMON_RDEPEND in the
ebuild. Or you could just use a syntax form that allows you to directly
inline that up front, rather than having to muck around w/ intermediate
vars.
Thanks /very/ much! You said you weren't sure you were being clear, but
this is the first time I've /really/ understood what must surely be the
root, at any reasonable level at all.

Let me see if I've got it right:

Yes, in some ways all we're dealing with here is "optics", but the
_problem_ is that with the proposed proliferation in detailed depend-
types, what is now a simple CDEPEND and thus conceptually easy to handle,
breaks into 10/20/50/whatever-large-number different shards, and what's
conceptually easy to handle /now/ becomes many many times more difficult
to handle, both conceptually for package maintainers and practically for
iterative resolution in the PMs, due to the interplay of all the
resulting *CDEPENDs.

The proposed solution to that explosion in conceptual complexity not only
changes the "optics", but by making most of those detail-depend-types
absolute/authoritative, allows both package managers (the programs,
machine) and package maintainers (the humans) to consider each depend-
type separately, thus decreasing both conceptual complexity to a once
again manageable level for package maintainers (humans), and practical
complexity for package managers (machine), increasing efficiency,
reducing resolution time and probably eventually memory/installed-db/
cache size as well.


Of course now I better understand Ciaran's argument for labels as well,
since it would extend the absolute/authoritative principle even further,
into the actual deps specification method in ebuilds/eclasses, thereby
reducing conceptual context load even further via more explicitly
absolute deps at the local level.

But like you, in practice I don't see that going anywhere in gentoo, in
the near/short-intermediate future, primarily due to political realities,
but practically, also due to the conceptual leap it'd require from devs
(as Ciaran himself points out in response to your statistical analysis of
exherbo's repo, former gentoo devs simply don't tend to take advantage of
this aspect of labels in exherbo either; the conceptual leap is in
practice simply too much). Thus, while academically, his label approach
is arguably better in terms of efficiency of absolutes, in practice,
there's little or no difference between how it's used, and how your
filtering approach will be used. Further, given the conceptual distance
between labels and gentoo's current approach, with filters falling in
between and political reality, the pragmatic filters approach at least
has /some/ chance of passing the dev-debate stage and being approved,
implemented and actually available for use in something like a reasonable
timeframe (say EAPI-6, in a year's time, a bit more for actual in-tree
use, given the historic EAPI-a-year processing). But exherbo style
labels support altho academically better, given political reality, in all
likelihood would take at least 2-3 years to pass and be usable in-tree.
And even then, its practical use as demonstrated in exherbo wouldn't take
advantage of the differences for another couple years after that, at
least.

Given that, having use of the useful pragmatic approach in a year's time
or so seems best, as opposed to an arguably (as you've pointed out, no
practical demonstration of it in exherbo yet, at least that you've been
able to find) more academically ideal approach in three, without any real
benefit over the pragmatic for five or more.
Post by Ian Stakenvicius
Besides, this isn't actually a -problem- as there's nothing which
really requires one to use such helpers; ebuild writers just, well,
can. :)
Getting to this point; yes, people could hack around it manually. Pretty
sure that hasn't been in doubt.
That's clearer now. Yes, people could continue to hack around it via
CDEPENDS, etc. But the number of common vars (or alternative
RDEPEND="$DEPEND ..." assignments) and the resulting conceptual load on
the human maintainer is set to increase exponentially as the number of
depend-types increases linearly. At some point it's just no longer
practically maintainable, and the whole process breaks down.

What the single dependencies variable aims to do in both the filtering
and labels forms is prevent that ultimate conceptual overload and
resulting process breakdown by allowing direct compound assignments, thus
eliminating the intermediate assignments and their exponential
proliferation.

Thanks again, Brian. Much clearer now, indeed, at least for me, and
presumably for others who previously had the same problem I was having.

=:^)
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
Ian Stakenvicius
2012-10-02 17:47:23 UTC
Permalink
Post by Brian Harring
Pardon the belated response; responding to emails that are quick
where possible, but lagging on -dev. Missed this one however...
No worries, there's a lot going on.. :D
Post by Brian Harring
Post by Ian Stakenvicius
Post by Michael Orlitzky
Post by Duncan
yngwin has a point that I've not seen addressed.
What /is/ wrong with the whole CDEPEND intermediate var idea?
The problem appears as we introduce more DEPEND variables
(which is what prompted the proposal, IIRC). If we have
ADEPEND, BDEPEND, CDEPEND, and DDEPEND, and there's only some
(i.e. not total) sharing going on then the COMMON_DEPEND
pattern starts to fall apart. You potentially need,
AB_DEPEND AC_DEPEND AD_DEPEND BC_DEPEND BD_DEPEND CD_DEPEND
ABC_DEPEND ABD_DEPEND ACD_DEPEND BCD_DEPEND ABCD_DEPEND
(COMMON_DEPEND)
This obviously gets worse as more DEPEND vars are introduced.
Well not really, no -- the additional *DEPENDs that are being
proposed (or at least mentioned) for new EAPI will either remove
atoms from COMMON_DEPEND/DEPEND/RDEPEND or will be used so
tersely that a COMMON_DEPEND or other intermediate variable won't
really be necessary for them.
It depends on the dep type actually, and how we're viewing those
deps- if they must be complete or not. [ .. Snip! .. ]
The point I'm trying to make here is that each dep phase should be
authorative; in doing so, you start getting a lot of potential
subsets (DEPEND is a subset of TDEPEND, TDEPEND isn't completely,
but mostly a subset of RDEPEND as RDEPEND is a likely a superset of
DEPEND; PDEPEND is a superset of RDEPEND).
So... you could do COMMON_DEPEND, COMMON_TDEPEND, COMMON_RDEPEND in
the ebuild. Or you could just use a syntax form that allows you
to directly inline that up front, rather than having to muck around
w/ intermediate vars.
... I think what you've just described here might be where the primary
difference in thinking is for most of us, between moving to
DEPENDENCIES and keeping the current *DEPENDs -- to me, from an ebuild
writer's perspective, the *DEPENDS -shouldn't- be authoritative. IE,
instead of thinking of PDEPEND as a superset of RDEPEND I consider
they are two separate sets, which should not intersect, and are
unioned together to form the full set of runtime dependencies. IE:
FULL_RUNTIME_DEPEND="$RDEPEND $PDEPEND" somewhere inside portage, if
portage actually needed it this way.

And I see this as how many of the other proposed new *DEPENDs would
work too, ie, they are a refined subset of the bigger sets and should
not intersect with the 'parent' *DEPEND that was used instead on older
EAPIs.

So if this were to change, it might make sense (as Duncan i think
pointed out in his response to this message), to a debate on whether
or not ebuilds must specify an authoritative list for each dep phase.
(I haven't read through PMS but I'm going to assume that it doesn't
specify this anywhere yet--and if it does, i'm sure Ciaran or someone
will quote it in response :)
Ben de Groot
2012-10-03 04:00:39 UTC
Permalink
Post by Ian Stakenvicius
Post by Michael Orlitzky
Post by Duncan
yngwin has a point that I've not seen addressed.
What /is/ wrong with the whole CDEPEND intermediate var idea?
The problem appears as we introduce more DEPEND variables
(which is what prompted the proposal, IIRC). If we have
ADEPEND, BDEPEND, CDEPEND, and DDEPEND, and there's only some
(i.e. not total) sharing going on then the COMMON_DEPEND
pattern starts to fall apart. You potentially need,
AB_DEPEND AC_DEPEND AD_DEPEND BC_DEPEND BD_DEPEND CD_DEPEND
ABC_DEPEND ABD_DEPEND ACD_DEPEND BCD_DEPEND ABCD_DEPEND
(COMMON_DEPEND)
This obviously gets worse as more DEPEND vars are introduced.
Well not really, no -- the additional *DEPENDs that are being
proposed (or at least mentioned) for new EAPI will either remove
atoms from COMMON_DEPEND/DEPEND/RDEPEND or will be used so
tersely that a COMMON_DEPEND or other intermediate variable won't
really be necessary for them.
Another thing I wanted to point out is that those "potential" extra
variables are not needed in practice. We already have 98% of the tree
(if I got the previously mentioned stats right) that does fine with
just one or two ({R,}DEPEND). The majority of that other 2% needs just
one more variable. There may be corner cases where more vars would be
needed, but those will never be more than a few ebuilds.

It's just not worth it to completely change the way we do things (or
use two systems in parallel) just for a few ebuilds that would
significantly benefit.

If we were a new distro and designing our ebuild format from scratch,
then yes, I would say your proposal has merit. But we aren't. We have
hundreds of people and tens of thousands of ebuilds using *DEPEND just
fine. There are no big problems, only corner-cases. We're not talking
about incremental improvements either (such as was the case e.g. with
use deps).

Let's just keep things simple, and refrain from "fixing" what isn't broken.
--
Cheers,

Ben | yngwin
Gentoo developer
Gentoo Qt project lead, Gentoo Wiki admin
Steven J. Long
2012-10-07 14:09:27 UTC
Permalink
Post by Brian Harring
Post by Ian Stakenvicius
Post by Michael Orlitzky
The problem appears as we introduce more DEPEND variables (which is
what prompted the proposal, IIRC). If we have ADEPEND, BDEPEND,
CDEPEND, and DDEPEND, and there's only some (i.e. not total)
sharing going on then the COMMON_DEPEND pattern starts to fall
apart. You potentially need,
AB_DEPEND AC_DEPEND AD_DEPEND BC_DEPEND BD_DEPEND CD_DEPEND
ABC_DEPEND ABD_DEPEND ACD_DEPEND BCD_DEPEND ABCD_DEPEND
(COMMON_DEPEND)
This obviously gets worse as more DEPEND vars are introduced.
Well not really, no -- the additional *DEPENDs that are being proposed
(or at least mentioned) for new EAPI will either remove atoms from
COMMON_DEPEND/DEPEND/RDEPEND or will be used so tersely that a
COMMON_DEPEND or other intermediate variable won't really be necessary
for them.
It depends on the dep type actually, and how we're viewing those deps-
if they must be complete or not.
Consider test depends for example.
I'd like to see some concrete discussion of link-depends/lib-depends, since
they represent a much more fundamental type of dependency, and would imo
remove atoms from a lot of other places.

Discussing test dependencies first, seems like putting the cart before the
horse.
Post by Brian Harring
The point I'm trying to make here is that each dep phase
should be authorative; in doing so, you start getting
a lot of potential subsets (DEPEND is a subset of TDEPEND, TDEPEND
isn't completely, but mostly a subset of RDEPEND as RDEPEND is a
likely a superset of DEPEND; PDEPEND is a superset of RDEPEND).
And all of them are a superset of LDEPEND (which is why I say it's more
fundamental, and class it as an omission from the original conversion
from ports to portage. The fact that link-dependencies are required to be
installed before build can begin, does not make them BUILD_DEPENDS, of
which DEPEND is the equivalent in Gentoo.)

In terms of what has been mentioned before, the set of dependencies
required at a particular point, LDEPEND are required at all of them.

As for the authoritative set at each point, you've already shown that the
package mangler has to build that currently, so there's no implementation
cost in continuing to do so.
Post by Brian Harring
So... you could do COMMON_DEPEND, COMMON_TDEPEND, COMMON_RDEPEND in
the ebuild. Or you could just use a syntax form that allows you to
directly inline that up front, rather than having to muck around w/
intermediate vars.
Or you could address the gaping hole in the spec, and allow people to simply
specify the most fundamental dependency, without having to hack it in.

Then TDEPEND would just be what was required in addition during TEST phases,
a large amount of those common dependency variables would not be needed, and
similarly for C/HOST_DEPENDS (aka the new DEPEND under the awfully-specified
HDEPEND EAPI) since by definition a library dependency lives in the same
ROOT as the installed package.
Post by Brian Harring
Post by Ian Stakenvicius
Besides, this isn't actually a -problem- as there's nothing which
really requires one to use such helpers; ebuild writers just, well,
can. :)
Getting to this point; yes, people could hack around it manually.
Pretty sure that hasn't been in doubt.
The question is what do we *gain* from making them do so. No one here
has stated what we gain from COMMON_DEPENDS in comparison to
DEPENDENCIES (where they can just inline it).
LDEPEND enables much better tracking of linkage, which is pretty fundamental
to building and maintaining a distribution. It's also something that
programmers specify upfront in configure and makefiles. As shown above,
it eliminates a lot of common dependencies, and at least some HDEPEND (if
that variable name is used accurately.)

Additionally, separating out the variable (which yes, can also be done with
a label, but it does need to be done) means linkage can be tracked
automatically based on the sort of data that every package supplies, every
other distribution provides, and every developer has to think about when
writing an ebuild; instead of on yet another syntax obfus^W complication that
was designed for java and python plugins.

Doing it in a separate variable as opposed to labelling it, means that
variable can be treated differently, should that be desired in a new EAPI.
Such a change in how LDEPEND were specified, would not impact on any of the
rest of the single variable format (which is more complex, and more
interdependent, at least in its scripted form) and would not need to be
verified against all the other types of specification in the rest of that
"integrated" variable. So a separate variable is more flexible.

It's also a *lot* more transparent.

For instance, if we'd had that before, do you agree we'd have been scanning
binaries to check for linkage outside that set and system? (After all, we
scan for a lot of other stuff already; scanelf is a gentoo success story.)

How would usage of revdep-rebuild been affected? What about implementation
of preserved-libs?
Post by Brian Harring
And as I'm trying to lay out, where dependencies *should* be going
towards, this issue will get worse if we're doing it out of band from
DEPENDENCIES.
You've given a convincing argument for the package mangler using a unified
variable internally, and if the PM authors all agree, using that in the
cache format. (ie the extra parsing complexity and maintenance is worth it
for the difference in IO across the cache.)

But you've also shown exactly how all that information can be inferred from
the existing ebuild format (and once inferred would actually be simpler than
the whole variable specified in the ebuild.) As such, it appears more like
something that should be in the back-end, without requiring such a shift in
how people write ebuilds.

Your additional arguments as to why this gets more complex later down the
line are predicated on ignoring LDEPEND.

If that requires a bit more runtime at cache-generation, we're already paying
that price. Saving a small amount of time there, where most users never see
it, doesn't seem worth the mental effort, and maintenance cost, required
to change the format. "Just go ahead and use it in the cache," seems like
the best solution.

Regards,
steveL.

Michał Górny
2012-09-16 07:56:27 UTC
Permalink
On Sat, 15 Sep 2012 18:20:26 -0700
Post by Brian Harring
Post by Michał Górny
On Sat, 15 Sep 2012 13:33:18 -0700
Post by Brian Harring
To demonstrate the gain of this, we basically take the existing
tree's deps, and re-render it into a unified DEPENDENCIES form.
But in order to do this, we first have to decide exactly what kind
of dependencies do we want to have. Then convert the tree to
a separate-variable form with new dependencies. Then we can compare
it with the DEPENDENCIES form and decide which one is better.
Funny you mentioned that, I just finished tweaking pquery to generate
real world example unified dependencies; these *are* accurate, just
to be clear.
But consider that for example Zac & AxS (correct me if I recall it
correctly) considered making changing the meaning of RDEPEND to install
them before the build, thus effectively making 'build,run' useless.
Post by Brian Harring
Total cache savings from doing this for a full tree conversion, for
our existing md5-cache format is 2.73MB (90 byes per cache entry).
Calculating the savings from the ebuild/eclass standpoint is
dependent on how the deps are built up, so I skipped that.
You're storing the cache in a tarball?
--
Best regards,
Michał Górny
Brian Harring
2012-09-16 11:10:01 UTC
Permalink
Post by Michał Górny
But consider that for example Zac & AxS (correct me if I recall it
correctly) considered making changing the meaning of RDEPEND to install
them before the build, thus effectively making 'build,run' useless.
I really am not trying to be a blatant dick to you, but this has
/zero/ relevance. RDEPEND means "required for runtime". That ain't
changing. If they were discussing changing what RDEPEND meant, then
they were high, period.

If zac/axs want to try and make the resolver install RDEPEND before
DEPEND... well, they're free to. That doesn't change the fact that
the deps still must be specified correctly; in short, build,run is
very much relevant.

What I suspect they were intending on doing is letting the resolver
work on RDEPENDS of a pkg in parallel to that pkg being built; this is
a parallelization scheduling optimization, still requires accurate
deps.

I'm trying to be nice here, but you're very confused on this matter.
Post by Michał Górny
Post by Brian Harring
Total cache savings from doing this for a full tree conversion, for
our existing md5-cache format is 2.73MB (90 byes per cache entry).
Calculating the savings from the ebuild/eclass standpoint is
dependent on how the deps are built up, so I skipped that.
You're storing the cache in a tarball?
Going to assume you're not trolling, and instead use this as a
way to point out that this actually *does* matter, although it's
admittedly not obvious if you don't know much about the guts of
package managers, or don't spend your saturday nights doing fun
things like optimizing ebuild package manager performance.

First, the figure is 3.204MB if default context is used; ~9.5% of the
content footprint for md5-cache specifically.

Little known fact; rsync transfers for gentoo are required to be
--whole-file; meaning no intra-file delta compression, it transfers
the whole file itself. This is done to keep cpu load on rsync nodes
low (else they'd be calculating minimally 97k md4's for every sync,
not counting the rolling adl32 chksum for all content dependent on
the window cut off threshold- sounds minor, but it's death by a
thousand cuts).

For obvious reasons, the cache is the hottest part of the tree due to
cascading updates due to eclass changes. In other words, that ~9.5%
reduction targest the core data actually transferered in a sync.

In terms of the total tree footprint, it's a 1% reduction; mostly lost
in blocksize overhead unless you're using squashfs (which a decent
number of folks do for speed reasons), or use tail packing FS for the
tree (again, more than you'd think- known primarily due to reiserfs
corruption bugs causing some hell on PM caches).

There's also the fact doing this means best case, 2 less inodes per
VDB entry (more once we start adding dependency types). For my vdb, I
have 15523 across 798 pkgs. 1331 of that is *DEPEND, converted to
DEPENDENCIES the file count is 748. Note that's preserving DEPEND,
although it's worthless at this stage of the vdb. So 5% reduction in
files in there. Whoopy-de-doo, right?

This one I can't test as well since the only rotational media I've got
these days is a hardware raid w/ a beefy cache; the closest I can
manage is local network nfs to an ssd FS, so it'll have to serve
as a stand in for cold cache/hot cache, and for a demonstration of
why having a backend that is a 101 small individual files is bad.

Best of 5 is displayed below:

Iterating over the vdb, and parsing and rendering all depends for our
current layout, w/ the vdb stored on nfs:

cold cache:
real 0m30.405s
user 0m1.046s
sys 0m0.390s

hot cache:
real 0m16.483s
user 0m0.883s
sys 0m0.168s

non-optimized, hacked to work (known slower for parsing in comparison
to the non quicky hack), iterating over the vdb, parsing all
depends and rendering said depends when it's stored as DEPENDENCIES;
literally, rendering DEPEND from it, RDEPEND, PDEPEND.

cold cache:
real 0m18.329s
user 0m0.908s
sys 0m0.280s

hot cache
real 0m12.185s
user 0m0.860s
sys 0m0.128s


You get the idea. See the various infamous cold cache/hot cache
performance tests in doubt; I can tell you that a similar trick, done
in '07, literally just skipping loading USE till it was needed for
provides parsing was enough to bring a 5400RPM drive's run time
down from 15s to 12s for cold cache- for parsing provides *alone*,
nothing else. Either way, do your own investigation, it's a
good education on performance.


Hopefully for the others listening, that last section was a random but
useful tidbit of info; if not, pardon, just being through to make sure
this point is not raised again.

~harring
Michał Górny
2012-09-16 11:21:26 UTC
Permalink
On Sun, 16 Sep 2012 04:10:01 -0700
Post by Brian Harring
Post by Michał Górny
But consider that for example Zac & AxS (correct me if I recall it
correctly) considered making changing the meaning of RDEPEND to
install them before the build, thus effectively making 'build,run'
useless.
I really am not trying to be a blatant dick to you, but this has
/zero/ relevance. RDEPEND means "required for runtime". That ain't
changing. If they were discussing changing what RDEPEND meant, then
they were high, period.
If zac/axs want to try and make the resolver install RDEPEND before
DEPEND... well, they're free to. That doesn't change the fact that
the deps still must be specified correctly; in short, build,run is
very much relevant.
I don't think we have made up our mind what *exactly* we want from
deps. Just because we have something semi-correct right now, doesn't
mean that we don't want to change that. But I guess with the whole
amount of noise in here I won't ever get any definitive answer.
Post by Brian Harring
There's also the fact doing this means best case, 2 less inodes per
VDB entry (more once we start adding dependency types). For my vdb,
I have 15523 across 798 pkgs. 1331 of that is *DEPEND, converted to
DEPENDENCIES the file count is 748. Note that's preserving DEPEND,
although it's worthless at this stage of the vdb. So 5% reduction in
files in there. Whoopy-de-doo, right?
So we can modify vdb now? What about all those applications which
obviously are broken due to that?
--
Best regards,
Michał Górny
Brian Harring
2012-09-16 11:49:21 UTC
Permalink
Post by Michał Górny
On Sun, 16 Sep 2012 04:10:01 -0700
Post by Brian Harring
Post by Michał Górny
But consider that for example Zac & AxS (correct me if I recall it
correctly) considered making changing the meaning of RDEPEND to
install them before the build, thus effectively making 'build,run'
useless.
I really am not trying to be a blatant dick to you, but this has
/zero/ relevance. RDEPEND means "required for runtime". That ain't
changing. If they were discussing changing what RDEPEND meant, then
they were high, period.
If zac/axs want to try and make the resolver install RDEPEND before
DEPEND... well, they're free to. That doesn't change the fact that
the deps still must be specified correctly; in short, build,run is
very much relevant.
I don't think we have made up our mind what *exactly* we want from
deps.
Are we now expecting deps to give us ponies or something? We know
*exactly* what we want from deps, and their current definition- the
problem isn't the definition, it's that we don't have the forms we
need.
Post by Michał Górny
Just because we have something semi-correct right now, doesn't
mean that we don't want to change that.
This is a no-op argument against the proposal: "we can't
change the deps because we might want to change the deps". It's also
irrelevant due to the core basis of it being broken as fuck (described
above).
Post by Michał Górny
Post by Brian Harring
There's also the fact doing this means best case, 2 less inodes per
VDB entry (more once we start adding dependency types). For my vdb,
I have 15523 across 798 pkgs. 1331 of that is *DEPEND, converted to
DEPENDENCIES the file count is 748. Note that's preserving DEPEND,
although it's worthless at this stage of the vdb. So 5% reduction in
files in there. Whoopy-de-doo, right?
So we can modify vdb now? What about all those applications which
obviously are broken due to that?
You're misunderstanding the problem of the VDB. We cannot change the
core structure of it- certain ebuilds currently expect to be able to
find USE/IUSE at specific pathways. That's where we're blocked.

Adding a new metadata key can, and has been done (defined_phases,
properties, required_use, etc). Removing keys can be done. For
efficiency, not writing a key if it's empty, can be and is being done.

Basically, your retort again, is wildly off mark and misunderstanding
the problem.

I strongly suggest you go do some real research into the existing
PMs, and the problems being discussed. Go read some code; you've
minimally got 3 different codebases to work from, and that's not
counting ML archives, tools, PMS, or the devmanul.

If per you're statements, you're actually serious about writing your
own PM, you're going to need to know this sort of shit /anyways/ to
actually do the basic legwork of a PM, so please go do the necessary
legwork if you want to persist in arguing about internals.

~harring
Michał Górny
2012-09-16 12:02:24 UTC
Permalink
On Sun, 16 Sep 2012 04:49:21 -0700
Post by Brian Harring
Post by Michał Górny
On Sun, 16 Sep 2012 04:10:01 -0700
Post by Brian Harring
Post by Michał Górny
But consider that for example Zac & AxS (correct me if I recall
it correctly) considered making changing the meaning of RDEPEND
to install them before the build, thus effectively making
'build,run' useless.
I really am not trying to be a blatant dick to you, but this has
/zero/ relevance. RDEPEND means "required for runtime". That
ain't changing. If they were discussing changing what RDEPEND
meant, then they were high, period.
If zac/axs want to try and make the resolver install RDEPEND
before DEPEND... well, they're free to. That doesn't change the
fact that the deps still must be specified correctly; in short,
build,run is very much relevant.
I don't think we have made up our mind what *exactly* we want from
deps.
Are we now expecting deps to give us ponies or something? We know
*exactly* what we want from deps, and their current definition- the
problem isn't the definition, it's that we don't have the forms we
need.
No, the problem is that we think we need more than we have now. Unless
you're considering the whole point of this thread is cosmetics... then
please leave that to Fedora or other people who are paid to change
stuff just because they can.
Post by Brian Harring
Post by Michał Górny
Just because we have something semi-correct right now, doesn't
mean that we don't want to change that.
This is a no-op argument against the proposal: "we can't
change the deps because we might want to change the deps". It's also
irrelevant due to the core basis of it being broken as fuck
(described above).
What I'm trying to say is that you're making a lot of noise about
cosmetics while we haven't even agreed on what's supposed to be inside.
So, are we introducing this obtuse syntax for three DEPEND variables,
of which the third is almost never used?
--
Best regards,
Michał Górny
Brian Harring
2012-09-16 13:38:36 UTC
Permalink
Post by Michał Górny
On Sun, 16 Sep 2012 04:49:21 -0700
Post by Brian Harring
Post by Michał Górny
On Sun, 16 Sep 2012 04:10:01 -0700
Post by Brian Harring
Post by Michał Górny
But consider that for example Zac & AxS (correct me if I recall
it correctly) considered making changing the meaning of RDEPEND
to install them before the build, thus effectively making
'build,run' useless.
I really am not trying to be a blatant dick to you, but this has
/zero/ relevance. RDEPEND means "required for runtime". That
ain't changing. If they were discussing changing what RDEPEND
meant, then they were high, period.
If zac/axs want to try and make the resolver install RDEPEND
before DEPEND... well, they're free to. That doesn't change the
fact that the deps still must be specified correctly; in short,
build,run is very much relevant.
I don't think we have made up our mind what *exactly* we want from
deps.
Are we now expecting deps to give us ponies or something? We know
*exactly* what we want from deps, and their current definition- the
problem isn't the definition, it's that we don't have the forms we
need.
No, the problem is that we think we need more than we have now.
Read what I wrote. "we don't have the forms we need"; a more proper
statement is "we don't have all of the forms we need".

Please read what I write, rather than just responding blindly. You
may have time to waste, but I don't, nor do the people on this ml need
to see you respond 13 minutes after I send an email when you
can't even be bothered to read the fucking content properly.
Post by Michał Górny
Unless
you're considering the whole point of this thread is cosmetics... then
please leave that to Fedora or other people who are paid to change
stuff just because they can.
This isn't productive; frankly it's childish. Take it to the forums
if you want to continue on tangents like this.
Post by Michał Górny
Post by Brian Harring
Post by Michał Górny
Just because we have something semi-correct right now, doesn't
mean that we don't want to change that.
This is a no-op argument against the proposal: "we can't
change the deps because we might want to change the deps". It's also
irrelevant due to the core basis of it being broken as fuck
(described above).
What I'm trying to say is that you're making a lot of noise about
cosmetics while we haven't even agreed on what's supposed to be inside.
So, are we introducing this obtuse syntax for three DEPEND variables,
of which the third is almost never used?
Reiterating the points again, and for the final time for you since you
seem intent on riding the short bus for this particular subject:

1) This unifies the existing syntax down into a collapsed form. In
doing so, there are measurable gains across the board for PM
efficiency and rsync alone.

2) In unifying the syntax via reusing our /existing fucking syntax/,
we formalize the adhoc common dependency assignments devs already are
doing in the tree.

3) In moving to a unified syntax, it positions us to easily introduce
new dependency types without introducing more redundancy. Easier to
add new dep types, faster to add new dep types, more efficient in
doing so in comparison to existing approaches, and done in a fashion
that devs can reuse existing conditionals.

4) It is not exherbo's DEPENDENCIES. Meaning it is not label based.
Meaning you do not need to knee-jerk attack it because of some notion
it's ciaran based/related.

Honestly, stop wasting my (and others time) and please read this email
full and through, including the /full thread you're blindly
responding to/ before responding again.

There is no prive for having the fastest turn around time in
responding to an email; not unless you consider a permenant /ignore
and killfile addition to be a prize.

~harring
Paweł Hajdan, Jr.
2012-09-07 16:12:08 UTC
Permalink
Post by Alexis Ballier
I actually do like the concept but I'm not sure we can reach consensus
about '*DEPEND vs DEPENDENCIES'
I also like at least significant parts of the DEPENDENCIES concept,
especially when we start adding more dep variables like HDEPEND.

My understanding is that COMMON_DEPEND can be easily and uniformly
expressed as build+run.

Do we have some list of pros and cons, so that the discussion can be
more focused?
Post by Alexis Ballier
a possibility to get people used to it
could be to have two parallel EAPIs, like 6 and 6-dependencies, where
the former will keep the old style and the latter use DEPENDENCIES.
How about just allowing older style DEPEND/RDEPEND, but not both
DEPEND/RDEPEND and DEPENDENCIES in the same ebuild? Note that this may
interact with eclasses (which is also true for above idea I think).
Michał Górny
2012-09-07 16:43:47 UTC
Permalink
On Fri, 07 Sep 2012 18:12:08 +0200
Post by Paweł Hajdan, Jr.
Post by Alexis Ballier
I actually do like the concept but I'm not sure we can reach
consensus about '*DEPEND vs DEPENDENCIES'
I also like at least significant parts of the DEPENDENCIES concept,
especially when we start adding more dep variables like HDEPEND.
My understanding is that COMMON_DEPEND can be easily and uniformly
expressed as build+run.
Do we have some list of pros and cons, so that the discussion can be
more focused?
There's one on the wiki but I'm not sure if it's up-to-date now. I made
it on top of the previous thread.

http://wiki.gentoo.org/wiki/Future_EAPI/DEPENDENCIES_variable
--
Best regards,
Michał Górny
Ciaran McCreesh
2012-09-07 16:10:59 UTC
Permalink
On Fri, 7 Sep 2012 12:46:41 -0300
Post by Alexis Ballier
For example, what is the HDEPEND equivalent for DEPENDENCIES ? exherbo
documentation doesn't seem to mention an equivalent label.
DEPENDENCIES is essentially independent of what label names we
introduce. I get the impression Gentoo will be bikeshedding, er, I mean
selecting shorter names for some of the labels than what Exherbo is
using. So HDEPEND could be 'host' if you like.

In any case, the reason you don't see a 'host' label on Exherbo is
because it's called 'build'. Exherbo's taken a more comprehensive
approach to handling ROOT-related dependencies -- dependency resolution
for ROOT!=/ still uses / for satisfying not-purely-runtimeish
dependencies, and then has a way of locking versions on / to versions
in ROOT. It does rely upon having a fully-ROOT-and-/-aware resolver,
though, so it may not be suitable for Gentoo.
--
Ciaran McCreesh
Zac Medico
2012-09-07 16:53:46 UTC
Permalink
Post by Michał Górny
On Fri, 7 Sep 2012 12:46:41 -0300
Post by Alexis Ballier
For example, what is the HDEPEND equivalent for DEPENDENCIES ? exherbo
documentation doesn't seem to mention an equivalent label.
DEPENDENCIES is essentially independent of what label names we
introduce. I get the impression Gentoo will be bikeshedding, er, I mean
selecting shorter names for some of the labels than what Exherbo is
using. So HDEPEND could be 'host' if you like.
In any case, the reason you don't see a 'host' label on Exherbo is
because it's called 'build'. Exherbo's taken a more comprehensive
approach to handling ROOT-related dependencies -- dependency resolution
for ROOT!=/ still uses / for satisfying not-purely-runtimeish
dependencies, and then has a way of locking versions on / to versions
in ROOT. It does rely upon having a fully-ROOT-and-/-aware resolver,
though, so it may not be suitable for Gentoo.
If you're insinuating that Portage may not have a
"fully-ROOT-and-/-aware resolver", then I can assure you that this is
not a problem. The only reason that Portage currently doesn't have "a
way of locking versions on / to versions in ROOT" is that none of the
existing EAPIs have a way to express this kind of dependency constraint.
--
Thanks,
Zacd
Ciaran McCreesh
2012-09-07 16:58:02 UTC
Permalink
On Fri, 07 Sep 2012 09:53:46 -0700
Post by Zac Medico
If you're insinuating that Portage may not have a
"fully-ROOT-and-/-aware resolver", then I can assure you that this is
not a problem.
In that case, why do we need HDEPEND at all?
--
Ciaran McCreesh
Ian Stakenvicius
2012-09-07 17:02:32 UTC
Permalink
Post by Ciaran McCreesh
Post by Zac Medico
If you're insinuating that Portage may not have a
"fully-ROOT-and-/-aware resolver", then I can assure you that
this is not a problem.
In that case, why do we need HDEPEND at all?
We don't, actually; HDEPEND is essentially DEPEND. what we need is
TDEPEND.
Zac Medico
2012-09-07 17:40:46 UTC
Permalink
Post by Ian Stakenvicius
Post by Ciaran McCreesh
Post by Zac Medico
If you're insinuating that Portage may not have a
"fully-ROOT-and-/-aware resolver", then I can assure you that
this is not a problem.
In that case, why do we need HDEPEND at all?
We don't, actually; HDEPEND is essentially DEPEND. what we need is
TDEPEND.
We could do either one (or do both, and get rid of DEPEND). In
discussions on the chromium-os-dev list [1] (people who could have been
using HDEPEND for years now), the dominant preference was to use HDEPEND
since they felt that it would require the least amount of adjustment to
existing DEPEND settings.

[1]
https://groups.google.com/a/chromium.org/forum/?fromgroups=#!topic/chromium-os-dev/yVAcpfZHrOE
--
Thanks,
Zac
Ian Stakenvicius
2012-09-07 17:58:00 UTC
Permalink
Post by Zac Medico
Post by Ian Stakenvicius
On Fri, 07 Sep 2012 09:53:46 -0700 Zac Medico
Post by Zac Medico
If you're insinuating that Portage may not have a
"fully-ROOT-and-/-aware resolver", then I can assure you
that this is not a problem.
In that case, why do we need HDEPEND at all?
We don't, actually; HDEPEND is essentially DEPEND. what we need
is TDEPEND.
We could do either one (or do both, and get rid of DEPEND). In
discussions on the chromium-os-dev list [1] (people who could have
been using HDEPEND for years now), the dominant preference was to
use HDEPEND since they felt that it would require the least amount
of adjustment to existing DEPEND settings.
[1]
https://groups.google.com/a/chromium.org/forum/?fromgroups=#!topic/chromium-os-dev/yVAcpfZHrOE
Thanks
for clarifying this; after reading through the bug I wasn't
sure if the recommendation was to add HDEPEND only or to deprecate
DEPEND entirely for HDEPEND/TDEPEND.

Just to clarify the work involved in converting to this; since DEPEND
on EAPI<=4 is essentially HDEPEND , wouldn't migration to the new EAPI
(with HDEPEND/DEPEND) generally mean that we would need to
s/DEPEND/HDEPEND/ for the vast majority of ebuilds (ie all the trivial
ones)?
Zac Medico
2012-09-07 18:18:28 UTC
Permalink
Post by Zac Medico
Post by Zac Medico
Post by Ian Stakenvicius
On Fri, 07 Sep 2012 09:53:46 -0700 Zac Medico
Post by Zac Medico
If you're insinuating that Portage may not have a
"fully-ROOT-and-/-aware resolver", then I can assure you
that this is not a problem.
In that case, why do we need HDEPEND at all?
We don't, actually; HDEPEND is essentially DEPEND. what we
need is TDEPEND.
We could do either one (or do both, and get rid of DEPEND). In
discussions on the chromium-os-dev list [1] (people who could
have been using HDEPEND for years now), the dominant preference
was to use HDEPEND since they felt that it would require the
least amount of adjustment to existing DEPEND settings.
[1]
https://groups.google.com/a/chromium.org/forum/?fromgroups=#!topic/chromium-os-dev/yVAcpfZHrOE
Thanks
for clarifying this; after reading through the bug I wasn't sure if
the recommendation was to add HDEPEND only or to deprecate DEPEND
entirely for HDEPEND/TDEPEND.
Just to clarify the work involved in converting to this; since
DEPEND on EAPI<=4 is essentially HDEPEND , wouldn't migration to
the new EAPI (with HDEPEND/DEPEND) generally mean that we would
need to s/DEPEND/HDEPEND/ for the vast majority of ebuilds (ie all
the trivial ones)?
In the linked chromium-os-dev discussion, the consensus seemed to be
that migrating deps from DEPEND to HDEPEND would result in fewer
overall changes than migrating deps from DEPEND to TDEPEND. For this
reason, the dominant preference was to go with HDEPEND.
- --
Thanks,
Zac
Zac Medico
2012-09-07 18:23:35 UTC
Permalink
Post by Zac Medico
Post by Zac Medico
Post by Zac Medico
Post by Ian Stakenvicius
On Fri, 07 Sep 2012 09:53:46 -0700 Zac Medico
Post by Zac Medico
If you're insinuating that Portage may not have a
"fully-ROOT-and-/-aware resolver", then I can assure you
that this is not a problem.
In that case, why do we need HDEPEND at all?
We don't, actually; HDEPEND is essentially DEPEND. what we
need is TDEPEND.
We could do either one (or do both, and get rid of DEPEND). In
discussions on the chromium-os-dev list [1] (people who could
have been using HDEPEND for years now), the dominant
preference was to use HDEPEND since they felt that it would
require the least amount of adjustment to existing DEPEND
settings.
[1]
https://groups.google.com/a/chromium.org/forum/?fromgroups=#!topic/chromium-os-dev/yVAcpfZHrOE
Thanks
for clarifying this; after reading through the bug I wasn't sure
if the recommendation was to add HDEPEND only or to deprecate
DEPEND entirely for HDEPEND/TDEPEND.
Just to clarify the work involved in converting to this; since
DEPEND on EAPI<=4 is essentially HDEPEND , wouldn't migration to
the new EAPI (with HDEPEND/DEPEND) generally mean that we would
need to s/DEPEND/HDEPEND/ for the vast majority of ebuilds (ie
all the trivial ones)?
In the linked chromium-os-dev discussion, the consensus seemed to
be that migrating deps from DEPEND to HDEPEND would result in
fewer overall changes than migrating deps from DEPEND to TDEPEND.
For this reason, the dominant preference was to go with HDEPEND.
Also, if the HDEPEND proposal is accepted, then I would recommend to
treat the migration of dependencies to HDEPEND as entirely optional.
This is similar to the approach that we took with the introduction of
prefix support in EAPI 3, where EAPI 3 ebuilds were not obligated to
support prefix.
- --
Thanks,
Zac
Ciaran McCreesh
2012-09-07 18:23:08 UTC
Permalink
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Fri, 07 Sep 2012 11:18:28 -0700
Post by Zac Medico
In the linked chromium-os-dev discussion, the consensus seemed to be
that migrating deps from DEPEND to HDEPEND would result in fewer
overall changes than migrating deps from DEPEND to TDEPEND. For this
reason, the dominant preference was to go with HDEPEND.
They're looking at "minimum number of changes to make it appear to
work", not "minimum number of changes to express dependencies
correctly".

- --
Ciaran McCreesh
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iEYEARECAAYFAlBKO44ACgkQ96zL6DUtXhHGDwCgi66mZCVH0UWdB8yfJxl8lcuR
QPkAnRgziWGLpMbJ1i92CXjRJ1Q2tWzJ
=S+
Michał Górny
2012-09-07 18:23:23 UTC
Permalink
On Fri, 07 Sep 2012 13:58:00 -0400
Post by Ciaran McCreesh
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Post by Zac Medico
Post by Ian Stakenvicius
On Fri, 07 Sep 2012 09:53:46 -0700 Zac Medico
Post by Zac Medico
If you're insinuating that Portage may not have a
"fully-ROOT-and-/-aware resolver", then I can assure you
that this is not a problem.
In that case, why do we need HDEPEND at all?
We don't, actually; HDEPEND is essentially DEPEND. what we need
is TDEPEND.
We could do either one (or do both, and get rid of DEPEND). In
discussions on the chromium-os-dev list [1] (people who could have
been using HDEPEND for years now), the dominant preference was to
use HDEPEND since they felt that it would require the least amount
of adjustment to existing DEPEND settings.
[1]
https://groups.google.com/a/chromium.org/forum/?fromgroups=#!topic/chromium-os-dev/yVAcpfZHrOE
Thanks
for clarifying this; after reading through the bug I wasn't
sure if the recommendation was to add HDEPEND only or to deprecate
DEPEND entirely for HDEPEND/TDEPEND.
Just to clarify the work involved in converting to this; since DEPEND
on EAPI<=4 is essentially HDEPEND , wouldn't migration to the new EAPI
(with HDEPEND/DEPEND) generally mean that we would need to
s/DEPEND/HDEPEND/ for the vast majority of ebuilds (ie all the trivial
ones)?
I can't agree unless I am missing something. Doesn't the majority of
ebuilds actually require most of DEPEND (well, the part common with
RDEPEND) to be installed on the target? I'm thinking of the shared
libraries mostly.
--
Best regards,
Michał Górny
Ciaran McCreesh
2012-09-07 18:31:16 UTC
Permalink
On Fri, 7 Sep 2012 20:23:23 +0200
Post by Michał Górny
I can't agree unless I am missing something. Doesn't the majority of
ebuilds actually require most of DEPEND (well, the part common with
RDEPEND) to be installed on the target? I'm thinking of the shared
libraries mostly.
"The part common with RDEPEND" is a different issue. We're talking
about what the usual thing to do is for dependencies that are in
DEPEND but not RDEPEND. A typical example here is a binary that is
executed as part of the build process.
--
Ciaran McCreesh
Michał Górny
2012-09-07 18:46:48 UTC
Permalink
On Fri, 7 Sep 2012 19:31:16 +0100
Post by Ciaran McCreesh
On Fri, 7 Sep 2012 20:23:23 +0200
Post by Michał Górny
I can't agree unless I am missing something. Doesn't the majority of
ebuilds actually require most of DEPEND (well, the part common with
RDEPEND) to be installed on the target? I'm thinking of the shared
libraries mostly.
"The part common with RDEPEND" is a different issue. We're talking
about what the usual thing to do is for dependencies that are in
DEPEND but not RDEPEND. A typical example here is a binary that is
executed as part of the build process.
Thanks for keeping me on the topic.

Now, let me remind you because you probably fail to know the world
outside your dreamworld:

(with HDEPEND/DEPEND) generally mean that we would need to
s/DEPEND/HDEPEND/ for the vast majority of ebuilds (ie all the trivial
ones)?

That does effectively refer to the common depends as well. You know, in
the real world where there is no magical variables which do miracles
behind your back.
--
Best regards,
Michał Górny
Ciaran McCreesh
2012-09-07 18:52:05 UTC
Permalink
On Fri, 7 Sep 2012 20:46:48 +0200
Post by Michał Górny
Now, let me remind you because you probably fail to know the world
(with HDEPEND/DEPEND) generally mean that we would need to
s/DEPEND/HDEPEND/ for the vast majority of ebuilds (ie all the
trivial ones)?
That does effectively refer to the common depends as well. You know,
in the real world where there is no magical variables which do
miracles behind your back.
Uhm, no, it doesn't. Things in both DEPEND and RDEPEND are an
*entirely* different case when it comes to destinations, since RDEPEND
goes to ROOT.

The distinction between DEPEND and HDEPEND is relevant only for
dependencies that are not also in RDEPEND.
--
Ciaran McCreesh
Michał Górny
2012-09-07 19:11:22 UTC
Permalink
On Fri, 7 Sep 2012 19:52:05 +0100
Post by Ciaran McCreesh
On Fri, 7 Sep 2012 20:46:48 +0200
Post by Michał Górny
Now, let me remind you because you probably fail to know the world
(with HDEPEND/DEPEND) generally mean that we would need to
s/DEPEND/HDEPEND/ for the vast majority of ebuilds (ie all the
trivial ones)?
That does effectively refer to the common depends as well. You know,
in the real world where there is no magical variables which do
miracles behind your back.
Uhm, no, it doesn't. Things in both DEPEND and RDEPEND are an
*entirely* different case when it comes to destinations, since RDEPEND
goes to ROOT.
And? RDEPEND isn't required to be merged before building the package.
--
Best regards,
Michał Górny
Ciaran McCreesh
2012-09-07 19:13:19 UTC
Permalink
On Fri, 7 Sep 2012 21:11:22 +0200
Post by Michał Górny
On Fri, 7 Sep 2012 19:52:05 +0100
Post by Ciaran McCreesh
On Fri, 7 Sep 2012 20:46:48 +0200
Post by Michał Górny
Now, let me remind you because you probably fail to know the world
(with HDEPEND/DEPEND) generally mean that we would need to
s/DEPEND/HDEPEND/ for the vast majority of ebuilds (ie all the
trivial ones)?
That does effectively refer to the common depends as well. You
know, in the real world where there is no magical variables which
do miracles behind your back.
Uhm, no, it doesn't. Things in both DEPEND and RDEPEND are an
*entirely* different case when it comes to destinations, since
RDEPEND goes to ROOT.
And? RDEPEND isn't required to be merged before building the package.
I repeat: "things in both DEPEND and RDEPEND".

I also repeat: HDEPEND is relevant only to things that are in DEPEND
and not RDEPEND.
--
Ciaran McCreesh
Michał Górny
2012-09-07 19:21:42 UTC
Permalink
On Fri, 7 Sep 2012 20:13:19 +0100
Post by Ciaran McCreesh
On Fri, 7 Sep 2012 21:11:22 +0200
Post by Michał Górny
On Fri, 7 Sep 2012 19:52:05 +0100
Post by Ciaran McCreesh
On Fri, 7 Sep 2012 20:46:48 +0200
Post by Michał Górny
Now, let me remind you because you probably fail to know the
(with HDEPEND/DEPEND) generally mean that we would need to
s/DEPEND/HDEPEND/ for the vast majority of ebuilds (ie all the
trivial ones)?
That does effectively refer to the common depends as well. You
know, in the real world where there is no magical variables
which do miracles behind your back.
Uhm, no, it doesn't. Things in both DEPEND and RDEPEND are an
*entirely* different case when it comes to destinations, since
RDEPEND goes to ROOT.
And? RDEPEND isn't required to be merged before building the
package.
I repeat: "things in both DEPEND and RDEPEND".
I also repeat: HDEPEND is relevant only to things that are in DEPEND
and not RDEPEND.
So... what is your issue in here, sir?
--
Best regards,
Michał Górny
Ciaran McCreesh
2012-09-07 19:25:58 UTC
Permalink
On Fri, 7 Sep 2012 21:21:42 +0200
Post by Michał Górny
So... what is your issue in here, sir?
The issue is what Zac, Ian and I were discussing, before you jumped in
and started yelling. Repeating it for you:

We want to know, for dependencies that are in DEPEND and not RDEPEND,
whether or not most of them will become HDEPENDs, if dependencies are
being expressed properly. If that is the case, then it makes more sense
to introduce TDEPEND than HDEPEND.
--
Ciaran McCreesh
Michał Górny
2012-09-07 20:07:30 UTC
Permalink
On Fri, 7 Sep 2012 20:25:58 +0100
Post by Ciaran McCreesh
On Fri, 7 Sep 2012 21:21:42 +0200
Post by Michał Górny
So... what is your issue in here, sir?
The issue is what Zac, Ian and I were discussing, before you jumped in
We want to know, for dependencies that are in DEPEND and not RDEPEND,
whether or not most of them will become HDEPENDs, if dependencies are
being expressed properly. If that is the case, then it makes more
sense to introduce TDEPEND than HDEPEND.
The only person yelling here is you. I have politely asked a question,
and then you come with your wisdom not answering it at all. And if you
haven't noticed, my question was directed to Ian who -- unlike you --
may be able to say something meaningful in the topic rather than
diverging from it just to prove a random point only you know.
--
Best regards,
Michał Górny
Ciaran McCreesh
2012-09-07 20:15:03 UTC
Permalink
On Fri, 7 Sep 2012 22:07:30 +0200
Post by Michał Górny
On Fri, 7 Sep 2012 20:25:58 +0100
Post by Ciaran McCreesh
On Fri, 7 Sep 2012 21:21:42 +0200
Post by Michał Górny
So... what is your issue in here, sir?
The issue is what Zac, Ian and I were discussing, before you jumped
We want to know, for dependencies that are in DEPEND and not
RDEPEND, whether or not most of them will become HDEPENDs, if
dependencies are being expressed properly. If that is the case,
then it makes more sense to introduce TDEPEND than HDEPEND.
The only person yelling here is you. I have politely asked a question,
and then you come with your wisdom not answering it at all. And if you
haven't noticed, my question was directed to Ian who -- unlike you --
may be able to say something meaningful in the topic rather than
diverging from it just to prove a random point only you know.
Ian and I are asking essentially the same question regarding the same
issue, and I believe you're the only interested person so far who has
had any difficulty understanding that. However, if hearing it from
someone other than me makes it easier for you to accept that there is
something to discuss, then you're welcome to pretend that everything
that I say from now on was written by someone other than me.
--
Ciaran McCreesh
Ian Stakenvicius
2012-09-07 20:08:53 UTC
Permalink
Post by Ciaran McCreesh
Post by Michał Górny
Now, let me remind you because you probably fail to know the
(with HDEPEND/DEPEND) generally mean that we would need to
s/DEPEND/HDEPEND/ for the vast majority of ebuilds (ie all the
trivial ones)?
That does effectively refer to the common depends as well. You
know, in the real world where there is no magical variables which
do miracles behind your back.
Uhm, no, it doesn't. Things in both DEPEND and RDEPEND are an
*entirely* different case when it comes to destinations, since
RDEPEND goes to ROOT.
The distinction between DEPEND and HDEPEND is relevant only for
dependencies that are not also in RDEPEND.
Bringing it back to the issue it's solving:

Afaict, for migration:

- - DEPEND changes to HDEPEND
- - the new DEPEND now will be used for things that are *currently* in
RDEPEND and DEPEND (so that things will work) but are not actually
run-time dependencies. Said atoms will then be removed from RDEPEND
(and not be included in the new HDEPEND either) as they aren't really
supposed to be there in the first place.

Right?

(Note, I have no idea how this will play with PDEPEND but maybe some
of the current circular dependencies will also disappear?)
Ciaran McCreesh
2012-09-07 20:14:36 UTC
Permalink
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Fri, 07 Sep 2012 16:08:53 -0400
Post by Ian Stakenvicius
- - DEPEND changes to HDEPEND
If we're going by Chromium, AFAICS they're only making this change when
they find they actually need it to get the resolver to give "the right
answer", and otherwise leaving DEPEND as-is. This strikes me as being
heavily in Doing It Wrong territory.
Post by Ian Stakenvicius
- - the new DEPEND now will be used for things that are *currently* in
RDEPEND and DEPEND (so that things will work) but are not actually
run-time dependencies. Said atoms will then be removed from RDEPEND
(and not be included in the new HDEPEND either) as they aren't really
supposed to be there in the first place.
I'm not entirely sure that there are more than a handful of very
special cases that would be covered by the second point. Can anyone
provide examples?

- --
Ciaran McCreesh
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iEYEARECAAYFAlBKVbQACgkQ96zL6DUtXhHcUwCfdNq3MSMyYBAx19ImoOtWFMRM
l2UAoM6DfYJOCL4tLwZ3s6Jeh/6CzBCI
=FIrN
-----E
Ian Stakenvicius
2012-09-07 20:28:40 UTC
Permalink
On Fri, 07 Sep 2012 16:08:53 -0400 Ian Stakenvicius
Post by Ian Stakenvicius
- - DEPEND changes to HDEPEND
If we're going by Chromium, AFAICS they're only making this change
when they find they actually need it to get the resolver to give
"the right answer", and otherwise leaving DEPEND as-is. This
strikes me as being heavily in Doing It Wrong territory.
Post by Ian Stakenvicius
- - the new DEPEND now will be used for things that are
*currently* in RDEPEND and DEPEND (so that things will work) but
are not actually run-time dependencies. Said atoms will then be
removed from RDEPEND (and not be included in the new HDEPEND
either) as they aren't really supposed to be there in the first
place.
I'm not entirely sure that there are more than a handful of very
special cases that would be covered by the second point. Can
anyone provide examples?
Bug 263343 , the 'fontconfig' dep for some package i wasn't able to
find easily

Bug 317337 (original HDEPEND proposal) mentions the
x11-proto/xcb-proto dep for libxcb (and i assume anything else
depending on xcb-proto)
Ciaran McCreesh
2012-09-07 20:40:19 UTC
Permalink
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Fri, 07 Sep 2012 16:28:40 -0400
Post by Ian Stakenvicius
Post by Ciaran McCreesh
Post by Ian Stakenvicius
- - the new DEPEND now will be used for things that are
*currently* in RDEPEND and DEPEND (so that things will work) but
are not actually run-time dependencies. Said atoms will then be
removed from RDEPEND (and not be included in the new HDEPEND
either) as they aren't really supposed to be there in the first
place.
I'm not entirely sure that there are more than a handful of very
special cases that would be covered by the second point. Can
anyone provide examples?
Bug 263343 , the 'fontconfig' dep for some package i wasn't able to
find easily
Do we have an explanation as to *why* fontconfig has to be on ROOT
here? Is it because $ROOT/var/cache/fontconfig needs to exist at build
time, but not at runtime? If so, is this better fixed by using a
temporary directory?
Post by Ian Stakenvicius
Bug 317337 (original HDEPEND proposal) mentions the
x11-proto/xcb-proto dep for libxcb (and i assume anything else
depending on xcb-proto)
That's a BADEPEND.

- --
Ciaran McCreesh
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iEYEARECAAYFAlBKW7cACgkQ96zL6DUtXhG0iQCfdYgzb5kFYE/fN6iTAEJriyuR
zxwAoJi0PvK7JZVqaltliyJFlCZARQRj
=Tzgv
-----END PGP SIGNAT
Ian Stakenvicius
2012-09-07 19:42:05 UTC
Permalink
On Fri, 7 Sep 2012 19:31:16 +0100 Ciaran McCreesh
On Fri, 7 Sep 2012 20:23:23 +0200 Michał Górny
Post by Michał Górny
I can't agree unless I am missing something. Doesn't the
majority of ebuilds actually require most of DEPEND (well, the
part common with RDEPEND) to be installed on the target? I'm
thinking of the shared libraries mostly.
"The part common with RDEPEND" is a different issue. We're
talking about what the usual thing to do is for dependencies that
are in DEPEND but not RDEPEND. A typical example here is a binary
that is executed as part of the build process.
Thanks for keeping me on the topic.
Now, let me remind you because you probably fail to know the world
(with HDEPEND/DEPEND) generally mean that we would need to
s/DEPEND/HDEPEND/ for the vast majority of ebuilds (ie all the
trivial ones)?
That does effectively refer to the common depends as well. You
know, in the real world where there is no magical variables which
do miracles behind your back.
RDEPEND is defined in an earlier post (by Zac i think?) as target
build+run dependencies; there are very few actual target build-only
dependencies (ie, deps that need to be installed on the target and
don't need to be installed on the host, and don't need to be on the
target after build time). As such, HDEPEND since it means build-host
build-time deps will still need to have these common dependencies
specified but that doesn't have anything to do with the target's
requirements, which right now are already entirely covered by RDEPEND
(and not DEPEND) at all.
Zac Medico
2012-09-07 17:31:15 UTC
Permalink
Post by Ciaran McCreesh
On Fri, 07 Sep 2012 09:53:46 -0700
Post by Zac Medico
If you're insinuating that Portage may not have a
"fully-ROOT-and-/-aware resolver", then I can assure you that this is
not a problem.
In that case, why do we need HDEPEND at all?
With existing EAPIs, it's possible to use DEPEND for host buildtime-only
deps, RDEPEND for target build+run deps, and PDEPEND for target
runtime-only deps. However, there's no way to to specify buildtime-only
target deps (that aren't needed at runtime). The HDEPEND proposal
involves solves this by using HDEPEND for host buildtime-only deps, and
DEPEND for target buildtime-only deps.
--
Thanks,
Zac
Michael Orlitzky
2012-09-07 22:55:10 UTC
Permalink
Post by Ciaran McCreesh
Since DEPENDENCIES hasn't been written up in a Gentoo-friendly
manner, and since the Exherbo documentation doesn't seem to suffice
to explain the idea here, here's some more details on the
DEPENDENCIES proposal.
It seems to me that the problem this solves is just one of ontology.
It's analogous to trying to stick files named "foo", "bar", "baz",
etc. into directories named "depend", "rdepend", "hdepend", and so on.

There are a few well-known ways to organize things in a hierarchy, and
which one is most efficient depends on the categories and objects that
you have. Given the way that most software is built (see:
COMMON_DEPEND), I think DEPENDENCIES would work better than what we're
doing now, but it also seems more complex.

I think that dependencies are ultimately not hierarchical, and this
can force duplication in DEPENDENCIES as well. Has anyone considered
tagging the package atoms with a list of dependency types? For example,

* foo/bar: ( build run host )
* baz/one: baz? ( build )
* baz/two, baz/three: baz? ( build run )
...

This would eliminate duplication of the objects (package atoms) in
favor of duplication of the categories (dependency types). Since the
package atoms are what we really care about, I think the tradeoff is
beneficial. Maintainers get to express each dependency exactly once.
Ciaran McCreesh
2012-09-08 06:43:58 UTC
Permalink
On Fri, 07 Sep 2012 18:55:10 -0400
Post by Michael Orlitzky
I think that dependencies are ultimately not hierarchical
Situations like foo? ( bar? ( || ( a ( b c ) ) ) ) do happen, so any
new syntax would have to be able to deal with that.
--
Ciaran McCreesh
Michael Orlitzky
2012-09-08 13:01:48 UTC
Permalink
On Fri, 07 Sep 2012 18:55:10 -0400 Michael Orlitzky
Post by Michael Orlitzky
I think that dependencies are ultimately not hierarchical
Situations like foo? ( bar? ( || ( a ( b c ) ) ) ) do happen, so
any new syntax would have to be able to deal with that.
The deps in both cases are just a collection of atom/type pairs, so
anything possible in one must be possible in the other. I think this
means, if USE=bar, then we need either a or (b and c)? It could be
written,

|| (
a: bar? ( build run )
b,c: bar? ( build run )
)

Or if we wanted to make it even easier, allow the USE conditional at
the top level like we do now:

bar? ( || (
a: ( build run )
b,c: ( build run )
))

I'm just wondering if it wouldn't be nicer to think in terms of
package atoms instead of the dependency types. Right now, we've got
buckets named DEPEND, RDEPEND, etc. and we put the package atoms in
those buckets. The above syntax would make the package atoms the
buckets, and we would be putting the dependency types into the buckets
instead.
Michał Górny
2012-09-08 07:27:15 UTC
Permalink
On Fri, 07 Sep 2012 18:55:10 -0400
Post by Michael Orlitzky
Post by Ciaran McCreesh
Since DEPENDENCIES hasn't been written up in a Gentoo-friendly
manner, and since the Exherbo documentation doesn't seem to suffice
to explain the idea here, here's some more details on the
DEPENDENCIES proposal.
It seems to me that the problem this solves is just one of ontology.
It's analogous to trying to stick files named "foo", "bar", "baz",
etc. into directories named "depend", "rdepend", "hdepend", and so on.
There are a few well-known ways to organize things in a hierarchy, and
which one is most efficient depends on the categories and objects that
COMMON_DEPEND), I think DEPENDENCIES would work better than what we're
doing now, but it also seems more complex.
I think that dependencies are ultimately not hierarchical, and this
can force duplication in DEPENDENCIES as well. Has anyone considered
tagging the package atoms with a list of dependency types? For
example,
* foo/bar: ( build run host )
* baz/one: baz? ( build )
* baz/two, baz/three: baz? ( build run )
...
This would eliminate duplication of the objects (package atoms) in
favor of duplication of the categories (dependency types). Since the
package atoms are what we really care about, I think the tradeoff is
beneficial. Maintainers get to express each dependency exactly once.
This is nowhere near friendly to a developer...
--
Best regards,
Michał Górny
Patrick Lauer
2012-09-08 01:02:39 UTC
Permalink
Post by Ciaran McCreesh
Since DEPENDENCIES hasn't been written up in a Gentoo-friendly manner,
and since the Exherbo documentation doesn't seem to suffice to explain
the idea here, here's some more details on the DEPENDENCIES proposal.
There's change, and there's progress.

This is change, just to have things change.

Like the /usr move, merging udev into some random package no one uses,
and some other things that people do to keep their fingers busy.


Can we please focus on progress instead?
Matt Turner
2012-09-09 03:32:46 UTC
Permalink
On Fri, Sep 7, 2012 at 4:45 AM, Ciaran McCreesh
Post by Ciaran McCreesh
Since DEPENDENCIES hasn't been written up in a Gentoo-friendly manner,
and since the Exherbo documentation doesn't seem to suffice to explain
the idea here, here's some more details on the DEPENDENCIES proposal.
I like this. It seems well planned and thought out and very flexible.
If I understand correctly paludis already supports this, so a working
implementation is already available which is clearly beneficial for a
proposal.

Thanks a lot, Ciaran, for writing this up.

Matt
Loading...