[tycho-user] Why is Tycho manifest-first?

[Aaron Digulla]

Hello,

I'm wondering why you're using p2 technology (with all it's drawbacks)
instead of using Maven's fast&reliable dependency resolution to create a
MANIFEST.MF from that?

Wouldn't it be much more simple to add a manifest builder plugin which I
can configure from a POM that produces a MANIFEST.MF?

Regards,

[Igor Fedorenko]

There is a plugin for that already [1].

[1] http://felix.apache.org/site/apache-felix-maven-bundle-plugin-bnd.html

--
Regards,
Igor

Post by Aaron Digulla
Hello,
I'm wondering why you're using p2 technology (with all it's drawbacks)
instead of using Maven's fast&reliable dependency resolution to create a
MANIFEST.MF from that?
Wouldn't it be much more simple to add a manifest builder plugin which I
can configure from a POM that produces a MANIFEST.MF?
Regards,

[Max Rydahl Andersen]

…manifest-first makes it much easier to migrate to from PDE-build and still be able to work with Eclipse PDE without additional overhead.

/max

Post by Aaron Digulla
Hello,
I'm wondering why you're using p2 technology (with all it's drawbacks) instead of using Maven's fast&reliable dependency resolution to create a MANIFEST.MF from that?
Wouldn't it be much more simple to add a manifest builder plugin which I can configure from a POM that produces a MANIFEST.MF?
Regards,
--
Aaron "Optimizer" Digulla a.k.a. Philmann Dark
"It's not the universe that's limited, it's our imagination.
Follow me and I'll show you something beyond the limits."
http://blog.pdark.de/
_______________________________________________
tycho-user mailing list
https://dev.eclipse.org/mailman/listinfo/tycho-user

/max
http://about.me/maxandersen

[Igor Fedorenko]

FWIW, latest m2e-tycho lets you work on pom-first projects using Eclipse
PDE without additional overhead either, so I would not use this to
choose between pom-first and manifest first.

The real difference comes from dependencies. If most of project
dependencies come from p2 repositories, then manifest-first makes
dependency resolution easier, if not the only viable option. If,
however, all project dependencies are available from Maven repositories,
then pom-first works much better.

Developer background and personal preferences also play a role here.
Personally, I find it easier to have bundle manifest generated for me,
but I know other developers prefer maintaing manifests by hand. One
of the crazier ideas I recently had was to rework Eclipse Platform build
using pom-first approach, so that entire eclipse platform becomes
natively available from a Maven repository. I gave up on this idea
mostly because I did not see how to convince platform developers let a
tool generate bundle manifests for them.

--
Regards,
Igor

Post by Max Rydahl Andersen
…manifest-first makes it much easier to migrate to from PDE-build and still be able to work with Eclipse PDE without additional overhead.
/max

Post by Aaron Digulla
Hello,
I'm wondering why you're using p2 technology (with all it's drawbacks) instead of using Maven's fast&reliable dependency resolution to create a MANIFEST.MF from that?
Wouldn't it be much more simple to add a manifest builder plugin which I can configure from a POM that produces a MANIFEST.MF?
Regards,
--
Aaron "Optimizer" Digulla a.k.a. Philmann Dark
"It's not the universe that's limited, it's our imagination.
Follow me and I'll show you something beyond the limits."
http://blog.pdark.de/
_______________________________________________
tycho-user mailing list
https://dev.eclipse.org/mailman/listinfo/tycho-user

/max
http://about.me/maxandersen
_______________________________________________
tycho-user mailing list
https://dev.eclipse.org/mailman/listinfo/tycho-user

[Hamza Soussou]

when you deal with package dependencies (import-package in manifest.mf),
which would be better: pom-first or manifest-first? I would vote for
manifest-first approach

Maybe one of the drawbacks of p2 repositories is you can't distinguish
between releases and snapshots

Post by Igor Fedorenko
FWIW, latest m2e-tycho lets you work on pom-first projects using Eclipse
PDE without additional overhead either, so I would not use this to
choose between pom-first and manifest first.
The real difference comes from dependencies. If most of project
dependencies come from p2 repositories, then manifest-first makes
dependency resolution easier, if not the only viable option. If,
however, all project dependencies are available from Maven repositories,
then pom-first works much better.
Developer background and personal preferences also play a role here.
Personally, I find it easier to have bundle manifest generated for me,
but I know other developers prefer maintaing manifests by hand. One
of the crazier ideas I recently had was to rework Eclipse Platform build
using pom-first approach, so that entire eclipse platform becomes
natively available from a Maven repository. I gave up on this idea
mostly because I did not see how to convince platform developers let a
tool generate bundle manifests for them.
--
Regards,
Igor

Post by Max Rydahl Andersen
…manifest-first makes it much easier to migrate to from PDE-build and
still be able to work with Eclipse PDE without additional overhead.
/max

Post by Aaron Digulla
Hello,
I'm wondering why you're using p2 technology (with all it's
drawbacks) instead of using Maven's fast&reliable dependency
resolution to create a MANIFEST.MF from that?
Wouldn't it be much more simple to add a manifest builder plugin
which I can configure from a POM that produces a MANIFEST.MF?
Regards,
--
Aaron "Optimizer" Digulla a.k.a. Philmann Dark
"It's not the universe that's limited, it's our imagination.
Follow me and I'll show you something beyond the limits."
http://blog.pdark.de/
_______________________________________________
tycho-user mailing list
https://dev.eclipse.org/mailman/listinfo/tycho-user

/max
http://about.me/maxandersen
_______________________________________________
tycho-user mailing list
https://dev.eclipse.org/mailman/listinfo/tycho-user

_______________________________________________
tycho-user mailing list
https://dev.eclipse.org/mailman/listinfo/tycho-user

[Eric Gwin]

Maybe one of the drawbacks of p2 repositories is you can't distinguish between releases and snapshots

You can if the P2 repos is release, milestone, or snapshot specific.

Post by Igor Fedorenko
FWIW, latest m2e-tycho lets you work on pom-first projects using Eclipse
PDE without additional overhead either, so I would not use this to
choose between pom-first and manifest first.
The real difference comes from dependencies. If most of project
dependencies come from p2 repositories, then manifest-first makes
dependency resolution easier, if not the only viable option. If,
however, all project dependencies are available from Maven repositories,
then pom-first works much better.
Developer background and personal preferences also play a role here.
Personally, I find it easier to have bundle manifest generated for me,
but I know other developers prefer maintaing manifests by hand. One
of the crazier ideas I recently had was to rework Eclipse Platform build
using pom-first approach, so that entire eclipse platform becomes
natively available from a Maven repository. I gave up on this idea
mostly because I did not see how to convince platform developers let a
tool generate bundle manifests for them.
--
Regards,
Igor

Post by Max Rydahl Andersen
…manifest-first makes it much easier to migrate to from PDE-build and
still be able to work with Eclipse PDE without additional overhead.
/max

Post by Aaron Digulla
Hello,
I'm wondering why you're using p2 technology (with all it's
drawbacks) instead of using Maven's fast&reliable dependency
resolution to create a MANIFEST.MF from that?
Wouldn't it be much more simple to add a manifest builder plugin
which I can configure from a POM that produces a MANIFEST.MF?
Regards,
--
Aaron "Optimizer" Digulla a.k.a. Philmann Dark
"It's not the universe that's limited, it's our imagination.
Follow me and I'll show you something beyond the limits."
http://blog.pdark.de/
_______________________________________________
tycho-user mailing list
https://dev.eclipse.org/mailman/listinfo/tycho-user

/max
http://about.me/maxandersen
_______________________________________________
tycho-user mailing list
https://dev.eclipse.org/mailman/listinfo/tycho-user

_______________________________________________
tycho-user mailing list
https://dev.eclipse.org/mailman/listinfo/tycho-user

_______________________________________________
tycho-user mailing list
https://dev.eclipse.org/mailman/listinfo/tycho-user

[Aaron Digulla]

Post by Hamza Soussou
when you deal with package dependencies (import-package in
manifest.mf), which would be better: pom-first or manifest-first? I
would vote for manifest-first approach

I prefer an approach which allows repeatable builds. When I build
something, I want to be sure what I get. This is very hard to do right
with p2.

Regards,

[Mickael Istria]

Hi,

MANIFEST.MF is the file that is used at dev-time by PDE and at runtime
by your OSGi Container to deal with modularization. Maven is build-time
modularization, based on pom.xml to express dependencies. Then the
general issue with Maven and OSGi is that there are 2 ways to express
dependencies. Tycho probably made the right choice choosing the approach
that is the state-of-the-art: relying on MANIFEST.MF.
If you want to have a pom-first approach, how do you deal with PDE at
dev-time without a good MANIFEST.MF?

Supporting files such as build.properties is quite convenient for
everybody in the Eclipse world. Without that, I think Tycho would not
have been so successful at Eclipse. But since these files are originally
specific to the PDE build, I think and I hope they'll tend to disappear
to have build customization in pom.xml instead of build.properties.

About p2, I can tell you that there is not any drawback when dealing
with Eclipse projects builds. Maybe you should elaborate more your
specific use-case to teach us the limitations some people (including me)
are not aware of.

My 2c.

Post by Aaron Digulla

Post by Hamza Soussou
when you deal with package dependencies (import-package in
manifest.mf), which would be better: pom-first or manifest-first? I
would vote for manifest-first approach

I prefer an approach which allows repeatable builds. When I build
something, I want to be sure what I get. This is very hard to do right
with p2.
Regards,

[Aaron Digulla]

Post by Mickael Istria
Hi,
MANIFEST.MF is the file that is used at dev-time by PDE and at
runtime by your OSGi Container to deal with modularization. Maven is
build-time modularization, based on pom.xml to express dependencies.
Then the general issue with Maven and OSGi is that there are 2 ways
to express dependencies. Tycho probably made the right choice
choosing the approach that is the state-of-the-art: relying on
MANIFEST.MF.
If you want to have a pom-first approach, how do you deal with PDE
at dev-time without a good MANIFEST.MF?

Simple: I'm not using PDE if I can avoid it. While the Maven
integration in Eclipse isn't perfect (for various reasons), PDE with
its infinitely recursive Ant scripts is worse.

So far, I have tried to build these projects with PDE: Equinox, SWT,
Platform, Console. I failed every time because of some obscure error
message in line 15000 of some Ant script at recursion level 97.
Enabling debug gives me a nice 150MB log file which is too big to
understand.

Error messages usually are incomprehensible to me and any Eclipse
developer that I asked.

For example, it took me 4 hours to set up a BIRT build using Maven
(that includes finding all the sources and removing any trace of PDE).
That build takes exactly 53 seconds for 4000 source files. The build
creates a working bundle and I know exactly what goes into the build.
No p2 repo is considered (see below), and no PDE script/tool is used.

My build is stable, it doesn't depend on some CVS server on the other
side of the planet, I know what it is doing, it's fast and reliable.

When I tried to build the same project with PDE, I ran into these problems:

1. The build only works on a single machine because the PDE scripts
contain absolute paths. For some reason, -Dkey=value is sometimes
ignored. All my attempts to override the paths didn't work. Even when
I changed the file, the build would checkout the files from the CVS
server and overwrite my changes.

2. The CVS checkout URLs only work inside the Eclipse network.

3. During the build, I saw error messages that no one ever saw before.
A developer from the BIRT team tried to help but he simply didn't know
what the error message meant or how to fix it. It was a typical "but
it works on my machine" situation.

4. Unlike Maven projects, PDE projects always depend on external
factors like the Target Platform. There is no simple, sure-fire way to
create a PDE project which will build out of the box for everyone.

The main problem with PDE is that it gives developers too many options.

On top of that, I haven't found a way to build a site with exactly the
version of features/plugins that I want. Even if it works perfectly,
all plugin qualifiers get updated even though the plugin code didn't
change.

In my view, one can argue that MANIFEST.MF is "state-of-the-art". But
PDE is definitely not.

I tried Buckminster but wasn't too impressed. Then we have lots of
small bugs in the PDE UI (menus sometime don't do anything without an
error, incomplete builds, the target file editor is very slow,
sometimes adding a feature doesn't work, plugins can't be found, ...)

Post by Mickael Istria
Supporting files such as build.properties is quite convenient for
everybody in the Eclipse world. Without that, I think Tycho would
not have been so successful at Eclipse. But since these files are
originally specific to the PDE build, I think and I hope they'll
tend to disappear to have build customization in pom.xml instead of
build.properties.
About p2, I can tell you that there is not any drawback when dealing
with Eclipse projects builds. Maybe you should elaborate more your
specific use-case to teach us the limitations some people (including
me) are not aware of.

I have several concerns with p2:

- Unlike Maven Central, p2 repositories are very unreliable. Someone
uploads a new version of a plugin and my builds change.

- If you're not in the US, starting a Maven build which uses the
official Indigo repo can take up to 15 minutes for the connection
phase alone! This is unacceptable. To drive this home: "mvn clean"
takes 15 minutes!!!

[Mickael Istria]

Post by Aaron Digulla

Post by Mickael Istria
Hi,
MANIFEST.MF is the file that is used at dev-time by PDE and at
runtime by your OSGi Container to deal with modularization. Maven is
build-time modularization, based on pom.xml to express dependencies.
Then the general issue with Maven and OSGi is that there are 2 ways
to express dependencies. Tycho probably made the right choice
choosing the approach that is the state-of-the-art: relying on
MANIFEST.MF.
If you want to have a pom-first approach, how do you deal with PDE at
dev-time without a good MANIFEST.MF?

Simple: I'm not using PDE if I can avoid it. While the Maven
integration in Eclipse isn't perfect (for various reasons), PDE with
its infinitely recursive Ant scripts is worse.

I think you are confusing PDE (IDE Tooling) and PDE Build.
Also I am not sure of what you mean by "worse". This is a contextual
word, without the context, it is not easy to get what are the concrete
issues ;)

Post by Aaron Digulla
So far, I have tried to build these projects with PDE: Equinox, SWT,
Platform, Console. I failed every time because of some obscure error
message in line 15000 of some Ant script at recursion level 97.
Enabling debug gives me a nice 150MB log file which is too big to
understand.
Error messages usually are incomprehensible to me and any Eclipse
developer that I asked.

That's true, and that's why Tycho is meant for: replace PDE build with a
more transparent build system. I think it is no more time to say the PDE
Build is not that good. PDE Build is past, everybody on this mailing
list agrees with it.

Post by Aaron Digulla
For example, it took me 4 hours to set up a BIRT build using Maven
(that includes finding all the sources and removing any trace of PDE).
That build takes exactly 53 seconds for 4000 source files. The build
creates a working bundle and I know exactly what goes into the build.
No p2 repo is considered (see below), and no PDE script/tool is used.

Ok.

Post by Aaron Digulla
My build is stable, it doesn't depend on some CVS server on the other
side of the planet, I know what it is doing, it's fast and reliable.

So is a Tycho-based build. It does not rely on a CVS server; it relies
on p2 repositories; you know what it is doing, it is reliable. But IMHO,
it is still much easier as a plugin developer to have MANIFEST-first
approach, because the tooling (IDE) works well doing so. Also the
MANIFEST.MF is a very important file for the developer, I prefer editing
it than having it generated.

Post by Aaron Digulla
1. The build only works on a single machine because the PDE scripts
contain absolute paths. For some reason, -Dkey=value is sometimes
ignored. All my attempts to override the paths didn't work. Even when
I changed the file, the build would checkout the files from the CVS
server and overwrite my changes.
2. The CVS checkout URLs only work inside the Eclipse network.
3. During the build, I saw error messages that no one ever saw before.
A developer from the BIRT team tried to help but he simply didn't know
what the error message meant or how to fix it. It was a typical "but
it works on my machine" situation.
4. Unlike Maven projects, PDE projects always depend on external
factors like the Target Platform. There is no simple, sure-fire way to
create a PDE project which will build out of the box for everyone.
The main problem with PDE is that it gives developers too many options.
On top of that, I haven't found a way to build a site with exactly the
version of features/plugins that I want. Even if it works perfectly,
all plugin qualifiers get updated even though the plugin code didn't
change.
In my view, one can argue that MANIFEST.MF is "state-of-the-art". But
PDE is definitely not.

Ok, PDE *Build* is now bad. I agree. But I meant PDE Tooling in the IDE.

Post by Aaron Digulla
- Unlike Maven Central, p2 repositories are very unreliable. Someone
uploads a new version of a plugin and my builds change.

That's true, it is up to project providing repositories to set up a
policy to have reliable p2 repositories. However, having your build
changing is not that important IMHO, because if the p2 repository is
updated, you have a build that changes, but it follows update of
different projects. Think about how your end-users will consume your
product: in Eclipse, if they use a p2 repository to install it, they'll
probably have also the latest versions of other repositories. As a
plugin developer, I am pretty happy to see my build automatically
switching to the maintenance release of Indigo because the content of
the repo changed. It ensures me people who get a latest release of
Eclipse will see the project working.

Post by Aaron Digulla
- If you're not in the US, starting a Maven build which uses the
official Indigo repo can take up to 15 minutes for the connection
phase alone! This is unacceptable. To drive this home: "mvn clean"
takes 15 minutes!!!

That's true. I've never reached 15 minutes, but that's long. Maybe the
idea behing this issue is the lack of mirroring support in p2 or Maven
in general. You should open a bug against p2 explaining that an install
should be able to delegate downloads to a mirror. That would be a
valuable improvement.

[Igor Fedorenko]

None of the problems you describe are inherent limitations of
manifest-first approach. It can be argued that pom-first is slightly
better at managing compile-time dependencies while manifest-first does a
slightly better job during integration testing, but neither has decisive
advantage. If project dependencies are available in both maven and p2
format, then choice between pom-first and manifest-first is down to
personal preferences and historical reasons, like pre-existing
hand-written manifests, like was the case in JBossTools for example.

--
Regards,
Igor

Post by Mickael Istria
Hi,
MANIFEST.MF is the file that is used at dev-time by PDE and at runtime
by your OSGi Container to deal with modularization. Maven is
build-time modularization, based on pom.xml to express dependencies.
Then the general issue with Maven and OSGi is that there are 2 ways to
express dependencies. Tycho probably made the right choice choosing
the approach that is the state-of-the-art: relying on MANIFEST.MF.
If you want to have a pom-first approach, how do you deal with PDE at
dev-time without a good MANIFEST.MF?

Simple: I'm not using PDE if I can avoid it. While the Maven integration
in Eclipse isn't perfect (for various reasons), PDE with its infinitely
recursive Ant scripts is worse.
So far, I have tried to build these projects with PDE: Equinox, SWT,
Platform, Console. I failed every time because of some obscure error
message in line 15000 of some Ant script at recursion level 97. Enabling
debug gives me a nice 150MB log file which is too big to understand.
Error messages usually are incomprehensible to me and any Eclipse
developer that I asked.
For example, it took me 4 hours to set up a BIRT build using Maven (that
includes finding all the sources and removing any trace of PDE). That
build takes exactly 53 seconds for 4000 source files. The build creates
a working bundle and I know exactly what goes into the build. No p2 repo
is considered (see below), and no PDE script/tool is used.
My build is stable, it doesn't depend on some CVS server on the other
side of the planet, I know what it is doing, it's fast and reliable.
1. The build only works on a single machine because the PDE scripts
contain absolute paths. For some reason, -Dkey=value is sometimes
ignored. All my attempts to override the paths didn't work. Even when I
changed the file, the build would checkout the files from the CVS server
and overwrite my changes.
2. The CVS checkout URLs only work inside the Eclipse network.
3. During the build, I saw error messages that no one ever saw before. A
developer from the BIRT team tried to help but he simply didn't know
what the error message meant or how to fix it. It was a typical "but it
works on my machine" situation.
4. Unlike Maven projects, PDE projects always depend on external factors
like the Target Platform. There is no simple, sure-fire way to create a
PDE project which will build out of the box for everyone.
The main problem with PDE is that it gives developers too many options.
On top of that, I haven't found a way to build a site with exactly the
version of features/plugins that I want. Even if it works perfectly, all
plugin qualifiers get updated even though the plugin code didn't change.
In my view, one can argue that MANIFEST.MF is "state-of-the-art". But
PDE is definitely not.
I tried Buckminster but wasn't too impressed. Then we have lots of small
bugs in the PDE UI (menus sometime don't do anything without an error,
incomplete builds, the target file editor is very slow, sometimes adding
a feature doesn't work, plugins can't be found, ...)

Post by Mickael Istria
Supporting files such as build.properties is quite convenient for
everybody in the Eclipse world. Without that, I think Tycho would not
have been so successful at Eclipse. But since these files are
originally specific to the PDE build, I think and I hope they'll tend
to disappear to have build customization in pom.xml instead of
build.properties.
About p2, I can tell you that there is not any drawback when dealing
with Eclipse projects builds. Maybe you should elaborate more your
specific use-case to teach us the limitations some people (including
me) are not aware of.

- Unlike Maven Central, p2 repositories are very unreliable. Someone
uploads a new version of a plugin and my builds change.
- If you're not in the US, starting a Maven build which uses the
official Indigo repo can take up to 15 minutes for the connection phase
alone! This is unacceptable. To drive this home: "mvn clean" takes 15
minutes!!!

[Eric Jain]

- Unlike Maven Central, p2 repositories are very unreliable. Someone uploads
a new version of a plugin and my builds change.

I don't see this as an issue with p2 itself, but with specific
projects that don't provide stable update sites (i.e. the update sites
contains just the latest n versions; even within eclipse.org there is
no consistent approach), and the fact that there is nothing like Maven
Central for p2.

Note that a target platform can require specific versions, so the
build will fail rather than get an untested version.

One solution is to install dependencies into a local p2 repository;
unfortunately I haven't found a good way to manage such repositories
(Nexus Pro doesn't appear to help, and using the p2 director or Tycho
has issues as well:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=360474).

- If you're not in the US, starting a Maven build which uses the official
Indigo repo can take up to 15 minutes for the connection phase alone! This
is unacceptable. To drive this home: "mvn clean" takes 15 minutes!!!

Are you sure the slowness is due to the repository being remote? See
https://bugs.eclipse.org/bugs/show_bug.cgi?id=356888.

[Igor Fedorenko]

What parts of p2 make reproducible builds harder?

--
Regards,
Igor

Post by Aaron Digulla

Post by Hamza Soussou
when you deal with package dependencies (import-package in
manifest.mf), which would be better: pom-first or manifest-first? I
would vote for manifest-first approach

I prefer an approach which allows repeatable builds. When I build
something, I want to be sure what I get. This is very hard to do right
with p2.
Regards,

[Aaron Digulla]

This post might be inappropriate. Click to display it.

[Igor Fedorenko]

No offense, but this sounds like a badly setup project, there is nothing
in p2 or tycho that prevents reproducible builds.

--
Regards,
Igor

Post by Aaron Digulla

Post by Igor Fedorenko
What parts of p2 make reproducible builds harder?

It's a set of features and missing documentation.
It starts with how OSGi treats versions. With Maven, 2.0.0 means 2.0.0
and nothing else. In OSGi/PDE/p2, it means 2.0.0 or better. If you're
not careful, you'll build against a 3.0 or 4.0 or 5.0. If that happens,
you won't notice unless the API has changed and you get compile errors.
If you get compile errors, that still doesn't tell you much because the
MANIFEST says "version=2.0.0".
The we have the features which bundle bundles. Which version of the
bundle is going into the feature? Probably the latest but that's just a
guess by me - I don't know for sure.
Then we have the site which is built from features. When I say "Build
All", I sometimes get new JAR files for every bundle in my workspace,
sometimes only for a few. All of the time, the ".qualifier" in the
version of my feature will be replaced with a timestamp but I don't
understand why. Why is PDE creating new versions of all bundles
sometimes and sometimes just a few? In the end, I always clean the site
project, just to be sure.
This all amounts to a build that I can't trust because I don't know what
goes into it, how I can control it, or how I can find out how to control
it. The documentation in that regard follows the approach "it just
works" but I want to understand what is going on.
Regards,

Post by Igor Fedorenko
--
Regards,
Igor

Post by Aaron Digulla

Post by Hamza Soussou
when you deal with package dependencies (import-package in
manifest.mf), which would be better: pom-first or manifest-first? I
would vote for manifest-first approach

I prefer an approach which allows repeatable builds. When I build
something, I want to be sure what I get. This is very hard to do right
with p2.
Regards,

_______________________________________________
tycho-user mailing list
https://dev.eclipse.org/mailman/listinfo/tycho-user

[Aaron Digulla]

Post by Igor Fedorenko
No offense, but this sounds like a badly setup project, there is nothing
in p2 or tycho that prevents reproducible builds.

Let me put it differently: Maven is easy to configure to get
reproducible builds. With p2 and OSGi, you need to know a lot about
version management and p2 repositories and it can still break.

Regards,

Post by Igor Fedorenko
--
Regards,
Igor

Post by Aaron Digulla

Post by Igor Fedorenko
What parts of p2 make reproducible builds harder?

It's a set of features and missing documentation.
It starts with how OSGi treats versions. With Maven, 2.0.0 means 2.0.0
and nothing else. In OSGi/PDE/p2, it means 2.0.0 or better. If you're
not careful, you'll build against a 3.0 or 4.0 or 5.0. If that happens,
you won't notice unless the API has changed and you get compile errors.
If you get compile errors, that still doesn't tell you much because the
MANIFEST says "version=2.0.0".
The we have the features which bundle bundles. Which version of the
bundle is going into the feature? Probably the latest but that's just a
guess by me - I don't know for sure.
Then we have the site which is built from features. When I say "Build
All", I sometimes get new JAR files for every bundle in my workspace,
sometimes only for a few. All of the time, the ".qualifier" in the
version of my feature will be replaced with a timestamp but I don't
understand why. Why is PDE creating new versions of all bundles
sometimes and sometimes just a few? In the end, I always clean the site
project, just to be sure.
This all amounts to a build that I can't trust because I don't know what
goes into it, how I can control it, or how I can find out how to control
it. The documentation in that regard follows the approach "it just
works" but I want to understand what is going on.
Regards,

Post by Igor Fedorenko
--
Regards,
Igor

Post by Aaron Digulla

Post by Hamza Soussou
when you deal with package dependencies (import-package in
manifest.mf), which would be better: pom-first or manifest-first? I
would vote for manifest-first approach

I prefer an approach which allows repeatable builds. When I build
something, I want to be sure what I get. This is very hard to do right
with p2.
Regards,

_______________________________________________
tycho-user mailing list
https://dev.eclipse.org/mailman/listinfo/tycho-user

_______________________________________________
tycho-user mailing list
https://dev.eclipse.org/mailman/listinfo/tycho-user

[Igor Fedorenko]

Yes, you need to know OSGi version syntax to work on OSGi projects.
Beyond that, though, there is a rough parity between Maven and Tycho
when it comes to reproducible builds.

--
Regards,
Igor

Post by Aaron Digulla

Post by Igor Fedorenko
No offense, but this sounds like a badly setup project, there is nothing
in p2 or tycho that prevents reproducible builds.

Let me put it differently: Maven is easy to configure to get
reproducible builds. With p2 and OSGi, you need to know a lot about
version management and p2 repositories and it can still break.
Regards,

Post by Igor Fedorenko
--
Regards,
Igor

Post by Aaron Digulla

Post by Igor Fedorenko
What parts of p2 make reproducible builds harder?

It's a set of features and missing documentation.
It starts with how OSGi treats versions. With Maven, 2.0.0 means 2.0.0
and nothing else. In OSGi/PDE/p2, it means 2.0.0 or better. If you're
not careful, you'll build against a 3.0 or 4.0 or 5.0. If that happens,
you won't notice unless the API has changed and you get compile errors.
If you get compile errors, that still doesn't tell you much because the
MANIFEST says "version=2.0.0".
The we have the features which bundle bundles. Which version of the
bundle is going into the feature? Probably the latest but that's just a
guess by me - I don't know for sure.
Then we have the site which is built from features. When I say "Build
All", I sometimes get new JAR files for every bundle in my workspace,
sometimes only for a few. All of the time, the ".qualifier" in the
version of my feature will be replaced with a timestamp but I don't
understand why. Why is PDE creating new versions of all bundles
sometimes and sometimes just a few? In the end, I always clean the site
project, just to be sure.
This all amounts to a build that I can't trust because I don't know what
goes into it, how I can control it, or how I can find out how to control
it. The documentation in that regard follows the approach "it just
works" but I want to understand what is going on.
Regards,

Post by Igor Fedorenko
--
Regards,
Igor

Post by Aaron Digulla

Post by Hamza Soussou
when you deal with package dependencies (import-package in
manifest.mf), which would be better: pom-first or manifest-first? I
would vote for manifest-first approach

I prefer an approach which allows repeatable builds. When I build
something, I want to be sure what I get. This is very hard to do right
with p2.
Regards,

_______________________________________________
tycho-user mailing list
https://dev.eclipse.org/mailman/listinfo/tycho-user

_______________________________________________
tycho-user mailing list
https://dev.eclipse.org/mailman/listinfo/tycho-user

[Eric Jain]

Post by Igor Fedorenko
Yes, you need to know OSGi version syntax to work on OSGi projects.
Beyond that, though, there is a rough parity between Maven and Tycho
when it comes to reproducible builds.

In theory...

In practice, many Maven dependencies are available from Maven Central,
which is more reliable than depending on various p2 repositories with
different policies.

Also, maintaining a local Maven repository for 3rd party artifacts
isn't a big deal, but I have yet to figure out how to maintain a p2
repository for the same purpose (Nexus Pro, for instance, doesn't
appear to support this).

[Hamza Soussou]

Post by Aaron Digulla

Post by Igor Fedorenko
No offense, but this sounds like a badly setup project, there is nothing
in p2 or tycho that prevents reproducible builds.

Let me put it differently: Maven is easy to configure to get
reproducible builds. With p2 and OSGi, you need to know a lot about
version management and p2 repositories and it can still break.

it took me a while to switch from old plain compilation environments to
Maven since I though that Maven was not easy to understand with its
dependency, plugins, snapshot/release repositories ...
take a deeper look at p2 and you may change your mind ;)

Post by Aaron Digulla
Regards,

Post by Igor Fedorenko
--
Regards,
Igor

Post by Aaron Digulla

Post by Igor Fedorenko
What parts of p2 make reproducible builds harder?

It's a set of features and missing documentation.
It starts with how OSGi treats versions. With Maven, 2.0.0 means 2.0.0
and nothing else. In OSGi/PDE/p2, it means 2.0.0 or better. If you're
not careful, you'll build against a 3.0 or 4.0 or 5.0. If that happens,
you won't notice unless the API has changed and you get compile errors.
If you get compile errors, that still doesn't tell you much because the
MANIFEST says "version=2.0.0".
The we have the features which bundle bundles. Which version of the
bundle is going into the feature? Probably the latest but that's just a
guess by me - I don't know for sure.
Then we have the site which is built from features. When I say "Build
All", I sometimes get new JAR files for every bundle in my workspace,
sometimes only for a few. All of the time, the ".qualifier" in the
version of my feature will be replaced with a timestamp but I don't
understand why. Why is PDE creating new versions of all bundles
sometimes and sometimes just a few? In the end, I always clean the site
project, just to be sure.
This all amounts to a build that I can't trust because I don't know what
goes into it, how I can control it, or how I can find out how to control
it. The documentation in that regard follows the approach "it just
works" but I want to understand what is going on.
Regards,

Post by Igor Fedorenko
--
Regards,
Igor

Post by Aaron Digulla

Post by Hamza Soussou
when you deal with package dependencies (import-package in
manifest.mf), which would be better: pom-first or manifest-first? I
would vote for manifest-first approach

I prefer an approach which allows repeatable builds. When I build
something, I want to be sure what I get. This is very hard to do right
with p2.
Regards,

_______________________________________________
tycho-user mailing list
https://dev.eclipse.org/mailman/listinfo/tycho-user

_______________________________________________
tycho-user mailing list
https://dev.eclipse.org/mailman/listinfo/tycho-user

[Hamza Soussou]

Post by Aaron Digulla

Post by Hamza Soussou
when you deal with package dependencies (import-package in
manifest.mf), which would be better: pom-first or manifest-first? I
would vote for manifest-first approach

I prefer an approach which allows repeatable builds. When I build
something, I want to be sure what I get. This is very hard to do right
with p2.

you can make reproductible builds by setting a strict version to your
dependencies [1.0.0.RELEASE,1.0.0.RELEASE] for example (and this is the
same thing when you deal with maven dependencies: you need to specify a
strict dependency to a given version)
and as you can notice, manifest-first approach can allow you to depend
on a single version or a range of versions

Post by Aaron Digulla
Regards,

[Dahanne, Anthony]

Igor,
This crazy idea sounds so cool ;-)
That would be awesome !
It reminds of a war project (built with Maven) depending on OSGi bundles (built with Tycho) only published to p2 repositories...

Is there a way for this war project to consume those already built OSGi bundles ?

(for now I only use Tycho to build equinox plugins and features...)

Regards,
Anthony




 
Le contenu de ce courriel s'adresse au destinataire seulement. Il contient de l'information pouvant être confidentielle. Vous ne devez ni le copier ni l'utiliser ni le divulguer à qui que ce soit à moins que vous soyez le destinataire ou une personne désignée autorisée. Si vous le receviez par erreur, veuillez nous aviser immédiatement et le détruire.
 
The contents of this e-mail are intended for the named addressee only. It contains information that may be confidential. Unless you are the named addressee or an authorized designee, you may not copy or use it, or disclose it to anyone else. If you received it in error please notify us immediately and then destroy it.
 

De : tycho-user-***@eclipse.org [mailto:tycho-user-***@eclipse.org] De la part de Igor Fedorenko
Envoyé : 16 novembre 2011 16:10
À : tycho-***@eclipse.org
Objet : Re: [tycho-user] Why is Tycho manifest-first?

FWIW, latest m2e-tycho lets you work on pom-first projects using Eclipse
PDE without additional overhead either, so I would not use this to
choose between pom-first and manifest first.

The real difference comes from dependencies. If most of project
dependencies come from p2 repositories, then manifest-first makes
dependency resolution easier, if not the only viable option. If,
however, all project dependencies are available from Maven repositories,
then pom-first works much better.

Developer background and personal preferences also play a role here.
Personally, I find it easier to have bundle manifest generated for me,
but I know other developers prefer maintaing manifests by hand. One
of the crazier ideas I recently had was to rework Eclipse Platform build
using pom-first approach, so that entire eclipse platform becomes
natively available from a Maven repository. I gave up on this idea
mostly because I did not see how to convince platform developers let a
tool generate bundle manifests for them.

--
Regards,
Igor

...manifest-first makes it much easier to migrate to from PDE-build and still be able to work with Eclipse PDE without additional overhead.
/max

Post by Aaron Digulla
Hello,
I'm wondering why you're using p2 technology (with all it's drawbacks) instead of using Maven's fast&reliable dependency resolution to create a MANIFEST.MF from that?
Wouldn't it be much more simple to add a manifest builder plugin which I can configure from a POM that produces a MANIFEST.MF?
Regards,
--
Aaron "Optimizer" Digulla a.k.a. Philmann Dark
"It's not the universe that's limited, it's our imagination.
Follow me and I'll show you something beyond the limits."
http://blog.pdark.de/
_______________________________________________
tycho-user mailing list
https://dev.eclipse.org/mailman/listinfo/tycho-user

/max
http://about.me/maxandersen
_______________________________________________
tycho-user mailing list
https://dev.eclipse.org/mailman/listinfo/tycho-user

[Max Rydahl Andersen]

Post by Igor Fedorenko
FWIW, latest m2e-tycho lets you work on pom-first projects using Eclipse
PDE without additional overhead either, so I would not use this to
choose between pom-first and manifest first.

Might be, but I would not have considered/moved to Tycho if it required me to delete all my manifest.mf/build.properties etc.
and rely on full m2e with maven setup being configured to use/move to it.

/max

Post by Igor Fedorenko
--
Regards,
Igor

Post by Max Rydahl Andersen
…manifest-first makes it much easier to migrate to from PDE-build and still be able to work with Eclipse PDE without additional overhead.
/max

Post by Aaron Digulla
Hello,
I'm wondering why you're using p2 technology (with all it's drawbacks) instead of using Maven's fast&reliable dependency resolution to create a MANIFEST.MF from that?
Wouldn't it be much more simple to add a manifest builder plugin which I can configure from a POM that produces a MANIFEST.MF?
Regards,
--
Aaron "Optimizer" Digulla a.k.a. Philmann Dark
"It's not the universe that's limited, it's our imagination.
Follow me and I'll show you something beyond the limits."
http://blog.pdark.de/
_______________________________________________
tycho-user mailing list
https://dev.eclipse.org/mailman/listinfo/tycho-user

/max
http://about.me/maxandersen
_______________________________________________
tycho-user mailing list
https://dev.eclipse.org/mailman/listinfo/tycho-user

_______________________________________________
tycho-user mailing list
https://dev.eclipse.org/mailman/listinfo/tycho-user

/max
http://about.me/maxandersen

[Volker Wegert]

Post by Max Rydahl Andersen
…manifest-first makes it much easier to migrate to from PDE-build and still be able to work with Eclipse PDE without additional overhead.

I'm new to both Maven and Tycho, so maybe I got this wrong - but as far as I understand: If you've got a p2 repository you can install stuff from, you can use it as a dependency in an MANIFEST.MF-first build approach. If you're out for a Mavenistic pom.xml-first approach, all dependencies need to be retrievable by Maven. This would limit the usable dependencies to the projects already converted to this approach, right?

FWIW, I did't touch Maven until ECE2011, and I'm impressed how easy this stuff is. Great job.

Volker

[Igor Fedorenko]

Post by Volker Wegert

Post by Max Rydahl Andersen
…manifest-first makes it much easier to migrate to from PDE-build
and still be able to work with Eclipse PDE without additional
overhead.

I'm new to both Maven and Tycho, so maybe I got this wrong - but as
far as I understand: If you've got a p2 repository you can install
stuff from, you can use it as a dependency in an MANIFEST.MF-first
build approach. If you're out for a Mavenistic pom.xml-first
approach, all dependencies need to be retrievable by Maven. This
would limit the usable dependencies to the projects already converted
to this approach, right?

Not exactly. Generating p2 metadata for artifacts built using pom-first
approach is a straightforward mechanical process and there is even an
experimental nexus plugin that does this transparently for pom-first
artifacts deployed to a nexus hosted repository.

Going the other way, i.e. converting p2 repository to maven, is a much
harder problem. It is probably impossible to to correct(!) p2->maven
conversion in all cases, but some approximation is likely possible in
most/all cases.

--
Regards,
Igor

[Volker Wegert]

Igor,

Post by Igor Fedorenko

Post by Volker Wegert
I'm new to both Maven and Tycho, so maybe I got this wrong - but as
far as I understand: If you've got a p2 repository you can install
stuff from, you can use it as a dependency in an MANIFEST.MF-first
build approach. If you're out for a Mavenistic pom.xml-first
approach, all dependencies need to be retrievable by Maven. This
would limit the usable dependencies to the projects already converted
to this approach, right?

Not exactly. Generating p2 metadata for artifacts built using pom-first
approach is a straightforward mechanical process and there is even an
experimental nexus plugin that does this transparently for pom-first
artifacts deployed to a nexus hosted repository.
Going the other way, i.e. converting p2 repository to maven, is a much
harder problem. It is probably impossible to to correct(!) p2->maven
conversion in all cases, but some approximation is likely possible in
most/all cases.

That's exactly what I wanted to say. But that infers a kind of reverse dependency, doesn't it? If I wanted to build a plugin using Maven pom-first, I would have to wait until all of my dependencies provided a pom for me to pull in, right?

BTW and a bit OT, is there a good Maven book anyone would recommend?

Thanks
Volker

[Dahanne, Anthony]

Post by Volker Wegert
BTW and a bit OT, is there a good Maven book anyone would recommend?

This is the (up to date) reference :
http://www.sonatype.com/books/mvnref-book/reference/



 
Le contenu de ce courriel s'adresse au destinataire seulement. Il contient de l'information pouvant être confidentielle. Vous ne devez ni le copier ni l'utiliser ni le divulguer à qui que ce soit à moins que vous soyez le destinataire ou une personne désignée autorisée. Si vous le receviez par erreur, veuillez nous aviser immédiatement et le détruire.
 
The contents of this e-mail are intended for the named addressee only. It contains information that may be confidential. Unless you are the named addressee or an authorized designee, you may not copy or use it, or disclose it to anyone else. If you received it in error please notify us immediately and then destroy it.
 

De : tycho-user-***@eclipse.org [mailto:tycho-user-***@eclipse.org] De la part de Volker Wegert
Envoyé : 17 novembre 2011 13:07
À : Tycho user list
Objet : Re: [tycho-user] Why is Tycho manifest-first?

Igor,

Post by Volker Wegert

Post by Volker Wegert
I'm new to both Maven and Tycho, so maybe I got this wrong - but as
far as I understand: If you've got a p2 repository you can install
stuff from, you can use it as a dependency in an MANIFEST.MF-first
build approach. If you're out for a Mavenistic pom.xml-first
approach, all dependencies need to be retrievable by Maven. This
would limit the usable dependencies to the projects already converted
to this approach, right?

Not exactly. Generating p2 metadata for artifacts built using pom-first
approach is a straightforward mechanical process and there is even an
experimental nexus plugin that does this transparently for pom-first
artifacts deployed to a nexus hosted repository.
Going the other way, i.e. converting p2 repository to maven, is a much
harder problem. It is probably impossible to to correct(!) p2->maven
conversion in all cases, but some approximation is likely possible in
most/all cases.

That's exactly what I wanted to say. But that infers a kind of reverse dependency, doesn't it? If I wanted to build a plugin using Maven pom-first, I would have to wait until all of my dependencies provided a pom for me to pull in, right?

BTW and a bit OT, is there a good Maven book anyone would recommend?

Thanks
Volker

[Igor Fedorenko]

Post by Volker Wegert
Igor,

Post by Igor Fedorenko

Post by Volker Wegert
I'm new to both Maven and Tycho, so maybe I got this wrong - but
as far as I understand: If you've got a p2 repository you can
install stuff from, you can use it as a dependency in an
MANIFEST.MF-first build approach. If you're out for a Mavenistic
pom.xml-first approach, all dependencies need to be retrievable
by Maven. This would limit the usable dependencies to the
projects already converted to this approach, right?

Not exactly. Generating p2 metadata for artifacts built using
pom-first approach is a straightforward mechanical process and
there is even an experimental nexus plugin that does this
transparently for pom-first artifacts deployed to a nexus hosted
repository.
Going the other way, i.e. converting p2 repository to maven, is a
much harder problem. It is probably impossible to to correct(!)
p2->maven conversion in all cases, but some approximation is likely
possible in most/all cases.

That's exactly what I wanted to say. But that infers a kind of
reverse dependency, doesn't it? If I wanted to build a plugin using
Maven pom-first, I would have to wait until all of my dependencies
provided a pom for me to pull in, right?

Right. If your dependencies are only available from p2 repositories,
then manifest-first is easier.

Post by Volker Wegert
BTW and a bit OT, is there a good Maven book anyone would recommend?

I am biased, but you may want to have a look at Sonatype maven books


[1] http://www.sonatype.com/Support/Books

--
Regards,
Igor

[Aaron Digulla]

Post by Igor Fedorenko

Post by Volker Wegert

Post by Max Rydahl Andersen
…manifest-first makes it much easier to migrate to from PDE-build
and still be able to work with Eclipse PDE without additional
overhead.

I'm new to both Maven and Tycho, so maybe I got this wrong - but as
far as I understand: If you've got a p2 repository you can install
stuff from, you can use it as a dependency in an MANIFEST.MF-first
build approach. If you're out for a Mavenistic pom.xml-first
approach, all dependencies need to be retrievable by Maven. This
would limit the usable dependencies to the projects already converted
to this approach, right?

Not exactly. Generating p2 metadata for artifacts built using pom-first
approach is a straightforward mechanical process and there is even an
experimental nexus plugin that does this transparently for pom-first
artifacts deployed to a nexus hosted repository.
Going the other way, i.e. converting p2 repository to maven, is a much
harder problem. It is probably impossible to to correct(!) p2->maven
conversion in all cases, but some approximation is likely possible in
most/all cases.

I've created a pretty good tool for this and it's mostly automatic.

Right now, it doesn't handle JAR bundles (OSGI bundles with just a
single JAR in them) correctly but that's the only case where I know that
it fails.

See http://wiki.eclipse.org/Maven_Tools_4_Eclipse

I'm in the middle of migrating the tool from Python (works but has some
odd dependencies) to Groovy. The migrations is 80% done; the most
important missing point is the ability to patch POMs.

Regards,

Post by Igor Fedorenko
--
Regards,
Igor
_______________________________________________
tycho-user mailing list
https://dev.eclipse.org/mailman/listinfo/tycho-user