RFC: Adding BBDB to Emacs core

[Thomas Fitzsimmons]

Hi,

Now that BBDB is copyright clear and available in GNU ELPA, thanks to
Roland Winkler, I'd like to see what people think about also adding it
to Emacs core.

While maintaining EUDC in Emacs core, I've encountered many references
to BBDB that are unresolved. The only reason for this, as far as I can
tell, is that historically BBDB's copyright status didn't allow it to be
included in core. Otherwise it probably would have been included all
along. Now it's possible to fix this properly.

I've started an integration attempt on the scratch/eudc-bbdb-3 branch.
I merged a recent version of BBDB from GNU ELPA into lisp/bbdb, then I
resolved references to BBDB in EUDC. For example, we can remove things
like:

(declare-function bbdb-record-phones "ext:bbdb" t) ; via bbdb-defstruct

and apply changes like:

--- a/lisp/net/eudc-export.el
+++ b/lisp/net/eudc-export.el
@@ -31,10 +31,8 @@
;;; Code:

(require 'eudc)
-
-;; NOERROR is so we can compile it.
-(require 'bbdb nil t)
-(require 'bbdb-com nil t)
+(require 'bbdb)
+(require 'bbdb-com)

(defun eudc-create-bbdb-record (record &optional silent)
"Create a BBDB record using the RECORD alist.

This makes the code cleaner and easier to maintain. We can also rely
only on the version of BBDB in GNU Emacs (or a later one in GNU ELPA)
and so all the BBDB < 3 compatibility code can be deleted without risk
of breaking people's package sets (BBDB >= 3 auto-converts BBDB < 3
databases to the updated format).

I think applying this same type of effort to the other BBDB-dependent
core packages would simplify them too.

I'd like BBDB to become the default out-of-the-box local contact
management library for GNU Emacs, in particular so that Gnus/EUDC/BBDB
work together to provide email completion and snarfing out-of-the-box,
without extra configuration or package installation.

Thoughts?

Thomas

[Joshua Branson]

That sounds pretty awesome, but does the bbdb package have any info
documentation? Not that it really matters, but it would be nice to have.

Also, may I ask about ebdb? Does ebdb ever have
a chance at making emacs core?

P.S. I'm not a developer (at least not yet). Just curious.

Post by Thomas Fitzsimmons
Hi,
Now that BBDB is copyright clear and available in GNU ELPA, thanks to
Roland Winkler, I'd like to see what people think about also adding it
to Emacs core.
While maintaining EUDC in Emacs core, I've encountered many references
to BBDB that are unresolved. The only reason for this, as far as I can
tell, is that historically BBDB's copyright status didn't allow it to be
included in core. Otherwise it probably would have been included all
along. Now it's possible to fix this properly.
I've started an integration attempt on the scratch/eudc-bbdb-3 branch.
I merged a recent version of BBDB from GNU ELPA into lisp/bbdb, then I
resolved references to BBDB in EUDC. For example, we can remove things
(declare-function bbdb-record-phones "ext:bbdb" t) ; via bbdb-defstruct
--- a/lisp/net/eudc-export.el
+++ b/lisp/net/eudc-export.el
@@ -31,10 +31,8 @@
(require 'eudc)
-
-;; NOERROR is so we can compile it.
-(require 'bbdb nil t)
-(require 'bbdb-com nil t)
+(require 'bbdb)
+(require 'bbdb-com)
(defun eudc-create-bbdb-record (record &optional silent)
"Create a BBDB record using the RECORD alist.
This makes the code cleaner and easier to maintain. We can also rely
only on the version of BBDB in GNU Emacs (or a later one in GNU ELPA)
and so all the BBDB < 3 compatibility code can be deleted without risk
of breaking people's package sets (BBDB >= 3 auto-converts BBDB < 3
databases to the updated format).
I think applying this same type of effort to the other BBDB-dependent
core packages would simplify them too.
I'd like BBDB to become the default out-of-the-box local contact
management library for GNU Emacs, in particular so that Gnus/EUDC/BBDB
work together to provide email completion and snarfing out-of-the-box,
without extra configuration or package installation.
Thoughts?
Thomas

[Eric Abrahamsen]

Post by Joshua Branson
That sounds pretty awesome, but does the bbdb package have any info
documentation? Not that it really matters, but it would be nice to have.
Also, may I ask about ebdb? Does ebdb ever have
a chance at making emacs core?

Probably not! There's already push-back on including more packages in
core, and the main argument for making an exception for BBDB (many
packages are already "aware" of it) doesn't apply to EBDB.

Personally, I belong to the "don't put more stuff in core" camp, and am
happy keeping EBDB in ELPA.

E

[Joshua Branson]

Post by Eric Abrahamsen

Post by Joshua Branson
That sounds pretty awesome, but does the bbdb package have any info
documentation? Not that it really matters, but it would be nice to have.
Also, may I ask about ebdb? Does ebdb ever have
a chance at making emacs core?

Probably not! There's already push-back on including more packages in
core, and the main argument for making an exception for BBDB (many
packages are already "aware" of it) doesn't apply to EBDB.
Personally, I belong to the "don't put more stuff in core" camp, and am
happy keeping EBDB in ELPA.

That makes sense. The more packages you have, the harder it must be to
maintain a stable emacs.

Post by Eric Abrahamsen
E

[Bozhidar Batsov]

You can add my voice to "I'd rather just have in ELPA (and we should be
moving more and more packages there).

There was some cool idea a while ago to have even Emacs stable releases
package some core deps directly from ELPA (similar to
what languages like Ruby do). That simplifies the maintenance of Emacs
itself and gives more flexibility to users to update stuff while waiting
for new Emacs releases.

Post by Joshua Branson

Post by Eric Abrahamsen

Post by Joshua Branson
That sounds pretty awesome, but does the bbdb package have any info
documentation? Not that it really matters, but it would be nice to

have.

Post by Eric Abrahamsen

Post by Joshua Branson
Also, may I ask about ebdb? Does ebdb ever have
a chance at making emacs core?

Probably not! There's already push-back on including more packages in
core, and the main argument for making an exception for BBDB (many
packages are already "aware" of it) doesn't apply to EBDB.
Personally, I belong to the "don't put more stuff in core" camp, and am
happy keeping EBDB in ELPA.

That makes sense. The more packages you have, the harder it must be to
maintain a stable emacs.

Post by Eric Abrahamsen
E

[John Wiegley]

BB> You can add my voice to "I'd rather just have in ELPA (and we should be
BB> moving more and more packages there).

Same here. I really don't want to see BBDB moved into core. I do want ELPA
packages to become more "first class" than they are now, so that the desire to
add such packages to core would no longer have the same appeal.

[Thomas Fitzsimmons]

Post by John Wiegley
BB> You can add my voice to "I'd rather just have in ELPA (and we should be
BB> moving more and more packages there).
Same here. I really don't want to see BBDB moved into core. I do want ELPA
packages to become more "first class" than they are now, so that the desire to
add such packages to core would no longer have the same appeal.

I think the end result from the perspective of core Emacs maintainers'
maintenance burden would not be different in beneficial ways than BBDB
just being in core.

Quoting Stefan later in the thread, the (1) "fetch some packages from
elpa.git while building the Emacs tarball" method may allow for nice
out-of-the-box BBDB integration for an Emacs release, and solve the
problems I'm trying to solve for users of Emacs major release tarballs
(but not users/developers who build the Emacs they use out of git, and
I'd worry about last-minute integration of all this stuff -- who makes
sure it all works together, at what point before release?).

But method (1) wouldn't solve the EUDC package maintenance aspects for
me (EUDC requiring something not in the tree). For that, we'd need
Stefan's solution (2) "also clone elpa.git when you clone emacs.git"
solution. That may solve both cases if it's done completely, see below.

To get the same benefits for BBDB as it being in core, I'd want solution
(2) to ensure that core maintainers always clone BBDB into their tree,
so that they always build it. Then they can check for compile errors,
and usage of new features, as they change the core parts of Emacs around
BBDB. I get very useful patches to EUDC from the core maintainers from
time to time even though they don't know EUDC functionality or
internals; I would hope that with solution (2) I'd still get those types
of patches.

Assuming (2) achieves all that, from the core maintainer's perspective,
what's the difference? The downside is they have two repos to deal
with, and the interactions between the two to always consider (e.g., do
we branch all of ELPA to match Emacs branches (probably not), or write
all ELPA packages to work on any Emacs branch (probably), etc.).

Or is there some other way of bumping ELPA packages up to first class
status that you're envisioning? I'm willing to help experiment with
different approaches to solve EUDC/BBDB issues, FWIW, but as yet I can't
envision the end result.

(Philosophical aside: I really don't want to see Emacs major releases
become just an Elisp language runtime, class libraries and package
management. That would be sad. Emacs is special, not just another
language environment. Package discovery via GNU ELPA (over the network)
just isn't the same as feature discovery within the running Emacs
instance -- there's always an extra level of annoyance, network access
and configuration associated with external packages. I'm hoping that by
Emacs 27.1 I'll have a window manager in my text editor.

Maybe Emacs could do like GNU/Linux distributions and publish e.g.,
emacs-minimal-27.1.tar.gz alongside emacs-27.1.tar.gz though.)

Thomas

[Stefan Monnier]

Before deciding on this, I think we'd need a clear picture of the ways
in which BBDB is needed/used by EUDC.

I see you saying things like "EUDC requiring something not in the tree",
but I don't see anything in EUDC that really requires BBDB, instead
I just see glue code that lets you use BBDB when it's available, just
like there's code that lets you use LDAP when it's available (but
there's clearly no corresponding push to try and add LDAP directly into
Emacs).

Could you give us a clear description of how BBDB is used by EUDC?
I'm thinking of a list of BBDB functions/variables that are used by
EUDC, along with a description of what they are used for (not what they
do themselves, but what EUDC uses them for).

Also good would be to describe concrete ways in which having BBDB in
Emacs itself would improve the user's experience (presumably for users
which don't already use BBDB).


Stefan

[Eric Abrahamsen]

Post by Stefan Monnier
Before deciding on this, I think we'd need a clear picture of the ways
in which BBDB is needed/used by EUDC.
I see you saying things like "EUDC requiring something not in the tree",
but I don't see anything in EUDC that really requires BBDB, instead
I just see glue code that lets you use BBDB when it's available, just
like there's code that lets you use LDAP when it's available (but
there's clearly no corresponding push to try and add LDAP directly into
Emacs).

I'm also curious about this, as I've started implementing EUDC support
for EBDB. On "my" side, I'm seeing `eudc-register-protocol', and a bunch
of calls to `eudc-protocol-set'. Shouldn't those thing be enough? If
EUDC provides a generalized framework, shouldn't the whole point be that
it doesn't need to know about its implementations?

[Thomas Fitzsimmons]

Post by Stefan Monnier
Before deciding on this, I think we'd need a clear picture of the ways
in which BBDB is needed/used by EUDC.

OK, I'll try to provide that.

Post by Stefan Monnier
I see you saying things like "EUDC requiring something not in the tree",
but I don't see anything in EUDC that really requires BBDB, instead
I just see glue code that lets you use BBDB when it's available,

When I say "require" there, I mean in the sense that to properly
byte-compile lisp/net/eudcb-bbdb.el and lisp/net/eudc-export.el, which
are part of EUDC, BBDB packages are require'd.

On master there are workarounds in place to stub out or ignore the
missing dependencies. One aspect of the BBDB inclusion proposal is that
I'm trying to get rid of those workarounds (in EUDC first and eventually
in Org and Gnus). See for example, these commits on the
scratch/eudc-bbdb-3 branch:

06c3557 EUDC: Require bbdb and bbdb-com without ignoring errors
f1ab95d EUDC: Require bbdb, bbdb-com without NOERROR
262dce0 EUDC: Remove external BBDB function declarations
e355a65 EUDC: Remove inline requires of bbdb in BBDB backend

And yes, to actually use the BBDB backend of EUDC at runtime, the BBDB
package needs to be loaded. On the branch, I've enabled the BBDB
backend by default in EUDC, which eliminates a configuration step for
email completion (see below):

193f9d5 EUDC: Enable BBDB backend by default

Post by Stefan Monnier
just like there's code that lets you use LDAP when it's available (but
there's clearly no corresponding push to try and add LDAP directly
into Emacs).

The Elisp part of Emacs's LDAP support, lisp/net/ldap.el, is already in
core. So EUDC's LDAP backend, lisp/net/eudcb-ldap.el, can just (require
'ldap) directly during byte-compilation.

ldap.el internally requires the ldapsearch command line utility, which
is provided on free operating systems as part of OpenLDAP packages.

I actually would prefer ldap.el to be a pure Elisp LDAP client (and thus
eliminate the OpenLDAP dependency) so that it could run with minimal
configuration even on non-free operating systems, but I haven't looked
at whether that would be practical to implement.

Post by Stefan Monnier
Could you give us a clear description of how BBDB is used by EUDC?
I'm thinking of a list of BBDB functions/variables that are used by
EUDC, along with a description of what they are used for (not what they
do themselves, but what EUDC uses them for).

The main function used is bbdb-search. Here's a backtrace of EUDC
expanding a contact:

eval((bbdb-search records "Examp"))
eudc-bbdb-query-internal(((firstname . "Examp")) (firstname lastname net))
eudc-query(((firstname . "Examp")) (firstname lastname net))
eudc-expand-inline()
funcall-interactively(eudc-expand-inline)

eudc-expand-inline uses bbdb-search to search for contacts in the BBDB
database using a search string, and return a list of contact records to
it. Then from those records, EUDC creates strings in the format
"[first-name] [last-name] <email-address>" (the output format is a
configurable property of EUDC); if there is only one result, it is
inserted at point; if there are multiple results, EUDC allows the user
to interactively select the one to insert at point.

Post by Stefan Monnier
Also good would be to describe concrete ways in which having BBDB in
Emacs itself would improve the user's experience (presumably for users
which don't already use BBDB).

OK, assuming the user wants to use Gnus, and they want to store contact
information they come across in Gnus locally, then BBDB would be the
recommended, "Emacs core endorsed" way of doing that, and the key
bindings, configuration etc. would all default to using BBDB.

Then EUDC would have the BBDB backend enabled by default (as shown on
the branch), and Gnus would have default keybindings for
`eudc-expand-inline' when point is in To: or Cc:.

Then without any setup, the user could snarf an address from within an
email via a keypress, then complete that contact's name and address when
composing a new email by pressing TAB.

If Gnus defaults to using `eudc-expand-inline`, then the user can also
get LDAP completion support just by adding an ldap entry to
`eudc-server-hotlist'.

I think that level of default integration would be hard to achieve and
maintain without having BBDB available in core.

Thomas

[Phillip Lord]

Post by Thomas Fitzsimmons

Post by John Wiegley
BB> You can add my voice to "I'd rather just have in ELPA (and we should be
BB> moving more and more packages there).
Same here. I really don't want to see BBDB moved into core. I do want ELPA
packages to become more "first class" than they are now, so that the desire to
add such packages to core would no longer have the same appeal.

Assuming (2) achieves all that, from the core maintainer's perspective,
what's the difference? The downside is they have two repos to deal
with, and the interactions between the two to always consider (e.g., do
we branch all of ELPA to match Emacs branches (probably not), or write
all ELPA packages to work on any Emacs branch (probably), etc.).

There is actually a problem here with the way the ELPA is structured,
since it already uses branches for (some) packages. This is not going to
marry easily with the use of branches for something else.

Post by Thomas Fitzsimmons
(Philosophical aside: I really don't want to see Emacs major releases
become just an Elisp language runtime, class libraries and package
management. That would be sad. Emacs is special, not just another
language environment. Package discovery via GNU ELPA (over the network)
just isn't the same as feature discovery within the running Emacs
instance -- there's always an extra level of annoyance, network access
and configuration associated with external packages. I'm hoping that by
Emacs 27.1 I'll have a window manager in my text editor.

The hope would be not that a major release doesn't seen significant
advances in Emacs functionality, but that, those who want to, will be
able to track that functionality earlier as it develops.

Phil

[Stefan Monnier]

Post by Phillip Lord

Post by Thomas Fitzsimmons
Assuming (2) achieves all that, from the core maintainer's perspective,
what's the difference? The downside is they have two repos to deal
with, and the interactions between the two to always consider (e.g., do
we branch all of ELPA to match Emacs branches (probably not), or write
all ELPA packages to work on any Emacs branch (probably), etc.).

There is actually a problem here with the way the ELPA is structured,
since it already uses branches for (some) packages. This is not going to
marry easily with the use of branches for something else.

Rather than a problem I see it as a defining aspect of whether a package
should be in elpa.git or in emacs.git, since the whole point of having
"unbundled packages" is so that their release schedule is not tied to
that of Emacs.


Stefan

[Phillip Lord]

Post by Stefan Monnier

Post by Phillip Lord

Post by Thomas Fitzsimmons
Assuming (2) achieves all that, from the core maintainer's perspective,
what's the difference? The downside is they have two repos to deal
with, and the interactions between the two to always consider (e.g., do
we branch all of ELPA to match Emacs branches (probably not), or write
all ELPA packages to work on any Emacs branch (probably), etc.).

There is actually a problem here with the way the ELPA is structured,
since it already uses branches for (some) packages. This is not going to
marry easily with the use of branches for something else.

Rather than a problem I see it as a defining aspect of whether a package
should be in elpa.git or in emacs.git, since the whole point of having
"unbundled packages" is so that their release schedule is not tied to
that of Emacs.

Not sure I understand.

If I have a package on master in ELPA, and I want to main one version
for Emacs-26 and another for Emacs-27, how do I do this? The obvious
solution is to put in a branch. But now I have branched all the other
packages to.

For externals, we can do this, although we can't call it "emacs-26" or
"emacs-27" because someone else will have that branch.

Phil

[Stefan Monnier]

Post by Phillip Lord
If I have a package on master in ELPA, and I want to main one version
for Emacs-26 and another for Emacs-27,

I would take this as a sign that the package shouldn't be in elpa.git.

elpa.git is for packages which are not specific to a particular Emacs version.


Stefan

[Phillip Lord]

Post by Stefan Monnier

Post by Phillip Lord
If I have a package on master in ELPA, and I want to main one version
for Emacs-26 and another for Emacs-27,

I would take this as a sign that the package shouldn't be in elpa.git.
elpa.git is for packages which are not specific to a particular Emacs version.

I think that this is wrong and I will give you a use case.

Imagine, for example, that I have a package which I add to ELPA when
Emacs-26 comes out. I maintain this package and expand it slowly. Then,
when Emacs-27 comes out, for example something significant has happened
(say, just for argument sake, the advice system got re-written --
unlikely, but it could happen).

As a package author, I have two choices here: either I move to Emacs-27
and ditch Emacs-26 support. Or I have to put runtime conditional logic
which supports both. I think package authors should have the choice of
freezing the Emacs-26 version of their package when Emacs-27 comes out.

This situation would get worse if we have an "unstable" version of
ELPA. I bet that there is some package already that depends on Emacs-27
at the bleeding edge.

Phil

[Stefan Monnier]

Post by Phillip Lord
As a package author, I have two choices here: either I move to Emacs-27
and ditch Emacs-26 support. Or I have to put runtime conditional logic
which supports both.

That's right.

Post by Phillip Lord
I think package authors should have the choice of freezing the
Emacs-26 version of their package when Emacs-27 comes out.

There's a case to be made for allowing ELPA archives to exports several
versions of a package at the same time, so older Emacsen can still
install the old version of a package without having to download the old
version by hand.

But I'm not moved by your scenario: adding runtime conditional logic has
been standard procedure "for ever" and comes with all kinds of
advantages, such as the ability for older Emacsen to benefit from other
improvements in the newer versions of your package, or an easier way to
install&use the package with several different versions of Emacs at the
same time, ...

Post by Phillip Lord
This situation would get worse if we have an "unstable" version of
ELPA.

I don't see why: "unstable" is just a preview of what will be "stable"
a few weeks/months later and packages usually preserve compatibility at
least with Emacsen that are a few years old.

Post by Phillip Lord
I bet that there is some package already that depends on Emacs-27
at the bleeding edge.

I strongly doubt it.


Stefan

[Paul Eggert]

Post by Stefan Monnier
adding runtime conditional logic has
been standard procedure "for ever" and comes with all kinds of
advantages,

It's a tradeoff, though, as the runtime logic makes the code harder to
maintain and (particularly) to test, and at some point it becomes more
trouble than it's worth. This is why Gnus dropped support for Emacs 22 a
couple of years ago. Even if ELPA packages should work on current and
previous Emacs versions, today I would think that they shouldn't have to
worry about porting to Emacs 23 or earlier, and that a package's
maintainer can determine whether the cutoff is 22, 23, or even 24.

[Stefan Monnier]

Post by Paul Eggert
It's a tradeoff, though, as the runtime logic makes the code harder to
maintain and (particularly) to test, and at some point it becomes more
trouble than it's worth. This is why Gnus dropped support for Emacs
22 a couple of years ago. Even if ELPA packages should work on current and
previous Emacs versions, today I would think that they shouldn't have to
worry about porting to Emacs 23 or earlier, and that a package's maintainer
can determine whether the cutoff is 22, 23, or even 24.

Of course, but that doesn't mean there needs to be several different
supported versions of the package. It just means you drop support for
old versions, as is the case in all packages. Nothing new here: no need
for different branches in elpa.git to go along with branches in
emacs.git just because of that.


Stefan

[Phillip Lord]

Post by Stefan Monnier

Post by Paul Eggert
It's a tradeoff, though, as the runtime logic makes the code harder to
maintain and (particularly) to test, and at some point it becomes more
trouble than it's worth. This is why Gnus dropped support for Emacs
22 a couple of years ago. Even if ELPA packages should work on current and
previous Emacs versions, today I would think that they shouldn't have to
worry about porting to Emacs 23 or earlier, and that a package's maintainer
can determine whether the cutoff is 22, 23, or even 24.

Of course, but that doesn't mean there needs to be several different
supported versions of the package. It just means you drop support for
old versions, as is the case in all packages. Nothing new here: no need
for different branches in elpa.git to go along with branches in
emacs.git just because of that.

If we support old versions, checkpointed at a particular time in
history, then would end up supporting tags. If we support tags, then we
support branches, if I understand git.

Phil

[Stefan Monnier]

Post by Phillip Lord

Post by Stefan Monnier
Of course, but that doesn't mean there needs to be several different
supported versions of the package. It just means you drop support for
old versions, as is the case in all packages. Nothing new here: no need
for different branches in elpa.git to go along with branches in
emacs.git just because of that.

If we support old versions, checkpointed at a particular time in
history, then would end up supporting tags.

No, the way I see it working is that the `archive-contents` file would
simply list all packages that are still present in the archive, rather
than only the latest. I.e. the Git side would be unaffected, only the
ELPA side.


Stefan

[Phillip Lord]

Post by Stefan Monnier

Post by Phillip Lord

Post by Stefan Monnier
Of course, but that doesn't mean there needs to be several different
supported versions of the package. It just means you drop support for
old versions, as is the case in all packages. Nothing new here: no need
for different branches in elpa.git to go along with branches in
emacs.git just because of that.

If we support old versions, checkpointed at a particular time in
history, then would end up supporting tags.

No, the way I see it working is that the `archive-contents` file would
simply list all packages that are still present in the archive, rather
than only the latest. I.e. the Git side would be unaffected, only the
ELPA side.

How would that be generated? I mean, it would be based on the packages
that are already there? I think that this would make generating ELPA
from a git clone would not work, unless you work through every commit
looking for changes in "Version:" number.

Phil

[Stefan Monnier]

Post by Phillip Lord
How would that be generated? I mean, it would be based on the packages
that are already there?

Yes.

Post by Phillip Lord
I think that this would make generating ELPA from a git clone would
not work,

It would work, but it would only give you the latest versions, indeed.
I don't see a problem with that: how many people/places do you know who
build an ELPA archive from the elpa.git repository?


Stefan

[Phillip Lord]

Post by Stefan Monnier

Post by Phillip Lord
How would that be generated? I mean, it would be based on the packages
that are already there?

Yes.

Post by Phillip Lord
I think that this would make generating ELPA from a git clone would
not work,

It would work, but it would only give you the latest versions, indeed.
I don't see a problem with that: how many people/places do you know who
build an ELPA archive from the elpa.git repository?

Probably not that many, but a repeatable build is always nice.

Phil

[Phillip Lord]

Post by Stefan Monnier

Post by Phillip Lord
As a package author, I have two choices here: either I move to Emacs-27
and ditch Emacs-26 support. Or I have to put runtime conditional logic
which supports both.

That's right.

Post by Phillip Lord
I think package authors should have the choice of freezing the
Emacs-26 version of their package when Emacs-27 comes out.

There's a case to be made for allowing ELPA archives to exports several
versions of a package at the same time, so older Emacsen can still
install the old version of a package without having to download the old
version by hand.

Yep. In practice, for my packages this would currently happen at either
24.1 (this is the oldest Emacs I can still compile, and even that
requires a bit of hacking) or 24.4 (the advice change). Being able to
leave a version at this point would be a good thing.

Post by Stefan Monnier
But I'm not moved by your scenario: adding runtime conditional logic has
been standard procedure "for ever" and comes with all kinds of
advantages, such as the ability for older Emacsen to benefit from other
improvements in the newer versions of your package, or an easier way to
install&use the package with several different versions of Emacs at the
same time, ...

Imagine I have left a package for an old version of Emacs. Now I find a
major security bug in the latest version, that I need to back-port to
the old version.

Post by Stefan Monnier

Post by Phillip Lord
This situation would get worse if we have an "unstable" version of
ELPA.

I don't see why: "unstable" is just a preview of what will be "stable"
a few weeks/months later and packages usually preserve compatibility at
least with Emacsen that are a few years old.

Post by Phillip Lord
I bet that there is some package already that depends on Emacs-27
at the bleeding edge.

I strongly doubt it.

I haven't tested it yet; to my knowledge there is no routine testing of
ELPA against multiple Emacs versions.

Phil

[Radon Rosborough]

Post by Stefan Monnier
There's a case to be made for allowing ELPA archives to exports several
versions of a package at the same time, so older Emacsen can still
install the old version of a package without having to download the old
version by hand.

For another perspective, see also source-based package managers like
straight.el [1], Borg [2], el-get [3], et cetera, which easily bypass
this problem by allowing you to install any version you want, and to
change your mind after installation.

[1]: https://github.com/raxod502/straight.el
[2]: https://github.com/emacscollective/borg
[3]: https://github.com/dimitri/el-get

[Phillip Lord]

Post by Radon Rosborough

Post by Stefan Monnier
There's a case to be made for allowing ELPA archives to exports several
versions of a package at the same time, so older Emacsen can still
install the old version of a package without having to download the old
version by hand.

For another perspective, see also source-based package managers like
straight.el [1], Borg [2], el-get [3], et cetera, which easily bypass
this problem by allowing you to install any version you want, and to
change your mind after installation.
[1]: https://github.com/raxod502/straight.el
[2]: https://github.com/emacscollective/borg
[3]: https://github.com/dimitri/el-get

More than I knew about. Replacing package.el would indeed be a way
forward (although you'd need to run both in parallel for a while).

Phil

[Stefan Monnier]

Post by Phillip Lord
More than I knew about. Replacing package.el would indeed be a way
forward (although you'd need to run both in parallel for a while).

They're so close to each other that rather than replacing them, they
need to be merged or modularized so that they build on top of each
other.

package.el is not nearly as constraining as people tend to think.


Stefan

[Radon Rosborough]

Post by Stefan Monnier
They're so close to each other that rather than replacing them, they
need to be merged or modularized so that they build on top of each
other.

If you have suggestions on how that could be accomplished, and what
the concrete advantages would be, I would be very glad to hear them. I
will, of course, be happy to provide any information about straight.el
that would be helpful to the discussion.

Post by Stefan Monnier
package.el is not nearly as constraining as people tend to think.

I will note that even if I did not perceive any technical reasons why
package.el couldn't be easily extended to support the features of
straight.el, the inertia generated by its membership in Emacs core
would still have been more than enough to convince me to start a new
project instead.

[Phillip Lord]

Post by Stefan Monnier

Post by Phillip Lord
More than I knew about. Replacing package.el would indeed be a way
forward (although you'd need to run both in parallel for a while).

They're so close to each other that rather than replacing them, they
need to be merged or modularized so that they build on top of each
other.
package.el is not nearly as constraining as people tend to think.

I'd just like it to cope with several versions and with Emacs version
dependencies.

Phil

[Stefan Monnier]

Post by Phillip Lord

Post by Stefan Monnier

Post by Phillip Lord
More than I knew about. Replacing package.el would indeed be a way
forward (although you'd need to run both in parallel for a while).

They're so close to each other that rather than replacing them, they
need to be merged or modularized so that they build on top of each
other.
package.el is not nearly as constraining as people tend to think.

I'd just like it to cope with several versions and with Emacs version
dependencies.

We agree. And the underlying design is perfectly able to cope with it.
I haven't checked, but it's even possible that package.el already
handles it correctly if/when the `archive-contents` file contains
various different versions of the same package.

IOW it might be a simple matter of adjusting the scripts that build
the archive(s).


Stefan

[Phillip Lord]

Post by Stefan Monnier

Post by Phillip Lord

Post by Stefan Monnier

Post by Phillip Lord
More than I knew about. Replacing package.el would indeed be a way
forward (although you'd need to run both in parallel for a while).

They're so close to each other that rather than replacing them, they
need to be merged or modularized so that they build on top of each
other.
package.el is not nearly as constraining as people tend to think.

I'd just like it to cope with several versions and with Emacs version
dependencies.

We agree. And the underlying design is perfectly able to cope with it.
I haven't checked, but it's even possible that package.el already
handles it correctly if/when the `archive-contents` file contains
various different versions of the same package.

That would be particularly good. If package.el can cope in some way,
then it's backdatable. I will check.

Phil

[Stefan Monnier]

Post by Phillip Lord
That would be particularly good. If package.el can cope in some way,
then it's backdatable. I will check.

It can definitely cope with different versions of the same package
offered by different ELPA archives, and I think the way it's implemented
it shouldn't really care whether it comes from different ELPA archives
or not.


Stefan

[Roland Winkler]

Post by John Wiegley
BB> You can add my voice to "I'd rather just have in ELPA (and we
BB> should be moving more and more packages there).
Same here. I really don't want to see BBDB moved into core. I do want
ELPA packages to become more "first class" than they are now, so that
the desire to add such packages to core would no longer have the same
appeal.

In general I tend to agree. Yet it appears to me that so far there is
one problem with GNU ELPA: I believe it would help to have a better
separation between ELPA being the repository for the distribution of
stable versions of packages to normal users and ELPA being the
repository for the development of such packages.

With core Emacs, we have a master branch with the latest and sometimes
buggy code. And occasionally there is a proper release with
significant efforts that these releases work reliably for normal users.
So far ELPA does not support such a model.

- Take BBDB as an example: It is not too difficult to break a user's
database by trying to improve some of BBDB's internal functions. That's
why right now I prefer to continue to use BBDB's repository on savannah
for code development. When a code change appears to be sufficiently
stable, it is also added to BBDB in ELPA.

Certainly, this is a non-ideal approach. I am wondering: Could it make
sense to have a separate infrastructure like "ELPA-devel" for the
on-going development of ELPA packages, where adventurous users find the
very latest code similar to the master branch of core Emacs. On the
other hand, ELPA becomes the place for stable versions of these
packages. My terminology "separate infrastructure" is intentionally
vague, because I do not yet have a clear idea how this can be
implemented efficiently.

A simple scheme could be a policy for naming development and release
branches of packages in the ELPA git repository (beyond the current
"externals" branches for individual ELPA packages), combined with tools
that allow one to access one or the other version of a package.

Or yet something different.

Roland

[John Wiegley]

RW> Certainly, this is a non-ideal approach. I am wondering: Could it make
RW> sense to have a separate infrastructure like "ELPA-devel" for the on-going
RW> development of ELPA packages, where adventurous users find the very latest
RW> code similar to the master branch of core Emacs. On the other hand, ELPA
RW> becomes the place for stable versions of these packages. My terminology
RW> "separate infrastructure" is intentionally vague, because I do not yet
RW> have a clear idea how this can be implemented efficiently.

Having an "-unstable" flavor of the package repository seems to be de rigeur
these days. I wonder if it would lead to any maintenance headaches. But it
would certainly improve ELPA's standing as a place both for development, and
distribution.

[Stefan Monnier]

Post by Roland Winkler
- Take BBDB as an example: It is not too difficult to break a user's
database by trying to improve some of BBDB's internal functions. That's
why right now I prefer to continue to use BBDB's repository on savannah
for code development. When a code change appears to be sufficiently
stable, it is also added to BBDB in ELPA.

FWIW, currently GNU ELPA supports this in the following way: a new GNU
ELPA package is only created once the "Version:" of a package is bumped.
So you can add experimental code to elpa.git's master branch without
worrying about its effect on the distributed package and when that code
is ready you can just bump the "Version:" header to cause a new release.


Stefan

[Roland Winkler]

Post by Stefan Monnier
FWIW, currently GNU ELPA supports this in the following way: a new GNU
ELPA package is only created once the "Version:" of a package is
bumped. So you can add experimental code to elpa.git's master branch
without worrying about its effect on the distributed package and when
that code is ready you can just bump the "Version:" header to cause a
new release.

It would be nice to be able to go beyond that. I am grateful if
adventurous users are willing to test the experimental code. So it
would be nice if there was infrastructure that somehow allowed these
users to subscribe to the experimental code, similar to how the
development of core emacs benefits from users using the master branch.

[Stephen Leake]

Post by Roland Winkler

Post by Stefan Monnier
FWIW, currently GNU ELPA supports this in the following way: a new GNU
ELPA package is only created once the "Version:" of a package is
bumped. So you can add experimental code to elpa.git's master branch
without worrying about its effect on the distributed package and when
that code is ready you can just bump the "Version:" header to cause a
new release.

It would be nice to be able to go beyond that. I am grateful if
adventurous users are willing to test the experimental code. So it
would be nice if there was infrastructure that somehow allowed these
users to subscribe to the experimental code, similar to how the
development of core emacs benefits from users using the master branch.

+1

I have an experimental version of ada-mode I'd like users to try.

[Phillip Lord]

Post by Roland Winkler

Post by John Wiegley
BB> You can add my voice to "I'd rather just have in ELPA (and we
BB> should be moving more and more packages there).
Same here. I really don't want to see BBDB moved into core. I do want
ELPA packages to become more "first class" than they are now, so that
the desire to add such packages to core would no longer have the same
appeal.

A simple scheme could be a policy for naming development and release
branches of packages in the ELPA git repository (beyond the current
"externals" branches for individual ELPA packages), combined with tools
that allow one to access one or the other version of a package.
Or yet something different.

I think this going to be scary because packages are distributed across
multiple branches, so names would end up getting namespaced. I think
that MELPA has it right here -- packages should be in different repos,
master is trunk, last tag is release. This could be expanded to support
multiple releases if package.el supported multiple versions of Emacs.

Phil

[Stefan Monnier]

Post by Roland Winkler
A simple scheme could be a policy for naming development and release
branches of packages in the ELPA git repository (beyond the current
"externals" branches for individual ELPA packages), combined with tools
that allow one to access one or the other version of a package.

I've had in my TODO list to add a "bleeding edge GNU ELPA" for a while
and that plan is much simpler than the above: just build this
alternative GNU ELPA archive from "master" by disregarding the
"Version:" header.

Currently, the only "official" way for users to use such a bleeding edge
archive is to clone elpa.git and do "make" inside it (it takes a few
more steps: check the README).


Stefan

[Roland Winkler]

Post by Stefan Monnier
I've had in my TODO list to add a "bleeding edge GNU ELPA" for a while
and that plan is much simpler than the above: just build this
alternative GNU ELPA archive from "master" by disregarding the
"Version:" header.

Sounds good to me. In my opinion, an important point is then to
advertise this as "ELPA for adventurous users", combined with tools
allowing one to use one or the other version of ELPA, and ideally with
an option that allows users to be adventurous with some packages only.

Roland

[Thomas Fitzsimmons]

Post by Joshua Branson
That sounds pretty awesome, but does the bbdb package have any info
documentation?

Not currently, only a stub .texi file exists in GNU ELPA.

Post by Joshua Branson
Not that it really matters, but it would be nice to have.

Agreed, it would be nice to have.

Thomas

[Glenn Morris]

I'd like to see what people think about also adding it to Emacs core.

Personally I think this is the opposite direction to the one in which
Emacs components should be moving.

is that historically BBDB's copyright status didn't allow it to be
included in core. Otherwise it probably would have been included all
along.

Bear in mind that ELPAs did not exist when bbdb was first created.

I'd like BBDB to become the default out-of-the-box local contact
management library for GNU Emacs, in particular so that Gnus/EUDC/BBDB
work together to provide email completion and snarfing out-of-the-box,
without extra configuration or package installation.

If GNU ELPA is a first-class citizen, then all the above can happen
without adding yet more stuff to the main Emacs repo. (Wistfully
thinking here yet again of the project to bundle GNU ELPA packages with
Emacs releases...)

[Radon Rosborough]

Post by Glenn Morris

I'd like to see what people think about also adding it to Emacs core.

Personally I think this is the opposite direction to the one in which
Emacs components should be moving.

I heartily agree here: as someone who is looking to modernize Emacs
package management, one of the biggest problems I see in the current
system is the fact that some (obsolete) versions of packages are
bundled with Emacs core. It is annoying in the same way that when
using a package manager on macOS, one must deal with the (obsolete)
versions of packages that are provided by macOS.

Post by Glenn Morris
(Wistfully thinking here yet again of the project to bundle GNU ELPA
packages with Emacs releases...)

My ideal situation would be one in which one can additionally install
a version of Emacs that does *not* come bundled with any packages.
AFAICT, this project would help make that happen, so I am also in
support of that.

Best,
Radon

[Thomas Fitzsimmons]

Post by Glenn Morris

I'd like to see what people think about also adding it to Emacs core.

Personally I think this is the opposite direction to the one in which
Emacs components should be moving.

I understand that position in general, but I think BBDB should be an
exception for two reasons: because there are already dependencies on it
in many core packages, and because it is a library that provides a
contact management API to other packages.

Thomas

[Phillip Lord]

Post by Glenn Morris

Post by Thomas Fitzsimmons
I'd like BBDB to become the default out-of-the-box local contact
management library for GNU Emacs, in particular so that Gnus/EUDC/BBDB
work together to provide email completion and snarfing out-of-the-box,
without extra configuration or package installation.

If GNU ELPA is a first-class citizen, then all the above can happen
without adding yet more stuff to the main Emacs repo. (Wistfully
thinking here yet again of the project to bundle GNU ELPA packages with
Emacs releases...)

I've part-written two different versions of this, both in git.

They work in different ways; but ultimately, I think we need to decide
what "ELPA as a first-class citizen" actually means.

This version:

http://git.savannah.gnu.org/cgit/emacs.git/log/?h=elparized-core

for example, just pulls out parts of ELPA using git magic, and copies
the files into core. Simple, straight-forward and it works. But,
ultimately, will it make maintaining core more easy? In the end, I think
not, because it is essentially an ad-hoc way of tying together emacs.git
and elpa.git.


This version:

http://git.savannah.gnu.org/cgit/emacs.git/log/?h=feature/integrated-elpa

uses package.el during the build process of Emacs, so that ELPA packages
could be added as packages. It requires more work. In the end, my own
feeling is that this is the right way. We could dramatically slim down
core Emacs to be enough to run package.el. The release would then be
"core plus what ever packages we think are important at the time".

This would decrease the complexity of the emacs git. But it might
increase the complexity of the release process, since you'd be dependent
on multiple other packages. I think ELPA and package.el need
to be able to cope with multiple versions of the same package,
supporting different versions of Emacs for this to work.


Phil

[Michael Welsh Duggan]

Post by Phillip Lord
http://git.savannah.gnu.org/cgit/emacs.git/log/?h=elparized-core
for example, just pulls out parts of ELPA using git magic, and copies
the files into core. Simple, straight-forward and it works. But,
ultimately, will it make maintaining core more easy? In the end, I think
not, because it is essentially an ad-hoc way of tying together emacs.git
and elpa.git.
http://git.savannah.gnu.org/cgit/emacs.git/log/?h=feature/integrated-elpa
uses package.el during the build process of Emacs, so that ELPA packages
could be added as packages. It requires more work. In the end, my own
feeling is that this is the right way. We could dramatically slim down
core Emacs to be enough to run package.el. The release would then be
"core plus what ever packages we think are important at the time".
This would decrease the complexity of the emacs git. But it might
increase the complexity of the release process, since you'd be dependent
on multiple other packages. I think ELPA and package.el need
to be able to cope with multiple versions of the same package,
supporting different versions of Emacs for this to work.

I would support any method that does _not_ require a working internet
connection in order to install packages. I have had to work on Emacs in
many, many places that have limited or non-existent network connections.
The ability to "move a package to external medium," then "install from
medium as a 'package' maintained by package-mode" would be very nice to
have.

This may not actually have anything to do with the text I quoted, but I
would like to put it out there anyway, just so it is in the back of
implementers' minds.

[Stefan Monnier]

Post by Michael Welsh Duggan
I would support any method that does _not_ require a working internet
connection in order to install packages.

We're discussing something quite different, tho: the proposal would be
something like "also clone elpa.git when you clone emacs.git", or "fetch
some packages from elpa.git while building the Emacs tarball".

Post by Michael Welsh Duggan
I have had to work on Emacs in
many, many places that have limited or non-existent network connections.
The ability to "move a package to external medium," then "install from
medium as a 'package' maintained by package-mode" would be very nice to
have.

There's `package-install-file` for that.


Stefan

[Achim Gratz]

Post by Phillip Lord
http://git.savannah.gnu.org/cgit/emacs.git/log/?h=feature/integrated-elpa
uses package.el during the build process of Emacs, so that ELPA packages
could be added as packages. It requires more work. In the end, my own
feeling is that this is the right way. We could dramatically slim down
core Emacs to be enough to run package.el. The release would then be
"core plus what ever packages we think are important at the time".

I think the point of contention previously was not from people using Git
anyway for all their builds, but from folks attuned to using tarballs.
So I think there'd need to be a way to create a tarball that looks like
a proper Git checkout of the release, probably with some info put
somewhere to record the exact state of these various checkouts at the
time of the archive creation and a build process that picks those up as
appropriate.

My personal preference would be if everything that isn't needed to
bootstrap Emacs moved to ELPA so that most of Emacs could be kept
up-to-date via the package manager.


Regards,
Achim.

[Eli Zaretskii]

Date: Mon, 16 Apr 2018 19:09:24 +0200
My personal preference would be if everything that isn't needed to
bootstrap Emacs moved to ELPA so that most of Emacs could be kept
up-to-date via the package manager.

Beware: XEmacs tried going that way, and failed, in that most users
ended up installing the "sumo" package. Let's not repeat past
mistakes, mmmkay?

[Achim Gratz]

Post by Eli Zaretskii
Beware: XEmacs tried going that way, and failed, in that most users
ended up installing the "sumo" package. Let's not repeat past
mistakes, mmmkay?

That was a different time when "most" users probably didn't have
always-on internet access. Again, you could have a "curated" set of
specially blessed ELPA packages if you wanted to and even provide it as a
meta package via ELPA.


Regards,
Achim.

[Phillip Lord]

Post by Achim Gratz

Post by Phillip Lord
http://git.savannah.gnu.org/cgit/emacs.git/log/?h=feature/integrated-elpa
uses package.el during the build process of Emacs, so that ELPA packages
could be added as packages. It requires more work. In the end, my own
feeling is that this is the right way. We could dramatically slim down
core Emacs to be enough to run package.el. The release would then be
"core plus what ever packages we think are important at the time".

I think the point of contention previously was not from people using Git
anyway for all their builds, but from folks attuned to using tarballs.
So I think there'd need to be a way to create a tarball that looks like
a proper Git checkout of the release, probably with some info put
somewhere to record the exact state of these various checkouts at the
time of the archive creation and a build process that picks those up as
appropriate.

This is actually quite straight-forward. One of my branche has this:

working/stream: Makefile.in
./bin/deploy-to-core 110b174643707 stream

110b174643707 is the git hash while "deploy-to-core" pulls out the code
for stream (from ELPA) and dumps it into the core structure.

Phil