Generators (iterators) for Gnu Emacs

[Michael Heerdegen]

Hello,

I want to contribute a package about generators to Gnu Emacs.

It is quite simple, but turned out to be surprisingly useful to have in
several situations. It is efficient and avoids recursion - iterating
across a generator's elements results in a simple `while' loop being
executed.

Daniel Colascione suggested to me to rename the thing to "iterators.el",
since this package doesn't use continuation passing style to implement
"yield". I guess that's reasonable.


Thanks,

Michael.

[Stefan Monnier]

Post by Michael Heerdegen
I want to contribute a package about generators to Gnu Emacs.

Thanks. Looks good. I generally prefer adding packages to GNU ELPA,
but I guess it also makes sense to add this directly to Emacs.

Post by Michael Heerdegen
Daniel Colascione suggested to me to rename the thing to "iterators.el",
since this package doesn't use continuation passing style to implement
"yield". I guess that's reasonable.

I don't have any opinion on the naming. A few comments below.


Stefan
Please state clearly which are all the basic operations one can perform on
a generator (AFAICT there's only one such operation, which is `gen-next').

It's great to see that you haven't needed anything else than `gen-next'.
I'd have expected a `gen-done-p' test to be needed/handy every once in
a while.

Post by Michael Heerdegen
(defmacro gen-make (&rest body)
"Return a generator that evaluates BODY to generate elements.
For each call of `gen-next' with the the returned generator, BODY
will be evaluated to produce an element."
(let ((this-element (make-symbol "this-element")))
`(let (,this-element)
(lambda ()

AFAICT this requires lexical-binding in the caller, so we might want to
signal an error if lexical-binding is nil.

Post by Michael Heerdegen
(if (eq ,this-element 'gen-done)
'gen-done

This code breaks down if your generator happens to return the symbol
`gen-done'. A better option is to have a (defconst gen-done <val>)
where the value is unique w.r.t `eq' (can be a string, a cons cell, an
uninterned symbol, you name it) and then to use this value rather than
hard-coding an interned symbol.

Post by Michael Heerdegen
(defun gen-next (generator)
(funcall generator))

A defalias would be more efficient.

Post by Michael Heerdegen
(if (not (null elements))

Aka (if elements

Post by Michael Heerdegen
The sequence of returned elements is starting with VALUE. Any

^^^
Add extra space (or break the line).

Post by Michael Heerdegen
successive element will be found by calling FUNCTION on the
preceding element."
(gen-append
(gen-from-elts value)
(gen-make (setq value (funcall function value)))))

I guess a `gen-cons' would be more efficient than this
(gen-append (gen-from-elts value) ...).

Post by Michael Heerdegen
(defun gen-number-range (&optional start end inc)
"Return a generator of a number sequence.
START denotes the first number and defaults to 1.

Tradition in computer science is to start counting from 0. There are
some notable exceptions (such as `point-min', which I regret), but
I strongly recommend to follow the tradition.

Post by Michael Heerdegen
The second optional argument END specifies the upper limit. If nil,
the returned generator is infinite. INC is the increment used between
numbers and defaults to 1."

Please be more precise about whether END is included or excluded.
Tradition (see again) is to exclude the upper bound (e.g. in dotimes).

Post by Michael Heerdegen
(defmacro gen-delay-expr (expression)

Have you made use of this? I'm not sure it really fits. I mean,
technically it works, but I'm not completely sure if pretending it's
a generator is a clever idea or not.

Post by Michael Heerdegen
(let ((done nil) el)
(gen-make
(if done 'gen-done
(setq el (gen-next generator))
(while (and (not (eq el 'gen-done))
(not (funcall predicate el)))
(setq el (gen-next generator)))
(if (eq el 'gen-done)
(setq done 'gen-done)
el)))))

Better move the `el' into the `gen-make' so it's not caught as a free
variable of the closure. And please apply de-Morgan's law.

Post by Michael Heerdegen
(let ((current (pop generators)) el (done nil) (skip (make-symbol "skip")))
(gen-delq
skip
(gen-make
(if done 'gen-done
(setq el (gen-next current))
(if (eq el 'gen-done)
(if (null generators)
(progn (setq done t) 'gen-done)
(setq current (pop generators))
skip)
el)))))))

Same here: sink `el' into the `gen-make'.

Post by Michael Heerdegen
(nreverse (gen-reduce (lambda (x y) (cons y x)) () generator)))

Too bad the order of arguments was not swapped so we could use #'cons
instead of this lambda.

Post by Michael Heerdegen
"Return true if PREDICATE is true for any element of GENERATOR."

^^^^
We call it "non-nil". Same for `gen-every'.

Post by Michael Heerdegen
(cl-callf or function #'identity)
(gen-max generator (and function (lambda (x) (- (funcall function x))))))

AFAICT, the first line makes sure `function' is not nil, so you can
remove the "(and function" on the second line.

[Leo Liu]

Post by Stefan Monnier
Thanks. Looks good. I generally prefer adding packages to GNU ELPA,
but I guess it also makes sense to add this directly to Emacs.

Isn't this a iterator (or stream) instead of generator?

Having generator would be great. Daniel, how mature is your generator
package?

Leo

[Daniel Colascione]

Post by Leo Liu

Post by Stefan Monnier
Thanks. Looks good. I generally prefer adding packages to GNU ELPA,
but I guess it also makes sense to add this directly to Emacs.

Isn't this a iterator (or stream) instead of generator?
Having generator would be great. Daniel, how mature is your generator
package?

It works well. The code generation in the CPS portions is fairly inefficient, but I don't think it matters all that much. Subroutines are compiled normally, of course, as are sub-forms that never yield.

[Leo Liu]

Post by Daniel Colascione
It works well. The code generation in the CPS portions is fairly
inefficient, but I don't think it matters all that much. Subroutines
are compiled normally, of course, as are sub-forms that never yield.

Stefan, any objection to make it part of emacs core?

Leo

[Stefan Monnier]

Post by Leo Liu

Post by Daniel Colascione
It works well. The code generation in the CPS portions is fairly
inefficient, but I don't think it matters all that much. Subroutines
are compiled normally, of course, as are sub-forms that never yield.

Stefan, any objection to make it part of emacs core?

Are we still talking about Micheal's code?


Stefan

[Leo Liu]

Post by Stefan Monnier
Are we still talking about Micheal's code?
Stefan

No. Daniel's package https://github.com/dcolascione/elisp-generators. Do
you have time to critique the code and make it an integral part of
elisp? Thanks ;)

Leo

[Stefan Monnier]

No. Daniel's package https://github.com/dcolascione/elisp-generators.
Do you have time to critique the code and make it an integral part of
elisp? Thanks ;)

I haven't had time to look at it in detail, but it looks nice.
Things I noticed:
- Drop the use of (require 'cl).
- Use `declare' for indentation and edebug specs.
- Most importantly: clean up the namespace use. `yield' is probably OK
(tho the global definition should probably not override a previous
definition) but `next' needs to turn into something like `gen-next',
`do-iterator' should probably turn into `gen-do', cl-loop's
`iterating' should probably turn into something that includes
"generator" in its name, `lambda-generator' should maybe be renamed to
`gen-lambda', ...

Once that's fixed, I'd be happy to see it included in Emacs.


Stefan

[Stefan Monnier]

Post by Leo Liu
No. Daniel's package https://github.com/dcolascione/elisp-generators. Do
you have time to critique the code and make it an integral part of
elisp? Thanks ;)

Oh, one more thing: is it technically important to sign the end by
signaling a `stop-iteration', rather than by returning a special value?
I think I'd prefer returning a special value (`condition-case' tends to
be on the costly side).


Stefan

[Daniel Colascione]

Post by Stefan Monnier

Post by Leo Liu
No. Daniel's package https://github.com/dcolascione/elisp-generators. Do
you have time to critique the code and make it an integral part of
elisp? Thanks ;)

Oh, one more thing: is it technically important to sign the end by
signaling a `stop-iteration', rather than by returning a special value?
I think I'd prefer returning a special value (`condition-case' tends to
be on the costly side).

I'd really rather signal to stop iteration. That way, it's harder to ignore end-of-iteration and loop forever and it remains possible to iterate over *any* value. condition-case is setjmp. I'm not *that* concerned by performance. The only other reasonable alternative is for iterators to yield a cons (terminal-p . value), and consing is going to be much worse than condition-case.

[Stefan Monnier]

Post by Daniel Colascione
I'd really rather signal to stop iteration. That way, it's harder to ignore
end-of-iteration and loop forever and it remains possible to iterate over
*any* value. condition-case is setjmp. I'm not *that* concerned by
performance. The only other reasonable alternative is for iterators to
yield a cons (terminal-p . value), and consing is going to be much worse
than condition-case.

The alternative I was thinking was to return a unique value, as in:

(defvar gen-eog (make-symbol "¡end-of-generator!"))
(defun gen-eog-p (x) (eq x gen-oeg))

In terms of efficiency it should be fine (contrary to consing, indeed).


Stefan

[Daniel Colascione]

Post by Stefan Monnier

Post by Daniel Colascione
I'd really rather signal to stop iteration. That way, it's harder to ignore
end-of-iteration and loop forever and it remains possible to iterate over
*any* value. condition-case is setjmp. I'm not *that* concerned by
performance. The only other reasonable alternative is for iterators to
yield a cons (terminal-p . value), and consing is going to be much worse
than condition-case.

(defvar gen-eog (make-symbol "¡end-of-generator!"))
(defun gen-eog-p (x) (eq x gen-oeg))

Right. That's the approach I don't like: now there's a value you can't enumerate. What if you wanted to enumerate, say, the values of all symbols?

[Daniel Colascione]

Post by Stefan Monnier

Post by Daniel Colascione
I'd really rather signal to stop iteration. That way, it's harder to ignore
end-of-iteration and loop forever and it remains possible to iterate over
*any* value. condition-case is setjmp. I'm not *that* concerned by
performance. The only other reasonable alternative is for iterators to
yield a cons (terminal-p . value), and consing is going to be much worse
than condition-case.

(defvar gen-eog (make-symbol "¡end-of-generator!"))
(defun gen-eog-p (x) (eq x gen-oeg))

Right. That's the approach I don't like: now there's a value you can't enumerate. What if you wanted to enumerate, say, the values of all symbols?

[Daniel Colascione]

Post by Stefan Monnier

Post by Daniel Colascione
I'd really rather signal to stop iteration. That way, it's harder to ignore
end-of-iteration and loop forever and it remains possible to iterate over
*any* value. condition-case is setjmp. I'm not *that* concerned by
performance. The only other reasonable alternative is for iterators to
yield a cons (terminal-p . value), and consing is going to be much worse
than condition-case.

(defvar gen-eog (make-symbol "¡end-of-generator!"))
(defun gen-eog-p (x) (eq x gen-oeg))

Right. That's the approach I don't like: now there's a value you can't enumerate. What if you wanted to enumerate, say, the values of all symbols?

[Daniel Colascione]

Post by Stefan Monnier

Post by Daniel Colascione
I'd really rather signal to stop iteration. That way, it's harder to ignore
end-of-iteration and loop forever and it remains possible to iterate over
*any* value. condition-case is setjmp. I'm not *that* concerned by
performance. The only other reasonable alternative is for iterators to
yield a cons (terminal-p . value), and consing is going to be much worse
than condition-case.

(defvar gen-eog (make-symbol "¡end-of-generator!"))
(defun gen-eog-p (x) (eq x gen-oeg))

Right. That's the approach I don't like: now there's a value you can't enumerate. What if you wanted to enumerate, say, the values of all symbols?

[Daniel Colascione]

Post by Stefan Monnier

Post by Daniel Colascione
I'd really rather signal to stop iteration. That way, it's harder to ignore
end-of-iteration and loop forever and it remains possible to iterate over
*any* value. condition-case is setjmp. I'm not *that* concerned by
performance. The only other reasonable alternative is for iterators to
yield a cons (terminal-p . value), and consing is going to be much worse
than condition-case.

(defvar gen-eog (make-symbol "¡end-of-generator!"))
(defun gen-eog-p (x) (eq x gen-oeg))

Right. That's the approach I don't like: now there's a value you can't enumerate. What if you wanted to enumerate, say, the values of all symbols?

[Daniel Colascione]

Post by Stefan Monnier

Post by Daniel Colascione
I'd really rather signal to stop iteration. That way, it's harder to ignore
end-of-iteration and loop forever and it remains possible to iterate over
*any* value. condition-case is setjmp. I'm not *that* concerned by
performance. The only other reasonable alternative is for iterators to
yield a cons (terminal-p . value), and consing is going to be much worse
than condition-case.

(defvar gen-eog (make-symbol "¡end-of-generator!"))
(defun gen-eog-p (x) (eq x gen-oeg))

Right. That's the approach I don't like: now there's a value you can't enumerate. What if you wanted to enumerate, say, the values of all symbols?

[Daniel Colascione]

Post by Stefan Monnier

Post by Daniel Colascione
I'd really rather signal to stop iteration. That way, it's harder to ignore
end-of-iteration and loop forever and it remains possible to iterate over
*any* value. condition-case is setjmp. I'm not *that* concerned by
performance. The only other reasonable alternative is for iterators to
yield a cons (terminal-p . value), and consing is going to be much worse
than condition-case.

(defvar gen-eog (make-symbol "¡end-of-generator!"))
(defun gen-eog-p (x) (eq x gen-oeg))

Right. That's the approach I don't like: now there's a value you can't enumerate. What if you wanted to enumerate, say, the values of all symbols?

[Stephen J. Turnbull]

Post by Daniel Colascione

Post by Stefan Monnier
(defvar gen-eog (make-symbol "¡end-of-generator!"))
(defun gen-eog-p (x) (eq x gen-oeg))

Right. That's the approach I don't like: now there's a value you
can't enumerate. What if you wanted to enumerate, say, the values
of all symbols?

The existence of `make-symbol' already means you can't do that.

[Daniel Colascione]

Post by Stefan Monnier

Post by Daniel Colascione
I'd really rather signal to stop iteration. That way, it's harder to ignore
end-of-iteration and loop forever and it remains possible to iterate over
*any* value. condition-case is setjmp. I'm not *that* concerned by
performance. The only other reasonable alternative is for iterators to
yield a cons (terminal-p . value), and consing is going to be much worse
than condition-case.

(defvar gen-eog (make-symbol "¡end-of-generator!"))
(defun gen-eog-p (x) (eq x gen-oeg))

Right. That's the approach I don't like: now there's a value you can't enumerate. What if you wanted to enumerate, say, the values of all symbols?

[Daniel Colascione]

Post by Stefan Monnier

Post by Daniel Colascione
I'd really rather signal to stop iteration. That way, it's harder to ignore
end-of-iteration and loop forever and it remains possible to iterate over
*any* value. condition-case is setjmp. I'm not *that* concerned by
performance. The only other reasonable alternative is for iterators to
yield a cons (terminal-p . value), and consing is going to be much worse
than condition-case.

(defvar gen-eog (make-symbol "¡end-of-generator!"))
(defun gen-eog-p (x) (eq x gen-oeg))

Right. That's the approach I don't like: now there's a value you can't enumerate. What if you wanted to enumerate, say, the values of all symbols?

[David Kastrup]

Post by Stefan Monnier

Post by Michael Heerdegen
(defmacro gen-make (&rest body)
"Return a generator that evaluates BODY to generate elements.
For each call of `gen-next' with the the returned generator, BODY
will be evaluated to produce an element."
(let ((this-element (make-symbol "this-element")))
`(let (,this-element)
(lambda ()

AFAICT this requires lexical-binding in the caller, so we might want to
signal an error if lexical-binding is nil.

Uh what? The whole point of the make-symbol abomination is to get a
symbol uniquely used by only one instance of the lambda function. This
is _exactly_ in order to _not_ require lexical binding.

Post by Stefan Monnier

Post by Michael Heerdegen
(if (eq ,this-element 'gen-done)
'gen-done

With lexical binding, this would look something like

(defmacro gen-mane (&rest body)
`(let (this-element)
(lambda ()
(if (eq this-element 'gen-done)
'gen-done
(setq this-element (progn ,@body))))))

And one would be tempted to see whether this really needs to be a macro.

[Stefan Monnier]

Post by David Kastrup
Uh what? The whole point of the make-symbol abomination is to get a
symbol uniquely used by only one instance of the lambda function.
This is _exactly_ in order to _not_ require lexical binding.

AFAICT this make-symbol is there to avoid name capture.


Stefan

[Michael Heerdegen]

Post by Stefan Monnier

Post by David Kastrup
Uh what? The whole point of the make-symbol abomination is to get a
symbol uniquely used by only one instance of the lambda function.
This is _exactly_ in order to _not_ require lexical binding.

AFAICT this make-symbol is there to avoid name capture.

Yes, that had been the intention. To make it work with dynamic callers,
I would have to defvar the uninterned symbol. Dunno if it's worth it.

Michael.

[Stefan Monnier]

Post by Michael Heerdegen
Yes, that had been the intention. To make it work with dynamic callers,
I would have to defvar the uninterned symbol.

Not only that, you'd have to build the lambda at run-time (i.e. make
the closures "by hand").

Post by Michael Heerdegen
Dunno if it's worth it.

It's not.


Stefan

[Michael Heerdegen]

Post by Stefan Monnier

Post by Michael Heerdegen
Yes, that had been the intention. To make it work with dynamic callers,
I would have to defvar the uninterned symbol.

Not only that, you'd have to build the lambda at run-time (i.e. make
the closures "by hand").

Would this version fix the issue?

--8<---------------cut here---------------start------------->8---
(defmacro gen-make (&rest body)
`(gen-make-1 (lambda () ,@body)))

(defun gen-make-1 (fun)
(let (this-element)
(lambda ()
(if (eq this-element 'gen-done)
'gen-done
(setq this-element (funcall fun))))))
--8<---------------cut here---------------end--------------->8---

Though, this `gen-make' is still inferior without lexical binding, but
at least should avoid the error.

Michael.

[Stefan Monnier]

Post by Michael Heerdegen
Would this version fix the issue?

No, with dynamic binding this just doesn't work. And we don't care,
because people can use lexical-binding just fine.


Stefan

[Michael Heerdegen]

Post by Stefan Monnier
A few comments below.

Thanks. Still using the term "generator" to avoid confusion, though I'm
going to rename the thing to "iterator".

Post by Stefan Monnier
It's great to see that you haven't needed anything else than `gen-next'.
I'd have expected a `gen-done-p' test to be needed/handy every once in
a while.

Dunno if that would be useful, I never missed it, but it should be
trivial to introduce.

Post by Stefan Monnier
I guess a `gen-cons' would be more efficient than this
(gen-append (gen-from-elts value) ...).

I'm not sure implementing `gen-cons' is a good idea. Generators are not
built from conses - but having `gen-cons' would allow to build generator
pseudo lists. This package is about introducing a different concept.
Plus, using `gen-cons' as a building block of generators would create
deeply nested lambdas that would soon hit `max-lisp-eval-depth' when
requesting elements. Avoiding recursion, which Emacs is not good at, is
crucial allover in this package.

Pushing to generators should not be needed repeatedly - if you think you
want this, use lists. In the above case, it is just coincidence that
the first generator has only one element. And I don't think efficiency
is an issue here. If you think it is I will rewrite that code, but
avoiding a `gen-cons'.

Post by Stefan Monnier

Post by Michael Heerdegen
(defmacro gen-delay-expr (expression)

Have you made use of this? I'm not sure it really fits. I mean,
technically it works, but I'm not completely sure if pretending it's
a generator is a clever idea or not.

I defined it because I need it in `gen-cache', and because it wasn't
existing yet in Emacs. Do you think delay-expression should be defined
somewhere else in Emacs? Then I would be happy to use it for
`gen-cache'.


Thanks,

Michael.

[Stefan Monnier]

Post by Michael Heerdegen

Post by Stefan Monnier
I guess a `gen-cons' would be more efficient than this
(gen-append (gen-from-elts value) ...).

I'm not sure implementing `gen-cons' is a good idea.

I only meant it as an optimization of (gen-append (gen-from-elts value) ...).
This form appears twice in your file.

Post by Michael Heerdegen

Post by Stefan Monnier

Post by Michael Heerdegen
(defmacro gen-delay-expr (expression)

Have you made use of this? I'm not sure it really fits. I mean,
technically it works, but I'm not completely sure if pretending it's
a generator is a clever idea or not.

I defined it because I need it in `gen-cache', and because it wasn't
existing yet in Emacs. Do you think delay-expression should be defined
somewhere else in Emacs? Then I would be happy to use it for
`gen-cache'.

Not sure. There's some overlap with url-future.el (which only has the
"url-" prefix because we didn't want to grab the "future-" prefix
without first getting more experience with the corresponding
abstraction).


Stefan

[Michael Heerdegen]

Post by Stefan Monnier
I only meant it as an optimization of (gen-append (gen-from-elts
value) ...). This form appears twice in your file.

Ok, I then can do it using some non-advertised helper function.

Post by Stefan Monnier
Not sure. There's some overlap with url-future.el (which only has the
"url-" prefix because we didn't want to grab the "future-" prefix
without first getting more experience with the corresponding
abstraction).

I would never have found that myself. But this looks good, it's like my
gen-delay plus better error handling. Hope it looses the url- prefix
some day.


Michael.

[Thierry Volpiatto]

Post by Michael Heerdegen
Hello,
I want to contribute a package about generators to Gnu Emacs.
It is quite simple, but turned out to be surprisingly useful to have in
several situations. It is efficient and avoids recursion - iterating
across a generator's elements results in a simple `while' loop being
executed.
Daniel Colascione suggested to me to rename the thing to "iterators.el",

We already have iterator.el.

[Stefan Monnier]

Post by Thierry Volpiatto
We already have iterator.el.

And it seems like it does pretty much the same thing as Michael's
library (at least `iter-next' looks eerily similar to `gen-next').

Is there a chance you two could merge those two libraries (I think
iterator.el is a good name, since generator.el is already taken by
Daniel's library which does something related yet different)?
Then we could install the result into Emacs?


Stefan

[Michael Heerdegen]

Post by Stefan Monnier
Is there a chance you two could merge those two libraries (I think
iterator.el is a good name, since generator.el is already taken by
Daniel's library which does something related yet different)?
Then we could install the result into Emacs?

I think we could do so, though, apart from the definition, our packages
seem quite disjoint.

Anyway, it would be good if Daniel's generators would then be valid
iterators in terms of the new iterator.el.

Michael.

[Stefan Monnier]

Post by Michael Heerdegen
I think we could do so, though, apart from the definition, our packages
seem quite disjoint.

I prefer to look at it as "complementary".

Post by Michael Heerdegen
Anyway, it would be good if Daniel's generators would then be valid
iterators in terms of the new iterator.el.

Indeed, that would be nice. Can you try and work that out?


Stefan

[Daniel Colascione]

Post by Stefan Monnier

Post by Michael Heerdegen
I think we could do so, though, apart from the definition, our packages
seem quite disjoint.

I prefer to look at it as "complementary".

Post by Michael Heerdegen
Anyway, it would be good if Daniel's generators would then be valid
iterators in terms of the new iterator.el.

Indeed, that would be nice. Can you try and work that out?

Assuming both packages are suitable for inclusion into core, we can definitely make them work together. I'd still very much like to use nonlocal control flow instead of a sentinel for enumeration termination, however.

That aside, I'm not sure I agree with your namespace preferences. `yield' is fundamentally a lexically-scoped macro. Making it defer to some prior definition makes no sense, since any code that can see this macro is already written with generators in mind.

I'd also strongly prefer using `next' as the iter-get function. If we're going to make the iteration protocol a core part of the environment, it deserves a prominent place in the global namespace. Using `funcall' is inadequate: first, it doesn't signal iteration in calling code, which can lead to confusion, and second, `funcall' doesn't work if we want to extent `next' to support things like lists, for which `next' would be equivalent to `pop'. AFAICT, nothing is already squatting on `next', so binding the symbol's function slot would be harmless for a compatibility perspective.

I'd also prefer using `iterating' as the cl-loop keyword. Because the keyword is only active in the lexical scope of a cl-loop construct, there's an even lower risk of collision, and the word "iterating" is exactly the right thing in the cl-loop language: (cl-loop for x iterating y collect x) reads very well.

[Michael Heerdegen]

Post by Daniel Colascione
Assuming both packages are suitable for inclusion into core, we can
definitely make them work together.

Thinking a bit about it, I guess my (only) constructor:

(defmacro gen-make (&rest body)
"Return a generator that evaluates BODY to generate elements.
(let ((this-element (make-symbol "this-element")))
`(let (,this-element)
(lambda ()
(if (eq ,this-element 'gen-done)
'gen-done
(setq ,this-element (progn ,@body)))))))

is just a special case of Daniel's `lambda-generator':

(defmacro gen-make (expr)
`(lambda-generator ()
(while t (yield ,expr))))

modulo (at least currently) the different kind of termination (right?).

So, all of my stuff can be expressed in terms of Daniels generators.
Which means that it would be a suitable extension to Daniels package,
defining some special generators, the transducers, and the cache, for
his more general kind of generators. It could then be named
"generator-x" or so.

Michael.

[Michael Heerdegen]

Post by Michael Heerdegen
So, all of my stuff can be expressed in terms of Daniels generators.
It could then be named "generator-x" or so.

The same is true for Thierry's code, I think, it could extend
Daniel's library too.


Michael.

[Stefan Monnier]

Post by Daniel Colascione
Assuming both packages are suitable for inclusion into core, we can
definitely make them work together.

That's the idea, yes.

Post by Daniel Colascione
I'd still very much like to use nonlocal control flow instead of
a sentinel for enumeration termination, however.

I'll let you guys decide together. I stated my preference, but I have
better things to worry about.

Post by Daniel Colascione
That aside, I'm not sure I agree with your namespace preferences. `yield' is
fundamentally a lexically-scoped macro.

The lexically scoped `yield' is OK. But the global (defmacro yield ...)
is not (tho, I'm OK with keeping it if wrapped in (unless (fboundp
'yield) ...)).

Post by Daniel Colascione
I'd also strongly prefer using `next' as the iter-get function.

I know, but this one is out of the question. All globally-visible names
need to be namespace-clean. So it can be `gen-next'.

Post by Daniel Colascione
I'd also prefer using `iterating' as the cl-loop keyword.

I know, but being globally visible, this is also out of the question: it
requires something with a "gen-" or at least some "generator" in its
name somewhere.

Post by Daniel Colascione
Because the keyword is only active in the lexical scope of a cl-loop
construct, there's an even lower risk of collision, and the word
(cl-loop for x iterating y collect x) reads very well.

But it doesn't say we're iterating on your particular kind of iteratable
construct.
[ BTW: regardless of namespace cleanliness, I don't find "iterating"
above to be a good choice, because "x" doesn't "iterate y".
The "cl-loop" doesn't do anything else than iterate, so "iterating" is
almost necessarily a misnomer in there. From an English language
point of view, I'd expect something more like "for x generated-by y"
or "for x in-generator y" or "for x in y". ]


Stefan

[Daniel Colascione]

Post by Stefan Monnier

Post by Daniel Colascione
That aside, I'm not sure I agree with your namespace preferences. `yield' is
fundamentally a lexically-scoped macro.

The lexically scoped `yield' is OK. But the global (defmacro yield ...)
is not (tho, I'm OK with keeping it if wrapped in (unless (fboundp
'yield) ...)).

Post by Daniel Colascione
I'd also strongly prefer using `next' as the iter-get function.

I know, but this one is out of the question. All globally-visible names
need to be namespace-clean. So it can be `gen-next'.

It *is* "namespace-clean", as I explained above. It's ludicrous to
expect that all future additions to elisp include horribly long,
hyphenated names even when they're describing fundamental features of
the environment. It's `car', goddammit it, not `seq-list-car'.

The point is to minimize the possibility of collision with useful user
code, not to religiously ensure that all symbols match ^[a-z]+-[a-z-]+$

[Stefan Monnier]

Post by Daniel Colascione
the environment. It's `car', goddammit it, not `seq-list-car'.

Yup. `car' sucks as well, but it's too late to fix it.
The prefix also helps discoverability, BTW.

Stefan

[Daniel Colascione]

Post by Stefan Monnier

Post by Daniel Colascione
the environment. It's `car', goddammit it, not `seq-list-car'.

Yup. `car' sucks as well, but it's too late to fix it.

No, `car' is fine: it's short and to the point. `next' would be too,
except that you insist that additions to the Emacs core look like a bowl
of corned beef hash, with chunks of `iter-'-this and `seq-' that
embedded all over the code for no good reason.

Prefixes are great for making sure out-of-tree packages don't conflict
with each other. For Emacs core functionality, requiring prefixes adds
friction and subtracts elegance, and because we manage all the names in
the core *anyway*, there's no risk of a conflict.

Any user code using a name like `next' for itself deserves to break, and
at least that pain is short-lived, while requiring some uselessly long
prefix for internal symbols makes life a little bit more difficult for
everyone forever.

[Stefan Monnier]

you insist that additions to the Emacs core look like a bowl of corned
beef hash, with chunks of `iter-'-this and `seq-' that embedded all
over the code for no good reason.

Indeed I do.


Stefan

[Daniel Colascione]

Post by Stefan Monnier

you insist that additions to the Emacs core look like a bowl of corned
beef hash, with chunks of `iter-'-this and `seq-' that embedded all
over the code for no good reason.

Indeed I do.

And you're really not interested in a discussion of your motives or the
circumstance under which you'd make an exception, are you? That's
supposed to motivate more contributions to core? Another recent thread
mentions the problem of too little development happening in-tree. I
wonder why that might be the case.

[Ted Zlatanov]

Post by Stefan Monnier

you insist that additions to the Emacs core look like a bowl of corned
beef hash, with chunks of `iter-'-this and `seq-' that embedded all
over the code for no good reason.

Indeed I do.

DC> And you're really not interested in a discussion of your motives or the
DC> circumstance under which you'd make an exception, are you? That's
DC> supposed to motivate more contributions to core? Another recent thread
DC> mentions the problem of too little development happening in-tree. I
DC> wonder why that might be the case.

I don't know Stefan's motives, but would love to see actual namespaces
in Emacs Lisp. The closest thing I've seen is
https://github.com/Bruce-Connor/names which can be installed from the
GNU ELPA. Perhaps that's an actual problem we keep ignoring because
it's easy to hack around it "just that once."

Ted

[Helmut Eller]

Post by Ted Zlatanov
I don't know Stefan's motives, but would love to see actual namespaces
in Emacs Lisp. The closest thing I've seen is
https://github.com/Bruce-Connor/names which can be installed from the
GNU ELPA. Perhaps that's an actual problem we keep ignoring because
it's easy to hack around it "just that once."

Thanks for the pointer.

I also wrote a namespace macro: https://github.com/ellerh/namespace.el
Another one I know is: https://github.com/chrisbarrett/elisp-namespaces

The experience with my own namespace macro was that it works fairly well
when writing a library from scratch, in particular when the code
consists mostly of functions and almost no quoted symbols. If it works
well, then the code looks a lot neater with namespaces.

But it was also rather useless when I tried it to use it with the SLIME
codebase. The existing code uses prefixes in inconsistent ways and
quoted symbols are needed surprisingly often.

This may also be a bit of a chicken-and-egg problem: if the existing
code can't be used with those namespace macros than nobody will use
namespaces, but as soon as some important libraries can be used with
namespaces it pays of to use namespaces and we get more usable
libraries.

Helmut

[Artur Malabarba]

Post by Helmut Eller

Post by Ted Zlatanov
I don't know Stefan's motives, but would love to see actual namespaces
in Emacs Lisp. The closest thing I've seen is
https://github.com/Bruce-Connor/names which can be installed from the
GNU ELPA. Perhaps that's an actual problem we keep ignoring because
it's easy to hack around it "just that once."

Thanks for the pointer.
I also wrote a namespace macro: https://github.com/ellerh/namespace.el
Another one I know is: https://github.com/chrisbarrett/elisp-namespaces
The experience with my own namespace macro was that it works fairly well
when writing a library from scratch, in particular when the code
consists mostly of functions and almost no quoted symbols. If it works
well, then the code looks a lot neater with namespaces.
But it was also rather useless when I tried it to use it with the SLIME
codebase. The existing code uses prefixes in inconsistent ways and
quoted symbols are needed surprisingly often.

Names.el partially solves that by namespacing sharp-quoted symbols as
function names (you have to use the sharp-quote on function names for this
to work, but that's good practice anyway). This catches the majority of the
uses of quotes for me.

Quoted variable names are impossible to identify with certainty, so Names
doesn't try to namespace them by default. But but it does offer the option
to namespace regular quoted symbols as variable names.

[Helmut Eller]

Post by Artur Malabarba
Names.el partially solves that by namespacing sharp-quoted symbols as
function names (you have to use the sharp-quote on function names for this to
work, but that's good practice anyway). This catches the majority of the uses
of quotes for me.

For me define-key was a problematic and common case. Until now I
believed that replacing 'COMMAND in (define-key ... 'COMMAND) with
#'COMMAND is inappropriate because I thought that #'F evaluates to
(symbol-function 'F).

But it turns out that my intuition was wrong and that (eq #'F 'F) => t
for all symbols F. So #' can be used to mark function names in more
cases than I thought. Which is good news for the namespace macro.

Helmut

[Artur Malabarba]

Post by Helmut Eller
But it turns out that my intuition was wrong and that (eq #'F 'F) => t
for all symbols F. So #' can be used to mark function names in more
cases than I thought. Which is good news for the namespace macro.

Indeed, now a days #' is really just a note to the compiler. See for
instance
http://endlessparentheses.com/get-in-the-habit-of-using-sharp-quote.html

Below is a short example of a package using the macro, in case anyone's
interested. I particularly find it a pleasure to write and read code like
this.
https://github.com/Bruce-Connor/camcorder.el

[Helmut Eller]

Post by Artur Malabarba
Below is a short example of a package using the macro, in case anyone's
interested. I particularly find it a pleasure to write and read code like
this.
https://github.com/Bruce-Connor/camcorder.el

Yes, in most cases dropping the "camcorder-" prefix is a win.
But this part:

(define-minor-mode mode nil nil "sc"
'(([f12] . camcorder-stop)
([f11] . camcorder-pause))
... )

doesn't look so great to me, because 1) "define-minor-mode mode" is
really useless for grep 2) it still uses the "camcorder-" prefix for key
bindings.

Helmut

[Artur Malabarba]

Post by Helmut Eller
Yes, in most cases dropping the "camcorder-" prefix is a win.
(define-minor-mode mode nil nil "sc"
'(([f12] . camcorder-stop)
([f11] . camcorder-pause))
... )
doesn't look so great to me,

Yes. That package is a good example because it portrays how often it works
and how often it doesn't.
I personally find these small issues completely worth the overall beauty of
the code, but that's entirely a matter of preference.

FWIW, Both issues are workaroundable (I'm guessing that's not a word :-)).

Post by Helmut Eller
because 1) "define-minor-mode mode" is
really useless for grep

This will be the case for any namespacing solution. You can work around by
defining the minor mode after the macro.
Also, I would be happy to add an option to avoid namespacing the
`define-minor-mode' form (and similar ones), if anyone wants it.

Post by Helmut Eller
2) it still uses the "camcorder-" prefix for key
bindings.

This can also be worked around by defining the keymap with a series of
`define-key' statements instead of doing it inside the `define-minor-mode'.
Which is something I see a lot of packages doing anyway.

Now that I think about it, I can also improve the macro to "look inside"
the fifth argument of a `define-minor-mode' call even if it's a quoted
list, since that argument is guaranteed to be a keymap.

[Dmitry Gutov]

Post by Artur Malabarba
This will be the case for any namespacing solution. You can work around by
defining the minor mode after the macro.

Not if the reader is namespace-aware. (See Clojure backquote).

[Artur Malabarba]

Post by Dmitry Gutov

Post by Artur Malabarba
This will be the case for any namespacing solution. You can work around by
defining the minor mode after the macro.

Not if the reader is namespace-aware. (See Clojure backquote).

I don't understand. What does that have to do with grep?

[Dmitry Gutov]

Post by Artur Malabarba

Post by Dmitry Gutov

Post by Artur Malabarba
This will be the case for any namespacing solution. You can work

around by

Post by Dmitry Gutov

Post by Artur Malabarba
defining the minor mode after the macro.

Not if the reader is namespace-aware. (See Clojure backquote).

I don't understand. What does that have to do with grep?

Sorry, the above indeed doesn't make sense, and was speed-reading, and
assumed the above referred to the item two.

What I meant to point out is that is the reader is namespace-aware, you
can resolve the local names with special syntax, instead of making the
namespace macro aware of `define-minor-mode' specifically.

[Helmut Eller]

Post by Artur Malabarba

Post by Helmut Eller
because 1) "define-minor-mode mode" is
really useless for grep

This will be the case for any namespacing solution. You can work around by
defining the minor mode after the macro.
Also, I would be happy to add an option to avoid namespacing the
`define-minor-mode' form (and similar ones), if anyone wants it.

Some commands like M-x gnus or M-x rmail don't have proper prefixes. I
suppose those would also need to be defined after the macro or with an
option to suppress automatic prefixing.

Post by Artur Malabarba

Post by Helmut Eller
2) it still uses the "camcorder-" prefix for key
bindings.

This can also be worked around by defining the keymap with a series of
`define-key' statements instead of doing it inside the `define-minor-mode'.
Which is something I see a lot of packages doing anyway.

Something like `(([f12] . ,#'stop)) would probably also work. Not
exactly an improvement, though.

Post by Artur Malabarba
Now that I think about it, I can also improve the macro to "look inside" the
fifth argument of a `define-minor-mode' call even if it's a quoted list,
since that argument is guaranteed to be a keymap.

I guess that temptation will come up often: just add a special case for
macro X.

Helmut

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider ]]]
[[[ whether defending the US Constitution against all enemies, ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Post by Helmut Eller
Some commands like M-x gnus or M-x rmail don't have proper prefixes.

Could you explain what you mean? The names 'gnus' and 'rmail' are fine.
They are commands to invoke packages with the same names.

[Artur Malabarba]

Post by Richard Stallman
[[[ To any NSA and FBI agents reading my email: please consider ]]]
[[[ whether defending the US Constitution against all enemies, ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Post by Helmut Eller
Some commands like M-x gnus or M-x rmail don't have proper prefixes.

Could you explain what you mean? The names 'gnus' and 'rmail' are fine.
They are commands to invoke packages with the same names.

He was just saying there's no way to define commands with these names
inside a Names.el namespace (because all definitions inside it are prefixed
with the package name).

He was just mentioning a limitation of Names.el, not criticising those
commands. (though, like I explained, it's not really a limitation).

[Helmut Eller]

Post by Richard Stallman

Post by Helmut Eller
Some commands like M-x gnus or M-x rmail don't have proper prefixes.

Could you explain what you mean?

Just that "gnus-" is not a prefix of "gnus". If the prefix "gnus-" is
inserted automatically one would get names like "gnus-gnus" or perhaps
"gnus-".

Post by Richard Stallman
The names 'gnus' and 'rmail' are fine.

Sure.

Helmut

[Drew Adams]

would love to see actual namespaces in Emacs Lisp.

+1

Prefixes are a lousy workaround. Something like Common Lisp
packages would be good. Explicit import/export, allowing
reference without the package prefix, etc.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider ]]]
[[[ whether defending the US Constitution against all enemies, ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Post by Ted Zlatanov
I don't know Stefan's motives, but would love to see actual namespaces
in Emacs Lisp.

The idea sounds nice in theory, but in practice multiple name spaces
do not fit into Lisp very well. Common Lisp packages are an unclean
kludge; this was clear to me when I implemented them in the 1980s in
the Lisp Machine. It is impossible to use them in the way one would
wish to use them.

In practice, you have to write the package prefix whenever you refer
to a symbol that has one. It might as well be part of the symbol name
itself. Thus, packages complicate the language definition while
providing no benefit.

So in GNU Emacs I decided to make them part of the symbol name itself
and not have packages.

[Daniel Colascione]

Post by Richard Stallman

Post by Ted Zlatanov
I don't know Stefan's motives, but would love to see actual namespaces
in Emacs Lisp.

The idea sounds nice in theory, but in practice multiple name spaces
do not fit into Lisp very well. Common Lisp packages are an unclean
kludge; this was clear to me when I implemented them in the 1980s in
the Lisp Machine. It is impossible to use them in the way one would
wish to use them.

The entire Common Lisp ecosystem is a counterexample so your rather
sweeping declaration. In practice, namespaces are rather useful despite
the need to *sometimes* fully *some* symbols. It's still better than
having to fully specify *every* symbol on *every* use.

Post by Richard Stallman
In practice, you have to write the package prefix whenever you refer
to a symbol that has one. It might as well be part of the symbol name
itself. Thus, packages complicate the language definition while
providing no benefit.

There's a lot of Lisp code out there that uses packages in the way they
were meant to be used, and your personal distaste for the mechanism
shouldn't keep namespaces out of elisp.

[Drew Adams]

Post by Daniel Colascione
The entire Common Lisp ecosystem is a counterexample so your rather
sweeping declaration. In practice, namespaces are rather useful
despite the need to *sometimes* fully *some* symbols. It's still
better than having to fully specify *every* symbol on *every* use.

That was my experience also.

Post by Daniel Colascione

Post by Richard Stallman
In practice, you have to write the package prefix whenever you
refer to a symbol that has one. It might as well be part of the
symbol name itself. Thus, packages complicate the language
definition while providing no benefit.

There's a lot of Lisp code out there that uses packages in the way
they were meant to be used, and your personal distaste for the
mechanism shouldn't keep namespaces out of elisp.

I wonder whether perhaps Richard ran into the need to use prefixes
so much because he was implementing the CL package system, and not
just making normal use of it. Just a conjecture.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider ]]]
[[[ whether defending the US Constitution against all enemies, ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Post by Drew Adams
I wonder whether perhaps Richard ran into the need to use prefixes
so much because he was implementing the CL package system, and not
just making normal use of it. Just a conjecture.

I implemented all the Common Lisp features (of which the package
system was one) in the Lisp Machine operating system, which included
lots of programs to do lots of things. (An Emacs-like editor was one
of them.)

In practice, the goal of using names without prefixes via importing
namespace did not work out.

I make the conjecture that name space importing is convenient when you
have lots of small modules, each with its own name space and not many
names, and each referring to just a few other modules. That's not how
the Lisp Machine system was written, and it's also mostly not how
Emacs was written.

[Stefan Monnier]

Post by Richard Stallman
I make the conjecture that name space importing is convenient when you
have lots of small modules, each with its own name space and not many
names, and each referring to just a few other modules. That's not how
the Lisp Machine system was written, and it's also mostly not how
Emacs was written.

In languages like SML, the common style is to always use qualified
names, tho the modules used in a file get locally renamed to use
shorter prefixes.


Stefan

[Ivan Shmakov]

Post by Stefan Monnier

Post by Richard Stallman
I make the conjecture that name space importing is convenient when
you have lots of small modules, each with its own name space and not
many names, and each referring to just a few other modules. That's
not how the Lisp Machine system was written, and it's also mostly
not how Emacs was written.

In languages like SML, the common style is to always use qualified
names, tho the modules used in a file get locally renamed to use
shorter prefixes.

FWIW, my own preference for namespace-aware languages (that’s
Scheme 48, Perl 5, and a few others) was always to import only
what’s strictly necessary.

Typically, of each module, only a few functions are used, so
even without a prefix, they don’t really clutter the namespace.

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider ]]]
[[[ whether defending the US Constitution against all enemies, ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Post by Daniel Colascione
The entire Common Lisp ecosystem is a counterexample so your rather
sweeping declaration.

I have no way of knowing what is proved by Common Lisp programming.
I don't have time to study people's Common Lisp programs.

In practice, namespaces are rather useful despite

Post by Daniel Colascione
the need to *sometimes* fully *some* symbols. It's still better than
having to fully specify *every* symbol on *every* use.

In the Lisp Machine system, importing proved to be unusable.

If it is true that it is usable in some programs, the question is:
what are the circumstances that make them usable in some circumstances
and not in others? In Emacs code, would they be usable or unusable?

I made a conjecture in another message just now.

[Drew Adams]

Post by Richard Stallman

Post by Drew Adams
I wonder whether perhaps Richard ran into the need to use
prefixes so much because he was implementing the CL package
system, and not just making normal use of it. Just a conjecture.

I implemented all the Common Lisp features (of which the package
system was one) in the Lisp Machine operating system, which included
lots of programs to do lots of things. (An Emacs-like editor was
one of them.)
In practice, the goal of using names without prefixes via importing
namespace did not work out.
I make the conjecture that name space importing is convenient when
you have lots of small modules, each with its own name space and not
many names, and each referring to just a few other modules. That's not
how the Lisp Machine system was written, and it's also mostly not how
Emacs was written.

Maybe. Dunno what constitutes small, not many, few, etc.
Perhaps others can chime in with more relevant info here.
Sorry; I don't have anything concrete to offer in this regard.

[Stefan Monnier]

Post by Daniel Colascione
And you're really not interested in a discussion of your motives or the
circumstance under which you'd make an exception, are you?

I think we've already discussed it at length in the cl-lib era.
It's largely an aesthetic issue: you want to save those 4 "gen-"
characters, whereas I want to keep the namespace clean. As mentioned
I also want to keep things discoverable and modular, and given the lack
of real modules, using a "gen-" prefix is the way to make things
discoverable and modular. Note that when I code in languages with good
module systems, I still use "prefixes", tho instead of "gen-" it would
probably be shortened to "G." or something like that.

As can be seen with the seq.el library, I want to move towards *more*
use of prefixes even for "core" functionality. This has been the case
for many many years. Even back in Emacs-21 times, my newcomment.el did
that, taking non-namespace-clean code from simple.el (or maybe it was
subr.el, can't remember) and making it use "comment-".

Over the years, my belief in the importance of using such prefixes has
only strengthened. I know many people disagree, but at this point
there's really not much room for negotiation.
We should simply agree to disagree.


Stefan


PS: In the case of "next", we could solve the namespace issue without
modules if Elisp had an equivalent to "foo.next". But the only OO-style
system we have uses methods which have global scope, so we'd need
another object system with a different method-call syntax like maybe
(foo :next).

[Nic Ferrier]

Post by Stefan Monnier
PS: In the case of "next", we could solve the namespace issue without
modules if Elisp had an equivalent to "foo.next". But the only OO-style
system we have uses methods which have global scope, so we'd need
another object system with a different method-call syntax like maybe
(foo :next).

So agree to add some more core things to make the problem easier.

A with-symbol for example.


Nic

[Stefan Monnier]

Post by Nic Ferrier
So agree to add some more core things to make the problem easier.

I think I'd be willing to add Objects, indeed. At least the
completion-tables scream "make me an object".

Post by Nic Ferrier
A with-symbol for example.

I don't know what `with-symbol' you're referring to.


Stefan

[Nic Ferrier]

Post by Stefan Monnier

Post by Nic Ferrier
So agree to add some more core things to make the problem easier.

I think I'd be willing to add Objects, indeed. At least the
completion-tables scream "make me an object".

Post by Nic Ferrier
A with-symbol for example.

I don't know what `with-symbol' you're referring to.

It's just a simple symbol-let, we discussed it before here I think. An
alternative to a name system is to define a namespace with a macro:

(defmacro my-package (&rest body)
(symbol-macrolet ((foo my-package--foo)
(bar my-package--bar))
,@body))

But that isn't exactly a friendly pattern is it?


Nic

[Daniel Colascione]

Post by Stefan Monnier

Post by Nic Ferrier
So agree to add some more core things to make the problem easier.

I think I'd be willing to add Objects, indeed. At least the
completion-tables scream "make me an object".

We *have* objects. Nothing is stopping you from using EIEIO for
completion tables. But I don't think you're talking about polymorphism
and encapsulation here: ITYM that we can solve the global namespace
problem by attacking the "global" part instead of the "namespace" part.
If we have your "Objects", many functions would be instead methods (or
messages or whatever terminology you want to choose) that have meaning
only in relation to one or more of the values acted upon by these
functions, without global names of their own.

That's a terrible idea. It makes it much harder to reason about the
correct operation of programs, since now it makes it virtually
impossible to implement find-definition functionality without lots of
type inference. There's also the runtime overhead of dynamic dispatch.
Using "Objects" doesn't even solve the problem it sets out to solve,
since you still need elaborately-mangled global names for the types of
the objects themselves.

Common Lisp solved this problem 20 years ago with namespaces. We should
just implement CL namespaces instead of trying to shoehorn Smalltalk
into the language. Would you accept a CL packages implementation?

[Drew Adams]

Post by Daniel Colascione
Common Lisp solved this problem 20 years ago with namespaces. We
should just implement CL namespaces instead of trying to shoehorn
Smalltalk into the language. Would you accept a CL packages
implementation?

+1

(And it was 30 years ago.)

[Stefan Monnier]

Post by Daniel Colascione
That's a terrible idea. It makes it much harder to reason about the
correct operation of programs, since now it makes it virtually
impossible to implement find-definition functionality without lots of
type inference.

That's indeed the main problem, and I agree it's a serious one.

Post by Daniel Colascione
There's also the runtime overhead of dynamic dispatch.

That's not relevant: one way or another there is dynamic dispatch,
regardless what system we use.

Post by Daniel Colascione
Common Lisp solved this problem 20 years ago with namespaces. We should
just implement CL namespaces instead of trying to shoehorn Smalltalk
into the language. Would you accept a CL packages implementation?

Last time this was discussed (a year ago, maybe?), we neded turning it
down, mostly because `M-x grep RET' can't be used for find-definition
and find-uses.


Stefan

[Daniel Colascione]

Post by Stefan Monnier

Post by Daniel Colascione
That's a terrible idea. It makes it much harder to reason about the
correct operation of programs, since now it makes it virtually
impossible to implement find-definition functionality without lots of
type inference.

That's indeed the main problem, and I agree it's a serious one.

Post by Daniel Colascione
There's also the runtime overhead of dynamic dispatch.

That's not relevant: one way or another there is dynamic dispatch,
regardless what system we use.

That's not true: if we send messages to objects, we can only know the
code to execute after examining the object. If we invoke functions
directly, we know *statically* what code runs where.

Post by Stefan Monnier

Post by Daniel Colascione
Common Lisp solved this problem 20 years ago with namespaces. We should
just implement CL namespaces instead of trying to shoehorn Smalltalk
into the language. Would you accept a CL packages implementation?

Last time this was discussed (a year ago, maybe?), we neded turning it
down, mostly because `M-x grep RET' can't be used for find-definition
and find-uses.

*Any* solution that frees developers from spelling all symbols in full
will have that disadvantage. At least namespaces make it relatively
symbol to determine the fully-resolved symbol from lexical context
without requiring a full type inference pass.

[Stefan Monnier]

Post by Daniel Colascione
*Any* solution that frees developers from spelling all symbols in full
will have that disadvantage.

That's right.


Stefan

[Daniel Colascione]

Post by Stefan Monnier

Post by Daniel Colascione
*Any* solution that frees developers from spelling all symbols in full
will have that disadvantage.

That's right.

So what? There are definitely offsetting benefits. Or are you saying M-x
grep is our top priority now?

[Stefan Monnier]

Post by Stefan Monnier

Post by Daniel Colascione
*Any* solution that frees developers from spelling all symbols in full
will have that disadvantage.

That's right.

So what? There are definitely offsetting benefits. Or are you saying M-x
grep is our top priority now?

I'm ambivalent on this issue, so take my word as an summary of my
understanding of the last discussion about it.

In other words, any improvement that tries to reduce the redundancy in
identifiers will require corresponding improvement in our tools to infer
the now-implicit information.

So we could bite the bullet and hope that someone will step up to the
plate and improve CEDET (or equivalent) to make up the difference.
Or we can sit here and wait for CEDET to improve first.
Or we can decide that this redundancy is really not that costly compared
to the benefits of being able to use dumb tools rather than being
dependent on more sophisticated packages.

When you said:

That's a terrible idea. It makes it much harder to reason about the
correct operation of programs, since now it makes it virtually
impossible to implement find-definition functionality without lots of
type inference.

you put your finger on the issue, indeed. Using namespaces/modules
rather than Smalltalk-style objects would undoubtedly require
significantly less sophisticated analysis, but the same kind of tradeoff
is at play.

So for now, I insist on `gen-next' and `cl-flet'. Even if it makes me
unpopular.

And note that I decided `cl-setf' was important/popular/frequent enough
to deserve to be promoted to `setf', and the same could happen at some
point to `gen-next' or `cl-flet'.


Stefan

[Thien-Thi Nguyen]

() Stefan Monnier <***@iro.umontreal.ca>
() Sun, 07 Dec 2014 22:23:58 -0500

In other words, any improvement that tries to reduce the
redundancy in identifiers will require corresponding
improvement in our tools to infer the now-implicit
information.

Right. Sez Alan Perlis:

Wherever there is modularity there is the potential for
misunderstanding: Hiding information implies a need to
check communication.

so the onerous activity here is the "check communication".
I would argue that the current FOO- scheme, being merely
convention, is actually more needful of inference than the
proposed FOO:: scheme. If modularity is directly supported,
we (programs and humans, both) can turn from heuristics to
algorithms, w/ corresponding reduction in worry and angst
that the "check" is incomplete or incoherent.

That said, my experience (largely positive) is entirely w/
Guile Scheme modules, and i imagine that Emacs' modules would
be similar. I don't know how Common Lisp does it -- maybe
someone who knows both systems could post a comparison?

[David Engster]

Post by Stefan Monnier

Post by Stefan Monnier

Post by Daniel Colascione
*Any* solution that frees developers from spelling all symbols in full
will have that disadvantage.

That's right.

So what? There are definitely offsetting benefits. Or are you saying M-x
grep is our top priority now?

I'm ambivalent on this issue, so take my word as an summary of my
understanding of the last discussion about it.
In other words, any improvement that tries to reduce the redundancy in
identifiers will require corresponding improvement in our tools to infer
the now-implicit information.
So we could bite the bullet and hope that someone will step up to the
plate and improve CEDET (or equivalent) to make up the difference.
Or we can sit here and wait for CEDET to improve first.

CEDET has an Elisp parser. It's a bit difficult for me to extend it to
parse namespaces without knowing the syntax for that, but I can try.

Seriously: Yes, the parser could be improved. But it still would have
several drawbacks compared to grep et al.:

. It is slow.
. You need to set up some kind of project beforehand so that Semantic
knows what to parse and where to find dependencies.
. It will only work inside Emacs.
. I guess that Changelog thingy will become more complicated.
. Did I mention it's slow?

I'm all for it.

-David

[Stefan Monnier]

Post by David Engster
Seriously: Yes, the parser could be improved. But it still would have
. It is slow.
. You need to set up some kind of project beforehand so that Semantic
knows what to parse and where to find dependencies.
. It will only work inside Emacs.
. I guess that Changelog thingy will become more complicated.
. Did I mention it's slow?
I'm all for it.

As mentioned in my sample grep command, it'd have to parse all the emacs
and elpa Elisp files, in order to find all potential references.


Stefan

[Dmitry Gutov]

Post by Stefan Monnier

Post by Daniel Colascione
Common Lisp solved this problem 20 years ago with namespaces. We should
just implement CL namespaces instead of trying to shoehorn Smalltalk
into the language. Would you accept a CL packages implementation?

Last time this was discussed (a year ago, maybe?), we neded turning it
down, mostly because `M-x grep RET' can't be used for find-definition
and find-uses.

Could we revisit this discussion, maybe? Having a Lisp VM at the core of
Emacs means we don't need to use grep for find-definition.

And find-uses can be implemented in a lisp-only namespace-aware
searching function.

Clojure (a rather popular Lisp) uses what sounds to be a similar
packages system, and the popular editing environments implement both
above operations without considerable problems.

[Stefan Monnier]

Post by Dmitry Gutov
Could we revisit this discussion, maybe? Having a Lisp VM at the core of
Emacs means we don't need to use grep for find-definition.

I'm sure it can be made to work, with enough work. But currently

M-x grep RET grep foo-bar {emacs,elpa}/**/*.el RET

works pretty well, for *very* little effort.


Stefan

[Dmitry Gutov]

Post by Stefan Monnier

Post by Dmitry Gutov
Could we revisit this discussion, maybe? Having a Lisp VM at the core of
Emacs means we don't need to use grep for find-definition.

I'm sure it can be made to work, with enough work. But currently
M-x grep RET grep foo-bar {emacs,elpa}/**/*.el RET
works pretty well, for *very* little effort.

Sure. In no way I'm trying to say that having to implement the above
would be much of a benefit by itself.

It's a cost, for having a modern-ish import system.

[David Kastrup]

Post by Stefan Monnier

Post by Daniel Colascione
the environment. It's `car', goddammit it, not `seq-list-car'.

Yup. `car' sucks as well, but it's too late to fix it.
The prefix also helps discoverability, BTW.

What was it again? "contents of address register" or something as
opposed to "contents of decrement register". Rather arbitrary terms for
first element and the tail of a list. If anything, I'd have expected
the tail to be in an address register.

But that's heritage. Not even Scheme dared touching it.

[Stephen J. Turnbull]

Post by David Kastrup
What was it again? "contents of address register" or something as
opposed to "contents of decrement register". Rather arbitrary terms for
first element and the tail of a list. If anything, I'd have expected
the tail to be in an address register.

Bigendian word packing, IIRC.