Discussion:
EWW improvements: open in new buffer, tags, quickmarks, search engines, ...
Pierre Neidhardt
2018-04-16 10:26:40 UTC
Permalink
I've recently hacked around EWW and since it's coming into good shape
I'm thinking of committing this upstream.

Before sending a patch, here is the current state of my hacks:

https://github.com/Ambrevar/dotfiles/blob/master/.emacs.d/lisp/init-eww.el

Summary:

- Add `eww-copy-page-title' that mirrors `eww-copy-page-url', it's
useful enough.

- Enhances eww-next-url / eww-previous-url so that when there is no next /
previous links, it tries to increment / decrement the last number in
the last element of the URL. ("do what you mean" style.)

- Add a `eww-reload-all' command to reload all *eww* buffers, it's
useful when using desktop-mode but when `eww-restore-desktop' is nil.

- Make `eww' so that the default value is in the prompt, ready to be
edited by the user. Very convenient in my opinion.

- Change `eww-open-in-new-buffer' so that it queries for a URL instead
of cloning the current buffer (which is not very useful in my
opinion).

- Make eww-update-header-line-format also update the buffer name so that
it contains the page title. _Very useful_ to browse / search *eww*
buffers (think Ivy/Helm).

- Ask for tags when saving a bookmark. Tags are stored under the key
:tags as a list of strings.

- Make `eww-add-bookmark' run a customizable function to decide when.
to error out. For instance, error out when a duplicate is detected
with protocol stripped out (https://foo.bar is seen as a duplicate of
http://foo.bar).

- Make `eww-write-bookmarks' run a customizable function before saving
the file. That function can be used, for instance, to detect
duplicates or to sort the bookmarks. This would make the eww-bookmarks
file more friendly to versioning.

- Bookmarks can have a mark which is a string saved under the key :mark.
The mark should be unique. It could be used like the "quickmark"
function found in some browsers: use it to quickly load a
bookmark. (Work in progress.)

- Bookmarks can have a search engine which is either appended to the
bookmark' URL if it does not start with "https?://", or used as-is
otherwise.
The search engine is stored as a string under the key :search.
A "%s" must be present in the search engine string as a place-holder
for the query.

- Make `eww-bookmark-prepare' only load bookmarks from file if not
already set. This makes it possible to display a custom / narrowed
list of bookmarks in the bookmark buffer.

- Make `eww-bookmark-prepare' display the mark, the tags and the search
engine, if available. Work in progress. I'm thinking of using a
different face for the mark if a search engine is present.

- Add a `eww-bookmarks-by-tags' command which queries the user for a
completing list of tags and then displays a bookmark buffers of all
the bookmarks which match the tags. The matching can be either
inclusive or exclusive (bookmarks which match at least one tag vs. all
of them).

- Make `eww--dwim-expand-url' follow a different logic to bind it all together:

- With a multi-word query, if first word is a mark of a bookmark with a search engine,
then use the said search engine over the rest of the query.

- With a single word query, if first word is a mark then open the
corresponding bookmark.

- Else query the default search engine.

- Fix `eww-forward-url' as it seems to corrupt the history. (Work in progress.)

Of course in its present state my hacks are what they are, very hacky.
It needs to be made more customizable and interfaceable.

What do you think?

--
Pierre Neidhardt

Power corrupts. Absolute power is kind of neat.
-- John Lehman, Secretary of the Navy, 1981-1987
Lars Ingebrigtsen
2018-04-16 10:54:49 UTC
Permalink
Post by Pierre Neidhardt
I've recently hacked around EWW and since it's coming into good shape
I'm thinking of committing this upstream.
Please send separate patches for each feature. :-)
Post by Pierre Neidhardt
- Add `eww-copy-page-title' that mirrors `eww-copy-page-url', it's
useful enough.
Yup.
Post by Pierre Neidhardt
- Enhances eww-next-url / eww-previous-url so that when there is no next /
previous links, it tries to increment / decrement the last number in
the last element of the URL. ("do what you mean" style.)
Hm... Are there many web sites where that is meaningful?
Post by Pierre Neidhardt
- Add a `eww-reload-all' command to reload all *eww* buffers, it's
useful when using desktop-mode but when `eww-restore-desktop' is nil.
Sure.
Post by Pierre Neidhardt
- Make `eww' so that the default value is in the prompt, ready to be
edited by the user. Very convenient in my opinion.
No, that's against Emacs' convention for prompting. The default is
always stashed in `M-n'.
Post by Pierre Neidhardt
- Change `eww-open-in-new-buffer' so that it queries for a URL instead
of cloning the current buffer (which is not very useful in my
opinion).
That's not what it's supposed to do -- it opens the link under point in
a new buffer.
Post by Pierre Neidhardt
- Make eww-update-header-line-format also update the buffer name so that
it contains the page title. _Very useful_ to browse / search *eww*
buffers (think Ivy/Helm).
Sounds nice, but also sounds like something not everybody would want, so
it should have a configuration option to switch on/off (but can default
to on).
Post by Pierre Neidhardt
- Ask for tags when saving a bookmark. Tags are stored under the key
:tags as a list of strings.
Good idea.
Post by Pierre Neidhardt
- Make `eww-add-bookmark' run a customizable function to decide when.
to error out. For instance, error out when a duplicate is detected
with protocol stripped out (https://foo.bar is seen as a duplicate of
http://foo.bar).
Hm... Doesn't really sound all that useful? But having that command
give feedback (and perhaps query the user about what to do) in that
situation would be handy.
Post by Pierre Neidhardt
- Make `eww-write-bookmarks' run a customizable function before saving
the file. That function can be used, for instance, to detect
duplicates or to sort the bookmarks. This would make the eww-bookmarks
file more friendly to versioning.
Sure.
Post by Pierre Neidhardt
- Bookmarks can have a mark which is a string saved under the key :mark.
The mark should be unique. It could be used like the "quickmark"
function found in some browsers: use it to quickly load a
bookmark. (Work in progress.)
A mark in addition to a tag? Sounds like a bit more than most users
would want to invest in a bookmarking system, but I don't object.
Post by Pierre Neidhardt
- Bookmarks can have a search engine which is either appended to the
bookmark' URL if it does not start with "https?://", or used as-is
otherwise.
The search engine is stored as a string under the key :search.
A "%s" must be present in the search engine string as a place-holder
for the query.
Hm. Sounds more complicated than users will want to do to me.
Post by Pierre Neidhardt
- Make `eww-bookmark-prepare' only load bookmarks from file if not
already set. This makes it possible to display a custom / narrowed
list of bookmarks in the bookmark buffer.
I don't quite follow... What about just adding narrowing and sorting to
that mode?
Post by Pierre Neidhardt
- Make `eww-bookmark-prepare' display the mark, the tags and the search
engine, if available. Work in progress. I'm thinking of using a
different face for the mark if a search engine is present.
Sure.
Post by Pierre Neidhardt
- Add a `eww-bookmarks-by-tags' command which queries the user for a
completing list of tags and then displays a bookmark buffers of all
the bookmarks which match the tags. The matching can be either
inclusive or exclusive (bookmarks which match at least one tag vs. all
of them).
Sounds nice.
Post by Pierre Neidhardt
- With a multi-word query, if first word is a mark of a bookmark with a search engine,
then use the said search engine over the rest of the query.
- With a single word query, if first word is a mark then open the
corresponding bookmark.
- Else query the default search engine.
Sounds confusing. :-) But also quite DWIM, which I like.
Post by Pierre Neidhardt
- Fix `eww-forward-url' as it seems to corrupt the history. (Work in progress.)
Of course in its present state my hacks are what they are, very hacky.
It needs to be made more customizable and interfaceable.
What do you think?
Sounds great to me. :-) Make each thing into a patch (with
documentation) and let the apply-to-Emacs-fest commence. That is, if
you've been through the assign-copyright-to-the-FSF-process. I don't
see you in the copyright.list file, but that's apparently out of date
these days...
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
Pierre Neidhardt
2018-04-16 11:18:52 UTC
Permalink
Post by Lars Ingebrigtsen
Post by Pierre Neidhardt
- Enhances eww-next-url / eww-previous-url so that when there is no next /
previous links, it tries to increment / decrement the last number in
the last element of the URL. ("do what you mean" style.)
Hm... Are there many web sites where that is meaningful?
Emacs (and all GNU) mailing list archives for a start! :)
Many websites use increments in URL while they forget the previous/next
hint.
Post by Lars Ingebrigtsen
Post by Pierre Neidhardt
- Change `eww-open-in-new-buffer' so that it queries for a URL instead
of cloning the current buffer (which is not very useful in my
opinion).
That's not what it's supposed to do -- it opens the link under point in
a new buffer.
Sorry, I meant when there is no link under point.
Right now there is no way to just open a new eww buffer.
Post by Lars Ingebrigtsen
Post by Pierre Neidhardt
- Make eww-update-header-line-format also update the buffer name so that
it contains the page title. _Very useful_ to browse / search *eww*
buffers (think Ivy/Helm).
Sounds nice, but also sounds like something not everybody would want, so
it should have a configuration option to switch on/off (but can default
to on).
Sure.
Post by Lars Ingebrigtsen
Post by Pierre Neidhardt
- Make `eww-add-bookmark' run a customizable function to decide when.
to error out. For instance, error out when a duplicate is detected
with protocol stripped out (https://foo.bar is seen as a duplicate of
http://foo.bar).
Hm... Doesn't really sound all that useful? But having that command
give feedback (and perhaps query the user about what to do) in that
situation would be handy.
Why not useful? `eww-add-bookmark' already does duplicate detection,
except that it's very dumb: it won't realize if two bookmarks are the
same up to the protocol.

The command would obviously explain what's wrong when erroring out.
Post by Lars Ingebrigtsen
Post by Pierre Neidhardt
- Bookmarks can have a mark which is a string saved under the key :mark.
The mark should be unique. It could be used like the "quickmark"
function found in some browsers: use it to quickly load a
bookmark. (Work in progress.)
A mark in addition to a tag? Sounds like a bit more than most users
would want to invest in a bookmarking system, but I don't object.
Maybe I should have explained the overall bookmark logic beforehand :p

- "tags" are optional words used to categorize bookmarks, e.g. "cinema", "emacs",
"books". They can be used to filter & lookup bookmarks.

- "mark" serves two purposes: it allows to open a link with a simple
keybinding (optional) + it serves as a prefix for search engines (in
which case it's no longer optional). For example, say github has mark
"gh", then

M-x eww RET gh

opens github, so does `C-c e g h' if `C-c e' is quickmark prefix key.
If github comes with a search engine `:search "/search?q=%s", then

M-x eww RET gh foobar

opens a github search query of "foobar".

The point of centralizing search engines, quickmarks and bookmarks is
that it makes it easier to configure and, most importantly, it allows
for searching for all URLs at the same spot (name the bookmarks).
Post by Lars Ingebrigtsen
Post by Pierre Neidhardt
- Bookmarks can have a search engine which is either appended to the
bookmark' URL if it does not start with "https?://", or used as-is
otherwise.
The search engine is stored as a string under the key :search.
A "%s" must be present in the search engine string as a place-holder
for the query.
Hm. Sounds more complicated than users will want to do to me.
See my previous comment.

I think this is actually simpler that adding search engine support with
seperate functions and variables. With my suggestion we re-use the
current implementation without adding any new variable / function.
Post by Lars Ingebrigtsen
Post by Pierre Neidhardt
- Make `eww-bookmark-prepare' only load bookmarks from file if not
already set. This makes it possible to display a custom / narrowed
list of bookmarks in the bookmark buffer.
I don't quite follow... What about just adding narrowing and sorting to
that mode?
We could also do, but that does not compose as well as my suggestion.
For instance, how would you filter bookmarks by tags?
Narrowind and sorting would not be enough.
Post by Lars Ingebrigtsen
Post by Pierre Neidhardt
- With a multi-word query, if first word is a mark of a bookmark with a search engine,
then use the said search engine over the rest of the query.
- With a single word query, if first word is a mark then open the
corresponding bookmark.
- Else query the default search engine.
Sounds confusing. :-) But also quite DWIM, which I like.
The description is, but in practice it's crystal clear and always does
what you want.
Post by Lars Ingebrigtsen
Sounds great to me. :-) Make each thing into a patch (with
documentation) and let the apply-to-Emacs-fest commence. That is, if
you've been through the assign-copyright-to-the-FSF-process. I don't
see you in the copyright.list file, but that's apparently out of date
these days...
I am assigned to Emms, does it count?

--
Pierre Neidhardt
Lars Ingebrigtsen
2018-04-16 12:08:44 UTC
Permalink
Post by Pierre Neidhardt
Post by Lars Ingebrigtsen
Post by Pierre Neidhardt
- Enhances eww-next-url / eww-previous-url so that when there is no next /
previous links, it tries to increment / decrement the last number in
the last element of the URL. ("do what you mean" style.)
Hm... Are there many web sites where that is meaningful?
Emacs (and all GNU) mailing list archives for a start! :)
Many websites use increments in URL while they forget the previous/next
hint.
Hm... The numbers in the URLs often (usually) don't refer to the
next/previous article, but are a global thing that count upwards.

But perhaps it'll work out nice on many web sites. I'm not against
adding this and see how it works out. If it turns out to be less than
helpful, we can remove it again.
Post by Pierre Neidhardt
Post by Lars Ingebrigtsen
Post by Pierre Neidhardt
- Change `eww-open-in-new-buffer' so that it queries for a URL instead
of cloning the current buffer (which is not very useful in my
opinion).
That's not what it's supposed to do -- it opens the link under point in
a new buffer.
Sorry, I meant when there is no link under point.
Right now there is no way to just open a new eww buffer.
Oh, I see. Yes, that's a good idea.
Post by Pierre Neidhardt
Post by Lars Ingebrigtsen
Post by Pierre Neidhardt
- Make `eww-add-bookmark' run a customizable function to decide when.
to error out. For instance, error out when a duplicate is detected
with protocol stripped out (https://foo.bar is seen as a duplicate of
http://foo.bar).
Hm... Doesn't really sound all that useful? But having that command
give feedback (and perhaps query the user about what to do) in that
situation would be handy.
Why not useful?
Because the vast majority of users won't create such a function. It's
more useful (i.e., for more people) if things work the way it should be
default.
Post by Pierre Neidhardt
`eww-add-bookmark' already does duplicate detection, except that it's
very dumb: it won't realize if two bookmarks are the same up to the
protocol.
But then it should be smarter, I think. :-)
Post by Pierre Neidhardt
Post by Lars Ingebrigtsen
A mark in addition to a tag? Sounds like a bit more than most users
would want to invest in a bookmarking system, but I don't object.
Maybe I should have explained the overall bookmark logic beforehand :p
- "tags" are optional words used to categorize bookmarks, e.g. "cinema", "emacs",
"books". They can be used to filter & lookup bookmarks.
- "mark" serves two purposes: it allows to open a link with a simple
keybinding (optional) + it serves as a prefix for search engines (in
which case it's no longer optional). For example, say github has mark
"gh", then
M-x eww RET gh
opens github, so does `C-c e g h' if `C-c e' is quickmark prefix key.
If github comes with a search engine `:search "/search?q=%s", then
M-x eww RET gh foobar
opens a github search query of "foobar".
Ah, I see. Yes, that sounds very nice and useful.
Post by Pierre Neidhardt
Post by Lars Ingebrigtsen
Post by Pierre Neidhardt
- Make `eww-bookmark-prepare' only load bookmarks from file if not
already set. This makes it possible to display a custom / narrowed
list of bookmarks in the bookmark buffer.
I don't quite follow... What about just adding narrowing and sorting to
that mode?
We could also do, but that does not compose as well as my suggestion.
For instance, how would you filter bookmarks by tags?
Add a "narrow to tag" command to the mode?
Post by Pierre Neidhardt
Post by Lars Ingebrigtsen
Sounds great to me. :-) Make each thing into a patch (with
documentation) and let the apply-to-Emacs-fest commence. That is, if
you've been through the assign-copyright-to-the-FSF-process. I don't
see you in the copyright.list file, but that's apparently out of date
these days...
I am assigned to Emms, does it count?
Hm... If it's specific to Emms, I don't think it's enough? Anybody
know?
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
Pierre Neidhardt
2018-04-16 12:22:22 UTC
Permalink
Post by Lars Ingebrigtsen
Post by Pierre Neidhardt
Post by Lars Ingebrigtsen
Post by Pierre Neidhardt
- Make `eww-add-bookmark' run a customizable function to decide when.
to error out. For instance, error out when a duplicate is detected
with protocol stripped out (https://foo.bar is seen as a duplicate of
http://foo.bar).
Hm... Doesn't really sound all that useful? But having that command
give feedback (and perhaps query the user about what to do) in that
situation would be handy.
Why not useful?
Because the vast majority of users won't create such a function. It's
more useful (i.e., for more people) if things work the way it should be
default.
Post by Pierre Neidhardt
`eww-add-bookmark' already does duplicate detection, except that it's
very dumb: it won't realize if two bookmarks are the same up to the
protocol.
But then it should be smarter, I think. :-)
Sorry if I wasn't clear: I would not only make it customizable but also
set the default to a smart function, so that both `eww-write-bookmark'
and `eww-add-bookmark' exhibit a smarter behaviour. If that would not
be smart enough to the users, they could still customize it.
Post by Lars Ingebrigtsen
We could also do, but that does not compose as well as my suggestion.
Post by Pierre Neidhardt
For instance, how would you filter bookmarks by tags?
Add a "narrow to tag" command to the mode?
I have to think about it a little more.
Post by Lars Ingebrigtsen
Post by Pierre Neidhardt
Post by Lars Ingebrigtsen
Sounds great to me. :-) Make each thing into a patch (with
documentation) and let the apply-to-Emacs-fest commence. That is, if
you've been through the assign-copyright-to-the-FSF-process. I don't
see you in the copyright.list file, but that's apparently out of date
these days...
I am assigned to Emms, does it count?
Hm... If it's specific to Emms, I don't think it's enough? Anybody
know?
I can sign another assignment, no problem with that :)

--
Pierre Neidhardt

It will be advantageous to cross the great stream ... the Dragon is on
the wing in the Sky ... the Great Man rouses himself to his Work.
Lars Ingebrigtsen
2018-04-16 12:27:00 UTC
Permalink
Post by Pierre Neidhardt
Sorry if I wasn't clear: I would not only make it customizable but also
set the default to a smart function, so that both `eww-write-bookmark'
and `eww-add-bookmark' exhibit a smarter behaviour. If that would not
be smart enough to the users, they could still customize it.
Oh, OK. Sounds good to me.
Post by Pierre Neidhardt
I can sign another assignment, no problem with that :)
Great; I'll send you the form off-list.
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
Lars Ingebrigtsen
2018-04-16 12:32:11 UTC
Permalink
Post by Pierre Neidhardt
I can sign another assignment, no problem with that :)
Uhm, I can't seem to find the assignment template now... Can somebody
else send it to Pierre?
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
Eli Zaretskii
2018-04-16 17:55:27 UTC
Permalink
Date: Mon, 16 Apr 2018 14:32:11 +0200
Post by Pierre Neidhardt
I can sign another assignment, no problem with that :)
Uhm, I can't seem to find the assignment template now... Can somebody
else send it to Pierre?
Sent off-list.
Charles A. Roelli
2018-04-17 11:18:28 UTC
Permalink
Date: Mon, 16 Apr 2018 15:56:40 +0530
I've recently hacked around EWW and since it's coming into good shape
I'm thinking of committing this upstream.
https://github.com/Ambrevar/dotfiles/blob/master/.emacs.d/lisp/init-eww.el
- ...
BTW, you may also be interested in Bookmark+'s support for EWW, which
includes:

- changing EWW to use standard Emacs bookmarks instead of EWW's own
implementation (so you get the annotation and position-saving
features for free!),
- automatic renaming of EWW buffers based on their URL or page title
(not switched on by default),

- and more generically (for every bookmark type under the sun),
- tagging and searching bookmarks via these tags,
- and dynamically sorting the bookmark list.

The Bookmark+ code specifically for EWW is at:

https://www.emacswiki.org/emacs/bookmark%2b-1.el

I don't know how interested people here would be in integrating
Bookmark+ (or just its EWW support) into Emacs itself, but it may be
worth looking at.
Pierre Neidhardt
2018-04-17 19:18:48 UTC
Permalink
Post by Charles A. Roelli
BTW, you may also be interested in Bookmark+'s support for EWW, which
- changing EWW to use standard Emacs bookmarks instead of EWW's own
implementation (so you get the annotation and position-saving
features for free!),
- automatic renaming of EWW buffers based on their URL or page title
(not switched on by default),
- and more generically (for every bookmark type under the sun),
- tagging and searching bookmarks via these tags,
- and dynamically sorting the bookmark list.
https://www.emacswiki.org/emacs/bookmark%2b-1.el
I don't know how interested people here would be in integrating
Bookmark+ (or just its EWW support) into Emacs itself, but it may be
worth looking at.
Very good point, thanks for bringing it up! I'll see if it's possible
to embed the mark and the search engine then.
--
Pierre Neidhardt
Pierre Neidhardt
2018-04-24 05:56:58 UTC
Permalink
Another feature I'd like to add: a global "history" file.
It would record all visited URI.

It's convenient to search URLs that we know we've visited, but not
necessarily only from a specific buffer.

To make the history persistent, I would store it in a format similar to
bookmarks but without any of the additional features I'm planning to add
(tags, search engine, etc.).
There would be the following keys:
- URL
- TITLE
- TIME

Defcustoms would include:

- The persistent file.
- The max number of entries to keep (0 for unlimited). Older entries
get deleted first.

--
Pierre Neidhardt
Lars Ingebrigtsen
2018-04-24 14:50:18 UTC
Permalink
Post by Pierre Neidhardt
Another feature I'd like to add: a global "history" file.
It would record all visited URI.
Yes, that would be very nice. And then ecomplete completion over the
results, like most other browsers have.

There's a privacy issue here, though, so it should be possible to switch
it off. And perhaps eww should have a porn, I mean anonymous, mode
where nothing is recorded: Neither cookies nor history.
Post by Pierre Neidhardt
To make the history persistent, I would store it in a format similar to
bookmarks but without any of the additional features I'm planning to add
(tags, search engine, etc.).
- URL
- TITLE
- TIME
Also the number of times it's been visited so that ecomplete can rank
the things you're most interested in first.
Post by Pierre Neidhardt
- The persistent file.
- The max number of entries to keep (0 for unlimited). Older entries
get deleted first.
These can be stored directly in ecomplete, so that's not necessary.
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
Stefan Monnier
2018-04-24 17:57:15 UTC
Permalink
Post by Lars Ingebrigtsen
There's a privacy issue here, though, so it should be possible to switch
it off. And perhaps eww should have a porn, I mean anonymous, mode
BTW, I think using "porn" as a humorous justification for secrecy is
detrimental since it promotes the view that secrecy is only used for
objectionable activities, along the lines of "I don't need secrecy
because I have nothing to hide".

The recent Cambridge Analytica blip (it seems to have already
disappeared from the media, sadly) is a prime example of why
"no-secrecy" ultimately means "no democracy" or "no power".


Stefan "who does realize that EWW's history probably won't be
shared on Facebook, so maybe he didn't choose the best
moment to bring up this general issue"
John Wiegley
2018-04-24 19:45:48 UTC
Permalink
SM> BTW, I think using "porn" as a humorous justification for secrecy is
SM> detrimental since it promotes the view that secrecy is only used for
SM> objectionable activities, along the lines of "I don't need secrecy because
SM> I have nothing to hide".

Indeed. As someone whose wife's friends in Iran are actually being imprisoned
on religious groups, the ability to promote education and other activities
deemed "subversive" by the current regime depends entirely on one's ability to
maintain private communications. This is what I think about when these issues
come up.
--
John Wiegley GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com 60E1 46C4 BD1A 7AC1 4BA2
Joost Kremers
2018-04-24 20:16:18 UTC
Permalink
Post by Stefan Monnier
Post by Lars Ingebrigtsen
There's a privacy issue here, though, so it should be possible to switch
it off. And perhaps eww should have a porn, I mean anonymous, mode
BTW, I think using "porn" as a humorous justification for
secrecy is
detrimental since it promotes the view that secrecy is only used for
objectionable activities, along the lines of "I don't need
secrecy
because I have nothing to hide".
Note that these "anonymous" modes that modern browser implement
really only serve to hide one's browsing history from family
members and flat/room/office mates that you share your computer
with. It has nothing to do with secrecy or anonymity on the
internet. So porn-mode really isn't that much of a misnomer.
--
Joost Kremers
Life has its moments
Stefan Monnier
2018-04-24 20:53:52 UTC
Permalink
Note that these "anonymous" modes that modern browser implement really only
serve to hide one's browsing history from family members and
flat/room/office mates that you share your computer with.
Until your computer is subpoena'd, stolen, hacked into, accidentally
mishandled, ...

Whether you personally feel like these are serious threats to you right
now is not the point. Information can easily be preserved for years, so
even those same pieces of data about which you don't care right now might
turn into very serious matters 10 years from now because your situation
or the world around you has changed.

To be honest, I feel like using exclusively Free Software, shying away
from cell phones and centralized systems like Facebook/Whatsapp/..., and
preferring good'ol cash makes me look pretty close to the prototypical
sophisticated criminal living "off-the-grid" in your average policial TV
series. The only saving grace is that I constantly post to public
mailing-lists like this one ;-)


Stefan
Lars Ingebrigtsen
2018-04-24 22:40:07 UTC
Permalink
Note that these "anonymous" modes that modern browser implement really
only serve to hide one's browsing history from family members and
flat/room/office mates that you share your computer with. It has
nothing to do with secrecy or anonymity on the internet. So porn-mode
really isn't that much of a misnomer.
Indeed. I had no idea that my lame joke would get Stefan's serious
reaction, but in this reality, all we can expect from "anonymous
browsing" is that the next person who opens our web browser won't be
confronted by our browsing history. That's not nothing -- it's a
valuable thing -- but we can't in any serious way present that mode as
being "anonymous".

We're still being tracked, and people at risk have to use completely
different methods to remain safe. I mean, safer.
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
Stefan Monnier
2018-04-24 22:44:17 UTC
Permalink
Post by Lars Ingebrigtsen
Indeed. I had no idea that my lame joke would get Stefan's serious
reaction, but in this reality, all we can expect from "anonymous
browsing" is that the next person who opens our web browser won't be
confronted by our browsing history. That's not nothing -- it's a
valuable thing -- but we can't in any serious way present that mode as
being "anonymous".
As mentioned in my signature, my response was not directed specifically
to the issue of the browsing history, but the use of "porn" as the
prototypical example of a need for secrecy. It trivializes secrecy and
makes it sound like something objectionable.


Stefan
Drew Adams
2018-04-24 16:24:33 UTC
Permalink
Post by Pierre Neidhardt
Another feature I'd like to add: a global "history" file.
It would record all visited URI.
It's convenient to search URLs that we know we've visited, but not
necessarily only from a specific buffer.
To make the history persistent, I would store it in a format similar to
bookmarks but without any of the additional features I'm planning to add
- URL
- TITLE
- TIME
- The persistent file.
- The max number of entries to keep (0 for unlimited). Older entries
get deleted first.
FWIW -

Charles mentioned earlier some of the Bookmark+ support for
EWW (normal Emacs bookmarks vs EWW's pseudo-bookmarks, plus
arbitrary tagging, sorting, filtering, multiple bookmark
files, org-mode annotations...).

It seems a bit silly for EWW to have its own, limited form
of "bookmarking". Perhaps it's time for it to graduate to
Emacs bookmarks?

Another Bookmark+ feature for EWW, relevant to your mail,
is minor mode `bmkp-eww-auto-bookmark-mode'. When it's
enabled a bookmark is automatically set whenever you visit
a URL with EWW. (A similar feature exists for Info nodes.)

When enabled, if user option `bmkp-eww-auto-type' is
`create-or-update' then such a bookmark is created for the
URL if none exists. If the option value is `update-only'
(the default) then no new bookmark is created automatically,
but an existing bookmark gets updated. You can toggle the
option value with command `bmkp-toggle-eww-auto-type'.

Emacs (real) bookmarks record not only the time of last
visit but also the number of visits. You can sort by
the number of visits, to easily see your weighted URL
visit history. (You can also edit the number of recorded
visits.)

In addition, EWW bookmarks with Bookmark+ have their own
visit history. You can cycle among just the EWW bookmarks
or a subset of them (e.g., those currently visible in the
bookmark-list display, in sort order and filtered). So
you can cycle among any set of EWW bookmarks, visiting
them in an order you choose. You need no programming to
do any of this - no knowledge of Lisp.

You can also bookmark a given bookmark-list display,
which persists its visibility/filtering settings and sort
order. So you can quickly switch among any number of
sequences (navigation lists) of EWW bookmarks to cycle
among.

Bookmark+ navigation lists give you, in this sense, any
number of EWW histories, easy to define and switch among.
And you can tag those different histories any way you
like, and switch among them by tag or tag combinations,
in addition to doing so by bookmark name.

AFAICT, everything you've mentioned so far as possible
improvements of EWW pseudo-bookmarking is already
available with Bookmark+ real bookmarks. Why reinvent
the wheel and remain incompatible with Emacs bookmarks?

Bookmark+ is in fact a superset of vanilla `bookmark.el'.
It could just replace it in Emacs.

https://www.emacswiki.org/emacs/BookmarkPlus#CyclingNavlist
Pierre Neidhardt
2018-04-25 07:00:14 UTC
Permalink
Post by Drew Adams
It seems a bit silly for EWW to have its own, limited form
of "bookmarking". Perhaps it's time for it to graduate to
Emacs bookmarks?
You mean bookmark+, right?

As far as I understand, Emacs native bookmarks won't work with EWW.
`M-x bookmark-set' complains the the EWW buffer does not point to a real file.

AFAICT, everything you've mentioned so far as possible
Post by Drew Adams
improvements of EWW pseudo-bookmarking is already
available with Bookmark+ real bookmarks. Why reinvent
the wheel and remain incompatible with Emacs bookmarks?
Bookmark+ is in fact a superset of vanilla `bookmark.el'.
It could just replace it in Emacs.
https://www.emacswiki.org/emacs/BookmarkPlus#CyclingNavlist
Thanks for all those good points.

For clarification, I never said I was opposed to bookmark+, I agree
merging it (or parts of it) would probably be the best path to follow.
I need to work on it.

That said, a few issues I can see at first glance:

- The package is published on Emacs Wiki. There does not seem to be any
version control.

- It's under GPL license, but I wonder if the lack of version control
could hinder the copyright assignment process. From the comments,
Drew Adams and Thierry Volpiatto seem to be the only two authors.
Drew, can you confirm?

- The package is _huge_! About 30k+ lines. It might require a lot of
work to review it and to integrate the needed parts into Emacs.

What do you think?

--
Pierre Neidhardt
Lars Ingebrigtsen
2018-04-25 12:30:45 UTC
Permalink
Post by Pierre Neidhardt
For clarification, I never said I was opposed to bookmark+, I agree
merging it (or parts of it) would probably be the best path to follow.
I need to work on it.
The eww history seems like a perfect fit for ecomplete, so I think you
should use that instead. :-)
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
Drew Adams
2018-04-25 15:29:46 UTC
Permalink
Post by Pierre Neidhardt
Post by Drew Adams
It seems a bit silly for EWW to have its own, limited form
of "bookmarking". Perhaps it's time for it to graduate to
Emacs bookmarks?
You mean bookmark+, right?
No, there I meant Emacs normal bookmarks vs the
pseudo-bookmarks that EWW currently uses.

Bookmark+ uses normal Emacs bookmarks. It does
provide additional features, some of which have
been mentioned (EWW support, tags, etc.).

But the point I was making there was that it seems
silly for EWW not to be using normal Emacs bookmarks.

The only reasons I can guess, so far, for why that
was done are (1) possibly the EWW author was not
familiar with Emacs bookmarks (but I doubt it) or
(2) there was a wish to let you keep EWW bookmarks
separate from other bookmarks.

#2 is not an issue at all with Bookmark+. It lets
you have all kinds of bookmarks and yet keep them
as separate as you like. It even has specific
"jumping" commands for different types of bookmarks,
e.g., complete only against names of bookmarks of
a specific type. There is no problem of mixing or
not mixing EWW bookmarks with other bookmarks.

But perhaps the EWW author can defend his choice
not to use normal Emacs bookmarks for EWW? Why go
against Occam's razor this way? What's the advantage?
Post by Pierre Neidhardt
As far as I understand, Emacs native bookmarks won't work with EWW.
`M-x bookmark-set' complains the the EWW buffer does not point to a real file.
No, normal Emacs bookmarks do not need a file as
target. Even vanilla Emacs can bookmark Dired and
Info nodes (albeit not so well). A bookmark handler
can do anything you want it to do.

The point is that vanilla Emacs, out of the box,
provides a healthy framework for bookmarking. That
includes the ability for anyone to define new kinds
of bookmarks.

Bookmark+ has defined new kinds of bookmarks, but
the framework (enhanced somewhat by Bookmark+) is
what Emacs provides.

My argument, first and foremost, is for EWW to use
Emacs bookmarks, not some ersatz, alternative
bookmarking system cooked up just for EWW.

Secondarily, I point out that Bookmark+ already
provides support for EWW using ordinary Emacs
bookmarks. What it does could be incorporated as
is, or Emacs could look to its code for inspiration.
The code is available for GNU Emacs/FSF, if it wants.
Post by Pierre Neidhardt
Thanks for all those good points.
For clarification, I never said I was opposed to bookmark+, I agree
merging it (or parts of it) would probably be the best path to follow.
I need to work on it.
Sounds good to me. Let me know how I might help.
Post by Pierre Neidhardt
- The package is published on Emacs Wiki. There does not seem to be any
version control.
It could be included in Emacs itself, maintained by
Emacs. For that, instead of requiring `bookmark.el',
its code that is needed would just be included in the
Bookmark+ files. Bookmark+ subsumes `bookmark.el'.

I have no special need to maintain it. I will do so,
if it's not included in Emacs. And I might do so even
if it is included. I still have improvements I want
to make to it, when I get time. But a priori I have
no objection to Emacs taking it over.

As for the doc: The doc I have for Bookmark+ is in the
form of comments - Commentary in file `bookmark+-doc.el'.
You can see the doc as readable plain-text in Finder mode
by using `M-x finder-commentary bookmark+-doc.el'.

The same doc is also in the form of HTML on Emacs Wiki
(maintained there by hand). If the doc is to be used
by Emacs it would presumably need to be converted to
texinfo/Info. Or it could just be dropped, providing
only the doc strings.

I would not want to do the conversion to texinfo/Info.
I have no special expertise in that. But I'd be glad
to help with any content questions or wording, if
needed.
Post by Pierre Neidhardt
- It's under GPL license, but I wonder if the lack of version control
could hinder the copyright assignment process. From the comments,
Drew Adams and Thierry Volpiatto seem to be the only two authors.
Drew, can you confirm?
I am the author, and have been from the outset, at
least as far back as 2004. Thierry worked only on
a couple features for a few months in 2009. Both
of us have signed FSF papers and have contributed
previously to GNU Emacs, AFAIK. There's no problem
with our contributing.
Post by Pierre Neidhardt
- The package is _huge_! About 30k+ lines. It might require a lot of
work to review it and to integrate the needed parts into Emacs.
What do you think?
23,114 lines, much of which is Commentary.
16,876 lines without comment-only or blank lines.
Much of the code is there to support multiple Emacs
versions. The byte-compiled code is 950KB.

It's not difficult to integrate. All that's needed is:

1. Integrate the little bit of `bookmark.el' code that
is currently needed by `require'. I've always tried
to supplement, replacing something only when I needed
to change its code. At this point, there is little
remaining of `bookmark.el' that is needed.

2. Remove support for older Emacs versions, if desired.
I try to support as many older versions as possible,
but Emacs doesn't do that. (Bookmark+ currently
works with Emacs releases back through Emacs 20.)

3. Remove optional support for other libraries when
they are available in `load-path': soft-required
e.g. (require 'thingatpt+ nil t) or via `fboundp'
or `boundp'.
Pierre Neidhardt
2018-05-03 06:52:05 UTC
Permalink
What do Emacs maintainers think about this?

Should we merge Bookmark+ (or parts of it) into Emacs to improve the
bookmark functionalities and extend its field of application to programs
like EWW?

--
Pierre Neidhardt
Pierre Neidhardt
2018-05-03 06:54:43 UTC
Permalink
Post by Drew Adams
Post by Pierre Neidhardt
As far as I understand, Emacs native bookmarks won't work with EWW.
`M-x bookmark-set' complains the the EWW buffer does not point to a real file.
No, normal Emacs bookmarks do not need a file as
target. Even vanilla Emacs can bookmark Dired and
Info nodes (albeit not so well). A bookmark handler
can do anything you want it to do.
Can you explain how you bookmark an EWW buffer with Emacs vanilla
bookmarks?

Besides it does not seem to effectively work with many other modes. For
instance, bookmarking a *Help* buffer works at first glance, but Emacs
fails to look read the bookmarks file upon restart.
--
Pierre Neidhardt

Captain Penny's Law:
You can fool all of the people some of the time, and
some of the people all of the time, but you Can't Fool Mom.
Drew Adams
2018-05-03 15:12:08 UTC
Permalink
Post by Pierre Neidhardt
Post by Drew Adams
Post by Pierre Neidhardt
As far as I understand, Emacs native bookmarks won't work with EWW.
`M-x bookmark-set' complains the the EWW buffer does not point to a real file.
No, normal Emacs bookmarks do not need a file as
target. Even vanilla Emacs can bookmark Dired and
Info nodes (albeit not so well). A bookmark handler
can do anything you want it to do.
Can you explain how you bookmark an EWW buffer with Emacs
vanilla bookmarks?
A new type of bookmark needs to be defined, with a handler
that "jumps" to the URL. Code from Bookmark+ or similar
could be used.

My point was that just to add EWW support for (real)
Emacs bookmarks it's not required to add all of Bookmark+
to Emacs. All that's needed are to define a format for
the bookmarks and define a hander function to jump to
such a bookmark.

In Bookmark+, the handler function is `bmkp-jump-eww',
and a bookmark with EWW format is made by function
`bmkp-make-eww-record'.

(defun bmkp-make-eww-record ()
"Make a record for an EWW buffer."
`(,(bmkp-eww-title)
(buffer-name . ,(bmkp-eww-new-buffer-name))
,@(bookmark-make-record-default 'NO-FILE)
(location . ,(bmkp-eww-url))
(handler . bmkp-jump-eww)))

The logic behind jumping to an EWW bookmark, and how
that is hooked into EWW is more complicated - see the
Bookmark+ code for how Bookmark+ does it.
Post by Pierre Neidhardt
Besides it does not seem to effectively work with many other modes. For
instance, bookmarking a *Help* buffer works at first glance, but Emacs
fails to look read the bookmarks file upon restart.
Dunno what that means. But yes, vanilla Emacs needs
to have some support for bookmarking arbitrary buffers,
as does Bookmark+, or else it needs to at least have
support for bookmarking some specific kinds of
bookmarks (*Info*, *Help* or whatever).

But I believe that vanilla Emacs already does have
support for bookmarking some kinds of buffers, such
as Info and Dired. I don't recall really, but I was
thinking it does when I wrote that. Emacs added
bookmark field `location', for example, and its value
need not be a file name.

In any case, the infrastructure is there in vanilla
Emacs to do any of this.

My point was:

1. Bookmark+ is freely available. It can be
integrated into Emacs; and it already provides
EWW support.

2. It's not _necessary_ to integrate Bookmark+ to add
EWW support to Emacs. That could be done either
by just copying the Bookmark+ EWW-support code to
Emacs or by Emacs rolling its own support.

3. In reply to your statement that vanilla Emacs
requires that a bookmark target a file I replied
that I don't think that's true.

Grepping the Emacs source code for "'location"
shows only one use of that property, in
`gnus-summary-bookmark-jump', and I might be
reading that wrong, so I could be wrong about this.

There are other ways than property `location' to do
this, so grepping `location" (no preceding quote')
might tell more.

And I see that an Info bookmark records both the
Info file and the node (so it does record a file
name).

Grepping for `bookmark-default-handler' can also
tell us whether some bookmark types do not record
a file name. Maybe `Man-bookmark-jump'?
`doc-view-bookmark-jump'?

T.V Raman
2018-04-24 15:58:05 UTC
Permalink
The url library already does this?
--
Pierre Neidhardt
2018-04-25 06:48:26 UTC
Permalink
Post by T.V Raman
The url library already does this?
I did not know about this library. I had a quick look and my first setup

(setq url-history-track t
url-history-file (expand-file-name "url/history" user-emacs-directory))

does not seem to be enough as it does not save anything with EWW. I'm not
sure how to link the two libraries.

Actually, which part of Emacs uses url-history?
--
Pierre Neidhardt
T.V Raman
2018-04-25 15:25:42 UTC
Permalink
The URL library was originally part of the old w3 browser; it was
refactored out as a separate url library about 15 years ago from
memory. I thought parts of EWW used the underlying url library --guess
it's not enabling the history bits.
--
Pierre Neidhardt
2018-04-25 16:25:02 UTC
Permalink
Post by T.V Raman
The URL library was originally part of the old w3 browser; it was
refactored out as a separate url library about 15 years ago from
memory. I thought parts of EWW used the underlying url library --guess
it's not enabling the history bits.
Do you have any example usage?
--
Pierre Neidhardt

Procrastination means never having to say you're sorry.
T.V Raman
2018-04-25 23:36:30 UTC
Permalink
Look in the w3 browser codebase,
--
Stefan Monnier
2018-04-26 12:46:24 UTC
Permalink
Post by T.V Raman
Look in the w3 browser codebase,
Which is conveniently available from GNU ELPA, of course,


Stefan
Eli Zaretskii
2018-04-24 16:49:28 UTC
Permalink
Date: Tue, 24 Apr 2018 11:26:58 +0530
To make the history persistent, I would store it in a format similar to
bookmarks but without any of the additional features I'm planning to add
(tags, search engine, etc.).
We already have features that save and restore various history lists
(savehist and desktop, to name just 2). Can this new history just be
added to those handled by these packages?
Pierre Neidhardt
2018-04-25 06:32:39 UTC
Permalink
Post by Eli Zaretskii
Post by Pierre Neidhardt
To make the history persistent, I would store it in a format similar to
bookmarks but without any of the additional features I'm planning to add
(tags, search engine, etc.).
We already have features that save and restore various history lists
(savehist and desktop, to name just 2). Can this new history just be
added to those handled by these packages?
I did not think about desktop-mode and you are right, I think it's a
right fit for a persistant history. Unless bookmark+ can do better (see
the rest of this thread).

Regarding savehist, I'm not sure how it could be used here since it
seems to be taylored for the minibuffer. EWW can visit URLs without
going through the minibuffer. Did I misunderstand you here?
--
Pierre Neidhardt
Eli Zaretskii
2018-04-25 15:55:53 UTC
Permalink
Date: Wed, 25 Apr 2018 12:02:39 +0530
Regarding savehist, I'm not sure how it could be used here since it
seems to be taylored for the minibuffer. EWW can visit URLs without
going through the minibuffer. Did I misunderstand you here?
savehist.el provides a defcustom, savehist-additional-variables, which
can be used to save and restore additional history variables. The
minibuffer history is just the default.
Pierre Neidhardt
2018-04-25 16:13:28 UTC
Permalink
Post by Eli Zaretskii
savehist.el provides a defcustom, savehist-additional-variables, which
can be used to save and restore additional history variables. The
minibuffer history is just the default.
OK, can we save additional information such as the page title and
the last-access date? If not, that would be too limiting I think.
--
Pierre Neidhardt
Eli Zaretskii
2018-04-25 16:19:26 UTC
Permalink
Date: Wed, 25 Apr 2018 21:43:28 +0530
Post by Eli Zaretskii
savehist.el provides a defcustom, savehist-additional-variables, which
can be used to save and restore additional history variables. The
minibuffer history is just the default.
OK, can we save additional information such as the page title and
the last-access date? If not, that would be too limiting I think.
It's fine with me if you decide savehist is not up to this job. My
point was that the candidate existing packages should be explored
before you decide that you need to roll your own. I'd be surprised to
learn that none of the existing capabilities fits your bill.

Thanks.
Lars Ingebrigtsen
2018-04-17 19:01:57 UTC
Permalink
Post by Pierre Neidhardt
- Enhances eww-next-url / eww-previous-url so that when there is no next /
previous links, it tries to increment / decrement the last number in
the last element of the URL. ("do what you mean" style.)
Apropos of this, have you by any chance read this news item?

http://www.cbc.ca/news/canada/nova-scotia/freedom-of-information-request-privacy-breach-teen-speaks-out-1.4621970

This teenager was put in jail for decrementing the number in an URL
and downloading the contents. Will this functionality open up all eww
users for prosecution? :-)
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
Loading...