kmail usability with IMAP

[Rob Kaper]

What's the point of downloading a 2MB e-mail every time you need to read
it? That's crap, to be honest. IMAP has good things, but also quite bad
things if the client is not ready enough.

I agree. IMAP was not designed to leave the e-mail data on servers
indefinitely, just the authorative header data. Obviously clients need to
fetch the data anyway if they want to access it. But IMAP does offer the
opportunity for clients to examine the headers before downloading all data.
so it only needs to be updated or retrieved when not in sync.

But what does KMail do? It totally ignores that feature and insists on
downloading every single bit of e-mail even when this is not necessary.

Nope, I'm switching to mozilla definetely.

Will you consider switching back when we show more sanity?

It makes no sense to cache the IMAP header information client-side (which
KMail does) while not being able to cache the associated data. Sure, it is
nice that you can store it on the server and retrieve it indefinitely, but
that does not at all imply you do not want the data stored locally as well.

Since we already provide a mean to store the e-mail headers between instances,
it should not be too hard to store the e-mail bodies as well and add a check
only to retrieve them when necessary.

KMail developers: please reconsider this or show me the relevant parts of the
code so I can work on this myself.

Rob

[Zack Rusin]

Post by Rob Kaper
KMail developers: please reconsider this or show me the relevant
parts of the code so I can work on this myself.

I like this idea, if Michael and Carsten won't object I'll put it on my
TODO list for 3.2 and promise to have it done by then.

Zack

[Unai Garro]

Post by Rob Kaper
Will you consider switching back when we show more sanity?

For the moment I keep receiving the e-mail using fetchmail and kmail. I gave
mozilla a try, but it looks like quite slow compared to kmail. :)

Post by Rob Kaper
Since we already provide a mean to store the e-mail headers between
instances, it should not be too hard to store the e-mail bodies as well and
add a check only to retrieve them when necessary.
KMail developers: please reconsider this or show me the relevant parts of
the code so I can work on this myself.

If I knew enough about kde/qt programming I'd like to help, but I've only made
small pieces of work so far under kde, so I don't trust myself doing a (very)
reliable code....

[Michael Häckel]

Post by Rob Kaper
I agree. IMAP was not designed to leave the e-mail data on servers
indefinitely, just the authorative header data. Obviously clients need to

Do I understand correctely, that you want to have to bodies of the message
deleted on the server and just the headers to stay there???

Post by Rob Kaper
fetch the data anyway if they want to access it. But IMAP does offer the
opportunity for clients to examine the headers before downloading all data.
so it only needs to be updated or retrieved when not in sync.

And KMail makes use of that feature.

Post by Rob Kaper
It makes no sense to cache the IMAP header information client-side (which

It does make sense, because it makes opening folders _much_ faster.

Post by Rob Kaper
KMail does) while not being able to cache the associated data. Sure, it is
nice that you can store it on the server and retrieve it indefinitely, but
that does not at all imply you do not want the data stored locally as well.

Of course it is also possible to cache the message bodies locally, but be
warned that caching and downloading a two completely different things.

Post by Rob Kaper
Since we already provide a mean to store the e-mail headers between
instances, it should not be too hard to store the e-mail bodies as well and
add a check only to retrieve them when necessary.
KMail developers: please reconsider this or show me the relevant parts of
the code so I can work on this myself.

Feel free to work on that feature for KDE 3.2.

Regards,
Michael Häckel

[Rob Kaper]

Post by Michael Häckel
Do I understand correctely, that you want to have to bodies of the message
deleted on the server and just the headers to stay there???

No! :-)

Post by Michael Häckel

Post by Rob Kaper
It makes no sense to cache the IMAP header information client-side (which

It does make sense, because it makes opening folders _much_ faster.

The ability to open e-mails _much_ faster would be nice too.

Post by Michael Häckel
Of course it is also possible to cache the message bodies locally, but be
warned that caching and downloading a two completely different things.

Yes, we already do the downloading and not the caching, my point exactly. ;-)

Post by Michael Häckel

Post by Rob Kaper
KMail developers: please reconsider this or show me the relevant parts of
the code so I can work on this myself.

Feel free to work on that feature for KDE 3.2.

Will do.

Thanks,

Rob

[Michael Häckel]

Post by Rob Kaper
The ability to open e-mails _much_ faster would be nice too.

That works of course only the second time. I read most of my mails only once
and wonder how much that would help though.

The headers are shown every time a folder is opened. Also for headers the
speed is on a fast network unlike for the bodies mainly a CPU power issue
(server and client side) not a network traffic issue.

Post by Rob Kaper

Post by Michael Häckel
Of course it is also possible to cache the message bodies locally, but be
warned that caching and downloading a two completely different things.

Yes, we already do the downloading and not the caching, my point exactly. ;-)

Under downloading I understand fetching the mails from the server, storing
them locally and deleting them on the server after they are stored locally.
This is what Unai Garro appears to want.

Caching means keeping a local copy of all information on the server. If a
message disappears on the server also the local copy is deleted. This appears
to be what you want.

Post by Rob Kaper
Will do.

I hope to see from you more than just words as from most other people that
critizised IMAP support in KMail so far and proved to have no idea of the
matter.

Regards,
Michael Häckel

[Carsten Burghardt]

Post by Michael Häckel

Post by Rob Kaper
The ability to open e-mails _much_ faster would be nice too.

That works of course only the second time. I read most of my mails only
once and wonder how much that would help though.

That depends if you use your kmail at work or at home.
At home I agree and the caching won't be worth the trouble. At work I've a lot
of emails that I need to open more than once (that's what imap is for, to
archive them). But even then I don't mind if the email needs to be transfered
because the network speed is high enough. The only problem with that are the
attachments but we need a separate solution for that.
The only really valid point for caching IMO is an imap-offline-mode.

Post by Michael Häckel
The headers are shown every time a folder is opened. Also for headers the
speed is on a fast network unlike for the bodies mainly a CPU power issue
(server and client side) not a network traffic issue.

Post by Rob Kaper

Post by Michael Häckel
Of course it is also possible to cache the message bodies locally, but
be warned that caching and downloading a two completely different
things.

Yes, we already do the downloading and not the caching, my point exactly. ;-)

Under downloading I understand fetching the mails from the server, storing
them locally and deleting them on the server after they are stored locally.
This is what Unai Garro appears to want.

The he can use POP.

Post by Michael Häckel
Caching means keeping a local copy of all information on the server. If a
message disappears on the server also the local copy is deleted. This
appears to be what you want.

Post by Rob Kaper
Will do.

I hope to see from you more than just words as from most other people that
critizised IMAP support in KMail so far and proved to have no idea of the
matter.
Regards,
Michael Häckel
_______________________________________________
KMail Developers mailing list
http://mail.kde.org/mailman/listinfo/kmail

[Michael Häckel]

Post by Carsten Burghardt

Post by Michael Häckel
Under downloading I understand fetching the mails from the server,
storing them locally and deleting them on the server after they are
stored locally. This is what Unai Garro appears to want.

The he can use POP.

He can't, because his server doesn't support it because his admin believes
that POP3 is unlike IMAP/SSL a security risk :-)
This is what the whole discussion that started on kde-devel is about.

Regards,
Michael Häckel

[Ingo Klöcker]

Post by Rob Kaper
Since we already provide a mean to store the e-mail headers between
instances, it should not be too hard to store the e-mail bodies as
well and add a check only to retrieve them when necessary.
KMail developers: please reconsider this or show me the relevant
parts of the code so I can work on this myself.

JFYI, if Michael wouldn't have more important things to do (like working
on his diploma) I'm sure he would have implemented local e-mail body
caching in the meantime. I know that it was on his TODO list.

You should really show some gratitude for what Michael has already
accomplished so far. Until KDE 3.0 he basically implemented the whole
IMAP support completely without the help of anybody else. And those
IMAP features which are already supported are faster and better than in
most other email clients (including Mozilla and OE).

Regards,
Ingo

[Don Sanders]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Post by Rob Kaper
Since we already provide a mean to store the e-mail headers
between instances, it should not be too hard to store the e-mail
bodies as well and add a check only to retrieve them when
necessary.
KMail developers: please reconsider this or show me the relevant
parts of the code so I can work on this myself.

JFYI, if Michael wouldn't have more important things to do (like
working on his diploma) I'm sure he would have implemented local
e-mail body caching in the meantime. I know that it was on his TODO
list.
You should really show some gratitude for what Michael has already
accomplished so far. Until KDE 3.0 he basically implemented the
whole IMAP support completely without the help of anybody else. And
those IMAP features which are already supported are faster and
better than in most other email clients (including Mozilla and OE).

I agree with Ingo's sentiments.

I find Rob's mail in poor taste and before denouncing us for being
against the idea he should have considered the possibility that the
developers simply haven't had time to implement every requested
feature yet.

Don.

[Rob Kaper]

Post by Don Sanders
I find Rob's mail in poor taste and before denouncing us for being
against the idea he should have considered the possibility that the
developers simply haven't had time to implement every requested
feature yet.

Well, there is no development roadmap for KMail on kmail.kde.org nor did I
find a TODO or similar in CVS, which is likely to cause these
misunderstandings.

The replies to the feature request were mostly "you don't need that" and one
even said something along the lines of "but if you insist, use a different
e-mail client" and I concluded that this was definitely not a consideration
then.

My intent to work on it myself clearly indicates that I *did* consider the
possibility that there will not be a complete rejection of the feature and
that a lack of time and other priorities are indeed the only showstoppers.

Rob

[Michael Häckel]

Post by Rob Kaper
Well, there is no development roadmap for KMail on kmail.kde.org nor did I
find a TODO or similar in CVS, which is likely to cause these
misunderstandings.

I'm against announcing development plans as I don't like people to complain
that the feature I promised to work on is still not there.

Post by Rob Kaper
The replies to the feature request were mostly "you don't need that" and

Well the start of the thread claimed that KMail is "half-usable" similar
statements followed which partly are not true. Therefore I tried to explain
why things are as they are for good reasons.

Post by Rob Kaper
one even said something along the lines of "but if you insist, use a
different e-mail client" and I concluded that this was definitely not a
consideration then.

fetchmail is not a mail client if you talk about that suggestion.

Post by Rob Kaper
My intent to work on it myself clearly indicates that I *did* consider the
possibility that there will not be a complete rejection of the feature and
that a lack of time and other priorities are indeed the only showstoppers.

I still don't understand what exactely you want to implement, sorry.

Regards,
Michael Häckel

[Don Sanders]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Post by Rob Kaper
Well, there is no development roadmap for KMail on kmail.kde.org
nor did I find a TODO or similar in CVS, which is likely to cause
these misunderstandings.

I'm against announcing development plans as I don't like people to
complain that the feature I promised to work on is still not there.

I agree with Michael. I think it's much better if people talk about
what they are working on this list.

That way you don't end up in a situation where someone plans to
implement a feature, adds it to the development web page, gives up,
never deletes their plans from the webpage and thus prevents other
people from implementing that the feature.

Post by Rob Kaper
The replies to the feature request were mostly "you don't need that" and

Well the start of the thread claimed that KMail is "half-usable"
similar statements followed which partly are not true. Therefore I
tried to explain why things are as they are for good reasons.
From my perspective as I don't read kde-devel all I saw was the rude

mail Rob sent to this list.

Post by Rob Kaper
one even said something along the lines of "but if you insist,
use a different e-mail client" and I concluded that this was
definitely not a consideration then.

fetchmail is not a mail client if you talk about that suggestion.

Post by Rob Kaper
My intent to work on it myself clearly indicates that I *did*
consider the possibility that there will not be a complete
rejection of the feature and that a lack of time and other
priorities are indeed the only showstoppers.

I'm don't mind the offer of help. From Rob's mail I got the impression
that he mistakenly assummed the KMail developers were against
implementing a feature that they simply hadn't had time to implement,
and proceeded to denounce developers for preventing that features
implementation.

That's bad behaviour from anyone but comming from a KDE developer it
shows a lack of understanding of the most basic principles of
contributing ot an open source project.

I don't think it would be out of place for Rob to issue the list an
apology.

I still don't understand what exactely you want to implement,
sorry.

I assumed he ment that he wants to cache the bodies of messages that
have been manually retrieved by users.

But I agree he needs to clarify what he wants implemented. From some
of his messages he seems to want to unconditionally download all
mails from the IMAP server which is in fact something that I feel
should not be implemented.

Even caching messages is not what everyone wants, lots of people run
IMAP servers on their home machines, and for them caching messages is
pointless.

Don Sanders

[Alexander Gretencord]

Post by Don Sanders
Even caching messages is not what everyone wants, lots of people run
IMAP servers on their home machines, and for them caching messages is
pointless.

Well I don run an IMAP Server at home and can say that caching is even in that
environment very neccessary. I takes quite some time to load all the headers
in a folder with 500+ msgs (at least it took KMail quite some time when it
had no caching). Still I think it would be great to be able to disable the
caching. We have 20MB quota at university and mail and browser caches can get
quite big so I have to disable/delete that regularly.


Alex

[Don Sanders]

Post by Alexander Gretencord

Post by Don Sanders
Even caching messages is not what everyone wants, lots of people
run IMAP servers on their home machines, and for them caching
messages is pointless.

Well I don run an IMAP Server at home and can say that caching is
even in that environment very neccessary. I takes quite some time
to load all the headers in a folder with 500+ msgs (at least it
took KMail quite some time when it had no caching). Still I think
it would be great to be able to disable the caching. We have 20MB
quota at university and mail and browser caches can get quite big
so I have to disable/delete that regularly.

I was talking about caching messages that is the complete messages
including the body, I wasn't talking about caching headers.

But to be honest I am surprised downloading the headers is slow for
just 500 headers on a local connection, especially as by all accounts
KMail's header retrieval is fairly optimized.

Don.

[Rob Kaper]

Post by Don Sanders
But to be honest I am surprised downloading the headers is slow for
just 500 headers on a local connection, especially as by all accounts
KMail's header retrieval is fairly optimized.

It varies.. most of the time Ctrl-L is extremely fast but sometimes it is a
lot slower and the funny thing is that the progress bar increments are
*always* 9% for me, from 0% all the way up to 99% and done. Might not be
relevant, but it did catch my attention.

It doesn't bother me much though, so I didn't even try to analyze whether
it is my IMAP server, KMail or the network itself causing these hiccups.

Rob

[Alexander Gretencord]

Post by Don Sanders
But to be honest I am surprised downloading the headers is slow for
just 500 headers on a local connection, especially as by all accounts
KMail's header retrieval is fairly optimized.

Well that was some time ago but it really was slow which is why I switched to
kmail cvs as soon as I could when header caching was implemented. Thats about
all I remember. No numbers just that "slow feeling" :)

Btw there are some other strange things I noticed when deleting messages
(Message-Id: <***@gmx.de>)


Alex

[Carsten Burghardt]

Post by Zack Rusin

Post by Rob Kaper
KMail developers: please reconsider this or show me the relevant
parts of the code so I can work on this myself.

I like this idea, if Michael and Carsten won't object I'll put it on my
TODO list for 3.2 and promise to have it done by then.

Go ahead.

[Rob Kaper]

Great, but then I still don't understand what you mean with "IMAP was not
designed to leave the e-mail data on servers indefinitely, just the
authorative header data."

I got the impression some people seemed to imply that just because the server
*does* have the data indefinitely, clients should not ever need to store it.
I should have said something along the lines of "exclusively".

Also be warned that an IMAP server has the right to reassign new UIDs to
all messages in a folder which requires the client to fetch all mails
again. Some servers especially UW-IMAP do this quite often. Therefore
caching can never be really for ever.

Of course clients will need to refetch when the message-id's change, but if
data didn't change the local cache would speed things up for some of us.
I never said that cached data would not need to be validated. And keeping a
cache would be completely optional. Current behaviour will be default. :-)

Rob

[Anthony Moulen]

This has nothing to do with Outlook or any other MUA. This feature is a
fairly common feature request and this is why I'm going to implement
it. It could/would _never_ be the default behavior, simply because it
"IMAP4rev1 also provides the capability for an offline
client to resynchronize with the server (see also [IMAP-DISC])."
which is exactly what's going to be implemented (for a complete
http://asg.web.cmu.edu/cyrus/rfc/draft-ietf-imap-disc-01.html )
I'm planning on adding an "Advanced" tab to our IMAP account dialog and
add option "Cache messages locally" (and move "Show hidden folders" in
there just as well). Don't like, don't need it - don't use it; simple
enough.
Zack

Sorry to swap destination on this (it was to kde-devel) but this seems like
the more appropriate list and I was away from my mail for the last week and
just now am catching up. I would like to help if I can on a IMAP caching
feature. I would personally prefer something in line with what Microsoft
does with Outlook (actually Notes allows for this as well) and be able to
cache individual IMAP folders instead of the entire IMAP mail store. I have
over 2 gig of email on my email server. Most of this is really old archives
of mail from previous jobs and school. I never want to see this mail sitting
on my laptop. I do occassionly have to go digging around it for some old
file or piece of information. I want something like my Inbox and a few key
folders besides cached for offline use.

Another thing that would be nice is if the IMAP implementation supported
proper IMAP deletion tagging. Deleted messages under IMAP are not supposed
to be pulled from the server but tagged for deletion and later purge is
supposed to be called. Deletion right now is painfully slow for large mail
boxes when I just want to completely empty it. Also very painful when
deleting messages over dialup that I had no intention of reading, which also
brings up a final point for this message, I should be able to tag messages
without them being previewed.

[t***@bishop.dhs.org]

Post by Anthony Moulen
Another thing that would be nice is if the IMAP implementation supported
proper IMAP deletion tagging. Deleted messages under IMAP are not supposed
to be pulled from the server but tagged for deletion and later purge is
supposed to be called. Deletion right now is painfully slow for large mail
boxes when I just want to completely empty it. Also very painful when
deleting messages over dialup that I had no intention of reading, which also
brings up a final point for this message, I should be able to tag messages
without them being previewed.

Set your trash folder for your imap account to a folder on the imap
server. Deleting becomes instant, painless, and basically everything
else that you ask for. It's in the config dialog for the imap account.

Have a nice day!

D.A.Bishop

[Rob Kaper]

Post by t***@bishop.dhs.org
Set your trash folder for your imap account to a folder on the imap
server. Deleting becomes instant, painless, and basically everything
else that you ask for. It's in the config dialog for the imap account.

Well, this has one shortfall and is a real pain when you use threading. With
the IMAP methology of having trash as a flag and not a different folder, it
is possible to keep trashed messages in the same view (Outlook strikes them
out).

This is particulary useful because when you use threading. With the current
behaviour, when you delete a message and jump to the next message, the
listview might be resorted because the Date of replies in a thread might be
more recent than that of messages that would (with the thread intact) be
sorted behind it.

Not sure if I explained this properly, but I can make a video recording of
this behaviour when I have my new camera. ;-)

Please note that this isn't IMAP specific though. Keeping threads intact by
flagging messages as trash instead of moving them should help for local
mailboxes as well. Especially mutt and Outlook converts will appreciate a
trash implementation that flags messages, as this is how these clients work
(and it is pretty neat).

But please note new features will have to wait until KDE 3.2, so this won't be
developed soon. Hm, I really should've made an attempt at joining development
much sooner, I've walked around with this itch for far too long! :-)

Rob

[David Bishop]

Post by Rob Kaper

Post by t***@bishop.dhs.org
Set your trash folder for your imap account to a folder on the imap
server. Deleting becomes instant, painless, and basically everything
else that you ask for. It's in the config dialog for the imap account.

Well, this has one shortfall and is a real pain when you use threading. With
the IMAP methology of having trash as a flag and not a different folder, it
is possible to keep trashed messages in the same view (Outlook strikes them
out).
This is particulary useful because when you use threading. With the current
behaviour, when you delete a message and jump to the next message, the
listview might be resorted because the Date of replies in a thread might be
more recent than that of messages that would (with the thread intact) be
sorted behind it.
Not sure if I explained this properly, but I can make a video recording of
this behaviour when I have my new camera. ;-)

No need, anyones thats used KMail with a mailing list knows exactly what
you mean.

Post by Rob Kaper
Please note that this isn't IMAP specific though. Keeping threads intact by
flagging messages as trash instead of moving them should help for local
mailboxes as well. Especially mutt and Outlook converts will appreciate a
trash implementation that flags messages, as this is how these clients work
(and it is pretty neat).

But here is where we disagree :-) I hate the strikeout functionality,
much more than I dislike having my messages rethreaded after every
delete. And my workaround for that is to simply read through an entire
thread before deleting the whole thing at once. I'm not saying that's
perfect, but it works for me ;-)

Post by Rob Kaper
But please note new features will have to wait until KDE 3.2, so this won't be
developed soon. Hm, I really should've made an attempt at joining development
much sooner, I've walked around with this itch for far too long! :-)

Hey, if you want to add that as functionality, more power to you. Heck,
make it the default, and force me to edit an rc file to get my way back,
I don't care. Just don't take the old way completely away B-)

HAND!

D.A.Bishop

[Zack Rusin]

Post by Anthony Moulen
Sorry to swap destination on this (it was to kde-devel) but this
seems like the more appropriate list and I was away from my mail for
the last week and just now am catching up. I would like to help if I
can on a IMAP caching feature. I would personally prefer something
in line with what Microsoft does with Outlook (actually Notes allows
for this as well) and be able to cache individual IMAP folders
instead of the entire IMAP mail store. I have over 2 gig of email on
my email server. Most of this is really old archives of mail from
previous jobs and school. I never want to see this mail sitting on
my laptop. I do occassionly have to go digging around it for some
old file or piece of information. I want something like my Inbox and
a few key folders besides cached for offline use.

First of all thanks for offering help but I think I and Rob will do just
fine as far as coding goes, what we could use would be testers :) You
see caching on a per-folder basis is not a problem. We have to check
the uidvalidity response anyway for each one of them. The problem with
caching IMAP messages is of course the fact that servers are permitted
to change UID of messages in order to keep them in ascending order. So,
if uidvalidty field is OK, then all we do is:
tag1 UID FETCH <lastSeenMessage+1>:* <descriptors>
to get the new messages and
tag2 UID FETCH 1:<lastSeenMessage> FLAGS
to update flags on the old ones. But we have a problem as soon as
uidvalidity changes. The 'Internet Draft: IMAP4 Disconnected Access'
suggests to use other message fields to create unique identifiers other
than uid's for cached messages, requires mua's to clean the queue from
all operations if uid's change between sessions and suggests presenting
a warning message to the users if this happens. In any case, I and Rob
will have do some brain-storming after 3.1 is released to come up with
some clean solutions.

Thanks,
Zack