Fixing things Akonadi doesn't with some SQL-fu

[Jerome Yuzyk]

So while I wait for Akonadi to reliaby do whatever magic was promised us long
ago it still left me with a buggered database that makes every new mail pick-
up an adventure, including this folder. Hopefully there are at least plans to
make akonadictl fsck actually fix things some day so it's easier to wait.

Every time KMail gags on a "Retrieving..." and I shut it down and have to
eventually kill a lingering kmail process and then cleanup after akonadictl
stop crashes I run akonadictl fsck and it gives me the usual list of

Item "122837" has no RID.
Item "138602" has no RID.
Item "170068" has no RID.
Item "201887" has no RID.
Item "201888" has no RID.
Item "201889" has no RID.
Item "201892" has no RID.

which has been growing.

I've gone into akonadiconsole and even though it crashes too when trying to
view one of those no-RID Items (all KMail messages) I can see that in fact the
RID column is empty for the Items that were flagged.

So how could I manually clear out those Items, either with akonadiconsole or
the SQL cli, or even PHPMyAdmin? I've read through https://techbase.kde.org/
KDE_PIM/Akonadi/Development_Tools#Access_to_the_Server_Database but before I
dive in on my own has anyone else ever done any Akonadi DB surgery using any
of these methods? I have some SQL DB experience, and have hand-edited MythTV
SQL tables using PHPMyAdmin before, and it's worth learning more so I can keep
using KMail.

Has anyone attempted such a thing?

[Paul Vixie]

Post by Jerome Yuzyk
...
Item "122837" has no RID.
Item "138602" has no RID.
Item "170068" has no RID.
Item "201887" has no RID.
Item "201888" has no RID.
Item "201889" has no RID.
Item "201892" has no RID.
which has been growing.
...
Has anyone attempted such a thing?

i periodically nuke my database directory and let it rebuild everything.
since there's a dozen gigabytes of e-mail on my imap account, this takes
a while. i do it once a month or so.

switching from mysql to pgsql speeds instantaneous kmail behaviour, but
corrupts more often, and rebuilds more slowly. so, a net loss for me.

perhaps an akonadi dev, for whom this never happens on their own
machine, would like to screen share with some of us some time.

[Peter Humphrey]

Post by Paul Vixie
i periodically nuke my database directory and let it rebuild everything.

Would that be ~/ .local/share/akonadi/db_data/mysql on my system?

Since I recovered a lot of folders from archive I'm getting more problems
with duplicates, messages revived suddenly from days or weeks ago, things
like those.

Post by Paul Vixie
since there's a dozen gigabytes of e-mail on my imap account, this takes
a while. i do it once a month or so.

Far fewer messages here: the daily archive (.tar.bz2) is only about 100M.

[Paul Vixie]

Post by Paul Vixie
i periodically nuke my database directory and let it rebuild everything.

Would that be ~/.local/share/akonadi/db_data/mysql on my system?

i think so, yes. i believe i just wipe out the ~/.local/share/akonadi
directory. note, this will disenfranchise your Sent and Trash folders in
each account, so you'll have to go back in and reconfigure those once
the new database is automatically built and repopulated.

[René J.V. Bertin]

When I read this kind of thread I have to wonder each time if Akonadi really regressed so much since moving to 5.x . Not that I don't have my share of little issues with KDEPIM 4.13 and 4.14, including the BSOD, sporadic need to run akonadi's fsck or just restart it, but it's nothing compared to what I read here.

I must say that I limit the retrieval interval for large folders like the GMail "All Mail" ones, and that I set all folder to "retrieve message bodies on demand", maybe that helps.

Maybe this would in fact be a solution for some: downgrade to 4.14 . Evidently you'd have to replace all of KDEPIM including Akonadi but that should still be possible.

[Paul Vixie]

Post by René J.V. Bertin
...
Maybe this would in fact be a solution for some: downgrade to 4.14 .
Evidently you'd have to replace all of KDEPIM including Akonadi but
that should still be possible.

if i wanted to run a dead mail client to which features aren't being
added and nobody but me cares about, i'd have stuck with MH.

[René J.V. Bertin]

Post by Paul Vixie
if i wanted to run a dead mail client to which features aren't being
added and nobody but me cares about, i'd have stuck with MH.

I didn't say I was proposing a definitive solution, for everyone. Those who find it more important to get every new feature as soon as it's conceived and somewhat implemented may indeed prefer to put up with instability. For me a MUA is a tool that I want to be productive with, not on, ie. spend time using it, not maintaining/updating/repairing it (except possibly in the source code, as far as I'm able to).

The only thing I new feature I think 5.x has that I miss (a bit) is support for GMail's non-legacy authentication protocol which supposed won't lock me out each time I connect from a new location. But my phone also needs the legacy login protocol so either way I'm stuck there.


FWIW, something else I do which may limit issues for me: I quit Kontact before suspending the computer because I'm pretty certain Akonadi doesn't really like connections dropping out from under it when in the middle of something.

R.

[Paul Vixie]

Post by René J.V. Bertin
...
FWIW, something else I do which may limit issues for me: I quit
Kontact before suspending the computer because I'm pretty certain
Akonadi doesn't really like connections dropping out from under it
when in the middle of something.

i understand. that's something that should have been tested before it
shipped, but, i understand.

[Martin Steigerwald]

[…]

Post by René J.V. Bertin
The only thing I new feature I think 5.x has that I miss (a bit) is support
for GMail's non-legacy authentication protocol which supposed won't lock me
out each time I connect from a new location. But my phone also needs the
legacy login protocol so either way I'm stuck there.

I found quite some notable performance improvements between KDEPIM 4.x and
KDEPIM 17.08. The last one is that KMail caches the threading in folders, so
switches even between large folders of more than 30000 mails are almost
instant once the initial threading has completed. This clearly increases my
productivity.

Also KAddressbook as a much nicer GUI and the crypto settings and handling in
KMail improved. There are likely quite some other notable changes, that I
currently miss, but remember the major work between KDEPIM 4.x and curent KF5
based KDEPIM was porting the whole thing to KF5 and Qt 5 which has been a
major task in itself. There are still applications that do net yet have a KF5
/ Qt 5 based release. This was maintenance work the KDEPIM developers had to
do. Work that didn´t automagically create new features or fix up all the
existing bugs. And work that had quite some potential for introducing
regressions.

Post by René J.V. Bertin
FWIW, something else I do which may limit issues for me: I quit Kontact
before suspending the computer because I'm pretty certain Akonadi doesn't
really like connections dropping out from under it when in the middle of
something.

Oh, in my impression this got a lot better.

I never quit KMail before hibernating or suspend. The work-related KMail with
the Exchange IMAP based account usually recovers just fine. Sometimes it needs
some time, but most of the time it gets it right. Even after a night of having
been hibernated. Not all KDEPIM 5.x versions have been this reliable for me,
there have been some improvements in recent versions.

Please note, that I can´t say much about KDEPIM 17.12 yet. I am still at
KDEPIM 17.08, but Sandro works on KDEPIM 17.12 packaging for Debian and I bet
it won´t take that him much longer to get it through the build daemons for all
currently supported architectures.

But even after cleaning out more than 1 million mails from archived folders
into xz compressed tarballs, the POP3 + Maildir setup still manages more than
one million of mails. Even folders with more than 50000+ mails are quite
usable meanwhile. The only thing that is still slow is threading within KMail.
If I disable threading after clicking on a folder with about that amount of
mails KMail + Akonadi just takes a few seconds here in order to display its
contents.

Is it perfect yet? No way, but for it got a long way. And IMHO it is much more
stable than the KDEPIM 4.x stuff and a lot faster too. Also in KDEPIM 3 times
I was never able to use folders with more than 30000 to 40000 mails. Non
Akonadi based KMail became just to slow. Of course, the laptop I had back then
still had a harddisk, a lot less memory and an older CPU, while this one has a
SSD, 16 GiB of RAM and at least a Sandybridge CPU. This definitely helps and I
throw 1 GiB towards the InnoDB buffer pool for MariaDB, but still… for me
KDEPIM works reasonable well. Not perfect, I do have some issues from time to
time, but I did not rebuild the whole Akonadi database in a long, long time –
simply cause it did not break for me. I remember I ranted about rebuilding the
Akonadi setup several times myself. I know how that felt to me, it was not a
pleasant experience, but it just did not break anymore for me, since a long
time. I think since I use KDEPIM 16.04 at least.

So while I totally get that you and others have major issues, I and I think
quite a lot of other users don´t have those. That said, my POP 3 based KMail
has a clean maildir only setup, no mixedmaildir anymore.

Thanks,

[Martin Steigerwald]

Post by Martin Steigerwald

Post by René J.V. Bertin
FWIW, something else I do which may limit issues for me: I quit Kontact
before suspending the computer because I'm pretty certain Akonadi doesn't
really like connections dropping out from under it when in the middle of
something.

Oh, in my impression this got a lot better.
I never quit KMail before hibernating or suspend. The work-related KMail
with the Exchange IMAP based account usually recovers just fine. Sometimes
it needs some time, but most of the time it gets it right. Even after a
night of having been hibernated. Not all KDEPIM 5.x versions have been this
reliable for me, there have been some improvements in recent versions.

I just tried it this morning. Switched to work related session, had the laptop
recover some of it from the SSD based swap – the current hibernation
implementation on Linux often stuffs some part of the used memory into swap
before creating the hibernation memory image – clicked on inbox and then after
some seconds all new mails appeared. And this is with an Exchange based IMAP,
which not the best and most reliable technical alternative available when it
comes to IMAP based mail servers.

And just tested it, all responsive and working just fine. This is with Akonadi
+ KDEPIM 17.08.

Thanks,

[René J.V. Bertin]

Post by Martin Steigerwald
KDEPIM 17.08. The last one is that KMail caches the threading in folders, so

That's a big difference with my settings; I use a "flat date view" aggregation. I think that's because I've never seen a client do really reliable threading that doesn't get in my way (except possibly the Mail app on my iPhone, but different rules apply there anyway).

More speedy everything would of course be nice but since in any case I attempt to keep my folders to a manageable-for-me size I rarely run into issues with that. Full syncs do take longer than I'd like to have to wait, but that's more a result of the number of accounts and on-server folders.

I understand there's less happening on the main thread in KDEPIM5, which is good - and which is something I might be able to hack into KDEPIM4 if it becomes "necessary".

Post by Martin Steigerwald
major task in itself. There are still applications that do net yet have a KF5
/ Qt 5 based release. This was maintenance work the KDEPIM developers had to
do. Work that didn´t automagically create new features or fix up all the
existing bugs. And work that had quite some potential for introducing
regressions.

THere's also been an explosion of the more or less monolithic source/library bundling which must have taken time and is something regular users will see very little of. I won't comment on whether or not this was a good move (but note that KDevelop made the opposite move and can now be built and maintained as a single project).

Post by Martin Steigerwald
I never quit KMail before hibernating or suspend. The work-related KMail with
the Exchange IMAP based account usually recovers just fine.

That's the keyword: "usually". That applies to my situation too, but the term is too flexible and thus not good enough. Restarting the app also has other benefits (purge memory for instance), and just not a problem if I do it on my own terms. Having to do so when I sit down at the awakened-and-supposedly-ready-to-go machine and discover that something just doesn't work right is a completely different scenario. If I can decrease the chance to avoid that by simply starting Kontact anew as part of the wake-up ritual, than I will. That's usually just 2 key presses as I have a dedicated terminal for KDEPIM (nowadays mostly to have feedback when Kontact quits and to see if some akonadi agent has been having the connectivity blues).

It's not that this recipe prevents all issues either (because I don't also quit akonadi itself?): there are still times where new mail just doesn't come in, for instance. Which is hard to detect if you don't have an independent means of checking for new email.

Post by Martin Steigerwald
some time, but most of the time it gets it right. Even after a night of having
been hibernated.

How about 2 nights, or a whole winter? O:^)

[to Paul]

Post by Martin Steigerwald
If you want to be unhappy, that is entirely and solely your choice

Can we please keep that kind of remark out of the (any) discussion? It's even less productive than "blaming, blaming and more blaming" and frankly arrogant to anyone who's ever had to deal with depression.

Post by Martin Steigerwald
i need kontact to Just Work, without giving it an SSD and 16GBytes of
RAM to live in.
perhaps that's where the disconnect lies.

Quite possibly - why would you not want a recent gaming-grade hot-rod to read your email and do some wordprocessing? ;)
Devs (not just of leading-edge applications) who work with older/slower/cheaper hardware seem to be decreasing in numbers.
Which is sad, to be honest. Linux used to be also the kind of OS you put on older hardware to extend its lifetime (without having to spend hundreds of € on a big enough SSD that's truly faster than spinning rust); if its applications lose that feature the OS will in a sense be a victim of its own success.

Post by Martin Steigerwald
... wouldn't that mean that akonadi can have all kind of trouble as soon as
the mail servers are not on the same physical lan? Which in fact I can confirm,
sort of

My experience also. The times I do have to restart akonadi there has usually been some kind of LAN glitch that wasn't picked up properly by the IP stack on the host (which *usually* happens when you suspend/wake the machine, change the WIFI network etc).

R

[Martin Steigerwald]

Dear René,

Post by René J.V. Bertin
[to Paul]

Post by Martin Steigerwald
If you want to be unhappy, that is entirely and solely your choice

Can we please keep that kind of remark out of the (any) discussion? It's
even less productive than "blaming, blaming and more blaming" and frankly
arrogant to anyone who's ever had to deal with depression.

Right.

I allowed myself to be triggered by the blaming tone in Paul´s note, wanting
to control how he behaves. But as I wrote, it is his choice to either continue
to give his power away by blaming what seems to appear external and
unchangeable to him, or constructively engage with helping to change the
current situation. Anyway, this clearly shows me I´d rather stop investing
time with replying to mails that put the main notion on blame.

I don´t agree with your "frankly arrogant to anyone who´s ever had to deal
with depression" accusation for several reasons (including that I do not share
the common notion on how dis-eases come into place after learning a lot on how
I create and perceive so called reality), but that reaches far into areas that
are off topic on this list. I did not mean it in disrespect to Paul or anyone
else.

Thanks,

[René J.V. Bertin]

Post by Martin Steigerwald
I don´t agree with your "frankly arrogant to anyone who´s ever had to deal
with depression" accusation for several reasons (including that I do not share
the common notion on how dis-eases come into place after learning a lot on how
I create and perceive so called reality), but that reaches far into areas that
are off topic on this list.

You'd indeed do better to keep those ideas to yourself unless you're a trained psyciatrist, psychologist or neuroscientist.

R.

[Paul Vixie]

René J.V. Bertin wrote:
...

Post by René J.V. Bertin
It's not that this recipe prevents all issues either (because I don't
also quit akonadi itself?): there are still times where new mail just
doesn't come in, for instance. Which is hard to detect if you don't
have an independent means of checking for new email.

i had a lot of trouble convincing kmail to check for upstream e-mail. F5
didn't force the matter. restarting kontact, no help. restarting
akonadi, no help. restarting the imap transport via akonadiconsole, no
help. telling the transport to resynch via akonadiconsole, no help.
rebooting the VM, no help.

then i found that if i carefully control which transport starts first,
so that kolabsys gets the first IMAP slot, which means IDLE will be
sent, then all is well. so now i restart everything, go offline, then
rescan my kolabsys inbox which asks if i want _that transport_ to be
brought online, and i say yes. then i bring the rest online.

this is more likely to be a bug in kolabsys, but it could be supported
by a leaning bug in akonadi -- it doesn't happen with my other IMAP
server. but i can tell you that i used to be technical, and noone who
was never technical is going to be able to find nor willing to use
workarounds at this level.

Post by René J.V. Bertin
[to Paul]

Post by Martin Steigerwald
If you want to be unhappy, that is entirely and solely your choice

Can we please keep that kind of remark out of the (any) discussion?

agreed, but do note, i am undaunted. "if you don't like the news, go out
there and make some of your own." akonadi is currently horrible for a
lot of people, and those people should not be discouraged from speaking
here. "boots in lab" is not an reasonable acceptance criteria.

Post by René J.V. Bertin

Post by Martin Steigerwald
i need kontact to Just Work, without giving it an SSD and 16GBytes
of RAM to live in.
perhaps that's where the disconnect lies.

Quite possibly - why would you not want a recent gaming-grade hot-rod
to read your email and do some wordprocessing? ;)

it's an x1c6 with a ~2GHz mobile pentium, 16G of RAM, and an SSD.
however, i don't want to dedicate it to kmail -- i have a lot of other
work i must also do. and i can't leave akonadi running except when i
need to use PGP, or else my battery will be dead in less than 1 hour.

Post by René J.V. Bertin

Post by Martin Steigerwald
... wouldn't that mean that akonadi can have all kind of trouble as
soon as the mail servers are not on the same physical lan? Which in
fact I can confirm, sort of

My experience also. The times I do have to restart akonadi there has
usually been some kind of LAN glitch that wasn't picked up properly
by the IP stack on the host (which *usually* happens when you
suspend/wake the machine, change the WIFI network etc).

race conditions are the norm now. we have to save endpoint state and be
very careful about how we restore it every time any part of the
distributed system becomes temporarily unreachable. that's the law (of
physics) and we can either respect it or lose.

[René J.V. Bertin]

Post by Paul Vixie
i had a lot of trouble convincing kmail to check for upstream e-mail. F5
didn't force the matter. restarting kontact, no help. restarting

[snip]

That looks like the times I seemingly am not getting any new email. But for me, restarting akonadi usually helps (with or without "connecting" kmail). Have you ever tried a "akonadictl vacuum" in this kind of situation?

Post by Paul Vixie
then i found that if i carefully control which transport starts first,
so that kolabsys gets the first IMAP slot, which means IDLE will be

Ah, IDLE. I seem to recall that only a single folder could use that at some point. I don't really remember if that was per account (e.g. all INBOX folders), but I'd hope that this limitation has been removed by now!

Post by Paul Vixie
agreed, but do note, i am undaunted.

I wasn't thinking about anyone in particular. I wasn't really daunted either - I know I can always go hang myself if all else fails ;)

Post by Paul Vixie
it's an x1c6 with a ~2GHz mobile pentium, 16G of RAM, and an SSD.

Not exactly an old, underpowered machine compared to my N3150 notebook with 8Gb and an SSHD.

Post by Paul Vixie
race conditions are the norm now.

Indeed.

Post by Paul Vixie
i don't think any of them can afford to interrupt their careers with a
one-month contract to fix akonadi

That leaves the solution of putting up the(se) task(s) as student projects, as well as on sites where developers roam who are looking specifically for contract-based projects. This ML isn't the best way to do that, of course; there are sites dedicated for that (even on stack-overflow you can set bounties). Here's such a site in France, for instance: https://emploi.developpez.com/ (it should be possible to post an ad in English).

[Paul Vixie]

Post by René J.V. Bertin
That looks like the times I seemingly am not getting any new email.
But for me, restarting akonadi usually helps (with or without
"connecting" kmail). Have you ever tried a "akonadictl vacuum" in
this kind of situation?

vacuum, fsck, restart, alone and in every combination/permutation.

Post by René J.V. Bertin

Post by Paul Vixie
i don't think any of them can afford to interrupt their careers
with a one-month contract to fix akonadi

That leaves the solution of putting up the(se) task(s) as student
projects, as well as on sites where developers roam who are looking
specifically for contract-based projects. This ML isn't the best way
to do that, of course; there are sites dedicated for that (even on
stack-overflow you can set bounties). Here's such a site in France,
for instance: https://emploi.developpez.com/ (it should be possible
to post an ad in English).

if i have to hire someone who isn't a kdepim dev to do this work, i know
how to do that. i just don't know how to train them up to speed on this
code base, nor how to get them and their work accepted by the existing
team. also, i don't want to bypass the existing team -- if somebody
wants paid contract work and loves to work on kdepim then we ought to be
keeping the money "in the family" so to speak.

[René J.V. Bertin]

Post by Paul Vixie
how to do that. i just don't know how to train them up to speed on this
code base, nor how to get them and their work accepted by the existing
team. also, i don't want to bypass the existing team -- if somebody
wants paid contract work and loves to work on kdepim then we ought to be
keeping the money "in the family" so to speak.

That could be part of the hiring conditions: introduce yourself to and work with the KDEPIM team. I have the impression that's how one of the more active members of the KDevelop team came aboard, with a somewhat comparable project (changing the parser backend; I may be wrong so I'm not calling any names).

R.

[Daniel Vrátil]

<snip>

Post by Paul Vixie

Post by René J.V. Bertin

Post by Paul Vixie
i don't think any of them can afford to interrupt their careers
with a one-month contract to fix akonadi

That leaves the solution of putting up the(se) task(s) as student
projects, as well as on sites where developers roam who are looking
specifically for contract-based projects. This ML isn't the best way
to do that, of course; there are sites dedicated for that (even on
stack-overflow you can set bounties). Here's such a site in France,
for instance: https://emploi.developpez.com/ (it should be possible
to post an ad in English).

if i have to hire someone who isn't a kdepim dev to do this work, i know
how to do that. i just don't know how to train them up to speed on this
code base

The Architecture overview document might help and of course we have mailing
list and IRC channel where any one can ask us to explain something to them or
help them with their work. Just because they are paid for the contributions
does not mean I wouldn't spend my spare time to help them succeed.

Post by Paul Vixie
nor how to get them and their work accepted by the existing
team.

You make it sound like we would treat them differently from voluntary
contributors. We won't. As long as they communicate with us and respect our
feedback there's no reason why we should not help them or not accept their
contributions.

After all, many of the current (and past) KDE PIM devs, myself included, did
some paid work on KDE in the past as part of our employment and we never had a
problem with our contributions not being accepted. If the maintainer has a
different vision than the customer you work for, a compromise can often be
found that suits both sides (as long as customer's vision is not removing
Akonadi for instance...).

Dan

[René J.V. Bertin]

Post by Daniel Vrátil
problem with our contributions not being accepted. If the maintainer has a
different vision than the customer you work for, a compromise can often be
found that suits both sides (as long as customer's vision is not removing
Akonadi for instance...).

You make it sound as if the Kontact/KMail front-end isn't considered as something that's valuable enough in its own right. That's a pity. I could easily imagine for instance that the Akonadi backend doesn't have the solution most suitable for embedding/bundling, as a mobile app for instance.
(I'm also pretty certain that the current architecture doesn't allow making an all-encompassing standalone Kontact.app Mac app bundle that will interact as it should with other similarly bundled KDE apps on Mac - but that's an independent issue.)

[Daniel Vrátil]

Post by René J.V. Bertin

Post by Daniel Vrátil
problem with our contributions not being accepted. If the maintainer has a
different vision than the customer you work for, a compromise can often be
found that suits both sides (as long as customer's vision is not removing
Akonadi for instance...).

You make it sound as if the Kontact/KMail front-end isn't considered as
something that's valuable enough in its own right.

Where did I say that?

What I meant by my comment is that as an upstream maintainer I would be very
hesitant to accept a contribution that does not fit into the vision of the
project and direction in which the project is heading just because someone was
paid to do it. Especially since this contributor would be gone once the money
run out and it would be up to us to maintain it.

Post by René J.V. Bertin
That's a pity. I could
easily imagine for instance that the Akonadi backend doesn't have the
solution most suitable for embedding/bundling, as a mobile app for
instance.

Historically Akonadi was running on Windows CE and it has means of running
multiple Resources in a single process. Since modern mobile OS have more
restrictions on multi-process, we would need to do a few little improvements
to also run Akonadi Server within the same process, but bundling this in to a
mobile app and having the whole thing run as a single-process application is
certainly possible and would not require any changes in the existing
architecture. The major issue is that we simply don't have any UI for mobile
PIM and nobody really has time to work on it.

I started working on an experimental touch client for PIM (using QML and
Kirigami), but currently it's on a backburner as Kirigami is not there yet and
there are more pressing issues that need my attention than mobile app (as sad
as it is).

Post by René J.V. Bertin
(I'm also pretty certain that the current architecture doesn't
allow making an all-encompassing standalone Kontact.app Mac app bundle that
will interact as it should with other similarly bundled KDE apps on Mac -
but that's an independent issue.)

I don't know how Mac bundles exactly work at runtime, but I suppose that as
long as other apps can access the same DBus session as the Kontact.app bundle
uses and they can access some of its config files to read how to connect to
Akonadi, then there would be no problem with integration/interaction. If you
mean opening-an-attachment-from-Kontact.app-bundle-in-Kate.app-bundle kind of
interaction, then it's hardly something you can blame Kontact or Akonadi
architecture for.

Btw we already have an Akonadi-based application working on Windows where you
basically need to ship everything as a "bundle" on its own. Creating a Mac
bundle is certainly possible, just needs someone to do it and maintain it.
None of the current devs have a Mac (or a desire to own one as far as I know).

[René J.V. Bertin]

Post by Daniel Vrátil
having the whole thing run as a single-process application is
certainly possible and would not require any changes in the existing
architecture.

I guess that would increase the memory footprint of that single process by a few orders of magnitude, but wouldn't it also remove whatever bottleneck there is currently because of interprocess communications?

Post by Daniel Vrátil
I don't know how Mac bundles exactly work at runtime, but I suppose that as

Nothing special, everything is simply set up such that required resources are found inside the bundle (which is just a directory structure). The dynamic linker/loader does have a few mods to make it possible to move the bundles around freely (think relative rpath entries) and there are SDK functions for finding an app bundle, its info dict etc. But basically you can achieve the same thing under Linux - GNUStep does for instance.

That also means everything doesn't have to be inside the app bundle; the minimum is the binary and an info dict (Info.plist) that informs the system what and how to do with the bundle in order to launch it.

Post by Daniel Vrátil
long as other apps can access the same DBus session as the Kontact.app bundle

That's probably the main hurdle. All-encompassing, standalone means everyone includes and runs their DBus, so this approach will only allow independent applications (or application suites) that cannot talk among each other via DBUS.

So yeah, it's possible, but at the very least it'd need an independent DBus install if that's the only interface through which interaction with other applications takes place.
FWIW, the current KDEPIM4 implementation in MacPorts uses a standard Unix way of installing stuff using a minimal app bundle for the front-ends and everything else installed in traditional fashion under a $prefix where all dependencies are also installed. That's also how I'd implement the KDEPIM5 ports.

Post by Daniel Vrátil
interaction, then it's hardly something you can blame Kontact or Akonadi
architecture for.

I'm not sure "blame" is the appropriate term, whom else could I "blame" for the fact that Kontact/Akonadi chose to use DBus instead of something else?;)

R

[Daniel Vrátil]

Post by René J.V. Bertin

Post by Daniel Vrátil
having the whole thing run as a single-process application is
certainly possible and would not require any changes in the existing
architecture.

I guess that would increase the memory footprint of that single process by a
few orders of magnitude,

Not neccesarily. On mobile you usually only have a single mail account, maybe
a calendar and you keep around only limited history, so the memory impact is
smaller in general. Whether you have 5 processes using less memory each or one
process using more memory does not really matter. And of course an opportunity
to make Akonadi less memory-hungry in general (although I'm amazed that people
complain about Akonadi using couple hundred megs of memory, but are completely
OK with shitty Electron-based desktop apps using half a gig of RAM each).

Assuming Android, I can imagine having Akonadi Server+agents as a single
process that performs background syncs and a separate GUI process that talks
to it.

Post by René J.V. Bertin
but wouldn't it also remove whatever bottleneck
there is currently because of interprocess communications?

Our IPC is extremely fast these days, usually the slowest part is the database
right now, which we are working on with Pablo.

Post by René J.V. Bertin

Post by Daniel Vrátil
I don't know how Mac bundles exactly work at runtime, but I suppose that as

Nothing special, everything is simply set up such that required resources
are found inside the bundle (which is just a directory structure). The
dynamic linker/loader does have a few mods to make it possible to move the
bundles around freely (think relative rpath entries) and there are SDK
functions for finding an app bundle, its info dict etc. But basically you
can achieve the same thing under Linux - GNUStep does for instance.
That also means everything doesn't have to be inside the app bundle; the
minimum is the binary and an info dict (Info.plist) that informs the system
what and how to do with the bundle in order to launch it.

Post by Daniel Vrátil
long as other apps can access the same DBus session as the Kontact.app bundle

That's probably the main hurdle. All-encompassing, standalone means everyone
includes and runs their DBus, so this approach will only allow independent
applications (or application suites) that cannot talk among each other via
DBUS.
So yeah, it's possible, but at the very least it'd need an independent DBus
install if that's the only interface through which interaction with other
applications takes place. FWIW, the current KDEPIM4 implementation in
MacPorts uses a standard Unix way of installing stuff using a minimal app
bundle for the front-ends and everything else installed in traditional
fashion under a $prefix where all dependencies are also installed. That's
also how I'd implement the KDEPIM5 ports.

Post by Daniel Vrátil
interaction, then it's hardly something you can blame Kontact or Akonadi
architecture for.

I'm not sure "blame" is the appropriate term, whom else could I "blame" for
the fact that Kontact/Akonadi chose to use DBus instead of something
else?;)

Well, I'm not porting Akonadi away from DBus just because Mac cannot get it
right ;-)

Post by René J.V. Bertin
R

[Martin Steigerwald]

Post by Daniel Vrátil

Post by Daniel Vrátil
long as other apps can access the same DBus session as the Kontact.app

Post by René J.V. Bertin
bundle

That's probably the main hurdle. All-encompassing, standalone means
everyone includes and runs their DBus, so this approach will only allow
independent applications (or application suites) that cannot talk among
each other via DBUS.
So yeah, it's possible, but at the very least it'd need an independent DBus
install if that's the only interface through which interaction with other
applications takes place. FWIW, the current KDEPIM4 implementation in
MacPorts uses a standard Unix way of installing stuff using a minimal app
bundle for the front-ends and everything else installed in traditional
fashion under a $prefix where all dependencies are also installed. That's
also how I'd implement the KDEPIM5 ports.

Post by René J.V. Bertin
interaction, then it's hardly something you can blame Kontact or Akonadi
architecture for.

I'm not sure "blame" is the appropriate term, whom else could I "blame" for
the fact that Kontact/Akonadi chose to use DBus instead of something
else?;)

Well, I'm not porting Akonadi away from DBus just because Mac cannot get it
right

I am happy about that. :)

I think it is more important to improve Akonadi + KDEPIM for the existing user
base, than to port it to whatever platform :)

First make it rock big time, and then probably other developers are willing to
invest the work to port the awesomeness to other platforms.

Also AFAIR haven´t there been attempts to use KDEPIM + Akonadi + Sqlite
backend on Plasma Mobile?

Thanks,

[Daniel Vrátil]

Post by Martin Steigerwald

Post by Daniel Vrátil

Post by Daniel Vrátil
long as other apps can access the same DBus session as the Kontact.app

Post by René J.V. Bertin
bundle

That's probably the main hurdle. All-encompassing, standalone means
everyone includes and runs their DBus, so this approach will only allow
independent applications (or application suites) that cannot talk among
each other via DBUS.
So yeah, it's possible, but at the very least it'd need an independent DBus
install if that's the only interface through which interaction with other
applications takes place. FWIW, the current KDEPIM4 implementation in
MacPorts uses a standard Unix way of installing stuff using a minimal app
bundle for the front-ends and everything else installed in traditional
fashion under a $prefix where all dependencies are also installed. That's
also how I'd implement the KDEPIM5 ports.

Post by René J.V. Bertin
interaction, then it's hardly something you can blame Kontact or Akonadi
architecture for.

I'm not sure "blame" is the appropriate term, whom else could I "blame" for
the fact that Kontact/Akonadi chose to use DBus instead of something
else?;)

Well, I'm not porting Akonadi away from DBus just because Mac cannot get it
right ;)

I am happy about that. :)
I think it is more important to improve Akonadi + KDEPIM for the existing
user base, than to port it to whatever platform :)

Sure, hence the winky smiley :) We are slowly pushing to Windows right now,
which is a huuuge market with some potential to attract more contributors. We
have Akonadi working there, no KDE PIM applications yet though.

Post by Martin Steigerwald
First make it rock big time, and then probably other developers are willing
to invest the work to port the awesomeness to other platforms.
Also AFAIR havenŽt there been attempts to use KDEPIM + Akonadi + Sqlite
backend on Plasma Mobile?

Yes, but since Plasma Mobile is basically Debian, they just compiled the whole
stack for ARM and deployed it. Back in the day there was Kontact Mobile, which
was QML1 port of Kontact running on top of Akonadi+SQLite. I remember using it
on my Nokia N900...:)

Post by Martin Steigerwald
Thanks,

[René J.V. Bertin]

On Thursday March 22 2018 14:19:05 Daniel Vrátil wrote:

My, we're staying on topic again, aren't we? :)

Post by Daniel Vrátil
Not neccesarily. On mobile you usually only have a single mail account, maybe

I wasn't just talking about mobile, but your assumption isn't correct for me at least. I haven't configured all my accounts on my iPhone, but that's in part because most of my email accounts feed into my GMail acount. Of course the email stack ignores all those big GMail folders unless I go in there, and it doesn't download messages until I open them.

Post by Daniel Vrátil
(although I'm amazed that people
complain about Akonadi using couple hundred megs of memory, but are completely
OK with shitty Electron-based desktop apps using half a gig of RAM each).

Don't you see it? If you only get and send a few simple emails a day, why would you want to reserve hundreds of Mb to the email client that could be spent so much better on that desktop you play so much more with? ;)

Post by Daniel Vrátil
Our IPC is extremely fast these days,

Even the one that involves DBus?

Post by Daniel Vrátil
Well, I'm not porting Akonadi away from DBus just because Mac cannot get it
right ;-)

The Mac is not to blame here and let's also not talk about KDE devs who think they know how things _have_ to be done on Mac (and that DBus is off-limits). In a sense Qt never got it right with their DBus wrapper which wraps the FreeDesktop.org D-Bus rather than providing an interface that hooks into existing platform APIs to provide *a* desktop bus.

[Daniel Vrátil]

Post by René J.V. Bertin
My, we're staying on topic again, aren't we? :)

Heh, yeah :)

Post by René J.V. Bertin

Post by Daniel Vrátil
Not neccesarily. On mobile you usually only have a single mail account, maybe

I wasn't just talking about mobile, but your assumption isn't correct for me
at least. I haven't configured all my accounts on my iPhone, but that's in
part because most of my email accounts feed into my GMail acount. Of course
the email stack ignores all those big GMail folders unless I go in there,
and it doesn't download messages until I open them.

The only thing we are missing in Akonadi is the "sliding window" that allows
to fetch only first 100 or so message headers and then fetch another 100 when
requested. Minor adjustment to the expiration policy would be needed too to
make it able to completely delete messages older than X days.

Post by René J.V. Bertin

Post by Daniel Vrátil
(although I'm amazed that people
complain about Akonadi using couple hundred megs of memory, but are
completely OK with shitty Electron-based desktop apps using half a gig of
RAM each).

Don't you see it? If you only get and send a few simple emails a day, why
would you want to reserve hundreds of Mb to the email client that could be
spent so much better on that desktop you play so much more with? ;)

I just quickly glanced into htop to see how much my Akonadi is eating right
now - I'm sure most people don't have 3 massive mail accounts, 5 calendar
accounts and everything compiled in full debug mode. For users with one gmail
account it's probably around a hundred or two hundred megs? Not saying we can
do better, but we are doing reasonably good IMO.

Post by René J.V. Bertin

Post by Daniel Vrátil
Our IPC is extremely fast these days,

Even the one that involves DBus?

Nowadays we only use DBus for simple calls here and there. Notifications go
through the binary protocol.

Post by René J.V. Bertin

Post by Daniel Vrátil
Well, I'm not porting Akonadi away from DBus just because Mac cannot get it
right ;-)

The Mac is not to blame here and let's also not talk about KDE devs who
think they know how things _have_ to be done on Mac (and that DBus is
off-limits). In a sense Qt never got it right with their DBus wrapper which
wraps the FreeDesktop.org D-Bus rather than providing an interface that
hooks into existing platform APIs to provide *a* desktop bus.

I see. Well, contributions are welcomed, I suppose :)

Dan

[Mathias Homann]

Post by René J.V. Bertin
FWIW, something else I do which may limit issues for me: I quit Kontact
before suspending the computer because I'm pretty certain Akonadi doesn't
really like connections dropping out from under it when in the middle of
something.

... wouldn't that mean that akonadi can have all kind of trouble as soon as
the mail servers are not on the same physical lan? Which in fact I can confirm,
sort of: my desktop at home talks to the mailserver via 1gbit ethernet, no
trouble. my latop, when it is at home, has no trouble. when I'm elsewhere and
connecting through vpn via whatever shady wifi is available: things go less
smoothly.

Connections "over the internet" drop all the time... hotel wifi being crap....
cellular data changing towers... etc etc etc.

cheers
MH

[Martin Steigerwald]

Hello Jerome.

Post by Jerome Yuzyk
So while I wait for Akonadi to reliaby do whatever magic was promised us
long ago it still left me with a buggered database that makes every new
mail pick- up an adventure, including this folder. Hopefully there are at
least plans to make akonadictl fsck actually fix things some day so it's
easier to wait.

Which version of Akonadi + KMail do you use?

Post by Jerome Yuzyk
Every time KMail gags on a "Retrieving..." and I shut it down and have to
eventually kill a lingering kmail process and then cleanup after akonadictl
stop crashes I run akonadictl fsck and it gives me the usual list of
Item "122837" has no RID.
Item "138602" has no RID.
Item "170068" has no RID.
Item "201887" has no RID.
Item "201888" has no RID.
Item "201889" has no RID.
Item "201892" has no RID.
which has been growing.
I've gone into akonadiconsole and even though it crashes too when trying to
view one of those no-RID Items (all KMail messages) I can see that in fact
the RID column is empty for the Items that were flagged.

Does it only crash when accessing items without RID? Probably this is just a
co-incidence.

Post by Jerome Yuzyk
So how could I manually clear out those Items, either with akonadiconsole or
the SQL cli, or even PHPMyAdmin? I've read through
https://techbase.kde.org/
KDE_PIM/Akonadi/Development_Tools#Access_to_the_Server_Database but before
I dive in on my own has anyone else ever done any Akonadi DB surgery using
any of these methods? I have some SQL DB experience, and have hand-edited
MythTV SQL tables using PHPMyAdmin before, and it's worth learning more so
I can keep using KMail.
Has anyone attempted such a thing?

I suggest you read through my second post in the thread:

Re: Review of database aspect of Akonadi, Akonadi concepts and a master plan
https://marc.info/?l=kde-pim&m=152137731726228&w=2

It gives more details about the meaning of these "Itam nnn has no RID"
messages. And the major change in Akonadi that will fix the issue: Server-side
change recording.

Just removing those entries from the database may result in data loss as item
without RID are items that only exist in the Akonadi database cause for some
reason the Akonadi resource was not able to play back the (changed) item to
the backend storage like for example an IMAP account.

If you really want to mess with the database, I suggest you also read through:

https://community.kde.org/KDE_PIM/Akonadi/Architecture

Dan summarizes there the major concepts of Akonadi. There are some bits still
missing, but as is it should already give a good overview on how Akonadi
works. At least it helped me a lot to understand it a bit better.

Also you may review the summary Pablo gave about the current state of the
database analysis in:

Re: Review of database aspect of Akonadi, Akonadi concepts and a master plan
https://marc.info/?l=kde-pim&m=152155153509020&w=2

Together with the writing of Dan, it should give you good understanding of the
underlying architecture.

In any case: The item without RID messages may not be the cause of the issues
you describe. So I suggest you´d rather not delete them until you are sure
that they are the issue, or you have made a backup of the Akonadi database
before.

Thanks,