Discussion:
Good way to back up Gmail when you've reached the size limit
(too old to reply)
ultred ragnusen
2018-03-02 23:01:30 UTC
Permalink
What's a good way to back up Gmail when you've reached the size limit?
Loading Image...

If you use the web for Gmail, and if you're full to the brim such that
Google has finally shut you down because you've used up all your disk
space, what is a good way to back up all your Gmail to a HDD on Windows so
that you can delete all the email and start fresh?
Jonathan N. Little
2018-03-03 12:24:33 UTC
Permalink
Post by ultred ragnusen
What's a good way to back up Gmail when you've reached the size limit?
http://wetakepic.com/images/2018/03/03/gmail_bck_01.jpg
If you use the web for Gmail, and if you're full to the brim such that
Google has finally shut you down because you've used up all your disk
space, what is a good way to back up all your Gmail to a HDD on Windows so
that you can delete all the email and start fresh?
Install Thunderbird. Add that account. Create message filters to copy
IMAP folders to analogous ones you can mirror in Thunderbird's Local
Folders. After mail downloaded purge mail on Google's server. Old
messages now accessible in Thunderbird, just remember they are mow on
your hard drive and preservation is your responsibility, i.e., local
backup plan required.
--
Take care,

Jonathan
-------------------
LITTLE WORKS STUDIO
http://www.LittleWorksStudio.com
J. P. Gilliver (John)
2018-03-03 13:45:56 UTC
Permalink
Post by Jonathan N. Little
Post by ultred ragnusen
What's a good way to back up Gmail when you've reached the size limit?
http://wetakepic.com/images/2018/03/03/gmail_bck_01.jpg
If you use the web for Gmail, and if you're full to the brim such that
Google has finally shut you down because you've used up all your disk
space, what is a good way to back up all your Gmail to a HDD on Windows so
that you can delete all the email and start fresh?
Install Thunderbird. Add that account. Create message filters to copy
IMAP folders to analogous ones you can mirror in Thunderbird's Local
Folders. After mail downloaded purge mail on Google's server. Old
messages now accessible in Thunderbird, just remember they are mow on
your hard drive and preservation is your responsibility, i.e., local
backup plan required.
If you're going to do this anyway, you might as well switch to POP.
(With local backup: as I said, if you're going to do that anyway.)
--
J. P. Gilliver. UMRA: 1960/<1985 MB++G()AL-IS-Ch++(p)***@T+H+Sh0!:`)DNAf

The fifth bestselling detail of all time: the Ford Transit. (RT/C4 2015-5-24.)
Jonathan N. Little
2018-03-03 14:40:34 UTC
Permalink
Post by J. P. Gilliver (John)
Post by Jonathan N. Little
Post by ultred ragnusen
What's a good way to back up Gmail when you've reached the size limit?
http://wetakepic.com/images/2018/03/03/gmail_bck_01.jpg
 If you use the web for Gmail, and if you're full to the brim such that
Google has finally shut you down because you've used up all your disk
space, what is a good way to back up all your Gmail to a HDD on Windows so
that you can delete all the email and start fresh?
Install Thunderbird. Add that account. Create message filters to copy
IMAP folders to analogous ones you can mirror in Thunderbird's Local
Folders. After mail downloaded purge mail on Google's server. Old
messages now accessible in Thunderbird, just remember they are mow on
your hard drive and preservation is your responsibility, i.e., local
backup plan required.
If you're going to do this anyway, you might as well switch to POP.
(With local backup: as I said, if you're going to do that anyway.)
That is what I would do, but maybe OP needs to access mail from multiple
devices. Then POP is not your best solution...
--
Take care,

Jonathan
-------------------
LITTLE WORKS STUDIO
http://www.LittleWorksStudio.com
Ian Jackson
2018-03-03 15:38:35 UTC
Permalink
Post by Jonathan N. Little
Post by J. P. Gilliver (John)
Post by Jonathan N. Little
Post by ultred ragnusen
What's a good way to back up Gmail when you've reached the size limit?
http://wetakepic.com/images/2018/03/03/gmail_bck_01.jpg
 If you use the web for Gmail, and if you're full to the brim such that
Google has finally shut you down because you've used up all your disk
space, what is a good way to back up all your Gmail to a HDD on Windows so
that you can delete all the email and start fresh?
Install Thunderbird. Add that account. Create message filters to
copy IMAP folders to analogous ones you can mirror in Thunderbird's
Local Folders. After mail downloaded purge mail on Google's server.
Old messages now accessible in Thunderbird, just remember they are
mow on your hard drive and preservation is your responsibility,
i.e., local backup plan required.
If you're going to do this anyway, you might as well switch to POP.
(With local backup: as I said, if you're going to do that anyway.)
That is what I would do, but maybe OP needs to access mail from
multiple devices. Then POP is not your best solution...
I (can) also access Google Mail through Thunderbird - and I had the
annoying experience of GM suddenly deleting most of what was in GM
online account (and also the webmail). [Don't know why, but apparently
it's sometimes a GM 'feature'.] It hasn't happened again, but I think
it's wise to move anything important to somewhere in Thunderbird's Local
Folders.
--
Ian
Jonathan N. Little
2018-03-03 17:05:21 UTC
Permalink
Post by Ian Jackson
I (can) also access Google Mail through Thunderbird - and I had the
annoying experience of GM suddenly deleting most of what was in GM
online account (and also the webmail). [Don't know why, but apparently
it's sometimes a GM 'feature'.] It hasn't happened again, but I think
it's wise to move anything important to somewhere in Thunderbird's Local
Folders.
Okay firstly "online account" is webmail. Whether on not email is
removed from the server depends on a couple of things. If you are
accessing gmail with Thunderbird via POP3 and if you are using the
default setting:

Server Settings > []Leave message on server

unchecked then once access your mail with Thunderbird the message is
downloaded to your local machine and is summarily deleted on the server.
Once deleted on the server you will not see it on Webmail but only on
the local machine where you have Thunderbird installed.

That is why I said POP is not always best if you want access from more
than one device. You would have to check that option so the message
remains on the google server.
--
Take care,

Jonathan
-------------------
LITTLE WORKS STUDIO
http://www.LittleWorksStudio.com
J. P. Gilliver (John)
2018-03-03 19:39:35 UTC
Permalink
Post by Ian Jackson
Post by Jonathan N. Little
Post by J. P. Gilliver (John)
Post by Jonathan N. Little
Post by ultred ragnusen
What's a good way to back up Gmail when you've reached the size limit?
http://wetakepic.com/images/2018/03/03/gmail_bck_01.jpg
 If you use the web for Gmail, and if you're full to the brim such that
Google has finally shut you down because you've used up all your disk
space, what is a good way to back up all your Gmail to a HDD on Windows so
that you can delete all the email and start fresh?
Install Thunderbird. Add that account. Create message filters to
copy IMAP folders to analogous ones you can mirror in Thunderbird's
Local Folders. After mail downloaded purge mail on Google's server.
Old messages now accessible in Thunderbird, just remember they are
mow on your hard drive and preservation is your responsibility,
i.e., local backup plan required.
If you're going to do this anyway, you might as well switch to POP.
(With local backup: as I said, if you're going to do that anyway.)
That is what I would do, but maybe OP needs to access mail from
multiple devices. Then POP is not your best solution...
I (can) also access Google Mail through Thunderbird - and I had the
annoying experience of GM suddenly deleting most of what was in GM
online account (and also the webmail). [Don't know why, but apparently
it's sometimes a GM 'feature'.] It hasn't happened again, but I think
it's wise to move anything important to somewhere in Thunderbird's
Local Folders.
I have at least two friends who use POP and multiple devices; they only
have "delete from server" on one (or if it's Thunderbird, "leave on
server" ticked!), and they know that when they download emails on that
device, they won't be on the server any more. It isn't rocket science. I
think some of their other devices may even be IMAP. (Do iPhones do POP?)
--
J. P. Gilliver. UMRA: 1960/<1985 MB++G()AL-IS-Ch++(p)***@T+H+Sh0!:`)DNAf

"Knowledge isnt elitist - that's rubbish! Why are we embarrassed by the idea
that people know things? It's not a conspiracy against the ignorant. Knowing
things is good!" - Jeremy Paxman, RT 14-20 August 2010
ultred ragnusen
2018-03-03 19:35:19 UTC
Permalink
Post by Jonathan N. Little
That is what I would do, but maybe OP needs to access mail from multiple
devices. Then POP is not your best solution...
We all used POP in the olden days, where POP downloaded the mail off the
server, which would be fine in this case because I would do that from
Windows and then I would have it delete the mail off the server so that the
result would be an empty Gmail account.

Then I could set Gmail back to IMAP so that I could access it from multiple
devices but from a fresh empty account.
ultred ragnusen
2018-03-03 19:28:44 UTC
Permalink
Post by Jonathan N. Little
Install Thunderbird. Add that account. Create message filters to copy
IMAP folders to analogous ones you can mirror in Thunderbird's Local
Folders. After mail downloaded purge mail on Google's server. Old
messages now accessible in Thunderbird, just remember they are mow on
your hard drive and preservation is your responsibility, i.e., local
backup plan required.
I used to have Eudora (way way long ago) where, if I recall, the
attachments were saved, by default, in some arbitrary file format (or, at
least not as separate attachments - maybe it was one big MIME encoded file
- I don't remember).

All I remember is that the archives that I saved from Eudora didn't have
attachments that were separate files like any decent backup would want them
(for compatibility).

Since the plan it to only use Thunderbird as the "backup method", the
question needs to be asked...

Q: Are Thunderbird's files saved in a common format that any file manager
can easily access, completely outside of Thunderbird itself?
J. P. Gilliver (John)
2018-03-03 19:48:29 UTC
Permalink
Post by ultred ragnusen
Post by Jonathan N. Little
Install Thunderbird. Add that account. Create message filters to copy
IMAP folders to analogous ones you can mirror in Thunderbird's Local
Folders. After mail downloaded purge mail on Google's server. Old
messages now accessible in Thunderbird, just remember they are mow on
your hard drive and preservation is your responsibility, i.e., local
backup plan required.
I used to have Eudora (way way long ago) where, if I recall, the
attachments were saved, by default, in some arbitrary file format (or, at
least not as separate attachments - maybe it was one big MIME encoded file
- I don't remember).
I don't _think_ it saved either emails or attachments as files.
Post by ultred ragnusen
All I remember is that the archives that I saved from Eudora didn't have
attachments that were separate files like any decent backup would want them
(for compatibility).
Since the plan it to only use Thunderbird as the "backup method", the
question needs to be asked...
Q: Are Thunderbird's files saved in a common format that any file manager
can easily access, completely outside of Thunderbird itself?
A: I don't know of _any_ email client which stores either emails or
attachments as separate files: they're all in one big file (or possibly
one big file per internal "folder" for some clients; I think Outlook
does that). Which has always struck me as perilous: damage that file and
you've lost lots of emails. I could - just about - see validity when
disc space was expensive, as emails (in those days) tended to be small,
and storing them all in one file made (some) sense, because of cluster
size (storing them separately would have meant even a 300 byte email
would have taken a whole cluster); however, it isn't so now.

Most - I think even Eudora - clients have the ability to export
(sometimes called save or other word) attachments (as their original
filetype), and in many cases then remove them from the email but leave
the email (sans attachment) in the inbox or wherever; but this is always
an action you have to deliberately take. Some, possibly most, also have
the ability to save emails as separate, but again that is something you
have to do deliberately. (From Outlook, the files are of type ".msg".)
--
J. P. Gilliver. UMRA: 1960/<1985 MB++G()AL-IS-Ch++(p)***@T+H+Sh0!:`)DNAf

"Knowledge isnt elitist - that's rubbish! Why are we embarrassed by the idea
that people know things? It's not a conspiracy against the ignorant. Knowing
things is good!" - Jeremy Paxman, RT 14-20 August 2010
ultred ragnusen
2018-03-05 12:55:21 UTC
Permalink
Post by J. P. Gilliver (John)
Most - I think even Eudora - clients have the ability to export
(sometimes called save or other word) attachments (as their original
filetype), and in many cases then remove them from the email but leave
the email (sans attachment) in the inbox or wherever; but this is always
an action you have to deliberately take. Some, possibly most, also have
the ability to save emails as separate, but again that is something you
have to do deliberately. (From Outlook, the files are of type ".msg".)
I found what I think may be the /easiest/ solution for Gmail archival.
https://support.google.com/accounts/answer/3024190?hl=en

I ran through those steps many hours ago, and I still haven't received my
email, where Google says it could take days for the 15GB to be archived.

Google says they'll save the archival link for a week, so in a day or two I
should have an answer for you.

In hindsight, had I labeled my mail by some sort of organizational system,
it would be easier to archive because Google will archive by label if you
want them to (although they couch that in these words):

Each message's labels are preserved in a special X-Gmail-Labels header
in your download file. While no mail client recognizes this header now,
most mail clients allow for extensions to be written that could make
use of these labels in the future.
VanguardLH
2018-03-03 13:01:30 UTC
Permalink
Post by ultred ragnusen
What's a good way to back up Gmail when you've reached the size limit?
http://wetakepic.com/images/2018/03/03/gmail_bck_01.jpg
If you use the web for Gmail, and if you're full to the brim such that
Google has finally shut you down because you've used up all your disk
space, what is a good way to back up all your Gmail to a HDD on Windows so
that you can delete all the email and start fresh?
If you use a local e-mail client that is extensible, you can use an
add-on that will search your e-mails to strip out all the attachments.
Some users think e-mail is FTP (file transfer protocol) and a file
server (file storage). Wrong. Attachments can be removed from e-mails
and either discarded if not needed or stored elsewhere with a link
pointing to them in the e-mail.

http://www.sperrysoftware.com/Email-Tools/product/attachment-save/
https://www.mapilab.com/outlook/attachments_processor/

There might be standalone bulke remove attachment processing tools to
strip attachments from e-mails and optionally save the attachments to
files in your OS; however, they would have to know how to parse the
message store for your local e-mail client. You're pretty screwed if
all you use is the e-mail providers webmail client. They rarely have
all the features available in a local e-mail client. For example:

http://steffondavis.com/delete-large-attachments-in-gmail-without-deleting-the-message/
http://www.zdnet.com/article/how-to-delete-large-attachments-to-save-storage-space-in-gmail/

The authors say Google forces their users to delete the huge e-mails due
to attachments (after first opting to save/extract the attachments to
elsewhere). What's the solution? Use a more robust local e-mail client
that lets you strip off the attachments. Archiving will also let you
move out old items to reduce the current message store (which gets
reflected to the server if using IMAP).

I'm not a slob with e-mail. If someone sends me a file via e-mail,
rarely is the e-mail of any importance so I save the attachment
elsewhere (if I want it) and DELETE the e-mail. If I'm sending someone
an e-mail with an attachment then obviously I don't need to keep that
e-mail with the attachment. I already have the file by itself that I
attached to the sent e-mail.

Most personal e-mail is fluff and should be deleted after a while. A
local e-mail client may have an archiving option that lets you delete
e-mails when they get old, like over 5 years old. Do you really need to
keep an old e-mail many years after it notified you about a change in a
get-together with your buddy?

Some companies need to keep e-mails for a long time because they involve
their customers. Auto-archiving can come in handy for that. Set
archiving after, say, 1 year to move old items into an archive file.
Old items get moved out the current message store into the archive file.
If using POP, the e-mails should already be getting deleted from the
server after retrieving new e-mails. If using IMAP, moving the items
out of the local client's current message store (into an archive file)
means those items disappear (get deleted) in the local message store
which syncs to the server to delete them from there. You can chain the
archiving operation so move items, say, over 2 years old from an archive
file into another archive file. So you could have an archive of the
last year's worth of e-mails, another archive that holds the prior
year's worth of e-mails, and so one with each archive hold a year's
worth of e-mails but one year older than the prior archive in the chain.
For archive chaining to work means the archive file must be opened in
the local e-mail client so it can run its archive operation on the
archive file(s). If you're just dumping all items older than, say, 5
years into a single archive folder, you don't need to keep the archive
file loaded in the local e-mail client. Its archiving function will
move the old items into the archive file whether that archive file is
loaded in the local e-mail client or not.

When items are moved due to archiving, they aren't in the local store
anymore so, for IMAP, they will also disappear up on the server. If you
are using POP and changed the local e-mail client to not delete new
e-mails from the server after retrieving them from the server (you want
to poll for new e-mails using multiple clients, like one on your desktop
PC and also get them on your smartphone or another computer), either do
the cleanup manually to flush the old turds or see if there is an option
to flush old e-mails after awhile (send a DELete to the server while
still retaining the local copy).

Webmail clients are for boobs or convenience, like when travelling.
They won't have anywhere need the functionality of a local e-mail client
which is what you'll need to strip off the attachment (and optionally
store them elsewhere) or to archive ancient messages. However, deleting
the fluff as you get it means incrementally doing the cleanup a little
at a time rather than waiting until a catastrophic event (out of space)
forces you to cleanup.
Ken Blake
2018-03-03 15:59:44 UTC
Permalink
Post by VanguardLH
Webmail clients are for boobs or convenience, like when travelling.
They won't have anywhere need the functionality of a local e-mail client
A very strong ditto! I've always used an e-mail client at home, and
for a while I used to use webmail when I traveled. Then I bought and
switched to a netbook, using an e-mail client, for traveling, and then
to a tablet.

Now I use my smart phone for e-mail when traveling. Each
step--netbook, tablet, smart phone--has been to something smaller,
lighter, and easier to travel with.
Jonathan N. Little
2018-03-03 17:16:20 UTC
Permalink
Post by Ken Blake
Post by VanguardLH
Webmail clients are for boobs or convenience, like when travelling.
They won't have anywhere need the functionality of a local e-mail client
A very strong ditto! I've always used an e-mail client at home, and
for a while I used to use webmail when I traveled. Then I bought and
switched to a netbook, using an e-mail client, for traveling, and then
to a tablet.
Now I use my smart phone for e-mail when traveling. Each
step--netbook, tablet, smart phone--has been to something smaller,
lighter, and easier to travel with.
Also more secure because email clients have the web features that make a
malicious message dangerous either disabled or not supported. Whereas
JavaScript, remote content, plugins, etc., are all present in *web
browsers* which you want otherwise web browsing would look like when you
are using Lynx, or turn the Internet into a BBS...
--
Take care,

Jonathan
-------------------
LITTLE WORKS STUDIO
http://www.LittleWorksStudio.com
ultred ragnusen
2018-03-03 19:39:33 UTC
Permalink
Post by VanguardLH
Webmail clients are for boobs or convenience, like when travelling.
They won't have anywhere need the functionality of a local e-mail client
which is what you'll need to strip off the attachment (and optionally
store them elsewhere) or to archive ancient messages.
I had a bad experience with Eurdora, of years past, where the attachments
weren't stripped out in the clear as separate files, so I definitely want
to have the attachments as separately accessed files.

I also have had a bad experience in the past of the mail being one huuuuuge
file (like a huge mbox.txt file!) which isn't what I want either.

Basically, I'm not sure what exactly I want, but thinking about it, I want
the mail accessed as a 'database' would be accessed. It would be great to
have a directory tree, with each thread in its own directory, for example,
and where all the attachments are separate.

If that's possible...
Big Al
2018-03-03 13:44:45 UTC
Permalink
Post by ultred ragnusen
What's a good way to back up Gmail when you've reached the size limit?
http://wetakepic.com/images/2018/03/03/gmail_bck_01.jpg
If you use the web for Gmail, and if you're full to the brim such that
Google has finally shut you down because you've used up all your disk
space, what is a good way to back up all your Gmail to a HDD on Windows so
that you can delete all the email and start fresh?
I like Johnathan's idea even if it puts the burden of backup on you.
Another thing you can do is taking a point from VanguardLH is purge the
send folder at Gmail webmail. I agree, you don't need all the sent
stuff. But that is only a small stopgap measure.

This is just a guess but if you're like my elderly sister, she won't
delete anything, zippo, nada! Even the "party has been cancelled"
email from 10 years ago. Or the 1000 spam emails that she subscribed to
but has not hit the unsubscribe link, so every week she gets another
spam email (not spam really but still it's something she doesn't want)
then a small modification in mail habits may be due.

She's on yahoo and I don't know their limit but I keep telling her she
may loose email one day!.
ultred ragnusen
2018-03-03 19:46:46 UTC
Permalink
Post by Big Al
I like Johnathan's idea even if it puts the burden of backup on you.
I'm OK with the one-time burden of backup.

I'm extremely organized with computers, but absolutely unorganized with
email. I don't know why the dichotomy, but mail never fit into a simple
folder structure for me. It's all basically random.

What I "think" I want is what I would think anyone would want, which is to
set a time point, like today, and archive everything in a way that it can
be accessed ten years from now.

Having been burned many times, as have all of you, by proprietary archive
formats, I don't want anything that can't save the information in separate
files, where there are really only a few "concepts" involved.

1. Attachments (that's relatively easy, at least with folder trees it is)
2. Threads (that's a lot harder, I think, to save in a folder tree)
3. Contacts (I think that's the easiest of all, with a simple export)

Since my need has to be nearly universal, there must be good solutions out
there that convert email threads to a folder tree on disk.
Frank Slootweg
2018-03-03 21:18:44 UTC
Permalink
ultred ragnusen <***@ragnusen.com> wrote:
[...]
Post by ultred ragnusen
Having been burned many times, as have all of you, by proprietary archive
formats, I don't want anything that can't save the information in separate
files, where there are really only a few "concepts" involved.
1. Attachments (that's relatively easy, at least with folder trees it is)
This (1.) requirement makes the format(s) proprietary by definition,
because - as far as standards go - any attachments are *part of* the RFC
5322 format message. I.e. the concept of seperate attachments is a
non-RFC one.
Post by ultred ragnusen
2. Threads (that's a lot harder, I think, to save in a folder tree)
Most MUAs can do this, so you wouldn't need to be tied to any
particular proprietary MUA.
Post by ultred ragnusen
3. Contacts (I think that's the easiest of all, with a simple export)
Since my need has to be nearly universal, there must be good solutions out
there that convert email threads to a folder tree on disk.
See above. My suggestion would be to convert the mail folders to mbox
format and then to split up the mbox-archive to individual mbox format
messages (if you indeed want on file per message).

Thunderbird has an Extension ('ImportExportTools') which can export to
mbox format, AFAICT both as single file per folder and as single file
per message. ImportExportTools also has some attachment-related options
which might be worth your while to have a look at.
ultred ragnusen
2018-03-03 22:20:13 UTC
Permalink
Post by Frank Slootweg
Post by ultred ragnusen
1. Attachments (that's relatively easy, at least with folder trees it is)
This (1.) requirement makes the format(s) proprietary by definition,
because - as far as standards go - any attachments are *part of* the RFC
5322 format message. I.e. the concept of seperate attachments is a
non-RFC one.
I'm think what you're saying is that if I strip out the attachments, en
masse, those attachments will lose their association to individual mail
messages ... is that what you're trying to tell me?

If so, I guess the basic need is to do two things, which is wasteful, but
which solves that problem:
1. Save attachments associated with their individual messages, and,
2. Save those same attachments, except outside their individual messages.
Post by Frank Slootweg
Post by ultred ragnusen
2. Threads (that's a lot harder, I think, to save in a folder tree)
Most MUAs can do this, so you wouldn't need to be tied to any
particular proprietary MUA.
Good, because while attachments are important, the individual threads far
outnumber the attachments in number, so, there will be a billion individual
folders, one for each thread (I presume).
Post by Frank Slootweg
Post by ultred ragnusen
3. Contacts (I think that's the easiest of all, with a simple export)
Since my need has to be nearly universal, there must be good solutions out
there that convert email threads to a folder tree on disk.
See above. My suggestion would be to convert the mail folders to mbox
format and then to split up the mbox-archive to individual mbox format
messages (if you indeed want on file per message).
Thunderbird has an Extension ('ImportExportTools') which can export to
mbox format, AFAICT both as single file per folder and as single file
per message. ImportExportTools also has some attachment-related options
which might be worth your while to have a look at.
As I recall, didn't Euroda do the "mbox" format?

The only thing I didn't like about Eurdora backups was that only Eudora
could read them, which is a failure in backing up if that's what happens
here.
J. P. Gilliver (John)
2018-03-04 00:15:15 UTC
Permalink
In message <***@40tude.net>, ultred ragnusen
<***@ragnusen.com> writes:
[]
Post by ultred ragnusen
If so, I guess the basic need is to do two things, which is wasteful, but
1. Save attachments associated with their individual messages, and,
2. Save those same attachments, except outside their individual messages.
[]
The association can be retained one way round, given a suitable mail
client: some will allow attachments to be saved then removed, but leave
text in the email saying "saved as file ...".
--
J. P. Gilliver. UMRA: 1960/<1985 MB++G()AL-IS-Ch++(p)***@T+H+Sh0!:`)DNAf

Life, liberty and the happiness of pursuit!
ultred ragnusen
2018-03-05 08:17:00 UTC
Permalink
Post by J. P. Gilliver (John)
The association can be retained one way round, given a suitable mail
client: some will allow attachments to be saved then removed, but leave
text in the email saying "saved as file ...".
I realize my need for archiving mail privately is probably nearly
universal, but I'm not sure what the universal solution would be with
respect to associating detached attachments to their individual messages.

My very first thought is that a "folder tree" type of archival would have
no trouble putting the detached attachment into the same folder as the
message it relates to.

It's hard to draw but it would be something like this folder tree:
.......................Mail
........................|
........Thread A...........Thread B.........Thread C
...........|
........Message 1...........Message 2.........Message 3
...........|
........Attachment x........Attachment y......Attachment z

Maybe I'm too romantic, but this seems so obvious as the first choice of
mail archival, that I simply /assume/ a tool to do this already exists.
Frank Slootweg
2018-03-04 11:39:16 UTC
Permalink
Post by ultred ragnusen
Post by Frank Slootweg
Post by ultred ragnusen
1. Attachments (that's relatively easy, at least with folder trees it is)
This (1.) requirement makes the format(s) proprietary by definition,
because - as far as standards go - any attachments are *part of* the RFC
5322 format message. I.e. the concept of seperate attachments is a
non-RFC one.
I'm think what you're saying is that if I strip out the attachments, en
masse, those attachments will lose their association to individual mail
messages ... is that what you're trying to tell me?
Yes.
Post by ultred ragnusen
If so, I guess the basic need is to do two things, which is wasteful, but
1. Save attachments associated with their individual messages, and,
2. Save those same attachments, except outside their individual messages.
That is a good approach, condidering that - as you mention (below) -
attachments are a minority, so the duplication is a lesser issue,
storage space wise.
Post by ultred ragnusen
Post by Frank Slootweg
Post by ultred ragnusen
2. Threads (that's a lot harder, I think, to save in a folder tree)
Most MUAs can do this, so you wouldn't need to be tied to any
particular proprietary MUA.
Good, because while attachments are important, the individual threads far
outnumber the attachments in number, so, there will be a billion individual
folders, one for each thread (I presume).
Post by Frank Slootweg
Post by ultred ragnusen
3. Contacts (I think that's the easiest of all, with a simple export)
Since my need has to be nearly universal, there must be good solutions out
there that convert email threads to a folder tree on disk.
See above. My suggestion would be to convert the mail folders to mbox
format and then to split up the mbox-archive to individual mbox format
messages (if you indeed want on file per message).
Thunderbird has an Extension ('ImportExportTools') which can export to
mbox format, AFAICT both as single file per folder and as single file
per message. ImportExportTools also has some attachment-related options
which might be worth your while to have a look at.
As I recall, didn't Euroda do the "mbox" format?
The only thing I didn't like about Eurdora backups was that only Eudora
could read them, which is a failure in backing up if that's what happens
here.
I've never used Eudora, but many MUAs can handle or at least import
(and export) in mbox format.

I'm currently using Thunderbird (for email), so I can always go to
mbox format if need be. mbox format is just a simple extension of the
RFC 5322 message format and AFAIK one of the few or even the only
standard format.

FWIW, I come from a (real) UNIX background, so mbox format is natural
for me.
ultred ragnusen
2018-03-05 08:27:44 UTC
Permalink
Post by Frank Slootweg
've never used Eudora, but many MUAs can handle or at least import
(and export) in mbox format.
Again, maybe I'm too romantic, but I would have thought all this mail
archival stuff would have been solved already.

There are two ways that are so obvious that I can't imagine they don't
exist, one of which is what you just said, which is the monolithic mbox
format.

I assume that this mbox format is the same as we had on UNIX in the days of
old, which was a huge file, where the attachments (which came later) were i
MIME-encoded format, so the whole monolithic file was text (not binary).

The other (romantic perhaps) choice, of course, is a basic file tree, with
each thread being its own folder, and each message being a folder below the
thread, and each attachment being in the same folder as the message it was
attached to.
Post by Frank Slootweg
I'm currently using Thunderbird (for email), so I can always go to
mbox format if need be. mbox format is just a simple extension of the
RFC 5322 message format and AFAIK one of the few or even the only
standard format.
The advantage, of course, of the mbox format, is that it's a standard which
should be portable ten or twenty years from now, which is what we need to
aim for when archiving.

It's just hard to search, although I guess on Linux, grep is our friend
when it comes to a huge mbox file.
Post by Frank Slootweg
FWIW, I come from a (real) UNIX background, so mbox format is natural
for me.
I've set up sendmail, in the days of old, but that was on SunOS or
something like that, where it was a boon when we moved from "mailx" (e.g.,
mailx -s "foo" user1 < attachment.mime) to ZMail (which was great. while it
lasted).

Did you ever use ZMail? It was the /best/ (bar none) MUA on Linux for a
long time running - but it was bought - and then instantly killed - I
forget by whom.
Frank Slootweg
2018-03-03 15:01:05 UTC
Permalink
Post by ultred ragnusen
What's a good way to back up Gmail when you've reached the size limit?
http://wetakepic.com/images/2018/03/03/gmail_bck_01.jpg
What happened to your 'privacy' paranoia?

You want your privacy, yet give your mail to Google!? Something is
seriously wrong with this picture!
Post by ultred ragnusen
If you use the web for Gmail, and if you're full to the brim such that
Google has finally shut you down because you've used up all your disk
space, what is a good way to back up all your Gmail to a HDD on Windows so
that you can delete all the email and start fresh?
Theoretically you could get more space, but that would conflict with
your other requirement that everything must be 'free'.

Having said that, I support the advice to move stuff into Thunderbird
by means of IMAP or POP. At least with TB you have a chance to move
things to yet another 'mailer', should the need arise. And whatever you
do, stay away from Windows [Live] Mail.
ultred ragnusen
2018-03-03 19:54:00 UTC
Permalink
Post by Frank Slootweg
Post by ultred ragnusen
What's a good way to back up Gmail when you've reached the size limit?
http://wetakepic.com/images/2018/03/03/gmail_bck_01.jpg
What happened to your 'privacy' paranoia?
You want your privacy, yet give your mail to Google!? Something is
seriously wrong with this picture!
Your point is perfectly valid, where /any/ mail server is going to store
your mail, but where the gmail mail server does two things better than
most:
1. Spam rejection
2. Free storage

For example, it probably took more than 5 years for me to fill up this
Gmail account, from scratch.
Post by Frank Slootweg
Theoretically you could get more space, but that would conflict with
your other requirement that everything must be 'free'.
Philosophically, I will /never/ have my data held hostage at a price.
So you are completely correct that I would never pay Google to store my
mail.
Post by Frank Slootweg
Having said that, I support the advice to move stuff into Thunderbird
by means of IMAP or POP.
Currently Gmail is at the default IMAP which works because I access it from
multiple devices - but for this one-time-every-five-years task, it doesn't
matter to me whether I archive my mail to a folder tree using IMAP or POP.

Once I've archived all the mail threads to a folder tree, I'll just switch
back to the default IMAP Gmail setting on the then-empty account.
Post by Frank Slootweg
At least with TB you have a chance to move
things to yet another 'mailer', should the need arise. And whatever you
do, stay away from Windows [Live] Mail.
To be clear, the goal is to archive the 5 years of mail to a folder tree,
that is completely accessible /outside/ of any MUA.

The idea to use Thunderbird as a one-time-only MUA is fine, since
Thunderbird would simply be the "mechanism" to convert five years of mail
into a Windows folder tree of individual threads and attachments.

Once I have the folder tree of individual threads and files, I would delete
Thunderbird, since the only purpose for Thunderbird was the conversion
mechanism of Google server email to my own HDD folder trees.
ultred ragnusen
2018-03-05 10:38:32 UTC
Permalink
Post by ultred ragnusen
What's a good way to back up Gmail when you've reached the size limit?
http://wetakepic.com/images/2018/03/03/gmail_bck_01.jpg
I realize Thunderbird might be the only viable method, but TB likely won't
create the kind of folder-style archive I'm hoping to find, so I'm saving
TB as the last-choice option (if nothing else works).

For the contacts, I've already archived them into a file, as shown here.
https://support.google.com/contacts/answer/1069522 (assumes older gmail!)

0. Create a folder hierarchy, e.g.,
"mkdir .\backup\gmail\{attach,contact,mbox}\"
1. Using a web browser, log in to your Google mail account
2. Click the down arrow to the right of the "Gmail" at the top left
3. Select "Contacts" from that short pulldown menu
4. Select the "hamburger' icon at top left in the blue bar
5. Click the "More" entry to expand the available options
6. Click the "Export" selection
7. Select the "GO TO OLD VERSION" option (it's your only choice)
8. Your browser may ask you to allow popups, which you'll have to allow
9. Click on the "More" button in the middle top of the web page
10. Select "Export contacts"
11. Change the default to "All contacts"
12. Select "Which export format" of Google or Outlook CSV or vCard format.
13. Click the blue "Export" button
14. The resulting file names will be, by default,
google.csv, contacts.csv, and contacts.vcf
which you'll likely wish to rename to an archival context such as
.\backup\gmail\contact\20180305youremailaddress_{format}.{ext}

The remaining question is about one-click archival to the local HDD of all
the mail (threads + messages) and their associated attachments.

This article implies Google will /create/ that archive for you:
http://smallbusiness.chron.com/move-gmail-hard-drive-78849.html
but the "Data Tools Tab" doesn't seem to exist anymore.

1. Using a web browser, log in to your Google mail account
2. Click your picture (or the circle with your email address first letter)
3. Click the button labeled "My Account"
4. Click the "Data Tools" tab
5. Click the "Select data to download" link in the Download Data section.
6. Click the "Create an archive" button
7. Click "Select all" to deselect all items
8. Select the "Mail" item
9. Click the "Create Archive" button
10. Check the "Email me when the archive is ready" box
11. Google sends you an email when you can download your archived mail
12. Click the download link and save the archive on your hard drive.

Giving up in despair that the "data tools tab" was apparently missing,
this seems to be a Google "backup and sync" (stub) tool though...
https://google.com/drive/download

I tried this "backup and sync" tool where it's a terribly badly designed
installer in that it doesn't even /ask/ you where you want to put it nor
does it even ask if you don't want it to pollute your desktop with
/multiple/ shortcut icons.

Running the tool, it's apparently for the purpose of "choosing folders to
continuously back up to Google Drive", which is the /opposite/ of what I
want to do (in every way).

So that first and second approach was a bust.
ultred ragnusen
2018-03-05 11:18:24 UTC
Permalink
Post by ultred ragnusen
http://smallbusiness.chron.com/move-gmail-hard-drive-78849.html
but the "Data Tools Tab" doesn't seem to exist anymore.
Given my mail is stopped, at 15GB, I am forced to try /something/.

Hence, I will try this approach which archives mail in MBOX format:
https://www.wikihow.com/Back-Up-Your-Gmail-Account

As tutorials often go stale, here's my writeup of the steps taken.
1. Using a web browser, log in to your Google mail account
2. Click your picture (or the circle with your email address first letter)
3. Click the button labeled "My Account"
4. Click "Personal info & privacy"
5. Scroll down to "Control your content" and "Download your data"
6. Click "CREATE ARCHIVE"
7. Click the "SELECT NONE" button (because all archives are on by default)
8. Turn on "all mail" or you can choose to archive mail by their labels.
9. Common labels are: all, inbox, sent, archived, drafts, starred, etc.
10. Press Next once you've selected "all mail"
11. Choose an archive format of either "zip" or "tgz"
12. Choose an archive size of (1, 2, 4, 10, or 50GB per file)
13. Note that Zip files larger than 2GB will be compressed in zip64 format
14. Choose a delivery method of "Send download link via email"
15. Click on the blue CREATE ARCHIVE button
16. A message will report:
"An archive of your Mail data is currently being prepared
Please note that archives may take a long time (hours or possibly days)
to create. You will receive an email when your archive is complete."

I'll let you know how it turns out, where the result should be a single
15GB file in "MBOX" format, or a set of smaller MBOX files depending on
what you decided to archive (e.g., sent versus inbox versus important,
etc.).
ultred ragnusen
2018-03-05 19:42:31 UTC
Permalink
Post by ultred ragnusen
https://www.wikihow.com/Back-Up-Your-Gmail-Account
I followed this method:
https://support.google.com/accounts/answer/3024190

Which took me to this Google "Takeout" location:
https://takeout.google.com/settings/takeout/

Where the resulting "Takeout" mbox file is monolithic (as expected) and
where there is an "Index" file (which was not expected) as shown in this
screenshot taken just now.
Loading Image...

The downloaded file unpacks to:
Takeout > Mail, index.html > All mail Including Spam and Trash.mbox

Unfortunately, vi is taking its sweet time trying to open that mbox so I
can take a look at it in more detail...so I'll likely scrap that method of
checking it out.

What I need to do is load a good MUA, where, I guess, TB is what most of
you use, so that we can figure out if TB is reasonably compatible with a
15GB MBOX archive from Google.
ultred ragnusen
2018-03-05 20:52:32 UTC
Permalink
Post by ultred ragnusen
What I need to do is load a good MUA, where, I guess, TB is what most of
you use, so that we can figure out if TB is reasonably compatible with a
15GB MBOX archive from Google.
It seems that the Google Takeout mechanism worked, at least so far:
Loading Image...

So I am trying to get Thunderbird settings to find the Index & 15GB mbox
files now patiently sitting on my HDD as a result of the Google takeout
download.
Loading Image...
Post by ultred ragnusen
Thunderbird does it one way, and another client that also uses mbox does
it differently. Some are compatible. The mails will be there for
reading, but specifically the index, all do it differently. Maybe you
will not know which post are read or replied to, maybe all will show as
"new".
I'm on Windows as the original problem was how to archive the Gmail, so I
might have to boot to Ubuntu 17.10 but at the moment, I'm trying to get the
Windows Thunderbird to find the Google Takeout mbox file and index.

TB has a "Local directory" setting, which defaults to:
C:\Users\ultred\AppData\Roaming\Thunderbird\Profiles\xxx.default\Mail\Local Folders

I should have experimented with a /smaller/ testcase as I'm not sure if the
Local Directory should be
C:\data\backup\gmail\ultred\mbox\Takeout\ (where index.html lies)
or C:\data\backup\gmail\ultred\mbox\Takeout\Mail (where the mbox lies)

Trial and error is a wonderful thing (on a smaller file!), but Takeout
didn't work so it's Mail that Thunderbird needs as its local folder.
Local Folder = C:\data\backup\gmail\ultred\mbox\Takeout\Mail
aka Local Directory = C:\data\backup\gmail\ultred\mbox\Takeout\Mail
Loading Image...

And where the message at the bottom of TB is:
"Building summary file for All mail Including Spam and Trash.mbox..."
And where the "mailbox" seems to be this:
mailbox:///C:/data/backup/gmail/ultred/mbox/Takeout/Mail/All mail Including Spam and Trash.mbox

Which seems to have worked on this intended-to-be-humorous test message:
Loading Image...
ultred ragnusen
2018-03-06 00:58:32 UTC
Permalink
Post by ultred ragnusen
http://wetakepic.com/images/2018/03/05/gmail_bck_06.jpg
One procedural faux pas I just learned the hard way, which nobody mentioned
explicitly, I don't think (or maybe I missed it), is that you can't just
use Thunderbird to download mail by setting POP in the Thunderbird
settings.

Gmail won't tell you that is the problem, and, in fact, Gmail from
Thunderbird will insist the login/password is wrong, but the real problem
is that you, apparently, have to change from IMAP to POP on the server
first, and /then/ you can download the mail via POP.

I wonder though, for safety, if you can download the mail on IMAP and
accomplish the same thing as downloading the mail with POP?

Seems to me both are equivalent on the desktop side of things if it's a
brand new installation of Thunderbird.

Can anyone concur?
Q: From the desktop archival standpoint, doesn't downloading all messages
to the desktop via IMAP or POP end up with exactly the same result on the
desktop file system?
Paul
2018-03-06 02:45:21 UTC
Permalink
Post by ultred ragnusen
Post by ultred ragnusen
http://wetakepic.com/images/2018/03/05/gmail_bck_06.jpg
One procedural faux pas I just learned the hard way, which nobody mentioned
explicitly, I don't think (or maybe I missed it), is that you can't just
use Thunderbird to download mail by setting POP in the Thunderbird
settings.
Gmail won't tell you that is the problem, and, in fact, Gmail from
Thunderbird will insist the login/password is wrong, but the real problem
is that you, apparently, have to change from IMAP to POP on the server
first, and /then/ you can download the mail via POP.
I wonder though, for safety, if you can download the mail on IMAP and
accomplish the same thing as downloading the mail with POP?
Seems to me both are equivalent on the desktop side of things if it's a
brand new installation of Thunderbird.
Can anyone concur?
Q: From the desktop archival standpoint, doesn't downloading all messages
to the desktop via IMAP or POP end up with exactly the same result on the
desktop file system?
You're playing with fire though, right ?

POP3 has the capability of deleting each mail transferred.
That means you have *one* opportunity to get it right.
If there is an option to leave the mail on the server,
you end up putting a great deal of trust in that setting.

IMAP shares mail over multiple client machines.
So I don't think it's opt-out driven. It probably
needs to have the "delete after download" bit turned on.
You have to "opt-in" to invite disaster.

In any case, I think there is a moderate amount of danger
here, of losing the whole thing. And you should be very
careful about your technique.

Perhaps you should be creating a test account, testing
your procedures for correctness first, then try it on
the real account. Doing experiments with a real account
is just asking for trouble.

Paul
ultred ragnusen
2018-03-06 03:13:53 UTC
Permalink
Post by Paul
You're playing with fire though, right ?
I agree with you Paul.

You hit upon a good analogy, which is that only a fool would use
Thunderbird to download their messages for the purpose of archival, simply
because it's an extremely confusing complex software for such a simple
task.

For example, I can't count the number of times TB has been downloading
messages, where there are all sorts of sync settings and other chances to
destroy your data the moment you delete it off the server (which is the
goal, after all - since Gmail is filled up).

So, having tried the Google Takeout method, and having tried the
download-with-Thunderbird method, I know the answer, emphatically, which is
that, unpless you're an expert with Thunderbird, you're playing with fire
to use Thunderbird as your archival mechanism.

The /best/ you can hope for is to attempt to download as much as you can,
and then archive the directories, which, by default, are in the dumbest
places you can possibly imagine (by default).
Post by Paul
POP3 has the capability of deleting each mail transferred.
That means you have *one* opportunity to get it right.
If there is an option to leave the mail on the server,
you end up putting a great deal of trust in that setting.
I agree with you Paul. I was only asking about Pop3 because people
mentioned it alongside IMAP4, where only a fool would use Thunderbird for
the stated goal of backing up mail prior to deletion.

It's just too risky unless ... unless you're an expert in Thunderbird -
which has so many idiotic settings it's not even funny.

I mean, WTF, why does it force me to argue with it to NOT link NNTP (of all
things, which is nothing like email) and what idiotic programmer decided on
those moronic default directory trees as just two examples of the
non-intuitive nature of the tool.

It has options for cookies, for heavens sake, that you have to deal with,
in a mail user agent, of all things.
Post by Paul
IMAP shares mail over multiple client machines.
So I don't think it's opt-out driven. It probably
needs to have the "delete after download" bit turned on.
You have to "opt-in" to invite disaster.
My plan is simple, but I'm convinced, unless you're an expert already in
Thunderbird, it's the /last/ method I'd suggest someone use to archive mail
prior to deletion.

The google method is far more foolproof.
Post by Paul
In any case, I think there is a moderate amount of danger
here, of losing the whole thing. And you should be very
careful about your technique.
Actually, I planned ahead. Remember, I have the Google archive already.
But I do agree with you.

Unless you're /already/ an expert in the intricacies of Thunderbird, it's
the /last/ thing I'd recommend for someone to archive mail prior to
deleting it off the server.
Post by Paul
Perhaps you should be creating a test account, testing
your procedures for correctness first, then try it on
the real account. Doing experiments with a real account
is just asking for trouble.
Again, I'm fine because I have the Google archive.

My plan is simple, but I've been playing with the absolutely insane user
interface of Thunderbird for about four or five hours - and it's just like
it's written by a bunch of confused kindergarten kids who got a job as
programmers. It's that bad.

Remember, I had used Thunderbird a lot, when it first came out, and when
Zmail died, but Jesus, did they fuck it up royally. It's a mess. Ten times
more tool than you need. Defaults in the craziest locations.

I mean, do we really need the idiotic browser profile directory tree that
Mozilla fomented upon us years ago, that only coders, who know nothing
about decent design, could imagine?

As just one example of the idiocy that is Thunderbird, look at this
/default/ folder location for heaven's sake!
C:\Users\ultred\AppData\Roaming\Thunderbird\Profiles\83ujpegn.default\Mail\Local Folders

That's just stupid. Nobody with a brain needs to dump mail 10 levels deep!
(Of course it can be changed, but that's my point. And yours. It's so
horridly complex that, unless you're an expert, you'll fuck it up when you
can't afford to fuck it up).

And what's with the idiotic organization of menus?
Why do you need "Tools > Options" to be separate from "Tools > Account
Settings". Did a kindergarten kid design this user interface?

Anyway, it's horrid. It can be learned. But it's horrid. It can be tamed.
But it's horrid. It can be organized. But it's horrid.

In short, I agree with you. Only a fool would use this method alone to
archive mail, if they weren't /already/ an expert in Thunderbird.

Period. It's that bad of a design.
ultred ragnusen
2018-03-06 03:52:08 UTC
Permalink
Post by ultred ragnusen
I mean, do we really need the idiotic browser profile directory tree that
Mozilla fomented upon us years ago, that only coders, who know nothing
about decent design, could imagine?
When your goal is to archive /important/ files, you can't believe the
amount of utter crap that is in the profile directory of Thunderbird.

It's as insane as a Firefox profile directory, with sql files, and json
files, and ini files and msf files and mab files and db files and dat files
and db files, etc. none of which are useful for the purpose of archival.

It's like a hundred coders built the thing, each dumping his crap in a
folder with no rhyme or reason attached to the organizational structure.
http://kb.mozillazine.org/Profile_folder_-_Thunderbird

Jesus. It's horrid. Utterly horrid. As bad as Firefox is.
Anyway, Thunderbird is the crap that it is and I can't change that.

So my simple plan is, well, is simple:

1. Download the mail (which isn't as simple as you think because /every/
time I run Thunderbird, the numbers change, and there is nothing happening
to my email - so I'm confused why it takes five hours to simply download
the mail and then it does it again and again and again and again.

2. Then delete everything /except/ the email and zip it up.

3. Then re-install Thunderbird, and try to re-populate it with the mail,
which seems to be in one location (mostly) but which is, idiotically,
scattered in multiple folders ten levels deep (which is again, just
insane).

The mail is scattered, but /most/ of the mail is in this one mbox, which is
1/10th the size it's supposed to be though... and the size keeps changing,
even though nothing is happening in my Gmail account.
C:\Users\ultred\AppData\Roaming\Thunderbird\Profiles\idiotic-name\ImapMail\imap.gmail.com\INBOX-1
--
PS: If I sound frustrated, it's because I am. Thunderbird is, IMHO, a POS
architecture that isn't a MUA but a suite of crap pile up upon a browser
architecture.
Frank Slootweg
2018-03-06 16:53:48 UTC
Permalink
ultred ragnusen <***@ragnusen.com> wrote:
[...]
Post by ultred ragnusen
PS: If I sound frustrated, it's because I am. Thunderbird is, IMHO, a POS
architecture that isn't a MUA but a suite of crap pile up upon a browser
architecture.
Thanks for the laughs!

Just because you lack any basic understanding and expertise in the
subject matter, something is "a POS architecture and a suite of crap
pile"! Makes perfect sense.

So I gather Thunderbird will be joining those frail USB drives, won't
it!?

Oh wait! Crap, *now* I get it! Too many clicks! Duh!
ultred ragnusen
2018-03-06 18:30:54 UTC
Permalink
Post by Frank Slootweg
Just because you lack any basic understanding and expertise in the
subject matter, something is "a POS architecture and a suite of crap
pile"! Makes perfect sense.
I only tell the facts as I see them in terms of the solution requested.

A. Someone asks what's a good way to back up Gmail safely.
B. The answer of Thunderbird works - but is inherently more complex than..
C. The answer of Google Takeout is vastly simpler & safer, and,
D. The answer of UpSafe, kindly provided by Zaidy036, is even simpler.

We all appreciate, Frank, that you did provide a potential answer, that of
the Thunderbird extension "ImportExportTools"), which has yet to be tested
and reported upon, since other more-direct tests have obviously been taken.

We'll get to your suggestion soon, we hope, so that the tribal knowledge of
this ng will, eventually, contain the best practices for the given
question.

Thanks for keeping on technical target with the goal of a solution suitable
for all to try and benefit from in the tribal knowledge archives.
Jonathan N. Little
2018-03-06 03:59:09 UTC
Permalink
Post by ultred ragnusen
Post by ultred ragnusen
http://wetakepic.com/images/2018/03/05/gmail_bck_06.jpg
One procedural faux pas I just learned the hard way, which nobody mentioned
explicitly, I don't think (or maybe I missed it), is that you can't just
use Thunderbird to download mail by setting POP in the Thunderbird
settings >
Gmail won't tell you that is the problem, and, in fact, Gmail from
Thunderbird will insist the login/password is wrong, but the real problem
is that you, apparently, have to change from IMAP to POP on the server
first, and /then/ you can download the mail via POP.
Yes, error messages minimal by default. You can set it though

<https://wiki.mozilla.org/MailNews:Logging#Windows>
Post by ultred ragnusen
I wonder though, for safety, if you can download the mail on IMAP and
accomplish the same thing as downloading the mail with POP?
Similar, yes. Just with IMAP you will have to manually delete messages
on server and setup a filter to move mail to local folders. Whereas POP
automatically downloads and you do have the option to opt-in to leave
messages on the server. A bit of an inverse of defaults.
Post by ultred ragnusen
Seems to me both are equivalent on the desktop side of things if it's a
brand new installation of Thunderbird.
Can anyone concur?
Q: From the desktop archival standpoint, doesn't downloading all messages
to the desktop via IMAP or POP end up with exactly the same result on the
desktop file system?
--
Take care,

Jonathan
-------------------
LITTLE WORKS STUDIO
http://www.LittleWorksStudio.com
ultred ragnusen
2018-03-06 14:56:45 UTC
Permalink
Post by Jonathan N. Little
Post by ultred ragnusen
I wonder though, for safety, if you can download the mail on IMAP and
accomplish the same thing as downloading the mail with POP?
Similar, yes. Just with IMAP you will have to manually delete messages
on server and setup a filter to move mail to local folders. Whereas POP
automatically downloads and you do have the option to opt-in to leave
messages on the server. A bit of an inverse of defaults.
That's sage advice. Thanks Jonathan for answering the quetion, especially
as I may have appeared frustrated (because I was) that the suggestion of
Thunderbird is, in actuality, a /terrible/ suggestion in light of the
simplicity and safety of the Google Takeout answer.

Admittedly, if you /already/ are familiar with the machinations of
Thunderbird, then it's a fine answer - but it's so fraught with peril for a
noob that my assessment is that it's just not viable for the stated purpose
of archival and deleted.

As just one example of the complexities, my Thunderbird, which I left
"running" all last night (checking at ten minute intervals), has /still/
not downloaded all the mail that it on the Gmail server. Why? Hell if I
know why. How hard can it be to tell Thunderbird to 'check mail' for
heaven's sake.

But even now, the INBOX, which we know to be huge, is still only a measly
3GB, and climbing ever so slowly. It's not the net speed - it's some
intricacy in Thunderbird - or - just as likely - some noob incorrect
settting or assessment on my part.

But, since the mail is intended to be /deleted/ off the server, these
worries don't make for Thunderbird being even a good suggestion for the
stated purpose. It's bad. Bad. Bad. Bad. Bad. It's just wrong.

The /safe/ and /simple/ and /reliable/ answer, so far, to backing up Gmail,
is to ask Gmail Takeout to send it to you for download, either in one fell
swoop or in labeled chunks.

Thanks for everyone's help. You won't know how to do this until you do it
with data you care about. It's sort of like raising kids versus grandkids,
where you make more mistakes when you are closer to the daily results.
ultred ragnusen
2018-03-08 15:10:36 UTC
Permalink
Post by ultred ragnusen
Thanks for everyone's help. You won't know how to do this until you do it
with data you care about. It's sort of like raising kids versus grandkids,
where you make more mistakes when you are closer to the daily results.
BTW, I've been continuing to test methods for the tribal knowledge archives
as to what's the best (or better) method of archiving Gmail, where the
Linux GetMail method was tried.

After 24 hours, and three failed attempts, it worked to download both:
a. A mbox file
b. Individual messages

Here's a screenshot, where the main unanswered anomaly is why a GetMail
mbox archive file is ~19GB while both the Thunderbird & Gmail archive mbox
files were around 15GB.

Loading Image...
Paul
2018-03-08 17:57:04 UTC
Permalink
Post by ultred ragnusen
Post by ultred ragnusen
Thanks for everyone's help. You won't know how to do this until you do it
with data you care about. It's sort of like raising kids versus grandkids,
where you make more mistakes when you are closer to the daily results.
BTW, I've been continuing to test methods for the tribal knowledge archives
as to what's the best (or better) method of archiving Gmail, where the
Linux GetMail method was tried.
a. A mbox file
b. Individual messages
Here's a screenshot, where the main unanswered anomaly is why a GetMail
mbox archive file is ~19GB while both the Thunderbird & Gmail archive mbox
files were around 15GB.
http://i.cubeupload.com/t5pSer.jpg
On Windows:

https://mh-nexus.de/en/hxd/

It's simply more practical than a proper text tool.

Maybe the line endings on the file are different, but
that probably doesn't account for that large a difference.

*******

I got fed up with using the usual approach to text
the other day, and ended up using "less" in Win10 Bash
shell to review a big file. Not very convenient, but
less annoying than waiting for some text editor to
load a file.

Paul

Zaidy036
2018-03-05 09:57:33 UTC
Permalink
Post by ultred ragnusen
What's a good way to back up Gmail when you've reached the size limit?
http://wetakepic.com/images/2018/03/03/gmail_bck_01.jpg
If you use the web for Gmail, and if you're full to the brim such that
Google has finally shut you down because you've used up all your disk
space, what is a good way to back up all your Gmail to a HDD on Windows so
that you can delete all the email and start fresh?
Gmail is free. Open another gmail address and then set the old one to
forward new emails to that account.
--
Zaidy036
ultred ragnusen
2018-03-05 18:34:36 UTC
Permalink
Post by Zaidy036
Gmail is free. Open another gmail address and then set the old one to
forward new emails to that account.
That's an option where it's easy to create a Gmail address and where mail
forwarders are out there so it's certainly another solution to the problem.

Each solution has its pros and cons, of course, as it makes searching
miserable since you have to search /two/ email addresses if you don't know
where the message or attachment you're looking for lies.

But it's certainly an option, which, we should note, can be done
concurrently with the method I found from Google, which is to simply ask
Google periodically to mail you a link to all your files in one huge mbox
format.
Loading Image...

Here's a test I suggest others try as mine has been taking /hours/ to
download so I don't suggest your first test be on a 10GB file.

1. Log into Gmail and label a handful of messages as, say, "test".
2. Go here and archive just the messages labeled "test"
3. https://takeout.google.com/settings/takeout/

Google will send you a message exactly like this one I received today:
Loading Image...
ultred ragnusen
2018-03-05 18:51:11 UTC
Permalink
Post by ultred ragnusen
Post by Zaidy036
Gmail is free. Open another gmail address and then set the old one to
forward new emails to that account.
That's an option where it's easy to create a Gmail address and where mail
forwarders are out there so it's certainly another solution to the problem.
Actually ... wait ... I think you have a /great/ idea!
Scratch what I just said about it being just another option!

What's a bad idea is:
a. Set up a new Gmail account,
b. Forward all new mail to the new Gmail account
c. Use the old account only when you need to search for messages.

What is a GREAT idea (that I didn't even think of) is:
a. Set up a new Gmail account
b. Load the old archive MBOX into the new Gmail account
c. Use the new account only when you need to search for messages.

Wow. It's GREAT when great minds work together!
I /like/ this idea!

Pros:
- You /know/ the mail will fit because it came from Gmail
- You know it's compatible because it came from Gmail
- You know it's searchable because it came from Gmail

Cons:
- Not many. You have to remember a new email address & passwd
- But more importantly, if rarely used, you have to save the challenges
- As Gmail makes your life a living hell with those challenges

Yep. I /like/ your idea! Why didn't anyone else suggest it!
I love it. It all makes sense. It really does.

A. You archive out of mail account 1.
B. You load that archive back in, into mail account 2.
C. You search using mail account 2.

This is a great idea! Thanks!
Zaidy036
2018-03-06 06:34:57 UTC
Permalink
Post by ultred ragnusen
What's a good way to back up Gmail when you've reached the size limit?
http://wetakepic.com/images/2018/03/03/gmail_bck_01.jpg
If you use the web for Gmail, and if you're full to the brim such that
Google has finally shut you down because you've used up all your disk
space, what is a good way to back up all your Gmail to a HDD on Windows so
that you can delete all the email and start fresh?
Free program <https://www.upsafe.com/free-gmail-backup/>
--
Zaidy036
ultred ragnusen
2018-03-06 15:23:41 UTC
Permalink
Post by Zaidy036
Post by ultred ragnusen
What's a good way to back up Gmail when you've reached the size limit?
Free program <https://www.upsafe.com/free-gmail-backup/>
Thanks for adding value to the question, for the tribal knowledge to
benefit from, of how best (safest and easiest and most effective) way to
back up Gmail.

Loading Image...

UpSafe seems to be a cloud service (even though they don't use that word,
probably for marketing reasons) backs up all your gmail, on the side, while
you use Gmail.

You download, install, and run the UpSafe software, and you hit an "accept"
button, where the UpSafe folks never see your Gmail credentials.

What I think is useful, and interesting, about UpSafe, is for someone like
me, who gets locked out of Gmail all the time (mostly due to VPN and
changing my MUA and using a different browser fingerprint every day, etc.),
although it's read only (AFAICT).

UpSafe insists that it's free, so, one always has to wonder about the
catch, where I presume the catch is that they want a relationship with you
to sell other things (which is fine).

Running the UpSafe setup.exe, I see it's a badly written installer right
off the bat, which doesn't even ask where to go, but it's quick and done in
a second after you agree to their terms.

It opens with an easily seen big red "sign in with Google" button, and
then, even when on VPN, it logs into Gmail asking five permissions:
1. Read, send, delete, and manage your email
2. Know who you are on Google
3. View your email address
4. View and modify but not delete your email
5. Insert mail into your mailbox

Once you accept those terms, there's a note that you can't have a history
of backups or backup by schedule without "registering" UpSafe, for free,
(whatever that means to "register"), but there's also a big green very
inviting button to "Start backup".

Only after I hit that big inviting button, I noticed it didn't ask me
/anything/ so I decided to hunt around while it's backing up.

There's a link for "Backup options", and "Archiving options" (what's the
difference?) [see later for that answer]

Looking at "Backup options", there are settings signing out automatically
when the backup is done, and a decent set of clickbox basic filters such as
mail from a certain person or a given subject, etc., so it's not an
all-or-nothing affair (which is good).

There is an "Archiving" tab (which doesn't belong in backup options if it
was a link outside of backup options, but hey, the GUI design is still
pretty good, even with that classic faux pas) where the difference between
backup and archiving seems to be that archiving /deletes/ the mail after
backup. Hmmm... OK. That makes sense.

Rooting around for /where/ UpSafe is putting my gmail, I find that there is
a "Storage" tab where you get to choose /where/ the backup goes, where, by
default, mine is grayed out as it's working, and going into:
C:\Users\ultred\AppData\Local\UpSafe\Gmail Backup\Data\

Who knows what format the end result will be in, but let's hope that it's
not a /proprietary/ format, as the whole goal is to save for use ten years
from now.

This may indeed be a /great/ solution ... if ... if the results are usable.
Time will tell... (as will I). :)
http://i.cubeupload.com/bqCs0i.jpg
ultred ragnusen
2018-03-06 16:43:14 UTC
Permalink
Post by ultred ragnusen
This may indeed be a /great/ solution ... if ... if the results are usable.
Time will tell... (as will I). :)
http://i.cubeupload.com/bqCs0i.jpg
Hmmmm.... UpSafe failed an hour into the program where a look into the log
file it saved says, ad infintum...
C:\Users\ultred\AppData\Local\UpSafe\GMail Backup\ logs\Service.log
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at UpSafe.MailBackup.Service.Retriever.MailRetriever.<ProcessMessageInternalAsync>d__38.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at UpSafe.MailBackup.Service.Retriever.MailRetriever.<ProcessMessageAsync>d__37.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at UpSafe.MailBackup.Service.Retriever.MailRetriever.<RetrieveAndProcessMessagesAsync>d__36.MoveNext()
etc.

Nonetheless, UpSafe did save /something/ where we look to see what at
C:\Users\ultred\AppData\Local\UpSafe\GMail Backup\ Data\
Where I see three files:
ServiceData.sdf (binary)
ServiceData_2011.zip
ServiceData_2012.zip

Looking inside those zip files, we find a hierarchy.
ServiceData_2011\***@gmail.com\[Gmail]\All Mail\(filenames).eml

Hmmmm.... what are these zillions of eml files?

Sorting on size, and opening the smallest EML files in vim, I see each
appears to be its own individual email message, complete with header and
body.

Opening the big ones in vim, it's the same thing, only with mime-encoded
attachments, so, it's pretty clear, the EML files are the individual
emails.

The naming system seems to be algorithmic,
2011-05-06T23_1Re_ Stupid Gmai_Jane L_ _johng0845e976-1bb8-4f0d-94db-6e4ffef90eab.eml
2011-05-06T23_5Re_ Stupid_Gmai_Donna Apt Johncb293622-37d7-4c38-a362-17043c1e22d0.eml
2011-05-07T01_2Check out sampsSierraCirc_aol.45a35c22-6541-40ff-a9bc-c646e0a14021.eml
2011-05-07T01_2Fwd_ Did you seSierraCirc_aol.427f328d-37ab-4537-8ea1-4e9bcfb50e8a.eml
2011-05-07T01_2Fwd_ Your Orde.SierraCirc_aol.de50cf38-f21a-44c2-9577-2e9255a4014e.eml

I'll try again.
ultred ragnusen
2018-03-08 15:26:52 UTC
Permalink
Post by ultred ragnusen
What's a good way to back up Gmail when you've reached the size limit?
http://wetakepic.com/images/2018/03/03/gmail_bck_01.jpg
After sucessfully using the GetMail method on Linux
http://i.cubeupload.com/t5pSer.jpg

I also went back to the Gmail Takeout archival method of asking for
archives in 2GB chunks instead of in a 15GB chunk, so that I could archive
the mbox files to DVD.

One caveat that I learned that I wish to impart upon the rest of you for
the tribal knowledge archive is that the Gmail Takeout archival method will
allow you to select just the inbox or just the sent or any particular label
such as important or starred that you have set, and it will do all mail,
but ...

But if you try too many options (as I did), Google will lock you out,
telling you that you've asked for too many archives in too short a period
of time.

Also, there's a concept in the Google Takeout archival method of "Archived"
mails, and "IMAP/Archived" mails, whatever that difference happens to be.

So plan ahead when you use the Gmail Takeout archival method.

Here's one suggestion:
a. Use labels daily so that you can archive more easily
b. When you archive, select 2GB chunks which can go 2 to a DVD (we hope)
c. Expect the process to take days elapsed time

If others have useful knowledge gleaned from using the Google Takeout
archival method, please let us all benefit from your experience.
Loading...