Offlineimap - can't delete files from Gmail 'All Mail' folder

firecat53 · 2013-11-16 22:21:28

I found one reference to this issue in an offlineimap mailing list post, but no resolution:

Using Mutt, if I delete messages from the Gmail 'All Mail' folder (or 'archive' as it's mapped to on my local machine) by copying them to the 'trash' folder, the first sync appears to work, but then they get copied back down from Gmail to my laptop on the next sync after that. Driving me a little batty....

Copying messages to the 'trash' folder works fine to 'delete' a message for any other label except the 'All Mail' folder.

Am I missing something obvious here??

Thanks,
Scott

EDIT: After playing with it some more, it appears that the emails that can't be deleted all have duplicates within the 'All Mail' folder. Deleting one or both emails from the folder (copying to 'trash' or even just removing from the folder) still results in them _not_ actually being deleted from Gmail and reappearing on the next sync. This kind of leads me to suspect an offlineimap bug...

EDIT #2: Please help. I think I'm going to scream. I can't get rid of this @#$@#$.

~/.offlinemaprc

[general]
metadata = ~/.offlineimap
accounts = firecat53
maxsyncaccounts = 2
ui = quiet
fsync = false

ignore-readonly = no

[Account firecat53]
localrepository = firecat53_local
remoterepository = firecat53_remote
autorefresh = 0
status_backend = sqlite

[Repository firecat53_local]
type = Maildir
localfolders = ~/mail/user1
sep = .
restoreatime = no
maxconnections = 5
nametrans = lambda folder: re.sub('spam', '[Gmail].Spam',
                           re.sub('drafts', '[Gmail].Drafts',
                           re.sub('sent', '[Gmail].Sent Mail',
                           re.sub('flagged', '[Gmail].Starred',
                           re.sub('trash', '[Gmail].Trash',
                           re.sub('archive', '[Gmail].All Mail', folder))))))

Last edited by firecat53 (2013-11-21 19:00:02)

skottish · 2013-11-17 17:48:16

I'm glad that you brought this up as I'm setting up Mutt again now and am using stuff from some of your posts. From https://support.google.com/mail/answer/78892?hl=en

Google wrote:

Deleting:
Do not save deleted messages on the server. Messages that are deleted from an IMAP folder (except for those in [Gmail]/Spam or [Gmail]/Trash) only have that label removed and still exist in All Mail. Hence, your client doesn't need to store an extra copy of a deleted message.
Do not save deleted messages to your [Gmail]/Trash folder because this will delete a message in all folders.
Do not save deleted messages to your [Gmail]/All Mail folder as some clients will try to empty this folder and ultimately fail. This can lead to delayed mail access or excessive battery consumption on a mobile device.

I'm not sure what this all means quite yet, but I'm working on it.

Last edited by skottish (2013-11-17 17:49:21)

WonderWoofy · 2013-11-17 17:56:36

It seems to work fine for me to simply not use the trash folder at all. I use mutt to delete the mail (with 'd') then I sync it (I think the default it '$'). So the next time it actually cycles through the imap sync, it picks up these changes. I'm not at all user if this is how I am supposed to be using gmail's imap, but it seems to work.

Oh, and I don't use offlineimap, I use mbsync, which seems far superior to offlineimap. I used offlineimap for about a year, and while it did in fact work pretty well, I could always tell it was syncing because it would cause my processor (i5 3210m) to 17-20% and my fans would kick on. This does not happen with mbsync/isync as the processor only jumps a couple percentage points and no detectable thermal change can be felt on my lap.

skottish · 2013-11-17 18:03:56

WonderWoofy wrote:

It seems to work fine for me to simply not use the trash folder at all. I use mutt to delete the mail (with 'd') then I sync it (I think the default it '$'). So the next time it actually cycles through the imap sync, it picks up these changes. I'm not at all user if this is how I am supposed to be using gmail's imap, but it seems to work.

Confirmed.

WonderWoofy wrote:

Oh, and I don't use offlineimap, I use mbsync, which seems far superior to offlineimap. I used offlineimap for about a year, and while it did in fact work pretty well, I could always tell it was syncing because it would cause my processor (i5 3210m) to 17-20% and my fans would kick on. This does not happen with mbsync/isync as the processor only jumps a couple percentage points and no detectable thermal change can be felt on my lap.

This is good to know. I'm in the early stages of (re)setting up mutt and offlineimap has always seemed to be buggy to me. But, like many Archers, brisbin33's mutt pages are where I started.

Last edited by skottish (2013-11-17 18:04:46)

firecat53 · 2013-11-17 18:50:59

WonderWoofy wrote:

It seems to work fine for me to simply not use the trash folder at all. I use mutt to delete the mail (with 'd') then I sync it (I think the default it '$'). So the next time it actually cycles through the imap sync, it picks up these changes. I'm not at all user if this is how I am supposed to be using gmail's imap, but it seems to work.

But can you do this in the 'All Mail' folder? It works normally for me, too...'archive' a message by 'd' from the particular folder view, or delete (copy to trash) with the 'D' macro:

macro index,pager D "<save-message>+user1/trash<enter>"

except from the 'All Mail' folder...and especially if it's a duplicate message! Which is the issue. I ended up with a bunch of duplicates (mostly from cronjobs emailed to myself via Gmail's SMTP servers).

WonderWoofy wrote:

Oh, and I don't use offlineimap, I use mbsync, which seems far superior to offlineimap. I used offlineimap for about a year, and while it did in fact work pretty well, I could always tell it was syncing because it would cause my processor (i5 3210m) to 17-20% and my fans would kick on. This does not happen with mbsync/isync as the processor only jumps a couple percentage points and no detectable thermal change can be felt on my lap.

I haven't tried mbsync yet, but I got around the CPU usage issue you're talking about by only syncing my INBOX and news folders every couple of minutes and doing a full sync every 15 min. Makes a huge difference!

*/2 * * * *            ID=offlineimap_inbox    /usr/bin/offlineimap -f INBOX,News &> /dev/null
*/15 * * * *   ID=offlineimap  /usr/bin/offlineimap &> /dev/null

Scott

WonderWoofy · 2013-11-17 20:14:00

skottish wrote:

<snip>
But, like many Archers, brisbin33's mutt pages are where I started.

Indeed, this is where I started as well, as he did an amazing job documenting and explaining his setup.

@firecat77, I also set up offlineimap to sync only inbox and spam, then would do everything ever couple of hours. But even then, I would still see a drastic jump in cpu usage, though it only lasted a moment with this setup. I also tried to run offlineimap as a daemon, but that started some >75 threads, and seemed to be hugely inefficient. Since I have a laptop, this actually makes a difference in battery life.

I highly recommend just setting up isync/mbsync over going through the whole offlineimap setup. Here is what I use for one of my gmail configs (I changed the account name to 'gmail' to make it more generic, and masked my email as pw for obvious reasons). The Arch wiki is also a pretty good resource for setting this up. It optionally gives much finer grained control over what gets synced in which direction, but I don't use that, as I want everything to go in both directions, which makes it work just like offlineimap.

firecat53 · 2013-11-21 00:05:53

Well, I think after battling with my email for about 5 days now, I've finally got it all sorted out. I did end up switching to mbsync....which was painful in it's own right...and I still seem to have a minor outstanding bug/misconfiguration. I'm actually not convinced it's that much faster than offlineimap, but I'll give it a try for awhile. I'm attempting to use the 'AltMap' Maildir store option because I was having issues with conflicting UID's when copying mails bewteen tags/folders. Either I didn't have that issue with offlineimap, or I just never noticed it before...either is equally possible! I was also able to get Gmail's subfolder feature working with mbsync, which a lot of people have had trouble with (using the 'Flatten' directive). If it all seems stable after a few weeks using it, I'll add my configs to the Wiki (I can post them here if anyone else wants to test them).

I think what finally solved the problem of not being able to delete mail and duplicates from the 'All Mail' folder via Mutt/{mbsync,offlineimap} was setting on the Gmail Site 'Settings->Forwarding and POP/IMAP->Auto-Expunge Off and When I mark a message in IMAP as deleted: Immediately delete the message forever'. Hit 'd' to remove them from the 'All Mail' folder in Mutt, but don't try to save them to the 'trash' folder like other mails.

Funny how checking one simple thing can lead right down the rabbit hole....

Scott

WonderWoofy · 2013-11-21 01:51:47

Oh, I didn't mean to try to lead you into thinking that mbsync would be faster. I find that it is probably about the same speed at actually syncing the mail. But offlineimap is written in python, and mbsync in C, so it is just far less taxing on the system itself. Offlineimap also does this funky thing where to starts to take up mass amounts of RAM as a given mailbox gets larger. So if your 'All Mail' box (tag, label, whatever) is huge, then it just seems to use far far more resources to achieve the same thing. I have a laptop, as my primary machine, so this means that using offlineimap has potential negative effects on battery life if I set it to sync too often.

firecat53 · 2013-11-21 02:55:05

Yeah, I've been running offlineimap for years and I do notice some of that, especially on older machines. However I did something similar to you and just synced my inbox (and rss2email folder) every couple of minutes and the rest every 15. I'm continuing that tradition with mbsync since I've got 2 accounts and a fair number of folders... Just to reduce the overall runtime and network traffic. Maybe unnecessary now, but I'll have to keep an eye on it.

Scott

WonderWoofy · 2013-11-21 03:21:40

I used to do a similar frequency, I think I had something like 4 minutes for my inbox and 20 minutes for everythign else. But recently I have been doing the inbox (and spam since it is nearly empty, and sometimes good stuff ends up there) every 10 minutes, and I pushed back the everything sync to 3 hours. So far, I haven't really missed the frequency. From time to time I know an email should be in my inbox, so I manually sync. I have that set up to a keybind in mutt. mbsync seems to handle it very well if it is mid sync and I try to do it manually.

skottish · 2013-11-21 17:32:59

firecat53 wrote:

I'm attempting to use the 'AltMap' Maildir store option because I was having issues with conflicting UID's when copying mails bewteen tags/folders. Either I didn't have that issue with offlineimap, or I just never noticed it before...

I'm also seeing this with mbsync and things are simply weird using it (stuff deleted out of All Mail isn't being deleted out of INBOX or Sent Mail). There's a workaround on their website, but right now I'm not feeling very trusting. I've gone back to offlineimap and it seems to both be working properly and no errors are being reported on the console. This is on my workstation though, so I'm not concerned about laptop related issues.

firecat53 · 2013-11-21 19:03:49

Yeah, well I'm about there now too. I marked this as unsolved again. I've got laptop and server that _should_ be seeing the same messages, but what I'm seeing on each is different from each other _and_ different from Gmail's site :-/ It seems to be ok for smaller folders/tags, but with big tags/folders like 'All Mail' and 'Sent' it seems to be off. Now I don't know if this is a Gmail issue with their IMAP implementation and the way I'm using it, or if it's an mbsync issue or all of the above. I'm about ready to delete all my mail from Gmail and start over with what I have on my laptop, which seems to be correct....

Grrrrr

skottish · 2013-11-21 21:00:17

This is preliminary and will probably go to shit right after I write this, but it seems like that once I get All Mail synchronized with all other folders from the web interface that it works from there. But, I am deleting to the on-line trash first before I use mutt's 'd'. Otherwise it seems to leave the tags in place and magically reappear at some random time. Hopefully things just work from here. I'm hopeful.

firecat53 · 2013-11-22 00:44:13

I think part of my issue may be my use of the 'archivemail' utility. I've been using it to keep only the last 180 days of mail active and storing the rest in a gzipped mbox file (that archivemail produces). I wrote a script that

1. Archives mail in the 'All Mail' directory to the mbox file older than 180 days
2. Deletes mail from each folder/tag older than 180 days
3. Deletes mail from spam and trash folders older than 20 days.

I think I screwed up the order of that -- you can't delete/remove something from the archive folder if it still exists in another tag/folder. So it _should_ be:

1. Deletes mail from each folder/tag older than 180 days
2. Archives mail in the 'All Mail' directory to the mbox file older than 180 days
3. Deletes mail from spam and trash folders older than 20 days.

In addition, in order to delete something from the 'All Mail' folder, you have to copy it to the 'trash' folder. Well, archivemail doesn't do that...it just deletes it from that folder (which would work for a _normal_ IMAP implementation). SO... I discovered a little script called gmaillabelpurge that actually correctly purges Gmail before a certain date. So now I'm going to try (still untested...I'm thinking this out in writing so I try not to screw it up again):

1. Delete mail from each folder/tag older than 180 days using 'archivemail --delete' (except archive, spam and trash). This will still leave a copy of everything in 'All Mail'.
2. Archives mail in the 'All Mail' directory to the mbox file older than 180 days using 'archivemail'. This only affects the 'archive' folder on the local machine.
3. Use 'gmaillabelpurge' to delete all the old mail from the Gmail server directly

The archivemail man page mentions that it will abort if a 3rd party (like offlineimap?) modifies the mailbox during archiving, so I'm not sure if I need to figure out how to turn of offlineimap syncing during the archive process. <sigh> why do I like to make my life difficult to make it easy? <erm>

Also FYI, I --think-- I've finally determined that to delete something from the 'All Mail' folder with offlineimap you need to:
1. Make sure it doesn't exist in any other folder
2. Make sure the Gmail IMAP Expunge setting I mentioned in a previous post is set to its default.
3. Copy the file to the trash.

At this point I'm not sure I have an outstanding question...but I still need to finish up some things and let it sit for a week or so before I mark this as solved.

Scott

mattington · 2015-04-10 18:04:31

This post was very helpful for me in figuring out Gmail and offlineimap (I did not switch, though I do have performance issues on my machine as well. Annoying.). Anyway, my tip is to disable the "Important" tag in Gmail, this was flagging my mails as still being part of another folder, preventing them from being expunged from the system. Check into that if you have problems, as the above setup works well.

firecat53 · 2015-04-10 18:52:54

Thanks for the feedback...I actually put this on my todo list and never got back to it <cough> 1.5 years later....

Scott

Arch Linux

#1 2013-11-16 22:21:28

Offlineimap - can't delete files from Gmail 'All Mail' folder

#2 2013-11-17 17:48:16

Re: Offlineimap - can't delete files from Gmail 'All Mail' folder

#3 2013-11-17 17:56:36

Re: Offlineimap - can't delete files from Gmail 'All Mail' folder

#4 2013-11-17 18:03:56

Re: Offlineimap - can't delete files from Gmail 'All Mail' folder

#5 2013-11-17 18:50:59

Re: Offlineimap - can't delete files from Gmail 'All Mail' folder

#6 2013-11-17 20:14:00

Re: Offlineimap - can't delete files from Gmail 'All Mail' folder

#7 2013-11-21 00:05:53

Re: Offlineimap - can't delete files from Gmail 'All Mail' folder

#8 2013-11-21 01:51:47

Re: Offlineimap - can't delete files from Gmail 'All Mail' folder

#9 2013-11-21 02:55:05

Re: Offlineimap - can't delete files from Gmail 'All Mail' folder

#10 2013-11-21 03:21:40

Re: Offlineimap - can't delete files from Gmail 'All Mail' folder

#11 2013-11-21 17:32:59

Re: Offlineimap - can't delete files from Gmail 'All Mail' folder

#12 2013-11-21 19:03:49

Re: Offlineimap - can't delete files from Gmail 'All Mail' folder

#13 2013-11-21 21:00:17

Re: Offlineimap - can't delete files from Gmail 'All Mail' folder

#14 2013-11-22 00:44:13

Re: Offlineimap - can't delete files from Gmail 'All Mail' folder

#15 2015-04-10 18:04:31

Re: Offlineimap - can't delete files from Gmail 'All Mail' folder

#16 2015-04-10 18:52:54

Re: Offlineimap - can't delete files from Gmail 'All Mail' folder

Board footer