You are not logged in.

#1 2018-07-11 13:29:24

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 20,445
Website

[SOLVED] isync/mbsync with OAuth2 for google accounts?

[SOLVED NOTE] while not working for my particular situation, I believe the information provided in this thread may be sufficient to set up mbsync for OAauth2 on other domains.

---

My new work email uses google/gmail accounts (with a different domain @schoolname.edu) as did my previous work place.  However, I was unable to translate my mbsync configuration to retrieve mail from the new system.

I discovered the new system may not allow any normal IMAP(S) access, but requires the use of google's OAuth2.  I've found several online discussions of this, and it seems most mbsync users just toggle the gmail setting for 'enable less secure applications'*.  Unfortunately, this option has been disabled at the domain level.  I'll see if I can convince IT to re-enable that option (as it's actually more secure), but from what I've heard that's unlikely.

This leaves getting mbsync to work with google's OAuth2 mechanism which is, to me, a complete mystical black box (or poo-brown box really).  In one mailing list thread I can no longer track down someone mentioned there was an OAuth2 plugin for mysnc, but it was so hard to use that one should just enable "less secure apps".  As the latter isn't an option, I'd like to try the hard to use plugin.  However I've been unable to find any other references to it.

I did find one blog post that suggests it can be done (even in arch):
http://taitran.ca/2016/12/22/oauth2-with-mutt.html

However, those steps as they are written seem troublesome as they install everything as root bypassing pacman.  I believe isync/mbsync in the repos is already built with SASL so there should be no need to rebuild that:

$ mbsync --help
...
Compile time options:
  +HAVE_LIBSSL +HAVE_LIBSASL +HAVE_LIBZ +USE_DB +HAVE_IPV6

So this would seem to leave the OAuth2 tool.  I cloned that from github and built it.  But neither it, nor the blog post, have any documentation or usage information, nor is there any --help output:

$ ./oauth2 --help
$ ./oauth2
No arguments given, do nothing
$

I also tried checking the repos for any OAuth2 tools and found extra/signon-plugin-oauth2, but this has the same issue of having absolutely no documentation, no wiki page, and no --help output from any of it's binaries.

So I'm at a loss at how to proceed or whether this is even a viable strategy - e.g., is getting mbsync (or any other IMAP or mail client for that matter) working with google's OAuth2 a complete non-starter, or is there some documentation on how it can be done?

---

* A bit ironic: "less secure applications" are those that would require password authentication over an ecrypted connection every time one wanted to access the mail.  While the OAuth2 mechanism I find quite disturbing: I've logged in once, and really can't seem to log out: anyone who visits gmail.com on my computer just gets instant access to all my email.  That to me is less secure!

Last edited by Trilby (2018-07-11 21:17:17)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#2 2018-07-11 15:18:08

progandy
Member
Registered: 2012-05-17
Posts: 3,195

Re: [SOLVED] isync/mbsync with OAuth2 for google accounts?

I think you need to register with google for a client id and client secret, then write those in the config.h as well as a redirect_uri. Change the username as well ("abc@gmail.com") and then create/refresh the token with

oauth2 your_account@example.com

The code tires to write the refresh token to /home/tait/.cache/oauth_ ... That should probably be fixed as well...

https://developers.google.com/identity/protocols/OAuth2

Edit: Apparently, it should also work with key anonmous and secret anonymous and a display name, but I don't know how to incorporate that in the tait code.
https://gist.github.com/tsycho/1970639

https://github.com/gitrc/nagios-plugins … d/OAuth.pm Probably outdated.

Edit: Using oauth2l (the python version) to create the tokens might work instead of the tait oauth binary.
https://github.com/google/oauth2l

Last edited by progandy (2018-07-11 16:00:12)


| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

Online

#3 2018-07-11 17:44:35

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 20,445
Website

Re: [SOLVED] isync/mbsync with OAuth2 for google accounts?

Thanks.  I see there is a python-oauth2client in the repos as well (edit: although this seems to be something different).  I was hoping there was a way to do this without writing my own tools, but perhaps using the python libs will not be bad.  It's still quite a black box to me on how I get mbsync to ... do whatever needs to be done to connect even when I get python code running.  I see there is a PassCmd option for mbsync which runs some other program to get a password, perhaps that's the connection.

EDIT: actually I'm not really sure what that github link really does.  It seems to require being passed the name of yet another tool that will actually do the SSO work and generate a token.  I beleive if I had such a tool, that'd be all I need, but in the absence of such a tool, I don't think that python code will do anything for me.

Last edited by Trilby (2018-07-11 17:55:56)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#4 2018-07-11 18:24:58

progandy
Member
Registered: 2012-05-17
Posts: 3,195

Re: [SOLVED] isync/mbsync with OAuth2 for google accounts?

I found something for you:
https://github.com/google/gmail-oauth2- … /oauth2.py

Edit: You can then use the generated access token as your password or create a script for passcmd that takes the refresh token and automatically requests a new access token if it expires.

Those tokens are basically the same as passwords. OAuth2 is more secure than a normal password since you get a new token on each device. You can then remotely disable the token and therefore prevent access, so a stolen laptop is not the same as a stolen password even if your data is unencrypted. If you never store passwords, then oauth is less secure.

You can also revoke a token without using the google ui and do an API call instead
https://developers.google.com/identity/ … okenrevoke

Last edited by progandy (2018-07-11 18:47:27)


| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

Online

#5 2018-07-11 18:45:28

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 20,445
Website

Re: [SOLVED] isync/mbsync with OAuth2 for google accounts?

Thanks, but unfortunately there's still "magic" in step one of getting the initial authroziation token that is required by that code.  I have no such token, and the google api page which can allegedly be used to create one for a new 'app' is disabled and cannot be accessed.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#6 2018-07-11 18:53:43

progandy
Member
Registered: 2012-05-17
Posts: 3,195

Re: [SOLVED] isync/mbsync with OAuth2 for google accounts?

I've never created an API key. I suggest you try to create the app with a normal google account. The keys should then work with any google or organization account, they identify the application and not the user.
Edit: Organizations can probably restrict access for third-party apps, but I doubt that this is done.

Edit: If you are evil, you can try to extract the keys from thunderbird big_smile

Last edited by progandy (2018-07-11 19:01:36)


| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

Online

#7 2018-07-11 19:30:24

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 20,445
Website

Re: [SOLVED] isync/mbsync with OAuth2 for google accounts?

Ah, you are right on that point.  I got that python code running and it seems to successfully retreive refresh and access tokens.  Unfortunately these do not work with mbsync (yet).  I've placed the path to the script that generates an access token in PassCmd in ~/.mbsyncrc but I get the same error as when I just tried to use my actual password:

$ mbsync work
C: 0/1  B: 0/1  M: +0/0 *0/0 #0/0  S: +0/0 *0/0 #0/0
IMAP command 'AUTHENTICATE PLAIN <authdata>' returned an error: NO [AUTHENTICATIONFAILED] Invalid credentials (Failure)
C: 1/1  B: 0/1  M: +0/0 *0/0 #0/0  S: +0/0 *0/0 #0/0

Reading the mbsync man page for PassCmd, I'm not sure if that's how an OAuth token should be used as PassCmd really just runs and generates output that is then treated like the Password field.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#8 2018-07-11 19:36:51

progandy
Member
Registered: 2012-05-17
Posts: 3,195

Re: [SOLVED] isync/mbsync with OAuth2 for google accounts?

Try to set the "Access Token" in the password field without the PassCmd.

The sasl-oauth module should take the username and oauth token and create the correct login credentials. In the first test you can even remove the password settings and let it prompt you I think.

Last edited by progandy (2018-07-11 19:38:37)


| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

Online

#9 2018-07-11 19:41:18

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 20,445
Website

Re: [SOLVED] isync/mbsync with OAuth2 for google accounts?

Can you clarify a bit, do you mean litterally having a line like the following in the config:

Pass Access Token

If so, how would mbsync no what script to run to generate a new access token?  That has to be provided somewhere.
I tried the above:

$ mbsync work
/home/jmcclure/.mbsyncrc:13: excess token 'Token'

Line 13 is that line of the config.

EDIT, I also tried removing all password lines from the config, and upon running it prompted for the password for which I entered a newly generated access token, but I got the following:

$ mbsync work
C: 0/1  B: 0/1  M: +0/0 *0/0 #0/0  S: +0/0 *0/0 #0/0
Password (work):
IMAP command 'AUTHENTICATE PLAIN <authdata>' returned an error: NO [AUTHENTICATIONFAILED] Invalid credentials (Failure)
C: 1/1  B: 0/1  M: +0/0 *0/0 #0/0  S: +0/0 *0/0 #0/0

Last edited by Trilby (2018-07-11 19:45:14)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#10 2018-07-11 19:43:51

progandy
Member
Registered: 2012-05-17
Posts: 3,195

Re: [SOLVED] isync/mbsync with OAuth2 for google accounts?

I meant to set the password to the access token you received from the oauth2 script or not setting a password at all and let it ask for it.

The access token is valid for about an hour, so to verify that it works just try it without the refresh.

Edit: Hmm, I don't know. You enabled die Gmail API for the API key, right?

Edit: You have to set AuthMechs to XOAUTH2 I think. Your error message still contains "PLAIN"
Edit: You have isync with sasl support. In addition you need the sasl2-oauth plugin.
Edit: Here is a PKGBUILD

pkgname=sasl2-oauth
pkgver=0.1+gc1d7cd07
pkgrel=1
arch=(x86_64)
_commit=c1d7cd0719c233c89307b7406f92a01602a85993
source=("$pkgname-$pkgver::https://github.com/robn/sasl2-oauth/archive/$_commit.zip")
sha256sums=('e14b03317c31efc01b4a69442ba2b9118778488f29c9c86a3350576868fab910')

build() {
	cd "$pkgname-$_commit"
	autoreconf -f -i
	./configure --prefix=/usr
}

package() {
	cd "$srcdir/$pkgname-$_commit"
	make DESTDIR="$pkgdir" install
}

Last edited by progandy (2018-07-11 20:10:10)


| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

Online

#11 2018-07-11 20:23:14

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 20,445
Website

Re: [SOLVED] isync/mbsync with OAuth2 for google accounts?

Great, thanks - yes I needed the sasl2-oauth plugin, that got a big step closer to working (previously AuthMech XOAUTH2 gave an error about not being a recognized method).  Now I get this:

$ mbsync -V work
Reading configuration file /home/jmcclure/.mbsyncrc
C: 0/1  B: 0/1  M: +0/0 *0/0 #0/0  S: +0/0 *0/0 #0/0
Channel work
Opening master store work-remote...
Resolving imap.gmail.com... ok
Connecting to imap.gmail.com (74.125.69.108:993)...
Opening slave store work-local...
Connection is now encrypted
Logging in...
Authenticating with SASL mechanism XOAUTH2...
Password (work):
Error: SASL(-13): authentication failure: server rejected XOAUTH2: {"status":"400","schemes":"Bearer","scope":"https://mail.google.com/"}

So it looks like the XOAUTH2 method is indeed being used, and the key is sent, but rejected.  I suspect this is due to the API key being for mail.google.com but server is actually imap.gmail.com ... which is rather ridiculous as the API key was generated by google for gmail.  I'll see if I can generate a new API key for the proper scope.

EDIT: Eh ... I don't see anywhere on the google API websites to change the scope associated with the key.  Gmail API keys are for mail.google.com but gmail cannot be retrieved from there, only from imap.gmail.com.

Last edited by Trilby (2018-07-11 20:29:09)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#12 2018-07-11 20:28:33

progandy
Member
Registered: 2012-05-17
Posts: 3,195

Re: [SOLVED] isync/mbsync with OAuth2 for google accounts?

The scope should be OK.
https://developers.google.com/gmail/ima … _20_scopes

Can you test with

oauth2.py --test_imap_authentication --user ... --access_token ...

| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

Online

#13 2018-07-11 20:33:52

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 20,445
Website

Re: [SOLVED] isync/mbsync with OAuth2 for google accounts?

That test fails:

$ ./run

  30:09.73 > CPDP1 AUTHENTICATE XOAUTH2
  30:09.81 < +
  30:09.81 write literal size 228
  30:10.01 < + eyJzdGF0dXMiOiI0MDAiLCJzY2hlbWVzIjoiQmVhcmVyIiwic2NvcGUiOiJodHRwczovL21haWwuZ29vZ2xlLmNvbS8ifQ==
  30:10.01 write literal size 228
  30:10.21 < CPDP1 NO [AUTHENTICATIONFAILED] Invalid credentials (Failure)
  30:10.21 NO response: [AUTHENTICATIONFAILED] Invalid credentials (Failure)
Traceback (most recent call last):
  File "/tmp/oauth2", line 320, in <module>
    main(sys.argv)
  File "/tmp/oauth2", line 307, in main
    base64_encode=False))
  File "/tmp/oauth2", line 248, in TestImapAuthentication
    imap_conn.authenticate('XOAUTH2', lambda x: auth_string)
  File "/usr/lib/python2.7/imaplib.py", line 366, in authenticate
    raise self.error(dat[-1])
imaplib.error: [AUTHENTICATIONFAILED] Invalid credentials (Failure)

I tried regenerating a new access token, but that is currently failing with a python error:

Traceback (most recent call last):
  File "/tmp/oauth2", line 320, in <module>
    main(sys.argv)
  File "/tmp/oauth2", line 284, in main
    print 'Access Token: %s' % response['access_token']
KeyError: 'access_token'

The python error looks like it's due to the oauth.py code non getting the response it expects.

EDIT:  I started back at the first step of the oauth.py code to get new refresh and access tokens, then I tested that access token:

  36:45.54 > JAFH1 AUTHENTICATE XOAUTH2
  36:45.65 < +
  36:45.65 write literal size 228
  36:46.18 < + eyJzdGF0dXMiOiI0MDAiLCJzY2hlbWVzIjoiQmVhcmVyIiwic2NvcGUiOiJodHRwczovL21haWwuZ29vZ2xlLmNvbS8ifQ==
  36:46.19 write literal size 228
  36:46.41 < JAFH1 NO [ALERT] IMAP access is disabled for your domain. Please contact your domain administrator for questions about this feature. (Failure)
  36:46.41 NO response: [ALERT] IMAP access is disabled for your domain. Please contact your domain administrator for questions about this feature. (Failure)
Traceback (most recent call last):
  File "/tmp/oauth2", line 320, in <module>
    main(sys.argv)
  File "/tmp/oauth2", line 307, in main
    base64_encode=False))
  File "/tmp/oauth2", line 248, in TestImapAuthentication
    imap_conn.authenticate('XOAUTH2', lambda x: auth_string)
  File "/usr/lib/python2.7/imaplib.py", line 366, in authenticate
    raise self.error(dat[-1])
imaplib.error: [ALERT] IMAP access is disabled for your domain. Please contact your domain administrator for questions about this feature. (Failure)

As it looks like this is due to the specific college's own domain settings, I may be SOL.

Last edited by Trilby (2018-07-11 20:38:53)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#14 2018-07-11 20:51:30

progandy
Member
Registered: 2012-05-17
Posts: 3,195

Re: [SOLVED] isync/mbsync with OAuth2 for google accounts?

I just did it myself and the imap test in oauth2.py worked.
* Create new Google Developer Project
* Switch to Project
* Add Gmail API
* Go to APIs & Services
* Create API Key (probably unnecessary)
* Set mail and name for oauth consent screen
* Create Credentials "OAuth-Client-ID" with type "Other"
* request token with oauth2.py and test it.

Edit: ... They did disable IMAP. How "progressive"... Why....
Edit: Maybe if you ask nicely. You can always hope.

Last edited by progandy (2018-07-11 21:00:56)


| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

Online

#15 2018-07-11 21:16:02

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 20,445
Website

Re: [SOLVED] isync/mbsync with OAuth2 for google accounts?

progandy wrote:

Edit: ... They did disable IMAP. How "progressive"... Why....
Edit: Maybe if you ask nicely. You can always hope.

I'll try, but it will take an aweful lot of hope.  In any case, thanks for all the detailed help.  Perhaps others can find this thread useful if their domain admin isn't quite so crazy.  I suppose I'll also mark this as SOLVED as it's as far as it can go within the external constraints.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#16 2018-07-11 21:29:45

progandy
Member
Registered: 2012-05-17
Posts: 3,195

Re: [SOLVED] isync/mbsync with OAuth2 for google accounts?

The only thing missing is a command for PassCmd that caches the access_token and only requests a new one when the old one expires. A naive approach would request a new one every half hour.

Edit: It is extremely unlikely, but maybe that error message is the same if you can enable IMAP for your account but omitted doing that. Open the mailbox in gmail and in the settings look for something like "forwarding and pop/imap". Can you enable imap here?

Last edited by progandy (2018-07-11 21:34:27)


| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

Online

#17 2018-07-11 21:35:53

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 20,445
Website

Re: [SOLVED] isync/mbsync with OAuth2 for google accounts?

Whenever the refresh key is used to get a new access key, a key is returned with a time till expiration.  So there shouldn't be a need to guess - just cache the access code and the expiration time.  Then on each call to the PassCmd command it can check if the current key is still valid, then use it, or request a new one.

EDIT: RE forwarding pop/imap, I can find no such settings.  That in itself is also disturbing as my worst-case backup plan was to just never really use that account but forward everything from it to my better accounts.

Last edited by Trilby (2018-07-11 21:40:14)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#18 2018-07-11 22:14:38

progandy
Member
Registered: 2012-05-17
Posts: 3,195

Re: [SOLVED] isync/mbsync with OAuth2 for google accounts?

That is really evil. https://support.google.com/a/answer/2491924?hl=en
Lucky that I am my own admin.

For completeness sake, here is a script that should do the key refresh in PassCmd. It imports the oauth2.py from post #4.

#!/usr/bin/python2
  
import oauth2
import sys
import json
import time

len(sys.argv)

if len(sys.argv) != 5:
    print "gmailtoken.py tokencache_file client_id client_secret refresh_token"
    sys.exit(1)

try:
    # try to use a cached access token
    with open(sys.argv[1]) as json_file:
        data = json.load(json_file)
        # only use it if does not expire in the next 60 seconds
        if 60+time.time() < 0+data['expires_time']:
            print data["access_token"]
            sys.exit(0)
except IOError:
    pass

response = oauth2.RefreshToken(sys.argv[2], sys.argv[3], sys.argv[4])
print response['access_token']
response['expires_time']=time.time()+response['expires_in']
with open(sys.argv[1], "w") as json_file:
    json.dump(response, json_file)

Edit: It is necessary to recompile isync with an increased buffer size for PassCmd in src/drv_imap.c (function ensure_password):

char buffer[80];
==>
char buffer[800];

Note: sasl2_oauth seems to throw authentication errors even though the authentication is successful and mbsync continues with the sync. It can be improved with this change in the sasl2-oauth source  file "xoauth2.c":

    if (text->state > 0) {
        static char buf[1024];
...

==>

    if (text->state > 0) {
        if (serverinlen == 0 || (serverinlen == 1 && serverin[0] == '\0') )
          return SASL_OK;
        static char buf[1024];
...

Last edited by progandy (2018-07-12 14:57:17)


| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

Online

#19 2018-07-12 15:16:48

progandy
Member
Registered: 2012-05-17
Posts: 3,195

Re: [SOLVED] isync/mbsync with OAuth2 for google accounts?

I got an initial sync working with my account, so all pieces should be here now. I won't continue since I currently prefer thunderbird.

@Trilby: I suggest you try gmailieer (AUR) to create a local notmuch mailbox. It uses the GMail API instead of IMAP, so it may work for you.
Sending messages and drafts are not implemented, if you want that, you'd have to write something
https://developers.google.com/gmail/api/v1/reference/
Edit: A minimal mta for gmail api exists: gmail-relay

Last edited by progandy (2018-07-12 15:50:24)


| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

Online

#20 2018-07-12 16:50:05

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 20,445
Website

Re: [SOLVED] isync/mbsync with OAuth2 for google accounts?

Thanks for the suggestions, I may look into those soon.  But I may have to balance the mild frustration of setting up and maintaining that with the potentially milder frustration of just telling everyone to use my preferred addresses and never use my @workplace email as I may not get it or respond in any reasonable time frame.

EDIT: I just tried gmailieer.  It works, more or less.  It does indeed retreive mails which is very helpful.  But apparently everything is just dumped into 'cur' mail and nothing is ever flagged as new.  So this is better than nothing, but still very far from a reasonable set up for actually using email.

Last edited by Trilby (2018-07-25 20:24:30)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

Board footer

Powered by FluxBB