Using fetchmail for one time email extraction from gmail

Question

• I'm trying to use fetchmail in terminal to extract e-mails from my gmail account.
• I configured my ~/.fetchmailrc with:

poll imap.gmail.com protocol POP3
   user "[email protected]" is oren here
   password 'verysecretpassword'

(Of course with real username+password).

• Then I tried to naively extract emails with: $ fetchmail.
• Sadly nothing happened, and all I got was:

fetchmail: 6.3.26 querying imap.gmail.com (protocol POP3) at Mon 03 Feb 2020 14:34:46 IST: poll started
Trying to connect to <ADDRESS> ... connection failed.

Answer 1

Using Fetchmail

It looks like the config is set to poll an IMAP server but then specifying POP3 protocol. Try something like this for the ~/.fetchmailrc file:

set postmaster "local_user"
set daemon 600
poll pop.gmail.com with proto POP3
   user 'gmail_user_name' there with password 'app_password' is local_user here options ssl fetchlimit 400

where:

• local_user is some local account where undeliverable mail should go (the "last ditch effort" before failing permanently).
• gmail_user_name is everything to the left of the @ in the email address.
• app_password is a specially generated password that is restricted to the gmail application (go here: https://myaccount.google.com/ and click Security, then app passwords and generate a new app password)

What to do at this point will depend on your local setup. Fetchmail will... fetch mail (clearly).... and then deliver it to the local machine's delivery system. If you have sendmail (a pretty safe bet), this might work:

$ fetchmail -d0 -avNk -m "/usr/sbin/sendmail -i -f %F -- %T" pop.gmail.com

Mail should start flowing in. Messages can be read using the mail command or get the raw content from /var/mail/[username]. This might not get everything in one shot; it very likely won't if the address has accumulated even a small amount of history. Let it finish and check that it worked as expected. If everything looks good, then it's time to start fetchmail as a daemon process and let it download the entire mailbox. First, configure fetchmail with appropriate polling interval and batch size settings¹.

• Confirm that the polling interval is configured in the ~/.fetchmailrc by the line daemon 600 (ie. 10 minute polling interval).
• Confirm that the polling option fetchlimit 400 is set in the ~/.fetchmailrc under the options section in the poll pop.gmail.com stanza. This is the maximum number of messages to fetch per poll.
• Start fetchmail using the same command as above, but omit the -d0 switch.

Fetchmail should start as a true daemon and continue to periodically download batches of messages until the whole mailbox is downloaded. You will need to remember to kill the daemon process if you don't want it to continue syncing until the next reboot.

---

Using Google Takeout

You can do this super easy using Google Takeout. Log in, click the "deselect" option at the top of the list, then scroll down to Mail and check just that. You can choose to get the data in a .zip or .tgz file. They will send you an email when the archive is ready for download. It is packaged in an mbox file but that is pretty straightforward to convert to other formats.

This is probably the easiest way to accomplish a one time export, and I think they have an option to set up a recurring export too. It probably isn't offering as much control compared to using the developer API directly, but it's a lot less hassle.

---

_{1: I believe Google has some rate limiting in place, so I am adding some steps to accommodate those limits. These are conservative values, since I don't know exactly what the limits are (or even for sure if they exist). If you know more, or care to research it, adjust these values to whatever you think is best.}