Most efficient way to get new messages

Question

I have a process that looks for new messages in a users Gmail. The message is added to an external database if it meets certain address criteria.

We have been using Users.History.List which returns all messages that have had a change made to them. This is quite inefficient as we have to subsequently check each message to see if we have already processed it.

We are looking at alternatively using Users.Messages.List and checking the MsgId to see if it is larger than the previous check (we store the Id from that). The assumption here is that the MsgId will keep getting larger. Is this approach flawed? What are others doing?

Many Thanks.

Answer 1

Here is an example how to get only new message (using Java client api)

List<History> histories = new ArrayList<History>();
ListHistoryResponse response = service.users().history().list(userUID).setStartHistoryId(startHistoryId).execute();

//get all history events and not only the first page
while (response.getHistory() != null) {
    histories.addAll(response.getHistory());
    if (response.getNextPageToken() != null) {
        String pageToken = response.getNextPageToken();
        response = service.users().history().list(userUID)
                .setPageToken(pageToken)
                .setStartHistoryId(startHistoryId).execute();
    } else {
        break;
    }
}

//for each history element find the added messages
for (History history : histories) {
    List<HistoryMessageAdded> addedMessages = history.getMessagesAdded();

    if (addedMessages == null){
        continue;
    }

    //call to message.get for each HistoryMessageAdded
    for (HistoryMessageAdded addedMsg : addedMessages) {
        Message message = addedMsg.getMessage();
            Message rawMessage = service.users().messages().get(userUID, message.getId()).setFormat("raw").execute();

    }
}

Probably there is a similar implementation in other languages/REST API.

You can use other history events such as:messagesDeleted, labelsAdded and labelsRemoved Reference: https://developers.google.com/gmail/api/v1/reference/users/history/list

Answer 2

Message ID is unique and its value is never changed. To get new messages, you can use history.list(), and give historyId of whatever the largest historyId you've previously for the message.

Here is the example response:

{
 "history": [
  {
   "id": "1825077",
   "messages": [
    {
     "id": "14b4c0dbc6ba9a57",
     "threadId": "14b4b4ae8cfbea5c"
    }
   ]
  },
  {
   "id": "1825087",
   "messages": [
    {
     "id": "14b4c0dc3ab5e49b",
     "threadId": "14b4b4ae8cfbea5c"
    }
   ]
  },
  {
   "id": "1825097",
   "messages": [
    {
     "id": "14b4c0e07e0f6545",
     "threadId": "14b4b4ae8cfbea5c"
    }
   ]
  }
 ]
}

1825097 is the largest historyId for the message "14b4c0e07e0f6545". Also Msgid didn't change here only history id is changed.

If you give 1825097 as history id and there is no change in the message, then the response will be 200 with headers. If you get response 404 error, you'll need to use messages.list() instead to perform a full sync.