How can I group individual emails by conversation, so that I know for sure (100%) that one email is in the same conversation as the other?

Question

Question

How can I group individual emails by conversation when using the Exchange WebServices API so that I know for sure (100%) that one email is in the same conversation as the other?

By individual I mean: I should be able to download 1 email at a time, since when a late response is sent (let's say as a reply to an email from last month), I don't want to bulk-download the whole conversation again, but just that one email to be able to store it in my own database.

context

I am working on a project where I need to put all fresh emails in a MySQL database so that we can add extra meta data from out of our CRM application. Right now I am doing some prototyping with the Exchange WebServices API over XML so that I know what data is available and what it looks like. I am doing this in PHP (also the language our CRM is built in).

Problem

When running a simple XML request (see below) I receive some emails from my inbox/sentitems folder. Since I know I had a conversation with a colleague to test the ConversationIndex I found out that the ConversationIndex doesn't seem to be an index as I expected it to be. I expected it to be equal on all emails so that I could easily group one conversation together.

But that doesn't appear to be true. Because when I lay the ConversationIndexes of my inbox and snetitems (of one conversation) next to each other I get the following result: (--> sentitem, <-- inbox)

1. --> Ac9IKUBx8yX3vSXYQSWcb8ggCfC7FQ== (first sentitem)
2. <-- Ac9IKUBx8yX3vSXYQSWcb8ggCfC7FQAAAT1A (reply to 1)
3. --> Ac9IKUBx8yX3vSXYQSWcb8ggCfC7FQAAAT1AAAAHYQA= (reply to 2)
4. <-- Ac9IKUBx8yX3vSXYQSWcb8ggCfC7FQAAAT1AAAAHYQAAAAScMA== (reply to 3)
5. --> Ac9IKUBx8yX3vSXYQSWcb8ggCfC7FQAAAT1AAAAHYQAAAAScMAAACBsQ (reply to 4)
6. <-- Ac9IKUBx8yX3vSXYQSWcb8ggCfC7FQAAAT1AAAAHYQAAADX4EA== (reply to 5)
7. <-- Ac9IKUBx8yX3vSXYQSWcb8ggCfC7FQAAP7ZQ (reply to 1)

Example of other conversation ConversationIndex: Ac9IB53hRlE85QAKR9qCJ8pbv8gikwAAtByQ

As you can see the ConversationIndex changes for every email except for the first 30 characters. Another thing I found out by consulting the answer on this question is that I should use a BitConverter and modify the ConverationIndex but that is for C#. I couldn't find a PHP solution for this neither.

Also, I can't find any information on this subject in the EWS docs (for example: http://msdn.microsoft.com/en-us/library/ms528174(v=exchg.10).aspx).

Possible alternatives

I saw a References key in the XML response from the Exchange server, but using those InternetMessageId's to determine what conversation an email is in would become a true hell.

Another thing I noticed was the ConversationId (http://msdn.microsoft.com/en-us/library/ff387041%28v=office.12%29.aspx) but that doesn't seem to be what I'm look for neither.

The XML request, just for reference

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
               xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types">
    <soap:Body>
        <FindItem xmlns="http://schemas.microsoft.com/exchange/services/2006/messages"
                  xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types"
                  Traversal="Shallow">
            <ItemShape>
                <t:BaseShape>AllProperties</t:BaseShape>
            </ItemShape>
            <IndexedPageItemView MaxEntriesReturned="5" Offset="0" BasePoint="Beginning" />
            <ParentFolderIds>
                <t:DistinguishedFolderId Id="inbox OR sentitems"/>
            </ParentFolderIds>
        </FindItem>
    </soap:Body>
</soap:Envelope>

UPDATE 1

It looks like I have to use the ConversationId instead of the index. Although I am working on Exchange 2007 at the moment we are upgrading to 2013 very soon. Can somebody verify that I have to use the ConversationId indeed?

Answer 1

Jelmer-

I recently wrote the How to: Work with conversations by using EWS in Exchange article on MSDN. The GetConversationItems EWS operation, which was introduced in Exchange 2013, is what you want to use to retrieve everything in a specific conversation. GetConversationItems uses the ConversationId element, which wasn't around in Exchange 2007. ConversationTopic and ConversationIndex were around in Exchange 2007 - and I would assume you could use another operation to find all items with a specific ConversationTopic, but I haven't had to do that. If you want help with that approach, let me know.

The ConversationThread is the value you identified above (Ac9IKUBx8yX3vSXYQSWcb8ggCfC7FQ), on the first message, then the ConversationIndex is incremented every time the item is responded to, forwarded, etc.

From the article: "Exchange applies the same ConversationTopic value to replies to the first message and then updates the ConversationIndex value to represent the message’s position relative to the original message. If the subject of the email thread changes, Exchange applies a new ConversationTopic value and new ConversationIndex values to the new conversation."

I noticed that what you referred to in your post as the EWS docs is actually the CDO docs. The EWS reference content for Exchange 2013 is located here: http://msdn.microsoft.com/en-us/library/bb204119(v=exchg.150).aspx. And the EWS reference content for Exchange 2007 (if you want to compare), is here: http://msdn.microsoft.com/en-us/library/bb204119(v=exchg.80).aspx.

Hope this helps and please mark the post as answered if I did indeed answer your question.

Thanks, Mimi

---

Updated on 3/27 to respond to additional questions about ConversationId:

I just played around with this to be sure and the ConversationId does change if the subject is changed, but it stays the same so long as the message is just forwarded and replied to. My original ConversationId was:

<ConversationId Id="AAQkADI2MDY4MjQzLWQ1YzQtNGZkNi1iZmE4LTRjZmZmNGEyZjQ0YwAQAJJCqNI7ZfhJmIh7j2kgJvk=" />
<ConversationTopic>Original</ConversationTopic>

but after changing the subject twice it looked like this:

<ConversationId Id="AAQkADI2MDY4MjQzLWQ1YzQtNGZkNi1iZmE4LTRjZmZmNGEyZjQ0YwAQAEjf5yU/W9VMo6HxuanvhYE=" />
<ConversationTopic>2nd subject</ConversationTopic>

So the first X characters are the same, but then there's divergence.