Adium

Of message history, and the thing that powers it

Adium has a feature called “message history”. When you open a new chat with a person, message history shows you the last n messages from your previous chat with that person, to remind you of the discussion from last time. This feature underwent some major changes in Adium 1.0.


Storage

Before Adium 1.0, Adium wrote those last n messages to a plist file when the chat closed. The problem with that was that when Adium crashed, the plist file would not be written out. This meant that when the user reopened the chat, the user would often be alarmed to find that the chat he had just been in (when he crashed) was not reflected in the message history, and would then conclude that the log of that chat had been lost.

This was never the case—we’ve always updated the log as messages are received and sent, never deferring writing to the end of the chat. But the confusion is understandable.

In Adium 1.0, we did away with the separate plist storage; we now use the logs (now called transcripts) for message history. This prevents the “where’d my history go?” problem, and ends the redundancy of writing out messages once to the logs and again to the plist.

Format

Before Adium 1.0, Adium stored its logs in a faux-HTML format. It was sufficient for display, and moderately parseable, but it was not very extensible, and not a true log format unto itself. (This is the format that Chat Transcript Manager understands.)

In Adium 1.0, we switched to a new real transcript format, which we call Unified Logging Format, and which is based on XML.


The problem now was how to quickly retrieve the last n messages from an XML transcript for the purpose of displaying the message history.

We had two options:

  1. Solution A: Use an existing XML parser to read the entire transcript, ignoring all messages except the last n.

    This solution doesn’t scale. A large transcript would cause a noticeable delay in opening a chat. This would be especially bad for chats opened from the remote side, since the user would not expect the beachball that would appear while Adium loads all those messages, discarding most of them on its way to the last n.

  2. Solution B: Write a new parser that parses the XML data in reverse.

Solution B is what I did.

I named the new parser “LMX” because it parses XML backwards (get it?). This allows Adium to read the last n without reading any of the messages before them.

13 Responses to “Of message history, and the thing that powers it”

  1. Mike Peel Says:

    Why didn’t you just store the logs in reverse order in the XML? Then the latest entries would have been at the top, meaning that solution A would have been fine.

  2. Martijn Says:

    Not sure if this is the easiest solution, but pretty cool nonetheless. :-D

  3. Peter Hosey Says:

    Mike Peel: Because the log is updated after every message, and file I/O doesn’t have an insert mode. We would have had to overwrite the rest of the file after every message, which would get very expensive really fast and probably not be atomic (so kiss that log goodbye in a hardware failure or power failure).

    And we’re certainly not going to switch to updating the log when the chat ends. That invites data loss of the kind that users of the old message history thought they had.

  4. alex Says:

    A cool hack indeed :)

  5. Anonymous Says:

    So Adium logs in “real time”? As in it updates the log file after every message is sent?

  6. David Says:

    anonymous: That is correct.

  7. Anonymous Says:

    Ah, the disadvantages of XML, you random access. Strange XML was picked as a _new_ format to fix an existing problem, and yet the existing problem still wasn’t fixed (well). With reverse parsing, wouldn’t that also cause problems if the XML was from another source? say, [transcript][message /][message /][message /][/transcript][transcript][message /][/transcript] Retrieving the last [message] may have unpredictable results.

  8. Anonymous Says:

    Oops, forgot the word lose =/ You lose random access.

  9. Peter Hosey Says:

    Anonymous: What existing problem? We didn’t need full random access; all we wanted was the last n elements, and LMX gives us that.

    And yes, borked XML can cause problems. Adium currently doesn’t allow you to view transcripts that aren’t in your Logs folder, so this is not going to be a very common problem—I doubt that most people will go drop transcripts from untrusted sources into their Logs folder. I’m sure we’ll robustify it when Adium has a UI more conducive to opening any transcript anywhere.

  10. rmann Says:

    What about keeping an index file, with date/file/offset for each message?

  11. Anonymous Says:

    I really miss the feature though where the chat transcripts were shown as consolidated by date.

  12. JoeyJoeJoe Says:

    So, is there some way to retrieve an entire conversation after Adium has been closed? Or do the files only store the last n lines of the conversation?

  13. pedro Says:

    hi I need to email some conversations i had in adium a couple of months ago for business purposes, I want to zip the entire chat folder and I dont want to copy/paste it in some word doc, could you please tell me where can i find my chat conversations files on my mac, where do they get storaged, could you please answer me via email, thanks, Ill appriciate it. pjgm84@hotmail.com
    PEDRO