How to model interpretations of rap music

Question

I just started working on a website that will help people understand what rappers are talking about. Users will see the lyrics to a rap song and they'll be able to click certain lyrics to see an explanation. Here's a screenshot (you can also check out the site itself here):

alt text http://img146.imageshack.us/img146/6882/clocal.png

(Original lyrics censored; click here to see them)

Anyway, my question is how to model these annotations in my application. Right now, I'm storing the lyrics and annotations as one big blob of HTML in this format:

<div class="lyrics">   With the goons I spy   <a href="#note1">Stay in tune with ma</a>   <a href="#note2">She like damn   This the realest since 'Kumbaya'</a>   Kumbayay Killa Cam my lord  </div>  <div class="annotations">   <div id="note1"> "Ma" refers to ladies, generally, and specifically also the woman singing the hook;  "Stay in tune" is a musical metaphor: he literally stays in tune with the singer and also in the sense that he has game.   </div>   <div id="note2"> Kumbaya is a campfire singalong.   </div> </div> 

and then processing it with this method for output:

class Song < ActiveRecord::Base   include ActionView::Helpers    def annotated_lyrics     lyrics = read_attribute('annotated_lyrics')     return if lyrics.blank?      require 'hpricot'     doc = Hpricot lyrics      doc.at('.lyrics').inner_html = doc.at('.lyrics').inner_html.strip     doc.search("a[@href^='#note']").set('class', 'tooltip').each do |t|       t.inner_html = t.inner_html.strip     end     doc.search("div[@id^='note']").set('class', 'annotation').each do |a|       a.inner_html = auto_link(a.inner_html.strip, :all, :target => '_blank')     end     simple_format doc.html.strip   end end 

and the rest I do with jQuery and the fantastic qTip plugin.

This works fine for display, but since my application doesn't know about the relationship between annotations and lyrics, it will be hard to, say, add an interface for updating an individual annotation inline (or at all, really).

On the other hand, I don't really know the best way to represent this in ActiveRecord. I suppose a song could "have_many" annotations, but how would I represent which lyrics were annotated? I could store the start and end word index, but this seems painful and sensitive to minor changes in the lyrics.

Answer 1

What about presenting the lyrics like this (with thanks to the People's Champ)?

Well it's that [grain grippa][1] from Houston, Tex That bar sippa, that bar no plex I'm straight up outta that [Swishahouse][2] Where G. Dash write all the checks So [check the neck, check the wrist][3] I'm balla status from head to toe  [1]Referring to the wood grain steering wheel common to luxury cars [2]Swisha House is the record label Paul Wall records for [3]"Look at my watch and necklace because they are expensive" 

Just an idea, I was inspired by the markup used to add comments on this site.

So, for the database, create Lyric, LyricLine and Annotation tables. Annotations have LyricLineIds, StartChar and EndChar values and a Meaning or Description field. LyricLines are the text of each line, related to the Lyric entity by LyricIds. Lyrics store song info, language info, whatever.

This format should be pretty easy to generate off of the database and has the benefit of being more "human readable" than XML and editable in-place, so you can test it a lot easier before you have to develop a whole UI.

I have this question favorited, and look forward to watching the site progress. Interesting work!

Answer 2

1. Tokenize your lyrics, so that you can identify a word in the lyrics by using e.g. a line and word number. Another option is to use character positions for your annotations. In any case, as always, take care of the character encoding of the lyrics.
2. Further, never touch the lyrics anymore. Better not store them as html, but as xml or as plain text.
3. Don't annotate within lyrics. Use a model wherein you can attach a position in the lyrics to an annotation. Use stand-off annotation.

Stand-off annotation will allow you to add more features over time, such as letting many users annotate the same lyrics. Generating the HTML you store as a blob is easy to do from stand-off annotations.

You might be interested in the (xml) data models of annotation tools that are quite well known among linguists: e.g. MMAX2 and Callisto. These are easily convertible to database models.