Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Hadoop - composite key

Suppose I have a tab delimited file containing user activity data formatted like this:

timestamp  user_id  page_id  action_id

I want to write a hadoop job to count user actions on each page, so the output file should look like this:

user_id  page_id  number_of_actions

I need something like composite key here - it would contain user_id and page_id. Is there any generic way to do this with hadoop? I couldn't find anything helpful. So far I'm emitting key like this in mapper:

context.write(new Text(user_id + "\t" + page_id), one);

It works, but I feel that it's not the best solution.

like image 864
Jacek Chmielewski Avatar asked Sep 14 '12 15:09

Jacek Chmielewski


2 Answers

Just compose your own Writable. In your example a solution could look like this:

public class UserPageWritable implements WritableComparable<UserPageWritable> {

  private String userId;
  private String pageId;

  @Override
  public void readFields(DataInput in) throws IOException {
    userId = in.readUTF();
    pageId = in.readUTF();
  }

  @Override
  public void write(DataOutput out) throws IOException {
    out.writeUTF(userId);
    out.writeUTF(pageId);
  }

  @Override
  public int compareTo(UserPageWritable o) {
    return ComparisonChain.start().compare(userId, o.userId)
        .compare(pageId, o.pageId).result();
  }

}

Although I think your IDs could be a long, here you have the String version. Basically just the normal serialization over the Writable interface, note that it needs the default constructor so you should always provide one.

The compareTo logic tells obviously how to sort the dataset and also tells the reducer what elements are equal so they can be grouped.

ComparisionChain is a nice util of Guava.

Don't forget to override equals and hashcode! The partitioner will determine the reducer by the hashcode of the key.

like image 80
Thomas Jungblut Avatar answered Oct 19 '22 03:10

Thomas Jungblut


You could write your own class that implements Writable and WritableComparable that would compare your two fields.

Pierre-Luc Bertrand

like image 21
Pierre-Luc Bertrand Avatar answered Oct 19 '22 03:10

Pierre-Luc Bertrand