Is there a common place to store data schemas in Hadoop?

Question

I've been doing some investigation lately around using Hadoop, Hive, and Pig to do some data transformation. As part of that I've noticed that the schema of data files doesn't seem to attached to files at all. The data files are just flat files (unless using something like a SequenceFile). Each application that wants to work with those files has its own way of representing the schema of those files.

For example, I load a file into the HDFS and want to transform it with Pig. In order to work effectively with it I need to specify the schema of the file when I load the data:

EMP = LOAD 'myfile' using PigStorage() as { first_name: string, last_name: string, deptno: int};

Now, I know that when storing a file using PigStorage, the schema can optionally be written out along side it, but in order to get a file into Pig in the first place it seems like you need to specify a schema.

If I want to work with the same file in Hive, I need to create a table and specify the schema with that too:

CREATE EXTERNAL TABLE EMP ( first_name string
                          , last_name string
                          , empno int)
LOCATION 'myfile';

It seems to me like this is extremely fragile. If the file format changes even slightly then the schema must be manually updated in each application. I'm sure I'm being naive but wouldn't it make sense to store the schema with the data file? That way the data is portable between applications and the barrier to using another tool would be lower since you wouldn't need to re-code the schema for each application.

So the question is: Is there a way to specify the schema of a data file in Hadoop/HDFS or do I need to specify the schema for the data file in each application?

Answer 1

It looks like you are looking for Apache Avro. With Avro your schema is embedded in your data, so you can read it without having to worry about schema issues and it makes schema evolution really easy.

The great thing about Avro is that it is completely integrated in Hadoop and you can use it with a lot of Hadoop sub-projects like Pig and Hive.

For example with Pig you could do:

EMP = LOAD 'myfile.avro' using AvroStorage();

I would advise looking at the documentation for AvroStorage for more details.

You can also work with Avro with Hive as described here but I have not used that personally but it should work the same way.

Answer 2

What you need is HCatalog which is

"Apache HCatalog is a table and storage management service for data created using Apache Hadoop.

This includes:

• Providing a shared schema and data type mechanism.
• Providing a table abstraction so that users need not be concerned with where or how their data is stored.
• Providing interoperability across data processing tools such as Pig, Map Reduce, and Hive."

You can take a look at the "data flow example" in the docs to see exactly the scenario you are talking about