Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to use sqoop to export the default hive delimited output?

Tags:

hadoop

hive

sqoop

I have a hive query:

insert override directory /x
select ...

Then I'm try to export the data with sqoop

sqoop export --connect jdbc:mysql://mysqlm/site --username site --password site --table x_data --export-dir /x  --input-fields-terminated-by 0x01 --lines-terminated-by '\n'

But this seems to fail to parse the fields according to delimiter What am I missing? I think the --input-fields-terminated-by 0x01 part doesn't work as expected?

I do not want to create additional tables in hive that contains the query results.

stack trace:

 2013-09-24 05:39:21,705 ERROR org.apache.sqoop.mapreduce.TextExportMapper: Exception: 
 java.lang.NumberFormatException: For input string: "9-2"
    at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
    at java.lang.Integer.parseInt(Integer.java:458)
 ...

The vi view of output

16-09-2013 23^A1182^A-1^APub_X^A21782^AIT^A1^A0^A0^A0^A0^A0.0^A0.0^A0.0
16-09-2013 23^A1182^A6975^ASoMo Audience  Corp^A2336143^AUS^A1^A1^A0^A0^A0^A0.2^A0.0^A0.0
16-09-2013 23^A1183^A-1^APub_UK, Inc.^A1564001^AGB^A1^A0^A0^A0^A0^A0.0^A0.0^A0.0
17-09-2013 00^A1120^A-1^APub_US^A911^A--^A181^A0^A0^A0^A0^A0.0^A0.0^A0.0
like image 543
Julias Avatar asked Sep 24 '13 10:09

Julias


People also ask

What is the default delimiter in Sqoop?

You can change delimiter in SQOOP command. By default, it takes delimiter as a tab but if you want to explicitly define it you need to use this command.

How do I export data from Hive table to mysql using Sqoop?

To export data into MySQL from HDFS, perform the following steps: Step 1: Create a database and table in the hive. Step 2: Insert data into the hive table. Step 3: Create a database and table in MySQL in which data should be exported.

What is the default file format to import data using Apache Sqoop?

i) Delimited Text File Format This is the default file format to import data using Sqoop. This file format can be explicitly specified using the –as-textfile argument to the import command in Sqoop.


1 Answers

Using

--input-fields-terminated-by '\001' --lines-terminated-by '\n'

as flags in the sqoop export command seems to do the trick for me.

So, in your example, the full command would be:

sqoop export --connect jdbc:mysql://mysqlm/site --username site --password site --table x_data --export-dir /x  --input-fields-terminated-by '\001' --lines-terminated-by '\n'
like image 194
Mark Grover Avatar answered Oct 23 '22 16:10

Mark Grover