Logo Questions Linux Laravel Mysql Ubuntu Git Menu
HTML CSS JAVASCRIPT SQL PYTHON PHP BOOTSTRAP JAVA JQUERY R React Kotlin
 

Different ways to import files into HDFS

Tags:

import

hadoop

hdfs

I want to know what are the different ways through which I can bring data into HDFS.

I am a newbie to Hadoop and was a java web developer till this time. I want to know if I have a web application that is creating log files, how can i import the log files into HDFS.

like image 505
Gaurav Avatar asked Sep 26 '15 06:09

Gaurav

People also ask

How do you import data from Rdbms to hdfs?

You create a single Sqoop import command that imports data from diverse data sources, such as a relational database, into HDFS. You enter the Sqoop import command on the command line of your cluster to import data from a data source into HDFS.

Which file are imported for Hadoop?

You can import the following file types from a Hadoop Distributed File System (HDFS): . avro, . csv, . json, .

1 Answers

There are lot's of ways on how you can ingest data into HDFS, let me try to illustrate them here:

  1. hdfs dfs -put - simple way to insert files from local file system to HDFS
  2. HDFS Java API
  3. Sqoop - for bringing data to/from databases
  4. Flume - streaming files, logs
  5. Kafka - distributed queue, mostly for near-real time stream processing
  6. Nifi - incubating project at Apache for moving data into HDFS without making lots of changes

Best solution for bringing web application logs to HDFS is through Flume.

like image 60
Ashrith Avatar answered Oct 19 '22 04:10

Ashrith
Sign in to Comment

Related questions

Cassandra timeout during read query at consistency ONE (1 responses were required but only 0 replica responded)
Elastic Map Reduce External Jars
Pydoop on Amazon EMR
how can i work with large number of small files in hadoop?
Change Block size of existing files in Hadoop
Change Hive Database location
How to use the ResourceManager web interface as an user
Why does YARN job not transition to RUNNING state?
AWS connection timeout when running Spark job on EMR
JAVA_HOME not setting
Mapper and Reducer for K means algorithm in Hadoop in Java
Launch a mapreduce job from eclipse
Where are my files(dir) stored when i used the hadoop fs -mkdir?
HDFS File Checksum
How to convert String object to IntWritable Object in Hadoop
package org.apache.hadoop.conf does not exist after setting classpath
unable to create hive table with primary key
HADOOP / YARN - Are the ResourceManager and the hdfs NameNode always installed on the same host?
Hive query stuck at 99%
What is the difference between Statement.setMaxRows vs Statement.setFetchsize in Hive

Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!