Menu
As per my understanding sqoop is used to import or export table/data from the Database to HDFS or Hive or HBASE.
And we can directly import a single table or list of tables. Internally mapreduce program (i think only map task) will run.
My doubt is what is sqoop direct and what when to go with sqoop direct option?
633
By default, Sqoop uses JDBC to connect to the database. However, depending on the database, there may be a faster, database-specific connector available, which you can use by using the --direct option. So, you go with --direct option when you want to use a different database connector than the default.
Sqoop has two main functions: importing and exporting. Importing transfers structured data into HDFS; exporting moves this data from Hadoop to external databases in the cloud or on-premises. Importing involves Sqoop assessing the external database's metadata before mapping it to Hadoop.
when we don't mention the number of mappers while transferring the data from RDBMS to HDFS file system sqoop will use default number of mapper 4. Sqoop imports data in parallel from most database sources. Sqoop only uses mappers as it does parallel import and export.
Sqoop supports two types of incremental imports: append and lastmodified . You can use the --incremental argument to specify the type of incremental import to perform. You should specify append mode when importing a table where new rows are continually being added with increasing row id values.
Just read the Sqoop documentation!
Some databases can perform imports in a more high-performance fashion by using database-specific data movement tools (...)
Some databases provides a direct mode for exports as well (...)
Details about use of direct mode with each specific RDBMS, installation requirements, available options and limitations can be found in Section 25
Bottom line: "direct mode" means different things for different databases.
For MySQL or PostgreSQL it relates to bulk loader/unloader utilities (i.e. completetely bypassing JDBC); while for Oracle it relates to "direct path INSERT" i.e. with JDBC but in a non-transactional mode (so you'd better use a temp table, or you might end up with duplicates in a PK and a corrupt table).
90
To be short and precise,its the mode for fast import which doesn't runs any mappers or reducers.
sqoop import --connect jdbc:mysql://db.foo.com/corp --table EMPLOYEES --direct
Notes:
--direct is only supported in mysql and postgresql.BLOB, CLOB, or LONGVARBINARY columns.
40
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
