Я новичок в HDFS и пытаюсь импортировать данные из базы данных oracle 12c. У меня есть таблица EMP, ее нужно импортировать в hdfs, а также в таблицы hive.
Мои данные создаются внутри hdfs («/user/hdfs» создается папка «EMP»). Но когда я открываю редактор запросов куста и набираю «показать таблицы», я не вижу здесь никаких таблиц. Мне нужно, чтобы таблицы создавались и внутри HIVE.
Я запускаю следующие команды.
1. Since am running sqoop as root user
usermod -a -G supergroup hardik
2.
export SQOOP_HOME=/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop
export HIVE_HOME=/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hive
export HADOOP_CLASSPATH=/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/ojdbc7.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hive/lib/*
export HADOOP_USER_NAME=hdfs
3.
export PATH=$PATH:$HIVE_HOME/bin
Теперь я запускаю команду импорта SQOOP, и я получаю следующее на консоли
4.
sqoop import --connect jdbc:oracle:thin:@bigdatadev2:1521/orcl --username BDD1 --password oracle123 --table EMP --hive-import -m 1 --create-hive-table --hive-table EMP
[root@bigdatadev1 ~]# sqoop import --connect jdbc:oracle:thin:@bigdatadev2:1521/orcl --username BDD1 --password oracle123 --table EMP --hive-import -m 1 --create-hive-table --hive-table EMP
Warning: /opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
16/04/07 22:15:23 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.5.1
16/04/07 22:15:23 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
16/04/07 22:15:23 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override
16/04/07 22:15:23 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.
16/04/07 22:15:23 INFO oracle.OraOopManagerFactory: Data Connector for Oracle and Hadoop is disabled.
16/04/07 22:15:23 INFO manager.SqlManager: Using default fetchSize of 1000
16/04/07 22:15:23 INFO tool.CodeGenTool: Beginning code generation
16/04/07 22:15:24 INFO manager.OracleManager: Time zone has been set to GMT
16/04/07 22:15:24 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM EMP t WHERE 1=0
16/04/07 22:15:24 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce
Note: /tmp/sqoop-root/compile/fcb6484db042a7b4295d911956145a4e/EMP.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
16/04/07 22:15:25 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/fcb6484db042a7b4295d911956145a4e/EMP.jar
16/04/07 22:15:25 INFO manager.OracleManager: Time zone has been set to GMT
16/04/07 22:15:25 INFO manager.OracleManager: Time zone has been set to GMT
16/04/07 22:15:25 INFO mapreduce.ImportJobBase: Beginning import of EMP
16/04/07 22:15:25 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
16/04/07 22:15:25 INFO manager.OracleManager: Time zone has been set to GMT
16/04/07 22:15:26 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
16/04/07 22:15:26 INFO client.RMProxy: Connecting to ResourceManager at bigdata/10.103.25.39:8032
16/04/07 22:15:30 INFO db.DBInputFormat: Using read commited transaction isolation
16/04/07 22:15:30 INFO mapreduce.JobSubmitter: number of splits:1
16/04/07 22:15:30 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460040138373_0007
16/04/07 22:15:31 INFO impl.YarnClientImpl: Submitted application application_1460040138373_0007
16/04/07 22:15:31 INFO mapreduce.Job: The url to track the job: http://bigdata:8088/proxy/application_1460040138373_0007/
16/04/07 22:15:31 INFO mapreduce.Job: Running job: job_1460040138373_0007
16/04/07 22:15:37 INFO mapreduce.Job: Job job_1460040138373_0007 running in uber mode : false
16/04/07 22:15:37 INFO mapreduce.Job: map 0% reduce 0%
16/04/07 22:15:43 INFO mapreduce.Job: Task Id : attempt_1460040138373_0007_m_000000_0, Status : FAILED
Error: EMP : Unsupported major.minor version 52.0
16/04/07 22:15:56 INFO mapreduce.Job: Task Id : attempt_1460040138373_0007_m_000000_1, Status : FAILED
Error: EMP : Unsupported major.minor version 52.0
16/04/07 22:16:03 INFO mapreduce.Job: map 100% reduce 0%
16/04/07 22:16:04 INFO mapreduce.Job: Job job_1460040138373_0007 completed successfully
16/04/07 22:16:04 INFO mapreduce.Job: Counters: 31
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=137942
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=87
HDFS: Number of bytes written=12
HDFS: Number of read operations=4
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Failed map tasks=2
Launched map tasks=3
Other local map tasks=3
Total time spent by all maps in occupied slots (ms)=20742
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=20742
Total vcore-seconds taken by all map tasks=20742
Total megabyte-seconds taken by all map tasks=10619904
Map-Reduce Framework
Map input records=3
Map output records=3
Input split bytes=87
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=53
CPU time spent (ms)=2090
Physical memory (bytes) snapshot=207478784
Virtual memory (bytes) snapshot=2169630720
Total committed heap usage (bytes)=134217728
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=12
16/04/07 22:16:04 INFO mapreduce.ImportJobBase: Transferred 12 bytes in 38.6207 seconds (0.3107 bytes/sec)
16/04/07 22:16:04 INFO mapreduce.ImportJobBase: Retrieved 3 records.
16/04/07 22:16:05 INFO manager.OracleManager: Time zone has been set to GMT
16/04/07 22:16:05 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM EMP t WHERE 1=0
16/04/07 22:16:05 INFO hive.HiveImport: Loading uploaded data into Hive
Logging initialized using configuration in jar:file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/hive-common-1.1.0-cdh5.5.1.jar!/hive-log4j.properties
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. AlreadyExistsException(message:Table EMP already exists)
Я пробовал все варианты команд импорта sqoop, но ни один из них не преуспел. Сегодня я еще больше запутался. Пожалуйста помоги. Пожалуйста, не отмечайте это как дубликат.
