sqoop导入问题


#1

使用sqoop将DB2内容导入hdfs中,导入时一部分数据库表会出现如下错误:

[east@cbrcxm06 test]$ sqoop import --connect jdbc:db2://10.72.0.61:50000/eastst4 -username NYYH -password 660426Zw --table T_KP_JJKXX -m 1 --fields-terminated-by “\001” --target-dir /user/east/tmp/NYYH/JJKXX/ --lines-terminated-by “\n”
Warning: /home/east/TDH-Client/sqoop/…/hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/east/TDH-Client/sqoop/…/accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
2018-04-13 15:39:11,212 INFO [main] sqoop.Sqoop (Sqoop.java:(92)) - Running Sqoop version: 1.4.5-transwarp-5.0.0
2018-04-13 15:39:11,228 WARN [main] tool.BaseSqoopTool (BaseSqoopTool.java:applyCredentialsOptions(1009)) - Setting your password on the command-line is insecure. Consider using -P instead.
2018-04-13 15:39:11,430 INFO [main] manager.SqlManager (SqlManager.java:initOptionDefaults(98)) - Using default fetchSize of 1000
2018-04-13 15:39:11,430 INFO [main] tool.CodeGenTool (CodeGenTool.java:generateORM(92)) - Beginning code generation
2018-04-13 15:39:21,875 INFO [main] manager.SqlManager (SqlManager.java:execute(749)) - Executing SQL statement: SELECT t.* FROM T_KP_JJKXX AS t WHERE 1=0
2018-04-13 15:39:21,943 INFO [main] manager.SqlManager (SqlManager.java:execute(749)) - Executing SQL statement: SELECT t.* FROM T_KP_JJKXX AS t WHERE 1=0
2018-04-13 15:39:21,980 INFO [main] orm.CompilationManager (CompilationManager.java:findHadoopJars(94)) - HADOOP_MAPRED_HOME is /home/east/TDH-Client/hadoop/hadoop-mapreduce
注: /tmp/sqoop-east/compile/e2dffaebed0d6469fcff08d70577e101/T_KP_JJKXX.java使用或覆盖了已过时的 API。
注: 有关详细信息, 请使用 -Xlint:deprecation 重新编译。
2018-04-13 15:39:23,768 INFO [main] orm.CompilationManager (CompilationManager.java:jar(330)) - Writing jar file: /tmp/sqoop-east/compile/e2dffaebed0d6469fcff08d70577e101/T_KP_JJKXX.jar
2018-04-13 15:39:23,803 INFO [main] mapreduce.ImportJobBase (ImportJobBase.java:runImport(225)) - Beginning import of T_KP_JJKXX
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/east/TDH-Client/hadoop/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/east/TDH-Client/hyperbase/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/east/TDH-Client/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2018-04-13 15:39:24,067 INFO [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1181)) - mapred.jar is deprecated. Instead, use mapreduce.job.jar
2018-04-13 15:39:24,072 INFO [main] manager.SqlManager (SqlManager.java:execute(749)) - Executing SQL statement: SELECT t.* FROM T_KP_JJKXX AS t WHERE 1=0
2018-04-13 15:39:24,682 INFO [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1181)) - mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
2018-04-13 15:39:29,730 INFO [main] client.ConfiguredRMFailoverProxyProvider (ConfiguredRMFailoverProxyProvider.java:performFailover(100)) - Failing over to rm2
2018-04-13 15:39:50,213 INFO [main] db.DBInputFormat (DBInputFormat.java:setTxIsolation(192)) - Using read commited transaction isolation
2018-04-13 15:39:50,458 INFO [main] mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(198)) - number of splits:1
2018-04-13 15:39:50,732 INFO [main] mapreduce.JobSubmitter (JobSubmitter.java:printTokens(287)) - Submitting tokens for job: job_1523030024488_0015
2018-04-13 15:39:52,393 INFO [main] impl.YarnClientImpl (YarnClientImpl.java:submitApplication(273)) - Submitted application application_1523030024488_0015
2018-04-13 15:39:52,464 INFO [main] mapreduce.Job (Job.java:submit(1294)) - The url to track the job: http://cbrcxm02:8088/proxy/application_1523030024488_0015/
2018-04-13 15:39:52,465 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1339)) - Running job: job_1523030024488_0015
2018-04-13 15:40:16,674 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1360)) - Job job_1523030024488_0015 running in uber mode : false
2018-04-13 15:40:16,678 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1367)) - map 0% reduce 0%
2018-04-13 15:40:59,608 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1367)) - map 100% reduce 0%
2018-04-13 15:41:25,805 INFO [main] mapreduce.Job (Job.java:printTaskEvents(1406)) - Task Id : attempt_1523030024488_0015_m_000000_0, Status : FAILED
Error: java.io.IOException: SQLException in nextKeyValue
at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:277)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:556)
at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1891)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: com.ibm.db2.jcc.b.SqlException: [jcc][t4][1065][12306][3.53.70] Caught java.io.CharConversionException. See attached Throwable for details. ERRORCODE=-4220, SQLSTATE=null
at com.ibm.db2.jcc.b.bd.a(bd.java:660)
at com.ibm.db2.jcc.b.bd.a(bd.java:60)
at com.ibm.db2.jcc.b.bd.a(bd.java:112)
at com.ibm.db2.jcc.b.gc.a(gc.java:1807)
at com.ibm.db2.jcc.b.gc.n(gc.java:503)
at com.ibm.db2.jcc.b.gc.F(gc.java:1285)
at com.ibm.db2.jcc.b.ul.d(ul.java:1035)
at com.ibm.db2.jcc.b.ul.getString(ul.java:1013)
at org.apache.sqoop.lib.JdbcWritableBridge.readString(JdbcWritableBridge.java:71)
at com.cloudera.sqoop.lib.JdbcWritableBridge.readString(JdbcWritableBridge.java:61)
at T_KP_JJKXX.readFields(T_KP_JJKXX.java:254)
at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:244)
… 12 more
Caused by: java.nio.charset.MalformedInputException: Input length = 21981
at com.ibm.db2.jcc.b.v.a(v.java:19)
at com.ibm.db2.jcc.b.gc.a(gc.java:1803)
… 20 more
Caused by: sun.io.MalformedInputException
at sun.io.ByteToCharUTF8.convert(ByteToCharUTF8.java:125)
at com.ibm.db2.jcc.b.v.a(v.java:16)
… 21 more

2018-04-13 15:41:26,840 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1367)) - map 0% reduce 0%
2018-04-13 15:42:02,082 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1367)) - map 100% reduce 0%
2018-04-13 15:42:09,136 INFO [main] mapreduce.Job (Job.java:printTaskEvents(1406)) - Task Id : attempt_1523030024488_0015_m_000000_1, Status : FAILED
Error: java.io.IOException: SQLException in nextKeyValue
at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:277)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:556)
at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1891)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: com.ibm.db2.jcc.b.SqlException: [jcc][t4][1065][12306][3.53.70] Caught java.io.CharConversionException. See attached Throwable for details. ERRORCODE=-4220, SQLSTATE=null
at com.ibm.db2.jcc.b.bd.a(bd.java:660)
at com.ibm.db2.jcc.b.bd.a(bd.java:60)
at com.ibm.db2.jcc.b.bd.a(bd.java:112)
at com.ibm.db2.jcc.b.gc.a(gc.java:1807)
at com.ibm.db2.jcc.b.gc.n(gc.java:503)
at com.ibm.db2.jcc.b.gc.F(gc.java:1285)
at com.ibm.db2.jcc.b.ul.d(ul.java:1035)
at com.ibm.db2.jcc.b.ul.getString(ul.java:1013)
at org.apache.sqoop.lib.JdbcWritableBridge.readString(JdbcWritableBridge.java:71)
at com.cloudera.sqoop.lib.JdbcWritableBridge.readString(JdbcWritableBridge.java:61)
at T_KP_JJKXX.readFields(T_KP_JJKXX.java:254)
at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:244)
… 12 more
Caused by: java.nio.charset.MalformedInputException: Input length = 4306
at com.ibm.db2.jcc.b.v.a(v.java:19)
at com.ibm.db2.jcc.b.gc.a(gc.java:1803)
… 20 more
Caused by: sun.io.MalformedInputException
at sun.io.ByteToCharUTF8.convert(ByteToCharUTF8.java:125)
at com.ibm.db2.jcc.b.v.a(v.java:16)
… 21 more

2018-04-13 15:42:10,144 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1367)) - map 0% reduce 0%
2018-04-13 15:42:39,355 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1367)) - map 100% reduce 0%
2018-04-13 15:42:42,379 INFO [main] mapreduce.Job (Job.java:printTaskEvents(1406)) - Task Id : attempt_1523030024488_0015_m_000000_2, Status : FAILED
Error: java.io.IOException: SQLException in nextKeyValue
at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:277)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:556)
at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1891)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: com.ibm.db2.jcc.b.SqlException: [jcc][t4][1065][12306][3.53.70] Caught java.io.CharConversionException. See attached Throwable for details. ERRORCODE=-4220, SQLSTATE=null
at com.ibm.db2.jcc.b.bd.a(bd.java:660)
at com.ibm.db2.jcc.b.bd.a(bd.java:60)
at com.ibm.db2.jcc.b.bd.a(bd.java:112)
at com.ibm.db2.jcc.b.gc.a(gc.java:1807)
at com.ibm.db2.jcc.b.gc.n(gc.java:503)
at com.ibm.db2.jcc.b.gc.F(gc.java:1285)
at com.ibm.db2.jcc.b.ul.d(ul.java:1035)
at com.ibm.db2.jcc.b.ul.getString(ul.java:1013)
at org.apache.sqoop.lib.JdbcWritableBridge.readString(JdbcWritableBridge.java:71)
at com.cloudera.sqoop.lib.JdbcWritableBridge.readString(JdbcWritableBridge.java:61)
at T_KP_JJKXX.readFields(T_KP_JJKXX.java:254)
at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:244)
… 12 more
Caused by: java.nio.charset.MalformedInputException: Input length = 4306
at com.ibm.db2.jcc.b.v.a(v.java:19)
at com.ibm.db2.jcc.b.gc.a(gc.java:1803)
… 20 more
Caused by: sun.io.MalformedInputException
at sun.io.ByteToCharUTF8.convert(ByteToCharUTF8.java:125)
at com.ibm.db2.jcc.b.v.a(v.java:16)
… 21 more

2018-04-13 15:42:43,387 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1367)) - map 0% reduce 0%
2018-04-13 15:43:11,566 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1367)) - map 100% reduce 0%
2018-04-13 15:43:16,608 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1380)) - Job job_1523030024488_0015 failed with state FAILED due to: Task failed task_1523030024488_0015_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

2018-04-13 15:43:16,764 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1385)) - Counters: 8
Job Counters
Failed map tasks=4
Launched map tasks=4
Other local map tasks=4
Total time spent by all maps in occupied slots (ms)=162816
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=162816
Total vcore-milliseconds taken by all map tasks=162816
Total megabyte-milliseconds taken by all map tasks=166723584
2018-04-13 15:43:16,776 WARN [main] mapreduce.Counters (AbstractCounters.java:getGroup(234)) - Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
2018-04-13 15:43:16,777 INFO [main] mapreduce.ImportJobBase (ImportJobBase.java:runJob(183)) - Transferred 0 bytes in 232.0679 seconds (0 bytes/sec)
2018-04-13 15:43:16,785 WARN [main] mapreduce.Counters (AbstractCounters.java:getGroup(234)) - Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2018-04-13 15:43:16,785 INFO [main] mapreduce.ImportJobBase (ImportJobBase.java:runJob(185)) - Retrieved 0 records.
2018-04-13 15:43:16,785 ERROR [main] tool.ImportTool (ImportTool.java:run(617)) - Error during import: Import job failed!


#2


可以在参考这篇文章对操作流程进行比对


#3

请问您知道tdh内sqoop驱动的路径么


#4

jar包需要放在每台机器的/usr/lib/sqoop/lib 下面,


#6

我的数据库时db2的,不是用mysql的jar包吧?不是应该用db2jcc.jar?


#7

对,需要用ODBC的jar包


#8

问题得到解决了吗


#9

尚未解决。
昨日测试发现,该无法导入的数据库表,按某字段分区作视图分视图导入,
部分视图能够导入,另一部分报错无法导入。


#10

db2的jdbc驱动与数据库版本不符所致。

--db2 版本查看
SELECT service_level, fixpack_num FROM TABLE
  (sysproc.env_get_inst_info()) as INSTANCEINFO;