René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"java.lang.RuntimeException: Error while encoding: java.lang.ArrayIndexOutOfBoundsException: 1","articleBody":"I got an error when I was trying to join two datasets which are from postgres and csv file\r\nthe error message is like this:\r\norg.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 14.0 failed 1 times, most recent failure: Lost task 0.0 in stage 14.0 (TID 14, localhost, executor driver): java.lang.RuntimeException: Error while encoding: java.lang.ArrayIndexOutOfBoundsException: 1\r\nstaticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, targetString), StringType), true, false) AS targetString#205\r\nstaticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, deviceName), StringType), true, false) AS deviceName#206\r\nstaticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, alarmDetectionCode), StringType), true, false) AS alarmDetectionCode#207\r\n\tat org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:292)\r\n\tat org.apache.spark.sql.SparkSession$$anonfun$4.apply(SparkSession.scala:593)\r\n\tat org.apache.spark.sql.SparkSession$$anonfun$4.apply(SparkSession.scala:593)\r\n\tat scala.collection.Iterator$$anon$11.next(Iterator.scala:410)\r\n\tat scala.collection.Iterator$$anon$11.next(Iterator.scala:410)\r\n\tat scala.collection.Iterator$$anon$11.next(Iterator.scala:410)\r\n\tat org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:149)\r\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)\r\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)\r\n\tat org.apache.spark.scheduler.Task.run(Task.scala:121)\r\n\tat org.apache.spark.executor.Executor$TaskRunner$$anonfun$11.apply(Executor.scala:407)\r\n\tat org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:413)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\r\n\tat java.lang.Thread.run(Thread.java:748)\r\nCaused by: java.lang.ArrayIndexOutOfBoundsException: 1\r\n\tat org.apache.spark.sql.catalyst.expressions.GenericRow.get(rows.scala:174)\r\n\tat org.apache.spark.sql.Row$class.isNullAt(Row.scala:191)\r\n\tat org.apache.spark.sql.catalyst.expressions.GenericRow.isNullAt(rows.scala:166)\r\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.StaticInvoke_1$(generated.java:94)\r\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(generated.java:36)\r\n\tat org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:289)\r\n\t... 15 more\r\n\r\nDriver stacktrace:\r\n\tat org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1890)\r\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1878)\r\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1877)\r\n\tat scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)\r\n\tat scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)\r\n\tat org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1877)\r\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:929)\r\n\tat org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:929)\r\n\tat scala.Option.foreach(Option.scala:257)\r\n\tat org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:929)\r\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2111)\r\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2060)\r\n\tat org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2049)\r\n\tat org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)\r\n\tat org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:740)\r\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:2073)\r\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:2094)\r\n\tat org.apache.spark.SparkContext.runJob(SparkContext.scala:2113)\r\n\tat org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:365)\r\n\tat org.apache.spark.sql.execution.CollectLimitExec.executeCollect(limit.scala:38)\r\n\tat org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$collectFromPlan(Dataset.scala:3383)\r\n\tat org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2544)\r\n\tat org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2544)\r\n\tat org.apache.spark.sql.Dataset$$anonfun$53.apply(Dataset.scala:3364)\r\n\tat org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)\r\n\tat org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)\r\n\tat org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)\r\n\tat org.apache.spark.sql.Dataset.withAction(Dataset.scala:3363)\r\n\tat org.apache.spark.sql.Dataset.head(Dataset.scala:2544)\r\n\tat org.apache.spark.sql.Dataset.take(Dataset.scala:2758)\r\n\tat org.apache.spark.sql.Dataset.getRows(Dataset.scala:254)\r\n\tat org.apache.spark.sql.Dataset.showString(Dataset.scala:291)\r\n\tat org.apache.spark.sql.Dataset.show(Dataset.scala:745)\r\n\tat org.apache.spark.sql.Dataset.show(Dataset.scala:704)\r\n\tat org.apache.spark.sql.Dataset.show(Dataset.scala:713)\r\n\tat jp.co.nec.necdas.commons.customize.service.dataset.ALMTriggerProcessLogic.process(ALMTriggerProcessLogic.java:295)\r\n\tat jp.co.nec.necdas.commons.customize.service.dataset.ALMTriggerProcessLogic.execute(ALMTriggerProcessLogic.java:179)\r\n\tat jp.co.nec.necdas.commons.customize.service.dataset.ALMTriggerProcessLogic.execute(ALMTriggerProcessLogic.java:1)\r\n\tat jp.co.nec.necdas.commons.service.StandardExecutorService.execute(StandardExecutorService.java:65)\r\n\tat jp.co.nec.necdas.commons.service.StandardExecutorServiceTest.testExecute(StandardExecutorServiceTest.java:26)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n\tat java.lang.reflect.Method.invoke(Method.java:498)\r\n\tat org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)\r\n\tat org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)\r\n\tat org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)\r\n\tat org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)\r\n\tat org.springframework.test.context.junit4.statements.RunBeforeTestExecutionCallbacks.evaluate(RunBeforeTestExecutionCallbacks.java:73)\r\n\tat org.springframework.test.context.junit4.statements.RunAfterTestExecutionCallbacks.evaluate(RunAfterTestExecutionCallbacks.java:83)\r\n\tat org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75)\r\n\tat org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86)\r\n\tat org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84)\r\n\tat org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)\r\n\tat org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:251)\r\n\tat org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:97)\r\n\tat org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)\r\n\tat org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)\r\n\tat org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)\r\n\tat org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)\r\n\tat org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)\r\n\tat org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)\r\n\tat org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)\r\n\tat org.junit.runners.ParentRunner.run(ParentRunner.java:363)\r\n\tat org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:190)\r\n\tat org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)\r\n\tat org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)\r\n\tat org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:538)\r\n\tat org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:760)\r\n\tat org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:460)\r\n\tat org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:206)\r\nCaused by: java.lang.RuntimeException: Error while encoding: java.lang.ArrayIndexOutOfBoundsException: 1\r\nstaticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, targetString), StringType), true, false) AS targetString#205\r\nstaticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, deviceName), StringType), true, false) AS deviceName#206\r\nstaticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, alarmDetectionCode), StringType), true, false) AS alarmDetectionCode#207\r\n\tat org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:292)\r\n\tat org.apache.spark.sql.SparkSession$$anonfun$4.apply(SparkSession.scala:593)\r\n\tat org.apache.spark.sql.SparkSession$$anonfun$4.apply(SparkSession.scala:593)\r\n\tat scala.collection.Iterator$$anon$11.next(Iterator.scala:410)\r\n\tat scala.collection.Iterator$$anon$11.next(Iterator.scala:410)\r\n\tat scala.collection.Iterator$$anon$11.next(Iterator.scala:410)\r\n\tat org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:149)\r\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)\r\n\tat org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)\r\n\tat org.apache.spark.scheduler.Task.run(Task.scala:121)\r\n\tat org.apache.spark.executor.Executor$TaskRunner$$anonfun$11.apply(Executor.scala:407)\r\n\tat org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)\r\n\tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:413)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\r\n\tat java.lang.Thread.run(Thread.java:748)\r\nCaused by: java.lang.ArrayIndexOutOfBoundsException: 1\r\n\tat org.apache.spark.sql.catalyst.expressions.GenericRow.get(rows.scala:174)\r\n\tat org.apache.spark.sql.Row$class.isNullAt(Row.scala:191)\r\n\tat org.apache.spark.sql.catalyst.expressions.GenericRow.isNullAt(rows.scala:166)\r\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.StaticInvoke_1$(generated.java:94)\r\n\tat org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(generated.java:36)\r\n\tat org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:289)\r\n\t... 15 more\r\n\r\nIt looked like mismatch happened when spark application was joining two dataset \r\nwith different schema , but I don't know how it happened.\r\n\r\n\r\nMy java code:\r\nDataset\u003cRow\u003e result = null;\r\nresult = deviceInfoDataset.join(searchInfo,deviceInfoDataset.col(\"deviceName\").equalTo(searchInfo.col(\"deviceName\"))); \r\nresult.show();\r\n\r\nDataset schema:\r\ndevice\r\n+--------+----------+----------+\r\n|ctgry_cd|deviceInfo|deviceName|\r\n+--------+----------+----------+\r\n\r\nsearchinfo\r\n\r\n+------------+----------+------------------+\r\n|targetString|deviceName|alarmDetectionCode|\r\n+------------+----------+------------------+","author":{"url":"https://github.com/yuanwuab","@type":"Person","name":"yuanwuab"},"datePublished":"2019-10-11T02:35:09.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/305/CaffeOnSpark/issues/305"}