java执行spark查询hbase的jar包出现错误提示：ob aborted due to stage failure: Master removed our application: FAILED
浏览数：121 /
时间：2015年06月11日
执行java调用scala 打包后的jar时候出现异常
/14 23:57:08 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory 15/04/14 23:57:23 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory 15/04/14 23:57:38 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory 15/04/14 23:57:39 INFO AppClient$ClientActor: Executor updated: app-20150414235011-0003/9 is now EXITED (Command exited with code 1) 15/04/14 23:57:39 INFO SparkDeploySchedulerBackend: Executor app-20150414235011-0003/9 removed: Command exited with code 1 15/04/14 23:57:39 ERROR SparkDeploySchedulerBackend: Application has been killed. Reason: Master removed our application: FAILED 15/04/14 23:57:39 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 15/04/14 23:57:39 INFO TaskSchedulerImpl: Cancelling stage 0 15/04/14 23:57:39 INFO DAGScheduler: Failed to run count at SparkSelect03.scala:55 Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Master removed our application: FAILED at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1049) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1033) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1031) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1031) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:635) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:635) at scala.Option.foreach(Option.scala:236) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:635) at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1234) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498) at akka.actor.ActorCell.invoke(ActorCell.scala:456) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237) at akka.dispatch.Mailbox.run(Mailbox.scala:219) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
问题1：
/14 23:57:08 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory
15/04/14 23:57:23 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory
15/04/14 23:57:38 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memor
分析：这个是内存不足？
我spark-env.sh的配置文件信息如下
export JAVA_HOME=/home/hadoop/jdk1.7.0_75
export SCALA_HOME=/home/hadoop/scala-2.11.6
export HADOOP_HOME=/home/hadoop/hadoop-2.3.0-cdh5.0.2
export HADOOP_CONF_DIR=/home/hadoop/hadoop-2.3.0-cdh5.0.2/etc/hadoop
export SPARK_CLASSPATH=/home/hadoop/hbase-0.96.1.1-cdh5.0.2/lib/*
export SPARK_MASTER_IP=master
export SPARK_MASTER_PORT=17077
export SPARK_MASTER_WEBUI_PORT=18080
export SPARK_WORKER_CORES=1
export SPARK_WORKER_MEMORY=1g
export SPARK_WORKER_WEBUI_PORT=18081
export SPARK_WORKER_INSTANCES=1
问题2：
15/04/14 23:57:39 INFO DAGScheduler: Failed to run count at SparkSelect03.scala:55
这句话的代码：
val count = hbaseRDD.count()
println("HBase RDD Count:" + count)
hbaseRDD.cache()
问题3：
in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Master removed our application: FAILED
有遇到过类似的或者知道怎么解决的可以留言下
郑重声明：本站内容如果来自互联网及其他传播媒体，其版权均属原媒体及文章作者所有。转载目的在于传递更多信息及用于网络分享，并不代表本站赞同其观点和对其真实性负责，也不构成任何其他建议。
