I'm using Spark Standalone Mode tutorial page to install Spark in Standalone mode.
1- I have started a master by:
./sbin/start-master.sh
2- I have started a worker by:
./bin/spark-class org.apache.spark.deploy.worker.Worker spark://ubuntu:7077
Note: spark://ubuntu:7077
is my master name, which I can see it in Master-WebUI
.
Problem: By second command, a worker started successfully. But it couldn't associate with master. It tries repeatedly and then give this message:
15/02/08 11:30:04 WARN Remoting: Tried to associate with unreachable remote address [akka.tcp://sparkMaster@ubuntu:7077]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: Connection refused: ubuntu/127.0.1.1:7077
15/02/08 11:30:04 INFO RemoteActorRefProvider$RemoteDeadLetterActorRef: Message [org.apache.spark.deploy.DeployMessages$RegisterWorker] from Actor[akka://sparkWorker/user/Worker#-1296628173] to Actor[akka://sparkWorker/deadLetters] was not delivered. [20] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
15/02/08 11:31:15 ERROR Worker: All masters are unresponsive! Giving up.
What is the problem?
Thanks
I usually start from spark-env.sh template. And I set, properties that I need. For simple cluster you need:
Then, create a file called "slaves" in the same directory as spark-env.sh and slaves ip's (one per line). Assure you reach all slaves through ssh.
Finally, copy this configuration in every machine of your cluster. Then start the entire cluster executing start-all.sh script and try spark-shell to check your configuration.
> sbin/start-all.sh
> bin/spark-shell
Thanks for answer. I have SPARK_MASTER_IP=192.168.84.128 in my spark-env.sh file and 192.168.84.129 in my slaves file. I copy these two files to the worker and master's conf folder. but when I execute ./start-all.sh in my master system I receive this error: starting org.apache.spark.deploy.master.Master, logging to /home/omid/Programs/spark-1.2.0/sbin/../logs/spark-omid-org.apache.spark.deploy.master.Master-1-ubuntu.out 192.168.84.129: ssh: connect to host 192.168.84.129 port 22: Connection refused
I can ping 192.168.84.129 from Master system. Isn't it sufficient? What is about ssh connection?
No, machines must be able to connect through ssh without keys. Install open-ssh, generate ssh keys (private and public) and add them to the authorized_keys for all of them.
Oh, Thanks very much. I can see my workers in my master WebUI. It's great.
Is it necessary that workers need to be started in their respective machines manually?