TUNANのBlogHomeArchivesTagsCategories
Spark使用Yarn模式解决Jar乱飞情况
|Spark
Word count: 97|Reading time: 1 min

  1. 在本地创建zip文件

    # 在spark的jars目录下
    zip spark.zip ./*

  2. HDFS上创建存放spark jar的目录

    hadoop fs -mkdir -p  /spark-yarn/jars

  3. 将$SPARK_HOME/jars下的spark.zip包上传至刚建的HDFS路径

    [hadoop@hadoop jars]$ hadoop fs -put ./spark.zip /spark-yarn/jars/

  4. 在 spark-defaults.conf中添加(也可以在启动的时候–conf指定)

    spark.yarn.archive=hdfs://hadoop:9000/spark-yarn/jars/spark.zip

  5. 查看Spark log

    yarn log -applicationID xxx
Author: Tunan
Link: http://yerias.github.io/2019/10/05/spark/5/
Copyright Notice: All articles in this blog are licensed under CC BY-NC-SA 4.0 unless stating additionally.
Spark