Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Highest
    • Resolution: Done
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.2.0
    • Component/s: hadoop
    • Labels:

      Description

      I have three nodes, one has metaserver and spark master, the second one is chunkserver, the third one is spark slave.
      All these nodes has Linux qfs account, same uid and gid:

      qfs:x:502:503::/home/qfs:/bin/bash

      qfs servers and spark severs are running under qfs account to keep things simple.

      The metaserver config file(MetaServer.prp) has following:

      metaServer.rootDirUser = 502
      metaServer.rootDirGroup = 503
      metaServer.rootDirMode = 0755

      metaServer.defaultLoadUser = 502
      metaServer.defaultLoadGroup = 503
      metaServer.defaultLoadFileMode = 0644
      metaServer.defaultLoadDirMode = 0755

      I can run qfsshell on node 3 under qfs account and create a dir successfully, but when running spark-shell on node 3, still under qfs account, I ran into the problem.

      Interesting thing is, the /tmp/hive folder is created when I run my scala script in spark-shell, but cannot write to this folder then.

      How to fix this? Any log can help me to diagnose this kind of problem?

      scala> val df = spark.read.option("delimiter","\t").csv("/data/files/gsm*");
      java.lang.RuntimeException: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rwx-wx--x
      at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
      at org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:189)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
      at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:258)
      at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:359)
      at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:263)
      at org.apache.spark.sql.hive.HiveSharedState.metadataHive$lzycompute(HiveSharedState.scala:39)
      at org.apache.spark.sql.hive.HiveSharedState.metadataHive(HiveSharedState.scala:38)
      at org.apache.spark.sql.hive.HiveSharedState.externalCatalog$lzycompute(HiveSharedState.scala:46)
      at org.apache.spark.sql.hive.HiveSharedState.externalCatalog(HiveSharedState.scala:45)
      at org.apache.spark.sql.hive.HiveSessionState.catalog$lzycompute(HiveSessionState.scala:50)
      at org.apache.spark.sql.hive.HiveSessionState.catalog(HiveSessionState.scala:48)
      at org.apache.spark.sql.hive.HiveSessionState$$anon$1.<init>(HiveSessionState.scala:63)
      at org.apache.spark.sql.hive.HiveSessionState.analyzer$lzycompute(HiveSessionState.scala:63)
      at org.apache.spark.sql.hive.HiveSessionState.analyzer(HiveSessionState.scala:62)
      at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:49)
      at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:64)
      at org.apache.spark.sql.SparkSession.baseRelationToDataFrame(SparkSession.scala:382)
      at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:143)
      at org.apache.spark.sql.DataFrameReader.csv(DataFrameReader.scala:413)
      at org.apache.spark.sql.DataFrameReader.csv(DataFrameReader.scala:349)
      ... 48 elided
      Caused by: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rwx-wx--x
      at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:612)
      at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:554)
      at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:508)
      ... 71 more

      scala> :q

        Attachments

          Activity

            People

            • Assignee:
              kstinson Kevin Stinson
              Reporter:
              csfreebird Dean Chen
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: