QFS Hadoop Shim Doesn't Honor Filesystem Create Paramters
When using the Hadoop shim to integrate Apache Spark with QFS, QFS doesn't honor any file creation parameters set via Spark. For example, if I add the following line to Spark's spark-defaults.conf configuration file:
Files created by Spark will still be created with the default Reed-Solomon encoding.
Based on some advice from , I was able to hack com.quantcast.qfs.hadoop.QFSImpl.java's as in this commit, which then enables QFS to honor any Spark configuration for QFS file creation.