When using the Hadoop shim to integrate Apache Spark with QFS, QFS doesn't honor any file creation parameters set via Spark. For example, if I add the following line to Spark's spark-defaults.conf configuration file:
Files created by Spark will still be created with the default Reed-Solomon encoding.
Based on some advice from , I was able to hack com.quantcast.qfs.hadoop.QFSImpl.java's as in this commit, which then enables QFS to honor any Spark configuration for QFS file creation.
Thanks,