QFS Hadoop Shim Doesn't Honor Filesystem Create Paramters

Description

When using the Hadoop shim to integrate Apache Spark with QFS, QFS doesn't honor any file creation parameters set via Spark. For example, if I add the following line to Spark's spark-defaults.conf configuration file:

Files created by Spark will still be created with the default Reed-Solomon encoding.

Based on some advice from , I was able to hack com.quantcast.qfs.hadoop.QFSImpl.java's as in this commit, which then enables QFS to honor any Spark configuration for QFS file creation.

Assignee

Michael Ovsiannikov

Reporter

Michael Kamprath

Labels

None

Components

Priority

Medium
Configure