Proposal: Implement CSVReader Configuration

[dixonwhitmire]

This proposal supports the addition of a new configuration model, CsvReaderParams, to the existing DataContract model.

CsvReaderParams collects the existing "csv reader params" within the General and FileDefinition, so that all "csv reader params" are managed within a single document. Collected parameters include:

• emptyFieldValues
• valueDelimiter
• convertColumnsToString
• headers
• skiprows

CsvReaderParams is implemented as an Optional field within the General and FileDefinition configurations. When CsvReaderParams is defined in both General and FileDefinition models, the FileDefinition.CsvReaderParams overrides values set in General.CsvReaderParams.

If CsvReaderParams is not configured within the General model, a default instance is created with appropriate defaults, based on the defaults for the 1.1.0 release.

• emptyFieldValues = None
• valueDelimiter = ","
• convertColumnsToString = True
• headers = None
• skipRows = None

Finally the existing fields will remain where they are for a "reasonable" amount of time to allow users to migrate to the new configuration model.

[hammadk373]

We have a requirement to configure some pandas reader params (chunk size to be specific) per fileDefinition. I think this ticket would be the correct place to implement that functionality.

@dixonwhitmire I will schedule a call with the maintainers to discuss how we want to implement this including support for arbitrary reader params.

[dixonwhitmire]

In our meeting today with @evbaron , @hammadk373 , @LisaWellman , @klwhaley we decided to opt for a dictionary based configuration. While we aren't locking down specific implementation details we will:

• Define our "default" csv reader configuration in a dictionary object.
• Load user provided csv reader config within a separate dictionary.
• Merge the two dictionaries with update() allowing the user provided config to override default settings.
• Provide documentation on default settings and how to override those defaults.

This solution isn't as tightly coupled to the reader, while still allowing us to have documented defaults. We can also use this solution for other Pandas compatible APIs such as Dask.