Infer schema for NDJSON

[Smotrov]

I have a data in NDJSON format.
The thing is some fields there are optional and populated when needed.
As I understand, DF is using the first row only by default to infer schema. Than when it see that certain fields are missing in som of the next rows it shows an error Fail to merge schema field 'my_fieldname' because the from data_type ... does not equal ....

So is it possible to specify N rows which should be used to infer schema?

[jonahgao]

Could you provide a small reproducible example? I’m not familiar with this area, but I may be able to help with debugging.

[Jefffrey]

Should be able to use schema_infer_max_records from NdJsonReadOptions to specify how many rows to use for inference now:

datafusion/datafusion/core/src/datasource/file_format/options.rs

Lines 445 to 468 in d24eb4a

	/// Options that control the reading of Line-delimited JSON files (NDJson)
	///
	/// Note this structure is supplied when a datasource is created and
	/// can not not vary from statement to statement. For settings that
	/// can vary statement to statement see
	/// [`ConfigOptions`](crate::config::ConfigOptions).
	#[derive(Clone)]
	pub struct NdJsonReadOptions<'a> {
	/// The data source schema.
	pub schema: Option<&'a Schema>,
	/// Max number of rows to read from JSON files for schema inference if needed. Defaults to `DEFAULT_SCHEMA_INFER_MAX_RECORD`.
	pub schema_infer_max_records: usize,
	/// File extension; only files with this extension are selected for data input.
	/// Defaults to `FileType::JSON.get_ext().as_str()`.
	pub file_extension: &'a str,
	/// Partition Columns
	pub table_partition_cols: Vec<(String, DataType)>,
	/// File compression type
	pub file_compression_type: FileCompressionType,
	/// Flag indicating whether this file may be unbounded (as in a FIFO file).
	pub infinite: bool,
	/// Indicates how the file is sorted
	pub file_sort_order: Vec<Vec<SortExpr>>,
	}

(#18920 adds a nicer API to configure this setting)