Fix/sparse schema and allow column drops

[hentzthename]

Hi Nico, side note, thanks for your work on the _dlt_loads table, I was looking for it, and all I had to do was install your latest release 😄

The 'sparse data' problem may sound similar to my previous PR: #10

But that PR addressed the scenario where dlt didn't know about the existing wide schema in a fresh container. Once the schema is known (either from _dlt_version or derivation), I discovered these downstream problems:

Problems

When ingesting sparse data (subsequent runs with fewer columns than the established schema), 3 problems surface:

1. False SchemaEvolutionError -- validate_schema_changes raises when columns are "dropped" (present in table but absent in incoming data), even though the incoming data is not requesting a schema change. The columns should remain in the Iceberg table and new rows should receive nulls.

   SchemaEvolutionError: Schema evolution validation failed:
     - Columns dropped (not safe): col1, col2, ...
   

2. cast_table_safe crashes on missing columns -- Even if the error above were bypassed, cast_table_safe calls table.select(target_field_names) which raises a KeyError for columns in the target Iceberg schema that don't exist in the source Arrow table. validate_cast already documents the correct intent -- "Field X exists in target but not in source (will be null)" -- but the cast logic never followed through.

3. allow_column_drops=True was a no-op -- dropped_fields was computed and validated but never passed to apply_schema_evolution, so columns were never actually removed from the Iceberg schema regardless of the flag value.

---

Solution

• Sparse data (allow_column_drops=False, default): the Iceberg table schema stays unchanged and new rows receive null for columns they don't contain.
• Column drops (allow_column_drops=True): columns missing from incoming data are removed from the Iceberg schema via update.delete_column().
• No changes to destination.py -- allow_column_drops=False remains the correct default.

---

Changes

schema_evolution.py

Function	Change
validate_schema_changes	Removed the SchemaEvolutionError for dropped columns. Neither case warrants an error -- allow_column_drops=True removes columns via apply_schema_evolution, and allow_column_drops=False leaves them in the schema with nulls filled at write time.
apply_schema_evolution	Added dropped_fields parameter. When provided, calls update.delete_column() for each field -- the actual implementation of allow_column_drops=True that was previously missing.
evolve_schema_if_needed	Logs sparse columns as a warning when allow_column_drops=False. Passes dropped_fields to apply_schema_evolution only when allow_column_drops=True. Returns early without schema changes for the sparse case.

schema_casting.py

Function	Change
cast_table_safe	Before table.select(target_field_names), adds a null column (pa.nulls) for any field in the target schema missing from the source table. Completes what validate_cast already documents as the intended behavior.

---

Tests

• New test_sparse_schema.py covering all three problems
• Existing test_schema_evolution.py updated to reflect corrected behavior

[nicosuave]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d21214d771

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Reject omitted required columns instead of null-filling

When the existing Iceberg schema has a required column (for example from a dlt column with nullable=False) and a later sparse batch omits it, this path appends an all-null array for that non-nullable target field. validate_cast only treats missing target fields as warnings, so even strict_casting=True will proceed and produce rows that violate the required Iceberg schema, leading to append failures or invalid data. Please only fill nullable/defaulted fields, and raise a casting error for omitted required columns.

Useful? React with 👍 / 👎.