generate_statistics_from_pyarrow table or parquet

[tanguycdls]

/type feature
Hi, since TF records are already converted to Pyarrow Tables to compute statistics, how hard would it be to add an option to read directly Pyarrow file or Parquet file?

data-validation/tensorflow_data_validation/utils/stats_gen_lib.py

Line 106 in bf40237

| 'DecodeData' >> tf_example_decoder.DecodeTFExample(

If my understanding of that code is correct we could replace beam.io.textio.ReadFromText by beam.io.parquetio.ReadFromParquet? if so will we need to extract features or the Pyarrow schema would be enough ?

My aim would be to use TFDV to extract data features and visualise them using facets.

Thanks

[gowthamkpr]

In this issue, its mentioned that the Towards the goal of adding support for computing statistics over structured data (e.g., arbitrary protocol buffers, parquet data), GenerateStatistics API will take Arrow tables as input instead of Dict[FeatureName, ndarray]

[caveness]

One thing to note is that the GenerateStatisticsAPI will only accept Arrow tables whose columns are ListArray of primitive types (e.g., int8, int16, int32, int64, uint8, uint16, uint32, uint64, float16, float32, float64, binary, string, unicode), so Arrow tables that are not in that format will not work with the API.

[tanguycdls]

Hi thanks for the prompt answer ! Sorry I did not search enough before sending the issue.

About the limitation of the input data to Arrow Tables of ListArray of primitives: It does not include List of List dtypes? If I use those dtypes what can I do ?

When converted to TfRecords it seems it worked when printing the facets: I had access to the quantiles of length of the records.

[jlafaye]

Many people use pandas or spark in their data preparation stage. Those tools dump parquet files in which every column is an array of 'primitive type', not an array of 'list of primitive types'.
I noticed that and tried to work around this limitation by attempting to cast between the two types but pyarrow does not support these kind of casts:

Traceback (most recent call last): File "apache_beam/runners/common.py", line 813, in apache_beam.runners.common.DoFnRunner.process File "apache_beam/runners/common.py", line 448, in apache_beam.runners.common.SimpleInvoker.invoke_process File "apache_beam/runners/common.py", line 928, in apache_beam.runners.common._OutputProcessor.process_outputs File "beam_test.py", line 64, in process yield table.cast(schema) File "pyarrow/table.pxi", line 1334, in itercolumns File "pyarrow/table.pxi", line 203, in pyarrow.lib.ChunkedArray.cast File "pyarrow/error.pxi", line 86, in pyarrow.lib.check_status pyarrow.lib.ArrowNotImplementedError: No cast implemented from int32 to list<item: int32>

Just wanted to share my interest in this kind of feature. I would by happy to help if needed.

[tanguycdls]

import numpy as np
import pyarrow as pa
import numpy as np
import pyarrow as pa
from tensorflow_data_validation.statistics import stats_options as options
from apache_beam.io.filesystem import CompressionTypes
import apache_beam as beam
from typing import Any, List, Optional, Text
from tensorflow_metadata.proto.v0 import statistics_pb2
import os
import tempfile
import tensorflow as tf
from tensorflow_data_validation import constants
from tensorflow_data_validation.api import stats_api
from tensorflow_data_validation import load_statistics

def convert_to_list_array(column):
        if isinstance(column.type, pa.lib.ListType):
            return column
        type_ = column.type.to_pandas_dtype()
        if hasattr(type_, 'kind'):
            if type_.kind == 'M':
                # tfdv does not support dates and pyarrow cannot cast from datetime to string natively
                return pa.lib.ListArray.from_arrays(np.arange(len(column)+1), column.to_pandas().astype(str))
        assert column.num_chunks == 1, 'does not work with more than one chunk'
        #Wrap in a list
        return pa.lib.ListArray.from_arrays(np.arange(len(column)+1), column.chunks[0])
    
def transform_2_valid_pyarrow(path_to_file, output_path):
    table = pa.parquet.read_table(path_to_file)
    res = {}
    for col in table.column_names:
        res[col] = convert_to_list_array(table[col])
    tb = pa.table(res)
    pa.parquet.write_table(tb, output_path)
    return output_path

def generate_statistics_from_arrow(
    data_location: Text,
    output_path: Optional[bytes] = None,
    stats_options: options.StatsOptions = options.StatsOptions(),
    pipeline_options: Optional[PipelineOptions] = None,
    compression_type: Text = CompressionTypes.AUTO,
) -> statistics_pb2.DatasetFeatureStatisticsList:
    if output_path is None:
        output_path = os.path.join(tempfile.mkdtemp(), 'data_stats.tfrecord')
    output_dir_path = os.path.dirname(output_path)
    if not tf.io.gfile.exists(output_dir_path):
        tf.io.gfile.makedirs(output_dir_path)

    batch_size = (stats_options.desired_batch_size if stats_options.desired_batch_size
      and stats_options.desired_batch_size > 0 else
      constants.DEFAULT_DESIRED_INPUT_BATCH_SIZE)
    with beam.Pipeline(options=pipeline_options) as p:
        _ = (
        p
        | 'ReadData' >> beam.io.parquetio.ReadFromParquetBatched(
        file_pattern=data_location, )
        | 'GenerateStatistics' >> stats_api.GenerateStatistics(stats_options)
        | 'WriteStatsOutput' >> beam.io.WriteToTFRecord(
        output_path,
        shard_name_template='',
        coder=beam.coders.ProtoCoder(
        statistics_pb2.DatasetFeatureStatisticsList)))
    return load_statistics(output_path)

The following should work and not cost too much in terms of memory !

[brills]

one caveat to that solution is that your original columns must not contain null (nil, None, etc).

[tanguycdls]

one caveat to that solution is that your original columns must not contain null (nil, None, etc).

Hi thanks for your answer indeed my dataframe was already null safe (replaced by 0) so i did not notice it...
So the easiest solution to use that would be to convert to TFrecords and let it handle all the conversions ?

If i can access the null mask of the array it should not be too hard to make the List array correctly but i'm not sure how to get it.

Thanks

[brills]

I think there may be an easy patch to your existing solution so it handles Nulls correctly, but I'm not familiar with pandas APIs.

Basically, we want to translate something like pd.Series([1, 2, None, 3]) to pa.array([[1], [2], None, [3]))

Note that pa.array([1], [2], None, [3]) is essentially

pa.ListArray.from_arrays(
  # mask[i] == True <==> list_array[i] is null
  offsets=pa.array([0,1,2,2,3], mask=np.array([False, False, True, False, False]),
  values=pa.array([1, 2, 3]))

So essentially, you can write a function that:

1. identifies the indices of all the nulls in a pd.Series
2. removes all the nulls
3. assemble a ListArray like above.