Ability to handle wide tables?

[roll]

https://frictionlessdata.slack.com/archives/C0369HZ2SLT/p1682673195434089 (by mlell)

Hello everyone! 👋 I am working in computational biology and found great promise in saving my data as CSV with documentation following the frictionless tabular resource spec. I have encountered one problem though. The standard works nicely for tabular data where the number of columns is limited (and also the number of levels in categorical data, but thats outside of this question). But I sometimes have to save microarray data which has one row for each individual and one column for each marker (locus of interest on the genome). This means I can have thousands (20k) of columns. It makes no sense to include a field entry for each column, because they are all the same (except the first few, which contain information about individuals. Is there a way to define a "default" column data type/description?
I am aware that I could just convert my data to long format, but this means an additional data transformation step which makes it harder for people with less technical background (collaboration with people who only know Excel is common in Biology) to work with the format. Also it increases my data size as I have to duplicate individual and marker names for each row in the output. I also know that there is the VCF format, but that requires a reference sequence which is not defined in my case (especially in industry, marker platforms might evolve over time using different or even no reference sequences) and a unique marker mapping position (same).

[mlell]

From the lack of reaction I feel that either this proposal is not terribly popular or people do not really know how to deal with it given the available time. Therefore I would propose something that is maybe easier. If there is a large number of columns with the same description, they will also have the same data type. Therefore, could the specs alternatively be extended to include two-dimensional data (a matrix). It would look very similar to the tabular data resource except that there is not an arbitrary number of fields but just (for example) "rownames", "colnames", "cells":

{
  "profile": "matrix-data-resource??",
  "name": "example-snp-data",
  "path": "...",
  "title": "Just some made-up data that looks like microarray calls",
  "description": "........",
  "format": "csv",
  "mediatype": "text/csv",
  "encoding": "utf-8",
  "hash": "",
  "schema": {
    "rownames": {
      /* "name" is not useful here
      "title": "Sample name",
      "type": "string",
      "description": "The name of plant variety as registered at the German BSA"
    },
   "colnames": {
      "title": "Marker name",
      "type": "string",
      "description": "The marker name, as defined in ArrayExpress ID XYZ"   
    },
   "cells": {
      "title": "Single nucleotide polymorphism call",
      "type": "string",
      "description": "The single nucleotide polymorphism detected, one letter nucleotide code. Heterozygous calls are encoded in IUPAC ambiguity code. Missing data is encoded as '-'."
    },
  },
  "dialect": {
    "delimiter": ";",
    "doubleQuote": true
  },
  "sources": [{
.....
    }],
    "licenses": [{
....
    }]
}

to annotate a table like (making up some values, names inspired from Wang et al. (2014) Plant Biotechnol J, 12(6): 787). Note that in reality there can be thousands of rows and columns:

"Marker";"Julius";"Donovan"
"CAP8_s9818_178";"A";"A"
"CAP8_s9855_165";"A";"A"
"CAP8_s9973_290";"G";"-"
"D_contig00066_616";"G";"R"
"D_contig00075_218";"C";"C"

Is there hope for a format like this? It would fill a gap in that annotating very large CSV files is currently not practically possible following the frictionless standard.

For matrices that have no row/column names, the respective parts of "schema" could be omitted.

[roll]

Hi @mlell,

Thanks a lot for your input! I think wide tables support has always been a popular topic for discussion so there is no reaction just because the specs are very stable, and while there is no current work for the next version (which will be happening soon), people outside of core contributors just don't watch new issues.

That's a really interesting proposal... Something like matrixSchema