Avoid hard-coding for concrete implementations

[andreasnoack]

I'm wondering to what extent it would be practical to avoid hard-coding for specific implementations in this package. In particular for NullableArrays and DataTables since I'd like to use the functionality here with DataFrames and DataArrays. Commenting out specific uses of NullableArrays and removing the uses of DataTable signatures (could be some kind of AbstractTable in the future) made some of the basic functionality work. Some definitions like https://github.com/JuliaStats/StatsModels.jl/blob/master/src/modelframe.jl#L108 seem misplaced here in any case. I believe they should be defined in their home packages. @nalimilan why do you define the underscore versions of these functions? Other uses might be harder to get rid of. In particular the use of the Categorical types. I wondered if we could define an abstract categorical which also PooledDataArrays could be subtypes of.

A very different approach could be to use https://github.com/davidagold/Relations.jl/ as the unifying framework here. I think it is very interesting and might be a way to go beyond in-memory data frame representations. However, there are probably a lot to sort out before could consider such a change whereas the changes discussed above are pretty simple.

[nalimilan]

The _unique methods are needed to unwrap nullables. Fortunately, special-casing Nullable does not interfere at all with DataArray support, which does not need unwrapping. Since this code is working on complete cases, it will never see a null or a NA and should work fine with either. See also JuliaStats/DataArrays.jl#237.

Actually, it won't be too hard to stop hard-coding CategoricalArray. In #15, I remove half of the uses, which are no longer needed. As you suggest, it would make sense to define AbstractCategoricalArray (probably in AbstractTables again), since the remaining occurrence of CategoricalArray in the code base is for modelmat_cols, where it's much more efficient to use the integer codes than to compare strings one by one to get the index of the column they correspond to. That's pretty easy change to make, with the slight complication that we also need AbstractNullableCategoricalArray, which won't be used by PooledDataArray since no unwrapping is required for it.

Then the other easy step is to replace AbstractDataTable with AbstractTable and make DataFrame and DataTable inherit from the latter.

Longer-term, I think it would indeed be great to move to a tuple-based approach like Relations.jl, which would avoid lots of copies and allow supporting any kind of source. But for now we don't really need it to support both DataFrames and DataTables.

EDIT: Regarding the _unique(x::CategoricalArray) method you refer to, it can be made more generic by dispatching on AbstractCategoricalArray. This method is needed since we want to sort levels of e.g. Vector{String} in alphabetical order, but we want to keep the user-defined order for categorical arrays. Let's discuss another possible approach at JuliaStats/DataArrays.jl#237.

[kleinschmidt]

I agree about moving to a tuple-based interface, but generalizing on DataFrames and DataTables is a great first step.

[davidanthoff]

I think I'm super close to have everything in place for a very generic solution to this problem in Query.jl.

Query is based on iterators of NamedTuples. I have a branch where the old modeling features in DataFrames all work with any source that is an iterator of NamedTuples, so one can use that already to essentially use any of the Query data sources with the modeling features (i.e. DataFrames, DataTables, TypedTables, any DataStreams source etc.). This is also a completely streaming interface, so it works really nicely for non-in-memory sources.

The main thing I still need to do is move this over to a traits based dispatch story. I think I've figured pretty much everything out for that, main problem is to find an afternoon where I can finalize all of this.

[ararslan]

Nice work, @davidanthoff! @quinnj, perhaps inspiration could be taken from there for defining things in AbstractTables.

[davidanthoff]

I think what I should do is actually move all of that functionality out into a package called IterableTables. It would be a super minimal package that provides a minimum interface for tabular data exchange.

[ararslan]

That's precisely what the plan is for AbstractTables, actually. 🙂

[ararslan]

It would be good to coordinate this effort. As I understand it, @quinnj had been planning to lead the effort on this end (Jacob--sorry if that's incorrect and I've volunteered you).

[davidanthoff]

Alright, I took a first stab at a traits based iterable tables implementation in Query in queryverse/Query.jl#93.

Here is what this does so far:

• DataTable, DataFrame and TypedTables are now iterable table sources. And any uncollected Query query (haha) that produces NamedTuples will also work.
• DataTable and DataFrame have new constructors that dispatch on this iterable table trait, so you can call them with DataFrame(x), and as long as x is an iterable table (right now any of the three iterable table sources from the previous bullet point, plus any iterable that produces NamedTuples).
• I also added a DataFrames.ModelFrame constructor that dispatches on the iterable table trait. So you can now call ModelFrame(@formula(a~b), q), and as long as q has the iterable table trait, it will just work. If we want functions like lm in GLM to work with this, we would just need to add a method there that dispatches on that trait.

I'm being a bit sloppy with the term "iterable table" here, that is actually not a trait in my current implementation, the details are slightly more complicated, but probably not super important.

[davidanthoff]

And I just added this trait support for any DataStream source, so you can also do things like this now:

using Query, DataTables, DataFrames, CSV

df = DataFrame(CSV.Source("test.csv"))

mf = ModelFrame(@formula(a~b), CSV.Source("test.csv"))

dt = DataTable(CSV.Source("test.csv"))

and pretty much all permutations of this. And this should work for any DataStreams source, i.e. feather, SQLite etc.

[davidanthoff]

I pushed one more thing, so that now things like lm from GLM.jl work with any data source that has the table trait. So this now works:

using Query, DataFrames, DataTables, GLM

dt = DataTable(X=[1,2,3], Y=[2,4,7])

lm(@formula(Y~X), dt)

[andreasnoack]

It looks almost too good to be true. Can you clarify the implied dependencies that this would require? A new AbstractTables/IterableTables that both DataFrames and DataTables should depend on, right? Where is Query in this chain?