Simplify verbNound for combining results over "outer" collections.

[Affie]

As an example, we have getBlobentriesVariables (which I find confusing) to collect Blobentries over variables matching some criteria.

I think a new verb might work best in this case (also considered an adjective).
So far, I have collect as the verb, ie. collectBlobentries. Also considered aggregate

The result will be a Vector{Pair{Symbol, Vector{Blobentry}}} and not a flattened collection as with getBlobentriesVariables to be more useful.

Edit: fixed collectBlobentries typo as pointed out bellow and result type missing Vector

[Affie]

The implementation will change to this:

function getBlobentries(
    dfg::AbstractDFG,
    label::Symbol;
    labelFilter::Union{Nothing, Function} = nothing,
)
    entries = getBlobentries(getVariable(dfg, label))
    filterDFG!(entries, labelFilter)
    return entries
end

function collectBlobentries(
    dfg::AbstractDFG;
    labelFilter::Union{Nothing, Function} = nothing,
    solvableFilter::Union{Nothing, Function} = nothing,
    tagsFilter::Union{Nothing, Function} = nothing,
    typeFilter::Union{Nothing, Function} = nothing,
    variableLabelFilter::Union{Nothing, Function} = nothing,
)
    vls = listVariables(
        dfg;
        solvableFilter,
        tagsFilter,
        typeFilter,
        labelFilter = variableLabelFilter,
    )
    return map(vls) do vl
        return vl=>getBlobentries(dfg, vl; labelFilter)
    end
end

[dehann]

we have getBlobentriesVariables (which I find confusing)

side comment, in cases like this I have we the logic is getVariablesBlobentries, since blobentries are satellites on variables. Also why I like the dispatch version of just get / collect since the order of the Nouns is basically just a footgun, as this example shows ... but newer suggestion likely better anyway...

ie. collectBlobEntries

I thought the noun here is Blobentries (not BlobEntries)

---

TL;DR; this double noun should maybe not be part of DFG v1.0.

The implementation will change to this:

would your example implementation of getBlobentries return Pair? Going from the text it sounds like that is the case?

The result will be a Vector{Pair{Symbol, Blobentry}}

a Vector{Pair{Symbol, Blobentry}} and not a flattened collection as with getBlobentriesVariablesgetVariablesBlobentries to be more useful.

My worry here is "to be more useful" sounds like a deviation from collect producing Vector{<:T}. Which would mean different signatures using verb collect return different "useful" return types. The problem is what is useful for one person is not useful for another who wants to do something slightly different.

In general, the double noun signatures for me does not immediately suggest you get back a Pair{NT1, NT2}. If that were the case then every double noun function should always return a Pair. E.g. getVariableBlobentry(_) should then return Pair{Symbol, Blobentry} and that would be weird.

Oh, and the other problem with non-dispatch signatures (i.e. verbNounNoun(::NounT1, ::NounT2) is that the double noun makes the order of Pair ambigious. E.g. does collectBlobentriesVariables return the opposite Pair{Blobentry, Symbol}?

Part of why I believe double Noun should be avoided for core low level implementation, and work to resolve a dispatch implementation -- and alias only as a last resort (fwiw Python comes after a last resort).

new verb might work best ... collect

I think collect implies a flattened list of the primary Noun (i.e. ::Vector{<:Blobentry}, so here's where my brain goes:

• collectBlobentries(::Variable; filter::Function) is going to return ::Vector{<:Blobentry}
• [overly wide example] collectVariablesBlobentry(::Factorgraph, blobLabel::Symbol) is going to return ::Vector{<:Blobentry} (loss of variable info). But confusing since Variables is the primary noun, so why not Vector{::Variable}?
• collect( getBlobentry.(getVariable.(::Factorgraph; filter::Function), blobLabel::Symbol) ) which returns a ::Vector{<:Blobentry}, but uses a lot of memory/bandwidth
• res = Vector{Blobentry}(); for vLb in ls(fg) (getBlobentry(getVariable(fg, vLb), blobLb) |> be->isnothing(be) ? nothing : push!(res,be)); end; res which returns a ::Vector{Blobentry} -- risk here is ::Blobentry should be concrete, but is much lower memory/bandwidth usage.
• etc.

If you really want to use collect for ::Vector{<:Pair} then the best way would be to set up an internal function that returns the pair and collect over that:

• res = Vector{Pair{Symbol, Blobentry}}(); ls(fg) .|> v->(getBlobentry(::Factorgraph, v::Variable) |> be -> isnothing(be) ? nothing : push!(res, getLabel(v) => be)) -- same risk of concrete ::Blobentry, but this version can leverage threading while keeping bandwidth usage minimal.
• Or trusty callback, ls(fg; filter=s::Symbol->hasBlobentry(fg, s, bLb::Symbol)) .|> v->getLabel(v) => getBlobentry(v, bLb) -- in this case the collect is already implied in the broadcast .|>

Recommend

I recommend perhaps think more aboutaggregate and don't export in DFG v1.0. Idea is to require some form of user provided callback to establishes the return. type E.g.

function aggregate(vv::Vector{T}; filter::Function) where T
  res = Vector{Any}() # todo resolve type of filter from vv, cannot just assume typeof(filter(vv[1])
  for v in vv
    s = filter(v)
    !isnothing(s) && push!(res, s)
  end
  return res
end 

And usage would be something like

res::Vector{<:Pair} = aggregate(
  ls(fg); 
  filter = l->begin 
    v=getVariable(fg, l); 
    be=getBlobentry(v, blobLb)); 
    isnothing(be) ? nothing : l=>be 
  end
)

This has the advantage of being really efficient on bandwidth side, but without solving TODO creates a ::Vector{Union{Nothing, Pair{Symbol, Blobentry}} -- maybe okay to use the dynamic dispatch a little bit in this case since it's a rather complicated query, also aggregate would not be exported in DFG v1.0, and can always be solved by completing the TODO. Maybe Julia already has some magic core function barrier like dropnothing/collectnotnothing which strips Union{Nothing,_} from the return type.

[Affie]

side comment, in cases like this I have we the logic is getVariablesBlobentries, since blobentries are satellites on variables. Also why I like the dispatch version of just get / collect since the order of the Nouns is basically just a footgun, as this example shows ... but newer suggestion likely better anyway...

My guess here is that your idea was that Variables in getBlobentriesVariables was used as the adjective, ie. getBlobentries_by_Variable

[Affie]

I thought the noun here is Blobentries (not BlobEntries)

Jip just a typo. So used to camel case.

[Affie]

The definition of aggregate in this context does not fit with what this function (verb) does, a few definitions:

• wikipedia: In database management, an aggregate function or aggregation function is a function where multiple values are processed together to form a single summary statistic.
• Use aggregate if the function also performs a summarization, reduction, or combines the entries in a more meaningful way (e.g., summing, averaging, grouping).
• "Aggregate" generally means to combine or collect into a whole or mass. It can refer to a sum total, a collection of diverse things, or specific materials like construction aggregate (sand, gravel, etc.). In statistics, it often means combining data into a total or average.
• Statistical Context: Definition: The process of combining numerical data to create a single value, like a total or average.

[Affie]

Sure we can leave it out of DFG v1.0. I think the current structure is easier to use without an extra function, replicating your example (slightly modiefied as be=getBlobentry(v, blobLb) does not return nothing as per get verb definition).

res = map(ls(fg)) do vl
    return vl=>getBlobentries(dfg, vl; labelFilter = startswith("IMG"))
end

A separate implementation of [collect/combine/gather]BlobEntries will only be for performance.

In this context, [collect/combine/gather] means to traverse a collection of variables in the factor graph and, for each variable, collect all associated Blobentry objects, returning them as a single flat vector of pairs (e.g., Vector{Pair{Symbol, Vector{Blobentry}}}). Each pair associates a variable label with one of its blob entries. This operation does not summarize or reduce the data, but simply brings together all relevant entries from across the graph into one list for further processing or inspection.

[dehann]

Hmm, gather returns ::Vector{<:Pair} sounds pretty good actually...

[dehann]

My guess here is that your idea was that Variables in getBlobentriesVariables was used as the adjective, ie. getBlobentries_by_Variable

My point is just the NounNoun can lead to a lot of confusion, so maximum dispatch around the verb and arguments just makes more sense to me. Probably why I dislike other programming languages so much too.

[dehann]

aggregate ... reduce

yeah, aggregate is not the right verb for this case. So we can just leave that.

[dehann]

does not return nothing as per get verb definition

oh sure, then those examples above would have to be modified a bit

[dehann]

So my knee jerk reactions are:

• collect returns flat vector (basically iterates over some collection with the purpose of returning a vector)
• combine is very close to union, merge, zip, ... so careful with this one
• gather returns ::Vector{<:Pair} where elements are source => obj::T -- source could be Symbol/String/UUID depending on whats being called.

[dehann]

A separate implementation of [gatherBlobentries] will only be for performance.

Think this is a good idea, and would still not export in DFG v1.0. Just let these gather signatures mature a bit before we risk breaking changes.