Optimization questions 3.17 vs 4.7

[qjhart]

I'm trying to understand some query optimizations that I see between versions 3.17 and 4.7.0 of jena as run inside fuseki. I feel like I'm not understanding some of the default query optimization parameters in fuseki, but I'm having trouble finding documentation explaining that.

They are each running the same database as a tdb2 file. For the sample query, version 3.17 includes the tdb2 stats.opt base reorders while the 4.70 version doesn't. I also don't understand why the fuseki query logging ja:context [ ja:cxtName "arq:logExec" ; ja:cxtValue "all" ] ; looks so much like the tdbquery plan (eg quadpattern) while the 4.7.0 looks like the arq plan (bgq).

Query

PREFIX oap: <http://oapolicy.universityofcalifornia.edu/vocab#>
select ?p ?b ?c
WHERE {
    ?p oap:begin-page ?b;
       oap:page-count ?c;
        .
 }

Only a few examples include page count.

Version 3.17 includes reordering from the stats.opt file:

fuseki_1  | 00:11:04 INFO  Fuseki          :: [6] Query = PREFIX oap: <http://oapolicy.universityofcalifornia.edu/vocab#>  select ?p ?b ?c WHERE { #  ?i oap:name "pagination"; #   ?i oap:pagination ?p; #     . #  ?i ?foo ?p.   ?p      oap:begin-page ?b;     oap:page-count ?c;   . }
fuseki_1  | 00:11:04 INFO  exec            :: QUERY
fuseki_1  |   PREFIX  oap:  <http://oapolicy.universityofcalifornia.edu/vocab#>
fuseki_1  |
fuseki_1  |   SELECT  ?p ?b ?c
fuseki_1  |   WHERE
fuseki_1  |     { ?p  oap:begin-page  ?b ;
fuseki_1  |           oap:page-count  ?c
fuseki_1  |     }
fuseki_1  | 00:11:04 INFO  exec            :: ALGEBRA
fuseki_1  |   (project (?p ?b ?c)
fuseki_1  |     (quadpattern
fuseki_1  |       (quad <urn:x-arq:DefaultGraphNode> ?p <http://oapolicy.universityofcalifornia.edu/vocab#begin-page> ?b)
fuseki_1  |       (quad <urn:x-arq:DefaultGraphNode> ?p <http://oapolicy.universityofcalifornia.edu/vocab#page-count> ?c)
fuseki_1  |     ))
fuseki_1  | 00:11:04 INFO  exec            :: TDB
fuseki_1  |   (project (?p ?b ?c)
fuseki_1  |     (quadpattern
fuseki_1  |       (quad <urn:x-arq:DefaultGraphNode> ?p <http://oapolicy.universityofcalifornia.edu/vocab#begin-page> ?b)
fuseki_1  |       (quad <urn:x-arq:DefaultGraphNode> ?p <http://oapolicy.universityofcalifornia.edu/vocab#page-count> ?c)
fuseki_1  |     ))
fuseki_1  | 00:11:04 INFO  exec            :: Execute
fuseki_1  |   ?p <http://oapolicy.universityofcalifornia.edu/vocab#page-count> ?c
fuseki_1  |   ?p <http://oapolicy.universityofcalifornia.edu/vocab#begin-page> ?b

While fuseki 4.7.0 doesn't

fuseki_1  | 00:49:39 INFO  Fuseki          :: [1] POST http://lab.dams.library.ucdavis.edu:3030/pubs/query
fuseki_1  | 00:49:39 INFO  Fuseki          :: [1] Query = PREFIX oap: <http://oapolicy.universityofcalifornia.edu/vocab#> select ?p ?b ?c WHERE {     ?p oap:begin-page ?b;        oap:page-count ?c;         .  }
fuseki_1  | 00:49:39 INFO  exec            :: QUERY
fuseki_1  |   PREFIX  oap:  <http://oapolicy.universityofcalifornia.edu/vocab#>
fuseki_1  |
fuseki_1  |   SELECT  ?p ?b ?c
fuseki_1  |   WHERE
fuseki_1  |     { ?p  oap:begin-page  ?b ;
fuseki_1  |           oap:page-count  ?c
fuseki_1  |     }
fuseki_1  | 00:49:39 INFO  exec            :: ALGEBRA
fuseki_1  |   (project (?p ?b ?c)
fuseki_1  |     (bgp
fuseki_1  |       (triple ?p <http://oapolicy.universityofcalifornia.edu/vocab#begin-page> ?b)
fuseki_1  |       (triple ?p <http://oapolicy.universityofcalifornia.edu/vocab#page-count> ?c)
fuseki_1  |     ))
fuseki_1  | 00:49:39 INFO  exec            :: TDB2
fuseki_1  |   (project (?p ?b ?c)
fuseki_1  |     (bgp
fuseki_1  |       (triple ?p <http://oapolicy.universityofcalifornia.edu/vocab#begin-page> ?b)
fuseki_1  |       (triple ?p <http://oapolicy.universityofcalifornia.edu/vocab#page-count> ?c)
fuseki_1  |     )) 

More info

stats.opt

This is the stats.opt file for the database.

stats
  (meta
    (count 229762653))
 (<http://oapolicy.universityofcalifornia.edu/vocab#page-count> 120)
  (<http://oapolicy.universityofcalifornia.edu/vocab#begin-page> 593829)

Fuseki 4.7.0

tdbquery

The tdbquery plan looks fine, where the execute stage reorders.

tdb2.tdbquery --explain --tdb=configuration/pubs.ttl --query=- <<<' PREFIX oap: <http://oapolicy.universityofcalifornia.edu/vocab#>      select ?p ?b ?c
      WHERE {
        ?p
          oap:begin-page ?b;
          oap:page-count ?c;
        .
      }'

results in

3:54:19 INFO  exec            :: QUERY
  PREFIX  oap:  <http://oapolicy.universityofcalifornia.edu/vocab#>

  SELECT  ?p ?b ?c
  WHERE
    { ?p  oap:begin-page  ?b ;
          oap:page-count  ?c
    }
23:54:19 INFO  exec            :: ALGEBRA
  (project (?p ?b ?c)
    (quadpattern
      (quad <urn:x-arq:DefaultGraphNode> ?p <http://oapolicy.universityofcalifornia.edu/vocab#begin-page> ?b)
      (quad <urn:x-arq:DefaultGraphNode> ?p <http://oapolicy.universityofcalifornia.edu/vocab#page-count> ?c)
    ))
23:54:19 INFO  exec            :: TDB2
  (project (?p ?b ?c)
    (quadpattern
      (quad <urn:x-arq:DefaultGraphNode> ?p <http://oapolicy.universityofcalifornia.edu/vocab#begin-page> ?b)
      (quad <urn:x-arq:DefaultGraphNode> ?p <http://oapolicy.universityofcalifornia.edu/vocab#page-count> ?c)
    ))
23:54:19 INFO  exec            :: Execute
  ?p <http://oapolicy.universityofcalifornia.edu/vocab#page-count> ?c
  ?p <http://oapolicy.universityofcalifornia.edu/vocab#begin-page> ?b

arq --explain

arq --explain --data=configuration/pubs.ttl --query=- <<<' PREFIX oap: <http://oapolicy.universityofcalifornia.edu/vocab#>
      select ?p ?b ?c
      WHERE {
        ?p
          oap:begin-page ?b;
          oap:page-count ?c;
        .
      }'

00:04:03 INFO  exec            :: QUERY
  PREFIX  oap:  <http://oapolicy.universityofcalifornia.edu/vocab#>

  SELECT  ?p ?b ?c
  WHERE
    { ?p  oap:begin-page  ?b ;
          oap:page-count  ?c
    }
00:04:03 INFO  exec            :: ALGEBRA
  (project (?p ?b ?c)
    (bgp
      (triple ?p <http://oapolicy.universityofcalifornia.edu/vocab#begin-page> ?b)
      (triple ?p <http://oapolicy.universityofcalifornia.edu/vocab#page-count> ?c)
    ))
00:04:03 INFO  exec            :: BGP
  ?p <http://oapolicy.universityofcalifornia.edu/vocab#begin-page> ?b
  ?p <http://oapolicy.universityofcalifornia.edu/vocab#page-count> ?c
00:04:03 INFO  exec            :: Reorder/generic
  ?p <http://oapolicy.universityofcalifornia.edu/vocab#begin-page> ?b
  ?p <http://oapolicy.universityofcalifornia.edu/vocab#page-count> ?c

Fuseki 3.17

The individual components for 3.17 seem almost identical to the 4.7.0 version.

tdbquery

tdb2.tdbquery --explain --tdb=configuration/pubs.ttl --query=- <<<' PREFIX oap: <http://oapolicy.universityofcalifornia.edu/vocab#>      select ?p ?b ?c
      WHERE {
        ?p
          oap:begin-page ?b;
          oap:page-count ?c;
        .
      }'

results in:

00:16:09 INFO  exec            :: QUERY
  PREFIX  oap:  <http://oapolicy.universityofcalifornia.edu/vocab#>

  SELECT  ?p ?b ?c
  WHERE
    { ?p  oap:begin-page  ?b ;
          oap:page-count  ?c
    }
00:16:09 INFO  exec            :: ALGEBRA
  (project (?p ?b ?c)
    (quadpattern
      (quad <urn:x-arq:DefaultGraphNode> ?p <http://oapolicy.universityofcalifornia.edu/vocab#begin-page> ?b)
      (quad <urn:x-arq:DefaultGraphNode> ?p <http://oapolicy.universityofcalifornia.edu/vocab#page-count> ?c)
    ))
00:16:09 INFO  exec            :: TDB
  (project (?p ?b ?c)
    (quadpattern
      (quad <urn:x-arq:DefaultGraphNode> ?p <http://oapolicy.universityofcalifornia.edu/vocab#begin-page> ?b)
      (quad <urn:x-arq:DefaultGraphNode> ?p <http://oapolicy.universityofcalifornia.edu/vocab#page-count> ?c)
    ))
00:16:09 INFO  exec            :: Execute
  ?p <http://oapolicy.universityofcalifornia.edu/vocab#page-count> ?c
  ?p <http://oapolicy.universityofcalifornia.edu/vocab#begin-page> ?b

arq --explain

arq --explain --data=configuration/pubs.ttl --query=- <<<' PREFIX oap: <http://oapolicy.universityofcalifornia.edu/vocab#>
      select ?p ?b ?c
      WHERE {
        ?p
          oap:begin-page ?b;
          oap:page-count ?c;
        .
      }'

results in:

01:08:57 INFO  exec            :: QUERY
  PREFIX  oap:  <http://oapolicy.universityofcalifornia.edu/vocab#>

  SELECT  ?p ?b ?c
  WHERE
    { ?p  oap:begin-page  ?b ;
          oap:page-count  ?c
    }
01:08:57 INFO  exec            :: ALGEBRA
  (project (?p ?b ?c)
    (bgp
      (triple ?p <http://oapolicy.universityofcalifornia.edu/vocab#begin-page> ?b)
      (triple ?p <http://oapolicy.universityofcalifornia.edu/vocab#page-count> ?c)
    ))
01:08:57 INFO  exec            :: BGP
  ?p <http://oapolicy.universityofcalifornia.edu/vocab#begin-page> ?b
  ?p <http://oapolicy.universityofcalifornia.edu/vocab#page-count> ?c
01:08:57 INFO  exec            :: Reorder/generic
  ?p <http://oapolicy.universityofcalifornia.edu/vocab#begin-page> ?b
  ?p <http://oapolicy.universityofcalifornia.edu/vocab#page-count> ?c

[afs]

You can remove all the subClassOf statements.

[qjhart]

Thanks Andy, I should have said that the pubs.ttl file was exactly as generated by the fuseki, via the new Dataset feature, I wanted to leave as created. I did remove those as a test, but it didn't affect the optimization issue.

[afs]

As of 4.8, they won't the the unnecessary declaration in them.

I have had a first go at recreating the situation but current I do not see what you report.

One thing to check is the location of the stats.opt file in the DB2 database especially if you have compacted the database at any time.

[qjhart]

I've done a little more digging, and I've discovered this was a bit of a red herring. Another dataset configuration seems to have somehow caused this error, and removing that configuration file fixed this issue. I'll try and figure out the exact reason, but It's not directly related to the configuration above.

[afs]

That explains the BGP vs quadpattern.

Some general dataset wrappers work on graphs and so block the quad evaluation.