Triples and quads are stored and queried separately in the same triplestore

[vemonet]

Currently Oxigraph stores and queries the ntriples and nquads completely separately in the same triplestore.

If I add a triple, lets say:

<http://st/> <http://pt/> <http://ot/> .

And a quad, lets say:

<http://sq/> <http://pq/> <http://oq/> <http://fq/> .

If I run a query without asking for graph

SELECT * WHERE { ?s ?p ?o . }

I am only getting the triple

To get the quad I need to also add the graph pattern:

SELECT * WHERE { GRAPH ?g { ?s ?p ?o . } }

Then I only get the quad

In those 2 queries I would expect to get the triple and quad returned (triple should be returned with the default graph URI as ?g)

I am not sure what is the stance of the SPARQL specs about this, but from a user point of view I find it quite confusing and unexpected (I think most other triplestores I use were always returning the quads stored even when I skipped the graph pattern).

I expect we should be able to query quads using directly the triples pattern without the need to add the graph pattern (which makes it easier to query triples across multiple graph without knowing exactly the graph composition).

A potential solution?

According to Oxigraph internal design docs: 9 tables are created to index the triples and quads, 3 for the default triple graphs. 6 for the nquads

It seems like depending on the user providing or not the graph pattern, then either the 3 tables for the default triple graph are used or the 6 tables for the quads

A solution could be to make oxigraph working only with nquads by default. With a default graph URI that can be set at startup, when triples are added without specifying the graph, they go to this graph

This would reduce the total number of tables from 9 to 6. And not using the graph pattern keyword could be interpreted as select * on the graph column

And there could be a switch to start oxigraph in triple mode, without graph, which would use only 3 tables

From a user and a admin point of view personally I think it is normal a triple store can be either supporting nquads or triples. As long as it is indicated in the service description (I think there is a metadata for this).

By default it would be supporting quads, and even if I add triples they will be converted to quads going to the default graph. But if I don't need the graph, and I value more efficiency and performance for querying my triples, then I can start the triplestore in triple mode.

Not sure if that makes sense, @Tpt you probably already thought about this, and I might be missing some information on implementation. But you were mentioning wanting to reduce the number of combinations, we can maybe also reduce the number of tables ;)

[Tpt]

Hey! Thank you so much for this detailed proposal. It's very appreciated.

According SPARQL spec by default the triple patterns only queries the default graph: Outside the use of GRAPH, matching is done using the default graph. in the spec. A lot of triple stores consider the default graph as the union of the default graph and all named graphs, hence also returning <http://sq/> <http://pq/> <http://oq/> when using the first query.

In Oxigraph I have made the choice of only querying the default graph by default (to enable use case like named graph representing not asserted facts) but provide an option for the "default graph is the union of all graphs behavior". This is the set_default_graph_as_union Rust option, the use_default_graph_as_union=True option of the query method in Python, the brand new use_default_graph_as_union: true option of the query method in JS... The fact that you have not seen it make me think this option is maybe not highlighted enough, feel free to point me places in the documentations where you think it would be great to talk about it (PR also welcome of course).

About indexes, triples in the default graph are added to the 3 triple indexes and triples in named graphs are added to the 6 quad indexes. This has been done to transparently use only 3 tables for triples in the default graph and so, provide nearly as good performances as a "triple-only mode" without having to set a configuration option explicitly. If I understand correctly your proposal, applying your change would mean 1. creating a configuration option that must be set before loading data and is hard to change 2. store triples in the default graph in 6 indices (the quad ones and not only 3 (the default graph ones), hence increasing the memory footprint significantly in the use case "a lot of triples in the default graph and few in named graphs" while only making query evaluation slightly faster (hopefully a bit less data chunks to merge when doing a scan).

I hope it helps, maybe there is something I am missing in your proposal. Thank you again!

[vemonet]

I must say I am not a big fan of requiring a specific argument at query time to get the expected results. I like that SPARQL can be queried in a standard way to get expected results without needing to add a wrapper. And people who query should not think of which arguments they need to set in order to get the results they expect with a triplestore

But if you are ok with adding it to the server and oxrdflib graph arguments, then that's perfect! That leaves the previous behavior unchanged by default, and that gives admins like me the possibility to set their own preferences. I'll look into adding it when I will have some time.

I have already played a bit with rust recently :) https://github.com/vemonet/nanopub-rs and https://github.com/biopragmatics/curies.rs , oxigraph was the main source of inspiration (and code), thanks a lot!

[vemonet]

@Tpt I had a quick look, and it seems we have 3 choices:

• Add a default_graph_as_union field to the Store struct, and create separate constructors Store::open_union() and Store::open_secondary_union(). This has the advantage to not change the arguments of existing constructors like Store::open() or Store::open_secondary(). But it adds more functions to the API, and requires to add a if/else to call the right constructor instead of just passing the variable.
• Otherwise we could add a default_graph_as_union boolean argument to the different constructors. It feels cleaner in term of API imo, but this would require to change all existing call to the open constructors, and introduce a breaking change for people who are using the rust library
• Last option does not require to change the Store struct at all. It could be directly picking up the default default_graph_as_union from environment variables in configure_and_evaluate_sparql_query(). Probably the worst in term of elegant API design, but this get the job done for most use-cases in 2 lines of code, without the need to touch to the Store.

Where goes your preference?

[adamreichold]

Probably the worst in term of elegant API design, but this get the job done for most use-cases in 2 lines of code, without the need to touch to the Store.

Just a bystander but I would strongly suggest to avoid environment variables as a primary configuration mechanism, they are ghosts.

[Tpt]

I like that SPARQL can be queried in a standard way to get expected results without needing to add a wrapper. And people who query should not think of which arguments they need to set in order to get the results they expect with a triple store.

I would argue here the "standard way" according to the actual standard is to only query the default graph by default (see my reference above) and it's why I would call the "expected" results. If I remind correctly (but I might be wrong) it's what rdflib Dataset, Jena and RDF4J are also doing and I guess also a good number of other implementations.

I must say I am not a big fan of requiring a specific argument at query time to get the expected results.

To play the devil's advocate, this is what the SPARQL 1.1 HTTP protocol allows with it's default-graph-uri argument that allows to change the default graph. It's why I have added the union-default-graph argument.

I'll look into adding it when I will have some time.

Great! Thanks! RDFlib is already providing the __UNION__ option to the Store.query method when targeting the union of all graph so I would tend to think the change should be done in rdflib itself and not in oxrdflib. But I have not investigated it properly.

I have already played a bit with rust recently :) https://github.com/vemonet/nanopub-rs and https://github.com/biopragmatics/curies.rs , oxigraph was the main source of inspiration (and code), thanks a lot!

Amazing! I will have a look at them.

Where goes your preference?

None are good imho sadly. The env variable is a non starter imho per @adamreichold argument and because it prevents to have both a Store with "default graph is union" and a Store with "default graph is default graph" in the same process. I am also not at ease with the boolean arguments because I am ready to bet there might an other dozen of boolean arguments one might want to add there two so we might end up with unreadable open("path", true, false, true, true). Similarly, custom constructors is not a good idea imho because of combinatorial explosion.

[vemonet]

Yes I agree none of these 3 solutions seemed clean, especially the env variable one

The best solution I know for this problem would be to use the builder pattern:

Instantiating a default store:

Store::open()
    .build()

With a custom union-default-graph:

Store::open()
    .set_default_graph_as_union()
    .build()

It will still require to update all calls to the Store constructor, but it is much more extensible, and it will allow you to add more configurable fields to the Store later easily without the need for more changes

Regarding the SPARQL specs, you are completely right, and it indeed behave this way with RDFLib, I just never noticed! To be fair, I am fine if you prefer to keep the union-default-graph just on the query, as long as we make it a bit clearer in the docs (maybe we could add a section about available params for HTTP query in https://github.com/oxigraph/oxigraph/blob/main/cli/README.md). But I still think it could be nice to set it at the server level, if you want to introduce the builder pattern :)