Using RDF4J in a multi-tenant environment

[imsdu]

Hello,
We plan to use RDF4J in a multi-tenant environment where each tenant needs isolation and therefore their dedicated repository.
Is there a limit on how many repositories a RDF4J deployment ?
What overhead does a new repository bring independently of the amount of data it holds ?

Also are there good practices to run RDF4J regarding deployment / backups / monitoring / statistics / etc ?
Like being able to keep track of the different seeing how much activity they have, how big there are / etc ?

Thanks :)

[kenwenzel]

Hi @imsdu,
have you already considered to use named graphs for separating the data?
Using multiple repositories is also possible but with the following limitations:

• each repository has its own values database for IRIs, blank nodes and literals
• each repository has its own caches
• NativeStore and LmdbStore both use memory mapping that might not work well if many repositories are created

[imsdu]

Thanks for the answer :)

The problem is that we already use named graphs to separate data within each tenant so that they can easily query / update / delete part of their data.

So repositories provide this level of isolation.
We are already doing that with Blazegraph for which RDF4J is a natural alternative.

In the long run, an idea would also be to archive and delete repositories for inactive tenants and restore them when they are active again to keep a lower number of repositories.

Another idea would be to introduce some sharding mechanism and route queries between different RDF4J instances according to the repository name.

I am currently writing some tests to see where and when I get problems with a single instance.

[imsdu]

Update: I have written a first test where I start the server/workbench and I create repositories / push data via the api using Gatling.
A test creating 10k repositories containing ~10k statements on my laptop (a Macbook with 32GB of RAM).

• With RDF4J/NativeStore,I get an OOM in RDF4J despite setting -Xmx to 16G ;
• With RDF4J/LmdbStore, the test completes in 60s/90s and the RAM consumption reaches ~20G of RAM ;
• With Blazegraph, it completes in ~18 minutes but it does not fully take advantages of the number of cores and memory.

Here is a gist of the simulation to get to this (there is probably some improvements to add to the configs ?):
https://gist.github.com/imsdu/9e7fe881128f3ebdfd02fef4ac98bc3f

RDF4J/LMDB:

Blazegraph:

I have a couple of questions:

• How experimental does the LmdbStore remain with RDF 5.x ?
• As the deployment will be shared between multiple tenants, are there configurations so that RDF4J is more adapted to this context, to avoid the noisy neighbor problem, make it more resilient to memory issues (possibly at the expense of performance), etc...

Thanks :)

[kenwenzel]

Are you sure that the data is actually written to disk? For the LmdbStore this would mean a throughput of more than 1 million triples per second.

The store is currently marked experimental because the internal data format is not yet fixed. For example, it might change to support RDF-star.

The LmdbStore mainly uses off-heap memory managed by the operating system. You can't directly influence how this memory is used.

[imsdu]

When stopping the instance and restarting it, I have my repositories back so it looks fine.
Having such numbers probably comes from having 50 concurrent users to create the repos.

[kenwenzel]

OK, good for the LmdbStore :-)

[hmottestad]

As for the general question of restricting resource use between instances. There is an option to set a timeout on queries, though I'm not sure if this can be controlled centrally without the users being able to override it at query time.

I'm not aware of us having any way to restrict CPU or memory usage. I think there is a way to restrict how many concurrent connections to the database are allowed but not sure if this might just be on the instance level rather than the repo level.