Volume testing in OpenRiak

[martinsumner]

A quick review of how volume testing in OpenRiak is done at the moment.

The primary branch used for testing is - https://github.com/OpenRiak/basho_bench/tree/mas-nhs-conditional

The test config used is - https://github.com/OpenRiak/basho_bench/blob/mas-nhs-conditional/examples/riakc_nhs_general.config. By default this will test using mainly PB API (except for 2i queries which use http). All of the operations have *_http flavour, and when resilience testing is being done, testing is done with a HTTP-load-balancer, using the http operations.

The test example assumes two buckets - a record bucket and a document bucket. The use of the document bucket is defined by:

{unique, {10000, key_order}}.
{operations, [ ... 
                {put_unique, 90}, {get_unique, 130}, {delete_unique, 25} ...
]}.

There is no mutation of the document bucket, objects are put in in key_order and are all of size 10KB. Sometimes the objects are deleted (in key order - so there is a range of active document items to fetch from). Gets are only for objects that exist within the range - not_found is not expected.

For the record bucket, use is defined by:

{postcode_indexcount, 6}.

{value_generator, {semi_compressible, 75000, 15000, 10, 0.05}}.

%% For alwaysget operations what is:
%% - the maximum number of keys per worker (max number of keys = this * concurrent)
%% - whether the inserts should be in key_order
{alwaysget, {2000000, 700000, skew_order}}.

{operations, [{alwaysget_pb, 620}, {alwaysget_updatewith2i, 130}, 
                ...
                {postcodequery_http, 2}, {dobquery_http, 3}]}.

With this configuration the minimum size of each object will be 75KB, but each object will be on average 15KB bigger (so the average object size is 90KB). The objects will be semi-compressible (a combination of compressible text and base64 encoded random bytes). 5% of the time though, an object will be PUT that is 10 x bigger than this (i.e. minimum 750KB, average of 900KB).

Each basho_bench worker keeps its own unique range of keys. There are up to 2M keys per worker (so with 100 workers this will be a key space of 200M record bucket entries). The skew_order defines that keys used will be in a random order (lexicographically). In the initial part of the test, until each worker has added 700K keys (so the database has grown to 70M records with 100 workers), no GETs will be performed. Once the minimum key volume has been reached then GET load will be added.

For each update operation a random key is selected. If the key has already been added, an update will be performed. If it has not been added a fresh PUT. As the size grows the volume of updates relative to inserts increase.

All GETs follow an 80/20 rule - 80% of the time a fetch will be taken from the same 20% of the key space - to simulate relatively hot ranges of keys.

The count of indexes is defined by postcode_indexcount. There is also a single "dob" index per object, and a single "lmd" index. the count of results returned by the 2i operations (postcodequery/dobquery) increases over time, and is periodically logged to the basho_bench console log.

[martinsumner]

Once the results have been returned they cna be collated by any means. The charts used recently for public presentations were produced in google sheets.

This is an example sheet - https://docs.google.com/spreadsheets/d/1Zzgt1l088jDCy6gto5X5zTXq1sxOkx7uKsZ-mjktMNs/edit?usp=sharing

Results can be imported into the "Input" sheet, and there is then a summary section at the end of that sheet, which can be copied into the "Summary" sheet. Comparisons are then made between runs by creating a sheet with a single "Accumulated updates" column with a transaction rate column for each test run.