Proposal: testing clients with a mock service

[hug-dev]

Integration testing in Parsec clients

Parsec clients need to be tested as well. The integration tests of clients can check that:

• requests are correctly serialiased according to the wire protocol and body format
• response are correctly deserialised
• any client logic for automatic provider/authenticator selection is correct
• any logic based on any kind of capability discovery is correct (not available in Parsec yet)

For this testing, clients developpers would often create the means of mocking the service side to check that the request serialised bytes are as expected and to reply to the client with specific response bytes, that are deserialised and checked as well. This is currently done in the Rust client and similarly proposed in the Go client, in the linked PR.

Moreover, it is fair to assume that any Parsec client would have structures and methods to be able to send requests with any opcode, any authentication method, to any provider and getting any response. Basically letting the client user tweak all possible parameters of the request header and set its own body/authentication data. Those with idiomatic and high level structure in the specific language. The Rust client has the notion of the OperationClient for that and the Go client has the operation method.

See this PR which started this idea.

Having a mock service to help

In order to stop duplicating the mocking test framework, to put in common and improve the effort for client testing and to help future client developpers to test it, I propose the following:

1. Create an independant mock service (probably in its own repo). This service will do nothing else but listening on a UDS for a byte buffer, comparing this buffer with some data and replying another data.
2. Create a set (or multiple) of test data.
3. Depending on its starting option, the mock service will use one of the set of test data.

The test data will consist of:

1. the full specification of a whole request, in high-level terms:

• the content of all fields of the fixed header
• the content of the authentication body
• high-level description of the content of the body

2. the same request, but serialised (shown as base64)
3. the full specification of a possible corresponding response to the request, can be an erroneous one, in high-level terms as well
4. the same response but serialised

A test will consist of the following steps:

• the client forms the request corresponding to 1, using the high-level facilities provided by its implementation, and send it to the mock service. I think the operation client or the basic client are good levels for this, if it's possible to still set all request parameters.
• the mock service compares the serialised request with 2, and errors out if it does not match.
• if it matches, the mock service responds with 4
• the client deserializes 4 with the facilities at hand and compares it with the expected response 3. It errors out if it does not match

The mock service will expect that there is a specific order to all the tests, and will process one after the other. Another way would be to use one of the field to specify the test ID, like the session identifier which is currently not used? However that will be a problem in the future if it gets used.

Example of test data

Here is an example of 1:

Header:
* Magic number: 0x5EC0A710
* Header size: 0x1E
* Major version number: 0x01
* Minor version number: 0x00
* Flags: 0x0000
* Provider: 0x00
* Session handle: 0x0000000000000000
* Content type: 0x00
* Accept type: 0x00
* Auth type: 0x01
* Content length: (don't know yet, to be calculated)
* Auth length: 0x0004
* Opcode: 0x00000009
* Status: 0x0000
Body: ListOpcodes operation with 0x01 as provider_id
Auth: "toto" UTF-8 encoded

Here is an example of 3:

Header:
* Magic number: 0x5EC0A710
* Header size: 0x1E
* Major version number: 0x01
* Minor version number: 0x00
* Flags: 0x0000
* Provider: 0x00
* Session handle: 0x0000000000000000
* Content type: 0x00
* Accept type: 0x00
* Auth type: 0x00
* Content length: (don't know yet, to be calculated)
* Auth length: 0x0000
* Opcode: 0x00000009
* Status: 0x0000
Body: ListOpcodes result with [1, 2, 3] as opcodes

Example of using it in the Rust client, the PARSEC_SERVICE_ENDPOINT variable would be set to point to the mock service.

use parsec_client::BasicClient;
let client = BasicClient::new_naked();
client.set_auth_data(Authentication::Direct("toto".to_string()));
let opcodes = client.list_opcodes(ProviderId::MbedCrypto).unwrap();
assert_eq!(opcodes, vec![1, 2, 3]); //simplified

Note that not all test data will be able to be converted using an high-level client like the BasicClient. I think the rule of thumb should be that the highest-level possible client should be used. If not all request parameters can be set or if not all response fields can be checked, a lower level client should be used instead.
The reason is that I think, the lowest level a client is, the fewer logic is actually tested with this.

Generating the test data and hosting the mock service

The mock service and test data would be in the same repository.
It should be the simplest possible and ideally not re-use one existing client, otherwise we might be testing against a buggy client!
Safety is not primordial here, so Python could be used for its convenience and its nice file handling. Rust could be used as well, but should not use any existing Parsec-related crate.
I think 1 and 3 should be written in human language, similarly than the specifications in our book.
The test data generator would contain the framework to easily create test files from the human-language specifications and produce test data files containing 1, 2, 3 and 4 that both the mock service and the clients' tests can use.
This part can be discussed, I am not too sure if using a specified format like JSON is a good idea to describe a full request as it will create one more thing to convert and doing that programmatically could lead to more bugs/errors than manually doing it.

[ionut-arm]

The mock service will expect that there is a specific order to all the tests, and will process one after the other. Another way would be to use one of the field to specify the test ID, like the session identifier which is currently not used? However that will be a problem in the future if it gets used.

I was thinking of something else: the mock service has a bunch of files (test1.req, test1.resp...) which it loads when it starts up an stores in a map between req and resp. When it gets a request, it checks in the map and replies with the corresponding resp.

We can differentiate between different tests issuing the same request and getting different responses by using the authentication field as a test ID.

[hug-dev]

I was thinking of something else: the mock service has a bunch of files (test1.req, test1.resp...) which it loads when it starts up an stores in a map between req and resp. When it gets a request, it checks in the map and replies with the corresponding resp.

That is a good idea! Then the check of the request is done via the fact that the entry is not found in the hash map. If that is the case, the tester can go to the repo and compare expected with actual payloads. Maybe in the future some kind of Parsec on the mock service side would still be useful to check which exact fields do not match for example.

We can differentiate between different tests issuing the same request and getting different responses by using the authentication field as a test ID.

I guess just using the auth field as a way to produce requests that have the same other fields but would produce another entry in the hash map would work!

[ionut-arm]

That is a good idea! Then the check of the request is done via the fact that the entry is not found in the hash map. If that is the case, the tester can go to the repo and compare expected with actual payloads. Maybe in the future some kind of Parsec on the mock service side would still be useful to check which exact fields do not match for example.

That's true, it would be difficult to tell which parameter was incorrect. Maybe if we make it so that the auth field is unique for all requests sent we can map them like that and be able to compare field by field.

Alternatively, we split the expected requests into folders (i.e. tests) and have them numbered (1.req, 1.resp, 2.req, 2.resp...) and you differentiate between tests using the auth field. Then you can compare what was expected with what was provided. (by this I mean we keep some state in the mock as to what request we're expecting for every test)

[jn9e9]

Like the idea of the mock client behind the UDS- should be worth noting that this test mocking is also worth doing in a unit test context with a mocked connection implementation (i.e. not needing the UDS) - this makes for fast simple tests with one level of complexity removed.

Making the test data and means to generate it common (with a good way to use it) will help a lot for client devs.

[jn9e9]

Are you saying that the test data generator would take a formalised human readable format as per 1, 3 and convert those to a simple json/yaml machine parsable model for client test suites? If not, then we should do that, please, or write specs in machine readable format and have translator to human friendly.

As a client developer, I want test data files to be machine readable, so that I can use existing unmarshalling libraries and so I don't have to create a parser for them as well as the protocol.

[hug-dev]

should be worth noting that this test mocking is also worth doing in a unit test context with a mocked connection implementation (i.e. not needing the UDS) - this makes for fast simple tests with one level of complexity removed.

Agree that generally more testing is anyway better! I think the hard part is making sure the test data is correct, and that's hard to do if you don't have one source of generation that you trust... Having one common place for this kind of test (specifically integration testing), will help for that and make sure that globally all clients are of the same "minimal correctness".

Are you saying that the test data generator would take a formalised human readable format as per 1, 3 and convert those to a simple json/yaml machine parsable model for client test suites?

In my originial idea there would not be any automatic conversion between the formalised human readable format (1, 3) and the serialised parts (2, 4).

The process I thought of would look like this.
On the mock service side, Bob would:

• come up with 1 and 3 that would be nice to test (test name: toto) and write the formalised human readable format in a markdown file: toto_request.md and toto_response.md
• uses the test generator tool to manually serialise 1 into 2 and 3 into 4 to create toto_request.base64 and toto_response.base64. "Manually" is a bit vague here but I was thinking literally writing a small Python program calling a master function with a lot of arguments each corresponding to the full specification of the request: the value of each field, the description of each payload. That would have to be done only once for each test. The call to the master method would probably have to be recorded in the repo somewhere.

On the client test's side, Alice would read and understand toto_request.md and toto_response.md and come up with either one or multiple tests that:

• generate the exact request as described in toto_request.md, using the API of her choice
• check that the response is as described in toto_response.md

Now, I agree that it seems strange (or crazy 🤯) to not replace the "formalised human readable format" by JSON/TOML but consider the following arguments:

• Writing the request header in JSON/TOML seems fine but describing the operation in this format would be equivalent of creating a new operation format and I don't think we want to go into that complexity. It is going to be very painful/tedious to describe PSA key attributes/algorithms in JSON/TOML. I think it is not worth it and the time taken to simply manually call a function won't take that long, specially because it is going to be copy and paste mostly. Also then, the function call could become the specification, as it would define programatically the full request/response. If it is Python, then basically we would have to write a minimal Python client and the Python method to generate the request/response can be the format.
• There is not a single way for Alice to create the test generating 2 from 1 and checking for 3 from 4. Alice could use any client (operation client or basic client) to write the test and could use them all. Also currently we are just thinking about checking serialisation functionnality but the mock service could also be useful to check for the BasicClient logic: for example checking that the correct behaviour happens when ListProviders return such and such provider, etc...

[hug-dev]

Going further with the above proposal.

Suggestion for the repository name: parsec-mock
Directory structure:

parsec-mock.py
generator/
    generator.py
    # probably other files
data/
    list-clients.py
    ping.py
    # and more with meaningfull names

Contents:

• the generator directory will contain the methods to easily create requests and responses, fully defined. This will contain the protobuf generated code and the wire header representation. The methods will produce base64 representations of requests and responses.
• the data directory will contain the files (here Python) containing the code to generate the request and response data for one particular test. For simplicity, each file can also directly contain, at the top, the base64 representation generated by the file itself (in comments).
• parsec-mock.py is the Parsec Mock service. On startup, it will browse all data files and look at the top of them the base64 requests and responses which it will load in its big hash table. Then it will listen for incoming requests, and for each request: check if there is one match in the hash table (if not error out) and return the corresponding response.

[hug-dev]

I created the repository here: https://github.com/parallaxsecond/parsec-mock

[jn9e9]

trying to get my head around the process of creating one of these tests - for the sake of argument TESTA. I think, I'm going to write your 1 and 3 constructs into a file called TESTA.spec. I will then create a gen_TESTA function in python (and add it to the list of generators to run). In that function, I will load TESTA.spec and get it to build the wire headers for request and response automatically (I really don't want to duplicate that, we'll introduce too many errors in the transposition). In that method, I also create (in code, by manually reading the spec) the request body and response body, as well as (potentially) the authentication data. The code can then automatically create the full request and response messages, and write them out in base64 encoded form to TESTA.req and TESTA.resp. - either that, or they get put into, say a json or yaml file that contains the test name+description and request and response.

[jn9e9]

i guess the mock client could be configured to work in one of a number of ways:

1 it could load a number of request/response pairs into hashmap and just send the responses that correspond to the request - this mode may also have a default reponse if nothing recieved.
2 it could load a number of request/response pairs and move through them for each recieved request, again, possibly with a default for non match.
3 it could, conceivably, have a hybrid, where we have a set of request response pairs, grouped up (x tests in group 1, y tests in group 2) and then advance through the groups, possibly with each request.
4 The test client could control the next set of test data using, e.g. a second unix socket, either setting the next test file, or the next group of tests

Suggest that the selection of tests is supplied to the mock client by a config file - yaml would do - could be something like this:

---
testsuite:
  description: option1 
  mode:  hashbag
  default_response: TESTQ
  tests:
    - TESTA
    - TESTB
    - TESTC
    - TESTZ
 ---
testsuite:
  description: option2 
  mode:  sequential
  default_response: TESTQ
  tests:
    - TESTA
    - TESTB
    - TESTC
    - TESTZ
---
testsuite:
  description: option3 
  mode:  sequential
  default_response: TESTQ
  groups:
    group
      name: first group
      - tests:
        - TESTA
        - TESTB    
    group
      name: second group
      - tests:
        - TESTD
        - TESTE
    
---
testsuite:
  description: option4 
  mode:  remote_control
    control_socket: ./mock.sock
  default_response: TESTQ
  groups:
    group
      name: first group
      - tests:
        - TESTA
        - TESTB    
    group
      name: second group
      - tests:
        - TESTD
        - TESTE

---
testsuite:
  description: option4.1 
  mode:  remote_control
    control_socket: ./mock.sock
  default_response: TESTQ
  tests:
    - TESTA
    - TESTB
    - TESTC
    - TESTZ

In any event, suggest we start with option 1 and move on.

I'm not a fan, BTW of embedding control messages in the protocol messages themselves - it restricts what you can test and also makes message matching very messy - without it we can cope with a simple equality/map, unless we get to the point of difficult authenticators where we can't guarantee the content.

If we're going to try and put effort into control messages, lets put it into making a dedicated control channel! The control channel, could, either just select the next lot of testing files from the configuration, or could actually configure the mock service on the fly: E.g.: activate "second group"; switch to test "TESTA"
or load hashbag with files TESTA, TESTB and TESTZ.

A control channel could also return status if required. A simple JSON request/response interface on a unix socket would be easy to implement in any language (I'm ignoring C for the momeent, but even C++ has brilliant json support!).

[hug-dev]

Ok so if we separate this in between the test generator and the mock service.

For the test generator, agreed with what you said. Having one function per test seems necessary to combine the automatic and manual steps involved. Maybe the wire header representation on the .spec file could actually be the Deserialised representation of its contents? I don't know Python too deeply (if Python is used), but it would be nice to automatically convert the wire header representation (which is easy enough to be JSON, YAML, TOML) into the serialised form that we want. As I explained above, this would be too cumbersome and complicated for the bodies (operations/responses/authentication).

For the mock service, I think having a control channel is a good idea! The information that needs to be shared is basically:

• from the client to the mock service: what test is going to be run
• from the mock service to the client: was the request sent correct? If not, maybe, what exactly did not match?

I would be happy to make things very simple for now and then having breaking changes later. As long as the mock service is versioned this is fine I think. As it is a test thing, I think it's ok to not come with a perfect future-proof design just now and we can allow making things extra simple and incrementally improve.

[jn9e9]

sounds sensible - we could follow a REST type convention - define the service using a path, which could include the version - e.g. /v1/setTest...

Next question is whether the results are polled for on the control channel or broadcast to subscribers - polling would be simpler to implement and likely simpler for a test client to implement.

[hug-dev]

Next question is whether the results are polled for on the control channel or broadcast to subscribers - polling would be simpler to implement and likely simpler for a test client to implement.

Instead of polling we could also use time outs on the socket? I guess the flow would be:

• client sends request and then blocks on waiting for response (for like 1 second)
• mock service checks that request is correct
  • if it is, sends response -> client knows it was correct
  • if it's not, does not send response -> clients time outs and then check for the control socket to see what was wrong

As the mock service does very little logic it should be quite fast so the timeout can be very low.

[ionut-arm]

Just wanted to point out something from the top comment:

The reason is that I think, the lowest level a client is, the fewer logic is actually tested with this.

I don't think the purpose of the mock service should be to test as much logic as possible in the client, but rather to test that the logic in the lower levels that handle the actual serialising/deserializing of requests and responses is correct. It's not really a complete solution, of course, because the cases where the lower levels reject a request because of inconsistencies (if they can happen) aren't considered. But if those layers do end up sending something over the wire and we can confirm that it is correctly formatted, then anything built on top should be fine from that point of view.

[jn9e9]

Next question is whether the results are polled for on the control channel or broadcast to subscribers - polling would be simpler to implement and likely simpler for a test client to implement.

Instead of polling we could also use time outs on the socket? I guess the flow would be:

* client sends request and then blocks on waiting for response (for like 1 second)

* mock service checks that request is correct
  
  * if it is, sends response -> client knows it was correct
  * if it's not, does not send response -> clients time outs and then check for the control socket to see what was wrong

As the mock service does very little logic it should be quite fast so the timeout can be very low.

we could put some sort of long poll with timeout in the logic in the control channel - wait for x seconds and either return failure or test result if it comes earlier - timeout at application layer rather than socket may be pleasanter. Toying with idea of just doing REST calls over unix socket (or even tcp as it probably doesn't matter for control channel??) Flask would make this pretty trivial either for either socket choice, and most languages have simple APIs for doing REST calls.

[ionut-arm]

Do we still need this issue? Now that we have a mock service repo, we could just host these conversations there.

Moved to the new repo!