Merge pull request #1 from jn9e9/initial-dev

Initial stab at test generator

Author: hug-dev

.github/workflows/build.yml

@@ -0,0 +1,33 @@
+name: Python build
+
+on: [push]
+
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        # pyyaml version we use only supporting python 3.6 upwards
+        python-version: [ 3.6, 3.7, 3.8]
+
+    steps:
+    - uses: actions/checkout@v2
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v2
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install flake8 pytest
+        if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
+    - name: Lint with flake8
+      run: |
+        # stop the build if there are Python syntax errors or undefined names
+        flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics --exclude generator/protobuf
+        # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
+        flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics --exclude generator/protobuf
+#    - name: Test with pytest
+#      run: |
+#        pytest

.gitignore

@@ -1,2 +1,4 @@
 /target
 *patch
+.devcontainer
+.vscode

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "parsec-operations"]
+	path = parsec-operations
+	url = https://github.com/parallaxsecond/parsec-operations

Makefile

@@ -0,0 +1,11 @@
+# Copyright 2021 Contributors to the Parsec project.
+# SPDX-License-Identifier: Apache-2.0
+PROTOC_OUTPUT_FILES=$(shell find parsec-operations/protobuf/ -name "*.proto" -exec basename {} .proto \; | awk '{print "generator/protobuf/"$$1"_pb2.py"}')
+.PHONY: protobuf all
+
+all: protobuf
+
+protobuf: ${PROTOC_OUTPUT_FILES}
+
+generator/protobuf/%_pb2.py: parsec-operations/protobuf/%.proto
+	@protoc -I=parsec-operations/protobuf --python_out=generator/protobuf $< > /dev/null

README.md

@@ -1,9 +1,164 @@
 # Parsec Mock
 
-This repository contains a mock service to test clients.
+This repository contains a mock service to test clients, and a test data generator to create test data files for the mock service, or to be used in testing of the parsec service, or parsec clients.
+
 See [this issue](https://github.com/parallaxsecond/parsec/issues/350) which describes the
 proposal.
 
+# Prerequisites
+To run the utilities in this repository, you will need python3 and pip installing.
+
+To install package prerequisites, run the following from this folder:
+
+```bash
+pip install -r requirements.txt
+```
+# Usage
+## Run Mock Service
+**TBD**
+
+## Generate Test Data
+To generate test data from test specs, run 
+
+```
+python generator/generator.py
+```
+
+This will parse the test specs in generator/testspecs and create test data in testdata/.  
+
+# Test Data
+The generator/generator.py script creates test data files in the testdata folder.  These test data files are intended for use, either with the mock service supplied here, or in other mock services that may be developed for the parsec service or parsec clients.  The test data files consist of the [test spec](#test-spec-format) (useful for the writer of a unit test), and test data, which contains base 64 encodes strings for a parsec request and a corresponding result.
+
+The overall test data file format is, therefore:
+
+```yaml
+spec:
+    # for format see below
+test_data:
+  request: EKfAXh4AAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAA
+  result: EKfAXh4AAQAAAAAAAAAAAAAAAAAAAAIAAAAAAAEAAAAAAAAACAE=
+```
+## Expected Use Of Test Data
+This repository does not dictate how the test data should be used by parsec client developers, but it was developed with the following use case in mind:
+
+A client developer wants to test a parsec request in their client.  
+- They select an appropriate test data file that fulfils the needs of their test (or creates a new one in this repository and uses that)
+- They write client code that should generate a message to match the request section of the [test spec](#test-spec-format).
+- They configure a mock service (the one here or a custom one) to expect the data defined in the test_data.request part of the test data file as well as the data to send if that request is received and what to send if it is not.
+- They stimulate their client under test to send the request message to the mock service.
+- The mock service will check to see if the request message it received matches the expected value (this is an exact byte match).
+  - If the match was successful, then it will return the configured match response
+  - Otherwise, it will return the configured non-match response
+- The client under test will attempt to decode the response.  The results can be checked by the test code.
+- If the spec.response.parse_should_succeed field is set to true, then the test code *should* expect a successful parse of the response data.  Otherwise, it *should* expect the parsing to fail.
+
+
+## Writing Tests
+To write a new test, a developer should create a test spec file in the generator/testspecs folder, using the [format below](#test-spec-format). 
+
+They then need to write a generator function in ```generator/generator_lib.py```.  This generator function takes no arguments, and should output a tuple of python binary strings.  
+
+The first element of the tuple should be the protocol buffer encoded value of the request's *operation* message that is described in the spec.request.body_description field of the test spec.  The second element of the tuple should be the protocol buffer encoded *result* value of the response message that is described in the spec.result.body_description field of the test spec.
+
+An example generator for a list opcodes test is shown below:
+
+```python
+def gen_list_opcodes_auth_direct():
+    op = protobuf.list_opcodes_pb2.Operation()
+    op.provider_id = 1
+
+    result = protobuf.list_opcodes_pb2.Result()
+    result.opcodes.extend([1,3,2])
+    return (op.SerializeToString(), result.SerializeToString())
+```
+The generated protocol buffers python library files are in generator/protobuf.
+
+Finally, the developer should add an entry to the generator dictionary at the bottom of the ```generator/generator_lib.py``` file.  The key must correspond to the value in the test spec's spec.generator field.  The value in the generator dictionary must be the new generator function name.  e.g.:
+
+```python
+generators = {
+    'ping_no_auth': gen_ping_no_auth,
+    'list_opcodes_auth_direct': gen_list_opcodes_auth_direct,
+}
+```
+**NOTE**  A generator function can be used in multiple test specs, if required.
+
+## Test Spec Format
+
+Test specs are defined as YAML.  An example file is shown below:
+
+```yaml
+# Copyright 2021 Contributors to the Parsec project.
+# SPDX-License-Identifier: Apache-2.0
+spec:
+  name: list_opcodes_auth_direct
+  generator: list_opcodes_auth_direct
+  description: List opcodes using direct authentication
+  request:
+    header:
+      magic_number: 0x5EC0A710
+      header_size: 0x1E
+      major_version_number: 0x01
+      minor_version_number: 0x00
+      flags: 0x0000
+      provider: 0x00
+      session_handle: 0x0000000000000000
+      content_type: 0x00
+      accept_type: 0x00
+      auth_type: 0x00
+      content_length: auto
+      auth_length: auto
+      opcode: 0x00000009
+      status: 0x0000
+    body_description: provider id 0x01
+    auth: 
+      type: direct
+      app_name: jimbob
+  result:
+    header:
+      magic_number: 0x5EC0A710
+      header_size: 0x1E
+      major_version_number: 0x01
+      minor_version_number: 0x00
+      flags: 0x0000
+      provider: 0x00
+      session_handle: 0x0000000000000000
+      content_type: 0x00
+      accept_type: 0x00
+      auth_type: 0x00
+      content_length: auto
+      auth_length: auto
+      opcode: 0x00000009
+      status: 0x0000
+    body_description: list opcodes result [1,3,2]
+    auth: 
+      type: direct
+      app_name: jimbob
+
+```
+The whole test spec is defined in the spec: object in the file.  In that spec, the fields are:
+| Field | Description |
+| --- | --- |
+| name | The name of the test spec.  Used to name the output test data file |
+| generator | The lookup for the operation and result generator in the generator library.  See [writing tests](#writing-tests).|
+| description | Description of test, not used in generation |
+| request | Settings for configuring the request data for the test data file |
+| request.header | Field values for the request header.  Meanings of these files and valid values can be found in the [parsec book](https://parallaxsecond.github.io/parsec-book/parsec_client/wire_protocol.html). |
+| request.header.content_length | If this field has the value ```auto``` then its value is calculated from the generated request content.  If the value is a number, then that value is used in the field instead. |
+| request.header.auth_length | If this field has the value ```auto``` then its value is calculated from the generated auth content.  If the value is a number, then that value is used in the field instead. |
+| request.body_description | A free form description of the contents of the request content.  Used for test writers (and generator writers) to understand how to construct the request object.  This field is not used by the test generator. |
+| request.auth.type | This can either be ```none```, which will cause the auth section of the message to be empty (equivalent of the No Authentication type of authentication); or it can be ```direct```, which will cause the auth section of the message to contain authentication data corresponding to the format for Direct Authentication.  If ```direct``` is set, then the request.auth.app_name field must be set.  Note that this field does not cause the header auth_type field to be set. |
+| request.auth.app_name | Used to populate the auth section of the message when ```direct``` authentication is selected |
+| response | Settings for configuration the response data for the test data file |
+| response.header | Field values for the response header.  Meanings of these files and valid values can be found in the [parsec book](https://parallaxsecond.github.io/parsec-book/parsec_client/wire_protocol.html). |
+| response.header.content_length | If this field has the value ```auto``` then its value is calculated from the generated response content.  If the value is a number, then that value is used in the field instead |
+| response.header.auth_length | If this field has the value ```auto``` then its value is calculated from the generated auth content.  If the value is a number, then that value is used in the field instead. |
+| response.body_description | A free form description of the contents of the response content.  Used for test writers (and generator writers) to understand how to construct the result object.  This field is not used by the test generator. |
+| response.auth.type | This can either be ```none```, which will cause the auth section of the message to be empty (equivalent of the No Authentication type of authentication); or it can be ```direct```, which will cause the auth section of the message to contain authentication data corresponding to the format for Direct Authentication.  If ```direct``` is set, then the result.auth.app_name field must be set.  Note that this field does not cause the header auth_type field to be set.  **NOTE:**  A Parsec client would not send authentication data in a result message, but this spec format allows test authors to create the message as they wish to excersice the parsec client code. |
+| response.auth.app_name | Used to populate the auth section of the message when ```direct``` authentication is selected |
+
+
+
 # License
 
 The software is provided under Apache-2.0. Contributions to this project are accepted under the same

generator/.gitignore

@@ -0,0 +1 @@
+__pycache__

generator/__init__.py

@@ -0,0 +1,13 @@
+# Copyright 2021 Contributors to the Parsec project.
+# SPDX-License-Identifier: Apache-2.0
+import protobuf.ping_pb2
+import protobuf.list_opcodes_pb2
+
+
+def gen_list_opcodes_auth_direct():
+    operation = protobuf.list_opcodes_pb2.Operation()
+    operation.provider_id = 1
+
+    result = protobuf.list_opcodes_pb2.Result()
+    result.opcodes.extend([1, 3, 2])
+    return (operation.SerializeToString(), result.SerializeToString())

generator/gen_list_operations.py

@@ -0,0 +1,13 @@
+# Copyright 2021 Contributors to the Parsec project.
+# SPDX-License-Identifier: Apache-2.0
+import protobuf.ping_pb2
+import protobuf.list_opcodes_pb2
+
+
+def gen_ping_no_auth():
+    op = protobuf.ping_pb2.Operation()
+    result = protobuf.ping_pb2.Result()
+    result.wire_protocol_version_maj = 1
+    result.wire_protocol_version_min = 0
+
+    return (op.SerializeToString(), result.SerializeToString())

generator/gen_ping_operations.py

@@ -0,0 +1,143 @@
+# Copyright 2021 Contributors to the Parsec project.
+# SPDX-License-Identifier: Apache-2.0
+import os
+from os import listdir
+from os.path import isfile, join
+
+from yaml import safe_load, dump
+
+from struct import pack
+import base64
+
+from generator_lib import generators
+
+
+class TestSpec(object):
+    """Class to represent a test specification.  Used to convert
+    dictionary created by pyaml to object format, making code easier to read."""
+
+    def __init__(self, dictionary):
+        def _traverse(key, element):
+            if isinstance(element, dict):
+                return key, TestSpec(element)
+            else:
+                return key, element
+
+        objd = dict(_traverse(k, v) for k, v in dictionary.items())
+        self.__dict__.update(objd)
+        self.basedict = dictionary
+
+    def is_valid(self):
+        return True
+
+
+def read_specs(folder):
+    """Read test specs from a folder"""
+    specfiles = [f for f in listdir(folder) if isfile(join(folder, f))]
+    specs = []
+    for file in specfiles:
+        print(f"Parsing spec file: {file}")
+        with open(os.path.join(folder, file), 'r') as f:
+            spec = safe_load(f)
+            testspec = TestSpec(spec["spec"])
+            if testspec.is_valid():
+                specs.append(testspec)
+            else:
+                print(f"Error loading test spec from {file}")
+    return specs
+
+
+def generate_data(specs, output_folder):
+    """Generate test data for a list of specs"""
+    for spec in specs:
+        if spec.generator in generators:
+            print(f"Generating test {spec.name}")
+            generate_spec_data(output_folder, spec, generators[spec.generator])
+        else:
+            print(f"No generator function found for spec {spec.name}, skipping...")
+
+
+def generate_spec_data(output_folder, spec, generator_fn):
+    """Generates data for a single spec and outputs it into the specified output folder."""
+    (operation, result) = generator_fn()
+
+    request_auth = create_auth(spec.request.auth)
+    request_content_len = spec.request.header.content_length
+    if request_content_len == 'auto':
+        request_content_len = len(operation)
+
+    request_auth_len = spec.request.header.auth_length
+    if request_auth_len == 'auto':
+        request_auth_len = len(request_auth)
+    request_header = pack_header(spec.request.header, request_auth_len, request_content_len)
+
+    response_content_len = spec.response.header.content_length
+    if response_content_len == 'auto':
+        response_content_len = len(result)
+
+    response_auth_len = spec.response.header.auth_length
+
+    response_header = pack_header(spec.response.header, response_auth_len, response_content_len)
+
+    request_buf = request_header + operation + request_auth
+    response_buf = response_header + result
+
+    out_data = {
+        "spec": spec.basedict,
+        "test_data": {
+            "request": base64.b64encode(request_buf).decode('ascii'),
+            "response": base64.b64encode(response_buf).decode('ascii'),
+        }
+    }
+    out_path = os.path.join(output_folder, spec.name + ".test.yaml")
+    print(f"Writing spec {spec.name} test data to {out_path}")
+    with open(out_path, 'w') as f:
+        dump(out_data, f, sort_keys=False)
+
+
+def pack_header(header, auth_len, body_len):
+    """Take a header data structure and convert it into binary representation."""
+    # pack function converts arguments into binary string, based on format string.
+    # < means integers are little endian.  Rest of format string is one character per input to indicate
+    # packed field interpretation.  See struct.pack docs for details.
+    return pack('<IHBBHBQBBBIHIHH',
+                header.magic_number,
+                header.header_size,
+                header.major_version_number,
+                header.minor_version_number,
+                header.flags,
+                header.provider,
+                header.session_handle,
+                header.content_type,
+                header.accept_type,
+                header.auth_type,
+                body_len,
+                auth_len,
+                header.opcode,
+                header.status,
+                0
+                )
+
+
+def create_auth(auth_spec):
+    """Creates auth body of message"""
+    if auth_spec.type == 'none':
+        return b''
+    if auth_spec.type == 'direct':
+        return auth_spec.app_name.encode('utf-8')
+    return b''
+
+
+def main():
+    specdir = os.path.abspath(os.path.join(os.path.dirname(__file__), 'testspecs'))
+    datadir = os.path.abspath(os.path.join(os.path.dirname(__file__), '../testdata'))
+    print("Generating test data.")
+    print(f"Reading test specs from {specdir}")
+    print(f"Generating test data to {datadir}")
+    specs = read_specs(specdir)
+
+    generate_data(specs, datadir)
+
+
+if __name__ == "__main__":
+    main()