Add basic autotests setup

README.md

@@ -59,6 +59,8 @@ config.json # Specify the name of the evaluation function in this file
    - Edit the `app/evaluation_tests.py` and `app/preview_tests.py` files to add tests which get run:
        - Every time you commit to this repo, before the image is built and deployed 
        - Whenever the `healthcheck` command is supplied to the deployed function
+   - Edit the `eval_tests.yaml` file to add tests that get run on the evaluation function as above,
+   but are also added to the documentation as examples.
    - Edit the `app/docs.md` file to reflect your changes. This file is baked into the function's image, and is made available using the `docs` command. This feature is used to display this function's documentation on our [Documentation](https://lambda-feedback.github.io/Documentation/) website once it's been hooked up!
 
 ---

app/evaluation_tests.py

@@ -1,11 +1,13 @@
 import unittest
+from autotests import auto_test
 
 try:
     from .evaluation import Params, evaluation_function
 except ImportError:
     from evaluation import Params, evaluation_function
 
 
+@auto_test("../eval_tests.yaml", evaluation_function)
 class TestEvaluationFunction(unittest.TestCase):
     """
     TestCase Class used to test the algorithm.

app/requirements.txt

@@ -0,0 +1 @@
+autotests @ git+https://github.com/lambda-feedback/evaluation-function-auto-tests@main

eval_tests.yaml

@@ -0,0 +1,9 @@
+# A placeholder test file.
+# The tests described here are run on the evaluation function by Github Actions,
+# and can be used to automatically generate examples for the documentation.
+# You should populate this file with tests for your evaluation function.
+# See https://github.com/lambda-feedback/evaluation-function-auto-tests for a
+# detailed explanation of the test file format.
+---
+title: placeholder
+tests: []