Add Cron job to run large submission tests

[jbygdell]

The test for finalize is expected to fail for now.

Related issue(s) and PR(s)

This PR closes #1504 .

Description

These tests create DB entries for 100k files, sends the messages to the MQ.

The test suite is run as a cronjob on Saturdays, all 4 tests are run in parallel in order to cope with the max runtime of 6h for a job.

Expected results

• finalize - failed test due to timeout
• ingest - passed test, 100k messages sent to the error queue
• mapper- passed test, no errors
• verify - passed test, 100k messages sent to the error queue

How to test

Once merged there is an option to manually trigger the tests.

[MalinAhlberg]

In general, this looks very good to me!

Some details are not clear to me, however. Shouldn't we check what error messages we get in the error queue? To make sure that they are due to files that can't be found, and not something else?

[jbygdell]

In general, this looks very good to me!

Some details are not clear to me, however. Shouldn't we check what error messages we get in the error queue? To make sure that they are due to files that can't be found, and not something else?

Well, the only test that is of relevance is the test for mapper, the other ones will behave the same regardless of the number of messages/files.

[pahatz]

Looks good!

[aaperis]

Looks good but I can't seem to remember why it's ok to have the finalize test failing.