Exceptions should be caught when setting up the workspaces and cause jobs to...

[Foxcapades]

What should happen when the creation/setup of a job fails in the platform before the job is executed?

• Should we assume it's a hiccup and clear out the job with a 500 so they can try resubmitting?

• Should we just fail the job before it starts?

• Submitting a job to the DB (we didn't even manage to record the job)

• Submitting a workspace to S3 (we recorded the job in the DB but S3 operation(s) failed)

• Submitting a job to the queue (we recorded the job in the DB and created an S3 workspace, but rabbit threw an exception)

[Foxcapades]

@dmgaldi points out that if we get a client timeout when creating and S3 workspace and fail the request, all subsequent requests to create that job will fail due to the workspace already existing (s3-workspaces throws an exception for double creating a workspace)