Write an exhaustive test suite

[pieterb]

Tobias Weigel wrote:

What about testing prior to every release? Do you e.g. have automatic CI running somewhere?

Ulrich Schwardmann wrote:

Yes, testing will become an important issue. We had this already discussed on monday. It is still open, whether each site will implement its own testing environment or there will be one testing server with the newest software stack. I would recommend the last way. At the moment SARA has a testing server in function, but this still uses the database of the production server, which is possible but still too risky from my point of view.

Tobias Weigel wrote:

My opinion: Testing the active development branch is a must, but testing individual developer's branches is also a very helpful idea to detect bugs early on.
At DKRZ, we've made solid experience with Jenkins [1]. What systems are currently in use at SARA and GWDG?
With Jenkins (and probably any other system), it is no problem to also set up some scripting to get individual branches tested in the same environment as the main branch, giving all the benefits of only having to maintain one base stack for all sites. But as you say, it's an open issue.

[pieterb]

Well, at the moment, we don't even have an exhaustive test suite, although lately one of my colleagues wrote some scripts that test the core web service functionality, as part of our Nagios-based service monitoring system at SARA.

And Tibor has started writing some Cucumber tests. I saw they've been checked in here at GitHub, but I didn't have time to take a look at them yet. I think that, ideally, we could take the following steps:

1. Complete an exhaustive test suite with Cucumber, which can serve as
   • runtime service monitoring test
   • version release quality test
   • development/integration test
2. Once we have this, we could start with Test Driven Development (TDD) or Behavior Driven Development (BDD), as Cucumber calls it.
3. As the development team grows, we might start with Continuous Integration (CI). I think that, in this phase of development, it's a bit overkill. Opinions are welcome.

[TobiasWeigel]

I don't think CI would be overkill if you have already decided to write tests. From writing tests to CI it is basicly just a few more steps:

1. set up a CI system - not too hard as there are ones that work out of the box
2. set up the testing env - this is the part that takes most work
3. wiring the test cases to the CI system

Regarding point 2., it is of course a main decision on how exhaustive to do this. Do you just load new code to an existing JRuby env or do you set up a completely new env, starting from scratch with no dependencies installed etc.?
The latter option would have uncovered the issue with d8e6762 even without exhaustive test cases, because it was a configuration error. Given that we did spent at least one collaborative hour trying to narrow this down, I don't think CI is too wasteful.
I do however agree that CI is not a top priority while we are still getting the governance and organizational framework together.

Hello Everyone,

in order to ensure that Tibor and my development on the EPIC API does not break any existing functionality new cucumber features using Httparty were written today. The features cover all CRUD operations of the service. However they are not an exhaustive test-suite. The tests can be found under the commit: 15689cc in the tkalman branch.

Best Regards Oliver