Cucumber Test Suite

The correct functionality of OSRM is supported by a test suite based on Cucumber (http://cukes.info). If you don't want to run tests, you can build and run OSRM without doing any of the following.

Cucumber itself is written in Ruby, and is distributed as a Ruby gem. All tests are written in Ruby using the natural language Gherkin syntax.

Prerequisites

• ruby: should be pre-installed on most systems.
• gems: cucumber, rake, osmlib-base. the easiest and safest is to first install the 'bundler' gem, then run - run 'bundle install' from the project folder.
• osmosis: command line tool for converting and cropping osm files. If you're on Mac OS X you can install it via homebrew package system with 'brew install osmosis'. Please consult the Osmosis Wiki page on how to install on your platform. Please make sure that the 'osmosis' command is found in one path of your path variables.

For some reason on Ruby 1.8.7 (on Ubuntu) some of the gems use an invalid date format. If you edit the gem specification which are raising the issue and remove the invalid date format, the issue will be resolved.

vi /var/lib/gems/1.8/specifications/term-ansicolor-1.0.7.gemspec

and change the s.date = line to the following: s.date = %q{2011-10-13} and save it.

Files

Tests are located in a subdirectory called 'features/' and organized into feature files, each containing scenarios.

Further subdirectories are 'test/' and 'sandbox/'. 'test/' is where all the tests are run. 'sandbox/' is where you can use the rake task to download real OSM data, run the server manually to test in a browser, etc. Both 'test/' and 'sandbox/' folders contain a .stxxl config file. The one in the 'test/' folder uses a 1MB disk, the other one in 'sandbox/' uses a 1GB disk.

Mechanics

For each test scenario, an OSM data file is constructed based on the description in the test scenario. The OSM file is converted to protobuffer and preprocessed with osrm-extract and osrm-prepare. The resulting osrm files are cached, so that this preprocessing is only repeated if either the osm data, the speedprofile or the binaries have changed. This helps to speed up the running of tests a lot.

'test/server.ini' and 'test/speedprofile.ini' is now rewritten to ensure that the right data files are used. osrm-routed is then launched, and routes are requested via http calls to port 5000 on localhost, and the response is compared to the expected result. All request are timed out after 5 seconds. both timeouts and crashes are caught and reported. To ensure a clean run of every test, all processes named osrm-extract, osrm-prepare and osrm-routed are killed before and after each test using a signal 9, no matter who launched them.

Most of the development of the test suite has been done and tested on Mac OS X 10.7. It should work on Linux, but process management can often behave differently depending on platform. Tests are not yet expected to work on Windows, for example the UNIX shell command 'ps' is used. There may be other issue too.

Writing tests

For Cucumber documentation, see https://github.com/cucumber/cucumber/wiki/A-Table-Of-Content.

The OSRM tests currently come in two main flavors.

With the first type, you specify a small 'node map', a list of ways, and optionally relations. You can then specify a list of routes to be tests, described by start and end points, and the expected route. Route are described by the ways traversed:

Scenario: Zig Zag
	Given the nodes
	 | a |   | c |
	 |   | b |   |

	And the ways
	 | nodes |
	 | ab    |
	 | bc    |
	
	When I route I should get
	 | from | to | route |
	 | a    | c  | ab,bc |
	 | c    | a  | bc,ab |

An OSM file is constructed based on the nodes, ways and relations specified, and used for testing.

By default, tests use the bicycle speedprofile in speedprofile/bicycle.ini, but this can be changed for each scenario by adding:

	Given the speedprofile "car"

You can also modify the speedprofile by using:

	Given the speedprofile "car"
	And the speedprofile settings
 	| obeyOneways | no |
 	| primary     | 50 |

Note that 'Given' and 'And' is interchangeable in cucumber, you should just use what creates a natural language flow.

The second flavor of tests is used for checking whether a way with a specific tag combination should be routable in the forward and backwards direction:

Scenario: Simplest possible oneway
	Given the speedprofile "car"
 	Then routability should be
	 | highway | oneway | forw | backw |
	 | primary |        | x    | x     |
	 | primary | yes    | x    |       |
	 | primary | -1     |      | x     |

An OSM file is constructed, containing one isolated way for each row. Column headers are used to specify tag keys. forw and backw has special meaning; if an forw column is present the way is tested for forward routability. if an backw row is present, it's tested for backwards routability.

Nodes are spaced on grid with origin at 1,1 and a grid size of 100m.

Running tests:

You can run all tests, or specific sets or scenarios.

• to run all test: cucumber (or 'rake test')
• to run test with a specific tag, use: cucumber --tags @restrictions
• to run a specific scenario: cucumber features/restrictions.feature:6

Routes are valided by parsing the instructions hash. During the tests, several log files are used. Output from osrm-extract and osrm-prepare are redirected to test/preprocess.log. Output from osrm-routed is redirected to test/osrm-routed.log.

Test results are shown in the terminal. Green rows indicate a passed route, yellow is an expected, but missing, result, while grey is an unexpected result. Failed row will be displayed right below the expected row. Note that there's currently an issue, which means that if there are incorrect rows, some additional correct rows might be marked as incorrect.

Additional info is logged to fail.log, containing details about the osm data used, the speedprofile, expected and returned routes, as well as actual query and response involved. Beware, there may be a number of failing tests, currently.

Rake task

A few useful rake task are included. You can run 'rake -T' to get a list of rake commands.

For example, you can run 'rake download ' to download and crop osm data into your sandbox folder. Currently, this is limited to a few fixed areas work, including:

kbh: openhagen dk: denmark

After downloading, you can run 'rake process' to process the downloaded data. To run your server in the terminal, use 'rake run', but you must download and process data beforehand.

You can also run the server in the background with 'rake up' and 'rake down'. Output is redirected to osrm-routed.log. Again, OSM data must be downloaded and preprocessed beforehand. Note however, that the included index.html file doesn't point to localhost, which is what you need to use the server you run locally.