- calibration - camera calibration script
- keypoint_detection - implementation of the keypoint detection approach
- masking - semantic segmentation network and scripts
- segmentation_optimization - optimization of the mask, main function, plotting
1.) Nvidia drivers should be at least Version 410.x or higher for GPU acceleration
2.) Install docker: https://docs.docker.com/install/
For Ubuntu 19.10 disco/ eoan might not work yet, just change the the package query to:
sudo add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
disco \
stable"
3.) Install nvidia-docker toolkit: https://github.com/NVIDIA/nvidia-docker
4.) Checkout our training docker with: sudo docker pull grosenick/robotmasking:1.2
Start the docker with:
sudo docker run --name masker --gpus all --rm --mount type=bind,source=/REPLACE/THIS/masking,target=/app \
--memory=12g -it grosenick/robotmasking:1.2 mask.sh
It will continuously scan for new images in the masking/masking_input directory and output them to masking/output
To change the default configuration change 'masking/mask_images.sh'. For the command line options run python3 masking/mask_images.py --help.
By default will continuously take all pictures inside the 'masking/masking_input' directory and output them to 'masking/output'. Additionally by default input pictures won't get deleted but moved into masking/masking_input/seen. You might want to configure which outputs you need. Defaults will generate all PNGs and NPYs, but no the scored mask.
The segmentation_optimization/main.py runs a optimization process of a given number of images and saves the results if wished. It collects all .jpg files in the input folder (excluding *_keypoint_result.jpg and *_detected_keypoints.jpg named ones) and optimizes the given amount of images starting with the first found image as default option.
Further configuration (like folders and optimization parameters) is possible in segmentation_optimization/configuration.py
usage: main.py [-h] [-n NUM_IMAGES] [--input_dir INPUT_DIR]
[--output_dir OUTPUT_DIR] [--mask_dir MASK_DIR]
[--start_image START_IMAGE] [--save_png] [--show]
[--multiprocessing] [--print_summary]
For a list of all options simply run:
python3 main.py --help
Input data folders can be either specified via command line or configured in the configuration.py.
All files belonging to one specific image need the same basename and be in the same directory.
INPUT_PATH (in the config) or --input_dir via command line needs the following files:
- basename.jpg - the image of the robot
- basename_points.json - the 3D keypoints of the robot in a json file
- basename_keypoint_predictions.json - the result of the keypoint prediction approach as a json file
- optionally basename_expected.json - list of 2D Points on the expected location on the image of the corresponding image, used for evaluation only (--print_summary need to be set)
MASK_DATA_DIR (in the config) or --mask_dir needs the following files:
- basename_binary.npy --> binary mask of the robot on the corresponding image, output from the neural network
- basename_bounding.csv --> csv file containing the coordinates of the bounding box on the corresponding image, output from the neural network
OUTPUT_PATH/--output_dir the following files are created:
- basename_result.png --> image of the result is saved only if the --save_png argument is set
- basename_result.json --> json file containing the optimized 2d points, the corresponding 3d points, optimized translation and rotation vector (with respect to given camera matrix and distortion coefficients)
Optimize all found images in the input folder (from configuration.py) and saves the results into the output directory (from configuration.py):
python3 segmentation_optimization/main.py --save_png
Optimizes all found images in the given directories using multithreading (all available threads!)
python3 segmentation_optimization/main.py --input_dir data/mini --output_dir data/mini_out \
--mask_dir data/nn_masking_complex --multiprocessing
A few handy functions for creating plots. Will show the plots interactively where they can then be saved.
usage: plot.py [-h] [--boxplot] [--histogram] [--scatterplot] [--single] directory
positional arguments:
directory directory of the result files as generated by the main.py
optional arguments:
-h, --help show this help message and exit
--boxplot generate boxplot
--histogram generate histogram
--scatterplot generate scatterplot
--single don't show comparison, generate only single graphs
To get access to the robot via scipt ROS needs to be installed. ROS is released as distributions that require specific Linux distributions. In the scope of this project, the ROS distribution Melodic and Ubuntu 18.04 was used. In addition to that, the ur modern driver https://github.com/ros-industrial/ur_modern_driver needs to be installed.
For reading the keypoints, the script keypoint3D_reader.py was made. To use this script, ROS needs to be connected to the robot. To do so, the following bash command needs to be executed, where ROBOT_IP_ADDRESS needs to replaced by the correct IP address.
roslaunch ur_modern_driver ur5_bringup.launch robot_ip:=ROBOT_IP_ADDRESS
In general the script listens to the keyboard and reads the output stream of the UR5 robot as soon as the space key is pressed. The read keypoints are saved as a json file.
Local optimzation of parameters in configuration.py. Initial values for optimization are defined in line 106 the variable 'init'.
The global variable conf_content is the fixed content of the configuration.py that is not optimized. The lines with parameters that are optimized, are written during the optimzation process. The reason for rewriting the whole configuration.py is that the function thats been optimized runs a subprocess of the main.py, that way, changing the values in configuration.py within this script, does not affect the values that are used in the subprocess.
usage: tweak_parameters.py [-h] [--images IMAGES]
optional arguments:
-h, --help show this help message and exit
--images IMAGES number of first n images being optimized