roboflex.realsense

Support for the Realsense D435 camera. We really just wrap the sdk and encode data into messages. It might work with other D4** cameras - YMMV.

System Dependencies

Sort of: none. We use cmake to build. If find_package(realsense2) fails (meaning cmake can't find some system-installed version of realsense2), we fetch and build it.

Install

pip install roboflex.realsense

Import

import roboflex.realsense as rr

Nodes

There is only one: RealsenseSensor

You can use a class method to discover the serial numbers of all the cameras currently connected to your computer:

rr.RealsenseSensor.get_connected_device_serial_numbers()

There are two ways to instantiate realsense sensors:

1. Via the constructor - you must know the serial number. See below for documentation of the Config type.

    sensor = rr.RealsenseSensor(
        serial_number: str,
        config: rr.Config,
        name: str = "RealsenseSensor",
    )
   

2. Via a class method - if you have a single realsense attached, this is the easiest way - you don't even have to know the serial number.

    sensor = rr.RealsenseSensor.get_one_sensor(
        config: rr.Config,
        name: str = "RealsenseSensor",
    )
   

Use it like so:

# must be started!
sensor.start()

# You can get the serial number:
sensor.serial_number -> str

# You can get the configuration back:
sensor.config -> rr.Config

# You can get the two camera-k matrices:
sensor.depth_camera_k -> np.array
sensor.color_camera_k -> np.array

# You can get various device values:
sensor.width_pixels_color -> int
sensor.height_pixels_color -> int
sensor.width_pixels_depth -> int
sensor.height_pixels_depth -> int
sensor.fps_color -> int
sensor.fps_depth -> int

# You can manually trigger a message event:
# (for the most part you won't need this - only
# if you have special need to control the message
# production), and don't want to use "start".
sensor.produce()

# You can turn the IR laser off and on
sensor.set_laser_on_off(False)
sensor.set_laser_on_off(True)

Messages

RealsenseSensor produces a single message type: RealsenseFrameset.

from roboflex.realsense import RealsenseFrameset

API:

# the timestamp just before reading from device
message.t0 -> Float

# the timestamp just after reading from device
message.t1 -> Float

# numpy array of shape=(height, width, 3), dtype=uint8
message.rgb -> np.ndarray

# numpy array of shape=(height, width), dtype=uint16
message.depth -> np.ndarray

# numpy array of shape=(height, width), dtype=uint8
message.ir1 -> np.ndarray

# numpy array of shape=(height, width), dtype=uint8
message.ir2 -> np.ndarray

# which camera (if any) the frame is aligned to
message.aligned_to -> rr.CameraType

# the serial number of the camera that produced this frameset
message.serial_number -> str

# the color camera k of the camera that produced this frameset
message.camera_k_rgb -> numpy array of (3, 3)

# the depth camera k of the camera that produced this frameset
message.camera_k_depth -> numpy array of (3, 3)

# the ir1 camera k of the camera that produced this frameset
message.camera_k_ir1 -> numpy array of (3, 3)

# the ir2 camera k of the camera that produced this frameset
message.camera_k_ir2 -> numpy array of (3, 3)

# the index, from the device, of this frameset
message.frame_number -> int

# the timestamp, from the device, of this frameset, in epoch seconds
message.timestamp -> float

DYNOFLEX:

d = DynoFlex.from_msg(message)

# the timestamp just before reading from device
d["t0"] -> Double

# the timestamp just after reading from device
d["t1"] -> Double

# numpy array of shape=(height, width, 3), dtype=uint8
d["rgb"] -> np.ndarray

# numpy array of shape=(height, width), dtype=uint16
d["depth"] -> np.ndarray

# numpy array of shape=(height, width), dtype=uint8
d["ir1"] -> np.ndarray

# numpy array of shape=(height, width), dtype=uint8
d["ir2"] -> np.ndarray

# which camera (if any) the frame is aligned to
d["aligned_to"] -> rr.CameraType

# the serial number of the camera that produced this frameset
d["serial_number"] -> str

# the color camera k of the camera that produced this frameset
d["camera_k_rgb"] -> numpy array of (3, 3)

# the depth camera k of the camera that produced this frameset
d["camera_k_depth"] -> numpy array of (3, 3)

# the camera k of the infrared camera 1 that produced this frameset
d["camera_k_ir1"] -> numpy array of (3, 3)

# the camera k of the infrared camera 2 that produced this frameset
d["camera_k_ir2"] -> numpy array of (3, 3)

# the index, from the device, of this frameset
d["n"] -> int

# the timestamp, from the device, of this frameset, in epoch seconds
d["t"] -> float

Other

Some types used for configuration

Some enums:

CameraType:
    RGB,
    DEPTH,
    IR1,    # raw infrared camera1
    IR2     # raw infrared camera2

# You can OR these together like so:

my_cameras = camera_type_or([CameraType.RGB, CameraType.DEPTH, etc])

# You can test for camera types:

has_rgb = camera_type_contains(my_camera_type, CameraType.RGB)

CameraAlignment:
    NONE,
    RGB,
    DEPTH

D400VisualPreset:
    CUSTOM,
    DEFAULT,
    HAND,
    HIGH_ACCURACY,
    HIGH_DENSITY,
    MEDIUM_DENSITY

TemporalFilterParameters:

f = rr.TemporalFilterParameters(
    alpha: float = 0.4,
    delta: float = 20.0,
    persistance_control: int = 7,
)

f.alpha -> float
f.delta -> float
f.persistance_control -> int

Config:

# all paramters are optional - defaults shown below
c = rr.Config(
    camera_type: rr.CameraType = CameraType.RGB | CameraType.DEPTH,
    align_to: rr.CameraAlignment = rr.CameraAlignment.NONE,
    prioritize_ae: Bool = False,
    rgb_settings: Dict[str, int] = {
        "fps": 0,
        "width": 0,
        "height": 0,
    },
    depth_settings: Dict[str, int] = {
        "fps": 0,
        "width": 0,
        "height": 0,
    },
    depth_visual_preset: rr.D400VisualPreset = rr.D400VisualPreset.DEFAULT,
    temporal_filter_parameters: rr.TemporalFilterParameters = None,
    hole_filling_mode: Optional[int] = None,
    decimation_filter: Optional[int] = None,
)

# which actual cameras are in use - this a bitmask, OR-ed together
c.camera_type -> rr.CameraType

# which camera the frames will be aligned to (if any)
c.align_to -> rr.CameraAligment

# When `true`, allows fps to drop in order to better expose
# frames, such as in dimly lit environments
c.prioritize_ae -> Bool

# fps, height, and width of the color camera
c.rgb_settings -> Dict[str, int]

# fps, height, and width of the depth camera
c.depth_settings -> Dict[str, int]

# the depth camera visual preset
c.depth_visual_preset -> rr.D400VisualPreset

# the temporal filter, if any - can be None
c.temporal_filter_parameters -> rr.TemporalFilterParameters

# the 'hole filling mode', if any - can be None
# 0: fill_from_left
# 1: farthest_from_around
# 2: nearest_from_around
c.hole_filling_mode -> int

# NOTE!
The realsense camera supports many more settings, such as laser power, etc. If there's something you want supported that's not
here, just let me know.