Skip to content

Latest commit

 

History

History
510 lines (451 loc) · 21.6 KB

customizing_pipeline_requests.md

File metadata and controls

510 lines (451 loc) · 21.6 KB

Customizing Video Analytics Pipeline Requests

| Request Format | Source | Destination | Parameters | Tags |

Pipeline requests are initiated to exercise the Intel(R) Deep Learning Streamer (Intel(R) DL Streamer) Pipeline Server REST API. Each pipeline in the Pipeline Server has a specific endpoint. A pipeline can be started by issuing a POST request and a running pipeline can be stopped using a DELETE request. The source and destination elements of Pipeline Server pipeline templates are configured and constructed based on the source and destination from the incoming requests.

Request Format

Note: This document shows curl requests. Requests can also be sent via vaclient using the --request-file option see VA Client Command Options

Pipeline requests sent to Pipeline Server REST API are JSON documents that have the following attributes:

Attribute Description
source Required attribute specifying the video source that needs to be analyzed. It consists of :
uri : the uri of the video source that needs to be analyzed
type : is the value uri
destination Optional attribute specifying the output to which analysis results need to be sent/saved. It consists of metadata and frame
parameters Optional attribute specifying pipeline parameters that can be customized when the pipeline is launched.
tags Optional attribute specifying a JSON object of additional properties that will be added to each frame's metadata.

Example Request

Below is a sample request using curl to start an object_detection/person_vehicle_bike pipeline that analyzes the video person-bicycle-car-detection.mp4 and sends its results to /tmp/results.json.

Note: Files specified as a source or destination need to be accessible from within the Pipeline Server container. Local files and directories can be volume mounted using standard docker runtime options. As an example the following command launches a Pipeline Server container with the local /tmp directory volume mounted. Results to /tmp/results.jsonl are persisted after the container exits.

docker/run.sh -v /tmp:/tmp
curl localhost:8080/pipelines/object_detection/person_vehicle_bike -X POST -H \
'Content-Type: application/json' -d \
'{
    "source": {
        "uri": "https://github.com/intel-iot-devkit/sample-videos/blob/master/person-bicycle-car-detection.mp4?raw=true",
        "type": "uri"
    },
    "destination": {
        "metadata": {
            "type": "file",
            "path": "/tmp/results.jsonl",
            "format": "json-lines"
        }
    },
    "parameters":{
        "threshold": 0.90
   }
}'
2

The number returned on the console is the pipeline instance id (e.g. 2). As the video is being analyzed and as objects are detected, results are added to the destination file which can be viewed using:

tail -f /tmp/results.jsonl
{"objects":[{"detection":{"bounding_box":{"x_max":0.7503407597541809,"x_min":0.6836109757423401,"y_max":0.9968345165252686,"y_min":0.7712376117706299},"confidence":0.93408203125,"label":"person","label_id":1},"h":97,"roi_type":"person","w":51,"x":525,"y":333}],"resolution":{"height":432,"width":768},"source":"https://github.com/intel-iot-devkit/sample-videos/blob/master/person-bicycle-car-detection.mp4?raw=true","timestamp":1916666666}
{"objects":[{"detection":{"bounding_box":{"x_max":0.7554543018341064,"x_min":0.6827328205108643,"y_max":0.9928492903709412,"y_min":0.7551988959312439},"confidence":0.92578125,"label":"person","label_id":1},"h":103,"roi_type":"person","w":56,"x":524,"y":326}],"resolution":{"height":432,"width":768},"source":"https://github.com/intel-iot-devkit/sample-videos/blob/master/person-bicycle-car-detection.mp4?raw=true","timestamp":2000000000}
<snip>

Source

The source attribute specifies the video source that needs to be analyzed. It can be changed to use media from different sources. Some of the common video sources are:

  • File Source
  • IP Camera (RTSP Source)
  • Web Camera

Note: See Source Abstraction to learn about GStreamer source elements set per request.

File Source

The following example shows a media source from a video file in GitHub:

curl localhost:8080/pipelines/object_detection/person_vehicle_bike -X POST -H \
'Content-Type: application/json' -d \
'{
    "source": {
        "uri": "https://github.com/intel-iot-devkit/sample-videos/blob/master/person-bicycle-car-detection.mp4?raw=true",
        "type": "uri"
    },
    "destination": {
        "metadata": {
            "type": "file",
            "path": "/tmp/results.jsonl",
            "format": "json-lines"
        }
    }
}'

A local file can also be used as a source. In the following example person-bicycle-car-detection.mp4 has been downloaded to /tmp and Pipeline Server was started as:

docker/run.sh -v /tmp:/tmp
curl localhost:8080/pipelines/object_detection/person_vehicle_bike -X POST -H \
'Content-Type: application/json' -d \
'{
    "source": {
        "uri": "file:///tmp/person-bicycle-car-detection.mp4",
        "type": "uri"
    },
    "destination": {
        "metadata": {
            "type": "file",
            "path": "/tmp/results.jsonl",
            "format": "json-lines"
        }
    }
}'

RTSP Source

RTSP streams from IP cameras can be referenced using the rtsp uri scheme. RTSP uris will normally be of the format rtsp://<user>:<password>@<ip_address>:<port>/<server_url> where <user> and password are optional authentication credentials.

The request source object would be updated to:

{
    "source": {
        "uri": "rtsp://<ip_address>:<port>/<server_url>",
        "type": "uri"
    }
}

Web Camera Source

Web cameras accessible through the Video4Linux api and device drivers are supported via type=webcam. device is the path of the v4l2 device, typically video<N>.

curl localhost:8080/pipelines/object_detection/person_vehicle_bike -X POST -H \
'Content-Type: application/json' -d \
'{
    "source": {
      "device": "/dev/video0",
      "type": "webcam"
    },
    "destination": {
        "metadata": {
          "type": "file",
          "path": "/tmp/results.jsonl",
          "format": "json-lines"
        }
    }
}'

Setting source properties

For any of the sources mentioned above, it is possible to set properties on the source element via the request.

Setting a property on source bin element

For example, to set property buffer-size on urisourcebin, source section can be set as follows:

{
    "source": {
        "uri": "file:///tmp/person-bicycle-car-detection.mp4",
        "type": "uri",
        "properties": {
            "buffer-size": 4096
        }
    }
}

Setting a property on underlying element

For example, if you'd like to set ntp-sync property of the rtspsrc element to synchronize timestamps across RTSP source(s).

Note: This feature, enabled via GStreamer source-setup callback signal is only supported for urisourcebin element.

{
    "source": {
        "uri": "rtsp://<ip_address>:<port>/<server_url>",
        "type": "uri",
        "properties": {
            "ntp-sync": true
        }
    }
}

Destination

Pipelines can be configured to output frames, metadata or both. The destination object within the request contains sections to configure each.

  • Metadata (inference results)
  • Frame

Metadata

For metadata, the destination type can be set to file, mqtt, or kafka as needed.

File

The following are available properties:

  • type : "file"
  • path (required): Path to the file.
  • format (optional): Format can be of the following types (default is json):
    • json-lines : Each line is a valid JSON.
    • json : Entire file is formatted as a JSON.

Below is an example for JSON format

curl localhost:8080/pipelines/object_detection/person_vehicle_bike -X POST -H \
'Content-Type: application/json' -d \
'{
    "source": {
        "uri": "https://github.com/intel-iot-devkit/sample-videos/blob/master/person-bicycle-car-detection.mp4?raw=true",
        "type": "uri"
    },
    "destination": {
        "metadata": {
            "type": "file",
            "path": "/tmp/results.json",
            "format": "json"
        }
    }
}'

MQTT

The following are available properties:

  • type : "mqtt"
  • host (required) expects a format of host:port
  • topic (required) MQTT topic on which broker messages are sent
  • timeout (optional) Broker timeout
  • mqtt-client-id (optional) Unique identifier for the MQTT client

Steps to run MQTT:

  1. Start the MQTT broker, here we use Eclipse Mosquitto, an open source message broker.
docker run --network=host eclipse-mosquitto:1.6
  1. Start the Pipeline Server with host network enabled
docker/run.sh -v /tmp:/tmp --network host
  1. Send the REST request : Using the default 1883 MQTT port.
curl localhost:8080/pipelines/object_detection/person_vehicle_bike -X POST -H \
'Content-Type: application/json' -d \
'{
  "source": {
      "uri": "https://github.com/intel-iot-devkit/sample-videos/blob/master/person-bicycle-car-detection.mp4?raw=true",
      "type": "uri"
  },
  "destination": {
      "metadata": {
          "type": "mqtt",
          "host": "localhost:1883",
          "topic": "pipeline-server",
          "mqtt-client-id": "gva-meta-publish"
      }
  }
}'
  1. Connect to MQTT broker to view inference results
docker run -it --network=host --entrypoint mosquitto_sub eclipse-mosquitto:1.6 --topic pipeline-server --id mosquitto-sub
{"objects":[{"detection":{"bounding_box":{"x_max":1.0,"x_min":0.11904853582382202,"y_max":0.9856844246387482,"y_min":0.019983917474746704},"confidence":0.5811731815338135,"label":"vehicle","label_id":2},"h":417,"roi_type":"vehicle","w":677,"x":91,"y":9}],"resolution":{"height":432,"width":768},"source":"https://github.com/intel-iot-devkit/sample-videos/blob/master/person-bicycle-car-detection.mp4?raw=true","timestamp":13916666666}
{"objects":[{"detection":{"bounding_box":{"x_max":0.3472719192504883,"x_min":0.12164716422557831,"y_max":1.0,"y_min":0.839308500289917},"confidence":0.6197869777679443,"label":"vehicle","label_id":2},"h":69,"roi_type":"vehicle","w":173,"x":93,"y":363}],"resolution":{"height":432,"width":768},"source":"https://github.com/intel-iot-devkit/sample-videos/blob/master/person-bicycle-car-detection.mp4?raw=true","timestamp":14333333333}
{"objects":[{"detection":{"bounding_box":{"x_max":0.3529694750905037,"x_min":0.12145502120256424,"y_max":1.0,"y_min":0.8094810247421265},"confidence":0.7172137498855591,"label":"vehicle","label_id":2},"h":82,"roi_type":"vehicle","w":178,"x":93,"y":350}],"resolution":{"height":432,"width":768},"source":"https://github.com/intel-iot-devkit/sample-videos/blob/master/person-bicycle-car-detection.mp4?raw=true","timestamp":14416666666}
  1. In the MQTT broker terminal, you should see the connection from client with specified mqtt-client-id
<snip>
1632949258: New connection from 127.0.0.1 on port 1883.
1632949258: New client connected from 127.0.0.1 as gva-meta-publish (p2, c1, k20).
1632949271: New connection from 127.0.0.1 on port 1883.
1632949271: New client connected from 127.0.0.1 as mosquitto-sub (p2, c1, k60).
1632949274: Client gva-meta-publish disconnected.

Kafka

The following are available properties:

  • type : "kafka"
  • host (required) expects a format of host:port
  • topic (required) Kafka topic on which broker messages are sent

Steps to run Kafka:

  1. Prepare to run a Kafka broker. Since Kafka relies on ZooKeeper for management, let's create docker-compose-kafka.yml with the following content:

    version: "2"
services:
  zookeeper:
    image: docker.io/bitnami/zookeeper:3.7
    ports:
      - "2181:2181"
    volumes:
      - "zookeeper_data:/bitnami"
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    image: docker.io/bitnami/kafka:2
    ports:
      - "9092:9092"
    volumes:
      - "kafka_data:/bitnami"
    environment:
      - KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE=true
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092
      - ALLOW_PLAINTEXT_LISTENER=yes
    depends_on:
      - zookeeper
volumes:
  zookeeper_data:
    driver: local
  kafka_data:
    driver: local

  2. Run the following command to launch Kafka broker as a detached service:

    docker-compose -p pipeline-server -f docker-compose-kafka.yml up -d

  3. Start the Pipeline Server with host network enabled:

    docker/run.sh -v /tmp:/tmp --network host

  4. Launch pipeline with parameters to emit on the Kafka topic we are listening for:

    ./vaclient/vaclient.sh start object_detection/person_vehicle_bike  \
https://github.com/intel-iot-devkit/sample-videos/blob/master/person-bicycle-car-detection.mp4?raw=true  \
--destination type kafka  \
--destination host localhost \
--destination port 9092  \
--destination topic pipeline-server.person_vehicle_bike

  5. Connect to Kafka broker to view inference results:

    docker exec -it pipeline-server_kafka_1 /opt/bitnami/kafka/bin/kafka-console-consumer.sh \
   --bootstrap-server localhost:9092 --topic pipeline-server.person_vehicle_bike
    {"objects":[{"detection":{"bounding_box":{"x_max":0.7448995113372803,"x_min":0.6734093427658081,"y_max":0.9991495609283447,"y_min":0.8781012296676636},"confidence":0.5402464866638184,"label":"person","label_id":1},"h":52,"roi_type":"person","w":55,"x":517,"y":379}],"resolution":{"height":432,"width":768},"source":"https://github.com/intel-iot-devkit/sample-videos/blob/master/person-bicycle-car-detection.mp4?raw=true","timestamp":1500000000}
{"objects":[{"detection":{"bounding_box":{"x_max":0.7442193031311035,"x_min":0.6763269901275635,"y_max":1.0,"y_min":0.8277983069419861},"confidence":0.5505848526954651,"label":"person","label_id":1},"h":74,"roi_type":"person","w":52,"x":519,"y":358}],"resolution":{"height":432,"width":768},"source":"https://github.com/intel-iot-devkit/sample-videos/blob/master/person-bicycle-car-detection.mp4?raw=true","timestamp":1666666666}
{"objects":[{"detection":{"bounding_box":{"x_max":0.7465137243270874,"x_min":0.6821863651275635,"y_max":1.0,"y_min":0.810469388961792},"confidence":0.6447391510009766,"label":"person","label_id":1},"h":82,"roi_type":"person","w":49,"x":524,"y":350}],"resolution":{"height":432,"width":768},"source":"https://github.com/intel-iot-devkit/sample-videos/blob/master/person-bicycle-car-detection.mp4?raw=true","timestamp":1750000000}
{"objects":[{"detection":{"bounding_box":{"x_max":0.7481285929679871,"x_min":0.6836653351783752,"y_max":0.9999656677246094,"y_min":0.7867168188095093},"confidence":0.8825281858444214,"label":"person","label_id":1},"h":92,"roi_type":"person","w":50,"x":525,"y":340}],"resolution":{"height":432,"width":768},"source":"https://github.com/intel-iot-devkit/sample-videos/blob/master/person-bicycle-car-detection.mp4?raw=true","timestamp":1833333333}

  6. NOTE: When finished, remember to close running containers with this command:

    docker-compose -f docker-compose-kafka.yml down

Frame

Frame is another aspect of destination and it can be set to RTSP.

RTSP

RTSP is a type of frame destination supported. The following are available properties:

  • type : "rtsp"
  • path (required): custom string to uniquely identify the stream
  • cache-length (default 30): number of frames to buffer in rtsp pipeline.
  • encoding-quality (default 85): jpeg encoding quality (0 - 100). Lower values increase compression but sacrifice quality.
  • sync-with-source (default True): rate limit processing pipeline to encoded frame rate (e.g. 30 fps)
  • sync-with-destination (default True): block processing pipeline if rtsp pipeline is blocked.

For more information, see RTSP re-streaming

Parameters

Pipeline parameters as specified in the pipeline definition file, can be set in the REST request. For example, below is a pipeline definition file:

{
	"type": "GStreamer",
	"template": ["uridecodebin name=source",
				" ! gvadetect model={models[object_detection][person_vehicle_bike][network]} name=detection",
				" ! gvametaconvert name=metaconvert ! gvametapublish name=destination",
				" ! appsink name=appsink"
			],
	"description": "Person Vehicle Bike Detection based on person-vehicle-bike-detection-crossroad-0078",
	"parameters": {
		"type": "object",
		"properties": {
			"detection-device": {
				"element": {
					"name": "detection",
					"property": "device"
				},
				"type": "string"
			},
			"detection-model-instance-id": {
				"element": {
					"name": "detection",
					"property": "model-instance-id"
				},
				"type": "string"
			},
			"inference-interval": {
				"element": "detection",
				"type": "integer"
			},
			"threshold": {
				"element": "detection",
				"type": "number"
			}
		}
	}
}

Any or all of the parameters defined i.e detection-device, detection-model-instance-id, inference-interval and threshold can be set via the request.

curl localhost:8080/pipelines/object_detection/person_vehicle_bike -X POST -H \
'Content-Type: application/json' -d \
'{
  "source": {
      "uri": "https://github.com/intel-iot-devkit/sample-videos/blob/master/person-bicycle-car-detection.mp4?raw=true",
      "type": "uri"
  },
  "destination": {
      "metadata": {
          "type": "file",
          "path": "/tmp/results.jsonl",
          "format": "json-lines"
      }
  },
  "parameters": {
      "detection-device": "GPU",
      "detection-model-instance-id": "1",
      "threshold": 0.90
  }
}'

For the example above as threshold was set to 0.90 (default value 0.5), the metadata would only contain results where the confidence exceeds 0.90

{"objects":[{"detection":{"bounding_box":{"x_max":0.7503407597541809,"x_min":0.6836109757423401,"y_max":0.9968345165252686,"y_min":0.7712376117706299},"confidence":0.93408203125,"label":"person","label_id":1},"h":97,"roi_type":"person","w":51,"x":525,"y":333}],"resolution":{"height":432,"width":768},"source":"https://github.com/intel-iot-devkit/sample-videos/blob/master/person-bicycle-car-detection.mp4?raw=true","timestamp":1916666666}
{"objects":[{"detection":{"bounding_box":{"x_max":0.7554543018341064,"x_min":0.6827328205108643,"y_max":0.9928492903709412,"y_min":0.7551988959312439},"confidence":0.92578125,"label":"person","label_id":1},"h":103,"roi_type":"person","w":56,"x":524,"y":326}],"resolution":{"height":432,"width":768},"source":"https://github.com/intel-iot-devkit/sample-videos/blob/master/person-bicycle-car-detection.mp4?raw=true","timestamp":2000000000}
{"objects":[{"detection":{"bounding_box":{"x_max":0.7566969394683838,"x_min":0.683247447013855,"y_max":0.9892041087150574,"y_min":0.7453113198280334},"confidence":0.95263671875,"label":"person","label_id":1},"h":105,"roi_type":"person","w":56,"x":525,"y":322}],"resolution":{"height":432,"width":768},"source":"https://github.com/intel-iot-devkit/sample-videos/blob/master/person-bicycle-car-detection.mp4?raw=true","timestamp":2083333333}
{"objects":[{"detection":{"bounding_box":{"x_max":0.7583206295967102,"x_min":0.6872420907020569,"y_max":0.9740238189697266,"y_min":0.7231987714767456},"confidence":0.95947265625,"label":"person","label_id":1},"h":108,"roi_type":"person","w":55,"x":528,"y":312}],"resolution":{"height":432,"width":768},"source":"https://github.com/intel-iot-devkit/sample-videos/blob/master/person-bicycle-car-detection.mp4?raw=true","timestamp":2166666666}

For more details on parameters, see Pipeline Parameters

Tags

Tags are pieces of information specified at the time of request, stored with frames metadata. In the example below, tags are used to describe the location and orientation of video input.

curl localhost:8080/pipelines/object_detection/person_vehicle_bike -X POST -H \
'Content-Type: application/json' -d \
'{
    "source": {
        "uri": "https://github.com/intel-iot-devkit/sample-videos/blob/master/person-bicycle-car-detection.mp4?raw=true",
        "type": "uri"
    },
    "destination": {
        "metadata": {
            "type": "file",
            "path": "/tmp/results.json",
            "format": "json"
        }
    },
    "tags": {
        "camera_location": "parking_lot",
        "direction" : "east"
    }
}'

Inference results are updated with tags

{
  "objects": [
      {
          "detection": {
              "bounding_box": {
                  "x_max": 0.7448995113372803,
                  "x_min": 0.6734093427658081,
                  "y_max": 0.9991495609283447,
                  "y_min": 0.8781012296676636
              },
              "confidence": 0.5402464866638184,
              "label": "person",
              "label_id": 1
          },
          "h": 52,
          "roi_type": "person",
          "w": 55,
          "x": 517,
          "y": 379
      }
  ],
  "resolution": {
      "height": 432,
      "width": 768
  },
  "source": "https://github.com/intel-iot-devkit/sample-videos/blob/master/person-bicycle-car-detection.mp4?raw=true",
  "tags": {
      "camera_location": "parking_lot",
      "direction": "east"
  },
  "timestamp": 1500000000
}