Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improved webhooks documentation #1174

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Improved webhooks documentation #1174

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
241 changes: 81 additions & 160 deletions docs/source/webhooks.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,116 +1,89 @@
Webhooks
========

Webhooks allow you to integrate external systems with |st2| using HTTP webhooks. Unlike sensors
Webhooks allow you to integrate external systems with |st2| using HTTP webhooks. Unlike sensors,
which use a "pull" approach, webhooks use a "push" approach. They push triggers directly to the
|st2| API using HTTP POST requests.

Sensors vs Webhooks
-------------------
## Sensors vs Webhooks

Sensors integrate with external systems and services using either a polling approach (sensors
periodically connect to an external system to retrieve data), or a passive approach, where they
listen on some port, receiving data using whatever custom protocol you define.
Sensors integrate with external systems using either a polling approach (sensors periodically
connect to an external system to retrieve data) or a passive approach, where they listen on some
port, receiving data using a custom protocol.

Webhooks provide a built-in passive approach for receiving JSON or URL-encoded form data, via
HTTP POST. This data must be "pushed" from an external system to |st2| when an interesting event
Webhooks, in contrast, provide a built-in passive approach for receiving JSON or URL-encoded form
data via HTTP POST. This data must be "pushed" from an external system to |st2| when an event
occurs.

Sensors are the preferred integration method since they offer a more granular and tighter
integration.
Sensors are the preferred integration method as they offer a more robust and customizable
approach.

On the other hand, webhooks come in handy when you have an existing script or software which you
can easily modify to send a webhook to the |st2| API when an interesting event occurs.
However, webhooks are useful when:

Another example where webhooks are useful is when you want to consume events from a 3rd party
service that already offers webhook integration - e.g. GitHub.
- You have an existing script or software that can easily send a webhook to the |st2| API.
- You want to consume events from a third-party service that already supports webhook
integrations, such as GitHub or Stripe.

Authentication
--------------
## Authentication

All requests to the ``/api/v1/webhooks`` endpoints need to be authenticated in the same way as other
API requests. There are two possible authentication approaches - :ref:`API keys
<authentication-apikeys>` and tokens. API keys are recommended for webhooks, as they do not
expire. Tokens have a fixed expiry.
All requests to ``/api/v1/webhooks`` must be authenticated, similar to other API requests. You
can use either API keys or tokens:

API key-based
~~~~~~~~~~~~~
| **Authentication Method** | **Usage** |
|--------------------------|---------------------------|
| **API Key (Recommended)** | ``St2-Api-Key`` header or ``?st2-api-key`` query parameter |
| **Token-Based** | ``X-Auth-Token`` header or ``?x-auth-token`` query parameter |

* `Header` : ``St2-Api-Key``
* `Query parameter` : ``?st2-api-key``
API keys are preferred for webhooks as they do not expire, unlike tokens that have a fixed expiry.

Token-based
~~~~~~~~~~~
Headers are recommended for authentication when using scripts, whereas query parameters are often
used when integrating with third-party services that allow only URL-based authentication.

* `Header` : ``X-Auth-Token``
* `Query parameter` : ``?x-auth-token``
## Request Body

Webhooks support two types of payload formats:

Both methods above support providing the authentication material as a header or query parameter.
A header is usually used with your scripts where you can control request headers while query
parameters are used with 3rd party services such as GitHub where you can only specify a URL.
| **Content-Type** | **Format** |
|-----------------|-------------------------------|
| ``application/json`` | JSON payload |
| ``application/x-www-form-urlencoded`` | URL-encoded form data |

Request Body
------------
All examples in this documentation assume JSON input.

The request body or so called trigger payload can be either JSON or URL encoded form data. The
body type is determined based on the value of the ``Content-Type`` header (``application/json``
for JSON and ``application/x-www-form-urlencoded`` for URL encoded form data).
## Registering a Webhook

All the examples below assume JSON and as such, provide ``application/json`` for the
``Content-Type`` header value.

Registering a Webhook
---------------------

You can register a webhook in |st2| by specifying the ``core.st2.webhook`` trigger inside a rule
To register a webhook in |st2|, you must specify the ``core.st2.webhook`` trigger in a rule
definition.

Here is an excerpt from a rule which registers a new webhook named ``sample``:
Example rule registering a webhook named ``sample``:

.. sourcecode:: yaml

...
trigger:
type: "core.st2.webhook"
parameters:
url: "sample"
...

The ``url:`` parameter above is added as a suffix to ``/api/v1/webhooks/`` to create the URL to
POST data to. So once you have created the rule above, you can use this webhook by POST-ing data
to your |st2| server at ``https://{$ST2_IP}/api/v1/webhooks/sample``.

The request body needs to be JSON and may contain arbitrary data which you can match against in
the rule criteria.

Note that all trailing and leading ``/`` of the ``url`` parameter are ignored by |st2|. e.g. a
value of ``/sample``, ``sample/``, ``/sample/`` and ``sample`` are all treated the same, i.e.
considered identical. They all result in an effective URL of ``/api/v1/webhooks/sample``.

POST-ing data to a custom webhook will cause a trigger with the following attributes to be
dispatched:
type: "core.st2.webhook"
parameters:
url: "sample"

* ``trigger`` - Trigger name.
* ``trigger.headers`` - Dictionary containing the request headers.
* ``trigger.body`` - Dictionary containing the request body.
Once registered, you can send data to this webhook by making a POST request to:
``https://{$ST2_IP}/api/v1/webhooks/sample``

This example shows how to send data to a custom webhook using ``curl`` and how to match on this
data using rule criteria:
### Example: Sending a Webhook Request

.. sourcecode:: bash

curl -X POST https://localhost/api/v1/webhooks/sample -H "X-Auth-Token: matoken" -H "Content-Type: application/json" --data '{"key1": "value1"}'
curl -X POST https://localhost/api/v1/webhooks/sample \
-H "X-Auth-Token: matoken" \
-H "Content-Type: application/json" \
--data '{"key1": "value1"}'

Rule:
### Rule to Match Webhook Data

.. sourcecode:: yaml

...
trigger:
type: "core.st2.webhook"
parameters:
url: "sample"
type: "core.st2.webhook"
parameters:
url: "sample"

criteria:
trigger.body.key1:
Expand All @@ -119,34 +92,32 @@ Rule:

action:
ref: "mypack.myaction"
parameters:
...

Using a Generic Webhook
-----------------------
## Using a Generic Webhook

By default, a special-purpose webhook with the name ``st2`` is already registered. Instead of
using ``core.st2.webhook``, it allows you to specify any trigger that is known to |st2| (either by
default or from custom sensors and triggers in packs), so you can use it to trigger rules that
aren’t explicitly set up to be triggered by webhooks.
A built-in webhook named ``st2`` allows triggering rules without explicitly registering a new webhook.
This is useful when you need to trigger rules dynamically.

The body of this request needs to be JSON and must contain the following attributes:
The request payload must contain:

* ``trigger`` - Name of the trigger (e.g. ``mypack.mytrigger``)
* ``payload`` - Object with a trigger payload.
| **Field** | **Description** |
|----------|----------------|
| ``trigger`` | Name of the trigger (e.g., ``mypack.mytrigger``) |
| ``payload`` | JSON object containing trigger payload |

This example shows how to send data to the generic webhook using ``curl``, and how to match this
data using rule criteria (replace ``localhost`` with your |st2| host if called remotely):
### Example: Sending a Generic Webhook

.. sourcecode:: bash

curl -X POST https://localhost/api/v1/webhooks/st2 -H "X-Auth-Token: matoken" -H "Content-Type: application/json" --data '{"trigger": "mypack.mytrigger", "payload": {"attribute1": "value1"}}'
curl -X POST https://localhost/api/v1/webhooks/st2 \
-H "X-Auth-Token: matoken" \
-H "Content-Type: application/json" \
--data '{"trigger": "mypack.mytrigger", "payload": {"attribute1": "value1"}}'

Rule:
### Rule to Match Data

.. sourcecode:: yaml

...
trigger:
type: "mypack.mytrigger"

Expand All @@ -157,91 +128,41 @@ Rule:

action:
ref: "mypack.myaction"
parameters:
...

The ``trigger.type`` attribute in the rule definition needs to be the same as the trigger name
defined in the webhook payload body.
## Listing Registered Webhooks

Listing Registered Webhooks
---------------------------

To list all registered webhooks, run:
To list all registered webhooks:

.. code-block:: bash

st2 webhook list

My Webhook Isn't Working!
-------------------------
## Debugging Webhook Issues

If you're encountering issues with webhooks, such as |st2| failing to recognize incoming webhooks, or trigger
instances not showing when expected, please see :doc:`Troubleshooting Webhooks</troubleshooting/webhooks>`.
If your webhook isn't triggering expected actions, check the logs and troubleshoot using the
``--debug`` flag.

When Not to Use Webhooks
------------------------
## Alternative Tools for Webhook Testing

While webhooks are useful, they do have two drawbacks:
To test webhooks locally before integrating with |st2|, you can use:

* **Not Bidirectional** - Webhooks simply submit data into |st2|. So if you want data back from
|st2|, or an action execution ID, you'll have to get that data in an asynchronous fashion.
* **No Guarantee of Execution** - Webhooks in |st2| do not guarantee an execution. It depends on
the rule configuration. Based upon the webhook contents, it may not execute any action, or may
execute multiple actions.
- **Beeceptor** (https://beeceptor.com) - Mock and inspect webhook requests.
- **Webhook.site** (https://webhook.site/) - Capture and debug webhooks.

If you always want to execute a specific action or workflow, and/or you're looking for a
guaranteed response, you can use the ``/v1/executions`` API. This is the same as explicitly
running an action from the CLI with ``st2 run <mypack>.<myaction>``.
These tools allow you to validate webhook payloads before sending them to |st2|.

We can get a little insight into how this work using the ``--debug`` flag:
## When Not to Use Webhooks

.. sourcecode:: bash
Consider using the ``/v1/executions`` API instead of webhooks if:

- You need a bidirectional response (webhooks only push data into |st2|).
- You require guaranteed execution (webhooks may not always trigger an action).

st2 --debug run core.local "date"
2017-03-31 08:21:18,706 DEBUG - Using cached token from file "/home/ubuntu/.st2/token-st2admin"
# -------- begin 140183979680208 request ----------
curl -X GET -H 'Connection: keep-alive' -H 'Accept-Encoding: gzip, deflate' -H 'Accept: */*' -H 'User-Agent: python-requests/2.11.1' -H 'X-Auth-Token: da5ecf3b0ab841008d663052fe95cddd' http://127.0.0.1:9101/v1/actions/core.local
# -------- begin 140183979680208 response ----------
{"name": "local", "parameters": {"cmd": {"required": true, "type": "string", "description": "Arbitrary Linux command to be executed on the local host."}, "sudo": {"immutable": true}}, "tags": [], "description": "Action that executes an arbitrary Linux command on the localhost.", "enabled": true, "entry_point": "", "notify": {}, "uid": "action:core:local", "pack": "core", "ref": "core.local", "id": "58c9663a49d4af4cbd56f84d", "runner_type": "local-shell-cmd"}
# -------- end 140183979680208 response ------------

# -------- begin 140183979680080 request ----------
curl -X GET -H 'Connection: keep-alive' -H 'Accept-Encoding: gzip, deflate' -H 'Accept: */*' -H 'User-Agent: python-requests/2.11.1' -H 'X-Auth-Token: da5ecf3b0ab841008d663052fe95cddd' 'http://127.0.0.1:9101/v1/runnertypes/?name=local-shell-cmd'
# -------- begin 140183979680080 response ----------
[{"runner_module": "local_runner", "uid": "runner_type:local-shell-cmd", "description": "A runner to execute local actions as a fixed user.", "enabled": true, "runner_parameters": {"sudo": {"default": false, "type": "boolean", "description": "The command will be executed with sudo."}, "timeout": {"default": 60, "type": "integer", "description": "Action timeout in seconds. Action will get killed if it doesn't finish in timeout seconds."}, "cmd": {"type": "string", "description": "Arbitrary Linux command to be executed on the host."}, "kwarg_op": {"default": "--", "type": "string", "description": "Operator to use in front of keyword args i.e. \"--\" or \"-\"."}, "env": {"type": "object", "description": "Environment variables which will be available to the command(e.g. key1=val1,key2=val2)"}, "cwd": {"type": "string", "description": "Working directory where the command will be executed in"}}, "id": "58c9663a49d4af4cbd56f847", "name": "local-shell-cmd"}]
# -------- end 140183979680080 response ------------

# -------- begin 140183979680976 request ----------
curl -X POST -H 'Connection: keep-alive' -H 'Accept-Encoding: gzip, deflate' -H 'Accept: */*' -H 'User-Agent: python-requests/2.11.1' -H 'content-type: application/json' -H 'X-Auth-Token: da5ecf3b0ab841008d663052fe95cddd' -H 'Content-Length: 69' --data-binary '{"action": "core.local", "user": null, "parameters": {"cmd": "date"}}' http://127.0.0.1:9101/v1/executions
# -------- begin 140183979680976 response ----------
{"status": "requested", "start_timestamp": "2017-03-31T08:21:18.828620Z", "log": [{"status": "requested", "timestamp": "2017-03-31T08:21:18.843043Z"}], "parameters": {"cmd": "date"}, "runner": {"runner_module": "local_runner", "uid": "runner_type:local-shell-cmd", "description": "A runner to execute local actions as a fixed user.", "enabled": true, "runner_parameters": {"sudo": {"default": false, "type": "boolean", "description": "The command will be executed with sudo."}, "timeout": {"default": 60, "type": "integer", "description": "Action timeout in seconds. Action will get killed if it doesn't finish in timeout seconds."}, "cmd": {"type": "string", "description": "Arbitrary Linux command to be executed on the host."}, "kwarg_op": {"default": "--", "type": "string", "description": "Operator to use in front of keyword args i.e. \"--\" or \"-\"."}, "env": {"type": "object", "description": "Environment variables which will be available to the command(e.g. key1=val1,key2=val2)"}, "cwd": {"type": "string", "description": "Working directory where the command will be executed in"}}, "id": "58c9663a49d4af4cbd56f847", "name": "local-shell-cmd"}, "web_url": "https://st2expect/#/history/58de117e49d4af083399181c/general", "context": {"user": "st2admin"}, "action": {"description": "Action that executes an arbitrary Linux command on the localhost.", "runner_type": "local-shell-cmd", "tags": [], "enabled": true, "pack": "core", "entry_point": "", "notify": {}, "uid": "action:core:local", "parameters": {"cmd": {"required": true, "type": "string", "description": "Arbitrary Linux command to be executed on the local host."}, "sudo": {"immutable": true}}, "ref": "core.local", "id": "58c9663a49d4af4cbd56f84d", "name": "local"}, "liveaction": {"runner_info": {}, "parameters": {"cmd": "date"}, "action_is_workflow": false, "callback": {}, "action": "core.local", "id": "58de117e49d4af083399181b"}, "id": "58de117e49d4af083399181c"}
# -------- end 140183979680976 response ------------

# -------- begin 140183979680976 request ----------
curl -X GET -H 'Connection: keep-alive' -H 'Accept-Encoding: gzip, deflate' -H 'Accept: */*' -H 'User-Agent: python-requests/2.11.1' -H 'X-Auth-Token: da5ecf3b0ab841008d663052fe95cddd' http://127.0.0.1:9101/v1/executions/58de117e49d4af083399181c
# -------- begin 140183979680976 response ----------
{"status": "succeeded", "start_timestamp": "2017-03-31T08:21:18.828620Z", "log": [{"status": "requested", "timestamp": "2017-03-31T08:21:18.843000Z"}, {"status": "scheduled", "timestamp": "2017-03-31T08:21:18.943000Z"}, {"status": "running", "timestamp": "2017-03-31T08:21:19.041000Z"}, {"status": "succeeded", "timestamp": "2017-03-31T08:21:19.242000Z"}], "parameters": {"cmd": "date"}, "runner": {"runner_module": "local_runner", "uid": "runner_type:local-shell-cmd", "enabled": true, "name": "local-shell-cmd", "runner_parameters": {"sudo": {"default": false, "type": "boolean", "description": "The command will be executed with sudo."}, "timeout": {"default": 60, "type": "integer", "description": "Action timeout in seconds. Action will get killed if it doesn't finish in timeout seconds."}, "cmd": {"type": "string", "description": "Arbitrary Linux command to be executed on the host."}, "kwarg_op": {"default": "--", "type": "string", "description": "Operator to use in front of keyword args i.e. \"--\" or \"-\"."}, "env": {"type": "object", "description": "Environment variables which will be available to the command(e.g. key1=val1,key2=val2)"}, "cwd": {"type": "string", "description": "Working directory where the command will be executed in"}}, "id": "58c9663a49d4af4cbd56f847", "description": "A runner to execute local actions as a fixed user."}, "elapsed_seconds": 0.378813, "web_url": "https://st2expect/#/history/58de117e49d4af083399181c/general", "result": {"failed": false, "stderr": "", "return_code": 0, "succeeded": true, "stdout": "Fri Mar 31 08:21:19 UTC 2017"}, "context": {"user": "st2admin"}, "action": {"runner_type": "local-shell-cmd", "name": "local", "parameters": {"cmd": {"required": true, "type": "string", "description": "Arbitrary Linux command to be executed on the local host."}, "sudo": {"immutable": true}}, "tags": [], "enabled": true, "entry_point": "", "notify": {}, "uid": "action:core:local", "pack": "core", "ref": "core.local", "id": "58c9663a49d4af4cbd56f84d", "description": "Action that executes an arbitrary Linux command on the localhost."}, "liveaction": {"runner_info": {"hostname": "st2expect", "pid": 1657}, "parameters": {"cmd": "date"}, "action_is_workflow": false, "callback": {}, "action": "core.local", "id": "58de117e49d4af083399181b"}, "id": "58de117e49d4af083399181c", "end_timestamp": "2017-03-31T08:21:19.207433Z"}
# -------- end 140183979680976 response -----------

id: 58de117e49d4af083399181c
status: succeeded
parameters:
cmd: date
result:
failed: false
return_code: 0
stderr: ''
stdout: Fri Mar 31 08:21:19 UTC 2017
succeeded: true

In addition to the "usual" output that shows the result of the execution, the ``--debug`` flag also
shows all the API calls made during the course of the entire interaction, in the form of ``curl``
commands.

That output shows the API calls made when executing the command from the |st2| host. If you are
accessing the API from a remote system, it will be proxied through nginx, using the ``/api`` URI.
So remote calls will take this form:
You can invoke an action synchronously using:

.. sourcecode:: bash

curl -X POST https://[ST2_IP]/api/v1/executions -H 'Connection: keep-alive' -H 'Accept-Encoding: gzip, deflate' -H 'Accept: */*' -H 'User-Agent: python-requests/2.11.1' -H 'content-type: application/json' -H 'X-Auth-Token: matoken' -H 'Content-Length: 69' --data-binary '{"action": "core.local", "user": null, "parameters": {"cmd": "date"}}'
curl -X POST https://[ST2_IP]/api/v1/executions \
-H "X-Auth-Token: matoken" \
-H "Content-Type: application/json" \
--data '{"action": "core.local", "parameters": {"cmd": "date"}}'