-
Notifications
You must be signed in to change notification settings - Fork 370
Source Insert Functions doc added to Functions [DOC-1109] #7686
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
sade-wusi
wants to merge
22
commits into
develop
Choose a base branch
from
source-insert-function-pubbeta-release
base: develop
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+121
−0
Open
Changes from 6 commits
Commits
Show all changes
22 commits
Select commit
Hold shift + click to select a range
82edc41
Source Insert Functions doc added to Functions
sade-wusi 3a1c850
added whitespace for [netlify-build]
sade-wusi e0f2a20
Update source-insert-functions [netlify-build]
sade-wusi 44c6a6f
Rename source-insert-functions to source-insert-functions.md
sade-wusi ea65258
Update source-insert-functions.md [netlify-build]
sade-wusi 92b80f0
formatting
sade-wusi 90eb4b8
Update src/connections/functions/source-insert-functions.md
sade-wusi d10eb7e
Update src/connections/functions/source-insert-functions.md
sade-wusi 3e25204
Update src/connections/functions/source-insert-functions.md
sade-wusi 629b0ee
Update src/connections/functions/source-insert-functions.md
sade-wusi 4b86de0
Update src/connections/functions/source-insert-functions.md
sade-wusi a39114d
Update src/connections/functions/source-insert-functions.md
sade-wusi 628241a
Update src/connections/functions/source-insert-functions.md
sade-wusi 35332e0
Update src/connections/functions/source-insert-functions.md
sade-wusi 4a813e2
Update source-insert-functions to include review fixes.md
sade-wusi 6676e9f
Update src/connections/functions/source-insert-functions.md
sade-wusi d6c6cc3
Update src/connections/functions/source-insert-functions.md
sade-wusi 6d86db6
Update src/connections/functions/source-insert-functions.md
sade-wusi f4a974c
Added PM changes + settings + testing section
sade-wusi 52bae26
Update src/connections/functions/source-insert-functions.md
sade-wusi 3f29890
Update src/connections/functions/source-insert-functions.md
sade-wusi 426060c
Update src/connections/functions/source-insert-functions.md
sade-wusi File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
@@ -0,0 +1,107 @@ | ||||||
--- | ||||||
title: Source Insert Functions | ||||||
--- | ||||||
|
||||||
> info "Source Insert Function is in Public Beta" | ||||||
> Source Insert Function is in public beta, and Segment is actively working on this feature. Some functionality may change before it becomes generally available. | ||||||
|
||||||
## Create source insert functions | ||||||
sade-wusi marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
There are two ways you can access source insert functions from your Segment workspace: | ||||||
- From the [Connections catalog](#using-the-catalog). | ||||||
- From the [Sources tab](#using-the-sources-tab). | ||||||
|
||||||
### Using the catalog | ||||||
|
||||||
To create a source insert function from Segment’s catalog: | ||||||
|
||||||
1. Navigate to **Connections** > **Catalog** > **Functions** and click **New Function**. | ||||||
sade-wusi marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
2. In the Select Function Type screen, select **Source Insert** and click **Next: Build Function**. | ||||||
|
||||||
3. Write and test your function code. Enter a sample event and click **Run** to test the function. | ||||||
|
||||||
4. Click **Next: Configure & Create** to add a function name, description (optional), and function logo (optional). | ||||||
|
||||||
5. Click **Create Function** to save your insert function. The new source insert function will be displayed in the Functions tab. | ||||||
sade-wusi marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
#### Coding the source insert function | ||||||
|
||||||
Insert functions can define handlers for the following message types: | ||||||
- `onIdentity` | ||||||
- `onTrack` | ||||||
- `onPage` | ||||||
- `onScreen` | ||||||
- `onGroup` | ||||||
- `onAlias` | ||||||
- `onDelete` | ||||||
|
||||||
The default source code template includes handlers for all event types. Just implement the types you need and skip the ones you don’t. For event types you want to send through, return the event in the respective event handler. | ||||||
sade-wusi marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
### Using the Sources tab | ||||||
sade-wusi marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
You can also create a source insert function from Sources. | ||||||
sade-wusi marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
1. Navigate to **Connections** > **Sources**. | ||||||
2. Select your source and go to the **Functions** tab. | ||||||
3. Click **Create insert function** to create your insert function from scratch. | ||||||
sade-wusi marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
### Connecting a source insert function to a source | ||||||
sade-wusi marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
For data to flow downstream after transformation, you need to connect your source insert function to a source. To do this, follow the steps below: | ||||||
sade-wusi marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
1. Select the insert function you want to connect to the source. You can edit, delete, and connect the insert function in the side pane. | ||||||
sade-wusi marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
2. Click *Connect a source**. | ||||||
3. Select the source you want to connect from the dropdown and click **Connect to Source** to connect. | ||||||
sade-wusi marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
### Errors and error handling | ||||||
|
||||||
Segment considers a function’s execution successful if it completes without error. You can `throw` an error to create a failure on purpose. Use these errors to validate event data before processing to make sure the function works as expected. | ||||||
|
||||||
You can `throw` the following pre-defined error types to indicate that the function ran successfully, but the data was not deliverable: | ||||||
|
||||||
- `EventNotSupported` | ||||||
- `InvalidEventPayload` | ||||||
- `ValidationError` | ||||||
- `RetryError` | ||||||
- `DropEvent` | ||||||
|
||||||
This is similar to Source, Destination, and Insert Functions. | ||||||
|
||||||
The errors listed are all permanent errors, except `RetryError`. When a `RetryError` is thrown, Segment will reattempt to run the function a set number of times before permanently erroring out. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
||||||
### Source insert functions FAQs | ||||||
sade-wusi marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
##### The Delivery Overview tab does not display any errors that occur within the source insert function. | ||||||
sade-wusi marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
If errors arise or events are dropped, they will appear as if the events never flowed from the source. | ||||||
sade-wusi marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
##### Can I test the function with different event types? | ||||||
|
||||||
You can test the function with different event types from the **Test** tab in the code editor, similar to other functions. | ||||||
|
||||||
##### Can I use source insert functions for all types of Source? | ||||||
sade-wusi marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
Source insert functions only work for event sources, not object sources or rETL. | ||||||
|
||||||
##### What is the expected latency for a source insert function? | ||||||
|
||||||
Typically, it takes a source insert function between 200 milliseconds to 5 seconds to complete, based on the complexity of the function. The default timeout is 5 seconds but can be increased to 60 seconds. | ||||||
sade-wusi marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
##### What is the runtime environment? | ||||||
|
||||||
Segment supports Node.js V18. The dependencies listed [here](https://segment.com/docs/connections/functions/insert-functions/#runtime-and-dependencies){:target=”_blank”} are installed in the function. The following dependencies are also installed: | ||||||
sade-wusi marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
- `@azure/identity` exposed as `azure.identity` | ||||||
- `@azure/event-hubs` exposed as `azure.eventHubs` | ||||||
- `@azure/synapse` exposed as `azure.synapse` | ||||||
- `jsftp` exposed as `jsftp` | ||||||
- `crypto-js` exposed as `cryptojslib.cryptojs` | ||||||
- `akeyless` exposed as `akeylessLabs.akeyless` | ||||||
- `akeyless-cloud-id` exposed as `akeylessLabs.akeylessCloudId` | ||||||
- `@onesignal/node-onesignal` exposed as `oneSignal.oneSignal` | ||||||
- `pg` exposed as `pg.pg` | ||||||
- `snowflake-sdk` exposed as `snowflakesdk.snowflakesdk` | ||||||
- `@clickhouse/client` exposed as `clickHouseClient.clickHouseClient` | ||||||
- `aws4` exposed as `aws4.aws4` | ||||||
- `@sentry/node` exposed as `sentrylib.sentry` | ||||||
|
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.