Skip to content

Commit 8c6f7f4

Browse files
forstisabellaforstisabella
authored
Merge pull request #6571 from segmentio/retl-docs-refresh
Reverse ETL 2.0 docs
2 parents b8eb32b + 73b9010 commit 8c6f7f4

File tree

13 files changed

+499
-387
lines changed

13 files changed

+499
-387
lines changed

‎src/_data/sidenav/main.yml

+7-1
Original file line numberDiff line numberDiff line change
@@ -184,8 +184,14 @@ sections:
184184
section:
185185
- path: /connections/reverse-etl
186186
title: Reverse ETL Overview
187+
- path: /connections/reverse-etl/setup
188+
title: Set Up Reverse ETL
189+
- path: /connections/reverse-etl/manage-retl
190+
title: Manage Reverse ETL Syncs
191+
- path: /connections/reverse-etl/system
192+
title: Reverse ETL System
187193
- path: /connections/reverse-etl/reverse-etl-catalog
188-
title: Reverse ETL Catalog
194+
title: Reverse ETL Destination Catalog
189195
- section_title: Reverse ETL Source Setup Guides
190196
slug: connections/reverse-etl/reverse-etl-source-setup-guides
191197
section:

‎src/connections/reverse-etl/faq.md

+25
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
---
2+
title: Reverse ETL FAQ
3+
beta: false
4+
---
5+
6+
Get answers to some frequently asked Reverse ETL questions.
7+
8+
## Why do my sync results show *No records extracted* when I select *Updated records* after I enable the mapping?
9+
It's expected that when you select **Updated records**, the records do not change after the first sync. During the first sync, the reverse ETL system calculates a snapshot of all the results and creates records in the `_segment_reverse_etl` schema. All the records are considered as *Added records* instead of *Updated records* at this time. The records can only meet the *Updated records* condition when the underlying values change after the first sync completes.
10+
11+
## Can I be notified when Reverse ETL syncs fail?
12+
Yes, you can sign up for Reverse ETL sync notifications.
13+
14+
To receive Reverse ETL sync notifications:
15+
1. Navigate to **Settings > User Preferences**.
16+
2. Select **Reverse ETL** in the **Activity Notifications** section.
17+
3. Enable the toggle for **Reverse ETL Sync Failed**.
18+
19+
In case of consecutive failures, Segment sends notifications for every sync failure. Segment doesn't send notifications for partial failures.
20+
21+
## Does Segment use Transport Layer Security (TLS) for the connection between Snowflake and Segment?
22+
Segment uses the [gosnowflake library](https://pkg.go.dev/github.com/snowflakedb/gosnowflake#pkg-variables){:target="_blank"} to connect with Snowflake, which internally uses TLS for the HTTP transport.
23+
24+
## Can I have multiple queries in the Query Builder?
25+
No. In Reverse ETL, Segment executes queries in a [common table expression](https://cloud.google.com/bigquery/docs/reference/standard-sql/query-syntax#with_clause){:target="_blank”}, which can only bind the results from **one single** subquery. If there are multiple semicolons `;` in the query, they'll be treated as several subqueries (even if the second part is only an inline comment) and cause syntax errors.

‎src/connections/reverse-etl/index.md

+74-373
Large diffs are not rendered by default.

‎src/connections/reverse-etl/manage-retl.md

+141
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,141 @@
1+
---
2+
title: Manage Reverse ETL Syncs
3+
beta: false
4+
---
5+
6+
View your sync history, gain insights into sync statuses, and restart or replay failed or partially successful syncs.
7+
8+
## Sync overview
9+
On the Reverse ETL sync overview tab for your destination, you can see information about your recent Reverse ETL syncs at a glance, search for recent syncs, and quickly access the mappings and models that power Reverse ETL.
10+
11+
![A screenshot of the sync overview page, which includes one failed sync and three successful syncs.](images/sync-overview.png)
12+
13+
You can view the following information about each sync:
14+
- **Latest sync**: The progress of your latest sync: syncs can either be **In progress**, **Successful**, or **Failed**. Also included is the timestamp of the sync start time.
15+
- **Mapping**: The named mapping that powered the sync and a hyperlink to the mapping's overview page.
16+
- **Model**: The name that you gave the SQL query used to withdraw information from your warehouse, with a hyperlink to the model overview page. Below the model name, you can see the warehouse source that Segment extracts information from.
17+
- **Action**: The Action that your destination uses to map information from your warehouse to your downstream destination.
18+
- **Mapping status**: The status of your mapping: either **Enabled** or **Disabled**.
19+
20+
You can also filter the sync overview table to return only the syncs that match your criteria.
21+
22+
You can filter for the following sync attributes:
23+
- **Sync status**: The status of your sync: In progress, Successful, Partially successful, or Failed.
24+
- **Start time**: Select a predefined time period, or create a custom date range.
25+
- **Model**: The model connected to your sync.
26+
- **Destination**: Select one or more of your connected destinations.
27+
- **Mapping status**: The status of your mapping: either **Enabled** or **Disabled**.
28+
29+
## Sync history
30+
Check the status of your data extractions and see details of your syncs. Click into failed records to view additional details on the error, sample payloads to help you debug the issue, and recommended actions.
31+
32+
To check the status of your extractions:
33+
1. Navigate to **Connections > Destinations** and select the **Reverse ETL** tab.
34+
2. Select the destination you want to view.
35+
3. Select the mapping you want to view.
36+
4. Click the sync you want to view to get details of the sync. You can view:
37+
* The status of the sync.
38+
* Details of how long it took for the sync to complete.
39+
* How many total records were extracted, as well as a breakdown of the number of records added, updated, and deleted.
40+
* The load results - how many successful records were synced as well as how many records were updated, deleted, or are new.
41+
5. If your sync failed, click the failed reason to get more details on the error and view sample payloads to help troubleshoot the issue.
42+
43+
> info "Segment automatically retries events that were extracted but failed to load"
44+
> Segment retries events for 14 days following a total or partial sync failure. Before loading the failed records on a subsequent sync, Segment checks for the latest changes in your data to ensure the data loaded into your warehouse isn't stale. If the error causing the load failure is coming from an upstream tool, you can fix the error in the upstream tool to ensure the record loads on the next sync.
45+
46+
## Reset syncs
47+
You can reset your syncs so that your data is synced from the beginning. This means that Segment resyncs your entire dataset for the model. During the next sync, all records extracted by the model are sent to your destination, not just the records that changed since the last sync.
48+
49+
To reset a sync:
50+
1. Select the three dots next to **Sync now**.
51+
2. Select **Reset sync**.
52+
3. Click **I understand what happens when I reset a sync state**.
53+
4. Click **Reset sync**.
54+
55+
## Replays
56+
You can choose to replay syncs. To replay a specific sync, contact [[email protected]](mailto:[email protected]). Keep in mind that triggering a replay resyncs all records for a given sync.
57+
58+
## Alerting
59+
You can opt in to receive email, Slack, and in-app alerts about Reverse ETL sync failures and partial successes.
60+
61+
To subscribe to alerts:
62+
1. Navigate to **Settings > User Preferences**.
63+
2. Select **Reverse ETL** in the **Activity Notifications** section.
64+
3. Click the Reverse ETL sync status that you'd like to receive notifications for. You can select one or more of the following sync statuses:
65+
- **Reverse ETL sync failed**: Receive a notification when your Reverse ETL sync fails.
66+
- **Reverse ETL sync partial success**: Receive a notification when your Reverse ETL sync is partially successful.
67+
4. Select one or more of the following alert options:
68+
- **Enable email notifications**: Enter an email address or alias that should receive alerts.
69+
- **Enable Slack notifications**: Enter a webhook URL and Slack channel name.
70+
- **Enable in-app notifications**: Select this option to see an in-app notification.
71+
5. Click **Create alert**.
72+
73+
> success ""
74+
> If you opted to receive notifications by email, you can click **View active email addresses** to see the email addresses that are currently signed up to receive notifications.
75+
76+
## Supported object and arrays
77+
78+
When you set up destination actions in Reverse ETL, depending on the destination, some [mapping fields](/docs/connections/reverse-etl/setup/#step-4-create-mappings) may require data as an [object](/docs/connections/reverse-etl/manage-retl/#object-mapping) or [array](/docs/connections/reverse-etl/manage-retl/#array-mapping).
79+
80+
### Object mapping
81+
You can send data to a mapping field that requires object data. An example of object mapping is an `Order completed` model with a `Products` column that’s in object format.
82+
83+
Example:
84+
85+
```json
86+
{
87+
"product": {
88+
"id": 0001,
89+
"color": "pink",
90+
"name": "tshirt",
91+
"revenue": 20,
92+
"inventory": 500
93+
}
94+
}
95+
```
96+
97+
To send data to a mapping field that requires object data, you can choose between these two options:
98+
99+
Option | Details
100+
------ | --------
101+
Customize object | This enables you to manually set up the mapping fields with any data from the model. If the model contains some object data, you can select properties within the object to set up the mappings as well.
102+
Select object | This enables you to send all nested properties within an object. The model needs to provide data in the format of the object.
103+
104+
> success ""
105+
> Certain object mapping fields have a fixed list of properties they can accept. If the names of the nested properties in your object don't match with the destination properties, the data won't send. Segment recommends you to use **Customize Object** to ensure your mapping is successful.
106+
107+
108+
### Array mapping
109+
To send data to a mapping field that requires array data, the model must provide data in the format of an array of objects. An example is an `Order completed` model with a `Product purchased` column that’s in an array format.
110+
111+
Example:
112+
113+
```json
114+
[
115+
{
116+
"currency": "USD",
117+
"price": 40,
118+
"productName": "jacket",
119+
"purchaseTime": "2021-12-17 23:43:47.102",
120+
"quantity": 1
121+
},
122+
{
123+
"currency": "USD",
124+
"price": 5,
125+
"productName": "socks",
126+
"quantity": 2
127+
}
128+
]
129+
```
130+
131+
To send data to a mapping field that requires array data, you can choose between these two options:
132+
133+
Option | Details
134+
------ | --------
135+
Customize array | This enables you to select the specific nested properties to send to the destination.
136+
Select array | This enables you to send all nested properties within the array.
137+
138+
> success ""
139+
> Certain array mapping fields have a fixed list of properties they can accept. If the names of the nested properties in your array don't match the destination properties, the data won't send. Segment recommends you to use the **Customize array** option to ensure your mapping is successful.
140+
141+
Objects in an array don't need to have the same properties. If a user selects a missing property in the input object for a mapping field, the output object will miss the property.

‎src/connections/reverse-etl/reverse-etl-catalog.md

+30-1
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,14 @@
11
---
22
title: Reverse ETL Catalog
3+
beta: false
34
---
45

5-
These destinations support [Reverse ETL](/docs/connections/reverse-etl/). If you don’t see your destination listed in the Reverse ETL catalog, use the [Segment Connections destination](/docs/connections/destinations/catalog/actions-segment/) to send data from your Reverse ETL warehouse to other destinations listed in the [catalog](/docs/connections/destinations/catalog/).
6+
Reverse ETL supports the entire Segment destination catalog - 30+ Actions destinations are natively supported and all other destinations are supported through the [Segment Connections](#segment-connections-destination) destination.
7+
8+
> success ""
9+
> Twilio Engage Premier Subscriptions users can use the [Segment Profiles](/docs/connections/destinations/catalog/actions-segment-profiles/) destination to enrich their warehouse data.
10+
11+
The following destinations natively support [Reverse ETL](/docs/connections/reverse-etl/). If you don’t see your destination listed in the Reverse ETL catalog, use the [Segment Connections destination](/docs/connections/destinations/catalog/actions-segment/) to send data from your Reverse ETL warehouse to other destinations listed in the [catalog](/docs/connections/destinations/catalog/).
612

713
<div class="destinations-catalog">
814
<div class="flex flex--wrap waffle waffle--xlarge">
@@ -37,3 +43,26 @@ These destinations support [Reverse ETL](/docs/connections/reverse-etl/). If you
3743
</div>
3844
</div>
3945

46+
## Segment Connections destination
47+
If you don’t see your destination listed in the Reverse ETL catalog, use the [Segment Connections destination](/docs/connections/destinations/catalog/actions-segment/) to send data from your Reverse ETL warehouse to other destinations listed in the [catalog](/docs/connections/destinations/catalog/).
48+
49+
The Segment Connections destination enables you to mold data extracted from your warehouse in [Segment Spec](/docs/connections/spec/) API calls that are then processed by [Segment’s HTTP Tracking API](/docs/connections/sources/catalog/libraries/server/http-api/). The requests hit Segment’s servers, and then Segment routes your data to any destination you want. Get started with the [Segment Connections destination](/docs/connections/destinations/catalog/actions-segment/).
50+
51+
> warning ""
52+
> The Segment Connections destination sends data to Segment’s Tracking API, which has cost implications. New users count as new MTUs and each call counts as an API call. For information on how Segment calculates MTUs and API calls, please see [MTUs, Throughput and Billing](/docs/guides/usage-and-billing/mtus-and-throughput/).
53+
54+
## Send data to Engage with Segment Profiles
55+
Engage Premier Subscriptions users can use Reverse ETL to sync subscription data from warehouses to destinations.
56+
57+
To get started with using Reverse ETL for subscriptions:
58+
1. Navigate to **Engage > Audiences** and select the **Profile explorer** tab.
59+
2. Click **Manage subscription statuses** and select **Update subscription statuses**.
60+
3. Select **Sync with RETL** as the method to update your subscription statuses.
61+
4. Click **Configure**.
62+
5. In the Reverse ETL catalog, select the Reverse ETL source you want to use.
63+
6. Set up the source. Refer to the [add a source](/docs/connections/reverse-etl/setup/#step-1-add-a-source) section for more details on how to set up the source.
64+
7. Add the Segment Profiles destination as your Reverse ETL destination. Refer to [add a destination](/docs/connections/reverse-etl/setup/#step-3-add-a-destination) for more details on how to set up the destination.
65+
8. Once your destination is set, go to the **Mappings** tab of your destination and click **Add Mapping**.
66+
9. Select the model you want to use and then select **Send Subscriptions**.
67+
10. Click **Create Mapping**.
68+
11. Follow the steps in the [Create Mappings](/docs/connections/reverse-etl/setup/#step-4-create-mappings) section to set your mappings.

‎src/connections/reverse-etl/reverse-etl-source-setup-guides/azure-setup.md

+2-2
Original file line numberDiff line numberDiff line change
@@ -61,7 +61,7 @@ To set up Azure as your Reverse ETL source:
6161
5. Navigate to **Connections > Sources** and select the **Reverse ETL** tab.
6262
6. Click **+ Add Reverse ETL source**.
6363
7. Select **Azure** and click **Add Source**.
64-
8. Enter the configuration settings for you Azure source based on the information from Step 3.
64+
8. Enter the configuration settings for your Azure source based on the information from Step 3.
6565
* Hostname:
6666
* Use `xxxxxxx.sql.azuresynapse.net` if you’re connecting to a dedicated SQL pool in Synapse workspace.
6767
* Use `xxxxxxx.database.windows.net` if you’re connecting to a dedicated SQL pool (formerly SQL DW)
@@ -72,5 +72,5 @@ To set up Azure as your Reverse ETL source:
7272
9. Click **Test Connection** to see if the connection works. If the connection fails, make sure you have the right permissions and credentials, then try again.
7373
10. Click **Add source** if the test connection is successful.
7474
75-
After you've successfully added your Azure source, [add a model](/docs/connections/reverse-etl/#step-2-add-a-model) and follow the rest of the steps in the Reverse ETL setup guide.
75+
After you've successfully added your Azure source, [add a model](/docs/connections/reverse-etl/setup/#step-2-add-a-model) and follow the rest of the steps in the Reverse ETL setup guide.
7676

‎src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md

+6-1
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,10 @@ To set up the Segment BigQuery connector:
2828
20. Click **Test Connection** to test to see if the connection works. If the connection fails, make sure you have the right permissions and credentials and try again.
2929
6. Click **Add source** if the test connection is successful.
3030

31-
After you've added BigQuery as a source, you can [add a model](/docs/connections/reverse-etl#step-2-add-a-model).
31+
After you've added BigQuery as a source, you can [add a model](/docs/connections/reverse-etl/setup/#step-2-add-a-model).
32+
33+
> info "BigQuery Reverse ETL sources support Segment's dbt extension"
34+
> If you have an existing dbt account with a Git repository, you can use [Segment's dbt extension](/docs/segment-app/extensions/dbt/) to centralize model management and versioning, reduce redundancies, and run CI checks to prevent breaking changes.
3235
3336
## Constructing your own role or policy
3437
When you construct your own role or policy, Segment needs the following permissions:
@@ -48,3 +51,5 @@ Permission | Details
4851
`bigquery.jobs.create` | This allows Segment to execute queries on any datasets or tables your model query references, and also allows Segment to manage tables used for tracking.
4952

5053
The `bigquery.datasets.*` permissions can be scoped only to the `__segment_reverse_etl` dataset.
54+
55+
After you've successfully added your BigQuery source, [add a model](/docs/connections/reverse-etl/setup/#step-2-add-a-model) and follow the rest of the steps in the Reverse ETL setup guide.

0 commit comments

Comments
 (0)