Skip to content

Commit 197838b

Browse files
fivetran-catfritzfivetran-catfritz
authored
Bug/renamed columns option3 (#48)
* bug/renamed-columns * add macro file * add casting * update yml * add casting for all * macro updates * bug/renamed-columns-dynamic * adjustments * add consistency tests * update yml * bug/renamed-columns-option3 * rename variables for readability * adjust casts * update yml * update version & changelog * update variable for readbility * small adjustment * revise consistency test * update changelog * update test * update changelog * update test & macro * update changelog * make breaking * regen docs * update packages * update test * update comments
1 parent b41bbcc commit 197838b

35 files changed

+542
-382
lines changed

‎CHANGELOG.md

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,19 @@
1+
# dbt_salesforce_source v1.1.0
2+
[PR #48](https://github.com/fivetran/dbt_salesforce_source/pull/48) includes the following updates:
3+
4+
## 🚨 Breaking Change 🚨
5+
- Added logic to support user-specified scenarios where the Fivetran Salesforce connector syncs column names using the original Salesforce API naming convention. For example, while Fivetran typically provides the column as `created_date`, some users might choose to receive it as `CreatedDate` according to the API naming. This update ensures the package is automatically compatible with both naming conventions.
6+
- Specifically, the package now performs a COALESCE, preferring the original Salesforce API naming. If the original naming is not present, the Fivetran version is used instead.
7+
- Renamed columns are now explicitly cast to prevent conflicts during the COALESCE.
8+
- ❗This change is considered breaking since the resulting column types may differ from prior versions of this package.
9+
10+
## Under the Hood
11+
- Added the following macros to support the mentioned bug fix logic:
12+
- `add_renamed_columns`: Determines the original names for each column and adds them to the list generated within each `get_*_columns` macro. By default, this macro processes column names by removing underscores and capitalizing each part that follows an underscore. This ensures all necessary columns are available for use in the `coalesce_rename` macro. Additionally, this macro tags each column with its renamed version to maintain tracking.
13+
- `column_list_to_dict`: Converts the list of dictionaries generated by the `get_*_columns` macros into a dictionary of dictionaries for use in the `coalesce_rename` macro. This conversion is necessary so that each column dictionary entry can be accessed by a key, rather than iterating through a list.
14+
- `coalesce_rename`: Utilizes the dictionary generated by `column_list_to_dict` to coalesce a column with its renamed counterpart, producing the final column. This macro also allows for the passing of a custom renamed spelling, datatype, and alias as arguments to override default values.
15+
- Added validation test to ensure the final column names generated before and after this update remain the same.
16+
117
# dbt_salesforce_source v1.0.1
218

319
[PR #44](https://github.com/fivetran/dbt_salesforce_source/pull/44) includes the following updates:

‎README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,7 @@ If you are **not** using the [Salesforce transformation package](https://github.
5151
```yaml
5252
packages:
5353
- package: fivetran/salesforce_source
54-
version: [">=1.0.0", "<1.1.0"] # we recommend using ranges to capture non-breaking changes automatically
54+
version: [">=1.1.0", "<1.2.0"] # we recommend using ranges to capture non-breaking changes automatically
5555
```
5656
## Step 3: Configure Your Variables
5757
### Database and Schema Variables (Using the standard Salesforce schema only)

‎dbt_project.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
config-version: 2
22
name: 'salesforce_source'
3-
version: '1.0.1'
3+
version: '1.1.0'
44
require-dbt-version: [">=1.3.0", "<2.0.0"]
55
models:
66
salesforce_source:

‎docs/catalog.json

Lines changed: 1 addition & 1 deletion
Large diffs are not rendered by default.

‎docs/index.html

Lines changed: 10 additions & 37 deletions
Large diffs are not rendered by default.

‎docs/manifest.json

Lines changed: 1 addition & 1 deletion
Large diffs are not rendered by default.

‎docs/run_results.json

Lines changed: 1 addition & 1 deletion
Large diffs are not rendered by default.

‎integration_tests/.gitignore

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,5 @@
22
target/
33
dbt_modules/
44
logs/
5-
env/
5+
env/
6+
package-lock.yml

‎integration_tests/dbt_project.yml

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,13 @@
11
name: 'salesforce_source_integration_tests'
2-
version: '1.0.1'
2+
version: '1.1.0'
33

44
config-version: 2
55

66
profile: 'integration_tests'
77

8+
models:
9+
+schema: "salesforce_source_{{ var('directed_schema','dev') }}"
10+
811
vars:
912
salesforce_source:
1013
salesforce_schema: salesforce_source_integrations_tests_3

‎integration_tests/tests/consistency/consistency_all_tables_and_columns.sql

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
{{ config(
2+
tags="fivetran_validations",
3+
enabled=var('fivetran_validation_tests_enabled', false)
4+
) }}
5+
6+
/* This test is to make sure the final columns produced are the same between versions.
7+
Only one test is needed since it will fetch all tables and all columns in each schema.
8+
!!! THIS TEST IS WRITTEN FOR BIGQUERY!!! */
9+
{% if target.type == 'bigquery' %}
10+
with prod as (
11+
select
12+
table_name,
13+
column_name,
14+
data_type
15+
from {{ target.schema }}_salesforce_source_prod.INFORMATION_SCHEMA.COLUMNS
16+
where table_name like 'stg_%'
17+
),
18+
19+
dev as (
20+
select
21+
table_name,
22+
column_name,
23+
data_type
24+
from {{ target.schema }}_salesforce_source_dev.INFORMATION_SCHEMA.COLUMNS
25+
where table_name like 'stg_%'
26+
),
27+
28+
prod_not_in_dev as (
29+
-- rows from prod not found in dev
30+
select * from prod
31+
except distinct
32+
select * from dev
33+
),
34+
35+
dev_not_in_prod as (
36+
-- rows from dev not found in prod
37+
select * from dev
38+
except distinct
39+
select * from prod
40+
),
41+
42+
final as (
43+
select
44+
*,
45+
'from prod' as source
46+
from prod_not_in_dev
47+
48+
union all -- union since we only care if rows are produced
49+
50+
select
51+
*,
52+
'from dev' as source
53+
from dev_not_in_prod
54+
)
55+
56+
select *
57+
from final
58+
59+
{% else %}
60+
{{ print('This is written to run on bigquery. If you need to run on another warehouse, add a version!') }}
61+
62+
{% endif %}

‎macros/add_renamed_columns.sql

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
{% macro add_renamed_columns(column_list) %}
2+
{# This macro determines the original names for each column and adds them to the list generated within each `get_*_columns` macro. By default, this macro processes column names by removing underscores and capitalizing each part that follows an underscore. This ensures all necessary columns are available for use in the `coalesce_rename` macro. Additionally, this macro tags each column with its renamed version to maintain tracking. #}
3+
4+
{%- set renamed_columns = [] %}
5+
6+
{%- for col in column_list %}
7+
8+
{%- set original_column_name = col.name %}
9+
10+
{%- if 'fivetran' not in original_column_name %}
11+
{# Use renamed_column_name value if it provided in the get_columns macro #}
12+
{%- set renamed_column_name = col.renamed_column_name | default(original_column_name.split('_') | map('capitalize') | join('')) %}
13+
14+
{# Add an entry to the list of renames to populate the filled columns if the rename is different #}
15+
{%- do renamed_columns.append({"name": renamed_column_name, "datatype": col.datatype, "is_rename": true}) if renamed_column_name|lower != original_column_name|lower %}
16+
17+
{# Update the original column with the renamed column name for use later. #}
18+
{%- set col = col.update({ "renamed_column_name": renamed_column_name, "is_rename": false}) %}
19+
{%- endif %}
20+
{%- endfor %}
21+
22+
{%- do column_list.extend(renamed_columns) %}
23+
24+
{% endmacro %}

‎macros/coalesce_rename.sql

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
{% macro coalesce_rename(
2+
column_key,
3+
column_dict,
4+
original_column_name=column_dict[column_key]["name"],
5+
datatype=column_dict[column_key]["datatype"],
6+
alias=column_dict[column_key]["alias"] | default(original_column_name),
7+
renamed_column_name=column_dict[column_key]["renamed_column_name"]
8+
) %}
9+
10+
{# This macro accomodates Fivetran connectors that keep the original salesforce field naming conventions without underscores #}
11+
{# Utilizes the dictionary generated by `column_list_to_dict` to coalesce a column with its renamed counterpart, producing the final column. This macro also allows for the passing of a custom renamed spelling, datatype, and alias as arguments to override default values. #}
12+
{%- if original_column_name|lower == renamed_column_name|lower %}
13+
cast({{ renamed_column_name }} as {{ datatype }}) as {{ alias }}
14+
15+
{%- else %}
16+
coalesce(cast({{ renamed_column_name }} as {{ datatype }}),
17+
cast({{ original_column_name }} as {{ datatype }}))
18+
as {{ alias }}
19+
20+
{%- endif %}
21+
{%- endmacro %}

‎macros/column_list_to_dict.sql

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
{% macro column_list_to_dict(column_list) %}
2+
{# This macro converts the list of dictionaries generated by the `get_*_columns` macros into a dictionary of dictionaries for use in the `coalesce_rename` macro. This conversion is necessary so that each column dictionary entry can be accessed by a key, rather than iterating through a list. #}
3+
{%- set column_dict = {} -%}
4+
{%- for col in column_list -%}
5+
{%- do column_dict.update({col.name: col}) if not col.is_rename -%}
6+
{%- endfor -%}
7+
{{ return(column_dict) }}
8+
9+
{% endmacro %}

‎macros/get_account_columns.sql

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
{% set columns = [
44

55
{"name": "_fivetran_synced", "datatype": dbt.type_timestamp()},
6-
{"name": "_fivetran_active", "datatype": "boolean"},
6+
{"name": "_fivetran_active", "datatype": dbt.type_boolean()},
77
{"name": "account_number", "datatype": dbt.type_string()},
88
{"name": "account_source", "datatype": dbt.type_string()},
99
{"name": "annual_revenue", "datatype": dbt.type_float()},
@@ -16,7 +16,7 @@
1616
{"name": "description", "datatype": dbt.type_string()},
1717
{"name": "id", "datatype": dbt.type_string()},
1818
{"name": "industry", "datatype": dbt.type_string()},
19-
{"name": "is_deleted", "datatype": "boolean"},
19+
{"name": "is_deleted", "datatype": dbt.type_boolean()},
2020
{"name": "last_activity_date", "datatype": dbt.type_timestamp()},
2121
{"name": "last_referenced_date", "datatype": dbt.type_timestamp()},
2222
{"name": "last_viewed_date", "datatype": dbt.type_timestamp()},
@@ -39,6 +39,8 @@
3939
{"name": "website", "datatype": dbt.type_string()}
4040
] %}
4141

42+
{{ salesforce_source.add_renamed_columns(columns) }}
43+
4244
{{ fivetran_utils.add_pass_through_columns(columns, var('salesforce__account_pass_through_columns')) }}
4345

4446
{{ return(columns) }}

‎macros/get_contact_columns.sql

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
{% set columns = [
44
{"name": "_fivetran_synced", "datatype": dbt.type_timestamp()},
5-
{"name": "_fivetran_active", "datatype": "boolean"},
5+
{"name": "_fivetran_active", "datatype": dbt.type_boolean()},
66
{"name": "account_id", "datatype": dbt.type_string()},
77
{"name": "department", "datatype": dbt.type_string()},
88
{"name": "description", "datatype": dbt.type_string()},
@@ -35,6 +35,8 @@
3535
{"name": "title", "datatype": dbt.type_string()},
3636
] %}
3737

38+
{{ salesforce_source.add_renamed_columns(columns) }}
39+
3840
{{ fivetran_utils.add_pass_through_columns(columns, var('salesforce__contact_pass_through_columns')) }}
3941

4042
{{ return(columns) }}

‎macros/get_event_columns.sql

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
{% set columns = [
44
{"name": "_fivetran_synced", "datatype": dbt.type_timestamp()},
5-
{"name": "_fivetran_active", "datatype": "boolean"},
5+
{"name": "_fivetran_active", "datatype": dbt.type_boolean()},
66
{"name": "account_id", "datatype": dbt.type_string()},
77
{"name": "activity_date", "datatype": dbt.type_timestamp()},
88
{"name": "activity_date_time", "datatype": dbt.type_timestamp()},
@@ -32,6 +32,8 @@
3232
{"name": "who_id", "datatype": dbt.type_string()}
3333
] %}
3434

35+
{{ salesforce_source.add_renamed_columns(columns) }}
36+
3537
{{ fivetran_utils.add_pass_through_columns(columns, var('salesforce__event_pass_through_columns')) }}
3638

3739
{{ return(columns) }}

‎macros/get_lead_columns.sql

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
{% set columns = [
44
{"name": "_fivetran_synced", "datatype": dbt.type_timestamp()},
5-
{"name": "_fivetran_active", "datatype": "boolean"},
5+
{"name": "_fivetran_active", "datatype": dbt.type_boolean()},
66
{"name": "annual_revenue", "datatype": dbt.type_float()},
77
{"name": "city", "datatype": dbt.type_string()},
88
{"name": "company", "datatype": dbt.type_string()},
@@ -48,6 +48,8 @@
4848
{"name": "website", "datatype": dbt.type_string()},
4949
] %}
5050

51+
{{ salesforce_source.add_renamed_columns(columns) }}
52+
5153
{{ fivetran_utils.add_pass_through_columns(columns, var('salesforce__lead_pass_through_columns')) }}
5254

5355
{{ return(columns) }}

‎macros/get_opportunity_columns.sql

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,9 +34,11 @@
3434
{"name": "record_type_id", "datatype": dbt.type_string()},
3535
{"name": "stage_name", "datatype": dbt.type_string()},
3636
{"name": "synced_quote_id", "datatype": dbt.type_string()},
37-
{"name": "type", "datatype": dbt.type_string()},
37+
{"name": "type", "datatype": dbt.type_string()}
3838
] %}
3939

40+
{{ salesforce_source.add_renamed_columns(columns) }}
41+
4042
{{ fivetran_utils.add_pass_through_columns(columns, var('salesforce__opportunity_pass_through_columns')) }}
4143

4244
{{ return(columns) }}

‎macros/get_opportunity_line_item_columns.sql

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
{% set columns = [
44
{"name": "_fivetran_synced", "datatype": dbt.type_timestamp()},
5-
{"name": "_fivetran_active", "datatype": "boolean"},
5+
{"name": "_fivetran_active", "datatype": dbt.type_boolean()},
66
{"name": "created_by_id", "datatype": dbt.type_string()},
77
{"name": "created_date", "datatype": dbt.type_timestamp()},
88
{"name": "description", "datatype": dbt.type_string()},
@@ -29,6 +29,8 @@
2929
{"name": "unit_price", "datatype": dbt.type_float()}
3030
] %}
3131

32+
{{ salesforce_source.add_renamed_columns(columns) }}
33+
3234
{{ fivetran_utils.add_pass_through_columns(columns, var('salesforce__opportunity_line_item_pass_through_columns')) }}
3335

3436
{{ return(columns) }}

‎macros/get_order_columns.sql

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
{% set columns = [
44
{"name": "_fivetran_synced", "datatype": dbt.type_timestamp()},
5-
{"name": "_fivetran_active", "datatype": "boolean"},
5+
{"name": "_fivetran_active", "datatype": dbt.type_boolean()},
66
{"name": "account_id", "datatype": dbt.type_string()},
77
{"name": "activated_by_id", "datatype": dbt.type_string()},
88
{"name": "activated_date", "datatype": dbt.type_timestamp()},
@@ -41,6 +41,8 @@
4141
{"name": "type", "datatype": dbt.type_string()},
4242
] %}
4343

44+
{{ salesforce_source.add_renamed_columns(columns) }}
45+
4446
{{ fivetran_utils.add_pass_through_columns(columns, var('salesforce__order_pass_through_columns')) }}
4547

4648
{{ return(columns) }}

‎macros/get_product_2_columns.sql

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
{% set columns = [
44
{"name": "_fivetran_synced", "datatype": dbt.type_timestamp()},
5-
{"name": "_fivetran_active", "datatype": "boolean"},
5+
{"name": "_fivetran_active", "datatype": dbt.type_boolean()},
66
{"name": "created_by_id", "datatype": dbt.type_string()},
77
{"name": "created_date", "datatype": dbt.type_timestamp()},
88
{"name": "description", "datatype": dbt.type_string()},
@@ -29,6 +29,8 @@
2929
{"name": "revenue_schedule_type", "datatype": dbt.type_string()},
3030
] %}
3131

32+
{{ salesforce_source.add_renamed_columns(columns) }}
33+
3234
{{ fivetran_utils.add_pass_through_columns(columns, var('salesforce__product_2_pass_through_columns')) }}
3335

3436
{{ return(columns) }}

‎macros/get_task_columns.sql

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
{% set columns = [
44
{"name": "_fivetran_synced", "datatype": dbt.type_timestamp()},
5-
{"name": "_fivetran_active", "datatype": "boolean"},
5+
{"name": "_fivetran_active", "datatype": dbt.type_boolean()},
66
{"name": "account_id", "datatype": dbt.type_string()},
77
{"name": "activity_date", "datatype": dbt.type_timestamp()},
88
{"name": "call_disposition", "datatype": dbt.type_string()},
@@ -33,6 +33,8 @@
3333
{"name": "who_id", "datatype": dbt.type_string()}
3434
] %}
3535

36+
{{ salesforce_source.add_renamed_columns(columns) }}
37+
3638
{{ fivetran_utils.add_pass_through_columns(columns, var('salesforce__task_pass_through_columns')) }}
3739

3840
{{ return(columns) }}

‎macros/get_user_columns.sql

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
{% set columns = [
44
{"name": "_fivetran_deleted", "datatype": "boolean"},
5-
{"name": "_fivetran_active", "datatype": "boolean"},
5+
{"name": "_fivetran_active", "datatype": dbt.type_boolean()},
66
{"name": "_fivetran_synced", "datatype": dbt.type_timestamp()},
77
{"name": "account_id", "datatype": dbt.type_string()},
88
{"name": "alias", "datatype": dbt.type_string()},
@@ -34,6 +34,8 @@
3434
{"name": "username", "datatype": dbt.type_string()},
3535
] %}
3636

37+
{{ salesforce_source.add_renamed_columns(columns) }}
38+
3739
{{ fivetran_utils.add_pass_through_columns(columns, var('salesforce__user_pass_through_columns')) }}
3840

3941
{{ return(columns) }}

‎macros/get_user_role_columns.sql

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
{% set columns = [
44
{"name": "_fivetran_deleted", "datatype": "boolean"},
5-
{"name": "_fivetran_active", "datatype": "boolean"},
5+
{"name": "_fivetran_active", "datatype": dbt.type_boolean()},
66
{"name": "_fivetran_synced", "datatype": dbt.type_timestamp()},
77
{"name": "developer_name", "datatype": dbt.type_string()},
88
{"name": "id", "datatype": dbt.type_string()},
@@ -12,6 +12,8 @@
1212
{"name": "rollup_description", "datatype": dbt.type_string()}
1313
] %}
1414

15+
{{ salesforce_source.add_renamed_columns(columns) }}
16+
1517
{{ fivetran_utils.add_pass_through_columns(columns, var('salesforce__user_role_pass_through_columns')) }}
1618

1719
{{ return(columns) }}

0 commit comments

Comments
 (0)