Feature/historical schedules (#55)

* stage changes

* seed

* bk

* add seed back in

* add using_holidays

* doc updates

* fix vars

* fix vars

* Update vars


* update yml

* update changelog & regen docs

* update var names, readme, regen docs

* update run models

* update changelog


* Apply suggestions from code review

Co-authored-by: Joe Markiewicz <[email protected]>

* Apply suggestions from code review

Co-authored-by: Jamie Rodriguez <[email protected]>

---------

Co-authored-by: Jamie Rodriguez <[email protected]>
Co-authored-by: Joe Markiewicz <[email protected]>

Author: 3 people

.buildkite/scripts/run_models.sh

@@ -21,4 +21,6 @@ dbt run --target "$db" --full-refresh
 dbt test --target "$db"
 dbt run --vars '{using_schedules: false, using_domain_names: false, using_user_tags: false, using_ticket_form_history: false, using_organization_tags: false}' --target "$db" --full-refresh
 dbt test --target "$db"
+dbt run --vars '{using_schedule_histories: true, using_holidays: false}' --target "$db" --full-refresh
+dbt test --target "$db"
 dbt run-operation fivetran_utils.drop_schemas_automation --target "$db"

CHANGELOG.md

@@ -1,3 +1,18 @@
+# dbt_zendesk_source v0.13.0
+[PR #55](https://github.com/fivetran/dbt_zendesk_source/pull/55) includes the following updates:
+
+## Breaking Changes
+- Introduced the `stg_zendesk__audit_log` table for capturing schedule changes from Zendesk's audit log.
+  - This model is disabled by default, to enable it set variable `using_schedule_histories` to `true` in your `dbt_project.yml`.
+  - While currently used for schedule tracking, this table has possible future applications, such as tracking user changes.
+
+## Features
+- Updated the `stg_zendesk__schedule_holidays` model to allow users to disable holiday processing (while still using schedules) by setting `using_holidays` to `false`.
+- Added field-level documentation for the `stg_zendesk__audit_log` table.
+
+## Under the Hood Improvements
+- Added seed data for `audit_log` to enhance integration testing capabilities.
+
 # dbt_zendesk_source v0.12.0
 [PR #53](https://github.com/fivetran/dbt_zendesk_source/pull/53) includes the following updates:
 ## Breaking changes

README.md

@@ -44,7 +44,7 @@ Include the following zendesk_source package version in your `packages.yml` file
 ```yaml
 packages:
   - package: fivetran/zendesk_source
-    version: [">=0.12.0", "<0.13.0"]
+    version: [">=0.13.0", "<0.14.0"]
 ```
 ### Step 3: Define database and schema variables
 By default, this package runs using your target database and the `zendesk` schema. If this is not where your Zendesk Support data is (for example, if your zendesk schema is named `zendesk_fivetran`), add the following configuration to your root `dbt_project.yml` file:
@@ -54,11 +54,13 @@ vars:
     zendesk_database: your_destination_name
     zendesk_schema: your_schema_name 
 ```
-### Step 4: Disable models for non-existent sources
-This package takes into consideration that not every Zendesk Support account utilizes the `schedule`, `domain_name`, `user_tag`, `organization_tag`, or `ticket_form_history` features, and allows you to disable the corresponding functionality. By default, all variables' values are assumed to be `true`. Add variables for only the tables you want to disable:
+### Step 4: Enable/Disable models for non-existent sources
+This package takes into consideration that not every Zendesk Support account utilizes the `schedule`, `schedule_holiday`, `ticket_schedule`, `daylight_time`, `time_zone`, `audit_log`, `domain_name`, `user_tag`, `organization_tag`, or `ticket_form_history` features, and allows you to disable the corresponding functionality. By default, all variables' values are assumed to be `true`, except for `using_schedule_histories`. Add variables for only the tables you want to enable/disable:
 ```yml
 vars:
-    using_schedules:            False         #Disable if you are not using schedules
+    using_schedule_histories:   True          #Enable if you are using audit_logs for schedule histories
+    using_schedules:            False         #Disable if you are not using schedules, which requires source tables ticket_schedule, daylight_time, and time_zone
+    using_holidays:             False         #Disable if you are not using schedule_holidays for holidays
     using_domain_names:         False         #Disable if you are not using domain names
     using_user_tags:            False         #Disable if you are not using user tags
     using_ticket_form_history:  False         #Disable if you are not using ticket form history

dbt_project.yml

@@ -1,6 +1,6 @@
 config-version: 2
 name: 'zendesk_source'
-version: '0.12.0'
+version: '0.13.0'
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 models:
   zendesk_source:
@@ -28,6 +28,7 @@ vars:
     user: "{{ source('zendesk', 'user') }}"
     time_zone: "{{ source('zendesk', 'time_zone') }}"
     daylight_time: "{{ source('zendesk', 'daylight_time') }}"
+    audit_log: "{{ source('zendesk', 'audit_log') }}"
 
   zendesk__ticket_passthrough_columns: []
   zendesk__user_passthrough_columns: []

docs/catalog.json

@@ -1,11 +1,12 @@
 config-version: 2
 
 name: 'zendesk_source_integration_tests'
-version: '0.12.0'
+version: '0.13.0'
 
 profile: 'integration_tests'
 
 vars:
+
   zendesk_schema: zendesk_source_integrations_tests_5
   zendesk_source:
     zendesk_organization_identifier:           "organization_data"
@@ -25,14 +26,11 @@ vars:
     zendesk_organization_tag_identifier:       "organization_tag_data"
     zendesk_user_identifier:                   "user_data"
     zendesk_user_tag_identifier:               "user_tag_data"
+    zendesk_audit_log_identifier:              "audit_log_data"
 
+  ## Uncomment for docs generation
+  # using_schedule_histories: True
 
-    using_schedules:            True
-    using_domain_names:         True
-    using_user_tags:            True
-    using_ticket_form_history:  True
-    using_organization_tags:    True
-    
 seeds:
   zendesk_source_integration_tests:
     +quote_columns: "{{ true if target.type == 'redshift' else false }}"

docs/manifest.json

@@ -0,0 +1,8 @@
+id,_fivetran_synced,action,actor_id,change_description,created_at,source_id,source_label,source_type
+579796,2024-05-28 21:53:06.793000,update,37253,"Workweek changed from {:sun=>{""01:45""=>""02:45""}, :mon=>{""09:00""=>""20:00""}, :tue=>{""09:00""=>""20:00""}, :wed=>{""08:00""=>""20:00""}, :thu=>{""08:00""=>""20:00""}, :fri=>{""08:00""=>""20:00""}} to {:sun=>{""03:00""=>""04:00""}, :mon=>{""08:00""=>""20:00""}, :tue=>{""08:00""=>""20:00""}, :wed=>{""07:15""=>""20:00""}, :thu=>{""07:15""=>""20:00""}, :fri=>{""07:15""=>""20:00""}}",2024-05-28 21:51:37.000000,18542,Workweek: Central US Schedule,zendesk/business_hours/workweek
+2679952,2024-05-28 16:18:58.471000,update,37253,"Workweek changed from {:thu=>{""09:00""=>""17:00""}, :fri=>{""09:00""=>""17:00""}, :mon=>{""09:00""=>""17:00""}, :tue=>{""09:00""=>""17:00""}, :wed=>{""09:00""=>""17:00""}} to {:mon=>{""09:00""=>""17:00""}, :tue=>{""09:00""=>""17:00""}, :wed=>{""09:00""=>""17:00""}, :thu=>{""09:00""=>""17:00""}, :fri=>{""09:00""=>""17:00""}}",2024-05-21 11:20:29.000000,267996,Workweek: New schedule here,zendesk/business_hours/workweek
+293556,2024-05-28 16:18:58.471000,update,37253,"Workweek changed from {} to {:mon=>{""09:00""=>""17:00""}, :tue=>{""09:00""=>""17:00""}, :wed=>{""09:00""=>""17:00""}, :thu=>{""09:00""=>""17:00""}, :fri=>{""09:00""=>""17:00""}}",2024-05-21 11:20:28.000000,267996,Workweek: New schedule here,zendesk/business_hours/workweek
+4441364,2024-05-28 16:18:58.471000,update,37253,"Workweek changed from {:wed=>{""09:00""=>""17:00""}, :thu=>{""09:00""=>""17:00""}, :mon=>{""09:00""=>""17:00""}, :tue=>{""09:00""=>""17:00""}, :fri=>{""09:00""=>""17:00""}} to {:mon=>{""09:00""=>""17:00""}, :tue=>{""09:00""=>""17:00""}, :wed=>{""09:00""=>""17:00""}, :thu=>{""09:00""=>""17:00""}, :fri=>{""09:00""=>""17:00""}}",2024-05-21 11:20:10.000000,267996,Workweek: New schedule 2,zendesk/business_hours/workweek
+70900,2024-05-28 16:18:58.471000,update,37253,"Workweek changed from {} to {:mon=>{""09:00""=>""17:00""}, :tue=>{""09:00""=>""17:00""}, :wed=>{""09:00""=>""17:00""}, :thu=>{""09:00""=>""17:00""}, :fri=>{""09:00""=>""17:00""}}",2024-05-21 11:20:09.000000,267996,Workweek: New schedule 2,zendesk/business_hours/workweek
+70901,2024-05-28 16:18:58.471000,update,37253,"Workweek changed from {"mon":{"10:00":"20:00"},"tue":{"10:00":"20:00"},"wed":{"10:00":"20:00"},"thu":{"10:00":"20:00"},"fri":{"10:00":"20:00"}} to {"mon":{"10:00":"22:00"},"tue":{"10:00":"22:00"},"wed":{"10:00":"22:00"},"thu":{"10:00":"22:00"},"fri":{"10:00":"22:00"}}",2024-05-21 11:20:09.000000,267996,Workweek: New schedule 2,zendesk/business_hours/workweek
+70902,2024-05-28 16:18:58.471000,update,37253,"Workweek changed from {:mon=>{""09:00""=>""10:45"", ""11:45""=>""12:45"", ""13:45""=>""14:45"", ""15:15""=>""16:15"", ""19:00""=>""20:00"", ""17:30""=>""18:30""}, :tue=>{""00:15""=>""13:15"", ""13:30""=>""18:30"", ""18:45""=>""21:45"", ""22:00""=>""24:00""}, :wed=>{""09:00""=>""21:00""}, :thu=>{""17:00""=>""18:00"", ""19:45""=>""20:45"", ""09:00""=>""10:45"", ""12:15""=>""13:15"", ""14:30""=>""15:30""}, :fri=>{""09:00""=>""12:45"", ""19:15""=>""22:30"", ""14:45""=>""15:45"", ""17:30""=>""18:30""}} to {:mon=>{""09:00""=>""10:45"", ""11:45""=>""12:45"", ""13:45""=>""14:45"", ""15:15""=>""16:15"", ""17:30""=>""18:30"", ""19:00""=>""20:00""}, :tue=>{""00:15""=>""13:15"", ""13:30""=>""18:30"", ""18:45""=>""21:45"", ""22:00""=>""24:00""}, :wed=>{""02:30""=>""21:45""}, :thu=>{""09:00""=>""10:45"", ""12:15""=>""13:15"", ""14:30""=>""15:30"", ""17:00""=>""18:00"", ""19:45""=>""20:45""}, :fri=>{""09:00""=>""12:45"", ""14:45""=>""15:45"", ""17:30""=>""18:30"", ""19:15""=>""22:30""}}",2024-05-21 11:20:09.000000,267996,Workweek: New schedule 2,zendesk/business_hours/workweek

docs/run_results.json

@@ -0,0 +1,17 @@
+{% macro get_audit_log_columns() %}
+
+{% set columns = [
+    {"name": "_fivetran_synced", "datatype": dbt.type_timestamp()},
+    {"name": "id", "datatype": dbt.type_int()},
+    {"name": "action", "datatype": dbt.type_string()},
+    {"name": "actor_id", "datatype": dbt.type_int()},
+    {"name": "change_description", "datatype": dbt.type_string()},
+    {"name": "created_at", "datatype": dbt.type_timestamp()},
+    {"name": "source_id", "datatype": dbt.type_int()},
+    {"name": "source_label", "datatype": dbt.type_string()},
+    {"name": "source_type", "datatype": dbt.type_string()}
+] %}
+
+{{ return(columns) }}
+
+{% endmacro %}

integration_tests/dbt_project.yml

@@ -13,6 +13,36 @@ sources:
 
 
     tables:
+      - name: audit_log
+        identifier: "{{ var('zendesk_audit_log_identifier', 'audit_log')}}"
+        config:
+          enabled: "{{ var('using_schedules', true) and var('using_schedule_histories', false) }}"
+        freshness: null
+        description: >
+          The `audit_log` table captures historical changes and actions within Zendesk. It provides a record of modifications 
+          made to tickets, schedules, and other objects, allowing for a detailed audit trail. Each row represents an action 
+          performed by an actor, including the time of the action, the affected entity, and a description of the changes. This 
+          table is especially useful for tracking schedule modifications and maintaining a history of schedule changes.
+        columns:
+          - name: id
+            description: The unique identifier for each audit log entry, representing a distinct action or change.
+          - name: action
+            description: Describes the specific action performed within Zendesk, such as ticket updates or schedule modifications.
+          - name: actor_id
+            description: The unique identifier of the user or system responsible for performing the action.
+          - name: change_description
+            description: A detailed description of the changes made during the action, capturing what was altered.
+          - name: created_at
+            description: The timestamp indicating when the action was performed and recorded in the audit log.
+          - name: source_id
+            description: The unique identifier of the entity affected by the action, such as a ticket or schedule.
+          - name: source_label
+            description: A label that provides additional context about the affected entity, typically related to its type or name.
+          - name: source_type
+            description: Specifies the type of entity impacted by the action, such as a ticket, schedule, or user.
+          - name: _fivetran_synced
+            description: The timestamp when the record was last synchronized by Fivetran, used to track data freshness.
+
       - name: ticket
         identifier: "{{ var('zendesk_ticket_identifier', 'ticket')}}"
         description: >
@@ -355,7 +385,7 @@ sources:
         description: Information about holidays for each specified schedule.
         freshness: null
         config:
-          enabled: "{{ var('using_schedules', true) }}"
+          enabled: "{{ var('using_schedules', true) and var('using_holidays', true) }}"
         columns:
           - name: end_date
             description: ISO 8601 representation of the holiday end date.

integration_tests/seeds/audit_log_data.csv

@@ -1,6 +1,32 @@
 version: 2
 
 models:
+  - name: stg_zendesk__audit_log
+    description: >
+      The `audit_log` table captures historical changes and actions within Zendesk. It provides a record of modifications 
+      made to tickets, schedules, and other objects, allowing for a detailed audit trail. Each row represents an action 
+      performed by an actor, including the time of the action, the affected entity, and a description of the changes. This 
+      table is especially useful for tracking schedule modifications and maintaining a history of schedule changes.
+    columns:
+      - name: audit_log_id
+        description: The unique identifier for each audit log entry, representing a distinct action or change.
+      - name: action
+        description: Describes the specific action performed within Zendesk, such as ticket updates or schedule modifications.
+      - name: actor_id
+        description: The unique identifier of the user or system responsible for performing the action.
+      - name: change_description
+        description: A detailed description of the changes made during the action, capturing what was altered.
+      - name: created_at
+        description: The timestamp indicating when the action was performed and recorded in the audit log.
+      - name: source_id
+        description: The unique identifier of the entity affected by the action, such as a ticket or schedule.
+      - name: source_label
+        description: A label that provides additional context about the affected entity, typically related to its type or name.
+      - name: source_type
+        description: Specifies the type of entity impacted by the action, such as a ticket, schedule, or user.
+      - name: _fivetran_synced
+        description: The timestamp when the record was last synchronized by Fivetran, used to track data freshness.
+
   - name: stg_zendesk__ticket
     description: >
       Tickets are the means through which your end users (customers) communicate with agents in Zendesk Support. Tickets can 

macros/get_audit_log_columns.sql

@@ -0,0 +1,45 @@
+{{ config(enabled=var('using_schedules', True) and var('using_schedule_histories', False)) }}
+
+with base as (
+
+    select * 
+    from {{ ref('stg_zendesk__audit_log_tmp') }}
+
+),
+
+fields as (
+
+    select
+        /*
+        The below macro is used to generate the correct SQL for package staging models. It takes a list of columns 
+        that are expected/needed (staging_columns from dbt_zendesk_source/models/tmp/) and compares it with columns 
+        in the source (source_columns from dbt_zendesk_source/macros/).
+        For more information refer to our dbt_fivetran_utils documentation (https://github.com/fivetran/dbt_fivetran_utils.git).
+        */
+        {{
+            fivetran_utils.fill_staging_columns(
+                source_columns=adapter.get_columns_in_relation(ref('stg_zendesk__audit_log_tmp')),
+                staging_columns=get_audit_log_columns()
+            )
+        }}
+        
+    from base
+),
+
+final as (
+    select 
+        cast(id as {{ dbt.type_string() }}) as audit_log_id,
+        action,
+        actor_id,
+        change_description,
+        cast(created_at as {{ dbt.type_timestamp() }}) as created_at,
+        source_id,
+        source_label,
+        source_type,
+        _fivetran_synced
+
+    from fields
+)
+
+select * 
+from final

models/src_zendesk.yml

@@ -1,5 +1,5 @@
---To disable this model, set the using_schedules variable within your dbt_project.yml file to False.
-{{ config(enabled=var('using_schedules', True)) }}
+--To disable this model, set the using_schedules or using_holidays variable within your dbt_project.yml file to False.
+{{ config(enabled=var('using_schedules', True) and var('using_holidays', True)) }}
 
 with base as (
 

models/stg_zendesk.yml

@@ -0,0 +1,4 @@
+{{ config(enabled=var('using_schedules', True) and var('using_schedule_histories', False)) }}
+
+select {{ dbt_utils.star(source('zendesk','audit_log')) }}  
+from {{ source('zendesk','audit_log') }} as audit_log_table

models/stg_zendesk__audit_log.sql

@@ -1,5 +1,5 @@
---To disable this model, set the using_schedules variable within your dbt_project.yml file to False.
-{{ config(enabled=var('using_schedules', True)) }}
+--To disable this model, set the using_schedules or using_holidays variable within your dbt_project.yml file to False.
+{{ config(enabled=var('using_schedules', True) and var('using_holidays', True)) }}
 
 select {{ dbt_utils.star(source('zendesk', 'schedule_holiday')) }}
 from {{ source('zendesk', 'schedule_holiday') }} as schedule_holiday_table