Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pre-build query not works - Job Latency Per Job (Batch Duration) #209

Open
hueiyuan opened this issue Apr 14, 2023 · 0 comments
Open

Pre-build query not works - Job Latency Per Job (Batch Duration) #209

hueiyuan opened this issue Apr 14, 2023 · 0 comments

Comments

@hueiyuan
Copy link

hueiyuan commented Apr 14, 2023

We founded that the pre-build query does not work about calculating job latency per job. It seems like that Properties_spark_app_id_s, Job_ID_d, Submission_Time_d, Completion_Time_d these fields does not existed in SparkListenerEvent_CL this table.

let results=SparkListenerEvent_CL
| where  Event_s  contains "SparkListenerJobStart"
| extend metricsns=columnifexists("Properties_spark_metrics_namespace_s",Properties_spark_app_id_s)
| extend apptag=iif(isnotempty(metricsns),metricsns,Properties_spark_app_id_s)
| project Job_ID_d,apptag,Properties_spark_databricks_clusterUsageTags_clusterName_s,
Submission_Time_d,TimeGenerated
| order by TimeGenerated asc  nulls last 
| join kind= inner (
        SparkListenerEvent_CL
    | where Event_s contains "SparkListenerJobEnd"
    | where Job_Result_Result_s contains "JobSucceeded"
    | project Event_s,Job_ID_d,Completion_Time_d,TimeGenerated
) on Job_ID_d;
results
| extend slice=strcat(Properties_spark_databricks_clusterUsageTags_clusterName_s,"-",apptag)
| extend jobDuration=Completion_Time_d - Submission_Time_d 
| summarize percentiles(jobDuration,10,30,50,90)  by bin(TimeGenerated,  1m), slice
| order by TimeGenerated asc nulls last

Please to update related query and fix it. Thank you

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant