SNOW-2084165 Add dataframe operation lineage on SnowparkSQLException

[sfc-gh-aalam]

1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR.

   Fixes SNOW-2084165

2. Fill out the following pre-review checklist:

   • I am adding a new automated test(s) to verify correctness of my new code
     • If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing
   • I am adding new logging messages
   • I am adding a new telemetry message
   • I am adding new credentials
   • I am adding a new dependency
   • If this is a new feature/behavior, I'm adding the Local Testing parity changes.
   • I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: Thread-safe Developer Guidelines
   • If adding any arguments to public Snowpark APIs or creating new public Snowpark APIs, I acknowledge that I have ensured my changes include AST support. Follow the link for more information: AST Support Guidelines

3. Please describe how your code solves the related issue.

   see doc.

[sfc-gh-snowflakedb-snyk-sa]

🎉 Snyk checks have passed. No issues have been found so far.

✅ security/snyk check is complete. No issues have been found. (View Details)

✅ license/snyk check is complete. No issues have been found. (View Details)

[sfc-gh-jkew]

In the hybrid client prototype we are using a slightly different method to get the source location; by just using inspect to walk the stack to the appropriate source location. We have to do this because modin is not using any of the AST stuff, but it's also relatively straight forward.

I sort of want to use your debugging tool for snowpandas as well; but we may want to refactor this so we don't require any of the protobuf work.

snowpark-python/src/snowflake/snowpark/modin/plugin/_internal/telemetry.py

Line 625 in 685fc31

def get_user_source_location(group: str) -> dict[str, str]:

[sfc-gh-jkew]

What about using this function?

snowpark-python/src/snowflake/snowpark/_internal/open_telemetry.py

Line 116 in f415c73

def context_manager_code_location(frame_info, func) -> Tuple[str, int]:

Essentially we seem to have three approaches to this problem. I'm /less/ of a fan of the AST because it doesn't help pandas for this type of debugging. but it seems like we might be able to consolidate w/ the open telemetry approach.

[sfc-gh-aling]

this is similar to what I would expect, see my comment on: #3380 (comment)

I'm thinking of the following to provide unified debug config experience.

@experimental(version="1.33.0")
def debug_config(
    *
    enable_eager_schema_validation=False,
    enable_dataframe_trace_on_error=False,
)

when users want to enable, they do

import snowflake.snowpark.context
snowflake.snowpark.context.debug_config(enable_eager_schema_validation=True)
# or
snowflake.snowpark.context.debug_config(
    enable_eager_schema_validation=True,
    enable_dataframe_trace_on_error=True
)

[sfc-gh-aalam]

good idea. @sfc-gh-jrose and I are aligned on the name. Let me add the @experimental decorator as well.

[sfc-gh-aalam]

could not import decorator due to circular import issues but added a warning there

[sfc-gh-jrose]

I think we should default to True. That way users that want a basic development mode can call this function without any parameters.

[sfc-gh-heshah]

Nit: I would argue that this isn't meant to be batch ID anymore. Within each Python session that imports the Snowpark module, each AST ID for a Table or Dataframe will be a UID.