feat: JIRA 1818 - Adding Types to Lineage Response for Special Queries in Prov-API.

[parth-kulkarni1]

Jira-1818 (Minor): Adding Types to Lineage Response for Special Queries in Prov-API.

JIRA Ticket 1818

Checklist

• If tests are required for this change, are they implemented?
• Are user documentation changes required, if so, is there a task to track it and/or is it completed?
• If developer/system documentation updates are required, is there a task to track it and/or is it completed?
• At least one developer has reviewed this change (unless PR is being used to mark a commit point without need for review)?

Description

This ticket involves making the Lineage Response more type-friendly and robust, so users can easily access the properties of the response via type hints.

Notes for reviewer

• Made two custom pydantic models.
• The "CustomLineageResponse" model overrides the "graph" field within LineageResponse.

This approach essentially allows you to access the LineageResponse in a typed manner. E.g. response.graph.nodes, response.graph.direction etc

[jyucsiro]

Can you create a test for this feature? I'd like to run the test using pytest to verify the feature functionality.

[jyucsiro]

i also tried adhoc.py in tests and it gave me this:

Traceback (most recent call last):
  File "/srv/repo/github/provena/provena-python-client/tests/adhoc.py", line 297, in <module>
    asyncio.run(main())
  File "/home/jon/miniconda3/envs/provena-client/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/home/jon/miniconda3/envs/provena-client/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/srv/repo/github/provena/provena-python-client/tests/adhoc.py", line 283, in main
    response = await client.prov_api.explore_upstream(
  File "/srv/repo/github/provena/provena-python-client/src/provenaclient/modules/prov.py", line 236, in explore_upstream
    typed_upstream_response = CustomLineageResponse.parse_obj(upstream_response.dict())
  File "pydantic/main.py", line 526, in pydantic.main.BaseModel.parse_obj
  File "pydantic/main.py", line 339, in pydantic.main.BaseModel.__init__
  File "pydantic/main.py", line 1074, in pydantic.main.validate_model
  File "pydantic/fields.py", line 895, in pydantic.fields.ModelField.validate
  File "pydantic/fields.py", line 1154, in pydantic.fields.ModelField._apply_validators
  File "pydantic/class_validators.py", line 304, in pydantic.class_validators._generic_validator_cls.lambda4
  File "/srv/repo/github/provena/provena-python-client/src/provenaclient/models/general.py", line 81, in convert_graph
    list_of_parsed_nodes: List[Node] = cls.parse_nodes(v.get('nodes', []))
AttributeError: 'CustomGraph' object has no attribute 'get'

[parth-kulkarni1]

Can you create a test for this feature? I'd like to run the test using pytest to verify the feature functionality.

Okay, I have added two small tests within the test_provenance_workflow section. I don't think this deserves a completely new test, as it can be checked elsewhere, where the special queries are being used and for efficiency purposes.

[parth-kulkarni1]

i also tried adhoc.py in tests and it gave me this:

Traceback (most recent call last):
  File "/srv/repo/github/provena/provena-python-client/tests/adhoc.py", line 297, in <module>
    asyncio.run(main())
  File "/home/jon/miniconda3/envs/provena-client/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/home/jon/miniconda3/envs/provena-client/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/srv/repo/github/provena/provena-python-client/tests/adhoc.py", line 283, in main
    response = await client.prov_api.explore_upstream(
  File "/srv/repo/github/provena/provena-python-client/src/provenaclient/modules/prov.py", line 236, in explore_upstream
    typed_upstream_response = CustomLineageResponse.parse_obj(upstream_response.dict())
  File "pydantic/main.py", line 526, in pydantic.main.BaseModel.parse_obj
  File "pydantic/main.py", line 339, in pydantic.main.BaseModel.__init__
  File "pydantic/main.py", line 1074, in pydantic.main.validate_model
  File "pydantic/fields.py", line 895, in pydantic.fields.ModelField.validate
  File "pydantic/fields.py", line 1154, in pydantic.fields.ModelField._apply_validators
  File "pydantic/class_validators.py", line 304, in pydantic.class_validators._generic_validator_cls.lambda4
  File "/srv/repo/github/provena/provena-python-client/src/provenaclient/models/general.py", line 81, in convert_graph
    list_of_parsed_nodes: List[Node] = cls.parse_nodes(v.get('nodes', []))
AttributeError: 'CustomGraph' object has no attribute 'get'

I have fixed this issue. It seems I over-engineered my initial approach, as I was converting into a dictionary and then back into a pydantic object, hence the issue of object has no attribute 'get'

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 65.11628% with 15 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/provenaclient/modules/prov.py	42.30%	15 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Files with missing lines	Coverage Δ
src/provenaclient/models/general.py	100.00% <100.00%> (ø)
src/provenaclient/modules/prov.py	48.51% <42.30%> (ø)

[jyucsiro]

Looking better now!

I was able to get my environment configured and able to run adhoc.py. A use case that I wanted to test was the ability to list all datasets. Some suggested tweak to adhoc.py - include a block to list all datasets:

e.g.

    print("Listing all datasets")
    for node in response.graph.nodes:
        if node.item_subtype == ItemSubType.DATASET:
            print(node.id, node.item_subtype) 

[parth-kulkarni1]

Looking better now!

I was able to get my environment configured and able to run adhoc.py. A use case that I wanted to test was the ability to list all datasets. Some suggested tweak to adhoc.py - include a block to list all datasets:

e.g.

    print("Listing all datasets")
    for node in response.graph.nodes:
        if node.item_subtype == ItemSubType.DATASET:
            print(node.id, node.item_subtype) 

I have added this now.

[jyucsiro]

This provides client functionality to type returned graph response from the provena API. Working well and able to filter subtypes now more easily.