issue with parsing metadata (DeviceServices) tag with a differnent namespace

[DrYSG]

I am using the package python pyodata and getting parse issues when the meta-data is returned from the root ODATA location. I don't own the site, so the ODATA interface is controlled by riverbed software.

The error is (It look like it is looking for the DataService tag.

  File "C:\Users\1455765990E\AppData\Local\Programs\Python\Python311\Lib\site-packages\pyodata\v2\model.py", line 2752, in build
    dataservices = next((child for child in edmx if etree.QName(child.tag).localname == 'DataServices'))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\1455765990E\AppData\Local\Programs\Python\Python311\Lib\site-packages\pyodata\v2\model.py", line 2752, in <genexpr>
    dataservices = next((child for child in edmx if etree.QName(child.tag).localname == 'DataServices'))
                                                    ^^^^^^^^^^^^^^^^^^^^^^
  File "src\lxml\etree.pyx", line 1849, in lxml.etree.QName.__init__
  File "src\lxml\apihelpers.pxi", line 1754, in lxml.etree._tagValidOrRaise
ValueError: Invalid tag name '<cyfunction Comment at 0x000001B5E651C5F0>'

and my code is: (I tired to tell it that it is a non-standard name space with a custom name space tag.

import requests
import pyodata
from pyodata.v2.model import PolicyFatal, PolicyWarning, PolicyIgnore, ParserError, Config

serviceURL = 'https://lms-odata.aternity.com/aternity.odata/v2.0/'
session = requests.Session()
session.auth = ('myself@mycom.com, 'pass')
session.verify = False

namespaces = {
    'edmx': 'http://docs.oasis-open.org/odata/ns/edmx',
    'edm': 'http://docs.oasis-open.org/odata/ns/edm',
    'riverbed': 'www.riverbed.com/api/contract'
}

custom_config = Config(
    xml_namespaces=namespaces,
    default_error_policy=PolicyIgnore()
)

with open('metadata.xml', 'rb') as metaFile:
    metadata = metaFile.read()

# Create instance of OData client
aternity = pyodata.Client(serviceURL, session, metadata=metadata, config=custom_config)

the start of the metadata file looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<edmx:Edmx xmlns:edmx="http://docs.oasis-open.org/odata/ns/edmx" xmlns:riverbed="www.riverbed.com/api/contract" Version="4.0" riverbed:aternity_api_version="2.0">
   <!--\n\n \tAternity Api version : 2.0 \n\n-->
   <edmx:DataServices>
      <Schema xmlns="http://docs.oasis-open.org/odata/ns/edm" Namespace="aternity.odata">
         <EntityType Name="RAW_DATA_TOP_PROCESSES_VIRTUAL_APP_SERVER_SESSIONS_Type">
            <Property Name="ACCOUNT_ID" Type="Edm.Int64" />
            <Property Name="APPLICATION_NAME" Type="Edm.String" MaxLength="1024" />
            <Property Name="APPLICATION_VERSION" Type="Edm.String" MaxLength="1024" />
            <Property Name="MEASUREMENT_START_TIMESTAMP" Type="Edm.DateTimeOffset" />
            <Property Name="PRC_CPU_UTIL_AVG" Type="Edm.Decimal" Precision="37" Scale="10" />
            <Property Name="PRC_IO_READ_RATE" Type="Edm.Decimal" Precision="37" Scale="10" />

...

[phanak-sap]

Hi @DrYSG
I just glanced oved the issue, did not yet tried to reproduce it locally. Anyway, for me the root cause seems to be the LXML parsing and presence of comments in the XML. This is what I didn't see so far in any $metadata file and it seems that we do not have it as a unit test case.

If this hypothesis is true, the fix should be pretty straightforward in the pyodata package - e.g. https://stackoverflow.com/questions/18313818/how-to-not-load-the-comments-while-parsing-xml-in-lxml/

Idea for a possible quick workaround - try to download the metadata file first using just the requests library - url = 'https://lms-odata.aternity.com/aternity.odata/v2.0/$metadata', then remove the HTML/XML comment from the file and then initialize the pyodata package from local file, according to documentation:
https://github.com/SAP/python-pyodata/blob/master/docs/usage/initialization.rst#get-the-service-with-local-metadata

[DrYSG]

Well, that is a step forward. Now, how would I track this new error down:

(BTW, I searched for a Precision="0" in the xml file, I don't see that anywhere.

python : Traceback (most recent call last):
At line:1 char:1
+ python .\pub-local.py
+ ~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (Traceback (most recent call last)::String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 
  File "C:\Users\1455765990E\OneDrive - United States Air Force\Documents\Software Projects\Aternity\pub-local.py", line 25, in <module>
    aternity = pyodata.Client(serviceURL, session, metadata=metadata, config=custom_config)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\1455765990E\AppData\Local\Programs\Python\Python311\Lib\site-packages\pyodata\client.py", line 91, in __new__
    return Client._build_service(logger, url, connection, odata_version, namespaces, config, metadata)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\1455765990E\AppData\Local\Programs\Python\Python311\Lib\site-packages\pyodata\client.py", line 110, in _build_service
    schema = pyodata.v2.model.MetadataBuilder(metadata, config=config).build()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\1455765990E\AppData\Local\Programs\Python\Python311\Lib\site-packages\pyodata\v2\model.py", line 2782, in build
    schema = Schema.from_etree(edm_schemas, self._config)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\1455765990E\AppData\Local\Programs\Python\Python311\Lib\site-packages\pyodata\v2\model.py", line 1276, in from_etree
    etype = EntityType.from_etree(entity_type, config)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\1455765990E\AppData\Local\Programs\Python\Python311\Lib\site-packages\pyodata\v2\model.py", line 1687, in from_etree
    etype = super(EntityType, cls).from_etree(type_node, config)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\1455765990E\AppData\Local\Programs\Python\Python311\Lib\site-packages\pyodata\v2\model.py", line 1513, in from_etree
    stp = StructTypeProperty.from_etree(proprty)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\1455765990E\AppData\Local\Programs\Python\Python311\Lib\site-packages\pyodata\v2\model.py", line 1930, in from_etree
    return StructTypeProperty(
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\1455765990E\AppData\Local\Programs\Python\Python311\Lib\site-packages\pyodata\v2\model.py", line 1814, in __init__
    super(StructTypeProperty, self).__init__(name, type_info, nullable, max_length, precision, scale, fixed_length)
  File "C:\Users\1455765990E\AppData\Local\Programs\Python\Python311\Lib\site-packages\pyodata\v2\model.py", line 813, in __init__
    self._check_scale_value()
  File "C:\Users\1455765990E\AppData\Local\Programs\Python\Python311\Lib\site-packages\pyodata\v2\model.py", line 892, in _check_scale_value
    raise PyODataModelError('Scale value ({}) must be less than or equal to precision value ({})'
pyodata.exceptions.PyODataModelError: Scale value (10) must be less than or equal to precision value (0)

[DrYSG]

metadata.xml.txt

[phanak-sap]

Hi @DrYSG.

First pointing out the obvious - this is a different, new problem.

You understood the error generally correctly, but ended searching for something that is simply not there at all - and that's the root cause of this new problem. We now moved from LXML parsing error (parsing of generic file as valid xml) to a pyodata parsing error (parsing generic xml as valid odata v2 metadata document).

The metadata file is invalid, in such way that even the loosest error handling policy PolicyIgnore cannot parse it without throwing out this exception.

This property is one example out of many with same problem:
<Property Name="USERS_COUNT_LAST_7_DAYS" Type="Edm.Decimal" Scale="10" />

Other Edm.Decimals are defined correctly and they have no problem, e.g.
<Property Name="PRC_CPU_UTIL_AVG" Type="Edm.Decimal" Precision="37" Scale="10" />

For more information, refer to chapter 6 - Primitive types here: https://www.odata.org/documentation/odata-version-2-0/overview/

Correct way is to fix the $metadata by the service API owner.

Workaround for you now is to manually search and replace all Type="Edm.Decimal" Scale="10" strings to just Type="Edm.Decimal" (Decimal type with uknown scale), since you are working with modified local file anyway :)

I tried that locally, using latest pyodata==1.10.1 on Python 3.10.2, and with that modification I can initialize pyodata client with

Metadata validation status: True

[DrYSG]

You get 5-stars for helpfulness, and even more for writing a complete answer. I completely agree with you about who needs to fix their code.

One small lesson you might want to take from this, that if the parser could throw the line that caused it to choke out in the stacktrace, Then one would not have to hunt to find the problem.

Your workaround is perfectly acceptable to us.

[phanak-sap]

OK, I will leave this issue open for a fix for parsing xml files with comments.

One small lesson you might want to take from this, that if the parser could throw the line that caused it to choke out in the stacktrace, Then one would not have to hunt to find the problem.

Yeah, we know that this would be nice basically for years. I would love it myself. But it is not that simple. It it basically a stream of chars in the end (or at least can be from networking libraries), so hard to backtrack to original line. But say even the character position would be enough... there seems to be nothing for such use case in the LXML parser itself, to go back from etree nodes back to original thing, if something during business logic of the etree parsing throws an error.

[DrYSG]

I was sorta expecting that it might be hard, especially if you are layering on 3rd party libs for the parse. I will close out the ticket as soon as I get the hotfix working.

It looks like there might be more edge cases, since I did (and double checked) all the cases you suggested, and it still gives the same error.

  File "C:\Users\1455765990E\AppData\Local\Programs\Python\Python311\Lib\site-packages\pyodata\v2\model.py", line 892, in _check_scale_value
    raise PyODataModelError('Scale value ({}) must be less than or equal to precision value ({})'
pyodata.exceptions.PyODataModelError: Scale value (10) must be less than or equal to precision value (0)

[phanak-sap]

I have just did the search and replace the Scale="10" for Decimals that have just that and this is my metadata file - https://github.com/phanak-sap/pyodata-issue-files/tree/master/%23249

[DrYSG]

@phanak-sap Thank you Petr, all is well now. Now to get to the real stuff about using it. Thank you for sticking around.

[DrYSG]

@phanak-sap My apologies if I should be doing this different. Probably this is a different ticket, Maybe I should be using Stack Overflow. But since you are already up to speed on this, and have the metadata.xml file, I figured this is something you have a very quick answer for me.

I am trying to do the textbook example you have for "get all entities in a set": https://pyodata.readthedocs.io/en/latest/usage/querying.html

The code is:

custom_config = Config(
    xml_namespaces=namespaces,
    default_error_policy=PolicyIgnore()
)

with open('metadata.xml', 'rb') as metaFile:
    metadata = metaFile.read()

# Create instance of OData client
aClient = pyodata.Client(serviceURL, session, metadata=metadata, config=custom_config)

raw = aClient.entity_sets.BUSINESS_ACTIVITIES_RAW.get_entities().select('LOCATION_CITY,DEVICE_NAME').execute()
for row in raw:
    print(row.DEVICE_NAME, row.LOCATION_CITY)

But it giving this error which I cannot decipher:

C:\Users\1455765990E\AppData\Local\Programs\Python\Python311\Lib\site-packages\urllib3\connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host 'lms-odata.aternity.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
Traceback (most recent call last):
  File "C:\Users\1455765990E\OneDrive - United States Air Force\Documents\Software Projects\Aternity\pub-local.py", line 27, in <module>
    raw = aClient.entity_sets.BUSINESS_ACTIVITIES_RAW.get_entities().select('LOCATION_CITY,DEVICE_NAME').execute()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\1455765990E\AppData\Local\Programs\Python\Python311\Lib\site-packages\pyodata\v2\service.py", line 349, in execute
    return self._call_handler(response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\1455765990E\AppData\Local\Programs\Python\Python311\Lib\site-packages\pyodata\v2\service.py", line 362, in _call_handler
    return self._handler(response)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\1455765990E\AppData\Local\Programs\Python\Python311\Lib\site-packages\pyodata\v2\service.py", line 1494, in get_entities_handler
    entities = content['d']
               ~~~~~~~^^^^^
KeyError: 'd'

It would also be nice to have a example that just gets all properties for a row (or tells me how to query the metadata to get a list of all properties for an entity, so that I can iterate through the properties).