get_attribute() return None in cdp mode

[Kamran-ov]

Hi Community,

Do you have any idea why I get None when I try to get the href attribute of an element in CDP mode?

with SB(uc=True, test=True) as sb:

sb.activate_cdp_mode(url)
sb.sleep(2.5)
sb.cdp.gui_click_element('button[id="onetrust-accept-btn-handler"]')
sb.sleep(1)
cats = sb.cdp.find_elements('li[data-testid="group-list-item"] a')
for cat in cats[1:2]:
    cat_name = cat.text
    print(cat_name)
    print("HTML:", cat.get_html())
    cat_url = cat.get_attribute('href')
    print("URL: ", cat_url)

Result:
======================== {nodriver_test.py:5:SB} starts ========================
3D Architecture
HTML: < a href="/3d-architecture-software/" class="sb color-mode-light link block font-bold no-underline">3D Architecture
URL: None
=================== {nodriver_test.py:5:SB} passed in 23.65s ===================

Note: I have put space on the element ("<a" >>>>> "< a", intentionally, output is not with space in terminal)

[mdmintz]

It's working for me in this example:

SeleniumBase/examples/cdp_mode/raw_kohls.py

Line 21 in 35dcd82

title = item.get_attribute("title")

[Kamran-ov]

Well, for some reason, neither get_attribute(), get_js_attributes() nothing worked for me inside the loop

In the end, I have decided to go with the custom function.
If anybody else has the same issue as I do, I will leave the script below.

   def get_custom_attribute(element, attribute):
       
       Extracts a custom attribute from the given element using regex.

       Args:
           element: The web element to extract the attribute from.
           attribute (str): The name of the attribute to extract.

       Returns:
           str or None: The value of the attribute if found, otherwise None.
       
       # Get the raw HTML of the element
       html = element.get_html()
       if not html:
           return None

       # Define a regex pattern to extract the given attribute
       pattern = fr'\s{attribute}=["\']([^"\']+)["\']'

       # Search for the pattern in the HTML
       match = re.search(pattern, html)

       # Return the extracted attribute value or None if not found
       return match.group(1) if match else None

[mdmintz]

It would be helpful if you could show an actual URL and element where get_attribute() didn't work.

[Kamran-ov]

sure,
website : kroger DOT com

sb = SB(uc=True,
       headless2=False,
       block_images=True,
       )
sb = sb.__enter__()
sb.activate_cdp_mode(base_url)
sb.wait(7)
sb.cdp.click('button#ExposedMenu-Category-Shop')
sb.wait(1)
uls = sb.cdp.find_elements('ul[data-testid="SecondCategoryDisplayer"]')

for idx, ul in enumerate(uls):
   if idx != 0:
       continue
   deps = ul.query_selector_all('li > button')
   for dep in deps:
       idx = dep.get_attribute('id')  # None
       attrs = dep.get_js_attributes()  # None
       print(idx)
       print(attrs)
       html = dep.get_html()  # Here it returns the element with all attributes.
       print(html)

Note that, without CDP same script was working normally, and I was getting the ID of each category without any issue (although my selectors were all XPATH). With CDP, I tried with XPATH not worked, then have converted them all to CSS thinking maybe it could help to fix the issue, but no success.

[mdmintz]

This is working for me:

from seleniumbase import SB

with SB(uc=True, test=True, locale_code="en") as sb:
    url = "https://www.kroger.com/"
    sb.activate_cdp_mode(url)
    sb.sleep(2.5)
    sb.cdp.click_if_visible("button#onetrust-accept-btn-handler")
    sb.sleep(1.2)
    sb.cdp.click('button#ExposedMenu-Category-Shop')
    sb.sleep(1)
    ul = sb.cdp.find_element('ul[data-testid="SecondCategoryDisplayer"]')
    deps = ul.query_selector_all('li > button')
    for dep in deps:
       idx = dep.get_attribute('id')  # None
       print(idx)

Output:

ExposedMenuContent-Link-Popular
ExposedMenuContent-Link-Seasonal Favorites
ExposedMenuContent-Link-Fresh
ExposedMenuContent-Link-Grocery
ExposedMenuContent-Link-Beer, Wine & Liquor
ExposedMenuContent-Link-Health, Beauty & Baby
ExposedMenuContent-Link-Cleaning & Household
ExposedMenuContent-Link-Home, Electronics & Toys
ExposedMenuContent-Link-Pet
ExposedMenuContent-Link-Our Brands
ExposedMenuContent-Link-Get it Shipped
ExposedMenuContent-Link-Gift Cards
ExposedMenuContent-Link-Explore More

[Kamran-ov]

I am not sure, then what the issue is. Mine returns None.
Well, My custom attribute extract helps me for now. All good. Thank you for checking.