updated test script for better handling of please enter based prompts… #22

Mohsinsiddi · 2025-01-17T16:19:55Z

Description

This PR introduces two main improvements to the test framework:

Enhanced Environment Variable Handling

Added dynamic config-based handling for "Please enter" prompts
Improved type detection for different input types (API keys, credentials, RPCs)
Better handling of Tenderly, Twitter, and other agent-specific inputs
Values are now correctly pulled from agent config files

Improved Test Logging Security

Added custom logging handler to mask sensitive information
Better organization of pexpect spawn configuration
Protected sensitive data like passwords, API keys, and credentials in logs
More consistent debug vs non-debug mode handling

Mech Service Enhancement

Added health check bypass for mech service
Fixed the check_service_health function to handle mech service specially:

if "mech" in config_path.lower():
    logger.info("Bypassing health check for mech service")
    return True, {
        'response_time': 0,
        'status_code': 200,
        'error': None,
        'successful_checks': 1,
        'total_checks': 1
    }

…, included mech test without healthcheck tho

OjusWiZard · 2025-01-17T17:48:06Z

tests/test_run_service.py

+        # Security/Authentication inputs
+        "password": lambda p: "password" in p,
+        "backup_owner": lambda p: "backup owner" in p,
+
+        # API Keys and Keys
+        "api_key": lambda p: any(x in p for x in [
+            "api key", 
+            "api_key", 
+            "apikey", 
+            "tenderly api", 
+            "coingecko api",
+            "gemini api"
+        ]),
+
+        # Tenderly specific inputs
+        "tenderly_info": lambda p: any(x in p for x in [
+            "tenderly account", 
+            "tenderly project",
+            "account slug",
+            "project slug"
+        ]),
+
+        # Twitter related inputs
+        "twitter_creds": lambda p: any(x in p for x in [
+            "twitter username", 
+            "twitter email", 
+            "twitter password"
+        ]),
+
+        # Technical configurations
+        "rpc": lambda p: any(x in p for x in ["rpc", "eth_newfilter"]),
+        "address": lambda p: any(x in p for x in ["address", "contract", "safe"]),
+
+        # Content/Description inputs
+        "persona": lambda p: "persona" in p,
+
+        # Navigation inputs
+        "choice": lambda p: "choice" in p,
+        "enter": lambda p: any(x in p for x in ["press enter", "continue"])


it appears to me that these agent specific checks are bloating the whole thing too much. If I'm not wrong then they were added to resolve that ENS issue right? If yes, then they are not needed anymore. We can clean up a lot here, wdyt?

yes, I shall remove this, once PR passes the test

OjusWiZard · 2025-01-20T11:43:45Z

tests/test_run_service.py

@@ -561,7 +570,7 @@ def get_config_files():
    logger.info(f"Found config files: {[f.name for f in config_files]}")

    # TODO: Support the test for memeooorr and mech


please remove this comment also

OjusWiZard · 2025-01-20T12:53:29Z

.github/workflows/python-tests.yml

-  pull_request:
-    branches:
-      - main
+  pull_request:  # Removed branches restriction to run on all PRs


please remove the comment, I guess you wanted me to read this comment? If that's the case then just comment it like this one in the PR. Because the comment in the code, doesn't make sense to a 3rd person.

OjusWiZard · 2025-01-20T12:54:32Z

tests/test_run_service.py

@@ -182,6 +182,17 @@ def check_docker_status(logger: logging.Logger, config_path: str) -> bool:
 def check_service_health(logger: logging.Logger, config_path: str) -> tuple[bool, dict]:
    """Enhanced service health check with metrics."""
    service_config = get_service_config(config_path)
+
+    if "mech" in config_path.lower():
+        logger.info("Bypassing health check for mech service")


Suggested change

logger.info("Bypassing health check for mech service")

logger.info("Bypassing health check for mech service because it's not supported")

OjusWiZard · 2025-01-20T12:59:54Z

tests/test_run_service.py

@@ -595,7 +626,6 @@ def get_base_config() -> dict:
        r"Enter your choice": base_config["STAKING_CHOICE"],
        r"Please input your backup owner \(leave empty to skip\)\:": base_config["BACKUP_WALLET"],  # Escape parentheses
        r"Press enter to continue": "\n",


Suggested change

r"Press enter to continue": "\n",

r"Press enter to continue": "\n",

r"Please enter .*": lambda output, logger: handle_env_var_prompt(output, logger, config_path)

why not keep this line here and then remove it from all the agent-specific parts?

I didnt do it because as its a get_base_config handler, didnt wanted to pass config paths into it

I updated it too

OjusWiZard · 2025-01-20T13:00:55Z

tests/test_run_service.py

-            time.sleep(0.5)
+        # Send input with a small delay
+        child.write(response + os.linesep)
+        time.sleep(2)


we can avoid these delay now right?

need to test it then! removed it

Mohsinsiddi added 2 commits January 17, 2025 16:28

updated test script for better handling of please enter based prompts…

1971b54

…, included mech test without healthcheck tho

updated ci file to trigger on every PR

6f7f1bc

OjusWiZard reviewed Jan 17, 2025

View reviewed changes

updated operate hash to fix latest issue

beb6a4e

OjusWiZard reviewed Jan 20, 2025

View reviewed changes

updated test script based on PR comments

b9d16d8

OjusWiZard approved these changes Jan 20, 2025

View reviewed changes

OjusWiZard merged commit 3060b03 into fix/ENS-error Jan 20, 2025
1 check passed

OjusWiZard deleted the feat/mech_tests branch January 20, 2025 13:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

updated test script for better handling of please enter based prompts… #22

updated test script for better handling of please enter based prompts… #22

Mohsinsiddi commented Jan 17, 2025

OjusWiZard Jan 17, 2025

Mohsinsiddi Jan 20, 2025

OjusWiZard Jan 20, 2025

OjusWiZard Jan 20, 2025

Mohsinsiddi Jan 21, 2025

OjusWiZard Jan 20, 2025

Mohsinsiddi Jan 21, 2025

OjusWiZard Jan 20, 2025

Mohsinsiddi Jan 21, 2025

Mohsinsiddi Jan 21, 2025

OjusWiZard Jan 20, 2025

Mohsinsiddi Jan 21, 2025

		@@ -561,7 +570,7 @@ def get_config_files():
		logger.info(f"Found config files: {[f.name for f in config_files]}")

		# TODO: Support the test for memeooorr and mech

	logger.info("Bypassing health check for mech service")
	logger.info("Bypassing health check for mech service because it's not supported")

	r"Press enter to continue": "\n",
	r"Press enter to continue": "\n",
	r"Please enter .*": lambda output, logger: handle_env_var_prompt(output, logger, config_path)

updated test script for better handling of please enter based prompts… #22

updated test script for better handling of please enter based prompts… #22

Conversation

Mohsinsiddi commented Jan 17, 2025

Description

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment