Skip to content

TCL QT Test Harness

Jump to bottom
KochynVolodymyr edited this page Nov 28, 2023 · 2 revisions

Note

This is a draft testing concept, please point out any issues/improvements you can think of.

Existing Testing Solutions

Testing is a full time job that usually has a dedicated team, writing our own solution will divert a large portion of developer time to creating and maintaining our own testing framework. If possible, using an existing solution could lighten the work load.

Below are a few solutions that might work for us, but haven’t been explored in in terms of license availability or integration with our tool.

  • Squish is the defacto ui automation tool for Qt (Now owned by the Qt coporation), but our open source nature seems to keep us from using it. If using Squish is possible in any way, we should use it instead as our implementation will never be as thorough as their coverage with a dedicated team working on.

  • QtMonkey has a BSD-3 Clause License, haven’t had a chance to try it, but it might provide the basic functionality we are going for w/ our tcl/qt harness approach.

  • SikuliX has an MIT license and appears to be a desktop level automation tool that uses computer vision to drive a UI with mouse and keyboard automation. Haven’t tested this, but it might provide a generic solution to automated testing.

Purpose

The goal of the tcl/qt test harness is to provide a way to automate the testing of our gui. Testing is intended to be done via tcl scripts allowing a testing team to create new tests w/o having to have access to the ui code.

Basic Overview of Harness

Helper functions that can search/manipulate the QT UI are contained within foedag. Tcl helper functions, also contained within foedag, provide an interface to these QT functions. The Tcl helper functions should parse arguments etc, but they should seek to leave actual functionality in the QT functions to allow re-use in other commands.

Needs more consideration: Currently, there are a couple higher level helper functions that simplify the use of the tcl helper functions stored in foedag. These high level helper functions call existing tcl functions or themselves. As such, it’s easier if they are implemented in pure tcl (ie the tcl function is defined in a tcl file instead of our c/tcl interface). For example: we have a waitForWidget tcl command that calls a repeatOnError tcl function which calls a sleep tcl function etc. Implementing this on the gui side would probably complicate things needlessly.

  • Currently these helpers are stored in ./tests/Harness/ui_testing_cmds.tcl if we can assume that all tests are run from the sandbox root, then a test can load these with source tests/Harness/ui_testing_cmds.tcl if tests can be run from any location, then how we load these functions might need some additional thought.

Current Commands

QT

QWidgetList findWidgets(FindWidgetRequest request)

This is a pure qt helper function that will use the values set in the request struct to search the UI for a QWidget that matches the request parameters. The basic approach is to use the request.widgetType string to determine what type of widget we are looking for. findChildren is called for the respective type to find all children of that type and then then results are filtered based on additional parameters set in the request.

QWidget* widgetAddrToPtr(const QString& addrStr, bool& convertStatus)

Pure qt helper function to convert widget pointer strings returned by the tcl helpers stored in foedag (for example the original qt test function qt_getWidget) to actual pointers that can be used by qt.

void openMenu(QWidget* searchWidget, const QStringList& menuEntries)

This searches for a top level menu and then tries to traverse it with each entry in menuEntries. Ex: To open File > New Project… you’d pass menuEntries as { “File”, “New Project…” }. This function will navigate through child menus based off menuEntries and then execute the final menu target action. This should exercise the opened of each child menu incase the content is dynamically generated, but the menus are currently just placed at (0,0) of the widget so the menu placement won’t look like an actual run. This can be improved later if menu placement matters.

bool setText(QWidget* ptr, const QString& newText)

This function attempts to cast ptr to qwidget types and then calls whatever method makes sense for that widget to set text. This function will need to be updated whenever we want to support a new widget as not all widgets have a setText concept.

TCL stored in FOEDAG (these are currently installed into foedag via registerBasicGuiCommands)

Note: qt_clickButton and qt_isEnabled are good examples of simple helper functions that will need to be written as we require more control for specific scenarios.

qt_findWidgets

This is a wrapper function that parses user input to construct a FindWidgetRequest that will be passed to findWidgets. The results are then converted to a string and returned to the tcl script/console. Currently supports the following parameters -parent, -type, -text, -object_name, -nth, -match_case, -include_hidden

qt_clickButton

Takes a QWidget pointer string as returned by qt_findWidgets or qt_getWidget converts it to a QAbstractButton* and then programmatically clicks the button

qt_isEnabled

Takes a QWidget pointer string as returned by qt_findWidgets or qt_getWidget converts it to a QWidget* and then returns the result of QWidget::isEnabled()

qt_openMenu

Adds each argument to a QStringList which is passed to openMenu

qt_setText

Takes a QWidget pointer string as returned by qt_findWidgets or qt_getWidget converts it to a QWidget* and then returns the result of setText from the QT commands above

qt_queue_cmds

Takes a set of tcl commands and executes them via lambda after a delay. This is designed to allow a user to execute tcl commands that would normally be blocked once a modal dialog is raised. See the Example Tcl Script for usage.

Script Level TCL

The following functions are probably going to be sourced from a known location to provide a few pure tcl helper functions that would work better in pure tcl

sleep and uniqkey (only used by sleep) are community created functions to provide a more c like sleep function as opposed to TCL’s standard after format which leads to hard to predict script timing

repeatOnError This tries a given tcl command, if it fails it waits a provided (or default) amount of time, then retries the command. The number of retries can be provide as well. If the try count is exceeded and error is thrown. This function is required by waitForWidgets

waitForWidgets This is a helper function to reduce the reliance on sleep commands in the script. This will pass all arguments to qt_findWidgets and execute it via repeatOnError (you can replace any working call to qt_findWidgets with waitForWidgets without changing arguments). The idea here is that since a tcl script will usually try to execute its code faster than the UI can respond, most qt_findWidgets calls after a ui change will fail because tcl will try to introspect the widgets before it exists. By using repeatOnError, we will try to find the widget a few times before failing out of the test.

waitForWidgetsEx This is the same as waitForWidgets but the first two arguments (retries, delay) allow the user to specify how many times to retry and how long to wait between attempts.

Example TCL script

# Load our helper commands
source tests/Harness/ui_testing_cmds.tcl

# If run via --replay, add gui_start to load the UI
#gui_start

# For generic widgets that don't have an objectName, qt_findWidgets allows you to query for the widget
# Here we are getting the New file button on the toolbar which doesn't have an objectName
# Note that qt_findWidgets automatically removes & from qpushbuttons and qactions which might have a format like &New
set newBtn [qt_findWidgets -type button -text "New"]
# at this point, newBtn will contain some pointer string like "QWidget(0x12345678)"
# now we can pass that "pointer" to qt_clickButton. (Not going to call this currently so following steps can be run)
# qt_clickButton $newBtn

# For generic, duplicate widgets, the -nth argument can be used
# This will find the first instance of "Edit Settings..." in the task window
set settings [waitForWidget -type button -text "Edit settings..." -nth 0]

# When opening a modal dialog, you need to queue the commands you want to execute while the ui blocks tcl execution
set cmds {
    # As more widgets in our UI have their objectName set, queries for the widget can be made more accurate
    # Note: Dialog's are top level widgets that require their own special search, as such, -type dialog is always required when looking for a qdialog
    set dlg [qt_findWidgets -type dialog -objectName tasksDlg_Synthesis__SettingsDialog]
    # Generic widgets like an cancel button can use the -parent argument to limit the search scope
    set cancel [qt_findWidgets -parent $dlg -type button -text cancel]
    qt_clickButton $cancel
}
# this will create a lambda with our queued commands and fire it after 1000ms
# this allows us to execute our tcl commands even though a blocking dialog is invoked in the next step
qt_queue_cmds 1000 $cmds

# open modal dialog
qt_clickButton $settings

# Not everything waits for widgets currently and you might run into a scenario where you need so delay for the test to pass
# adding sleep X gives the ui a bit more time to finish its last operation before executing the next steps
sleep 1000

# Menus can be traverse by providing the text of each menu entry and the subsequent child entry etc
# Note: in tcl, to pass a whitespace separated string as 1 argument, wrap it in {}
qt_openMenu File {Recent Projects} noname

Testing Harness Implementation Status

What is currently provided is a basic prototype to get an idea for how this testing could be done and to show how we can grow out the functionality as we need. The following topics/required features came to mind while writing this and we’ll need to address this moving forward.

611incompleteError handling, most of these helpers should cause the test to fail if some unexpected happens like not being able to find a widget etc592completeFinding Widget types (Partial widget coverage currently)

  • findWidgets should handle finding all widgets, something generic like searching by objectName can find most widgets w/o a type specified, but some widgets, like QDialog, require a special search method so findWidgets needs a case for QDialogs. When searching by text, you need to specify the widget type you want to find so that the code can find that object and looked up the “text” relative to that widget since not every QWidget supports text(). Special scenarios like these will be discovered as we try to write tests.

  • qt_findWidgets currently has a bug where it doesn’t error out on extra parameters

    • Ex: waitForWidget -parent $dirDlg -type button "blah" “blah” should be recognized as an unhandled param because the user forget -text.

593complete Click Buttons594incomplete Click Tabs595incompleteSelect item in a radio button group (might already be doable searching for buttons, but a helper function might be good)596incompleteSet checkbox value597incompleteSet spinbox value

  • Can be set via text entry or the up/down arrows, might need to test both to cover all logic

598completeOpen/traverse Menus to select a given option599incompleteSelect item in tree/listwidget etc600incompleteSelect cell in table601incompleteSelect entry in dropdown/combox

  • should support selecting via index or matching text

602incompleteSelecting text603completeEnter text in qlineedit

  • qt_set_text has support for qlineedit, other widgets will need to be added

604incompleteR-click item

  • Might by able to use openMenu once r-click has been fired

605incompleteDialog control622incompleteClose dialog623completeNon-modal dialogs624incompleteModal dialogs (these block tcl test execution, see Modality under Potential Issues)606incompleteTcl Testing Timing / Robustness607completesleep608completewaitForWidget609completerepeatOnError610incompleteGlobal timeout for hung tests (gui changes or broken tests might leave the UI in an unresponsive state. We need a way for individual tests to automatically fail after X minutes so that our github CI doesn’t hang on test failure)612incompleteScreenshots (done the road feature, but eventually we might want to screencap widgets for visual verification. Also screenshots on failure can provide the test writer w/ a hint to why things are failing, assuming they can access the image easily)613incompleteWidget introspection. Need a way to get data from a widget for verification purposes614completetext615incompletefindWidgets should be extended for all widget types to take their equivalent of text (not all widgets have a text() function) and store it in qt property “qt_test_Text” which can be generically compared against when filtering results.616completeisEnabled()617incompleteisVisible(), findWidgets has a filter option for visibility, but we don’t currently have a helper function to introspect the value618incompleteVerification helpers619incompleteHelper functions/asserts to error out of tcl scripts w/ proper error codes when a value isn’t correct620incompleteHelper function to grep a file for results621incompleteNeed to decide how these tests should be invoked, see Test Invocation in Potential Issues below.

Potential Issues

  • Modality appears to block the tcl test execution

    • waitOnWidget -type dialog -name "Open File" hangs on the Open File Dialog because it's opened modally. Non-modal dialogs seem to work, but we need some alternate solution for modal dialogs, maybe pre script the actions and have qt execute them all?

    • Did some testing with this, we can continue to interact with the UI if we use QTimer::singleShot with a lambda that contains all the UI control actions we want to do while the dialog is up.

    • Need to come up with a way to pre-queue desired dialog actions that we can then fire w/ singleShot while the modal dialog is blocking.

  • Haven’t tested, but I wonder if a tcl script calling some ui action that then calls another tcl command will cause any issue with the interpreter loop

  • Test Invocation

    • --script

    • ties its run status to the stop button which means we can’t introspect or interact w/ the start/stop buttons for verification purposes.

    • gui takes error messages so scripts need to be wrapped in a try/catch to capture errors

    • --replay

    • Need to call gui_start for the ui to fully loaded

    • dies when parsing a tcl file that has functions defined (because of new lines?)

    • We can partially avoid this issue by sourcing our functions from a different file

    • This will cause problems for modal tests where we need to define a command block

    • might need new arg to specify a test file to run if the above options won’t work

  • As the number of gui tests grow, we will see a considerable delay in CI run time

    • Might need a set of smoke tests to reduce test suite

    • Might need a way of parallelizing test runs to speed up tests

    • If gui tests prove to be too slow, might need to offload the testing to a scheduled cron job that reports failures to the developer

    • This is risky as the code is verified after merge

    • Gets complicated when multiple commits are being tested as you’ll need to figure out which commit caused the regression.

Clone this wiki locally