How can I run busted tests in a while loop until a fastest time is found?

[ColinKennedy]

I'd like to use busted to profile unittests. But variation can cause tests to perform differently across runs (for example a cold cache vs a warm cache).

What I'd like to do is be able to run busted in a while loop that goes something like this

local maximum_tries = 10
local counter = 10
local fastest_time = 2^1023

while true do
    local before = os.clock()
   
    run_busted_suite()  -- The lua equivalent of this terminal call `busted --helper spec/minimal_init.lua --output=my_cool_profiler .`

    local duration = os.clock() - before

    if duration < fastest_time then
        counter = maximum_tries
        fastest_time = duration
    else
        counter = counter - 1
    end

    if counter == 0 then
        break
    end
end

In the above example, a run that is the fastest of 10 consecutive runs is considered "probably the best time we're going to get". And then I'd use the profile results of that fastest run.

How can I achieve that easily with busted? I checked around it seems like busted isn't like other testing frameworks where you can call the test suite runner directly with lua.

I tried a real example using this:

local function _clear_arg()
    for key, _ in pairs(arg) do
        if key ~= 0 then
            arg[key] = nil
        end
    end
end

local function _keep_arg(caller)
    local original = vim.deepcopy(arg)

    caller()

    for key, value in pairs(original) do
        arg[key] = value
    end
end

local function _run_busted_suite(runner)
    _keep_arg(function()
        _clear_arg()

        arg[1] = "--ignore-lua"
        arg[2] = "--helper=spec/minimal_init.lua"
        arg[3] = "--output=busted.profile_using_flamegraph"

        runner({ standalone=false })
    end)
end

local function main()
    local maximum_tries = 10
    local counter = 10
    local fastest_time = 2^1023

    while true do
        print("running")
        local before = os.clock()

        local runner = require("busted.runner")
        _run_busted_suite(runner)
        -- NOTE: It looks like for some reason busted forces `runner()` to
        -- return an empty table if it is called more than once. Which is
        -- weird. So we have to force-remove the module so we can load it from
        -- scratch again.
        --
        package.loaded["busted.runner"] = nil

        local duration = os.clock() - before

        if duration < fastest_time then
            counter = maximum_tries
            fastest_time = duration
        else
            counter = counter - 1
        end

        if counter == 0 then
            break
        end
    end
end

main()

Because runner takes a combination of arg and options, the interface for this gets hacky. And then there's this
if loaded then return function() end else loaded = true end that prevents me from calling the runner more than once. I tried to get around it by forcing the file to reload with package.loaded["busted.runner"] = nil but it isn't working just yet.

Anyway I'm struggling to achieve the effect I'm looking for. Any advice would be appreciated. Maybe I'm just looking in the wrong place and there's an easy way to do this?