Bisect bad plugin

[AckslD]

Sometimes I have some issue in my config and I'm not completely sure what caused it. So I configured by setup and wrote a python script such that I could do a binary search of my plugins to find the culprit. Thought I would share the code here in case it's useful for anyone else. It's quite rough though so you might need to configure it to what you need

The way I used it was to run the python script below as eg:

bisect_lazy bisect <args>

where <args> are any args to nvim, such as a filepath to open.
It will then start a binary search for the bad plugin by opening nvim (using kitty in my case) and after you close it ask you if it was good or bad. Quite similar to git bisect.

There is also bisect_lazy single_remove (remove a single plugin one at a time), bisect_lazy single_keep (only keep one plugin at a time) and bisect_lazy random (use random subsets of the plugins).

For this to work you need something like the following when setting up lazy:

local lazypath = vim.fn.stdpath("data") .. "/lazy/lazy.nvim"
if not vim.loop.fs_stat(lazypath) then
  vim.fn.system({
    "git",
    "clone",
    "--filter=blob:none",
    "https://github.com/folke/lazy.nvim.git",
    "--branch=stable", -- latest stable release
    lazypath,
  })
end
vim.opt.rtp:prepend(lazypath)


local get_default_modules = function(ignores)
  -- TODO whatever way you usually generate your specs, can be just a static table you return
  ...
end

-- TODO update path to where you want
local bisect_path = vim.fs.normalize('some/path/bisect_lazy')

local handle_bisect = function(ignores)
  local bisect_in_path = bisect_path .. '/input.json'
  local bisect_out_path = bisect_path .. '/output.json'
  if vim.fn.filereadable(bisect_in_path) ~= 0 then
    local bisect_in = vim.fn.json_decode(vim.fn.readfile(bisect_in_path))
    if bisect_in.query == 'list' then
      local modules = get_default_modules(ignores)
      vim.fn.writefile(vim.fn.split(vim.fn.json_encode(modules), '\n'), bisect_out_path)
    elseif bisect_in.query == 'use_plugins' then
      return bisect_in.plugins
    end
  end
end

local make_plugins = function()
  local modules = handle_bisect() or get_default_modules()
  local specs = {}
  for _, module in ipairs(modules) do
    local spec = R('plugins.'..module)
    table.insert(specs, spec)
  end
  return specs
end

require("lazy").setup(make_plugins(), {})

and you can then run the following:

#!/usr/bin/python
import sys
import json
import random
from pathlib import Path
from subprocess import run
from contextlib import contextmanager

# TODO update path
BISECT_PATH = Path() / 'some' / 'path' / 'bisect_lazy'
BISECT_IN_PATH = BISECT_PATH / 'input.json'
BISECT_OUT_PATH = BISECT_PATH / 'output.json'


def bisect(*args):
    plugins = ask_current_plugins()
    print(f'default {len(plugins)}')
    print('trying to search for the bad plugin with binary search, let me know which half of plugins is good')
    start = 0
    end = len(plugins)
    while start < end:
        middle = (start + end) // 2
        print(f'try with plugins {start} until {middle - 1}\n{plugins[start:middle]}')
        left_good = check_if_good(plugins[start:middle], *args)
        print(f'try with plugins {middle} until {end - 1}:\n{plugins[middle:end]}')
        right_good = check_if_good(plugins[middle:end], *args)
        if left_good:
            if right_good:
                print('both halves are good, seems to not be the cause of a single plugin, quitting')
                return
            else:
                start = middle
        else:
            if right_good:
                end = middle
            else:
                print('both halves are bad, seems to not be the cause of a single plugin, quitting')


def single_remove(*args):
    plugins = ask_current_plugins()
    print(f'default {len(plugins)}')
    print('trying to remove each plugin one by one, let me know if any makes the setup good again')
    for i in range(len(plugins)):
        print(f'{i}: try without "{plugins[i]}"')
        if check_if_good(plugins[:i] + plugins[i + 1:], *args):
            print(f'seems "{plugins[i]}" was the issue, quitting')
            return


def single_keep(*args):
    plugins = ask_current_plugins()
    print(f'default {len(plugins)}')
    print('trying with only one plugin at a time, let me know if any makes the setup bad on its own again')
    for i in range(len(plugins)):
        print(f'{i}: try with only "{plugins[i]}"')
        if not check_if_good([plugins[i]], *args, default='y'):
            print(f'seems "{plugins[i]}" was the issue, quitting')
            return


def random_subsets(*args):
    plugins = ask_current_plugins()
    print(f'default {len(plugins)}')
    print('trying random subsets of plugins, let me know if any are good (quit with ctrl-c)')
    while True:
        size = random.randint(1, len(plugins) - 1)
        subset = random.choices(plugins, k=size)
        print(f'try with plugins "{subset}"')
        if check_if_good(subset, *args):
            print('found a good subset, quitting')
            return


def check_if_good(plugins, *args, default='n'):
    start_with_plugins(plugins, *args)
    return ask_if_good(default)


def ask_if_good(default='n'):
    answer = input(f'good? (y/n, default {default})')
    if answer == '':
        return default
    return answer == 'y'


def ask_current_plugins():
    query({'query': 'list'}, headless=True)
    return read()


def start_with_plugins(plugins, *args):
    query({'query': 'use_plugins', 'plugins': plugins}, *args)


def query(data, *args, **kwargs):
    with write(data):
        run_nvim(*args, **kwargs)


@contextmanager
def write(data):
    BISECT_IN_PATH.parent.mkdir(parents=True, exist_ok=True)
    BISECT_IN_PATH.touch(exist_ok=True)
    BISECT_IN_PATH.write_text(json.dumps(data))
    yield
    BISECT_IN_PATH.unlink()


def read():
    return json.loads(BISECT_OUT_PATH.read_text())


def run_nvim(*args, headless: bool = False):
    if headless:
        cmd = 'nvim --headless +q'
    else:
        cmd = 'kitty nvim'
    run(' '.join([cmd, *args]), shell=True)


MODES = {
    'bisect': bisect,
    'single_keep': single_keep,
    'single_remove': single_remove,
    'random': random_subsets,
}


if __name__ == '__main__':
    mode = sys.argv[1]
    if mode not in MODES:
        raise ValueError(f'unknown mode "{mode}"')
    MODES[mode](*sys.argv[2:])

Happy bug hunting! :)

[AckslD]

In my case I tried to figure out what made some highlighting weird so I couldn't do it programatically but of course you can also extend this to have the querying be done whether some command fails or not, much like git bisect run.

[luiz00martins]

@AckslD, in the lines:

  for _, module in ipairs(modules) do
    local spec = R('plugins.'..module)
    table.insert(specs, spec)
  end

What's the function R?
(Haven't found anything in the help files)

[AckslD]

@luiz00martins oh right that snuck in there, it's just my function to require my own modules:

R = function(module)
    modules[module] = true
    return require('acksld.'..module)
end

which also keeps track of them such that I can reload them easily.

[abeldekat]

Hello @AckslD,

Thanks for the script! I wrote lazyflex.nvim. Lazyflex.nvim also aims to facilitate bisecting a bad plugin.
I would be interested in your opinion.

Best regards!

[AckslD]

Hi @abeldekat, thanks for sharing! Looking at the README I don't see anything mentioning how to bisect?

[abeldekat]

Thanks for the reply!

Looking at the README I don't see anything mentioning how to bisect?

That's correct. The plugin does not offer an automated solution like git-bisect, or your script. I did not use the word bisect in the readme in order to avoid confusion.

The user, knowing his config, is expected to manually select plugins by a few well-chosen keywords. This works especially well when the user implements presets, a set of keywords for plugins representing distinctive parts of the configuration. LazyVim, for example, has "coding", "editor, "lsp", "treesitter", and "ui".

I think the most important part of :h bisect is supported: Start with half of the config, and continue from there... When the user suspects some part of his config to be the culprit, that thought can be immediately confirmed by only disabling that specific part.