LightningCLI - instantiate model from config

[i008]

So lets say i have a model and i'm using the newest CLI API to train it: The config uses sub modules and can look something like:

# config.yaml
model:
  class_path: pl_models_2.ModelPL
  init_args:
    margin: 0.3
    basemodel:
      class_path: pl_models_2.ModelBackbone
      init_args:
        base_model: resnet50
        pooling: both

data:
  batch_size: 32
  image_size: 224
  augmentation_strategy: medium2

Now i want to run some scripts or notebooks using the model i trained with this config and i would like to instantiate the model using this config file.
f/e. model = load_model(config.yaml)

I was digging how this happens in LightningCLI and jsonargsparse but gave up after a while.. and started trying to hack this around with importlib, which works but feels like reinventing the wheel - i mean this logic has to be somewhere already :) i just cant find it.

[tchaton]

@mauvilsa ?

[mauvilsa]

If you want to use that exact config (not stripping out everything except the model) you can do the following:

from jsonargparse import ArgumentParser

parser = ArgumentParser()
parser.add_argument('--model', type=ModelClass)
parser.add_argument('--data', type=dict) # to ignore data
config = parser.parse_path('config.yaml')
config_init = parser.instantiate_classes(config)

The instantiated model will be in config_init.model.

[cifkao]

@mauvilsa What I would like to do is setup the parser with all the correct parameters but without actually parsing any command line arguments or instantiating any classes. That would allow me to load a custom configuration via jsonargparse's parse_string() or parse_object() and/or tweak the configuration before instantiating anything.

run=False apparently prevents the subcommand parsers from being initialized, which then causes errors if the config file contains keys defined by these subcommands (which is the case for config files that are written by Lightning during training).

After much trial and error, I ended up doing something like this:

with open(CONFIG_PATH, "rb") as f:
    config_dict = yaml.load(f, Loader=yaml.SafeLoader)
config_dict["trainer"]["logger"] = False
del config_dict["seed_everything"], config_dict["ckpt_path"]

parser = LightningArgumentParser()
parser.add_class_arguments(MyModel, "model")
parser.add_class_arguments(MyDataModule, "data")
parser.add_class_arguments(Trainer, "trainer")
MyCLI.add_arguments_to_parser(None, parser)
config = parser.parse_object(config_dict)
objects = parser.instantiate_classes(config)

[mauvilsa]

@cifkao maybe you should create a new issue or discussion explaining better what you want. Mentioning parse_string() and parse_object() is okay. But better to know more about the use case to understand why these are needed. I am also a bit unsure about the code snippet. Deleting seed_everything or ckpt_path seems more like something to do by overriding via command line arguments,e.g. cli.py fit --config=path/to/config.yaml --seed_everything=null --ckpt_path=null.

the config file contains keys defined by these subcommands (which is the case for config files that are written by Lightning during training).

This shouldn't be the case. What version of pytorch-lightning are you using? This issue was fixed in pull request #11532.

[cifkao]

the config file contains keys defined by these subcommands (which is the case for config files that are written by Lightning during training).

This shouldn't be the case. What version of pytorch-lightning are you using? This issue was fixed in pull request #11532.

@mauvilsa I'm using version 1.5.10. If I remember correctly, my issue was that ckpt_path and seed_everything was not accepted by the parser (Configuration check failed :: No action for destination key "ckpt_path" to check its value.), apparently because the subcommands were not initialized. This is why I'm deleting these args. Perhaps I should indeed create an issue for this.

[cifkao]

@mauvilsa I now understand a bit better what run=False is supposed to do, and based on that it seems that it's not well suited for loading a model in a notebook since as you said in #12302, it basically defines a different CLI (and this is a feature, not a bug).

What method would you then recommend for loading a model in a notebook in a way that is good practice in terms of reproducibility?

[mauvilsa]

@cifkao my proposal for this is #18105.

[Jackiexiao]

@cifkao @mauvilsa
why can't we use

from pytorch_lightning.cli import LightningCLI
if __name__ == "__main__":
    cli = LightningCLI(run=False)
    model = cli.model

but load_from_checkpoint not working

    cli.model.load_from_checkpoint(ckpt_path)

it raise

ModelPL.__init__() missing 1 required positional arguments: 'basemodel', 

[mauvilsa]

The purpose of LightningCLI is to help in implementing configurable CLIs. When used it will try to get the configuration (sys.argv, env vars, etc.) and then always instantiate multiple classes (trainer, model). Simply, that class is not designed for only loading a model from a checkpoint, and doesn't really make sense to extend it such that it does.

Some time ago I created pull request #18105 with a proposal to allow loading from the checkpoint a model trained with LightningCLI. You are free to look at that pull request and comment.

[Jackiexiao]

thx for your reply!

[adosar]

@mauvilsa Is there anywhere documented how to load a model from a checkpoint as proposed in #18105 ?

[mauvilsa]

#18105 was merged on February 23rd and there have only been lightning patch releases since then. This is a new feature, so not part of any lightning release yet. You will have to wait for v2.3.

Regarding docs, you can see the changes in the diff of 18105. It is just using save_hyperparameters and then load_from_checkpoint. Though, that this is only for newly trained models.