RuntimeError: Invalid pretrained config, cannot load weights.

[tracek]

Hi! I am following your README to test the pre-trained model on the ClothoV2 benchmark, and running into the following error:

RuntimeError: Invalid pretrained config, cannot load weights. Use `pretrained=False` for random init.

Any clues as for what could be the issue? I have slightly different environment than yours (modules' versions + Ubuntu 22.04), but I don't think it should matter.

CUDA_VISIBLE_DEVICES=0 python -m experiments.ex_dcase24 cmd_test_on_clothov2 with \
data_loader.batch_size_eval=32 \
audio_features.segment_length=10 \
audio_features.model=passt \
sentence_features.model=roberta-large \
load_model=passt_roberta.ckpt
wandb: Using wandb-core as the SDK backend.  Please refer to https://wandb.me/wandb-core for more information.
wandb: Currently logged in as: lukasz-tracewski (lukasz-tracewski-datana-solutions). Use `wandb login --relogin` to force relogin
__main__ is running pid 36592 in module main:  __main__
WARNING - root - Added new config entry: "load_model"
WARNING - audio_retrieval - No observers have been added to this run
INFO - audio_retrieval - Running command 'cmd_test_on_clothov2'
INFO - audio_retrieval - Started
Initialize model...
passt_roberta.ckpt
/home/tracek/micromamba/envs/salsa/lib/python3.12/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
/home/tracek/micromamba/envs/salsa/lib/python3.12/site-packages/timm/models/helpers.py:7: FutureWarning: Importing from timm.models.helpers is deprecated, please import via timm.models
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.models", FutureWarning)
#### Using PaSST model with no overlap ####


 Loading PaSST pre-trained on AudioSet Patch 16 stride 16 structured patchout mAP=472 


ERROR - audio_retrieval - Failed after 0:00:02!
Traceback (most recent calls WITHOUT Sacred internals):
  File "/home/tracek/Projects/salsa/experiments/ex_dcase24.py", line 1051, in cmd_test_on_clothov2
    model = AudioRetrievalModel.load_from_checkpoint(load_model, strict=False)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tracek/micromamba/envs/salsa/lib/python3.12/site-packages/pytorch_lightning/utilities/model_helpers.py", line 125, in wrapper
    return self.method(cls, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tracek/micromamba/envs/salsa/lib/python3.12/site-packages/pytorch_lightning/core/module.py", line 1582, in load_from_checkpoint
    loaded = _load_from_checkpoint(
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tracek/micromamba/envs/salsa/lib/python3.12/site-packages/pytorch_lightning/core/saving.py", line 91, in _load_from_checkpoint
    model = _load_state(cls, checkpoint, strict=strict, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tracek/micromamba/envs/salsa/lib/python3.12/site-packages/pytorch_lightning/core/saving.py", line 165, in _load_state
    obj = instantiator(cls, _cls_kwargs) if instantiator else cls(**_cls_kwargs)
                                                              ^^^^^^^^^^^^^^^^^^
  File "/home/tracek/Projects/salsa/experiments/ex_dcase24.py", line 299, in __init__
    self.audio_embedding_model, audio_output_size = get_audio_embedding_model(
                                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tracek/Projects/salsa/models/audio/base.py", line 12, in get_audio_embedding_model
    model, emb_dim = get_passt(name, **model_config)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tracek/Projects/salsa/models/audio/passt.py", line 23, in get_passt
    model = get_model_passt("passt_s_p16_s16_128_ap468", input_tdim=1000, fstride=16, tstride=16,
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tracek/Projects/salsa/models/audio/external/hear21passt/hear21passt/models/passt.py", line 955, in get_model
    model = model_func(pretrained=pretrained, num_classes=n_classes, in_chans=in_channels,
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tracek/Projects/salsa/models/audio/external/hear21passt/hear21passt/models/passt.py", line 768, in passt_s_p16_s16_128_ap468
    model = _create_vision_transformer(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tracek/Projects/salsa/models/audio/external/hear21passt/hear21passt/models/passt.py", line 694, in _create_vision_transformer
    model = build_model_with_cfg(
            ^^^^^^^^^^^^^^^^^^^^^
  File "/home/tracek/Projects/salsa/models/audio/external/hear21passt/hear21passt/models/helpers/vit_helpers.py", line 253, in build_model_with_cfg
    load_pretrained(
  File "/home/tracek/micromamba/envs/salsa/lib/python3.12/site-packages/timm/models/_builder.py", line 156, in load_pretrained
    raise RuntimeError("Invalid pretrained config, cannot load weights. Use `pretrained=False` for random init.")
RuntimeError: Invalid pretrained config, cannot load weights. Use `pretrained=False` for random init.

[tracek]

Looking a bit further, what we seem to be missing here when loading the model is the pretrained_cfg key. It does not seem to be in the model, nor it is passed as an arg, hence the problem (?).

    pretrained_cfg = pretrained_cfg or getattr(model, 'pretrained_cfg', None)
    if not pretrained_cfg:
        raise RuntimeError("Invalid pretrained config, cannot load weights. Use `pretrained=False` for random init.")

[chungminho1]

Did you solve the issue?
adding "pretrained=False" at models/audio/passt.py might work.

    model = get_model_passt("passt_s_p16_s16_128_ap468", pretrained=False, input_tdim=1000, fstride=16, tstride=16,
                            s_patchout_t=s_patchout_t, s_patchout_f=s_patchout_f)