Add simplified model manager install API to InvocationContext

[lstein]

Summary

This adds two model manager-related methods to the InvocationContext uniform API. They are accessible via context.models.*:

1. load_and_cache_model(source: Path|str|AnyHttpURL, loader: Optional[Callable[[Path], Dict[str, Tensor]]] = None) -> LoadedModel

Load the model located at the indicated path, URL or repo_id.

This will download the model from the indicated location , cache it locally, and load it into the model manager RAM cache if needed. If the optional loader argument is provided, the loader will be invoked to load the model into memory. Otherwise the method will call safetensors.torch.load_file() or torch.load() (with a pickle scan) as appropriate to the file suffix. Diffusers models are supported via HuggingFace repo_ids.

Be aware that the LoadedModel object will have a config attribute of None.

Here is an example of usage:

def invoke(self, context: InvocatinContext) -> ImageOutput:
       model_url = 'https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.0/RealESRGAN_x4plus.pth'
       loadnet = context.models.load_and_cache_model(model_url)
       with loadnet as loadnet_model:
             upscaler = RealESRGAN(loadnet=loadnet_model,...)

---

2. download_and_cache_model( source: str | AnyHttpUrl, access_token: Optional[str] = None, timeout: Optional[int] = 0) -> Path

Download the model file located at source to the models cache and return its Path.

This will check models/.download_cache for the desired model file and download it from the indicated source if not already present. The local Path to the downloaded file is then returned.

Other Changes

This PR performs a migration, in which it renames models/.cache to models/.convert_cache, and migrates previously-downloaded ESRGAN, openpose, DepthAnything and Lama inpaint models from the models/core directory into models/.download_cache.

There are a number of legacy model files in models/core, such as GFPGAN, which are no longer used. This PR deletes them and tidies up the models/core directory.

Related Issues / Discussions

I have systematically replaced all the calls to download_with_progress_bar(). This function is no longer used elsewhere and has been removed.

QA Instructions

I have added unit tests for the three new calls. You may test that the load_and_cache_model() call is working by running the upscaler within the web app. On first try, you will see the model file being downloaded into the models .cache directory. On subsequent tries, the model will either load from RAM (if it hasn't been displaced) or will be loaded from the filesystem.

Merge Plan

Squash merge when approved.

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Tests added / updated (if applicable)
• Documentation added / updated (if applicable)

[lstein]

I have added a migration script that tidies up the models/core directory and removes unused models such as GFPGAN. In addition, I have renamed models/.cache to models/.convert_cache to distinguish it from the directory in which just-in-time models are downloaded into, which is models/.download_cache. While the size of models/.convert_cache is capped such that less-used models are cleared periodically, files in models/.download_cache are not removed unless the user does so manually.

[psychedelicious]

I'm not sure what I was expecting the implementation to be, but it definitely wasn't as simple as this - great work.

I've requested a few changes and there's one discussion item that I'd like to marinate on before we change the public invocation API.

[lstein]

@psychedelicious This is ready for your review now. There are now just two calls: load_and_cache_model() and download_and_cache_model() which return a locally cached Path and LoadedModel respectively. In addition, the model source can now be a URL, a local Path, or a repo_id. Support for the latter involved my refactoring the way that multifile downloads work.

[lstein]

@psychedelicious I just updated the whole thing to work properly with the new (and very nice) Pydantic-based events. I've also added a new migration. Please review when you can. I'm having to resolve merge conflicts fairly regularly!

[psychedelicious]

This is a notable API change. This means we cannot rely on a LoadedModel to have a config. Maybe this should be two separate classes, LoadedModelWithoutConfig and LoadedModel(LoadedModelWithoutConfig)...

[psychedelicious]

I removed a number of unnecessary changes in invocation_context.py, mostly extraneous type annotations. If mypy is complaining about these, then that's a mypy problem, because all the methods are annotated correctly.

I also moved load_model_from_url from the main model manager class into the invocation context.

[psychedelicious]

This function is unused - I think the logic to get the loader should be checking if it's a directory? I'm not sure how to fix this myself bc the diffusers_load_directory function has a different type signature than the other loader function options.

[psychedelicious]

Sorry for the delay in reviewing. I've tidied a few things and tested everything, working great!

Two minor issues noted.