[WIP] Hugging Face Integration #531

dcwil · 2023-09-11T12:37:36Z

#480
Whats been done:

Set up basic hugging face workflow (vaguely following the torchvision style)
uploaded some pretrained nets to my personal repo (ShallowFBCSP, BNCI2014001, same params except 200 epochs)
Extended trialwise decoding example to demonstrate the pretrained network

What would likely need to be added:

more models/datasets (see below)
a specific example for using pretrained networks
create (and use) braindecode HF repo
allow users to upload networks trained with braindecode to their own HF repo?
initialize_model variant or toggle for initialising a skorch classifier?

To move forwards need to know if:

we're happy with how the weights will be loaded (from a user perspective, and also internally)
exactly which models should be run, on what datasets and with which parameters etc (Do we need to do a hyperparameter exploration first?)

PierreGtch · 2023-09-11T16:47:30Z

braindecode/weights.py

+
+
+@dataclass
+class Weights:


Can you put a reference to torchvision in the docstring of this class to say we mimic their behavior?

PierreGtch · 2023-09-11T16:48:19Z

braindecode/weights.py

+
+
+@dataclass
+class Weights:


Can you put a reference to torchvision in the docstring of this class to say we mimic their behavior?

PierreGtch · 2023-09-11T16:48:24Z

braindecode/weights.py

+    path: str
+
+
+class WeightsEnum(Enum):


PierreGtch · 2023-09-11T16:58:35Z

braindecode/pretrained.py

+    Initialize and return a model specified by the `name` parameter. If `dataset_name` and
+    `subject_id` are provided, pretrained weights associated with those parameters will be downloaded
+    and used for initialization; otherwise, random initialization will be performed.


Not sure I like this. This forces models to be trained on one subject only. Loading pre-trained weights will be especially useful for general models that were trained on large/multiple datasets. And even on the same dataset, we can provide multiple pre-trained weights that were obtained with different training algorithms.

I think the weights name should not have constraints.

We could say in the case of passing a dataset name str, (or list of strs for multiple datasets) and subject_id=None implies you want to download a model that has been trained for all subjects on those given datasets?

For different training algorithms we could add another variable. It does then start to get slightly messy however, and I guess the alternative is we force the user to type the pretrained model name explicitly e.g:
model = initialize_model(name='ShallowFBCSPNet', weights=ShallowFBCSPNet['SomeDataset_S1_V2'])

I think you should just replace the parameters subject_id and dataset by weights_id and provide naming guidelines for weights_id with a few examples.
@bruAristimunha what do you think?

I agree - best would be to define convention but not restrict people to define model on subject_is anddataset

I agree too

PierreGtch · 2023-09-11T17:04:05Z

braindecode/pretrained.py

+MODELS_AND_WEIGHTS = {
+    "shallowfbcspnet": {"model": ShallowFBCSPNet, "weights": ShallowFBCSPNetWeights}
+    # Other models go here
+}


maybe we can follow the convention described in #524 and put that dict in models.utils

Also, why do you put the model name in lower case?

Oh yeah, lower case to make it easier to type the correct model name (the name gets .lower() in initialize_model), but we can probably trust the user to do this correctly.

PierreGtch · 2023-09-11T17:06:12Z

examples/model_building/plot_bcic_iv_2a_moabb_trial.py

+model = initialize_model('ShallowFBCSPNet', dataset_name=dataset_name, subject_id=subject_id)
+
+clf = EEGClassifier(
+    model,
+    device=device,
+)


We should find a way to be able to do directly something like this:

Suggested change

model = initialize_model('ShallowFBCSPNet', dataset_name=dataset_name, subject_id=subject_id)

clf = EEGClassifier(

model,

device=device,

)

clf = EEGClassifier.from_pretrained(

ShallowFBCSPNet,

device=device,

weights_name='...',

)

# downloads the model args,

# instanciates the EEGClassigier with these args,

# initializes the EEGClassifier (including the model),

# and loads the pre-trained weights in the model.

yeah we can also check other libraries what API they use for this, e.g. does skorch have some way already? Also would we even need separate from_pretrained parameter? Torchvision for example has weights as another parameter in the constructor at https://pytorch.org/vision/stable/models.html

…raindecode#524

sliwy · 2023-09-11T21:13:00Z

No clue if it should be done in this PR but if we plan to share models and load them easy into python maybe we should also think about a better format than .pkl (for example safer). Maybe safetensors https://github.com/huggingface/safetensors#yet-another-format-?

PierreGtch · 2023-09-12T06:49:32Z

@sliwy I agree but probably for another PR

bruAristimunha · 2023-09-26T12:10:42Z

Hey @dcwil, do you need some help with the new extra requirement?

# Conflicts: # braindecode/models/shallow_fbcsp.py

codecov · 2023-09-27T09:39:15Z

Codecov Report

Merging #531 (c85b47f) into master (9cdde83) will decrease coverage by 1.28%.
Report is 4 commits behind head on master.
The diff coverage is 62.90%.

@@            Coverage Diff             @@
##           master     #531      +/-   ##
==========================================
- Coverage   84.72%   83.45%   -1.28%     
==========================================
  Files          63       65       +2     
  Lines        4741     4878     +137     
==========================================
+ Hits         4017     4071      +54     
- Misses        724      807      +83

bruAristimunha · 2023-09-27T16:56:47Z

Hi @dcwil,

In the CI, it seems that the generation of documentation is failing due to something related to the new functionality.

dcwil · 2023-09-29T09:33:39Z

Hi @dcwil,

In the CI, it seems that the generation of documentation is failing due to something related to the new functionality.

I haven't had the chance to look into it very deeply yet, but it could be that the weights I uploaded in Paris are now out of date with the code somehow?

robintibor · 2023-11-06T14:29:38Z

Do you have any time to look into this @dcwil ?

dcwil · 2023-11-06T15:24:31Z

Do you have any time to look into this @dcwil ?

Not right now I'm afraid - it's on my to-do list but has been superseded by a couple of other things

dcwil added 4 commits September 11, 2023 10:29

basic setup for hf

bdf70c5

added weight fetching via dataset name and subject id

ad6632b

add rest of subjects

7394a2c

basic hugging face integration

c24afd3

dcwil added the wip label Sep 11, 2023

dcwil requested review from robintibor, gemeinl, PierreGtch and bruAristimunha September 11, 2023 12:37

dcwil self-assigned this Sep 11, 2023

PierreGtch reviewed Sep 11, 2023

View reviewed changes

dcwil added 2 commits September 11, 2023 19:48

add torchvision link to docstrings

81108d2

create weights_dict and switch model loading convention to match PR b…

6598e6c

…raindecode#524

bruAristimunha and others added 2 commits September 12, 2023 17:33

Merge branch 'master' into hugging_face

ce2112d

switch to weight_id from dataset_name and subject_id

d421377

sliwy mentioned this pull request Sep 12, 2023

Decide on format to be used to share models #536

Open

dcwil added 5 commits September 26, 2023 20:04

fix example

817af1a

Merge remote-tracking branch 'origin/master' into hugging_face

adaed9b

# Conflicts: # braindecode/models/shallow_fbcsp.py

add huggingface_hub to requirements

1c32763

add huggingface_hub to requirements

dc4f937

add huggingface_hub to requirements

29662ac

Remove suffix Weights from weights dict keys

12b10d3

fix bug in weights id

c85b47f

robintibor mentioned this pull request Nov 6, 2023

release 0.8 #555

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Hugging Face Integration #531

[WIP] Hugging Face Integration #531

dcwil commented Sep 11, 2023

PierreGtch Sep 11, 2023

PierreGtch Sep 11, 2023

PierreGtch Sep 11, 2023

PierreGtch Sep 11, 2023

dcwil Sep 11, 2023

PierreGtch Sep 11, 2023

sliwy Sep 11, 2023

bruAristimunha Sep 12, 2023

PierreGtch Sep 11, 2023

dcwil Sep 11, 2023

PierreGtch Sep 11, 2023 •

edited

robintibor Sep 12, 2023

sliwy commented Sep 11, 2023

PierreGtch commented Sep 12, 2023

bruAristimunha commented Sep 26, 2023

codecov bot commented Sep 27, 2023 •

edited

bruAristimunha commented Sep 27, 2023

dcwil commented Sep 29, 2023

robintibor commented Nov 6, 2023

dcwil commented Nov 6, 2023

-model = initialize_model('ShallowFBCSPNet', dataset_name=dataset_name, subject_id=subject_id)
-clf = EEGClassifier(
-    model,
-    device=device,
-)
+clf = EEGClassifier.from_pretrained(
+    ShallowFBCSPNet,
+    device=device,
+    weights_name='...',
+)
+# downloads the model args,
+# instanciates the EEGClassigier with these args,
+# initializes the EEGClassifier (including the model),
+# and loads the pre-trained weights in the model.

[WIP] Hugging Face Integration #531

Are you sure you want to change the base?

[WIP] Hugging Face Integration #531

Conversation

dcwil commented Sep 11, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PierreGtch Sep 11, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sliwy commented Sep 11, 2023

PierreGtch commented Sep 12, 2023

bruAristimunha commented Sep 26, 2023

codecov bot commented Sep 27, 2023 • edited

Codecov Report

bruAristimunha commented Sep 27, 2023

dcwil commented Sep 29, 2023

robintibor commented Nov 6, 2023

dcwil commented Nov 6, 2023

PierreGtch Sep 11, 2023 •

edited

codecov bot commented Sep 27, 2023 •

edited