[ENH] Add fine tuning methods using PEFT for HFTransformersForecaster

[geetu040]

To extend PEFT fine tuning methods in the existing huggingface interface, we need to wrap the model initialization with peft configuration.
Hypothetically, this can be done using this piece of code

from peft import LoraConfig, TaskType, get_peft_model
peft_config = LoraConfig(task_type=TaskType.SEQ_2_SEQ_LM, inference_mode=False, r=8, lora_alpha=32, lora_dropout=0.1)
self.model = get_peft_model(self.model, peft_config)

Although the same technique can be used to wrap the huggingface model implemented in HFTransformersForecaster, yet fine-tuning this wrapped model results in different errors that are either because of the nature of these Time Series transformers that are different from typical Large Language Models on which these peft techniques are originally used, or because the right parameters and configurations are not set.
If we apply this PEFT wrapper right before the model is passed for training in HFTransformersForecaster, like this

from peft import LoraConfig, TaskType, get_peft_model
peft_config = LoraConfig(task_type=TaskType.SEQ_2_SEQ_LM, target_modules=["v_proj"], inference_mode=False, r=8, lora_alpha=32, lora_dropout=0.1)
self.model = get_peft_model(self.model, peft_config)
# TypeError: AutoformerForPrediction.forward() got an unexpected keyword argument 'input_ids'

[geetu040]

FYI @benHeid , this is something maybe you can help with?

[benHeid]

@geetu040 I think the main issue is that you set the task_type. If you not set it than it should work.

[geetu040]

yes @benHeid that seems to solve the problem, Thanks