Training a model with a custom FairseqDataset implementation

[sidharthrajaram]

❓ Questions and Help

Before asking:

• search the issues.
• search the docs.

What is your question?

I have extended FairseqDataset and created a custom dataset implementation. How do I use this custom implementation to train against some particular model architecture using fairseq-train?

What have you tried?

Read the docs regarding Tasks and have a feeling that's the direction to go. Still curious about the extent to which the CLI tool fairseq-train can be used.

What's your environment?

• fairseq Version (e.g., 1.0 or main): main
• PyTorch Version (e.g., 1.0): 2.2.2
• OS (e.g., Linux): Ubuntu 20.04
• How you installed fairseq (pip, source): source
• Build command you used (if compiling from source): pip installed from source as described in README.
• Python version: 3.10
• CUDA/cuDNN version: CUDA 11.8

[sidharthrajaram]

For anyone encountering the same issue, it probably varies quite a bit depending on your dataset, but I had to implement custom Task and FairseqDataset classes and then override setup_task() and load_dataset() behavior in my custom Task class.