Neutone Model for Cross-Synthesis with DDSP

neutone models that perform a form of cross-synthesis between an input audio signal and the output of a DDSP timbre transfer model. The main use case is to alter the timbre of vocals while preserving the lyrics, but of course any creative "misuse" is encouraged.

See here for a video demonstration.

Using the Neutone Cross-Synthesis Models

1. Install the neutone plugin.

2. Download the models from the neutone_models folder and load them into the neutone plugin.

The four custom knobs are:

A. Pitch shift of the input signal. This is useful to align the input signal with the timbre model. 0.5 (knob at 12o'clock) is no shift.

B. Amount of cross-synthesis. 0 only uses the input harmonics, 1 only uses the model harmonics. In between, the harmonics are mixed together.

C. Volume of the filtered noise generated by the DDSP model. At 0.5, the noise is at "default" volume. At 0, the noise is completely muted.

D. Formant shift of the input signal. This is an additional tool to alter the sound. 0.5 (knob at 12o'clock) is no shift.

Creating Your Own

The two models provided were trained on violin recordings and the Helm synthesizer. If you want a cross-synthesis model trained on your own audio, you can follow these steps:

1. Train a DDSP timbre transfer model. Either follow the instructions in the realtimeDDSP repository, or use the corresponding Colab Notebook.

2. If you're satisfied with the timbre model, download the checkpoint file.

3. Clone this repository and install the dependencies.

git clone https://github.com/dsuedholt/ddsp_xsynth
cd ddsp_xsynth
pip install -r realtimeDDSP/requirements.txt

4. Export the model checkpoint to a neutone model.

python export.py PATH_TO_CKPT EXPORT_NAME [--dataset_name="..." --author_name="..."]