Use DoRA as diff to standardize with sdwebui #3298
Closed
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There is a discrepancy in the implementation of DoRA between sdwebui and comfyui.
sdwebui: Implements DoRA as a weight difference.
comfyui: Directly modifies the weight.
The DoRA paper does not include the strength. Setting lora strength to zero in comfyui modifies the output image which is unexpected.
Testing the two approaches in https://github.com/catid/dora results in the same final evaluation loss.
Testing:
Resources:
sdwebui diff code: https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/adadb4e3c7382bf3e4f7519126cd6c70f4f8557b/extensions-builtin/Lora/network.py#L189
DoRA trained with sdwebui diff approach: https://civitai.com/models/397590/looking-away-to-looking-at-viewer-slidersntcaixyz?modelVersionId=443385