Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Regexp Include filter ignored but identical filter without regexp [] works #4590

Open
arberg opened this issue Aug 12, 2021 · 1 comment · May be fixed by #5053
Open

Regexp Include filter ignored but identical filter without regexp [] works #4590

arberg opened this issue Aug 12, 2021 · 1 comment · May be fixed by #5053
Labels
filters

Comments

@arberg
Copy link

arberg commented Aug 12, 2021 •
edited

Environment info

Description

I found a special scenario where Duplicati fails to include by path when filter used as regexp:
--include="[/mnt/user/share/]"
and when identical used as non-regexp, it works:
--include="[/mnt/user/share/]"

I've included a zip with the data dir and two commands, one which fails, and one which works

It only seems to happen if a following exclude will exclude parent-folders of the source dir. However the first two filters ensures the include from the top of the source, and also there shouldn't be a difference between regexp and non-regexp as long as they both match. In other words I got configuration to work as I wanted by avoiding exclude [/.*/] and instead exclude from my source dir [/mnt/user/share/.*/]. I'm not certain whether this is coincidental or not.

Error-Command

docker exec duplicati mono /app/duplicati/Duplicati.CommandLine.exe test-filters  /mnt/user/media/TestDuplicati/ \
 --include="[/mnt/user/media/TestDuplicati/(cache|disk\d+|user)/]" \
 --include="[/mnt/user/media/TestDuplicati/(cache|disk\d+|user)/share/]" \
 --include="[/mnt/user/media/TestDuplicati/user/share/]" \
 --exclude="[/.*/]" \
 --include="[/.*]"

The second and third filter above should both have included user/share folder

Output from error variant, should have included /mnt/user/media/TestDuplicati/user/share/

$ ./testDuplicati-bad.sh
Including source path: /mnt/user/media/TestDuplicati/
Including path due to filter: /mnt/user/media/TestDuplicati/cache/ => ([((/mnt/user/media/TestDuplicati/(cache|disk\d+|user)/)|(/mnt/user/media/TestDuplicati/(cache|disk\d+|user)/share/))|(/mnt/user/media/TestDuplicati/user/share/)])
Including path due to filter: /mnt/user/media/TestDuplicati/user/ => ([((/mnt/user/media/TestDuplicati/(cache|disk\d+|user)/)|(/mnt/user/media/TestDuplicati/(cache|disk\d+|user)/share/))|(/mnt/user/media/TestDuplicati/user/share/)])
Excluding path due to filter: /mnt/user/media/TestDuplicati/user/otherShare/ => ([/.*/])
Excluding path due to filter: /mnt/user/media/TestDuplicati/user/share/ => ([/.*/])
Excluding path due to filter: /mnt/user/media/TestDuplicati/cache/share/ => ([/.*/])
Matched 0 files (0 bytes)

Success full Command

Note the only difference is removal of [] brackets around 3. filter. Same result if we remove the second (regexp) filter below..

docker exec duplicati mono /app/duplicati/Duplicati.CommandLine.exe test-filters  /mnt/user/media/TestDuplicati/ \
 --include="[/mnt/user/media/TestDuplicati/(cache|disk\d+|user)/]" \
 --include="[/mnt/user/media/TestDuplicati/(cache|disk\d+|user)/share/]" \
 --include="/mnt/user/media/TestDuplicati/user/share/" \
 --exclude="[/.*/]" \
 --include="[/.*]"

Output from success variant, which correctly includes /mnt/user/media/TestDuplicati/user/share/

$ ./testDuplicati-good-non-regexp-include.sh
Including source path: /mnt/user/media/TestDuplicati/
Including path due to filter: /mnt/user/media/TestDuplicati/cache/ => ([(/mnt/user/media/TestDuplicati/(cache|disk\d+|user)/)|(/mnt/user/media/TestDuplicati/(cache|disk\d+|user)/share/)]) || (/mnt/user/media/TestDuplicati/user/share/)
Including path due to filter: /mnt/user/media/TestDuplicati/user/ => ([(/mnt/user/media/TestDuplicati/(cache|disk\d+|user)/)|(/mnt/user/media/TestDuplicati/(cache|disk\d+|user)/share/)]) || (/mnt/user/media/TestDuplicati/user/share/)
Excluding path due to filter: /mnt/user/media/TestDuplicati/user/otherShare/ => ([/.*/])
Including path due to filter: /mnt/user/media/TestDuplicati/user/share/ => ([(/mnt/user/media/TestDuplicati/(cache|disk\d+|user)/)|(/mnt/user/media/TestDuplicati/(cache|disk\d+|user)/share/)]) || (/mnt/user/media/TestDuplicati/user/share/)
Including path due to filter: /mnt/user/media/TestDuplicati/user/share/user.log => ([/.*])
Excluding path due to filter: /mnt/user/media/TestDuplicati/cache/share/ => ([/.*/])
Matched 1 files (674 bytes)

PS: It would be nice if test-filters had an option to not list all included files but just number of files in each dir (or something).

TestDuplicati-support.zip
system-info.txt
@Jojo-1000
Copy link
Contributor

The problem is that the regexp expressions are combined into (((/mnt/user/media/TestDuplicati/(cache|disk\d+|user)/)|(/mnt/user/media/TestDuplicati/(cache|disk\d+|user)/share/))|(/mnt/user/media/TestDuplicati/user/share/)).
This matches the first part of the path: /mnt/user/media/TestDuplicati/user/ share/
Because it is tested if the match is the full length, this will fail to match the path.
A workaround is to put start ^ and end $ markers around each of the individual regexp strings..
To fix this in the code, we need to always wrap regexp filters in these markers.

Jojo-1000 added a commit to Jojo-1000/duplicati that referenced this issue Oct 12, 2023
@Jojo-1000
Fix incorrect filter behavior when combining multiple regexp.
38d098c 
Closes duplicati#4590

With multiple regexp filters, if an earlier filter matches only part of a path, any other filters were ignored.
Jojo-1000 added a commit to Jojo-1000/duplicati that referenced this issue Nov 2, 2023
@Jojo-1000
Fix incorrect filter behavior when combining multiple regexp.
4f7e414 
Closes duplicati#4590

With multiple regexp filters, if an earlier filter matches only part of a path, any other filters were ignored.
@Jojo-1000 Jojo-1000 linked a pull request Nov 2, 2023 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
filters
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix incorrect filter behavior when combining multiple regexp Jojo-1000/duplicati
3 participants
@arberg @warwickmm @Jojo-1000