fix: symlinks handling

[2ZeroSix]

fixes: #3143

• reverts to old behavior: symlinks to directories are handled as copy-paste Pub does not support publishing packages with directory symlinks #3143
• fix of pubignore/gitignore:
  • allow non-resolving symlinks if they are ignored
  • don't warn on ignored directory symlinks

[jonasfj]

I'm a bit unsure if we want to support this, how does it handle cycles?
What is the need for symlink directories?

Might it not be better to just ask advanced users to copy files if they need to include them in multiple locations. They can wrap their dart pub publish command with a bash script living in tool/ -- it's ugly, but how many people actually need this?

[2ZeroSix]

It’s common use case for macOS/iOS plug-ins, third party libraries in ffi plugins etc

In our case it forces us to remove directory with symlink to common scripts before publish

pubignore also does not solve it since this check is done prior to application of ignore rules

[2ZeroSix]

• added test for cycles
• fixed wrongly appended / to path consisted of single delimiter

The only failing test should be one which ensures lish is failing on package with symlink to directory.
This seems to be a regression and was introduced somewhere between flutter 2.5+ and 2.8- (dart 2.12 and 2.15)

Previous behavior (pub bundled in dart 2.12 and behavior in current PR):

• pub archives all filtered files in tar.gz

• consumer receives them as separate files after unpacking of tar.gz

• so, there is no OS-dependant problems for consumer and no headache for package developer

Current behavior:

• pub throws on any (even ignored) directory symlink beneath package directory
• package developers should apply workarounds to their pipelines
• consumers doesn't see any difference

If you insist this is expected behavior: I could try to move the check after resolving of ignored files and allow only ignored directory symlinks

[2ZeroSix]

Hmm, it seems that in windows it fails to list dirrectory with loops in symlinks

It differs from what is done on posix-like systems

02:34 +32 -1: test/package_list_files_test.dart: throws on symlink cycles
02:34 +32 -2: test/package_list_files_test.dart: throws on symlink cycles [E]
  Expected: throws <Instance of 'DataException'> with `message`: contains 'Pub does not support publishing packages with non-resolving symlink:'
    Actual: <Closure: () => List<String>>
     Which: threw FileSystemException:<FileSystemException: Directory listing failed, path = 'C:\Users\runneradmin\AppData\Local\Temp\dart_test_5e4ddcd9\myapp\subdir\symlink\subdir\symlink\subdir\symlink\subdir\symlink\subdir\symlink\subdir\symlink\subdir\symlink\subdir\symlink\subdir\symlink\subdir\symlink\subdir\symlink\subdir\symlink\subdir\symlink\*' (OS Error: The system cannot find the path specified.
                  , errno = 3)>
            stack dart:io                                   _Directory.listSync
                  package:pub/src/package.dart 248:48       Package.listFiles.<fn>
                  package:pub/src/ignore.dart 291:28        Ignore.listFiles
                  package:pub/src/package.dart 245:19       Package.listFiles
                  test\package_list_files_test.dart 115:30  main.<fn>.<fn>
                  package:test_api                          expect
                  test\package_list_files_test.dart 114:5   main.<fn>
                  
            which is not an instance of 'DataException'
  
  package:test_api                         expect
  test\package_list_files_test.dart 114:5  main.<fn>

[sigurdm]

Besides the cycle-handling this is looking quite good!

[2ZeroSix]

jfyi: I had no time to reslove windows-related bugs lately, will continue in a few days

[2ZeroSix]

Hi, today I finally found time to fix all problems on windows

I'm going to take a look at the overall code style tomorrow, but otherwise it seems to be ready for review

UPD: nope, as I've started to add comments with reasoning of this solution I found mistake, writing an additional test and fixing the problem now

UPD2: now it should be definitely ready for review 😅

[jonasfj]

@2ZeroSix, can you refresh my memory, how does this handle symlinks?

With this symlink folders will be considered the same as if the folder had been copied.
Thus, cycles won't be allowed. And symlinks will not be retained in the tarballs, instead the files will simply be duplicated.
Is that correct?

---

My concerns are:

• It might be much harder / impossible to verify package contents vs git revision? (cc, @simolus3)
• Wouldn't it be better to fix the Flutter plugin templates to not require duplication?

[jonasfj]

@2ZeroSix is this mostly for ios/macos dart:ffi based plugins? Or are the other use-cases?

[2ZeroSix]

@jonasfj

There is a few use cases:

• any shared code between plugins (macos+ios, ffi, etc)
• some local tooling (like build scripts or something) among packages in multi-package repositories via symlinks
• any other project specific directory symlinks that are not meant to be published
• etc

In most use cases besides "code" directory symlinks are not meant to be published, but existing code throws an exception even for symlink that points to empty directory or ignored (works as expected only if you ignore whole parent directory of symlink)

This pull request solves:

• allows directory symlinks to be published as directory
• allows directory symlinks if they are ignored:
  • gitignore validator seen directory symlinks as directories (but for git they are files), and listed them as gitignored but checked out
• allows non-resolving symlinks if they are ignored

It might be much harder / impossible to verify package contents vs git revision? (cc, @simolus3)

• existing solution: either duplicate code or add some pre-publish scripts to modify package structure
• after this: with minor adjustments it's possible to create archive from these files by running pub archive (for example) and compare them against published archive (there is still room for custom pre-publish scripts, but the space in which it is necessary will decrease)

[2ZeroSix]

If that doesn't seem convincing enough, here is another way:

here is spinoff of this pull request which allows anything (if it is ignored), but still forbids directory symlinks to be published
#3300 (it's a bit outdated and i've closed it as there was no activity)

After addition of proper loop detection it's pretty easy to adjust solution so it would throw only if symlinked directory or it's content isn't ignored

[jonasfj]

any other project specific directory symlinks that are not meant to be published

This I'm sold we should allow.. I thought we did fix that, or did we never land the PR?
But yes, .pubignore/.gitignore should be able to ignore symlinks.

[2ZeroSix]

Yeah, it was never landed.

So lets consider what is the best behavior for pub:

• allow any symlinks and throw only on cycles and non-resolving ones (if they are not ignored)
• disallow directory symlinks and only land a fix for ignored symlinks

It seems to me a bit illogical to allow file-symlinks but disallow directory-symlinks at the same time

Anyway, I'm ok with any decision as on daily basis I mostly use symlinks for internal tooling that are not meant to be published

[jonasfj]

@2ZeroSix

disallow directory symlinks and only land a fix for ignored symlinks

This seems like a no-brainer. We're just correctly ignoring symlinks.

I see how allowing non-cyclic directory symlinks could be nice, but the cost in terms of complexity (and ongoing maintenance) is non-trivial. So maybe this something we should reconsider, if there is a lot requests for it in the future. But right now it's not a highly requested feature, and in general most people are probably better off not using symlinks :D

[Gustl22]

I would simply allow it for the sake of expectation. One should be able to get work everything which is working right now through git.

Of course it's independent, but it's the most common workflow. I see that it could be misused by some devs, but honestly, so you can with code, your git history and everything else.
Linking provides much more flexibility and saves space and time for maintenance. Linking in general is fundamental for so many fields in informatics: packaging, memory, compiling, networks, file system.

The work every developer has to spend to circumvent this lack of feature takes more time and maintenance for all together and certainly brings more bugs, than having a collective and supervised mechanism for directory symlinks.

But I'm also no messias, so this may brings the linking hell. But did it in git?