analyzer takes a long time to process deletion of empty file on linux

[Hixie]

STEPS TO REPRODUCE

1. Run flutter analyze --watch in a big package, e.g. the flutter framework (https://github.com/flutter/flutter/tree/master/packages/flutter)
2. From a separate terminal, create an empty file in that package, e.g. touch foo.dart
3. Delete that file, e.g. rm foo.dart

EXPECTED RESULTS

Step 1 should take a while. Steps 2 and 3 should each trigger a very fast reanalysis.

ACTUAL RESULTS

Step 1 takes a while. Step 2 triggers a fast analysis. Step 3 triggers a multi-second analysis.

CONTEXT

This impacts users of IDEs that create temporary files with .dart extensions. For example, Emacs creates advisory lock files with names like .#animated_size_test.dart when a file is edited and not saved. (Actually in the case of Emacs it's not even a file, it's a symlink to a non-existent file, but the effect on the analyzer is the same.) When a file is saved, reanalysis takes multiple seconds every time even though nothing has changed, because the editor deletes this temporary file with a .dart extension.

[Hixie]

cc @DanTup

[bwilkerson]

@scheglov

[DanTup]

@scheglov I had a quick look at this, and with some extra logging I can see that the deletion seems to trigger analyzing a lot of files:

add new file

1714040979301:Watch:<unknown>:C::\Dev\Google\Flutter\Flutter main\packages\flutter\foo.dart:add
1714040979303:Noti:{"event"::"server.status","params"::{"analysis"::{"isAnalyzing"::true}}}
1714040979304:Info:Analyzing general file C::\Dev\Google\Flutter\Flutter main\packages\flutter\foo.dart
1714040979305:Noti:{"event"::"analysis.errors","params"::{"file"::"C::\\Dev\\Google\\Flutter\\Flutter main\\packages\\flutter\\foo.dart","errors"::[]}}
1714040979307:Noti:{"event"::"server.status","params"::{"analysis"::{"isAnalyzing"::false}}}

delete file

1714040984594:Watch:<unknown>:C::\Dev\Google\Flutter\Flutter main\packages\flutter\foo.dart:remove
1714040984596:Noti:{"event"::"analysis.flushResults","params"::{"files"::["C::\\Dev\\Google\\Flutter\\Flutter main\\packages\\flutter\\foo.dart"]}}
1714040984596:Noti:{"event"::"server.status","params"::{"analysis"::{"isAnalyzing"::true}}}
1714040984601:Info:Analyzing general file C::\Dev\Google\Flutter\Flutter main\packages\flutter\test_profile\basic_test.dart
1714040984674:Info:Analyzing general file C::\Dev\Google\Flutter\Flutter main\packages\flutter\test_release\diagnostics_test.dart
1714040984674:Info:Analyzing general file C::\Dev\Google\Flutter\Flutter main\packages\flutter\test_release\foundation\memory_allocations_test.dart
1714040984674:Info:Analyzing general file C::\Dev\Google\Flutter\Flutter main\packages\flutter\test_release\precondition_test.dart
1714040984674:Info:Analyzing general file C::\Dev\Google\Flutter\Flutter main\packages\flutter\test_release\rendering\memory_allocations_test.dart
1714040984674:Info:Analyzing general file C::\Dev\Google\Flutter\Flutter main\packages\flutter\test_release\widgets\memory_allocations_test.dart
1714040984675:Info:Analyzing general file C::\Dev\Google\Flutter\Flutter main\packages\flutter\lib\animation.dart
1714040984696:Info:Analyzing general file C::\Dev\Google\Flutter\Flutter main\packages\flutter\lib\cupertino.dart
1714040984696:Info:Analyzing general file C::\Dev\Google\Flutter\Flutter main\packages\flutter\lib\foundation.dart
1714040984696:Info:Analyzing general file C::\Dev\Google\Flutter\Flutter main\packages\flutter\lib\gestures.dart
1714040984696:Info:Analyzing general file C::\Dev\Google\Flutter\Flutter main\packages\flutter\lib\material.dart
1714040984704:Info:Analyzing general file C::\Dev\Google\Flutter\Flutter main\packages\flutter\lib\painting.dart

Looking in the code, I came across this line that adds all of addedFiles to _pendingFiles when a file is removed (and a TODO that may be related):

sdk/pkg/analyzer/lib/src/dart/analysis/file_tracker.dart

Lines 141 to 144 in 4b66657

	// TODO(paulberry): removing the path from [fsState] and re-analyzing all
	// files seems extreme.
	_fsState.removeFile(path);
	_pendingFiles.addAll(addedFiles);

I guess the intention is to ensure files that depend on the deleted file are re-analyzed. I don't know how much of that work is short-cut or whether it's possible to be more selective (in this case, 0 files depend on the deleted one). On my machine, the time spent re-analyzing for the Flutter repo is about 4s on the deletion.

[dart-github-bot]

Item	Details
Summary	Analyzer takes a long time to process deletion of empty file on Linux.
Triage to	area-analyzer

(what's this?)

[scheglov]

The analysis is correct, deleting a file is currently very expensive.
Improving here will require a lot of thinking and testing, for which we don't have time right now.

[DanTup]

@Hixie as a workaround for you, you might be able to change your editor to not write these lock files to the project - see https://www.emacswiki.org/emacs/LockFiles

It does say "but it might not be wise to do this" without an explanation though 🥲

[Hixie]

Yeah, I've changed my configuration already. :-)

[Hixie]

Does the analyzer have a graph of which files affect each other?