Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crashing isolate due to uncaught exception: ClientException #7482

Open
jonasfj opened this issue Feb 15, 2024 · 2 comments
Open

Crashing isolate due to uncaught exception: ClientException #7482

jonasfj opened this issue Feb 15, 2024 · 2 comments
Labels
Area: stackdriver error report Error reported through Stackdriver Error Reporting Type: bug

Comments

@jonasfj
Copy link
Member

jonasfj commented Feb 15, 2024

@isoos if we're using RetryClient from package:http am I correct in that it won't retry if partial response is received. Like in the case of ClientException?

I'm guessing this error isn't the end of the world, but we probably shouldn't crash due to this. I'm also guessing that this isn't the only place where we have this issue.

pub.analyzer: ERROR from index-builder isolate #1

Error:
    [Exception: Crashing isolate due to uncaught exception: ClientException: Connection closed before full header was received, uri=https://storage.googleapis.com/resumable/upload/storage/v1/b/dartlang-pub--search-snapshot/o?name=snapshot%2F2024.02.13.json.gz&uploadType=resumable&alt=json, #0      _wrapper.<anonymous closure> (package:pub_dev/service/entrypoint/_isolate.dart:316:9)
    #1      _RootZone.runBinary (dart:async/zone.dart:1666:54)
    #2      StackZoneSpecification._handleUncaughtError (package:stack_trace/src/stack_zone_specification.dart:161:20)
    #3      _Zone._processUncaughtError (dart:async/zone.dart:1081:14)
    #4      _CustomZone.handleUncaughtError (dart:async/zone.dart:1285:5)
    #5      Future._propagateToListeners (dart:async/future_impl.dart:760:16)
    #6      Future._completeError (dart:async/future_impl.dart:655:5)
    #7      Future._asyncCompleteError.<anonymous closure> (dart:async/future_impl.dart:745:7)
    #8      StackZoneSpecification._run (package:stack_trace/src/stack_zone_specification.dart:207:15)
    #9      StackZoneSpecification._registerCallback.<anonymous closure> (package:stack_trace/src/stack_zone_specification.dart:114:48)
    #10     _rootRun (dart:async/zone.dart:1399:13)
    #11     _CustomZone.run (dart:async/zone.dart:1301:19)
    #12     _CustomZone.runGuarded (dart:async/zone.dart:1209:7)
    #13     _CustomZone.bindCallbackGuarded.<anonymous closure> (dart:async/zone.dart:1249:23)
    #14     _microtaskLoop (dart:async/schedule_microtask.dart:40:21)
    #15     _startMicrotaskLoop (dart:async/schedule_microtask.dart:49:5)
    #16     _runPendingImmediateCallback (dart:isolate-patch/isolate_patch.dart:118:13)
    #17     _RawReceivePort._handleMessage (dart:isolate-patch/isolate_patch.dart:185:5)
    ]

Also related I think:

isolate.wrapper: Uncaught exception in isolate.

Error:
    ClientException: Connection closed before full header was received, uri=https://storage.googleapis.com/resumable/upload/storage/v1/b/dartlang-pub--search-snapshot/o?name=snapshot%2F2024.02.13.json.gz&uploadType=resumable&alt=json

Stack:
    package:http/src/io_client.dart 156:7                    IOClient.send
    ===== asynchronous gap ===========================
    dart:async                                               _Completer.completeError
    package:gcloud/src/storage_impl.dart 629:20              _MediaUploadStreamSink._completeError
    ===== asynchronous gap ===========================
    dart:async                                               Future.then
    package:gcloud/src/storage_impl.dart 659:10              _MediaUploadStreamSink._startResumableUpload
    package:gcloud/src/storage_impl.dart 541:9               new _MediaUploadStreamSink
    package:gcloud/src/storage_impl.dart 199:16              _BucketImpl.write
    package:pub_dev/shared/storage.dart 255:27               uploadWithRetry.<fn>
    package:pub_dev/shared/utils.dart 186:24                 retryAsync
    package:pub_dev/shared/storage.dart 253:9                uploadWithRetry
    package:pub_dev/shared/storage.dart 276:5                uploadBytesWithRetry
    package:pub_dev/shared/storage.dart 300:13               VersionedJsonStorage.uploadDataAsJsonMap
    package:pub_dev/search/backend.dart 212:32               SearchBackend.doCreateAndUpdateSnapshot
    ===== asynchronous gap ===========================
    dart:async                                               _CustomZone.registerUnaryCallback
    package:pub_dev/search/backend.dart 159:5                SearchBackend.doCreateAndUpdateSnapshot
    ===== asynchronous gap ===========================
    dart:async                                               _CustomZone.registerUnaryCallback
    package:pub_dev/search/backend.dart 159:5                SearchBackend.doCreateAndUpdateSnapshot
    package:pub_dev/search/backend.dart 106:17               SearchBackend.updateSnapshotInForeverLoop.<fn>
    package:pub_dev/task/global_lock.dart 81:22              GlobalLock.withClaim
    ===== asynchronous gap ===========================
    dart:async                                               _CustomZone.registerUnaryCallback
    package:pub_dev/task/global_lock.dart 40:15              GlobalLock.withClaim
    package:pub_dev/search/backend.dart 105:20               SearchBackend.updateSnapshotInForeverLoop
    ===== asynchronous gap ===========================
    dart:async                                               _CustomZone.registerUnaryCallback
    package:pub_dev/search/backend.dart 105:9                SearchBackend.updateSnapshotInForeverLoop
    package:pub_dev/service/entrypoint/analyzer.dart 78:23   _indexBuilderMain
    ===== asynchronous gap ===========================
    dart:async                                               _CustomZone.registerUnaryCallback
    package:pub_dev/service/entrypoint/analyzer.dart 77:3    _indexBuilderMain
    package:pub_dev/service/entrypoint/_isolate.dart 311:43  _wrapper.<fn>.<fn>
    package:pub_dev/service/services.dart 295:28             _withPubServices.<fn>.<fn>.<fn>
    dart:async                                               _CustomZone.run
    package:pub_dev/service/services.dart 291:10             _withPubServices.<fn>.<fn>
    package:gcloud/service_scope.dart 205:19                 _ServiceScope._fork.<fn>
    dart:async                                               runZoned
    package:gcloud/service_scope.dart 204:12                 _ServiceScope._fork
    package:gcloud/service_scope.dart 102:30                 fork
    package:pub_dev/service/services.dart 288:18             _withPubServices.<fn>
    ===== asynchronous gap ===========================
    dart:async                                               _CustomZone.registerUnaryCallback
    package:pub_dev/service/services.dart 225:7              _withPubServices.<fn>
    package:gcloud/service_scope.dart 205:19                 _ServiceScope._fork.<fn>
    dart:async                                               runZoned
    package:gcloud/service_scope.dart 204:12                 _ServiceScope._fork
    package:gcloud/service_scope.dart 102:30                 fork
    package:pub_dev/service/services.dart 222:10             _withPubServices
    package:pub_dev/service/services.dart 127:20             withServices.<fn>.<fn>
    ===== asynchronous gap ===========================
    dart:async                                               _CustomZone.registerUnaryCallback
    package:pub_dev/service/services.dart 84:26              withServices.<fn>.<fn>
    package:gcloud/service_scope.dart 205:19                 _ServiceScope._fork.<fn>
    dart:async                                               runZoned
    package:gcloud/service_scope.dart 204:12                 _ServiceScope._fork
    package:gcloud/service_scope.dart 102:30                 fork
    package:pub_dev/service/services.dart 82:18              withServices.<fn>
    package:appengine/src/appengine_internal.dart 37:51      withAppEngineServices.<fn>
    package:appengine/src/appengine_internal.dart 110:20     _withAppEngineServicesInternal.<fn>
    ===== asynchronous gap ===========================
    dart:async                                               _CustomZone.registerUnaryCallback
    package:appengine/src/appengine_internal.dart 102:45     _withAppEngineServicesInternal.<fn>
    package:gcloud/service_scope.dart 205:19                 _ServiceScope._fork.<fn>
    dart:async                                               runZoned
    package:gcloud/service_scope.dart 204:12                 _ServiceScope._fork
    package:gcloud/service_scope.dart 102:30                 fork
    package:appengine/src/appengine_internal.dart 101:13     _withAppEngineServicesInternal
    package:appengine/src/appengine_internal.dart 37:5       withAppEngineServices
    package:appengine/appengine.dart 150:29                  withAppEngineServices
    package:pub_dev/service/services.dart 78:10              withServices
    package:pub_dev/service/entrypoint/_isolate.dart 311:22  _wrapper.<fn>
    package:stack_trace                                      Chain.capture
    package:pub_dev/service/entrypoint/_isolate.dart 310:24  _wrapper
@jonasfj jonasfj added the Area: stackdriver error report Error reported through Stackdriver Error Reporting label Feb 15, 2024
@isoos isoos mentioned this issue Feb 16, 2024
Merged
@isoos
Copy link
Collaborator

isoos commented Feb 16, 2024

Normally this should have been caught by this block:
https://github.com/dart-lang/pub-dev/blob/master/app/lib/search/backend.dart#L104-L110

The only thing so far that may be suspicious is the concurrent processing inside it: if some exception would not have been caught while using Pool + Future.wait...

@isoos
Copy link
Collaborator

isoos commented Feb 16, 2024

The other potential cause could be the scheduled microtask in lock.withClaim:
https://github.com/dart-lang/pub-dev/blob/master/app/lib/task/global_lock.dart#L45

@isoos isoos mentioned this issue Feb 16, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Area: stackdriver error report Error reported through Stackdriver Error Reporting Type: bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jonasfj @isoos