Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Certain inputs to sky launch would trigger sky check, making it slow #3565

Closed
concretevitamin opened this issue May 20, 2024 · 0 comments · Fixed by #3567
Closed

Certain inputs to sky launch would trigger sky check, making it slow #3565

concretevitamin opened this issue May 20, 2024 · 0 comments · Fixed by #3567

Comments

@concretevitamin
Copy link
Collaborator

concretevitamin commented May 20, 2024

resources:
  accelerators: A100:8
  ordered:
    - cloud: kubernetes
    - cloud: aws
      use_spot: true

My sky check has AWS, GCP, ... enabled, no k8s.

On latest main branch: 19e8ed1, it took 16s to error out:

time sky launch t-multi.yaml                                                                                    
Task from YAML spec: t-multi.yaml
I 05-19 18:22:18 optimizer.py:983] Using user-specified accelerators list (will be tried in the listed order): Kubernetes({'A100': 8}), AWS([Spot], {'A100': 8})
INFO:googleapiclient.discovery_cache:file_cache is only supported with oauth2client<4.0.0
sky.exceptions.ResourcesUnavailableError: Task requires Kubernetes which is not enabled: Task(run=<empty>)
  resources: Kubernetes({'A100': 8}).
To enable access, run sky check , or change the cloud requirement
sky launch t-multi.yaml  7.12s user 2.06s system 57% cpu 16.084 total

py-spy:

sudo py-spy dump --pid  82883
Process 82883: /Users/zongheng/anaconda/envs/py39/bin/python /Users/zongheng/anaconda/envs/py39/bin/sky launch t-multi.yaml
Python v3.9.18 (/Users/zongheng/anaconda/envs/py39/bin/python3.9)

Thread 0x203261600 (idle): "MainThread"
    select (selectors.py:416)
    _communicate (subprocess.py:1995)
    communicate (subprocess.py:1134)
    run (subprocess.py:507)
    is_api_disabled (sky/clouds/gcp.py:107)
    check_credentials (sky/clouds/gcp.py:702)
    check_one_cloud (sky/check.py:34)
    check (sky/check.py:94)
    _fill_in_launchable_resources (sky/optimizer.py:1178)
    _optimize_dag (sky/optimizer.py:1000)
    optimize (sky/optimizer.py:127)

This appears to be caused by not having k8s enabled but it's requested in YAML. Is there a good optimization for such cases?

  • At the minimum, sky check should not check GCP, which is not mentioned in the YAML

Version & Commit info:

  • sky -v: PLEASE_FILL_IN
  • sky -c: PLEASE_FILL_IN
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant