dataset download error

[Thaumaturge2020]

Habitat-Lab and Habitat-Sim versions

Habitat-Lab: master

Habitat-Sim: master

when I try to download the habitat_test_scenes I got this error:

stderr: 'fatal: unable to access 'https://huggingface.co/datasets/ai-habitat/habitat_test_scenes.git/': gnutls_handshake() failed: Error in the pull function.'

the cmd I used was :

python -m habitat_sim.utils.datasets_download --uids habitat_test_scenes --data-path data/

How could i solve this problem?

[aclegg3]

Try again, this is working for me. I haven't seen this error personally, but it may be a simple network outage at the time of download.

[21stYouth]

i have the same err in download, have you solved it?

[aclegg3]

Hey folks, looks like this could very well be a local system package or firewall issue: https://stackoverflow.com/questions/52379234/git-gnutls-handshake-failed-error-in-the-pull-function

[21stYouth]

Unfortunately，i still have the problem.

Here is the err (when i use python -m habitat_sim.utils.datasets_download --uids habitat_test_scenes --data-path data/):

git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
cmdline: git fetch -v -- origin v1.6
stderr: 'fatal: unable to access 'https://huggingface.co/datasets/ai-habitat/ReplicaCAD_dataset.git/': Failed to connect to huggingface.co port 443 after 133412 ms: Connection timed out'

I went to the website https://huggingface.co/datasets/ai-habitat/ReplicaCAD_dataset.git/, it showed 404 to me. Is that normal, or just i have this result?

[xavierpuigf]

I am not sure of what could be cause the timing out issue. As for the link, the way you can see the dataset on the website is by removing the .git extension.

https://huggingface.co/datasets/ai-habitat/ReplicaCAD_dataset

[Lr-2002]

Are you in China Mainland？ This was because of the network ,you need to the proxy to solve it.

[21stYouth]

Thanks everyone, i finally found the solution of these problems.
May i repeat the key point of my err:
When i use python -m habitat_sim.utils.datasets_download --uids habitat_test_scenes --data-path data/ or python examples/example.py, the program will have err and said

stderr: 'fatal: unable to access 'https://huggingface.co/datasets/ai-habitat/ReplicaCAD_dataset.git/'

So here are my solutions:

1. Git over SSH in Huggingface: You can see details in https://huggingface.co/docs/hub/security-git-ssh
2. Find the git clone command in the err report, like

git clone --depth 1 --branch v1.6 https://huggingface.co/datasets/ai-habitat/ReplicaCAD_dataset.git /home/xxx/habitat-lab/data/versioned_data/replica_cad_dataset

3. Change the https://huggingface.co/ to git@hf.co:. So the command will be like

git clone --depth 1 --branch v1.6 git@hf.co:datasets/ai-habitat/ReplicaCAD_dataset.git /home/xxx/habitat-lab/data/versioned_data/replica_cad_dataset

4. Use this command to download the datasets again and again when having the err, until all the datasets are downloaded

[21stYouth]

By the way, the problem of

subprocess.CalledProcessError: Command '['git', 'clone', '--depth', '1', '--branch', 'v1.6', 'https://huggingface.co/datasets/ai-habitat/ReplicaCAD_dataset.git', '/home/xxx/habitat-lab/data/versioned_data/replica_cad_dataset']' returned non-zero exit status 128.

maybe can be solved by this way as well

[aclegg3]

@21stYouth

HF has deprecated use of username/password for authentication. By using git@hf.com you specify use of the SSH authentication channel.
Our downloader does this for authenticated datasets. I'm a bit surprised to see this is ever necessary for non-authenticated datasets.