[Question] How to bypass HTTPError 429: 'Too Many Requests'

[mbnoimi]

Checklist

• I'm asking a question
• I've looked through the README and FAQ for similar questions
• I've searched the bugtracker for similar questions including closed ones

Question

I want to know how to bypass HTTPError 429: 'Too Many Requests' error message. This issue frequently appears. To solve it I restart my router whenever it occurs (for renewing the public IP).
Is there any workaround to bypass it without needing to restart my router or waiting google for ending the restriction?

Log

$ youtube-dlc --newline -i --force-ipv4 --age-limit 30 -R 99 --hls-prefer-native -o "/home/laptop/Downloads/youtube-dl/downloads/Documentary/Nat Geo Abu Dhabi/%(title)s.%(ext)s" -f mp4 --merge-output-format mp4 --write-sub --verbose --continue https://www.youtube.com/user/natgeoabudhabime/videos
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--newline', '-i', '--force-ipv4', '--age-limit', '30', '-R', '99', '--hls-prefer-native', '-o', '/home/laptop/Downloads/youtube-dl/downloads/Documentary/Nat Geo Abu Dhabi/%(title)s.%(ext)s', '-f', 'mp4', '--merge-output-format', 'mp4', '--write-sub', '--verbose', '--continue', 'https://www.youtube.com/user/natgeoabudhabime/videos']
[debug] Loading archive file None
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dlc version 2020.10.09
[debug] Python version 3.8.5 (CPython) - Linux-5.4.0-51-generic-x86_64-with-glibc2.29
[debug] exe versions: ffmpeg 4.2.4, ffprobe 4.2.4
[debug] Proxy map: {}
[youtube:user] natgeoabudhabime: Downloading channel page
[youtube:playlist] UULiTe0aOHShx7hXGyqZ9UIw: Downloading webpage
[download] Downloading playlist: Uploads from Nat Geo Abu Dhabi
[youtube:playlist] UULiTe0aOHShx7hXGyqZ9UIw: Downloading page #1
[youtube:playlist] UULiTe0aOHShx7hXGyqZ9UIw: Downloading page #2
[youtube:playlist] UULiTe0aOHShx7hXGyqZ9UIw: Downloading page #3
[youtube:playlist] UULiTe0aOHShx7hXGyqZ9UIw: Downloading page #4
[youtube:playlist] UULiTe0aOHShx7hXGyqZ9UIw: Downloading page #5
[youtube:playlist] UULiTe0aOHShx7hXGyqZ9UIw: Downloading page #6
[youtube:playlist] UULiTe0aOHShx7hXGyqZ9UIw: Downloading page #7
[youtube:playlist] UULiTe0aOHShx7hXGyqZ9UIw: Downloading page #8
[youtube:playlist] UULiTe0aOHShx7hXGyqZ9UIw: Downloading page #9
[youtube:playlist] UULiTe0aOHShx7hXGyqZ9UIw: Downloading page #10
[youtube:playlist] UULiTe0aOHShx7hXGyqZ9UIw: Downloading page #11
[youtube:playlist] playlist Uploads from Nat Geo Abu Dhabi: Downloading 1119 videos
[download] Downloading video 1 of 1119
[youtube] R9K9NC2MpDs: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 429: Too Many Requests (caused by <HTTPError 429: 'Too Many Requests'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dlc with the --verbose flag and include its complete output.
  File "/home/laptop/.local/lib/python3.8/site-packages/youtube_dlc/extractor/common.py", line 632, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/home/laptop/.local/lib/python3.8/site-packages/youtube_dlc/YoutubeDL.py", line 2308, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)

[download] Downloading video 2 of 1119
[youtube] d4lwAIglTrE: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 429: Too Many Requests (caused by <HTTPError 429: 'Too Many Requests'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dlc with the --verbose flag and include its complete output.
  File "/home/laptop/.local/lib/python3.8/site-packages/youtube_dlc/extractor/common.py", line 632, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/home/laptop/.local/lib/python3.8/site-packages/youtube_dlc/YoutubeDL.py", line 2308, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
^C
ERROR: Interrupted by user
$

[mbnoimi]

NOTE: I tried to use --cookies but it didn't solve it.

[blackjack4494]

There is --sleep-interval however that will only sleep between downloads of files and not between (api) calls like Downloading page.
When I was working on the youtube history feed fix (which is not released btw)

I had to rate limit calls with sleep (internal function).
6 Seconds as in the example is a bit generous. It seems 1-3 seconds (where you should be completely safe with 3 seconds) is actually enough.
Furthermore it also seems this problem is rather related to youtube only as I haven't noticed any other service with similar problems.

[october262]

try using a proxy or a VPN - ytdl-org#22839

[blackjack4494]

@october262 that won't always work out. It's the anti bot design by YouTube. So the tools (youtube-dlc in this case) needs to adjust to that. As I already mentioned all that is needed is some timeout between calls. Current behaviour is easy to identify as bots. But normal user behaviour is not able to process so many request at once so you have to mirror this behaviour.