Arch Linux

Scriptor

Member

Registered: 2011-01-16

Posts: 146

AUR Rate limit reached

Hello,

I don't know if I am the only one to encounter this problem, but when I do:

yay -S <some_aur_package>

I get:

Rate limit reached

I read here that "he API rate is limited to a maximum of 4000 requests per day per IP", but I don't recall having made 4000 requests today.

Thank you very much in advance.

Trilby

Inspector Parrot

Registered: 2011-11-29

Posts: 30,360

Website

Re: AUR Rate limit reached

And can you make direct requests without yay?

---

"UNIX is simple and coherent" - Dennis Ritchie; "GNU's Not Unix" - Richard Stallman

V1del

Forum Moderator

Registered: 2012-10-16

Posts: 24,365

Re: AUR Rate limit reached

An automated update check script? GUI AUR Helpers checking in the background?

This is something that runs on your system, you have to find out what that is.

ewaller

Administrator

From: Pasadena, CA

Registered: 2009-07-13

Posts: 20,381

Re: AUR Rate limit reached

Moving to AUR issues

---

Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
The shortest way to ruin a country is to give power to demagogues.— Dionysius of Halicarnassus
---
How to Ask Questions the Smart Way

Scriptor

Member

Registered: 2011-01-16

Posts: 146

Re: AUR Rate limit reached

Hello,

Thank you for your quick response.

When opening, for example, https://aur.archlinux.org/rpc/?v=5&type=info&arg[]=foobar, I get an error: "Rate limit reached". But once again, I don't recall having reached 4000 requests.

Otherwise, I still can do a simple search in the aur web (https://aur.archlinux.org)

An automated update check script? GUI AUR Helpers checking in the background?

Indeed, I have an automated update check for my polybar. I use yay and checkupdate. I have been using it for a long time, but I have never encountered such a problem. Hopefully this will be corrected by tomorrow.

Thank you very much in advance for your help.

Last edited by Scriptor (2020-03-29 15:11:25)

Scimmia

Fellow

Registered: 2012-09-01

Posts: 12,634

Re: AUR Rate limit reached

And how often does your script check? And how does it check?

It'll be corrected tomorrow until you hit the limit again.

Scriptor

Member

Registered: 2011-01-16

Posts: 146

Re: AUR Rate limit reached

And how often does your script check? And how does it check?

I admit that the delay was too short. I increased it.

Trilby

Inspector Parrot

Registered: 2011-11-29

Posts: 30,360

Website

Re: AUR Rate limit reached

4000 checks per day just to see if there are updates available?  Do you have any need to know the exact second an update comes available?  There's generally not a need to update AUR packages even as often as repo packages - but even putting this aside, say you do them at a similar interval: updating your system once a day is more than sufficient.

This reminds me of a script I once saw that just updated a clock in a status bar.  The clock did not show seconds, just HH:MM, yet the update ran every half second.  There is no reason for an informational display to update at a higher resolution than the greater of the change in granularity of the change in the information or the rate at which you'd notice or care.

Last edited by Trilby (2020-03-29 16:18:23)

---

"UNIX is simple and coherent" - Dennis Ritchie; "GNU's Not Unix" - Richard Stallman

Scriptor

Member

Registered: 2011-01-16

Posts: 146

Re: AUR Rate limit reached

Don't worry, I know it's kinda stupid... At least I've learned something today

Trilby

Inspector Parrot

Registered: 2011-11-29

Posts: 30,360

Website

Re: AUR Rate limit reached

I hope you *drastically* increased the interval.  Adjusting to where you are just below the cutoff prevents you from having trouble, but you'd still be draining community resources for no reason.

---

"UNIX is simple and coherent" - Dennis Ritchie; "GNU's Not Unix" - Richard Stallman

Scriptor

Member

Registered: 2011-01-16

Posts: 146

Re: AUR Rate limit reached

I hope you *drastically* increased the interval.

Indeed, I, at least, multiplied it by a hundred.

Xyne

Administrator/PM

Registered: 2008-08-03

Posts: 6,965

Website

Re: AUR Rate limit reached

I hope you *drastically* increased the interval.

Indeed, I, at least, multiplied it by a hundred.

So then you're still checking at least 40 times a day, which is still too much. Please use the community resources responsibly.

---

My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone

Trilby

Inspector Parrot

Registered: 2011-11-29

Posts: 30,360

Website

Re: AUR Rate limit reached

He said he multiplied the interval by 100, if he previously reached the limit early in the day, now he could be just short of meeting the 4000 limit.

The problem is the interval was never specified.  Zero times 100 is still zero.

---

"UNIX is simple and coherent" - Dennis Ritchie; "GNU's Not Unix" - Richard Stallman

loqs

Member

Registered: 2014-03-06

Posts: 18,418

Re: AUR Rate limit reached

Also does your script issue one query for all AUR packages or one query per AUR package?

Scriptor

Member

Registered: 2011-01-16

Posts: 146

Re: AUR Rate limit reached

Also does your script issue one query for all AUR packages or one query per AUR package?

For all the packages.

So then you're still checking at least 40 times a day, which is still too much. Please use the community resources responsibly.

I'll increase it even more then. Thank you for your remark.

Xyne

Administrator/PM

Registered: 2008-08-03

Posts: 6,965

Website

Re: AUR Rate limit reached

The problem is the interval was never specified.  Zero times 100 is still zero.

Maybe the interval wasn't in seconds. It could be set to check every time a new case of covid-19 is confirmed.

---

My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone

merlock

Member

Registered: 2018-10-30

Posts: 259

Re: AUR Rate limit reached

@Scriptor:

Why don't you just make an account over on AUR, then select "enable notifications" for the packages you use.  You get a nice email, then you can use that nasty yay to update.

---

Eenie meenie, chili beanie, the spirits are about to speak -- Bullwinkle J. Moose
It's a big club...and you ain't in it -- George Carlin
Registered Linux user #149839
perl -e 'print$i=pack(c5,(41*2),sqrt(7056),(unpack(c,H)-2),oct(115),10); '

eschwartz

Fellow

Registered: 2014-08-08

Posts: 4,097

Re: AUR Rate limit reached

Incidentally, the rate limit was introduced specifically in order to block users who had, until then, been using conky scripts to check out of date packages at a rate of one API request per package...

... which ran in a tight loop, sending a new batch (of dozens of requests) once per 5 seconds, or however fast the previous "cower -u" succeeded.

Any decent AUR helper capable of checking for updates (that is, not cower) should only send *one* API request to check many packages at once, so if you check for updates once every 5 minutes you will still only use 288 API requests per day. You'd have to be checking for updates once every 20 seconds to hit the rate limiting at 4320 requests, one request per check (which means you'd run out less than 2 hours before the end of the day).

It is extremely easy to avoid getting rate limited. Even repology.org manages to do okay, and they scrape for all metadata for 60k packages by batching as many packages as can fit into one API request (a few hundred), and only scraping once every few hours.

---

Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

Trilby

Inspector Parrot

Registered: 2011-11-29

Posts: 30,360

Website

Re: AUR Rate limit reached

... which ran in a tight loop, sending a new batch (of dozens of requests) once per 5 seconds, or however fast the previous "cower -u" succeeded.

Damn Technology!  Turn it off.

---

"UNIX is simple and coherent" - Dennis Ritchie; "GNU's Not Unix" - Richard Stallman

SRG

Member

Registered: 2012-12-09

Posts: 33

Re: AUR Rate limit reached

I wonder if sometimes the rate limiter may be triggered for other reasons ...

I also have one "crontab update" script on each of my server, executed once a hour (so in the end, i should have ~24 requests per day * 5 servers = 120 + N manually done requests when installing something, so at worst, ~10 extra ones).
Today i got :

15:50 sergio@moon ~% yay
:: Synchronizing package databases...
 core is up to date
 extra is up to date
 community is up to date
 multilib is up to date
:: Starting full system upgrade...
warning: libxcrypt-compat: local (4.4.28-3) is newer than core (4.4.28-2)
 there is nothing to do
:: Searching databases for updates...
:: Searching AUR for updates...
 -> libxcrypt-compat: local (4.4.28-3) is newer than core (4.4.28-2)
 -> status 429: Rate limit reached

And two minutes later (time to search this on google and to find this topic) :

15:50 sergio@moon ~/% yay
:: Synchronizing package databases...
 core is up to date
 extra is up to date
 community is up to date
 multilib is up to date
:: Starting full system upgrade...
warning: libxcrypt-compat: local (4.4.28-3) is newer than core (4.4.28-2)
 there is nothing to do
:: Searching databases for updates...
:: Searching AUR for updates...
 -> libxcrypt-compat: local (4.4.28-3) is newer than core (4.4.28-2)
 -> Flagged Out Of Date AUR Packages:  chrome-gnome-shell  webkitgtk
 there is nothing to do
15:52 sergio@moon ~/% 

(no more issues after that)
(and first time ever i see that 429 error about yay)

Trilby

Inspector Parrot

Registered: 2011-11-29

Posts: 30,360

Website

Re: AUR Rate limit reached

I also have one "crontab update" script on each of my server, executed once a hour (so in the end, i should have ~24 requests per day * 5 servers = 120 + N manually done requests when installing something, so at worst, ~10 extra ones).

No, it's 24 runs of the script per server per day.  How many requests does the script make each time it runs?  And do you know how many requests yay makes per operation that you request?

Last edited by Trilby (2022-07-31 14:10:23)

---

"UNIX is simple and coherent" - Dennis Ritchie; "GNU's Not Unix" - Richard Stallman

Scimmia

Fellow

Registered: 2012-09-01

Posts: 12,634

Re: AUR Rate limit reached

Try again, there was a temporary glitch