You are not logged in.

#1 2020-03-29 14:49:58

Scriptor
Member
Registered: 2011-01-16
Posts: 144

AUR Rate limit reached

Hello,

I don't know if I am the only one to encounter this problem, but when I do:

yay -S <some_aur_package>

I get:

Rate limit reached

I read here that "he API rate is limited to a maximum of 4000 requests per day per IP", but I don't recall having made 4000 requests today.

Thank you very much in advance.

Offline

#2 2020-03-29 15:02:34

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,441
Website

Re: AUR Rate limit reached

And can you make direct requests without yay?


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#3 2020-03-29 15:05:30

V1del
Forum Moderator
Registered: 2012-10-16
Posts: 21,415

Re: AUR Rate limit reached

An automated update check script? GUI AUR Helpers checking in the background?

This is something that runs on your system, you have to find out what that is.

Online

#4 2020-03-29 15:06:35

ewaller
Administrator
From: Pasadena, CA
Registered: 2009-07-13
Posts: 19,739

Re: AUR Rate limit reached

Moving to AUR issues


Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
Sometimes it is the people no one can imagine anything of who do the things no one can imagine. -- Alan Turing
---
How to Ask Questions the Smart Way

Offline

#5 2020-03-29 15:08:38

Scriptor
Member
Registered: 2011-01-16
Posts: 144

Re: AUR Rate limit reached

Hello,

Thank you for your quick response.

When opening, for example, https://aur.archlinux.org/rpc/?v=5&type=info&arg[]=foobar, I get an error: "Rate limit reached". But once again, I don't recall having reached 4000 requests.

Otherwise, I still can do a simple search in the aur web (https://aur.archlinux.org)

An automated update check script? GUI AUR Helpers checking in the background?

Indeed, I have an automated update check for my polybar. I use yay and checkupdate. I have been using it for a long time, but I have never encountered such a problem. Hopefully this will be corrected by tomorrow.

Thank you very much in advance for your help.

Last edited by Scriptor (2020-03-29 15:11:25)

Offline

#6 2020-03-29 15:30:29

Scimmia
Fellow
Registered: 2012-09-01
Posts: 11,461

Re: AUR Rate limit reached

And how often does your script check? And how does it check?

It'll be corrected tomorrow until you hit the limit again.

Offline

#7 2020-03-29 15:32:10

Scriptor
Member
Registered: 2011-01-16
Posts: 144

Re: AUR Rate limit reached

And how often does your script check? And how does it check?

I admit that the delay was too short. I increased it.

Offline

#8 2020-03-29 16:02:45

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,441
Website

Re: AUR Rate limit reached

4000 checks per day just to see if there are updates available?  Do you have any need to know the exact second an update comes available?  There's generally not a need to update AUR packages even as often as repo packages - but even putting this aside, say you do them at a similar interval: updating your system once a day is more than sufficient.

This reminds me of a script I once saw that just updated a clock in a status bar.  The clock did not show seconds, just HH:MM, yet the update ran every half second.  There is no reason for an informational display to update at a higher resolution than the greater of the change in granularity of the change in the information or the rate at which you'd notice or care.

Last edited by Trilby (2020-03-29 16:18:23)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#9 2020-03-29 16:24:26

Scriptor
Member
Registered: 2011-01-16
Posts: 144

Re: AUR Rate limit reached

Don't worry, I know it's kinda stupid... At least I've learned something today smile

Offline

#10 2020-03-29 16:37:52

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,441
Website

Re: AUR Rate limit reached

I hope you *drastically* increased the interval.  Adjusting to where you are just below the cutoff prevents you from having trouble, but you'd still be draining community resources for no reason.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#11 2020-03-29 19:04:36

Scriptor
Member
Registered: 2011-01-16
Posts: 144

Re: AUR Rate limit reached

I hope you *drastically* increased the interval.

Indeed, I, at least, multiplied it by a hundred.

Offline

#12 2020-03-29 19:28:27

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: AUR Rate limit reached

Scriptor wrote:

I hope you *drastically* increased the interval.

Indeed, I, at least, multiplied it by a hundred.

So then you're still checking at least 40 times a day, which is still too much. Please use the community resources responsibly.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#13 2020-03-29 19:38:59

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,441
Website

Re: AUR Rate limit reached

He said he multiplied the interval by 100, if he previously reached the limit early in the day, now he could be just short of meeting the 4000 limit.

The problem is the interval was never specified.  Zero times 100 is still zero.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#14 2020-03-29 19:58:27

loqs
Member
Registered: 2014-03-06
Posts: 17,192

Re: AUR Rate limit reached

Also does your script issue one query for all AUR packages or one query per AUR package?

Offline

#15 2020-03-29 20:04:47

Scriptor
Member
Registered: 2011-01-16
Posts: 144

Re: AUR Rate limit reached

Also does your script issue one query for all AUR packages or one query per AUR package?

For all the packages.

So then you're still checking at least 40 times a day, which is still too much. Please use the community resources responsibly.

I'll increase it even more then. Thank you for your remark.

Offline

#16 2020-03-29 20:17:13

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: AUR Rate limit reached

Trilby wrote:

The problem is the interval was never specified.  Zero times 100 is still zero.

Maybe the interval wasn't in seconds. It could be set to check every time a new case of covid-19 is confirmed.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#17 2020-03-29 21:49:05

merlock
Member
Registered: 2018-10-30
Posts: 233

Re: AUR Rate limit reached

@Scriptor:

Why don't you just make an account over on AUR, then select "enable notifications" for the packages you use.  You get a nice email, then you can use that nasty yay to update.


Eenie meenie, chili beanie, the spirits are about to speak -- Bullwinkle J. Moose
It's a big club...and you ain't in it -- George Carlin
Registered Linux user #149839
perl -e 'print$i=pack(c5,(41*2),sqrt(7056),(unpack(c,H)-2),oct(115),10); '

Offline

#18 2020-03-29 23:02:24

eschwartz
Fellow
Registered: 2014-08-08
Posts: 4,097

Re: AUR Rate limit reached

Incidentally, the rate limit was introduced specifically in order to block users who had, until then, been using conky scripts to check out of date packages at a rate of one API request per package...

... which ran in a tight loop, sending a new batch (of dozens of requests) once per 5 seconds, or however fast the previous "cower -u" succeeded.

Any decent AUR helper capable of checking for updates (that is, not cower) should only send *one* API request to check many packages at once, so if you check for updates once every 5 minutes you will still only use 288 API requests per day. You'd have to be checking for updates once every 20 seconds to hit the rate limiting at 4320 requests, one request per check (which means you'd run out less than 2 hours before the end of the day).

It is extremely easy to avoid getting rate limited. Even repology.org manages to do okay, and they scrape for all metadata for 60k packages by batching as many packages as can fit into one API request (a few hundred), and only scraping once every few hours.


Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

Offline

#19 2020-03-29 23:15:28

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,441
Website

Re: AUR Rate limit reached

eschwartz wrote:

... which ran in a tight loop, sending a new batch (of dozens of requests) once per 5 seconds, or however fast the previous "cower -u" succeeded.

Damn Technology!  Turn it off.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#20 2022-07-31 13:59:13

SRG
Member
Registered: 2012-12-09
Posts: 9

Re: AUR Rate limit reached

I wonder if sometimes the rate limiter may be triggered for other reasons ...

I also have one "crontab update" script on each of my server, executed once a hour (so in the end, i should have ~24 requests per day * 5 servers = 120 + N manually done requests when installing something, so at worst, ~10 extra ones).
Today i got :

15:50 sergio@moon ~% yay
:: Synchronizing package databases...
 core is up to date
 extra is up to date
 community is up to date
 multilib is up to date
:: Starting full system upgrade...
warning: libxcrypt-compat: local (4.4.28-3) is newer than core (4.4.28-2)
 there is nothing to do
:: Searching databases for updates...
:: Searching AUR for updates...
 -> libxcrypt-compat: local (4.4.28-3) is newer than core (4.4.28-2)
 -> status 429: Rate limit reached

And two minutes later (time to search this on google and to find this topic) :

15:50 sergio@moon ~/% yay
:: Synchronizing package databases...
 core is up to date
 extra is up to date
 community is up to date
 multilib is up to date
:: Starting full system upgrade...
warning: libxcrypt-compat: local (4.4.28-3) is newer than core (4.4.28-2)
 there is nothing to do
:: Searching databases for updates...
:: Searching AUR for updates...
 -> libxcrypt-compat: local (4.4.28-3) is newer than core (4.4.28-2)
 -> Flagged Out Of Date AUR Packages:  chrome-gnome-shell  webkitgtk
 there is nothing to do
15:52 sergio@moon ~/% 

(no more issues after that)
(and first time ever i see that 429 error about yay)

Offline

#21 2022-07-31 14:09:31

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,441
Website

Re: AUR Rate limit reached

SRG wrote:

I also have one "crontab update" script on each of my server, executed once a hour (so in the end, i should have ~24 requests per day * 5 servers = 120 + N manually done requests when installing something, so at worst, ~10 extra ones).

No, it's 24 runs of the script per server per day.  How many requests does the script make each time it runs?  And do you know how many requests yay makes per operation that you request?

Last edited by Trilby (2022-07-31 14:10:23)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#22 2022-07-31 14:23:13

Scimmia
Fellow
Registered: 2012-09-01
Posts: 11,461

Re: AUR Rate limit reached

Try again, there was a temporary glitch

Offline

Board footer

Powered by FluxBB