Page 1 of 2

API Request

Posted: Mon Dec 21, 2020 12:49 am
by keith.f.kelly
I'm an individual user/collector with many thousands of games I need to scrape as I build out my RetroPie system. I keep running up against the anonymous/regular user throttling and caps while trying to scrape my large personal collection, so I need an API key. This will only be for my personal use, not redistributed or shared with anyone. I'm also happy to make a modest donation via Patreon or PayPal or something if that's what it takes to offset the costs of my personal usage -- just tell me where to send it. Thanks in advance!

Re: API Request

Posted: Sun Jan 03, 2021 6:37 am
by edirol
Hello.

I'm afraid you can't get API key as regular user because keys are intended for developers only.

Re: API Request

Posted: Fri Jan 08, 2021 10:37 pm
by keith.f.kelly
Then you all really need to increase the non-developer limits. The current limits are nowhere near adequate for users. If bandwidth or server capacity is an issue, then please provide a way for users to sign up for a paid subscription access or something. Thanks.

Re: API Request

Posted: Tue Jan 12, 2021 7:27 pm
by StainlessSteelRat
The scrapers shouldn't be targeting an app that uses the Api directly like that. They should collating all the data in some way, and using the update API to keep the db in sync. This could mean pushing an entire db to users (like the apps that use the API do) or them writing their own API.

Re: API Request

Posted: Wed Jan 13, 2021 9:07 am
by alanpotts
The scrapers should collate all the data in some way and use the update API to keep the db in sync.

Re: API Request

Posted: Fri Jan 15, 2021 10:10 am
by keith.f.kelly
StainlessSteelRat wrote:
Tue Jan 12, 2021 7:27 pm
The scrapers shouldn't be targeting an app that uses the Api directly like that.
What do you mean by "targeting an app"? The scrapers I am using (EmulationStation's built-in scraper, and the standalone "skyscraper" command-line scraper) don't "target an app"; they are the apps, and each of those apps queries one or more online databases (such as TheGamesDB) for each game to initially fetch metadata and art/screenshots.
They should collating all the data in some way, and using the update API to keep the db in sync. This could mean pushing an entire db to users (like the apps that use the API do) or them writing their own API.
What do you mean by "collating"? Do you mean "locally caching", as skyscraper does? Local caches must be initially populated by querying the server API for each game -- that is where I keep running up against the limit. I have tens of thousands of games in my collection, and I need to scrape metadata and art/screenshots for each of them at least once. With an apparent monthly per-user limit of only 20,000 queries, and with multiple RetroPie systems I'm separately building out using my game collection, that limit is easily exhausted.

Re: API Request

Posted: Sun Jan 17, 2021 8:38 am
by edirol
Mind the whole sentence - scrapers shouldn't be targeting an app that uses the Api directly like that.

Most preferrable API usage for any app is creating own mirrror/cache/etc and updating it using updates endpoint.

Re: API Request

Posted: Mon Jan 18, 2021 11:24 pm
by StainlessSteelRat
Yes, they should be providing users with a "starting point" and then syncing from there. If they can't do that, then they should at least start from the monthly JSON dump, use that to populate a local cache, and then sync from there.

Then all queries can hit the local DB instead of the API.

Re: API Request

Posted: Wed Jan 20, 2021 1:59 pm
by keith.f.kelly
Okay, that's fine and dandy... but it doesn't help end-users like me. I can't force a scraper developer to add features you say I need.

Re: API Request

Posted: Fri Jan 22, 2021 3:03 am
by keith.f.kelly
Is there some way for me to download the entire dB (including images, such as screenshots and box art) at one shot, and then use it to populate skyscraper's local cache? I'm willing to do it myself, if there's a way.