API Request

This is where you can request API access. Please be sure to mention what your use case is and if possible a link to the project.
Please provide a basic overview of your proposed use of the API.
Any request lacking this information will be denied pending an appropriate response.
Forum rules
Please provide a basic overview of your proposed use of the API.
Any request lacking this information will be denied pending an appropriate response.
keith.f.kelly
Posts: 9
Joined: Wed Nov 28, 2018 2:17 am

API Request

Post by keith.f.kelly »

I'm an individual user/collector with many thousands of games I need to scrape as I build out my RetroPie system. I keep running up against the anonymous/regular user throttling and caps while trying to scrape my large personal collection, so I need an API key. This will only be for my personal use, not redistributed or shared with anyone. I'm also happy to make a modest donation via Patreon or PayPal or something if that's what it takes to offset the costs of my personal usage -- just tell me where to send it. Thanks in advance!

User avatar
edirol
Posts: 264
Joined: Thu Jun 28, 2018 1:08 am

Re: API Request

Post by edirol »

Hello.

I'm afraid you can't get API key as regular user because keys are intended for developers only.
From Russia with love

keith.f.kelly
Posts: 9
Joined: Wed Nov 28, 2018 2:17 am

Re: API Request

Post by keith.f.kelly »

Then you all really need to increase the non-developer limits. The current limits are nowhere near adequate for users. If bandwidth or server capacity is an issue, then please provide a way for users to sign up for a paid subscription access or something. Thanks.

StainlessSteelRat
Posts: 29
Joined: Mon May 21, 2018 7:18 am

Re: API Request

Post by StainlessSteelRat »

The scrapers shouldn't be targeting an app that uses the Api directly like that. They should collating all the data in some way, and using the update API to keep the db in sync. This could mean pushing an entire db to users (like the apps that use the API do) or them writing their own API.

alanpotts
Posts: 1
Joined: Wed Jan 13, 2021 9:04 am

Re: API Request

Post by alanpotts »

The scrapers should collate all the data in some way and use the update API to keep the db in sync.

keith.f.kelly
Posts: 9
Joined: Wed Nov 28, 2018 2:17 am

Re: API Request

Post by keith.f.kelly »

StainlessSteelRat wrote:
Tue Jan 12, 2021 7:27 pm
The scrapers shouldn't be targeting an app that uses the Api directly like that.
What do you mean by "targeting an app"? The scrapers I am using (EmulationStation's built-in scraper, and the standalone "skyscraper" command-line scraper) don't "target an app"; they are the apps, and each of those apps queries one or more online databases (such as TheGamesDB) for each game to initially fetch metadata and art/screenshots.
They should collating all the data in some way, and using the update API to keep the db in sync. This could mean pushing an entire db to users (like the apps that use the API do) or them writing their own API.
What do you mean by "collating"? Do you mean "locally caching", as skyscraper does? Local caches must be initially populated by querying the server API for each game -- that is where I keep running up against the limit. I have tens of thousands of games in my collection, and I need to scrape metadata and art/screenshots for each of them at least once. With an apparent monthly per-user limit of only 20,000 queries, and with multiple RetroPie systems I'm separately building out using my game collection, that limit is easily exhausted.

User avatar
edirol
Posts: 264
Joined: Thu Jun 28, 2018 1:08 am

Re: API Request

Post by edirol »

Mind the whole sentence - scrapers shouldn't be targeting an app that uses the Api directly like that.

Most preferrable API usage for any app is creating own mirrror/cache/etc and updating it using updates endpoint.
From Russia with love

StainlessSteelRat
Posts: 29
Joined: Mon May 21, 2018 7:18 am

Re: API Request

Post by StainlessSteelRat »

Yes, they should be providing users with a "starting point" and then syncing from there. If they can't do that, then they should at least start from the monthly JSON dump, use that to populate a local cache, and then sync from there.

Then all queries can hit the local DB instead of the API.

keith.f.kelly
Posts: 9
Joined: Wed Nov 28, 2018 2:17 am

Re: API Request

Post by keith.f.kelly »

Okay, that's fine and dandy... but it doesn't help end-users like me. I can't force a scraper developer to add features you say I need.

keith.f.kelly
Posts: 9
Joined: Wed Nov 28, 2018 2:17 am

Re: API Request

Post by keith.f.kelly »

Is there some way for me to download the entire dB (including images, such as screenshots and box art) at one shot, and then use it to populate skyscraper's local cache? I'm willing to do it myself, if there's a way.

Post Reply