TheGamesDb.net

StainlessSteelRat wrote: ↑

Mon Jan 18, 2021 11:24 pm

they should at least start from the monthly JSON dump

What monthly JSON dump?

edirol wrote: ↑

Sun Jan 17, 2021 8:38 am

Mind the whole sentence - scrapers shouldn't be targeting an app that uses the Api directly like that.

I did "mind the whole sentence". It still doesn't make any sense. "Scrapers shouldn't be targeting an app" doesn't make any sense. A scraper is an app. How can an app target an app? And what do you even mean by "target" an app?

There are few ways to get info: API, SQL dump and JSON dump (viewtopic.php?f=5&t=1947).
For all questions regarding Skyscraper contact its dev.

And you still read only half of original phrase. Explanation is simple - scrapers shouldn't be the type of app that uses API each time users request data. I've already explained most preferrable model of using our API.

More to the point, and our developers can clarify, there's a huge issue with scrapers not batching queries, instead opting for 1:1 pulls that quickly hit the API limit. This quickly exhausts user limits, where the preferred method would have those projects periodically syncing the full database and correctly batching queries (per every download site ever, admins quickly get irritated by users refusing to queue downloads, either by their own malign intent OR through inaction by the devs responsible for application upkeep).

Our developers can fill in some of the blanks, like implementation of the API or quantification of usage limits; in our case, when correctly implemented, the number of available queries significantly outweighs the current size of the database, with intended options to also pull a semi-regular dump and only update listings that have changed.

edirol wrote: ↑

Fri Jan 22, 2021 4:56 pm

And you still read only half of original phrase.

No, I read the whole thing. What you wrote was simply unclear and non-sensical. I think you overestimate your own English fluency.

Regardless, I finally understand what you were trying to say, thanks to Leo_Pride's answer.

Leo_Pride wrote: ↑

Sat Jan 23, 2021 10:46 am

More to the point, and our developers can clarify, there's a huge issue with scrapers not batching queries, instead opting for 1:1 pulls that quickly hit the API limit.

Then you should re-examine your API documentation, because it's obviously not as clear as you think.

Meanwhile, if skyscraper and other RetroPie-provided scrapers are discouraged, then can you please recommend a scraper (that can produce EmulationStation-compatible gamelist.xml files) that works in the "well behaved" manner you describe? Preferably one that runs on Raspberry Pi?

And if you want people to only use "well behaved" scrapers, your web site should probably maintain a list of "recommended" and "discouraged" scrapers in a prominent spot on the front page. Otherwise, people are just going to keep using whatever RetroPie and its documentation tell them to use.