Suggested changes after AEL front-end development

Projects, Applications, and API development.
Post Reply
Wintermute0110
Posts: 3
Joined: Thu Jun 27, 2019 9:53 am

Suggested changes after AEL front-end development

Post by Wintermute0110 » Tue Aug 20, 2019 10:48 am

I just completed a rewriting of the scraper using TGDB for Advanced Emulator Launcher, my front-end addon for Kodi. During the development I have found a series of comments/issues I would like to share that may improve TGDB API and website.

Issue a) In the endpoint /Games/ByGameID developers, publishers and genres are returned as database ID numbers and not resolved to the correct string. Of course, this can be easily solved by calling the API. For example, calling /Genres to retrieve all the genres and then convert the genre database ID to the corresponding string. In my opinion, this is inefficient and consumes precious API calls. Of course the Developers, Publishers and Genres can be cached locally, but this increases the complexity of the front-end and creates problems, for example, how frequently the cache must be updated, etc.

Solution: Create new keywords in the /Games/ByGameID fields, for example developers_str, publishers _str and genres_str, that return strings instead of database IDs. An example in JSON:

Code: Select all

"games": [
        {
          "id": 1,
          "game_title": "Halo: Combat Evolved",
          "release_date": "2001-11-15",
          "developers": [1389],
          "genres": [8],
          "publishers": [1],
          "developers_str": ["Bungie"],
          "publishers_str": ["Microsoft"],
          "genres_str": ["First person shooter"],
        }
]
One last thing: developers seems to be always returned when calling /Games/ByGameID but if developers_str is introduced it can be changed from mandatory to optional though the fields parameter.

Issue b) The number of remaining API calls can be checked in every API call. In AEL I have implemented a functionality that allows users to check their remaining number of API calls. However, when the API number of calls is exhausted the server always returns status code "403 Forbidden" and the allowance_refresh_timer cannot be checked any more, and hence there is no way for the user to know how long he has to wait.

Would it be possible that when the API calls are exhausted the server returns a JSON like this??

Code: Select all

[
  {
    "code": 429,
    "status": "Too many requests. API number of calls exceeded",
    "remaining_monthly_allowance": 0,
    "allowance_refresh_timer": 1234567,
    "extra_allowance": 0
  }
]
Issue c) Will it be possible to differentiate between screenshots and Title screenshots? For example, the current image type screenshot is for in-game screenshots and a new titleshot for title screenshots. Of course this is a huge change but if you change the database then users will start contributing title screenshots little by little.

Issue d) I think many people have mentioned it in the forum, but 1500 API calls per month per IP is only useful to scrape really small collections. Would it be possible to increase it, for example, to 1000 calls per day?

Finally, kudos to TGDB developers/admins for maintaining such a superb site.

StainlessSteelRat
Posts: 12
Joined: Mon May 21, 2018 7:18 am

Re: Suggested changes after AEL front-end development

Post by StainlessSteelRat » Sat Aug 24, 2019 9:48 pm

The point of the low number of calls is to force high traffic users to mirror the data.

I’m not sure if you were with us before, but the serever was completely overloaded and therefore unstable. Sometimes going down for a very long time. The majority of those abusive calls came from the rom scrapers.

While it does increase complexity, but if you mirror, then your program will essentially have zero limits.

Wintermute0110
Posts: 3
Joined: Thu Jun 27, 2019 9:53 am

Re: Suggested changes after AEL front-end development

Post by Wintermute0110 » Wed Aug 28, 2019 4:31 am

StainlessSteelRat wrote:
Sat Aug 24, 2019 9:48 pm
The point of the low number of calls is to force high traffic users to mirror the data.

I’m not sure if you were with us before, but the serever was completely overloaded and therefore unstable. Sometimes going down for a very long time. The majority of those abusive calls came from the rom scrapers.

While it does increase complexity, but if you mirror, then your program will essentially have zero limits.
I have been around TGDB around 4 years now and yes, I noticed the server overload in the past. I apologise because I was partially responsible. Having an API in JSON is certainly much more efficient that scraping HTML code so definitely that was a very good step in reduce the server load.

Following you suggestion, I have implemented a disk cache so every piece of information downloaded with the API is stored locally and never downloaded again by the scraper, ever if the user decides to rescrape the collection. Stills, 1500 calls per month is very low. Suppose a user have a Genesis (2000 ROMs) and a SNES (3000 ROMs) collection to scrape. Every ROM scrap consumes 2 API calls, so it will require more than 3 months for the user to scrape the collection. With the disk cache, users will consume a lot of API calls when scanning their collection and then nothing, so I think is reasonable to increase the API limit so users can fill the local disk cache.

Besides, issues b) ability to check API allowance even when expired and c) differentiate between Title ans Snap screenshots still are of interest. Any comments to these?

User avatar
Zer0xFF
Posts: 264
Joined: Fri Apr 20, 2018 9:18 am

Re: Suggested changes after AEL front-end development

Post by Zer0xFF » Sat Sep 28, 2019 1:30 pm

yes, I should introduce an api call that only returns the limit (without using any of it).

as for rom scrappers, we added UID (unique ID) which can be used to batch request games.
for example PS2 games UID is whats know as serial number SLES-30231,SLUS-12345, which can be found in the disc data.
however, its a bit more complicated with old games such as (S/)NES since no such info existed, so we'd hopefully introduce hashes(/crc32), which will work the same way. however I know that it will be a slow process to populate the database.

and to readdress the limit issue, as per StainlessSteelRat reply, we're trying to strike a balance between limit and usability,
so we based on usage and server load, we have in the past moved the limit from 1000 to 1500 and we will consider bumping up soon.

just as well, Im trying to make server mirroring easier, 1 request, to allow high traffic apps to easily setup their own api stations.
Regards
Zer0xFF

Post Reply