ARC Browser

This is where you can request API access. Please be sure to mention what your use case is and if possible a link to the project.
Please provide a basic overview of your proposed use of the API.
Any request lacking this information will be denied pending an appropriate response.
Forum rules
Please provide a basic overview of your proposed use of the API.
Any request lacking this information will be denied pending an appropriate response.
Post Reply
User avatar
freelancer
Posts: 17
Joined: Fri Jun 29, 2018 4:53 pm
Location: Sweden

ARC Browser

Post by freelancer » Fri Jun 29, 2018 5:00 pm

Hi awesome people!

I'm the developer of ARC Browser, an emulator frontend for Android. Currently using your old API, but I'm of course interested in switching to the new API once it comes out of beta.

I'd like to know more about how you intend to limit API/key access, if you've made any decisions yet. Right now, using the old API, users of my app have two ways of scraping data from your database:

1) Directly using your API from the app. This is not the default.
2) Using my own scraping/search/metadata servers dedicated to my app. This is the default. My servers have their own database with all the game metadata from your database, and it's updated daily using the (old) Updates.php API. The artwork is still referred to your servers, but all metadata and search requests are handled by my servers.

What are your thoughts on this? I don't wanna end up in a situation where all my users are prohibited from accessing the database just because one user used up all the requests for my API key.

Also, I need an API key =) But I suppose that can wait until the new API is in production.

User avatar
Zer0xFF
Posts: 287
Joined: Fri Apr 20, 2018 9:18 am

Re: ARC Browser

Post by Zer0xFF » Fri Jun 29, 2018 5:59 pm

Hi freelancer,

Thanks for the interest in the new API, it's great to see someone offloading the load to their own server as thats a problem we had with the old API.
though we have nothing set in stone yet, as we'd like to see dev use the api and with their feedback we'll change it hopefully for the better of everyone.

a quick summary of how we currently have it setup, is that you'd receive an API keys which will work on a key/ip limit, in that each ip will have it's own limit per app, the current limit is set to 1000 requests a month, based on how your app utilise it, example requesting games via their id or through search, you could in theory request upto 20,000 game info.

but your use case is getting me to consider a 2nd private API key, with perhaps a higher limit, which can be used server side as oppose to client/app side

anyway, the new API is bound for release soon(TM), we have recently started redirecting to the beta site and the old legacy site/api is currently read only, once the switch is official you'll be granted an API key.

would love to hear any idea you might have, in this regards.

Regards
Zer0xFF
Regards
Zer0xFF

User avatar
freelancer
Posts: 17
Joined: Fri Jun 29, 2018 4:53 pm
Location: Sweden

Re: ARC Browser

Post by freelancer » Sat Jun 30, 2018 9:18 am

I'm not sure how you'd get 20000 games from 1000 requests, but maybe your math is better than mine. I can't load games by ID from the app as it doesn't yet know the IDs during scraping, so it has to search for every game. I think most monthly limits could potentionally be a problem for the power users though, sometimes I get new users who immediately want to scrape data for 30000 roms.

HOWEVER, thinking some more about it, maybe I should completely disable direct access from the app and just have everything go through my servers? I feel like that would benefit everyone. Less load on your servers, and me and my users wouldn't have to worry about running into any request limits. And I already have the infrastructure for it.

A private key sounds like a great idea. It could have a higher request limit, but would count it globally instead of per IP. Initially I'd (preferrably) need a higher-than-normal limit because I'd want to update every single game to make sure my database is up-to-date. After that though, I'd just make one request to check for updates every day and then download the data for any game that has been updated since my last successful request plus one day (I always add one extra day even though this means downloading most updated games' info twice, just to insulate myself against any timezones bugs in my code or yours or anything inbetween).

In the old API, Updates.php only returns a list of game IDs, and then I have to request the actual game info with new requests. Looking at the new API docs though, it looks like /Games/Updates could return all of the game info in the Updates request? If so, I would literally only have to make one or two requests per day. That would be great.

Leo_Pride
Posts: 461
Joined: Mon Apr 23, 2018 2:10 am

Re: ARC Browser

Post by Leo_Pride » Sat Jun 30, 2018 10:05 am

Just a quick note, per my understanding, each request can return up to 20 game listings, so 20000 from 1000. :)
We're curious to see how new projects use TGDB API.
If you have a new public project, please provide a link to it so we can highlight cool new applications! 8-)

User avatar
Zer0xFF
Posts: 287
Joined: Fri Apr 20, 2018 9:18 am

Re: ARC Browser

Post by Zer0xFF » Sun Jul 01, 2018 12:08 am

yes, update has been changed, and it's likely to be updated again in the next few days, and it will start returning actual user edits...

edit: this is what the updates API would now return, it's currently set to return 100 changes per page

Code: Select all

{
  "code": 200,
  "status": "Success",
  "data": {
    "count": 22,
    "updates": [
      {
        "game_id": 60006,
        "timestamp": "2018-06-30 12:36:29",
        "type": "game",
        "value": "[NEW]"
      },
      {
        "game_id": 60006,
        "timestamp": "2018-06-30 12:36:29",
        "type": "game_title",
        "value": "Ys VIII: Lacrimosa Of DANA Limited Edition"
      },
      {
        "game_id": 60006,
        "timestamp": "2018-06-30 12:36:29",
        "type": "overview",
        "value": "Ys returns with a brand new adventure for the first time in 8 years! Adol awakens shipwrecked and stranded on a cursed island. There, he and the other shipwrecked passengers he rescues form a village to challenge fearsome beasts and mysterious ruins on the isolated island. Amidst this, Adol begins to dream of a mysterious blue-haired maiden living in an unknown world. Join Adol as he unravels the riddle of the cursed isle and the blue-haired maiden Dana in Ys VIII: Lacrimosa of DANA!\r\n\r\nAn Epic Tale - Follow Adol on his latest adventure and uncover the mysteries of the cursed island.\r\nParty On! – Change your playable character on-the-fly as you battle ferocious foes in lighting-fast combat.\r\nAnother Side – Unravel the mystery of the blue-haired maiden Dana through her unique gameplay sections.\r\nIt Takes a Village! – Rescue your fellow shipwrecked passengers and bring them to your village where they will provide valuable services."
      },
      {
        "game_id": 60006,
        "timestamp": "2018-06-30 12:36:29",
        "type": "release_date",
        "value": "2018-06-26"
      },
      {
        "game_id": 60006,
        "timestamp": "2018-06-30 12:36:29",
        "type": "players",
        "value": "1"
      },
      {
        "game_id": 60006,
        "timestamp": "2018-06-30 12:36:29",
        "type": "coop",
        "value": "No"
      },
      {
        "game_id": 60006,
        "timestamp": "2018-06-30 12:36:29",
        "type": "developer",
        "value": "NIHON FALCOM"
      },
      {
        "game_id": 60006,
        "timestamp": "2018-06-30 12:36:29",
        "type": "publisher",
        "value": "NIS AMERICA, INC."
      },
      {
        "game_id": 60006,
        "timestamp": "2018-06-30 12:36:29",
        "type": "youtube",
        "value": "https://www.youtube.com/watch?v=-hCyZdbz9pI"
      },
      {
        "game_id": 60006,
        "timestamp": "2018-06-30 12:37:39",
        "type": "boxart",
        "value": "boxart/front/60006-1.jpg"
      },
      {
        "game_id": 60006,
        "timestamp": "2018-06-30 12:37:59",
        "type": "boxart",
        "value": "[REMOVED]"
      },
      {
        "game_id": 60006,
        "timestamp": "2018-06-30 12:37:59",
        "type": "boxart",
        "value": "boxart/front/60006-1.jpg"
      },
      {
        "game_id": 60007,
        "timestamp": "2018-06-30 12:44:01",
        "type": "game",
        "value": "[NEW]"
      },
      {
        "game_id": 60007,
        "timestamp": "2018-06-30 12:44:01",
        "type": "game_title",
        "value": "The Lost Child Limited Edition"
      },
      {
        "game_id": 60007,
        "timestamp": "2018-06-30 12:44:01",
        "type": "overview",
        "value": "Occult journalist Hayato Ibuki encounters a strange girl during his investigations into a series of mysterious suicides in Tokyo. She implores that Hayato must live, before bestowing upon him a mystical device that allows him to capture celestial and arcane creatures. He is now thrust into a world beyond our own, home to horrific monsters and celestial beings. In it, he must delve into dungeons called Layers and solve the mystery of the device. Where his journey leads him could decide the fate of our world...\r\n\r\n• Mythology Unbound -- Features 50+ entities from myths and legends to capture, each with 3 unique forms to progress through!\r\n• Spread the Wealth of Knowledge -- There are more than 250 skills to learn and distribute among your party of captured Astrals.\r\n• The Infinite Abyss Awaits -- Test yourself against massive dungeons with up to 99 floors of deathly dangers!"
      },
      {
        "game_id": 60007,
        "timestamp": "2018-06-30 12:44:01",
        "type": "release_date",
        "value": "2018-09-19"
      },
      {
        "game_id": 60007,
        "timestamp": "2018-06-30 12:44:01",
        "type": "players",
        "value": "1"
      },
      {
        "game_id": 60007,
        "timestamp": "2018-06-30 12:44:01",
        "type": "coop",
        "value": "No"
      },
      {
        "game_id": 60007,
        "timestamp": "2018-06-30 12:44:01",
        "type": "developer",
        "value": "KADOKAWA GAMES"
      },
      {
        "game_id": 60007,
        "timestamp": "2018-06-30 12:44:01",
        "type": "publisher",
        "value": "NIS AMERICA, INC."
      },
      {
        "game_id": 60007,
        "timestamp": "2018-06-30 12:44:01",
        "type": "youtube",
        "value": "https://www.youtube.com/watch?v=TIB1uHpawv4"
      },
      {
        "game_id": 60007,
        "timestamp": "2018-06-30 12:44:13",
        "type": "boxart",
        "value": "boxart/front/60007-1.jpg"
      }
    ]
  },
  "pages": {
    "previous": null,
    "current": "https://api.beta.thegamesdb.net/Games/Updates?apikey=Zer0&time=510&page=1",
    "next": null
  },
  "remaining_monthly_allowance": 962,
  "extra_allowance": 0
}
it will also return game deletion, image uploads etc etc etc
Regards
Zer0xFF

StainlessSteelRat
Posts: 12
Joined: Mon May 21, 2018 7:18 am

Re: ARC Browser

Post by StainlessSteelRat » Sun Jul 01, 2018 9:18 am

Am I the only one that thinks the update API is overcomplicated? It could be a simple bit mask representing major categories for a game (i.e, art changed, metadata changed, game deleted, etc...) or something like that.

Then you wouldn't even have to worry about paging since you won't have large fields like overviews crowding up the response (another over-complication IMO). I can't imagine anyone needing that granular of a change format.

I mean, it in essence forces us to have a complex parser for each conceivable field of a game. Changes in format will reek havoc. Changes in field name's will reek havoc.

User avatar
freelancer
Posts: 17
Joined: Fri Jun 29, 2018 4:53 pm
Location: Sweden

Re: ARC Browser

Post by freelancer » Sun Jul 01, 2018 12:01 pm

If it was just a simple bitmask, it'd be really hard to change the fields because changing a single bit to mean something else would create incompatibilities/inconsistencies that might not be noticed by the API user. And if the actual changed data wasn't returned in the Updates request, you have to make additional requests to actually get it. And changes in format will always cause problems, no matter how simple the current format is ;)

I think I could get behind the Updates API as described, as long as I could regularly get all the changes without cutting it too close to a request limit. But I'm confident we can work that out.

I'm not entirely sure why there's one object per field instead of one object per game though. For example, why this:

Code: Select all

{
        "game_id": 60007,
        "timestamp": "2018-06-30 12:44:01",
        "type": "game",
        "value": "[NEW]"
      },
      {
        "game_id": 60007,
        "timestamp": "2018-06-30 12:44:01",
        "type": "game_title",
        "value": "The Lost Child Limited Edition"
      },
      {
        "game_id": 60007,
        "timestamp": "2018-06-30 12:44:01",
        "type": "overview",
        "value": "Occult journalist Hayato Ibuki encounters a strange girl during his investigations into a series of mysterious suicides in Tokyo. She implores that Hayato must live, before bestowing upon him a mystical device that allows him to capture celestial and arcane creatures. He is now thrust into a world beyond our own, home to horrific monsters and celestial beings. In it, he must delve into dungeons called Layers and solve the mystery of the device. Where his journey leads him could decide the fate of our world...\r\n\r\n• Mythology Unbound -- Features 50+ entities from myths and legends to capture, each with 3 unique forms to progress through!\r\n• Spread the Wealth of Knowledge -- There are more than 250 skills to learn and distribute among your party of captured Astrals.\r\n• The Infinite Abyss Awaits -- Test yourself against massive dungeons with up to 99 floors of deathly dangers!"
      },
      {
        "game_id": 60007,
        "timestamp": "2018-06-30 12:44:01",
        "type": "release_date",
        "value": "2018-09-19"
      }
instead of this:

Code: Select all

{
        "game_id": 60007,
        "timestamp": "2018-06-30 12:44:01",
        "status": "[NEW]",
        "game_title": "The Lost Child Limited Edition",
        "overview": "Occult journalist Hayato Ibuki encounters a strange girl during his investigations into a series of mysterious suicides in Tokyo. She implores that Hayato must live, before bestowing upon him a mystical device that allows him to capture celestial and arcane creatures. He is now thrust into a world beyond our own, home to horrific monsters and celestial beings. In it, he must delve into dungeons called Layers and solve the mystery of the device. Where his journey leads him could decide the fate of our world...\r\n\r\n• Mythology Unbound -- Features 50+ entities from myths and legends to capture, each with 3 unique forms to progress through!\r\n• Spread the Wealth of Knowledge -- There are more than 250 skills to learn and distribute among your party of captured Astrals.\r\n• The Infinite Abyss Awaits -- Test yourself against massive dungeons with up to 99 floors of deathly dangers!",
        "release_date": "2018-09-19"
      }
? Is it because different fields can have different timestamps? In your example it looks like every field belonging to the same game has the same timestamp. I can handle both formats, but the latter would definitely be easier to parse.

User avatar
Zer0xFF
Posts: 287
Joined: Fri Apr 20, 2018 9:18 am

Re: ARC Browser

Post by Zer0xFF » Sun Jul 01, 2018 4:07 pm

I'm not entirely sure why there's one object per field instead of one object per game though.
Is it because different fields can have different timestamps? In your example it looks like every field belonging to the same game has the same timestamp.
yes they can and will have different timestamps, but you're looking at it the wrong way, but suppose the example that happens to be is also misleading.

the example above just so happens to be the last update to the database, which happens to be a user just adding a new game entry as such the `[New]` tag followed by all the other fields(so that you don't need to make another separate request for that game at all), however, the api will also return individual changes to different games at the same time, which obviously complicates things when it comes to returning a single game object, this seemed like the better compromise.

@StainlessSteelRat, as @freelancer said, simplifying the api as you suggest would have the knock on effect of complicating it from another point of view, aka more requests.

edit: grammer
Regards
Zer0xFF

Leo_Pride
Posts: 461
Joined: Mon Apr 23, 2018 2:10 am

Re: ARC Browser

Post by Leo_Pride » Sun Jul 01, 2018 6:35 pm

Taking all that into consideration, a bitmask may be appropriate for, say, different Ratings systems. Beyond that, perhaps a timestamp check is in order, so identically stamped fields are returned using the slightly more monolithic format to save calls. Something for the future, at least.
We're curious to see how new projects use TGDB API.
If you have a new public project, please provide a link to it so we can highlight cool new applications! 8-)

User avatar
Zer0xFF
Posts: 287
Joined: Fri Apr 20, 2018 9:18 am

Re: ARC Browser

Post by Zer0xFF » Wed Jul 04, 2018 3:22 am

freelancer,

You've been given access to the API and you can find your key here http://api.thegamesdb.net/key.php

Regards
Zer0xFF
Regards
Zer0xFF

Post Reply