Are you really supposed to use the api with your own server?

Need help with your application? Ask here.
Post Reply
RVDSoft
Posts: 7
Joined: Thu Jul 05, 2018 8:37 am

Are you really supposed to use the api with your own server?

Post by RVDSoft » Wed Sep 26, 2018 1:11 pm

Hi

I'm still a bit confused about the use case of the new API. Should developers without their own dedicated server move on to something else? Or is the 1000/month query limit going to change in the future if the server load is low? It would be more logical if the developer field and publisher field returned a name with the id as opposed to genres which can be looked up locally from your cached genres. I understand you guys want to keep the server load to an absolute minimum in the beginning to see how things go. And If thinigs keep going well, increase the query limit and/or query results a little bit. Then see how that goes. Is the plan something along these lines?

Also is their any broad estimate on when the old api will be shutdown? Like within two years?

regards,

RVDSoft

User avatar
Zer0xFF
Posts: 240
Joined: Fri Apr 20, 2018 9:18 am

Re: Are you really supposed to use the api with your own server?

Post by Zer0xFF » Tue Oct 02, 2018 11:34 pm

Hey,

the public key is rate limited per IP (which roughly translate to per user)(perhaps I'd need to reword the description), so that 1000 request limit and count down applies to you and you only, anyone else will have their own 1000 request/count down.
ideally we'd like you to have your server, but as long as you can stick to the 1000 request limit, it shouldn't be an issue.

as for when the legacy server shutdown,
we don't have a date. the legacy server was just there to allow existing apps to continue to work as dev transit to the new api, (it is not intended for new apps/devs), as such it was always a short term solution, though we don't have a date, I can assure you we won't keep it up for a full year.(however, since this was brought up, I do feel like we (internally) would need to discuss plans for shutdown)

Regards
Regards
Zer0xFF

RVDSoft
Posts: 7
Joined: Thu Jul 05, 2018 8:37 am

Re: Are you really supposed to use the api with your own server?

Post by RVDSoft » Thu Oct 04, 2018 7:52 pm

I know it's rate limited per IP but I to my estimation it's not enough for my needs. So I was wondering if there's a chance you'll ever increase the limit? Not for the sake of it but if a lot of developers would be complaining about the lilmit?

Leo_Pride
Posts: 330
Joined: Mon Apr 23, 2018 2:10 am

Re: Are you really supposed to use the api with your own server?

Post by Leo_Pride » Fri Oct 05, 2018 2:54 am

Every query can return 20 results, 10 if you count full artwork.
If I remember correctly, the one-time full sync covers 6000 queries, coming to 120,000 entries with basic data, 60,000 with full artwork. That's more entries than exist in the current database.
It's also a good reason to test on the per-IP key, which limits out at 1000 / 20000 / 10000 but resets every month.
We're curious to see how new projects use TGDB API.
If you have a new public project, please provide a link to it so we can highlight cool new applications! 8-)

User avatar
Zer0xFF
Posts: 240
Joined: Fri Apr 20, 2018 9:18 am

Re: Are you really supposed to use the api with your own server?

Post by Zer0xFF » Fri Oct 05, 2018 4:07 pm

RVDSoft wrote:
Thu Oct 04, 2018 7:52 pm
I know it's rate limited per IP but I to my estimation it's not enough for my needs. So I was wondering if there's a chance you'll ever increase the limit? Not for the sake of it but if a lot of developers would be complaining about the limit?
if 1000 request per IP is too little, you're doing something fundamentally wrong, either your app is making too many requests, or maybe by some chance your users have too many games, but if they do, you should really be looking for a smart solution into how to ping our server for requests.

so far, you're only the 3rd dev to think this is an issue (out of 125+), and with that said the other 2 complained before using the api, and they just deemed it too limited, they haven't said anything afterwards.

if you do need a higher limit, then the best option for us, you and your users, is to mirror the database and host it yourself. (if you used the old site/api before, you would have know that it at some time can take up to 5 minutes for a page or an api to return any results, which is something we'd like to avoid)
Regards
Zer0xFF

RVDSoft
Posts: 7
Joined: Thu Jul 05, 2018 8:37 am

Re: Are you really supposed to use the api with your own server?

Post by RVDSoft » Sun Oct 07, 2018 9:55 pm

if 1000 request per IP is too little, you're doing something fundamentally wrong, either your app is making too many requests, or maybe by some chance your users have too many games, but if they do, you should really be looking for a smart solution into how to ping our server for requests.

so far, you're only the 3rd dev to think this is an issue (out of 125+), and with that said the other 2 complained before using the api, and they just deemed it too limited, they haven't said anything afterwards.

if you do need a higher limit, then the best option for us, you and your users, is to mirror the database and host it yourself. (if you used the old site/api before, you would have know that it at some time can take up to 5 minutes for a page or an api to return any results, which is something we'd like to avoid)
I see, I'll have to figure something out.

I have two more questions:
1) I have read in another thread that there are plans to increase the result limit from 20 to 100?
2) To read the publishers and developers, I need to query all the publishers and developers beforehand and cache them somewhere. Am I supposed to store them locally in a database on the phone of the user? If so, how often do I need to redownload them?


EDIT: I answered the second question myself. I created a sqlite db with both the publishers and developers whichs comes down to about 350kb. If I just save them as text and then compress it into a zip file, then it comes down to 110kb. That's pretty reasonable in 2018. I could redownload the developers or publishers when I come across an id that's not stored in the cache.

User avatar
Zer0xFF
Posts: 240
Joined: Fri Apr 20, 2018 9:18 am

Re: Are you really supposed to use the api with your own server?

Post by Zer0xFF » Mon Oct 08, 2018 9:24 am

1) I have read in another thread that there are plans to increase the result limit from 20 to 100?
the update API returns 100 changes per page
2) To read the publishers and developers, I need to query all the publishers and developers beforehand and cache them somewhere. Am I supposed to store them locally in a database on the phone of the user? If so, how often do I need to redownload them?
ummm, this sounds to me like you're making more than 2 requests? calling `/Developers` would return all devs.
you wouldn't even need to download and compress them(you could), just have the user retrieve them on 1st boot, you'd only be using 2 requests, then update them as you mentioned whenever you come across a new entry.
Regards
Zer0xFF

Leo_Pride
Posts: 330
Joined: Mon Apr 23, 2018 2:10 am

Re: Are you really supposed to use the api with your own server?

Post by Leo_Pride » Tue Oct 09, 2018 4:14 am

I could swear we answered this once before, am I losing my mind? :lol:
We're curious to see how new projects use TGDB API.
If you have a new public project, please provide a link to it so we can highlight cool new applications! 8-)

RVDSoft
Posts: 7
Joined: Thu Jul 05, 2018 8:37 am

Re: Are you really supposed to use the api with your own server?

Post by RVDSoft » Tue Oct 09, 2018 8:38 pm

the update API returns 100 changes per page
Ah I must have misread it.
ummm, this sounds to me like you're making more than 2 requests? calling `/Developers` would return all devs.
you wouldn't even need to download and compress them(you could), just have the user retrieve them on 1st boot, you'd only be using 2 requests, then update them as you mentioned whenever you come across a new entry.
No just 2.

I could swear we answered this once before, am I losing my mind? :lol:
Just noticed there's a thread made already about publishers and developers but it didn't exactly answer my question. I was confused at first on why I needed to cache 1MB+ of data to read a couple of fields. Then it turned out it's about 10 times less when converted to csv and compressed. I also have to remind myself that you guys are not a company and you are taking special measures to keep server load as low as possible.



Btw I have noticed you guys still haven't updated your twitter or facebook about the new API. I wish you guys would have mentioned the new API in june because I had to find out about the new API through my users' bug reports and then had to quickly rush an update out. Is there any way I can receive updates about the API without checking the forum?

User avatar
Zer0xFF
Posts: 240
Joined: Fri Apr 20, 2018 9:18 am

Re: Are you really supposed to use the api with your own server?

Post by Zer0xFF » Wed Oct 10, 2018 9:13 am

Btw I have noticed you guys still haven't updated your twitter or facebook about the new API. I wish you guys would have mentioned the new API in june because I had to find out about the new API through my users' bug reports and then had to quickly rush an update out. Is there any way I can receive updates about the API without checking the forum?
sorry about that, social media announcement escaped me, since i know they haven't been active in years, its also why we kept the legacy active until now, to allow quick hot fixing while having sometime to update to the new api.
i dont know who controls the social media accounts at the moment. i'll check and see what we can do. (hopefully there is a plugin to auto share announcements).
Regards
Zer0xFF

Post Reply