Page 1 of 1

Skyscraper api key request

Posted: Sun Apr 22, 2018 2:38 pm
by muldjord
Hi guys,
Congrats on the new forum, api and site so far :) I'm the developer of Skyscraper, a game scraper for Debian Linux derivatives such as RetroPie. I am not a Discord user, so I am unsure how to request for access to the new json api. So I thought a good way to start would be to post on here.

So I hereby request an API key.

More on Skyscraper here: https://github.com/muldjord/skyscraper

And a small summary: A powerful and versatile yet easy to use game scraper written in C++ for use with multiple frontends running on a Linux system. It scrapes and caches various game resources from various web sources, including media such as screenshot, cover and video.

Re: Skyscraper api key request

Posted: Sun Apr 22, 2018 6:09 pm
by Zer0xFF
Hi muldjord,

please check you PM for your key.
Few things to note that the keys will most likely be reset once the beta is finished, the current beta database is kept separate from the main site as such it will not be up to date, currently the api is subject to change, as for example filters work by table fields name which is a bit inconsistent when it comes to naming, as such will be changed in time.

final thing to note, currently the key is note made to work in public code (aka it should be kept hidden), but we're currently looking into the best way to allow it to be use publicly such as oauth or user keys, if you've any suggestions in this regard or anything else regarding the API, we'd love to hear from you.

Re: Skyscraper api key request

Posted: Sun Apr 22, 2018 7:08 pm
by muldjord
Thank you, I'll take note that it will probably be reset. Concerning keeping the key hidden, I use obfuscation to hide api keys in Skyscraper. In other words; the key is not shown in my code directly anywhere and it would require quite a bit of coding knowledge to understand how to de-obfuscate it. Is this enough for me to use it?

Thoughts on user keys, just my five cents: I have nothing against the idea personally, as it also means that people take time to actually register, and thereby potentially help fill up the databases, but it does provide a "stone on the road" for casual users which is a bit of a hit on usability. It's a give and take I guess. But ultimately it is your database and your decision, so I'll try and adjust to your design choices. :)

EDIT: I don't have authorization to send PM's. Is this by design (fine by me, I'm just wondering)? I wanted to explain a bit further about the obfuscation, but don't want to post it here.

Re: Skyscraper api key request

Posted: Sun Apr 22, 2018 8:06 pm
by Zer0xFF
newly registered users can't PM other members to prevent spam, I've changed the settings for you now, so you should be able to PM me.

the thing is, we're only asking you to keep the key private, because if not someone else can use your allocated requests for the month.

Re: Skyscraper api key request

Posted: Sun Apr 22, 2018 9:05 pm
by muldjord
Oh ok. I'll have to figure out a way to deal with that. Depending on how many users a given software has, that would then potentially pose a problem each month for all users. I certainly understand the need for a limit, but I guess I'm thinking it should be on a user level, not at developer level. Just my five cents.

I could potentially have one user who decides to scrape 24/7 and shut down the use of Skyscraper for everyone else after only a few days, rendering the software unusable. I can't control that on my end unfortunately. With user based access and limits, that would be more balanced.

I currenly support this for other modules. Api key allows access in general, and then the users provide their personal credentials (in a user-read-only config file if they so choose) which set the limits.

Re: Skyscraper api key request

Posted: Sun Apr 22, 2018 9:55 pm
by Zer0xFF
muldjord wrote:
Sun Apr 22, 2018 9:05 pm
Oh ok. I'll have to figure out a way to deal with that.
I don't think this is something YOU'd need to deal with, well at least not at the moment, since this is a beta and we've still haven't decided how to manage the users on the other end of this (but if you do want to use this in production, then yes, you'd need to deal with it). Note that as a beta there could be API breaking changes, though we'd keep those to a minimal.
muldjord wrote:
Sun Apr 22, 2018 9:05 pm
Depending on how many users a given software has, that would then potentially pose a problem each month for all users. I certainly understand the need for a limit, but I guess I'm thinking it should be on a user level, not at developer level. Just my five cents.
I just want to say, I'm also a user of this API, and it's how I joined the team for the refresh. but my opinion on the matter is different, we're here to provide the devs access to the API and it's their job to manage their users, not ours, but I also understand the kind of restriction that will cause, because I'll have to deal with it myself (unfortunately from both ends). which is also why I'm open for suggestions.
muldjord wrote:
Sun Apr 22, 2018 9:05 pm
I could potentially have one user who decides to scrape 24/7 and shut down the use of Skyscraper for everyone else after only a few days, rendering the software unusable. I can't control that on my end unfortunately. With user based access and limits, that would be more balanced.
and thats what we're facing right now, just imagine that software is our database.

Re: Skyscraper api key request

Posted: Mon Apr 23, 2018 6:53 am
by muldjord
we're here to provide the devs access to the API and it's their job to manage their users, not ours
I agree with that in the sense that scrapers should be able to retrieve the restrictions for any given user from the database and enforce them correctly. In other words, if a user has scraped more than their limit, the database would return a "no more requests available" (monthly requests would make sense here) response and the scraper should act on that accodingly, thereby making it the responsibility of the scraper developer to deal with those server respones. That much is clear.
But I do not agree that it should be set on a developer key level instead of a user credentials level. It would be very hard for me to handle that for my users. I would have to save the number of requests they've made locally, which could easily be reset by editing a local file. The other solution would be to enforce a "requests per hour" limit directly in my software, which would only serve to make scraping really slow.

But as I said, it's your database, and as I understand nothing is currently set in stone. Some of the stuff I mention here is probably not even what you meant I am still a bit unsure if I understand you correctly. Either way, come the end of beta, I will try to follow your guidelines/restrictions as best I can - not matter what they might be.

I am not aware if you know about Skyscraper, but I put a lot of effort into caching scraped data for later use to ease the load on the sources. Basically any piece of information that Skyscraper scrapes, is labeled with type, source (in your case 'thegamesdb') and timestamp and saved locally. Then, if the user decides to rescrape a file later, it will automagically use the locally cached data instead of refetching from the source. This is completely transparent to the user.

Lastly, thank you for your work on this.