I've developed a custom data scraper for my own personal use to automate EmulationStation scraping on Arch Linux, without the whole RetroPie custom OS silliness. It'll remain in use for just one machine, on just one dataset. If that changes, and I decide it's stable enough and worthy of release, I'll re-apply for a new API key before I do anything crazy like release it. I'm aware you're just ensuring no one makes a profit off of your bandwidth and server infrastructure (or data by ripping it and re-uploading it) without sharing some back, and do not intend to do any such thing.
If you're interested in the technical details, the script has not yet been updated to the new API, but it previously had searched for the best match from the file name alone, waited a full second, then used the game ID of the best match to cache an XML file to a permanent location. After parsing an entire system that way, it then downloaded the front cover artwork (again to a permanent location, skipping previously downloaded artwork) for each game from that XML data, and generated the EmulationStation datafile from that permanent XML cache. In the event that no suitable match was found, it stored an empty XML file to prevent a duplicate searches on subsequent runs.
The gist is, each game gets searched only once ever, there's a minimum one second delay between general search and data retrieval from game ID, and no game is ever re-searched unless a file is physically deleted by hand. For each game, that's two DB hits and a single image download max, one time only. Oh, and as this is a BASH shell script, even if there were a malfunction the code moves so slowly it shouldn't cause much issue anyway. There's at least a half second delay each time CURL spins up, in addition to all the other delays.
d33733t
API Request for custom private my-use-only scraper
Forum rules
API Keys (unique userpage)
Usergroups (request API Users access here after posting a new topic below)
Please provide a basic overview of your proposed use of the API.
Any request lacking this information will be denied pending an appropriate response.
Please do not request keys as end users of another project, as those projects should request and properly implement their own keys. Key requests of this nature will be removed in the interest of more efficiently handling project requests. Assistance may be available upon contact for projects experiencing issues utilizing the current API.
API Keys (unique userpage)
Usergroups (request API Users access here after posting a new topic below)
Please provide a basic overview of your proposed use of the API.
Any request lacking this information will be denied pending an appropriate response.
Please do not request keys as end users of another project, as those projects should request and properly implement their own keys. Key requests of this nature will be removed in the interest of more efficiently handling project requests. Assistance may be available upon contact for projects experiencing issues utilizing the current API.
Re: API Request for custom private my-use-only scraper
Hi there,
You've been granted an API key which you can access through this page.
if you ever go public with your project, please post in this thread notifying us of that,
note if you go public you'll continue using the same key, we just request to be informed out of interest.
please check the Announcement forum on regular bases to ensure compliance with any changes or updates.
Regards
Zer0xFF
You've been granted an API key which you can access through this page.
if you ever go public with your project, please post in this thread notifying us of that,
note if you go public you'll continue using the same key, we just request to be informed out of interest.
please check the Announcement forum on regular bases to ensure compliance with any changes or updates.
Regards
Zer0xFF
Regards
Zer0xFF
Zer0xFF