I am wondering how many simultaneous requests per end user you feel is appropriate? Like I can scrape 1 game at once, or 20. Is there any guideline on this?
General étiquette would probably be one game at a time so you don’t hog resources.
Ok cool, I’d like to do maybe 3 at a time though, that would give an enormeous perfomance boost and still maybe not be too bad, or would that not be cool?
@Luke I suppose it depends in what way you are going to do that? Under what circumstances are you going to scrape? You should minimize all strain on the server, and that certainly includes multiple requests at a time.
It sounds like what you are after is having nearly all of the data, which means if you have every user do this, you’ll be a major resource hog.
You’ll want to scrape everything you care about, remove the rest, and create a small resource pack that just your clients can request. That would be the largest performance boost you could ask for, and also protects against when the site goes down (because the resources are being hogged by someone else!).