NES and SNES games not scraping (the games db scraper)



  • Guys this os driving me nuts, all systems are scraping fine and quick except NES and SNES… is anybody having this problem? Any way of fixing this???


  • moderators

    Are they not scraping at all, or are you seeing major delays first?
    If a couple games are scraping, it’s possible the API server is timing out.
    If nothing is scraping at all, we may have to drop stale queries.



  • Fist it loads for a few second and then is just doesnt scrape at all, but only with NES and SNES wich is weird


  • moderators

    Does it stall at the same title, or does it actually go a bit further each time?
    If it’s the same title, that entry may be malformed or incompletely removed.



  • For example genesis scrapas and rescrapes fine and quick but the nes games and snes they dont, theg dont rescrape either


  • moderators

    I’m asking that specific question for a reason: we’ve seen both behaviours, but the two solutions are very different.

    If the scrape fails on the same title every time, there’s likely a malformed entry that will need to be fixed or removed.

    If the scrape goes a bit further each time, there may be a server issue that requires a reboot to fix.



  • Hi there, I am having the same problem. All the mega drive games scrape quickly but NES and SNES won’t scrape at all. I just get no games found. If it help, I just installed retropie yesterday so the system had never been scraped before.I deleted all the games and re-loaded them and it still wouldn’t work. Thanks for the help!



  • Glad I wasnt the only one! Got a little worry there… hope is fixed soon, i have a huge collectio didnt wanna rescrape again


  • moderators

    I’ve put in an RFC with the development team, hopefully we’ll get to the bottom of this relatively soon.

    Update #1:
    The CloudFlare endpoint for the Philadelphia area seems to be experiencing undefined issues.
    https://www.cloudflarestatus.com/
    If you’re connecting from this area, this may be part of the problem; however, it seems unlikely.



  • @LeoPride Thank you. I appreciate all you guys do.



  • @LeoPride Im in California and its still not working for me…



  • @nestor1924 I’m in Illinois and still no luck for NES or SNES. What’s weird though is that it works on my other pie that has been scraped previously. I added new NES and SNES games and they scraped just fine.



  • @rallying83 damn so is it a hardware problem or what? Damn i hope not… are you using the same SD card for both pis???



  • @nestor1924 Yup they are identical with SD cards.



  • @rallying83 so you have an sd card for each pi? Did you try using the one that is not scraping on the other pi? Man this is driving me nuts, there has to be a way tl fix this, i dont wanna getba second pi…


  • moderators

    The upside of the Pi (and this covers any SD-based product, so Galileo and Curie aren’t immune either) is also its greatest weakness: removable flash media is notoriously susceptible to data corruption. As it’s removable - and, hence, easily replaceable - troubleshooting doesn’t necessarily require replacing the system itself, but it does somewhat complicate matters. Personally, I’ve corrupted two cards (one in RetroPie, the other in OSMC) before learning that flash media isn’t particularly well-suited for database storage (I still got a couple 32-gig Class 10 SanDisks on sale, because I’ll be damned if my computers dictate how I use them). However, in so doing, I’ve been trying to effectively transfer the databases to my NAS (with some success), for the sole reason that mechanical drives are designed to do this for years on end - I transferred my games there after previously losing years of archival work.

    Now, all that said, it is at least possible that an update stuffed up privs somewhere along the line.
    If you can SSH into your Pi (use PuTTy on Windows), you can check privs as follows:
    ls -lh ~/RetroPie/roms | grep nes

    The important bit is the first column, which should read
    drwxr-xr-x (or, possibly, 0755)
    If it doesn’t, there’s a good chance that the scraper can’t even access the directory.



  • I have no idea of what you just said… can you be more specific on how to fix this problem?.. this is a new sd card im usin cause the last one wasnt working so i transfer the image to this new one 3 dags ago…



  • This post is deleted!


  • @LeoPride this is what i got after i put this : ls -lh ~/RetroPie/roms | grep nes
    https://imgur.com/a/BkYeQ

    what does that mean?


  • moderators

    What it means is that directory privs are not the problem, here’s why:

    drwxr-xr-x 2 pi pi

    d is the directory bit.
    The first three characters (rwx, in this case) are owner privileges (read/write/execute).
    The second three characters (r-x) are group privileges (read and execute, no write).
    The final three characters (again, r-x) are global privileges (anyone who’s not owner or in the relevant group).
    I don’t generally give the digit much thought, but I doubt it’s at issue here.
    “pi” is, in the order listed, the owner and the group assigned to those directories listed.

    My first concern here is that only shows us data for the directories in question.
    I would try the same thing as before, only on the megadrive directory, to see if it’s different:
    ls -lh ~/RetroPie/roms | grep megadrive
    If it is, there’s the issue; if not, there’s a whole list of other things that could be causing the issue you’re seeing:

    • Were the files downloaded from a website (could be Dropbox or any number of sites, this isn’t meant as a trap) or transferred from another drive? If it was another drive, what type and OS?
    • Do you recall the scraper working before a particular time? Regressions do happen, and this can help to pinpoint the issue, whether it lies in the distribution or our site.
    • Is the Pi set up to use IPv6 by default? This is another longshot, but I’ve had too many issues stem from accidental partial updates to not ask.

    EDIT TO NOTE
    My systems are doing the same thing now, I strongly suspect a regression in the scraper and will look into this over the weekend (for added clarity, I checked: our last code revision of any type was at the end of 2016).



  • @LeoPride ill test the megadrive when i get home cause im at work at the moment, also are you saying this is happening to you aswell? Are there systems that are not scraping at all?


Log in to reply