Punctuation causes problems

  • After days of attempting to clean up my retropie data I’ve realized that punctuation causes no results nearly 100% of the time. (Ignoring entirely the problems where aka names don’t ever match.) If a game name has any sort of punctuation in it (I’ve seen the issue with dashes, apostrophes, periods, and exclamation points just off the top of my head) nothing will ever match. Not sure if this is an encoding issue (like dash gets changed to &emdash; or something when it is sent to the API) or if it’s a matching issue where the database can’t find something with punctuation or something like that. I’ve even tried manually inputting names that perfectly line up with the real name on thegamesdb (according to the website) and they still don’t come back with anything.

    Does anyone know of a way around this problem? (Or is anyone reading this capable of fixing the searches so they don’t barf on punctuation?) I’m willing to modify the emulation station xml manually and copy all the data and download the images myself, but if there is an easier way, I’d obviously prefer that.

  • moderators

    This has kicked my butt for the longest time: in the latest RetroPie, you can manually rescrape an individual title with literally zero changes and the search should find it if it’s in the database. No idea why this works.

  • Unfortunately I’ve tried redoing it several times now without having any luck. I think that approach definitely worked when the site was constantly crashing (just a couple weeks ago), but now they appear to have bumped up the hosting to fix that problem recently.

    A couple of examples… Games like B.O.B. for the SNES simply won’t come up with anything at all, no matter how I spell it, with or without punctuation, and with any version of the aka options. F-Zero worked correctly by removing the dash to match the non-aka version on the site. (I think someone might have edited the official name of F-Zero to be FZero specifically to resolve this issue.)

    Thanks for the theory either way though.

  • The search problem has existed for a very long time. It also unfortunately does not search the title alternates, which would alleviate the “f zero vs f-zero vs fzero” problem.

  • Related to this, I just realized that there appears to be a minimum word size for words that are included in the search. This causes problems with several titles that are either 3 characters, or made up of words that are 3 characters or less.

    Examples where neither word is long enough to count (so it looks like an empty search):

    • NHL 94 (or any of the other 5 or 6 years)
    • WWF Raw

    Examples where the whole name is too short (which makes punctuation maybe not the issue):

    • BOB
    • SOS

  • moderators

    I strongly recommend using the id: fallback (using the Game ID from the site) in the latest RetroPie to alleviate this issue until the site and API are more properly addressed for handling titles at single-character minimums, rather than the current three-character minimum. I suspect that this is currently in-play to ignore articles with little bearing on search relevance (The, And, An, Or, A, etc.) but something’s clearly gone a bit awry.

  • I’ve been manually modifying the XML by copying bits from the web site and downloading the images myself to get the games that can’t be scraped directly. I don’t know anything about this id: fallback approach (as I’ve never seen it mentioned anywhere else but here), do you think you can explain how to do that?

    I’ve tried entering in the manual input box of the scraper a couple variations and I still can’t get it to come up with any results.

    For example, I tried for NHL '94:

    • id: 3228
    • id:3228
    • id3228
    • 3228

    This was based on: http://thegamesdb.net/game/3228
    Is there another ID somewhere else I don’t know about? Am I just doing it wrong? I assume I’m just guessing wrong how to do it, so any pointers you have on how to make it work would be greatly appreciated. After all, manual XML creation is very tedious and slow.

  • moderators

    That second form should’ve worked on the most recent RetroPie / EmulationStation.
    When EmulationStation loads, what colour is the loading screen?

  • I’m running the latest version of RetroPie as of roughly a month ago. It’s version 4.2 (which is still the latest version as I write this). The Emulation Station loading screen is white with black letters, except the logo itself which uses blue.

    Just to make sure I’m not missing anything, I’ve done the process all over again and taken notes on exactly what I did, step by step. Here’s the list of steps I followed (sorry for it being super verbose):

    • Turned on the Pi and waited for it to finish booting.
    • Pressed Start, selected Scraper
    • Left the settings for Scrape From (thegamesdb) and Scrape Settings (on), and selected Scrape Now
    • Filter set to Only Missing Image, Systems set to just SNES, User Decides set to on, selected Start
    • First game came up as NHL '94, Results area shows “No Games Found - Skip”
    • Moved the cursor to select the Input button
    • Changed the default search from “NHL '94” to “id:3228” (no quotes used in actual searches)
    • Selected the Search button.
    • Returned to the game screen for NHL '94, Results area still shows “No Games Found - Skip”

    As mentioned previously, I’ve tried this with many variations and always get the same result. Also just to make sure, I tried it with a non-NHL game. I did SOS for SNES as “id:5961” and didn’t have any luck with that either. My last theory is that there is some mod I need to install that enables this feature in the scraper that I don’t know about and don’t have installed.

    Thanks again for your assistance with this, I’m sorry if I’m being a pain. Obviously I can just return to manual XML edits, but I was really excited about the ID thing as it would literally always work (if it worked at all) and it would be way faster.

  • moderators

    Unfortunately, I suspected as much. I’m not sure offhand exactly when the ID search was added, but I’m running 4.2.16 and it’s available. Basically, the tipoff here is that the latest updates use a Night loading screen (white text on black background), so I knew once you clarified which loading screen you have, that your version doesn’t support it yet (I’ve previously confirmed this solely out of curiosity).

    That said, you can run retropie-setup, then just update the setup script, then all packages (in that order). Should be smooth sailing after that (although, for your sanity’s sake, I seriously hope packages were used rather than source - bleeding edge is cool, right up until a 30-minute deploy turns into three days.)

  • I’ve now updated the retropie setup script, all packages, and the underlying OS just for good measure. My ES screen is now black, and the id:3228 search now works. Thanks a ton! I had no reason to suspect I was behind on anything since I was using the latest “official” release of retropie.

  • moderators

    Yeah, not a problem, I’m actually updating to .18 now, too.
    Don’t want to miss anything! :smile:

Log in to reply