I'm working through mirroring the database and I've already burned through a good number of my private key requests. I'm only showing 38,372 games. Old database had 54,000+ I believe.
But I'm not sure how to figure out which games I'm missing without stepping through the entire database again which would burn a lot of unnecessary requests.
My current method of scraping is:
Get All Platform IDs
Call /Games/ByPlatformID? and pass in all the platform ids
Go page by page adding each game and its images
It would be a lot more efficient for me and your server if I could request a list of all game IDs per platform ID. Then I could figure out which ones I'm missing very quickly.
Any chance this could be implemented? Or maybe even just a call that would return all Game IDs together?
Thanks!
Possible to get a list of all GameIDs per Platform?
- GamersDatabase
- Posts: 20
- Joined: Wed Jul 04, 2018 2:48 pm
Re: Possible to get a list of all GameIDs per Platform?
Hi there,
We'll find a way for you to mirror the rest of the games, but I'd like to debug how this happened, can you provide me with highest game id for each platform you have? should be easy enough If you're storing everything in a sql database.
few things to note, the DB currently contains just under 52k games, so you're about 14k off, the database has 108 consoles, of which 100 have games listed, the consoles IDs increment from 1, but at a certain point they jump to 4k+, all the games listed under consoles with 4k+ dd up to 9k which I would imagine account for most of your games.
We'll find a way for you to mirror the rest of the games, but I'd like to debug how this happened, can you provide me with highest game id for each platform you have? should be easy enough If you're storing everything in a sql database.
Code: Select all
select platform_id, MAX(id) from games group by platform_id
Regards
Zer0xFF
Zer0xFF
- GamersDatabase
- Posts: 20
- Joined: Wed Jul 04, 2018 2:48 pm
Re: Possible to get a list of all GameIDs per Platform?
I cross checked all platform IDs against the games I had already scraped and found a few platforms got skipped somehow. So I updated those platforms and now I'm up to 38,938 games. But that still leaves quite a few missing. The only platform IDs I don't have games for are the ones that don't have any listed on the website. So somehow some games within some platforms got skipped but I have no idea which...
edit:
Another oddity I just noticed... Game ID 50894 is listed on platform ID 4974 from the API data. Which checks out when I go to that game ID on the website.
https://thegamesdb.net/game.php?id=50894
But when I go to that platform page on the website, it doesn't list any games. Not sure if that's working as intended?
https://thegamesdb.net/platform.php?id=4974
Code: Select all
PlatformID MAX(GamesDBID) COUNT(PlatformID)
1 60415 6277
2 35693 606
3 36910 377
4 54983 644
5 46585 1118
6 60404 1907
7 60403 1376
8 60383 942
9 43639 1172
10 47931 1583
11 60402 2644
12 60421 1879
13 60384 1152
14 49003 698
15 60416 1441
16 55309 329
17 37175 439
18 37339 984
20 37048 358
21 36016 216
22 54195 465
23 54131 1588
24 35110 167
25 35420 150
26 37037 133
27 28406 67
28 30132 58
29 5268 18
30 35204 24
31 36834 164
32 36975 153
33 36170 40
34 54030 373
35 36903 344
36 50977 686
37 60417 96
38 60412 162
39 60422 535
40 55116 715
41 43056 564
4911 55337 523
4912 60397 349
4913 49624 297
4914 50755 267
4915 60409 178
4916 60410 383
4917 35403 166
4918 36351 25
4919 60424 1405
4920 60418 248
4921 32529 40
4922 36721 10
4923 36764 74
4924 31701 84
4925 45110 110
4926 42355 92
4927 37125 128
4928 35188 24
4929 37017 442
4930 35217 14
4931 35176 4
4932 32465 2
4933 31820 2
4934 36404 3
4935 32227 8
4936 34693 75
4937 48165 11
4938 34663 63
4939 53518 73
4940 32117 7
4941 37862 48
4942 35798 16
4943 35956 4
4944 32498 5
4945 33313 14
4946 32441 2
4947 32642 30
4949 36704 90
4950 37229 56
4951 34668 9
4952 37170 12
4953 51104 165
4954 35242 16
4955 60382 226
4956 36847 21
4957 55399 12
4958 54650 10
4960 42978 27
4961 40743 15
4963 55872 8
4965 40925 12
4966 40955 30
4967 40969 14
4968 45058 14
4969 55875 4
4971 60449 319
4974 50894 1
4975 50751 1
4976 47946 5
4979 52592 1
edit:
Another oddity I just noticed... Game ID 50894 is listed on platform ID 4974 from the API data. Which checks out when I go to that game ID on the website.
https://thegamesdb.net/game.php?id=50894
But when I go to that platform page on the website, it doesn't list any games. Not sure if that's working as intended?
https://thegamesdb.net/platform.php?id=4974
Re: Possible to get a list of all GameIDs per Platform?
sorry for the late reply Gamers, I've been rather busy.
anyone looking at those numbers, they make no sense at all, the few I checked tend to be on the last page or close to it, as such you must have passed through all the pages before getting this far.
page count also indicate correct listing, so something else is at play, if I had to guess, you're skipping the entire page listing if you an error takes place, unfortunately, only you would be able to tell me what that error is, assuming you had any forum of debugging/logging enabled.
as for the platform listing, yes this is intentional, the page only shows the latest releases, and since the game has no release date it can't be listed as one of those releases, so no listing is shown.
anyone looking at those numbers, they make no sense at all, the few I checked tend to be on the last page or close to it, as such you must have passed through all the pages before getting this far.
page count also indicate correct listing, so something else is at play, if I had to guess, you're skipping the entire page listing if you an error takes place, unfortunately, only you would be able to tell me what that error is, assuming you had any forum of debugging/logging enabled.
as for the platform listing, yes this is intentional, the page only shows the latest releases, and since the game has no release date it can't be listed as one of those releases, so no listing is shown.
Regards
Zer0xFF
Zer0xFF
Re: Possible to get a list of all GameIDs per Platform?
I really could use an api that returns just a list of game ids per platform too, it would be awesome if you ever decide to implement it.
Re: Possible to get a list of all GameIDs per Platform?
Any news about the possibility to have an api that returns a list of all game ids for a platform?
Thank you
Thank you