Sure, why not.

Great. 65.

Cecil

https://www.gog.com/u/cecil/wishlist

Palio de Monte

Not in for bler's GA ;)

Yeah, when I write webscraper code (which isn't very often) I put those kind of thoughts into it too, as I don't want to overburden a website with a massive DoS attack. I usually set up a queue with timers and either evenly spaced time delay between queries, or a random time delay within a range to blend in with other traffic more, and I try to err on the side of caution and underburden, handle errors, and try to detect unexpected scenarios and bail so an error doesn't send the script into a loop pounding a site and getting the same error over and over etc.

I'd have to think about it more but my first thought is to break it down into at least 2 operations. The first is finding potential nicknames to check for the existence of a shared wishlist, and the second is querying the wishlist and caching the results. I would probably start by scanning one page worth of threads in the General Discussion thread and then start walking the thread topics by queing and loading them one at a time in most recent post to least recent order of thread, and also most recent page to oldest page within a thread with the site configured to maximum posts per page for making less requests to the webserver.

I'd put a healthy time delay of a good 30s or more in between queries during testing until I got any kinks ironed out, then I'd probably set it to load pages with a random delay between loads varying from 17-37 seconds or so to more simulate human-like behaviour. The reason for the odd numbers instead of rounding to 10 is that it shows up in logs looking more random. :) If the bot gets banned for excessive queries, just increase the delays in a binary fashion to find the sweet spot to dodge the software detection. That puts it outside of the range considered problematic by a given server's configuration and thus "reasonable" IMHO.

Then the max post in each thread should be tracked in a database so subsequent scans don't redownload already processed data and waste more resources too. Next I'd probably sort the username list in order of most frequent poster to least frequent under the premise that someone who is more frequently posting is more interactive in the community and their wishlist is potentially more relevant (depending on what the purpose of gathering the data is to be used for of course). Then I'd probe for their wishlists and make a note of whether or not they had a public list or not and timestamp it in a database along with the result (yes/no). If they have a public wishlist I'd yank the data that is desired and remember the timestamp of that also.

For subsequent probes I would check the last-probe timestamps and only re-probe a given wishlist if N amount of time elapsed since the last probe. For people who do not have a public wishlist I would probably pick an arbitrary time interval to re-probe in case they open their wishlist public at some point in the future. I do like to get fancy and smart with this sort of thing too, so I'd probably keep a running count of how many times they were probed and did not have their list public, then use exponential fallback to degrade probing for them again. So for example, probe once per week by default and increment a counter for their "total probes resulting in non-public/not available count", then wait some multiple of that before the next probe. ie: 1 week then 2 weeks, 3 weeks, 4 weeks perhaps up to some maximum interval, or atlernatively double it each time 1/2/4/8. The purpose of the incremental or exponential fallback of course is to try to greatly reduce the load on the remote webserver on data that is predictably going to be false anyway, but to not give up entirely in case it does show up at some point. If someone makes their list public after a dry spell, reset the probe counter to 0 so it is probed regularly as long as it is public.

Anyhow, just some random ideas... I hope that makes some sense... No time to hack away at it myself though... WITCHER 3!!! :)

As I said, seems daunting.

Anyhow, 67 and counting.

If they are going to have a feature like public wishlists I think it would make sense to be able to set a variety of priorities to the games. Sure I'd like Whacky Wheels and Cannon Fodder but not nearly as much as I want The Dig or Rex Nebular and, of course, reversely, if I feel compelled to gift someone something from their list, I'd like to know if they want Spycraft: The Great Game more than Return to Zork and be able to surprise them with it rather than have to ask which they'd prefer and spoil the surprise.

Agree - even a simple scheme with 3 settings like "warm" "hot" "en fuego!" would be useful.

Interim report:

- Began importing the wishlists into the beta version of MaGog (will probably be released next week)
- Of the 67 wishlists, 63 are valid, 3 are no longer public, 1 is empty

I was thinking something similar too, but instead of priority something more generic like "categories" or the tags feature even. That way you could categorize them on arbitrary keywords/phrases and that could include making priorities however one sees fit. "MOST HIGHLY ANTICIPATED", "NEEDED TO COMPLETE SERIES", "KIND OF WANT TO TRY SORTAMAYBEIDUNNO" etc...

I'm actually thinking of going through my wishlist and removing games that are lower priority "kinda sorta maybe" stuff and bookmarking them in Firefox in a "GOG Games - Kinda Sorta Maybe I Dunno WTF BBQ" folder. That way my wishlist can be cleaner and reflect games I actually know I want more highly, which is also more useful to friends and others who might decide to do surprise birthday gifts or similar etc.

Bumpestest

The feature has been implemented.

Here are some of the many questions one may answer through this new-old feature. now based on the public wishlists:

1) [url=http://www.an-ovel.com/cgi-bin/magog.cgi?ver=520&scp=gdspur&dsp=ipgfsorlcmbah&ord=&flt=tcs~trails~&opt=&myf=SunMay240412182015_ZoLC_vpZDifk3]Who wishlisted Legend of Heroes: Trails in the Sky?[/url]

2) [url=http://www.an-ovel.com/cgi-bin/magog.cgi?ver=520&scp=gdspur&dsp=ipgfsorlcmbah&ord=&flt=hby~mrkgnao~&opt=&myf=SunMay240413392015_vR2Z5iIUsPV__]Which games are on mrkgnao's wishlist?[/url]

3) [url=http://www.an-ovel.com/cgi-bin/magog.cgi?ver=520&scp=gdspur&dsp=ipgfsorlcmbah&ord=&flt=hby~mrkgnao~gan~fan~&opt=&myf=SunMay240414292015_qBKMs1NHpa_Li]Which fantasy games are on mrkgnao's wishlist?[/url]

4) [url=http://www.an-ovel.com/cgi-bin/magog.cgi?ver=520&scp=gdspur&dsp=ipgfsorlcmbah&ord=h9&flt=hat~20~&opt=&myf=SunMay240415382015_F_ntg5glp9PhG]Which games have been wishlisted by at least 20 people (sorted by descending order of number of wishlisters)?[/url]

5) [url=http://www.an-ovel.com/cgi-bin/magog.cgi?ver=520&scp=gdspu&dsp=ipgfsorlcmbah&ord=&flt=pat~0.1~hby~00000~tds~%5BRemoved~&opt=&myf=SunMay240419342015_BHQ_qO_5UWHzn]Which non-free non-removed games have been wishlisted by no one?[/url]

If you haven't opted in yet, perhaps now is the time. Or perhaps later. Or perhaps never.

Regardless, this thread will remain active as long as MaGog does.

Here you go, this is interesting. https://www.gog.com/u/Jeets2/wishlist

dead link