*Max Earning Club - Printable Version

*Max Earning Club - Printable Version

+- [DEV] ISFL Forums (http://dev.sim-football.com/forums)
+-- Forum: Community (http://dev.sim-football.com/forums/forumdisplay.php?fid=5)
+--- Forum: Media (http://dev.sim-football.com/forums/forumdisplay.php?fid=37)
+---- Forum: Graded Articles (http://dev.sim-football.com/forums/forumdisplay.php?fid=38)
+---- Thread: *Max Earning Club (/showthread.php?tid=29304)

*Max Earning Club - Yeenoghu - 01-19-2021

It's been ages since I've made a media post and I don't have a league job at the moment, so ...

I had the day off work today and decided to tackle an annoying problem that I haven't been able to easily resolve since we migrated to the current forums. Back in the days of jcink, it was somewhat easier to get forum data in bulk to figure out which threads a given user had posted in. There are a few different use cases for being able to easily access this kind of data, and two jumped out to me immediately when I considered how to approach this topic. First, in the Chicago server (and I'm sure many others do something similar), we have a 'Max Earning Club'. Besides lending its name to this little project, Max Earning Club essentially involves @TheCC looking through profile histories, claim threads, update posts, etc. to ping us if we missed a point task or claim, all in the name of making sure we are earning as much TPE as possible. This inevitably results in heartbreak for some users, triumph for others, and post-ping anger from everyone, but it's certainly effective at keeping everyone on track, especially users who aren't visiting the site every day. Secondly, the forum migration really borked one tab of our scouting spreadsheet in London. Now we have to tediously go check on every user to see what tasks they completed leading up to the draft.

I've had this idea kicking around for a while since @r0tzbua posted something similar back in September, but I got a little sidetracked by diversions like sim testing and GMing. Now that I'm free from the shackles of DDSPF16, I unintentionally spent today engaged in my own little hackathon to see what I could figure out. Now, I don't want to give the impression that I am some fantastic coder--I think anyone who does this kind of thing for a living will think what I've built is pretty cute and potentially offensive. I'm just trying to get better at Python, so if you ask me why I did something a certain way, the answer is probably "I don't know! ¯\_(ツ)_/¯"

Okay, enough stalling. Here's a link to the github repo. There's a readme file there, but I'll add some screenshots here for more clarity.

Hopefully the code body is self-explanatory, but here is a brief rundown anyway. Open in your favorite Python IDE--it will become abundantly clear if you open this file that I am an amateur programmer, so any funny business you do with the command line instead is up to you. I'm sure it's rife with bugs, so please tell me about them.

As inputs, MaxEarningClub.py takes three possible values (but usually two)--

* Threads - This should be the thread ID ('tid') of any thread that you want to scrape. It's the number after 'tid=' in any thread URL. If left blank, the script will still attempt to query the Weekly Training thread for the specified user/team.
* Users - A list of one or more users. The values should match the forum names exactly.
* Teams - A list of exactly one team. Required abbreviations are listed in the .py file. If a team is specified, all users associated with the team will be queried and returned.

[Image: JkzSOyY.png]

Either Users or Teams is required, but probably not both. You can technically do both at once, but I wouldn't expect good results from the Weekly Training scraping if you start crossing the streams.

After the inputs are specified, run the script. In the event that you don't encounter an error, you should see a list of users and various thread names and URLs as outputs. These are threads that the user has not posted in (or, in the case of Weekly Training, has not posted in since Monday of the current week). Buy ice cream for users who return an empty set--that means they posted in every thread you specified. (Sorry in advance for putting my two teammates below on blast.)

[Image: fhavUqi.png]

The biggest outstanding issue with the script right now is related to the way it determines if users have completed Weekly Training. The timestamps on the forums were giving me fits, so the way it's implemented is that it works backwards until it finds the most recent training post on Sunday and gives credit to all of the people that completed training since then. If no one did training on a Sunday, I have no idea what will happen. Big Grin

If you've made it this far, you're probably Slate or mojo (or a non-mojo media grader). If I work on this enough, maybe it can help the audit team one day. At the very least, it should be very beneficial to us during scouting next offseason. Ping me on Discord at Maglubiyet#0479 if you have any issues or just want to talk. If none of this makes sense but you made it to the end anyway, I'm sure you can figure this stuff out if you find it interesting. Until next time! Good luck!

RE: Max Earning Club - Troen - 01-19-2021

(01-19-2021, 01:33 AM)Maglubiyet Wrote: The biggest outstanding issue with the script right now is related to the way it determines if users have completed Weekly Training. The timestamps on the forums were giving me fits, so the way it's implemented is that it works backwards until it finds the most recent training post on Sunday and gives credit to all of the people that completed training since then. If no one did training on a Sunday, I have no idea what will happen.

Can you parse the forum dates? If so, can you check something like

Code:
lastWeek =  datetime.now() - timedelta(days=7)

for each player:

    get most recent post

    if post.time < lastWeek:

        print that they didn't do it or whatever

(I don't actually know python)
That's obviously not perfect, but maybe a good starting heuristic?

RE: Max Earning Club - Yeenoghu - 01-19-2021

(01-19-2021, 02:26 AM)Troen Wrote:
(01-19-2021, 01:33 AM)Maglubiyet Wrote: The biggest outstanding issue with the script right now is related to the way it determines if users have completed Weekly Training. The timestamps on the forums were giving me fits, so the way it's implemented is that it works backwards until it finds the most recent training post on Sunday and gives credit to all of the people that completed training since then. If no one did training on a Sunday, I have no idea what will happen.

Can you parse the forum dates? If so, can you check something like

Code:
lastWeek = datetime.now() - timedelta(days=7) for each player: get most recent post if post.time < lastWeek: print that they didn't do it or whatever
(I don't actually know python)
That's obviously not perfect, but maybe a good starting heuristic?

Yeah, this is the way I originally started going about this! It works great for most posts, but here's what I run into. Old post timestamps look like this--

Code:
<span class="post_date"><i class="fa fa-clock-o"></i> Tuesday, July 21st, 2020, 10:18 am</span>

It's fairly straightforward to get the date/time value from the text of that field, but this is what "new" posts look like...

Code:
<span class="post_date"><i class="fa fa-clock-o"></i> <span title="Tuesday, January 19th, 2021, 09:06 am">5 minutes ago</span>

The same element now has a description of the date/time value, while the value itself is moved to the title tag. The same thing happens for posts that have been edited. Anyway, I do agree that this is the best way forward for the sake of accuracy if I can work around this issue. Thank you!

RE: *Max Earning Club - TheCC - 01-22-2021

Oh Mag, I love you.

RE: *Max Earning Club - Crodyman - 01-22-2021

WAIT HAVE I MISSED A CLAIM?

(some time later checking the S27 playoff prediction claim. . . )

[Image: s27.jpg]

the big sad is here again

RE: *Max Earning Club - Thiath - 01-23-2021

God you guys are nerds. What does any of this mean? How can I use it to theorize mock drafts?