[DEV] ISFL Forums
Work Smarter, Not Harder - Printable Version

+- [DEV] ISFL Forums (http://dev.sim-football.com/forums)
+-- Forum: Community (http://dev.sim-football.com/forums/forumdisplay.php?fid=5)
+--- Forum: Media (http://dev.sim-football.com/forums/forumdisplay.php?fid=37)
+---- Forum: Graded Articles (http://dev.sim-football.com/forums/forumdisplay.php?fid=38)
+---- Thread: Work Smarter, Not Harder (/showthread.php?tid=3197)

Pages: 1 2 3

Work Smarter, Not Harder - 7hawk77 - 08-09-2017

[div align=\\\"center\\\"]Hey Everyone,[/div]

Today I'm bringing you a great tool that you can use to automate TPE gathering. I decided to make this tool because @Noppadet and I are planning on collaborating to do some fun stuff with it.


Anyways, lets say you wanted to get all of the TPE for all of the players and then figure out how much TPE each team would have. This would probably take you like 20-30 minutes. Now this gets updated every week so if you wanted this weekly, it becomes even more of a pain.

Enter the NSFL TPE Webscraper.

(If you would like to see the code, it's on github here: )

Scraper Repository On Github


Now if you just want the tool, the instructions and download are here:
Release download

Be forewarned, you will probably need to be using windows. Also, you will also have to trust me since it's an executable from a "stranger on the internet." If this makes you uncomfortable, I guess I don't really care, just don't use it.

Alright so basically how this works is it goes to each team's player roster page, scrapes the data and then outputs it in to text files. The text files have abbreviated names such as AOPlayers for Arizona Outlaws Players or AOTeam for the team total of Arizona. I kept these separate because I think it will be more useful for Noppadet.



For example, I'll just go ahead and show the output for a few teams:

Code:
Arizona Outlaws:

(S3) - Jaxon Tuck - RB,TPE: 57,
(S3) - Godfrey King - DT,TPE: 73,
(S3) - Gavin Graham - S,TPE: 50,
(S3) - Mark Strike - QB,TPE: 68,
(S2) - Luke Luechly - LB,TPE: 164,
(S2) -  C.A Chess - WR,TPE: 166,
(S2) - Gui Worthy - OL,TPE: 75,
(S2) - Allen Davis - S,TPE: 115,
(S2) - Hingle Mcringleberry - LB,TPE: 50,
(S2) - Scott Timlin - OL,TPE: 145,
(S2) - Busters Brownce - OL,TPE: 50,
(S2) - Dustin Evans - WR,TPE: 198,
(S2) - Christiano Ronaldo - P,TPE: 199,
(S2) - Tim Pest - OL,TPE: 254,
(S2) - Teez Tobin - RB,TPE: 50,
(S2) - Jack Vanderjagt - K,TPE: 50,
(S2) - Connor MacManus - OL,TPE: 69,
(S1) - Luke Pierce - DT,TPE: 94,
(S1) - Fernando Bessel - RB,TPE: 50,
(S1) - Alex Thomas - OL,TPE: 123,
(S1) - Jason Williams - CB,TPE: 155,
(S1) - Dental Dam - DE,TPE: 236,
(S1) - Adrienne Nugent - K/P,TPE: 50,
(S1) - Joseph Tkachuk- TE,TPE: 105,
(S1) - Isaiah Rashad - S,TPE: 126,
(S1) - Arbin Asipi - DT,TPE: 137,
(S1) - Harrif Ernston - LB,TPE: 273,
(S1) - Jaylon Lee - LB,TPE: 308,
(S1) - Ryan Flock - S,TPE: 286,
(S1) - Jayce Tuck - DE,TPE: 319,
(S1) - King Bronko - QB,TPE: 236,
(S1) - Reg Mackworthy - RB,TPE: 270,
(S1) - Vincent Draxel - QB,TPE: 107,
(S1) - Mayran Jackson - WR,TPE: 275,

Arizona Outlaws' TPE: 4983

Baltimore Hawks:

(S3) DE - J.D. Boom,TPE: 70,
(S2) - Rafael Reyes - CB,TPE: 118,
(S2) - Dieter Koch - OL,TPE: 110,
(S2) - James Wright - LB,TPE: 101,
(S2) - Pat Pancake - OL,TPE: 50,
(S2) - Alonzo Charles - OL,TPE: 50,
(S2) - Carmel Gibson - TE,TPE: 50,
(S2) - Avon Blocksdale - OL,TPE: 107,
(S2) - Owen Taylor - DE,TPE: 139,
(S2) - Jorge Masvidal - S,TPE: 107,
(S2) - Darlane Farlane - RB,TPE: 152,
(S2) - Brokk Lee - OL,TPE: 167,
(S2) - John Baker - WR,TPE: 89,
(S2) - Darnell Turner - CB,TPE: 206,
(S2) - James Lillico - S,TPE: 66,
(S2) - Matthew Vincent - WR,TPE: 159,
(S2) - Jesse Sanderson - TE,TPE: 52,
(S1) - William Nasher - DT,TPE: 76,
(S1) - Matt Smith - S,TPE: 50,
(S1) - Scrub Kyubee - QB,TPE: 209,
(S1) - Kurt Hendrix - DE,TPE: 245,
(S1) - Jonathan Shaloiko - WR,TPE: 93,
(S1) - Antoine Delacour - CB,TPE: 227,
(S1) - Stephen Harrison - LB,TPE: 293,
(S1) - Bert Metas - DT,TPE: 168,
(S1) - Marcus Robinson - RB,TPE: 111,
(S1) - Erasmo Broadway - LB,TPE: 156,
(S1) - Cooper Christmas - WR,TPE: 228,
(S1) - Turk Turkleton - K/P,TPE: 232,

Baltimore Hawk's TPE: 3881

Colorado Yeti

(S3) - Antonio Sandoval - OL,TPE: 57,
(S2) - Blaster Blade - DE,TPE: 222,
(S2) - Big Bot - DE,TPE: 119,
(S2) - Steve Austin - LB,TPE: 93,
(S2) - Gadget Tech - OL,TPE: 94,
(S2) - Torque Lewith - S,TPE: 106,
(S2) - Jon Ross - WR,TPE: 117,
(S2) - Santana Moss - WR,TPE: 50,
(S2) - Roman Thomas - S,TPE: 135,
(S2) - Bernard Forthward - OL,TPE: 50,
(S2) - Rich Gucci - TE,TPE: 153,
(S1) - Kieran O'Connell - RB,TPE: 205,
(S1) - Johnathon Saint - LB,TPE: 281,
(S1) - Daniel Robicheaux - OL,TPE: 81,
(S1) - Bender Rodriguez - OL,TPE: 146,
(S1) - Philippe Carter - CB,TPE: 277,
(S1) - Kendrick Hendrix - WR,TPE: 288,
(S1) - Luke Tiernan - LB,TPE: 262,
(S1) - Jimmy Darkapple - K/P,TPE: 84,
(S1) - Mark Hargrove - TE,TPE: 56,
(S1) - Mike McNoodle - S,TPE: 81,
(S1) - Boss Tweed - RB,TPE: 297,
(S1) - Fuego Wozy - DE,TPE: 268,
(S1) - D.J. Law - WR,TPE: 211,
(S1) - Bojo Kicksit - K/P,TPE: 176,
(S1) - Wyatt Fulton - LB,TPE: 270,
(S1) - Logan Noble - QB,TPE: 337,
(S1) - Alex Hayden - CB,TPE: 241,


Colorado Yeti TPE: 4757


You get the idea...


So if you see any bugs or problems with it, let me know and I'll get them fixed soon.


Anyways, I have no idea how you want to grade this @124715, It took about 4-5 hours of work, and could generate about 1200 words of "media" per week. We can talk about it if you want. I'm also fine with waiting til noppadet finishes the other part too (so you can better understand the ramifications of this)


Cheers


GRADED

Work Smarter, Not Harder - kckolbe - 08-09-2017

Can confirm that this does take over 30 minutes to do manually, as I've been doing it every week by hand.

Work Smarter, Not Harder - 7hawk77 - 08-09-2017

Forgot to tag @Bzerkap



Work Smarter, Not Harder - Bzerkap - 08-09-2017

(08-09-2017, 05:32 PM)7hawk77 Wrote:Forgot to tag @Bzerkap
Don't you @ me bro! You wanna fight!?

Work Smarter, Not Harder - timeconsumer - 08-09-2017

(08-09-2017, 07:30 PM)kckolbe Wrote:Can confirm that this does take over 30 minutes to do manually, as I've been doing it every week by hand.

Dunno what you were doing but after the initial setup of my Excel tool it took me 5 minutes a week.

Work Smarter, Not Harder - kckolbe - 08-09-2017

(08-09-2017, 04:45 PM)timeconsumer Wrote:Dunno what you were doing but after the initial setup of my Excel tool it took me 5 minutes a week.

I've been hand-jamming them onto a notepad file and adding them up via calculator. Also, I only include who I think would be starters in the "base package" rather than the whole team or top 23. That part adds a little to it.

Work Smarter, Not Harder - timeconsumer - 08-09-2017

(08-09-2017, 07:49 PM)kckolbe Wrote:I've been hand-jamming them onto a notepad file and adding them up via calculator.  Also, I only include who I think would be starters in the "base package" rather than the whole team or top 23.  That part adds a little to it.


I see. My method was more focused on how teams/position groups are gaining TPE over time and what attributes they spent it on.

Work Smarter, Not Harder - Noppadet - 08-09-2017

Y'all will see some cool stuff from this. Thanks Hawk for getting this ready so quickly.

Work Smarter, Not Harder - 7hawk77 - 08-10-2017

So after playing around with this some more, Some teams have 3 pages of rosters to scrape while other team's only have two.

The next release will have some way to handle this if a team adds players and moves up to 3 pages or loses players and goes down to 2 within the configuration file. Currently I'd need to recompile and reupload the executable.

Work Smarter, Not Harder - timeconsumer - 08-10-2017

FYI, I track TPE changes via the roster pages on the Index after Monday night games. It's slightly less precise because I can't track partial attribute points, but it's close enough for my needs and is easily manipulated by Excel. Plus it allows tracking by attribute.

Not sure if it would make your current process easier or not, but figure I'd throw it out there,