Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,5 @@ root/mood.json
settings.json
round.csv.gz
round.csv
parseRoundInfoTable.py
parseRoundInfoTable.py
root/config/header_config.json
30 changes: 18 additions & 12 deletions README.MD
Original file line number Diff line number Diff line change
Expand Up @@ -4,31 +4,37 @@ This is a set of tools developed to aid in the tracking of maptick, td, and pop

It scrapes all available performance logs, processes them into csv files (One for each map, one for each server, and a few global ones), and then uses `gnuplot` to plot them in a few different ways I've judged as useful.

## Scraping
## Setup

We don't scrape all performance logs, since some of them from before 2020/11/11 don't scrape well and don't match the current system.
### Install Dependencies

## Parsing
Download and install the following:
- [Python 3.6](https://www.python.org/downloads/release/python-360/)
- [gnuplot 5+](http://www.gnuplot.info/)

A brief note on parsing, we compile each round into one data-point, attached to some other info about it.
Make sure both are in your path, then restart your computer. You should then run `Setup.bat`, this will setup the default header config in [root/config](root/config) and download any python modules needed through pip.
Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do you need to restart for this?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Personally I had to restart to get the path changes to propagate, and this is recommended for path changes in general.

Copy link
Copy Markdown
Owner

@LemonInTheDark LemonInTheDark Oct 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was under the impression that you just needed to open a new terminal to refresh environmentals

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was running these not through a terminal but instead by double clicking the batch files themselves, doing it through terminal might have worked. Want me to replace the restart mention with a recommendation to use a terminal?

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's odd that it what you describe wouldn't just work then. I thought the only restriction was needing to refresh a terminal's copy of environment vars after modifying them. are you sure a restart is needed in the situation you describe?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No I'm not but even the windows dialogue itself recommends restarting when you edit the path variable, which is why I repeated it here after it had fixed my own issue.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

huh. ok I guess this is fine then


## Updating your data
### Update data

Running `Update.bat` will pull all the logs you don't have (Outside some edgecases I don't have the brainpower to fix, read the python file) parse them into csv files, and generate a new plot from them.

## Single Round Parsing
## Usage Notes

If you'd like to graph a single round you've already pulled, use `GraphOne.bat`. It'll dump images for that round into [root/single_round](root/single_round)
### Scraping

## More Information
We don't scrape all performance logs, since some of them from before 2020/11/11 don't scrape well and don't match the current system.

If you'd like to learn more about any of this go read [root/README.md](root/README.md), it goes into more detail about how things work and how to interact with them
### Parsing

## Dependencies
A brief note on parsing, we compile each round into one data-point, attached to some other info about it.

### Single Round Parsing

If you'd like to graph a single round you've already pulled, use `GraphOne.bat`. It'll dump images for that round into [root/single_round](root/single_round)

This project was created with `python 3.6`, you'll need at least `python 3` for the logs parser to run properly.
## More Information

Additionally you will need gnuplot installed, with a version above 5. This is to support graphing several files onto one graph. Needs to be in your path too, because I'm a moron. [Have another Link](http://www.gnuplot.info/)
If you'd like to learn more about any of this go read [root/README.md](root/README.md), it goes into more detail about how things work and how to interact with them

## License

Expand Down
17 changes: 17 additions & 0 deletions Setup.bat
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
@ECHO OFF

SET CONFIGFOLDER=root\config\
SET TEMPLATEPATH=%CONFIGFOLDER%_template.json
SET CONFIGPATH=%CONFIGFOLDER%header_config.json

IF EXIST %CONFIGPATH% (
ECHO Config already exists, skipping.
) ELSE (
ECHO No config found, copying default...
COPY %TEMPLATEPATH% %CONFIGPATH%
)

ECHO Installing pip dependencies...
py -3.6 -m pip install requests

PAUSE
25 changes: 14 additions & 11 deletions Update.bat
Original file line number Diff line number Diff line change
@@ -1,12 +1,15 @@
#!/bin/sh

@echo off
pushd "%~dp0"
cd root
echo Grabbing Logs
py RawLogsScraper.py
echo Averaging CSV files
py MaptickAverager.py
echo Plotting Graphs
@ECHO OFF

PUSHD "%~dp0"
CD root

ECHO Grabbing Logs
py -3.6 RawLogsScraper.py

ECHO Averaging CSV files
py -3.6 MaptickAverager.py

ECHO Plotting Graphs
gnuplot maptick_plot.gnuplot
popd "%~dp0"

POPD "%~dp0"
2 changes: 1 addition & 1 deletion root/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ In the event of an error, this info can be found in [data/scraping.json](data/sc

Also has some internal procs you can use to look at how many holes you have. No way to self heal if they happen before buffering was introduced though.

Oh right uh, mood.json exists. Create one and edit it to match your server creds if for some reason you want to pull from raw logs (Don't forget to change the url)
Oh right uh, `header_config.json` exists. One should be created when `Setup.bat` was run in the [config](config) folder. Edit it to match your server creds if for some reason you want to pull from raw logs (Don't forget to change the url)

### [MaptickAverager](MaptickAverager.py)

Expand Down
2 changes: 1 addition & 1 deletion root/RawLogsScraper.py
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,7 @@ def clearDataFile():

#owning mso with facts and logic (you need beautiful soup to parse raw logs for reasons)
#from bs4 import BeautifulSoup
#cookie_file = "mood.json"
#cookie_file = "config/header_config.json"
#do_not_post_this_4head = open(cookie_file)
#fakingIdentity = json.load(do_not_post_this_4head) #Loads a .json file containing the cookie and other params to send to mso
#do_not_post_this_4head.close()
Expand Down
10 changes: 10 additions & 0 deletions root/config/_template.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{
"Host": "tgstation13.download",
"User-Agent": "Default Graphing Scraper (Lemon Scented)",
"Accept": "Remember to not accept zip files",
"Accept-Language": "hhhhhhhhhhh",
"Referer": "Pick Your Poison",
"Connection": "keep-alive",
"Cookie": "Do not post this publically anywhere 4head",
"Upgrade-Insecure-Requests": "1"
}