Home- Data Collection- Data Mining- GPL Software- Services- FAQ- Tips
Web Page Monitoring
OverviewThis tutorial covers applications were the data is available from a web page(http). The data we want to capture is the total number of bytes that have been sent and received from a firewall to the Internet. The firewall has a built in web server that reports many statistics. By recording time stamped traffic totals at regular intervals we can later calculate the transfer for given time periods.
To capture the data we are using a program called HTTPparse on the client. It uses the lynx browser to request a web page at http://192.168.1.254/cgi-bin/viewnet. The contents of the web page are written to a file and then the data is parsed out. It creates 2 time stamped records that are simply empty text files with some data in the file name. The files are written to /home/client/records/.
datatype timestamp station element value
n 19900101063811 003 0 1772633
n 19900101063811 003 1 72890
InstructionsWe are using 2 PCs to monitor the firewall. It can be done on just the sever PC if the server is behind the Firewall.
Dachstein LEAF Firewall
Download and compile HTTPparse in the /home/client/ directory.
HTTPparse needs to be added to cron. 15 minutes intervals seems to work well.
0,15,30,45 * * * * ./HTTPparse.exe 59 23 * * * ./HTTPparse.exeNote we added an extra execution just before midnight to make the search script a little more simple.
The records should be added the database when sync executes.
Firewall Traffic ChartFirewall Traffic Chart