Home- Data Collection- Data Mining- GPL Software- Services- FAQ- Tips

Web Page Monitoring Overview This tutorial covers applications were the data is available from a web page(http). The data we want to capture is the total number of bytes that have been sent and received from a firewall to the Internet. The firewall has a built in web server that reports many statistics. By recording time stamped traffic totals at regular intervals we can later calculate the transfer for given time periods. To capture the data we are using a program called HTTPparse on the client. It uses the lynx browser to request a web page at http://192.168.1.254/cgi-bin/viewnet. The contents of the web page are written to a file and then the data is parsed out. It creates 2 time stamped records that are simply empty text files with some data in the file name. The files are written to /home/client/records/. Format datatype timestamp station element value Example n 19900101063811 003 0 1772633 n 19900101063811 003 1 72890 Instructions We are using 2 PCs to monitor the firewall. It can be done on just the sever PC if the server is behind the Firewall. Server Setup Client Setup Monitored Firewall Dachstein LEAF Firewall Download and compile HTTPparse in the /home/client/ directory. Source Code HTTPparse needs to be added to cron. 15 minutes intervals seems to work well. As client crontab -e 0,15,30,45 * * * * ./HTTPparse.exe 59 23 * * * ./HTTPparse.exe Note we added an extra execution just before midnight to make the search script a little more simple. The records should be added the database when sync executes. Firewall Traffic Chart Firewall Traffic Chart

Home- Data Collection- Data Mining- GPL Software- Services- FAQ- Tips



http://sourceforge.net/projects/frontiertech/