Home- Data Collection- Data Mining- GPL Software- Services- FAQ- Tips

Web Page Monitoring


This tutorial covers applications were the data is available from a web page(http). The data we want to capture is the total number of bytes that have been sent and received from a firewall to the Internet. The firewall has a built in web server that reports many statistics. By recording time stamped traffic totals at regular intervals we can later calculate the transfer for given time periods.

To capture the data we are using a program called HTTPparse on the client. It uses the lynx browser to request a web page at The contents of the web page are written to a file and then the data is parsed out. It creates 2 time stamped records that are simply empty text files with some data in the file name. The files are written to /home/client/records/.

datatype timestamp station element value

n 19900101063811 003 0 1772633
n 19900101063811 003 1 72890


We are using 2 PCs to monitor the firewall. It can be done on just the sever PC if the server is behind the Firewall.
Server Setup
Client Setup

Monitored Firewall
Dachstein LEAF Firewall

Download and compile HTTPparse in the /home/client/ directory.

Source Code

HTTPparse needs to be added to cron. 15 minutes intervals seems to work well.
As client
crontab -e
0,15,30,45 * * * * ./HTTPparse.exe
59 23 * * * ./HTTPparse.exe
Note we added an extra execution just before midnight to make the search script a little more simple.

The records should be added the database when sync executes.

Firewall Traffic Chart

Firewall Traffic Chart

Home- Data Collection- Data Mining- GPL Software- Services- FAQ- Tips Logo