- Use the Source, Luke
Home | Register | News | Forums | Guide | MyLinks | Bookmark

Sponsored Links

Latest News
  General News
  Press Releases
  Off Topic

Back to files


ULW -- User-Agent Log Watch -- Instructions

Copyright (c) 2000 Edward V. POPKOV
Distributed under GPL, see COPYING for details

"Edward V. POPKOV" <>

$Id: README,v 0.8 2000/06/08 23:03:11 root Exp root $

$Log: README,v $
Revision 0.8 2000/06/08 23:03:11 root Minor touches, changed install dir, this is the last release

Revision 0.7 2000/03/30 19:57:33 root Added command line options for bash, gawk script rewritten

Revision 0.6 2000/03/15 16:00:59 root Incremental, cleanup release

Revision 0.5 2000/02/25 01:32:46 root URGENT bugfix release. Thanks, Kevin.

Revision 0.4 2000/02/23 00:53:48 root Added vhost/URL field to ulw.conf, known IPs file.

Revision 0.3 2000/02/19 22:41:13 root Added interface to mail/sendmail.

Revision 0.2 2000/02/17 22:51:04 root Added files, regexp in ulw.conf, %v to custom log format

Revision 0.1 2000/02/14 22:54:40 root Support for non-default custom log layout, bugfixes.

Revision 0.0 2000/02/14 15:26:19 root Initial check-in.


  1. What is ULW?
  2. Requirements
  3. Installation
  4. Distribution
  5. Credits
  6. How to contribute
  7. What is ULW?

ULW, User-Agent Log Watch, is a set of scripts to process and backup Apache log files of custom format that contain the following data:

-user agent
-remote IP address
-accessed host and requested URI

Proceed to 3. Installation to see how to tell Apache to write this kind of log for you.
When ULW bash script is invoked, it moves last custom log (value of --enable-log-dir configure script option, concatenated with --enable-log) to pre-defined location (--enable-savelog-dir + --enable-log + date.time ) and sends SIGHUP to Apache process to reset the log. Then gawk is called to read ULW config file (--sysconfdir + ulw.conf) and ULW gawk script (--sbindir + ulw.gawk), and process latest custom log. All matches, defined in ulw.conf are printed to stdout. To have them mailed, ULW bash script (--sbindir + ulw) must be invoked from crontab with MAILTO variable set.
Since version 0.7 ULW behaviour can be controlled via command line options:

ulw [-h] [-v] [-i] -m m|s|o

        -h      show brief help message and exit
        -v      show version and exit
        -i      ignore known IPs file
        -m      output mode: m- talk to mail, s- talk to sendmail, o- write to

If no option specified, ULW writes to stdout, if option not recognized ULW prints short usage help. Only first option is processed, all subsequent options are silently ignored.
configure script understands --enable-from, --enable-to and --enable-subject command line options to control emailing of ULW reports. See INSTALL. By default ULW assumes that the first field is user agent identifier, and the second is remote IP address. If your log format is different, see the bottom of INSTALL for options --enable-ia-field and --enable-ip-field explained.
Since version 0.2 ULW assumes that records in ulw.conf consist of regular expressions, rather than simple strings. So, you can either specify full user agent or remote IP, or set it to `*', or write your regexp. See gawk(1) for details. Note that you can not use `|' in your regular expressions, because this symbol is ULW's field delimiter. Refer to ulw.conf for examples. Since version 0.4 ULW keeps track of IP addresses known to it. Entries in ULW IP file start with IP= followed by IP address in decimal-dotted form. No spaces around =. This file is maintained automatically, however it's OK to edit it manually. Just keep in mind that unlike ulw.conf entries, these IP addresses are checked for exact match, ie they are not regular expressions. ULW won't send any messages for custom log entries for IP addresses in its known IPs file. File name is configured with --enable-ip-file option, and automatically prepended with --enable-savelog-dir. Since version 0.7 ULW can be told to ignore the IPs known to it. Due to specifics of gawk script, file of known IP addresses must contain at least one valid record, if you want ULW to maintain it automatically. Initially this file contains IP= If you don't want this feature, you can keep the file of known IPs empty.

2. Requirements

Development and testing platform is PC under Linux 2.0.38, libc5. The following software is required to be able to use ULW:

-GNU bash, version 1.14.7(1) (bash -version) -GNU Awk 3.0.4 (gawk --version)

ULW is supposed to work correctly with later versions of bash and gawk, and probably will work with earlier releases. Contact me if you experience compatibility issues.
Bash script also calls mv, kill and gzip.

3. Installation

Prepare Apache server. Find httpd.conf file. Back it up. Add custom log format definition:

LogFormat "%{User-agent}i|%a|%{%x %X}t|%v%U" user_agent

Note that in order for ULW to be able to parse the log, your delimiter should be `|'. You can adjust %{%x %X}t, which is date and time format, to meet your needs, see strftime(3).

Specify the file name:

CustomLog /var/log/httpd/apache/ua_log user_agent.

Depending upon subtleties of your setup, you may with to combine log records for virtual hosts into a single log file. All the necessary information can be found in Apache docs:

Apache docs -> Run-time configuration directives -> LogFormat -> Custom Log Formats.

Actual file name in CustomLog can be any file. Just note it, you'll need it to tune ULW. Now kick Apache so it notices the changes in config:

# /etc/rc.d/init.d/httpd stop
# /etc/rc.d/init.d/httpd start


# kill -SIGHUP `cat /var/run/`

To setup ULW do

# ./configure
# make -n install

check dirs and permissions

# make install

See the very bottom of INSTALL file to find what standard and additional options can be supplied to ULW configure script. Make sure you read it and supply appropriate command line options to configure script of your custom log format differs from the default one, shown above. In samples subdir you'll find Apache log rotator ral and proposed crontab to schedule ULW and ral. I believe they both are pretty straightforward. Please refer to your docs or contact me if you experience any problems with these.
Makefile has a phony target `uninstall', which removes all the ULW files from your system. It depends on configure command line options and enviroment, so if there's a chance that you'll be uninstalling ULW, backup Makefile before calling make distclean;

4. Distribution

ULW is GPL'ed, see COPYING for details.

5. Acknowledgements

ULW project sponsor is "Kevin M. LIANG" <>.

In my work I'm using:

Linux, bash, Apache, Lynx, vim, autoconf, make, sed, Netscape

Thanks to developers of these and numerous other free and open source tools.

6. Contributions

Get in touch with me if you want to add to ULW. 0.8 is the last release, so you'll most likely on your own now.

Thx and take care

Edward V. POPKOV
Independent Developer

Version: 3.1
GCS/CC d-(+) s:- a C$ UL+++>$ P+ L++(+++)>$ E--- W--(+) N o? K? w$ !O M? V PS+ PE Y PGP t? 5? X? R !tv b++@ DI D+ G++ e h* r++ y+++


Sponsored Links

Discussion Groups
  Networking / Security

About | FAQ | Privacy | Awards | Contact
Comments to the webmaster are welcome.
Copyright 2006 All rights reserved.