squidanalyzer/doc/squidanalyzer.3

.\" Automatically generated by Pod::Man 2.25 (Pod::Simple 3.16)
.\"
.\" Standard preamble:
.\" ========================================================================
.de Sp \" Vertical space (when we can't use .PP)
.if t .sp .5v
.if n .sp
..
.de Vb \" Begin verbatim text
.ft CW
.nf
.ne \\$1
..
.de Ve \" End verbatim text
.ft R
.fi
..
.\" Set up some character translations and predefined strings.  \*(-- will
.\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left
.\" double quote, and \*(R" will give a right double quote.  \*(C+ will
.\" give a nicer C++.  Capital omega is used to do unbreakable dashes and
.\" therefore won't be available.  \*(C` and \*(C' expand to `' in nroff,
.\" nothing in troff, for use with C<>.
.tr \(*W-
.ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p'
.ie n \{\
.    ds -- \(*W-
.    ds PI pi
.    if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch
.    if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\"  diablo 12 pitch
.    ds L" ""
.    ds R" ""
.    ds C` ""
.    ds C' ""
'br\}
.el\{\
.    ds -- \|\(em\|
.    ds PI \(*p
.    ds L" ``
.    ds R" ''
'br\}
.\"
.\" Escape single quotes in literal strings from groff's Unicode transform.
.ie \n(.g .ds Aq \(aq
.el       .ds Aq '
.\"
.\" If the F register is turned on, we'll generate index entries on stderr for
.\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index
.\" entries marked with X<> in POD.  Of course, you'll have to process the
.\" output yourself in some meaningful fashion.
.ie \nF \{\
.    de IX
.    tm Index:\\$1\t\\n%\t"\\$2"
..
.    nr % 0
.    rr F
.\}
.el \{\
.    de IX
..
.\}
.\"
.\" Accent mark definitions (@(#)ms.acc 1.5 88/02/08 SMI; from UCB 4.2).
.\" Fear.  Run.  Save yourself.  No user-serviceable parts.
.    \" fudge factors for nroff and troff
.if n \{\
.    ds #H 0
.    ds #V .8m
.    ds #F .3m
.    ds #[ \f1
.    ds #] \fP
.\}
.if t \{\
.    ds #H ((1u-(\\\\n(.fu%2u))*.13m)
.    ds #V .6m
.    ds #F 0
.    ds #[ \&
.    ds #] \&
.\}
.    \" simple accents for nroff and troff
.if n \{\
.    ds ' \&
.    ds ` \&
.    ds ^ \&
.    ds , \&
.    ds ~ ~
.    ds /
.\}
.if t \{\
.    ds ' \\k:\h'-(\\n(.wu*8/10-\*(#H)'\'\h"|\\n:u"
.    ds ` \\k:\h'-(\\n(.wu*8/10-\*(#H)'\`\h'|\\n:u'
.    ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'^\h'|\\n:u'
.    ds , \\k:\h'-(\\n(.wu*8/10)',\h'|\\n:u'
.    ds ~ \\k:\h'-(\\n(.wu-\*(#H-.1m)'~\h'|\\n:u'
.    ds / \\k:\h'-(\\n(.wu*8/10-\*(#H)'\z\(sl\h'|\\n:u'
.\}
.    \" troff and (daisy-wheel) nroff accents
.ds : \\k:\h'-(\\n(.wu*8/10-\*(#H+.1m+\*(#F)'\v'-\*(#V'\z.\h'.2m+\*(#F'.\h'|\\n:u'\v'\*(#V'
.ds 8 \h'\*(#H'\(*b\h'-\*(#H'
.ds o \\k:\h'-(\\n(.wu+\w'\(de'u-\*(#H)/2u'\v'-.3n'\*(#[\z\(de\v'.3n'\h'|\\n:u'\*(#]
.ds d- \h'\*(#H'\(pd\h'-\w'~'u'\v'-.25m'\f2\(hy\fP\v'.25m'\h'-\*(#H'
.ds D- D\\k:\h'-\w'D'u'\v'-.11m'\z\(hy\v'.11m'\h'|\\n:u'
.ds th \*(#[\v'.3m'\s+1I\s-1\v'-.3m'\h'-(\w'I'u*2/3)'\s-1o\s+1\*(#]
.ds Th \*(#[\s+2I\s-2\h'-\w'I'u*3/5'\v'-.3m'o\v'.3m'\*(#]
.ds ae a\h'-(\w'a'u*4/10)'e
.ds Ae A\h'-(\w'A'u*4/10)'E
.    \" corrections for vroff
.if v .ds ~ \\k:\h'-(\\n(.wu*9/10-\*(#H)'\s-2\u~\d\s+2\h'|\\n:u'
.if v .ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'\v'-.4m'^\v'.4m'\h'|\\n:u'
.    \" for low resolution devices (crt and lpr)
.if \n(.H>23 .if \n(.V>19 \
\{\
.    ds : e
.    ds 8 ss
.    ds o a
.    ds d- d\h'-1'\(ga
.    ds D- D\h'-1'\(hy
.    ds th \o'bp'
.    ds Th \o'LP'
.    ds ae ae
.    ds Ae AE
.\}
.rm #[ #] #H #V #F C
.\" ========================================================================
.\"
.IX Title "SQUIDANALYZER 1"
.TH SQUIDANALYZER 1 "2014-03-15" "perl v5.14.2" "User Contributed Perl Documentation"
.\" For nroff, turn off justification.  Always turn off hyphenation; it makes
.\" way too many mistakes in technical documents.
.if n .ad l
.nh
.SH "NAME"
SquidAnalyzer \- Squid access log report generation tool
.SH "DESCRIPTION"
.IX Header "DESCRIPTION"
SquidAnalyzer parse native access log format of the Squid proxy and
generate general statistics about hits, bytes, users, networks, top
url and top second level domain.
.PP
Statistic reports are oriented to user and bandwidth control, this
is not a pure cache statistics generator. SquidAnalyzer use flat
files to store data and don't need any \s-1SQL\s0, \s-1SQL\s0 Lite or Berkeley
databases.
.PP
This analyzer is incremental so it should be run in a daily cron.
Take care if you have rotate log enable to run it before rotation
is done.
.SH "REQUIREMENT"
.IX Header "REQUIREMENT"
Nothing is required than a modern perl version 5.8 or higher. Graphics
are based on the Flotr2 Javascript library so they are drawn at your
browser side without extra installation required.
.SH "INSTALLATION"
.IX Header "INSTALLATION"
.SS "Generic install"
.IX Subsection "Generic install"
If you want the package to be intalled into the Perl distribution just
do the following:
.PP
.Vb 3
\&    perl Makefile.PL
\&    make
\&    make install
.Ve
.PP
Follow the instruction given at the end of install. With this default install
everything configurable will be installed under /etc/squidanalyzer. The Perl
library SquidAnalyzer.pm will be installed under your site_perl directory and
the squid-analyzer Perl script will be copied under /usr/local/bin.
.PP
The default output directory for html reports will be /var/www/squidanalyzer/.
.PP
On FreeBSD, if make install is freezing and you have the following messages:
.PP
.Vb 3
\&        FreeBSD: Registering installation in the package database
\&        FreeBSD: Cannot determine short module description
\&        FreeBSD: Cannot determine module description
.Ve
.PP
please proceed as follow:
.PP
.Vb 3
\&        perl Makefile.PL INSTALLDIRS=site
\&        make
\&        make install
.Ve
.PP
as the issue is related to an install into the default Perl vendor installdirs
it will then use Perl site installdirs.
.SS "Custom install"
.IX Subsection "Custom install"
You can create your fully customized SquidAnalyzer installation by using the
Makefile.PL Perl script. Here is a sample:
.PP
.Vb 8
\&        perl Makefile.PL \e
\&                LOGFILE=/var/log/squid3/access.log \e
\&                BINDIR=/usr/bin \e
\&                CONFDIR=/etc \e
\&                HTMLDIR=/var/www/squidreport \e
\&                BASEURL=/squidreport \e
\&                MANDIR=/usr/man/man3 \e
\&                DOCDIR=/usr/share/doc/squidanalyzer
.Ve
.PP
If you want to build a distro package, there are two other options that you may use. The \s-1QUIET\s0 option is to tell to Makefile.PL to not show the default post install \s-1README\s0. The \s-1DESTDIR\s0 is to create and install all files in a package build base directory. For example for Fedora \s-1RPM\s0, thing may look like that:
.PP
.Vb 12
\&        # Make Perl and SendmailAnalyzer distrib files
\&        %{_\|_perl} Makefile.PL \e
\&            INSTALLDIRS=vendor \e
\&            QUIET=1 \e
\&            LOGFILE=/var/log/squid/access.log \e
\&            BINDIR=%{_bindir} \e
\&            CONFDIR=%{_sysconfdir} \e
\&            BASEDIR=%{_localstatedir}/lib/%{uname} \e
\&            HTMLDIR=%{webdir} \e
\&            MANDIR=%{_mandir}/man3 \e
\&            DOCDIR=%{_docdir}/%{uname}\-%{version} \e
\&            DESTDIR=%{buildroot} < /dev/null
.Ve
.PP
See spec file in packaging/RPM for full \s-1RPM\s0 build script.
.SS "Local install"
.IX Subsection "Local install"
You can also have a custom installation. Just copy the SquidAnalyzer.pm and the
squid-analyzer perl script into a directory, copy and modify the configuration
file and run the script from here with the \-c option.
.PP
Then copy files sorttable.js, squidanalyzer.css and logo\-squidanalyzer.png into
the output directory.
.SS "Post installation"
.IX Subsection "Post installation"
1. Modify your httpd.conf to allow access to \s-1HTML\s0 output like follow:
.PP
.Vb 8
\&        Alias /squidreport /var/www/squidanalyzer
\&        <Directory /var/www/squidanalyzer>
\&            Options \-Indexes FollowSymLinks MultiViews
\&            AllowOverride None
\&            Order deny,allow
\&            Deny from all
\&            Allow from 127.0.0.1
\&        </Directory>
.Ve
.PP
2. If necessary, give additional host access to SquidAnalyzer in httpd.conf.
Restart and ensure that httpd is running.
.PP
3. Browse to http://my.host.dom/squidreport/ to ensure that things are working
properly.
.PP
4. Setup a cronjob to run squid-analyzer daily or more often:
.PP
.Vb 2
\&        # SquidAnalyzer log reporting daily
\&        0 2 * * * /usr/local/bin/squid\-analyzer > /dev/null 2>&1
.Ve
.PP
or run it manually. For more information, see \s-1README\s0 file.
.PP
If your squid logfiles are rotated then cron isn't going to give the expected
result as there exists a time between when the cron is run and the logfiles
are rotated. It would be better to call squid-analyzer from logrotate, eg:
.PP
.Vb 12
\&        /var/log/proxy/squid\-access.log {
\&            daily
\&            compress
\&            rotate 730
\&            missingok
\&            nocreate
\&            sharedscripts
\&            postrotate
\&                test ! \-e /var/run/squid.pid || /usr/sbin/squid \-k rotate
\&                /usr/bin/squid\-analyzer \-d \-l /var/log/proxy/squid\-access.log.1
\&            endscript
\&        }
.Ve
.PP
You can also use network name instead of network ip addresses by using the
network-aliases file. Also if you don't have authentication enable and
want to replace client ip addresses by some know user or computer you
can use the user-aliases file to do so.
.PP
See the file squidanalyzer.conf to customized your output statistics and
match your network and file system configuration.
.SH "USAGE"
.IX Header "USAGE"
SquidAnalyzer can be run manually or by cron job using the squid-analyzer Perl
script. Here are authorized usage:
.PP
.Vb 1
\&    Usage: squid\-analyzer [ \-c squidanalyzer.conf ] [\-l logfile]
\&
\&        \-c | \-\-configfile filename : path to the SquidAnalyzer configuration file.
\&                                     By default: /etc/squidanalyzer.conf
\&        \-b | \-\-build_date date     : set the day to be rebuilt, format: yyyy\-mm\-dd,
\&                                     yyyy\-mm or yyyy. Used with \-r or \-\-rebuild.
\&        \-d | \-\-debug               : show debug informations.
\&        \-h | \-\-help                : show this message and exit.
\&        \-l | \-\-logfile filename    : path to the Squid logfile to parse.
\&                                     By default: /var/log/squid/access.log
\&        \-p | \-\-preserve number     : used to set the statistic obsolescence in
\&                                     number of month. Older stats will be removed.
\&        \-P | \-\-pid_dir directory   : set directory where pid file will be stored.
\&                                     Default /tmp/
\&        \-r | \-\-rebuild             : use this option to rebuild all html and graphs
\&                                     output from all data files.
\&        \-v | version               : show version and exit.
\&        \-\-no\-year\-stat             : disable years statistics, reports will
\&                                     start from month level only.
.Ve
.PP
There is special options like \-\-rebuild that force SquidAnalyzer to rebuild all
\&\s-1HTML\s0 reports, useful after an new feature or a bug fix. If you want to limit the
rebuild to a single day, a single month or year, you can use the \-\-build_date
option by specifying the date part to rebuild, format: yyyy-mm-dd, yyyy-mm or yyyy.
.PP
The \-\-preserve option should be used if you want to rotate your statistics and
data. The value is the number of months to keep, older reports and data will be
removed from the filesystem. Useful to preserve space, for example:
.PP
.Vb 1
\&        squid\-analyzer \-p 6 \-c /etc/squidanalyzer/squidanalyzer.conf
.Ve
.PP
will only preserve six month of statistics from the last run of squidanalyzer.
.SH "CONFIGURATION"
.IX Header "CONFIGURATION"
Unless previous version customization of SquidAnalyzer is now
done by a single configuration file squidanalyzer.conf.
.PP
Here follow the configuration directives used by Squid Analyzer.
.IP "Output output_directory" 4
.IX Item "Output output_directory"
Where SquidAnalyzer should dump all \s-1HTML\s0, data and images files.
You should give a path that can be read by a Web browser.
.IP "WebUrl" 4
.IX Item "WebUrl"
The \s-1URL\s0 of the SquidAnalyzer javascript, \s-1HTML\s0 and images files.
Default: /squidreport
.IP "CustomHeader" 4
.IX Item "CustomHeader"
This directive allow you to replace the SquidAnalyze logo by your custom
logo. The default value is defined as follow:
.Sp
.Vb 3
\&        <a href="$self\->{WebUrl}">
\&        <img src="$self\->{WebUrl}images/logo\-squidanalyzer.png" title="SquidAnalyzer $VERSION" border="0">
\&        </a> SquidAnalyzer
.Ve
.Sp
Feel free to define your own header but take care to not break current design.
For example:
.Sp
.Vb 2
\&        CustomHeader   <a href="http://my.isp.dom/"><img src="http://my.isp.dom/logo.png" title="My ISP link" border="0" width="100" height="110"></a> My ISP Company
\&                                                                                           126,1         Bas
.Ve
.IP "LogFile squid_access_log_file" 4
.IX Item "LogFile squid_access_log_file"
Set the path to the Squid log file.
.IP "UseClientDNSName        0" 4
.IX Item "UseClientDNSName        0"
If you want to use \s-1DNS\s0 name instead of client Ip address as username enable
this directive. When you don't have authentication, the username is set to
the client ip address, this allow you to use the \s-1DNS\s0 name instead.
Note that you must have a working \s-1DNS\s0 resolution and that it can really slow
down the generation of reports.
.IP "DNSLookupTimeout        0.0001" 4
.IX Item "DNSLookupTimeout        0.0001"
If you have enabled UseClientDNSName and have lot of ip addresses that do not
resolve you may want to increase the \s-1DNS\s0 lookup timeout. By default SquidAnalyzer
will stop to lookup a \s-1DNS\s0 name after 0.0001 second (100 ms).
.IP "NetworkAlias network\-aliases_file" 4
.IX Item "NetworkAlias network-aliases_file"
Set path to the file containing network alias name. Network are
show as Ip addresses so if you want to display name instead
create a file with this format:
.Sp
.Vb 1
\&    LOCATION_NAME IP_NETWORK_ADDRESS
.Ve
.Sp
Separator must be a tabulation.
.Sp
You can use regex to match and group some network addresses. See
network-aliases file for examples.
.IP "UserAlias user\-aliases_file" 4
.IX Item "UserAlias user-aliases_file"
Set path to the file containing user alias name. If you don't have
auth_proxy enable users are seen as ip addresses. So if you want to
show username or computer name instead, create a file with this format:
.Sp
.Vb 1
\&    FULL_USERNAME IP_ADDRESS
.Ve
.Sp
If you have auth_proxy enable but want to replace login name by full
user name for example, create a file with this format:
.Sp
.Vb 1
\&    FULL_USERNAME LOGIN_NAME
.Ve
.Sp
Separator for both must be a tabulation.
.Sp
You can use regex to match and group some user login or ip addresses. See
user-aliases file for examples.
.Sp
You can also replace default ip address by his \s-1DNS\s0 name by enabling
directive 'UseClientDNSName'.
.IP "AnonymizeLogin  0" 4
.IX Item "AnonymizeLogin  0"
Set this to 1 if you want to anonymize all user login. The username
will be replaced by an unique id that change at each squid-analyzer
run. Default disable.
.IP "OrderNetwork bytes|hits|duration" 4
.IX Item "OrderNetwork bytes|hits|duration"
.PD 0
.IP "OrderUser bytes|hits|duration" 4
.IX Item "OrderUser bytes|hits|duration"
.IP "OrderUrl bytes|hits|duration" 4
.IX Item "OrderUrl bytes|hits|duration"
.PD
Used to set how SquidAnalyzer sort Network, User and User detailed Urls
reports screen. Value can be: bytes, hits or duration. Default is bytes.
Note that OrderUrl is limited to User detailed Urls reports and does not
apply to Top Url and Top domain report where there is three reports each
already ordered.
.IP "OrderMime bytes|hits" 4
.IX Item "OrderMime bytes|hits"
Used to set how SquidAnalyzer sort Mime types report screen
Value can be: bytes or hits. Default is bytes.
.IP "UrlReport 0|1" 4
.IX Item "UrlReport 0|1"
Should SquidAnalyzer display user details. This will show all
\&\s-1URL\s0 read by user. Take care to have enougth space disk for large
user. Default is 0, no url detail report.
.IP "QuietMode 0|1" 4
.IX Item "QuietMode 0|1"
Run in quiet mode for batch processing or print debug information.
Default is 0, verbose mode.
.IP "CostPrice price/Mb" 4
.IX Item "CostPrice price/Mb"
Used to set a cost of the bandwith per Mb. If you want to generate
invoice per Mb for bandwith traffic this can help you. Value 0 mean
no cost, this is the default value, the \*(L"Cost\*(R" column is not displayed
.IP "Currency currency_abreviation" 4
.IX Item "Currency currency_abreviation"
Used to set the currency of the bandwith cost. Preferably the html
special character. Default is &euro;
.IP "TopNumber number" 4
.IX Item "TopNumber number"
Used to set the number of top url and second level domain to show.
Default is top 100.
.IP "TopUrlUser Use this directive to show the top N users that look at an \s-1URL\s0 or a domain. Set it to 0 to disable this feature. Default is top 10." 4
.IX Item "TopUrlUser Use this directive to show the top N users that look at an URL or a domain. Set it to 0 to disable this feature. Default is top 10."
.PD 0
.IP "Exclude exclusion_file" 4
.IX Item "Exclude exclusion_file"
.PD
Used to set client ip addresses, network addresses, auth login or
uri to exclude from report.
.Sp
You can define one by line exclusion by specifying first the type of the
exclusion (\s-1USER\s0, \s-1CLIENT\s0 or \s-1URI\s0) and a space separated list of valid regex.
.Sp
You can also use the \s-1NETWORK\s0 type to define network address with netmask
using the \s-1CIDR\s0 notation: xxx.xxx.xxx.xxx/n
.Sp
See example bellow:
.Sp
.Vb 7
\&        NETWORK        192.168.1.0/24 10.10.0.0/16
\&        CLIENT         192\e.168\e.1\e.2 
\&        CLIENT         10\e.169\e.1\e.\ed+ 192\e.168\e.10\e..*
\&        USER           myloginstr
\&        USER           guestlogin\ed+ guestdemo
\&        URI            http:\e/\e/myinternetdomain.dom.*
\&        URI            .*\e.webmail\e.com\e/.*\e/login\e.php.*
.Ve
.Sp
you can have multiple line of the same exclusion type.
.IP "Include inclusion_file" 4
.IX Item "Include inclusion_file"
Used to set client ip addresses, network addresses or auth login
to include into the report. All others will not be included. It
works as the opposite of the Include parameter.
.Sp
You can define one by line inclusion by specifying first the type of the
inclusion (\s-1USER\s0 or \s-1CLIENT\s0) and a space separated list of valid regex.
.Sp
You can also use the \s-1NETWORK\s0 type to define network address with netmask
using the \s-1CIDR\s0 notation: xxx.xxx.xxx.xxx/n
.Sp
See example bellow:
.Sp
.Vb 7
\&        NETWORK        192.168.1.0/24 10.10.0.0/16
\&        CLIENT         192\e.168\e.1\e.2 
\&        CLIENT         10\e.169\e.1\e.\ed+ 192\e.168\e.10\e..*
\&        USER           myloginstr
\&        USER           guestlogin\ed+ guestdemo
\&        URI            http:\e/\e/myinternetdomain.dom.*
\&        URI            .*\e.webmail\e.com\e/.*\e/login\e.php.*
.Ve
.Sp
you can have multiple line of the same inclusion type.
.IP "Lang language_file" 4
.IX Item "Lang language_file"
Used to set the translation file to be used. Value must be set to
a file containing all string translated. See the lang directory
for translation files. Default is defined internally in English.
.IP "DateFormat" 4
.IX Item "DateFormat"
Date format used to display date (year = \f(CW%y\fR, month = \f(CW%m\fR and day = \f(CW%d\fR)
You can also use \f(CW%M\fR to replace month by its 3 letters abbreviation.
Default: \f(CW%y\fR\-%m\-%d
.IP "SiblingHit" 4
.IX Item "SiblingHit"
Adds peer cache hit (\s-1CD_SIBLING_HIT\s0) to be taken has local cache hit.
Enabled by default, you must disabled it if you don't want to report
peer cache hit onto your stats.
.IP "TransfertUnit" 4
.IX Item "TransfertUnit"
Allow to change the default unit used to display transfert size. Default
is \s-1BYTES\s0, other possible values are \s-1KB\s0, \s-1MB\s0 and \s-1GB\s0.
.IP "MinPie" 4
.IX Item "MinPie"
Minimum percentage of data in pie's graphs to not be placed in the others
item. Lower values will be summarized into the others item.
.IP "Locale" 4
.IX Item "Locale"
Set this to your locale to display generated date in your language. Default
is to use the current locale of the system. If you want date in German for
example, set it to de_DE.
.Sp
.Vb 1
\&        Rapport genere le mardi 11 decembre 2012, 15:13:09 (UTC+0100).
.Ve
.Sp
with a Locale set to fr_FR.
.SH "AUTHOR"
.IX Header "AUTHOR"
Gilles \s-1DAROLD\s0 <gilles@darold.net>
.SH "COPYRIGHT"
.IX Header "COPYRIGHT"
Copyright (c) 2001\-2013 Gilles \s-1DAROLD\s0
.PP
This package is free software and published under the \s-1GPL\s0 v3 or above
license.