tyler.durden/loganalyzer
1
0
Fork 0
mirror of https://github.com/rsyslog/loganalyzer.git synced 2025-09-21 17:08:00 +02:00
Code Issues Packages Projects Releases Wiki Activity

added general doc and search description doc

Browse Source 
the search doc was actually done by Andre, I am just commiting it
This commit is contained in:
Rainer Gerhards 2008-10-01 16:48:30 +02:00
parent aca944b579
commit 84cac3e7e3
3 changed files with 274 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

176
doc/basics.html Normal file
View File

@ -0,0 +1,176 @@
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html><head>
<title>phpLogCon Basics</title>
</head>
<body>
<h1>phpLogCon Basics</h1>
<p>This documentation describes some basic concepts of phpLogCon. It
is meant as a useful guide for understanding concepts, the application itself
as well as the rest of the documentation.
<h2>Configuring phpLogCon</h2>
<p>phpLogCon is configured via a master configuration file. If the userDB system
(see below) is enabled, most settings can be made via the web-application itself
without any further need to change the config file.
<p>Intial configuration can also be done during a setup wizard, which writes
an initial configuration file itself. This wizard can also enable the userDB system,
so in this case you probably never need to touch the configuration file.
<h2>Data Sources</h2>
<p>A data source is a set of syslog (and other) data that is gathered. Data sources
can be text files or databases. Any text file is supported, as long as it contains
purely printable characters and LF is used as a line terminator. This applies to
all regular text files so in short you can use whatver is present in text format.
Obviously, these are files written by the syslogd, but this also includes any
other text file, e.g. written by an application as its log file. Note that at
this time, a file data source can contain exactly one file (and NOT multiple).
For the database, tabels in either MonitorWare format or the format used by
php-syslog-ng is supported. We support php-syslog-ng schema mainly for migration
scenarios: it is sub-optimal in that it uses text strings where integer values are sufficient.
This results in the need for more database space plus slower response time. If
you set up something new, be sure the use MonitorWare schema. If you use
<a href="http://www.rsyslog.com">rsyslog</a> to create the database, please note
that rsyslog uses MonitorWare schema by default, too. So you probably need not to
do anything special.
<h2>Searches</h2>
<p>With phpLogCon, you can search any datasource for a variety of properties.
By default, text is searched within the msg part of the message. However, quite
complex searches can be performed. It is suggested to use the "advanced search"
button to build these. Note that searches are done via http get requests. That
means you can copy and paste an url (or bookmark and email it) and that URL
will contain a complete source. This is actually a great way to send searches
to a co-worker or have some automatted process (eg via cron &quot; wget) pull specific
data on a periodic schedule.
<p>The search part of phpLogCon is used much like any major search engine
and hopefully is quite intuitive.
<p>Note that a search is currently limited to a single data source only. If you
would like to search across different data sources, you need to do this on the basis
of individual data sources. To support you at least a bit in the process, there
is a special page (for historical reasons called "the Oracle") which generates canned
searches for you, so that you only need to click the individual search links to perform
these searches. We know this is not perfect, but we hope this is useful.
<h3>Context Links and Menues</h3>
<p>At appropriate places, phpLogCon generates context links to potentially helpful
information. For example, links to domain or IP range owner lookup or troubleshooting
information (via the external <a href="http://kb.monitorware.com/">knowledge base</a>)
are generated. At other places, links to the cross-datasource search capability are
generated.
<p>Consider this example to understand why it is done and how it may be useful. Let's
assume you have two data sources, one with your firewall log and one with the mail log.
Now you wonder why a recent spam attack could happen. You review the mail log and find
indication of the spammer. Now, you can lookup the IP addresses and domain names used.
Probably more interesting, you can invoke the cross-datasource search tool and obtain
information on what the firewall log has recorded about the IPs in question. Finally,
you may want to check the online knowledge based if there is some information
recorded about this or a similar event (for example, if you found a message that
puzzels you). You may even ask you peers for help via the knowledge base.
<p>In addition to the links, there are helpful popup menues for most of the
properties being displayed. To find out what you can do, please simply click the values
(even those that do NOT look like a link) and see what the popup has to offer (and, yes,
we will improve doc on these topics... ;)).
<h2>Windows Event Log Data</h2>
<p>PhpLogCon contains automatic support for displaying Windows Event Log data
in a useful format if that data is generated by either the
<a href="http://www.eventreporter.com/">EventReporter</a>
or the
<a href="http://www.msagent.com/">MonitorWare Agent</a>
forwarding agents. This includes proper detection, and ability to filter, on
event-log specific properties (like Windows event id and such).
<h2>The UserDB System</h2>
<p>The so-called userDB system allows different user accounts to be created and
user-specific settings to be made. This is a great aid if multiple people share
a single instance of phpLogCon.
<p>The userDB system is disabled by default. This is because
a database is needed for the userDB system as user profiles are stored
inside it. Setting up the database tables requires some additonal work,
so we do not expect users to do this by default. Please do not confuse this with
database (log) data sources: these are not necessarily needed. You can use
the userDB system and still store the log data in text files (which may be
desirable from a performance point of view). To enable the userDB system,
the phpLogCon configuration file needs to be changed. Note that once the
userDB system is active, most system settings can be made via the web application.
<p>There are basically two types of users: admins and non-admins. Admins can
change anything, non-admins can only change their personal preferences.
<p>The userDB system is not yet a strong security tool, but helps greatly with
moderate security needs. User groups can be created and data sources be assigned
to a specific user group. So only users of this group can access the data
source in question. This is useful if you have a group of people caring about
the firewall logs and another group caring about the mail logs. You can then define
two different groups and assign the data source accordingly. Then, assign the
user's the group they should belong to. The end result is that every user only
sees what he or she is expected to see.
<h2>Do I need a Database?</h2>
<p>Not necessarily. PhpLogCon only needs databases if you plan to use
the userDB system or use database data sources. Without that, no database is
needed. A typical scenario, for example, is private review of server-based
syslog files. For this use case, no database is required.
<h2>Securing phpLogCon</h2>
<p>Log data is very valuable to an attacker. So it is highly suggested that you
secure access to any phpLogCon instance, especially if it contains live log
data. We suggest to place it on local, non-internet accessible servers, only.
In a hosting environment, it may be useful to place it on an internet-accessible
server. In this case, access should be protected on the http layer at least.
In any case, the use of https is suggested to prevent accidental loss
of confidentiality (this is important in the local network, too!).
<p>The userDB system can be used as a tool to tune user's ability to
view data sources (users can only view those sources that belong to
one of their groups). However, this is currently considered a secondary access
control mechanism. An Internet-accessible instance of phpLogCon should
NOT rely on that as the sole source of protection.
<p>Please note that this section gives just a few rough, common-sense
recommendations. Evaluate the risk yourself, check with your policies and
do not blame us if you made a mistake ;) <b>In short: use phpLogCon at your
sole risk</b>, and reduce this risk by thinking about what you do.
<p>You may consider to purchase
<a href="professional_services.html">professional services</a>
if you are serious about the risk implications in an (enterprise) environment.
<h2>External Tool Integration</h2>
<p>As already described, phpLogCon accesses external tools, most importantly the
<a href="http://kb.monitorware.com">MonitorWare Knowledge Base</a> to aid you
in your analysis effort. The goal is to provide useful information that helps
you get the job done better and quicker.
<p>No private data (except the obious one, eg a domain name for a domain search)
is provided to the external entity nor is anything recorded (except for what
can be seen from regular web logs). However, you need to decide yourself, as with
all external accesses, if this functionality fits into your security policy.
<p>Please note that the external tool is able to call back into your local
phpLogCon installation if you provide it with the local URL. This can be done via
the user profile in the external part. Note that both the local as well as
the external part are programmed in such a way that no private data (except
for the URL) needs to be kept externally and, most important, local data
is never visible to a third-party observer. The notable exception is if you
host your local phpLogCon on an external server without any further security
measure (e.g. https, access restrictions). Even in that case the external
part will not be able to access data from the local part, but an attacker
may find it easy to get hold of your log data (which is always the case
with such a local setup and not specifically related to the local/external integration).
<h2>Upgrades</h2>
<p>PhpLogCon is actively being developed. So chances are good you want to
upgrade to a later release at some time. In general, it is always a good idea
to backup everything before you upgrade. We highly recommend doing so. Other than that,
the upgrade should be fairly easy without the installed userDB system. If the
userDB system is installed, it may be necessary to upgrade the database schema. This
can be done via the admin center. No access to phpLogCon is possible unless the
database schema has been upgraded.
<p>Again, as a general precaution, you should have a solid backup available
before you run the upgrade procedure (and this specifically applies to any
database content!).
<p>[<a href="manual.html">manual index</a>] [<a href="http://www.phplogcon.com/">phpLogCon site</a>]</p>
<p><font size="2">This documentation is part of the
<a href="http://www.phplogcon.com/">phpLogCon</a>
project.<br>
Copyright &copy; 2008 by
<a href="http://www.adiscon.com/">Adiscon</a>.
Released under the GNU GPL version 3 or higher.</font></p>
</body></html>

3
doc/manual.html
View File

@ -18,7 +18,8 @@ Data can be obtained from databases but also from plain text files, for
example those that are written by the syslgod.
<p><b>Follow the links below for the</b><br></p>
<ul>
<li><xa href="">phpLogCon Search syntax</a></li>
<li><a href="basics.html">phpLogCon basics <b>Important - read this!</b></a></li>
<li><a href="searching.html">phpLogCon Search syntax</a></li>
<li><xa href="">phpLogCon configuration parameters</a></li>
<li><a href="troubleshoot.html">troubleshooting phpLogCon problems</a></li>
<li><a href="professional_services.html">phpLogCon professional services</a></li>

96
doc/searching.html Normal file
View File

@ -0,0 +1,96 @@
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html><head>
<title>phpLogCon Search Syntax</title>
</head>
<body>
<h1>phpLogCon Search Syntax</h1>
<p>This document describes how you can search in phpLogCon. Please note that
you can also always use the "advanced search" dialog to build the search string.
<p>A search term is always in the format "property:search" where property is the
data item you will search (for example the syslog tag, source system).
Available properties depend on the data source in use, parsers present,
and even the actual data. What usually is available is listed below. Please note that there
is a special case. Because it is so common to search inside the message text itself,
there is no property name for this - to perform this search, simply omit the
"property:" part.
<p>In phpLogCon we basically have two different types of data types which
are handled different from each other, these are: string and number. There is
also a date type filter and the message field, but these are handled separately
from the others fields. Every filter usually is delimited by spaces, and filters
from different fields are combined by AND.
<h3>String Fields</h3>
<p>For the string type fields, you can search for partially or full string
matches. Searching for full string matches can improve performance in your
logstream. If you have spaces in your search string, you need to replace them
with a + character, the reason is simple, because the space character splits the
filters. If you want to search for a single + sign, you need to replace it with
a double +, so + becomes ++ in your search string. To separate multiple search
values for the same fieldtype, you just need to separate them commas - but
remember, no spaces!
<p>String filters can be included in your search as well as excluded. All
included filter of a field type will be combined by OR which means if you search
for &quot;Search1&quot; and &quot;Search2&quot;, you will results matching both strings. This is
different if you exclude strings from your search. Excluded filters from a field
type will be combined with AND for the simple reason, if you want to exclude
&quot;Search1&quot; and &quot;Search2&quot;, you want that both are excluded, and one or the other.
<p>Below you will see how the syntax looks like:<ul>
<li>Search for a partially string match: property:search</li>
<li>Search for multiple partially string matches: property:search1,search2</li>
<li>Search for a full string match: property:=search1,search2</li>
<li>Search for multiple full string matches: property:=search</li>
<li>Exclude by a partially string match: property:-search</li>
<li>Exclude multiple partially string matches: property:-search1,-search2</li>
<li>Exclude by a full string match: property:-=search</li>
<li>Exclude multiple full string matches: property:-=search1,-=search2</li>
</ul>
<p>&nbsp;<h3>Number Fields</h3>
<p>Number fields work similar to string fields, except that there are partially
matches, and the search value obviously must be a number. There are a few
exceptions where you can use string representations instead of numbers, but
these will be explained later.
<p>Below you will see how the syntax looks like:<ul>
<li>Search for a number: property:1234</li>
<li>Search for multiple numbers: property:1234,5678</li>
<li>Exclude a number from the search: property:-1234</li>
<li>Exclude multiple numbers from the search: property:-1234,-5678</li>
</ul>
<p>&nbsp;</p>
<h3>Date Field</h3>
<p>There is currently only one date field possible, and so we have two different
kinds of date filters we can use on this date field. The first option is to
search for a time range where you need to use the datefrom and dateto
properties. The date/time format has to be ISO TIMESTAMP (see RFC 3339 for more)
for both properties. The other option is datelastx which will filter for the
last hour (filter value 1), last 12 hours (filter value 2), last 24 hours
(filter value 3), last 7 days (filter value 4) and last 31 days (filter value
5). </p>
<ul>
<li>This search will filter for events starting at 2007-10-01 and ending at
2008-10-01: datefrom:2007-10-1T00:00:00 dateto:2008-10-1T23:59:59</li>
<li>This search will show events from the last 24 hours: datelastx:3</li>
</ul>
<p>&nbsp;</p>
<h3>Message Field</h3>
<p>The message field is a string type field and does not require the use of a
property. You can have include and exclude filters here as well. The main
difference to ordinary string filters is, that include filters are combined with
AND instead of OR. </p>
<ul>
<li>This search will include &quot;test1&quot;: test1</li>
<li>This search will include &quot;test1&quot; and &quot;test2&quot;: test1 test2</li>
<li>This search will exclude &quot;test1&quot;: -test1</li>
<li>This search will exclude &quot;test1&quot; and &quot;test2&quot;: -test1 -test2</li>
</ul>
<p>&nbsp;<p>&nbsp;<p>[<a href="manual.html">manual index</a>] [<a href="http://www.phplogcon.com/">phpLogCon site</a>]</p>
<p><font size="2">This documentation is part of the
<a href="http://www.phplogcon.com/">phpLogCon</a>
project.<br>
Copyright &copy; 2008 by
<a href="http://www.adiscon.com/">Adiscon</a>.
Released under the GNU GPL version 3 or higher.</font></p>
</body></html>