2006-09-20 Sancho Lerena <slerena@artica.es>

* en/pandora.xml: Changes in copyright notice and other minor stuff.

	* en/pandora_introduction.xml: Major cut/paste info from older
	docs. Formatting and some graphics inline. Initial first version
	of introduction chapter for main/install pandora doc. Please
	review it MANY times before publish it.

	* en/pandora_install.xml: More formatting and text added. Second
	version, with new stuff from last revision. Please review it MANY
	times before publish it.

	* en/images/*.jpg: New files added for introduction chapter.
	



git-svn-id: https://svn.code.sf.net/p/pandora/code/trunk@180 c3f86ba8-e40f-0410-aaad-9ba5e7f4b01f

pandora_doc/en/pandora.xml

@@ -3,6 +3,7 @@
 	  "http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd" [
 <!ENTITY pandora_version "v1.2">
 <!ENTITY pandora "<emphasis>Pandora</emphasis>">
+<!ENTITY include_pandora_introduction SYSTEM "pandora_introduction.xml">
 <!ENTITY include_pandora_install SYSTEM "pandora_install.xml">
 <!ENTITY include_fdl SYSTEM "fdl.xml">
 <!ENTITY include_gpl SYSTEM "gpl.xml">
@@ -11,11 +12,12 @@
 <book lang="en">
   <bookinfo>
     <title>&pandora; &pandora_version;</title>
-    <graphic fileref="./img/pandora_logo.jpg" valign="bottom" align="center"/>
+    <graphic fileref="./images/pandora_logo.jpg" valign="bottom" align="center"/>
     <copyright>
       <year>2006</year>
       <holder>
-	Ártica Soluciones Tecnológicas S.L, Sancho Lerena, Esteban Sánchez y otros.
+	Ártica Soluciones Tecnológicas S.L, Sancho Lerena, Esteban
+	Sánchez y otros.
       </holder>
     </copyright>
     <date>11/09/2006</date>
@@ -42,14 +44,16 @@
 	<date>11 Sept 2006</date>
 	<revremark>Submitted.</revremark>
       </revision>
+
       <revision>
-	<revnumber>0.1</revnumber>
+	<revnumber>1.1</revnumber>
-	<date>11 Sept 2006</date>
+	<date>20 Sept 2006</date>
 	<revremark>First draft for review.</revremark>
       </revision>
     </revhistory>
   </bookinfo>
-  
+
+  &include_pandora_introduction; 
   &include_pandora_install;
   &include_fdl;
   &include_gpl;

pandora_doc/en/pandora_introduction.xml

@@ -0,0 +1,418 @@
+<?xml version="1.0" encoding="utf-8"?>
+<chapter>
+  <title>Introduction to Pandora</title>
+  <sect1><title>Pandora. The Free monitoring system</title>
+  <para>
+    Pandora is a monitoring application to watch systems and
+    applications. Pandora allows to know the status of any element of
+    your bussiness systems. Pandora watch for your hardware, your
+    software, your multilayer system and of course your Operating
+    System. Pandora could detect a network interface down and the
+    movement of any value of the NASDAQ new technology market. If you
+    want, Pandora could sent a SMS message when your systems
+    fails... or when Google value low below 330$.
+  </para>
+  <para>
+    Pandora adapt, like an octopus, to your systems and requirements,
+    because has been designed to be open, modular, multiplattform and
+    easy to customize.
+  </para>
+  </sect1>
+  <sect1>
+    <title>Pandora detailed introduction.</title>
+    <para>
+      &pandora; is a monitoring tool that allows a system
+      administrator to visually analise the status and efficiency of
+      Operating Systems, Servers, Applications and Hardware Systems -
+      such as firewalls, proxies, databases, Web servers, tunnelling
+      servers, routers, switches, processes, services, remote access
+      servers, etc. - all integrated into an open and distributed
+      architecture.  Pandora can be implemented over any operating
+      system, with specific agents for each platform. Pandora can also
+      monitor any TCP/IP hardware system, such as load balancers,
+      routers, switches, printers, etc.
+      <graphic fileref="images/esquema.png" scale="50" align="center"/>
+      Pandora architecture is formed of four main components:
+      <itemizedlist mark='bullet'>
+      <listitem>
+	<para>
+	  <emphasis>Web Console</emphasis>: Pandora's user
+	  interface. The user controls and operates the system with
+	  it. Several Web consoles can be implemented in a single
+	  system. The Web console is written in PHP, and rests on a
+	  database and a Web server. It is compatible with any
+	  platform - GNU/Linux, Solaris, Win2000, AIX, etc. However,
+	  the official supported platform is GNU/Linux.
+	</para>
+	<para>
+	  The console permits the user to control the status of the
+	  agents, view statistical information, generate graphs and
+	  data tables, keep a system incident control, as well as to
+	  generate reports and change the alerts, agents, and user
+	  profile settings.
+	</para>
+      </listitem>
+      <listitem>
+	<para>
+	  <emphasis>Server</emphasis>: The core server is the receptor
+	  of the data packages and generates the alerts - it is the
+	  brain of the system. Several servers can work alongside for
+	  larger systems. It has been developed in Perl and works over
+	  any platform, although, the official platform is GNU/Linux.
+	</para>
+	<para>
+	  The core server accesses Pandora database, which is shared
+	  with the Web server, and stores the processed data
+	  packages. Server executes as daemon, and processes the
+	  packages stored in its file system. Data is generated by the
+	  system agents. Despite the server's low system resources
+	  comsumption and simple installation and operation, the core
+	  server is the most critical element of the system. The core
+	  server receives and processes the produced data, and fires
+	  the alerts and the events.
+	</para>
+	<para>
+	  With the new Pandora 1.2 Network Server technology, Pandora
+	  Network Servers could monitorize remote systems using
+	  network resources like ICMP, TCP, UDP or SNMP
+	  Queries. Network Servers are acting itself like "Network
+	  Agents".
+	</para>
+      </listitem>
+      <listitem>
+	<para>
+	  <emphasis>Central Database</emphasis>: At the moment the
+	  system only supports MySQL. The central database keeps all
+	  the information Pandora needs to work - agent data,
+	  settings, user information, incidents, system settings,
+	  etc. The system can use a MySQL cluster to store the
+	  information, or a high disponibility solution for larger
+	  sytems.
+	</para>
+	<para>
+	  This database can work with any of the platform officially
+	  supported by MySQL. Pandora can be implemented with MySQL
+	  versions 3.0 and 4.0, although the latest is recommended.
+	</para>
+      </listitem>
+      <listitem>
+	<para>
+	  <emphasis>Pandora Agents</emphasis>: They collect all the
+	  system's data. They are executed in each local system,
+	  although they can also collect remote information by
+	  intalling monitoring sytems for the agent in several
+	  different machines - called satellite agents.
+	</para>
+	<para>
+	  They have been developed to work under a specific platform,
+	  making use of the specific tools of the used language:
+	  ShellScripting for Unix - which includes GNU/Linux, Solaris,
+	  AIX, HP-UX and BSD, as well as the Nokia's IPSO. Pandora
+	  agents can be developed in virtually any language, given its
+	  simple API and being open source. Windows agent are
+	  developed in a free development enviroment for C++ and uses
+	  the same interface and modularity than Unix agents.
+	</para>
+	<para>
+	  The old agent for Windows plattforms was developed on VBS
+	  Scripting language, and is deprecated with the new Pandora
+	  1.2 windows agent.
+	</para>
+      </listitem>
+      </itemizedlist>
+       <graphic fileref="images/pandora_arch1.jpg" scale="70" align="center"/>
+    </para>   
+    </sect1>
+  <sect1><title>What kind of systems/ services can be monitored?</title>
+  <para>
+    At present, with Pandora any process or system that through a
+    command returns a value can be monitored, as well as any value in
+    any Operating System log file or similar. Some examples of already
+    existing implementations can be the following ones:
+    <programlisting>      
+      Number of connections (sessions) of Checkpoint FW-1
+      Number of NAT sessions of Checkpoint FW-1
+      Number of connections of Linux NetFilter / IPTables firewall
+      Number of FW-1 logged packets 
+      Number of FW-1 dropped packets 
+      Number of FW-1 accepted packets
+      State of High Availability in FW1 NG
+      Last policy installed in a Firewall-1 module
+      Synchronization state of the modules in FW1 NG
+      CPU of the system: idle, user and system
+      Number of processes of the system
+      Temperature of the CPU of a system
+      Value of a MS Windows registry entry
+      Queued jobs in a generic dispatcher
+      Memory of the system: free, swap, kernel Fw-1, cache
+      Percentage of free space on disc (for different partitions)
+      Messages processed by a mail gateway
+      Existence of a string in a text file
+      IP traffic (filtering based on the connections of the firewall)
+      Hits of pages in HTTP Servers (Apache, iPlanet, IIS, Netscape)
+      Percentage of erroneous packets in a Gateway
+      Connections established in a Remote Access Server (RAS)
+      Size of a file
+      Open sessions by a VPN server
+      MySQL Performance: Threads, queries, sessions...
+      Snort system state
+      Reported events by IDS (Snort) up to six levels of priority
+      Network load
+      Number of local Connections (TCP, UDP, Unix sockets)
+      Detected viruses by a Web Antivirus Gateway
+      ICMP latency time towards a host
+      Rate of average transference in a file transfer tool
+      Number of DNS requests attended by a server (including types)
+      Number of FTP sessions attended by a FTP server
+      (Generic) State of any active process / service in the system
+      (Generic) State of any countable parameter of the system
+    </programlisting>
+  </para>
+  <sect2><title>Global architecture</title>
+  <para>
+    Pandora 1.2 has changed many things from 1.1 version, but this
+    graph representing Pandora architecture is very useful to
+      understand in a single graph, all components.
+      <graphic fileref="images/pandora_arch2.jpg" scale="45" align="center"/>
+  </para>
+  </sect2>
+  </sect1>
+  <sect1><title>Information gathering with Pandora agents</title>
+  <para>
+    Pandora agents are based on native languages in every platform:
+    scripts that can be written in any language. It's possible to
+    reproduce any agent in any programming language and can be
+    extended without difficulty the existing ones in order to cover
+    aspects not taken into account up to the moment.
+  </para>
+  <para>
+    These scripts are formed by modules that each one gathers a
+    "chunk" of information. Thus, every agent gathers several "chunks"
+    of information; this one is organized in a data set and stored in
+    a single file, called data file.
+  </para>
+  <para>
+    The process of transferring the data file from the agent to the
+    server is made regularly at a defined time interval in the agent
+    configuration file, pandora_agent.conf. It's possible to modify
+    that parameter to not fill the database with non-relevant
+    information, not to load the network or to not affect the system
+    performance. The default interval is 300 (seconds), which is
+    equivalent to five minutes. Minor values of 100 (seconds) are not
+    recommended since host performance can be affected, besides
+    loading excessively Database and the Operating System of Pandora
+    Server. Pandora is not a real time system; it's an applications
+    and systems general monitoring system in environments that are not
+    critical at real time.
+  </para>
+  <para>
+    Packets transfers are made via SSH, with DSA authentication
+    (although also RSA can be used). The process is completely safe
+    since neither any password nor unencrypted confidential
+    information is sent. Confidentiality, integrity and authentication
+    of the connections between the agent and the server are
+    ensured. In the Agents and Server Installation and Configuration
+    guides, the process of generation of keys to do the automatic SCP
+    transfer is detailed.
+  </para>
+  <para>
+    Also the transfer via FTP or any other file transfer system could
+    be made, although SSH has been chosen for security and
+    compatibility with most of the systems in the market.
+  </para>
+  <para>
+    Pandora Agents are thought to be executed from the agent from
+    which they gather information, although the agents can gather
+    information of accessible machines from the host where they are
+    installed. In this case those agents are called "Satellite
+    Agents". These Satellite Agents can use Telnet, SNMP or any other
+    commands to get the information.
+  </para>
+  <para>
+    We can also have a host with several agents: Some that gather
+    information from the accessible machines (acting as "satellite
+    agents") and the Standard Agent that monitors the host where it's
+    running.
+  </para>
+  <sect2><title>XML Data files</title>
+  <para>
+    The data file has the following syntax:
+    <programlisting>
+      hostname.serialnumber.data 
+    </programlisting>
+    This is an XML file, and its name is the combination of the
+    hostname where the agent runs, a different serial number for every
+    data package and the extension .data that indicates that it's a
+    data file.
+  </para>
+  <para>
+    We also have a control file for every data file:
+    <programlisting>
+      hostname.serialnumber.checksum
+    </programlisting>
+    This file has .checksum extension and contains a MD5 hash of the
+    data file. This allows checking that the information has not been
+    changed before being processed.
+  </para>
+  <para>   
+    The XML data file generated by every agent is the core of
+    Pandora. This file has the information gathered by the Agent. Its
+    easy structure allows that any user could create it's own
+    developments to be processed in Pandora, or use the included ones.
+    An example of the information included into the data file is the
+    following one:
+  <screen>
+    <![CDATA[
+    <agent data os_name="SunOS" os_version="5.8" timestamp="300"
+    agent_name="pdges01" version="1.0">
+      <module>
+        <name>SSH Daemon</name>
+        <type>generic_proc</type>
+        <data>1</data>
+      </module>
+      <module>
+        <name>FTP Daemon</name>
+        <type>generic_proc</type>
+        <data>0</data>
+      </module>
+      <module>
+        <name>DiskFree</name>
+        <type>generic_data</type>
+        <data>5200000</data>
+      </module>
+      <module>
+        <name>UsersConnected</name>
+        <type>generic_data_inc</type>
+        <data>119</data>
+        <min>1</min>
+        <max>250</max>
+        <description>Users currently connected</description>
+      </module>
+      <module>
+        <name>LastLogin</name>
+        <type>generic_data_string</type>
+        <data>slerena</data>
+      </module>
+    </agent_data>
+    ]]>
+  </screen>
+  </para>
+  </sect2>
+  <sect2><title>Pandora servers</title>
+  <para>
+    Pandora Server is a Perl script that processes the information
+    sent by the agents. The agents send the XML data file via SSH and
+    the server periodically verifies if it has new data files waiting
+    to be processed.
+  </para>
+  <para>
+    Data are extracted from the data file, identifying origin, type
+    and category. One classified, the data are inserted into the
+    Database by the same Perl script.
+  </para>
+  <para>
+    Pandora Server can work in High Availability and/or Load
+    Balancing. In a very big architecture, several Pandora Servers can
+    be arranged simultaneously to be able to manage big volumes of
+    information distributed by geographical or functional zones.
+  </para>
+  <para>
+    Pandora Server is always running (as a daemon) and permanently
+    verifies if some element causes to fire an alarm. If so, it
+    executes the action defined in the alarm, as to send a SMS, an
+    email, to activate the execution of a SCRIPT or to send an HTTP
+    form.
+  </para>
+  <para>
+    We could have several simultaneous servers, one of them is the
+    Main Server or "Master Server " and the rest servers are "Slave
+    Servers". The Master Server is the only one that verifies the
+    alarms if any agent goes down. The server who receives the data
+    file from the agent always fires the rest of alarms, defined in
+    the agents' modules. This is also important if this server changes
+    (due to configurations of high availability, load balancing or
+    clustering).
+  </para>
+  </sect2>
+  
+  <sect2><title>Pandora console</title>
+  <para>
+    The Web Console it's a web application that allows to see
+    graphical reports, state of every agent, and to access to the
+    information sent by the agent, to see every monitored parameter
+    and to see its evolution throughout the time, to form the
+    different nodes, groups and users of the system. It is the part
+    that interacts with the final user, and that allows you to
+    administer the system.
+  </para>
+  <para>
+    The Web Console is written in PHP and no plug-in, Flash, Java or
+    ActiveX is needed to access the console, only a browser that
+    supports HTML and CSS (IE5+ o Mozilla 4+). Pandora Web Console can
+    run in several servers, the only thing you need is to access
+    Pandora Database, where Pandora stores all the information.
+  </para>
+  </sect2>
+  
+  <sect2><title>Pandora database</title>
+  <para>
+    Pandora uses a SQL Database to store all the information. Pandora
+    maintains an asynchronous database with all the received data,
+    making a temporary cohesion of everything what it receives and
+    normalizing all the information from the different sources. Every
+    Agent data module generates an entry of information for every data
+    bundle, which implies that a real production system can have of
+    the order of ten million of data, or information atoms.
+  </para>
+  <para>
+    This information is managed automatically from Pandora, carrying
+    out a periodic and automatic maintenance of the database. This
+    allows that Pandora should need neither any type of administration
+    of database nor process attended by an operator or manager. This
+    is made by a periodic purge of the past information over a date
+    (by default 90 days), as well as a data compaction of the data
+    that have more than, by default, 30 days.
+  </para>
+
+  <sect3><title>Compacting data</title>
+  <para>
+    Data stored by Pandora are useful to see evolutions regard through
+    the time, to make statistics, to generate reports and to do
+    capacity planning, as well as other tasks of statistical
+    nature. For it, it isn't necessary to have all the data, but it's
+    enough to have a representative sample, of smaller resolution,
+    enough to carry out the task that is needed.
+  </para>
+  <para>
+    With that philosophy the compaction system has been
+    constructed. If we have a sample of 9.000 elements, distributed
+    during 90 days, for example, Pandora is going to take the data of
+    last month, which would be 3.000 elements and it's going them to
+    compress them in 300. In the graphs they will practically be seen
+    equal, which it will serve us for the reports, statistics and
+    other tasks. This is made by means of interpolation in temporary
+    strips, in a totally automatic and periodic way, without the user
+    or the administrator must himself or herself worry about it.
+  </para>
+  </sect3>
+  </sect2>
+</sect1>
+
+<sect1>
+  <title>About Pandora
+  </title>
+  <para>
+    Pandora is a project initiated and mainly developed by Sancho
+    Lerena, at present other people is working on it: Raúl Mateos,
+    David Villanueva, Esteban Sánchez, Jose Navarro and Jonathan
+    Barajas. We want to give thanks for many other people who help us
+    with translation, graphic design, bugs reporting and interesting
+    ideas.
+  </para>
+  <para>
+    Pandora is Free Software, and is published under GPL Licence. In
+    order to know the last features, go to the official web site of
+    the project in http://pandora.sourceforge.net.
+  </para>
+</sect1>
+</chapter>