tyler.durden/icingabeat
1
0
Fork 0
mirror of https://github.com/Icinga/icingabeat.git synced 2025-08-16 07:18:08 +02:00
Code Issues Packages Projects Releases Wiki Activity
icingabeat/vendor/github.com/elastic/beats/libbeat/docs/gettingstarted.asciidoc
Blerim Sheqa dd3104e1c6 Update to libbeat 6.1
2017-12-19 13:16:39 +01:00

493 lines
17 KiB
Plaintext
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

[[getting-started]]
== Getting started with Beats and the Elastic Stack
Looking for an "ELK tutorial" that shows how to set up the Elastic stack for Beats? You've
come to the right place. The topics in this section describe how to install and configure
the Elastic stack for Beats.
A regular _Beats setup_ consists of:
* Elasticsearch for storage and indexing. See <<elasticsearch-installation>>.
* Logstash (optional) for inserting data into Elasticsearch. See <<logstash-installation,Installing Logstash>>.
* Kibana for the UI. See <<kibana-installation>>.
* One or more Beats. You install the Beats on your servers to capture operational data. See <<installing-beats>>.
* Kibana dashboards for visualizing the data.
See the https://www.elastic.co/support/matrix[Elastic Support Matrix] for information
about supported operating systems and product compatibility.
NOTE: To get started, you can install Elasticsearch and Kibana on a
single VM or even on your laptop. The only condition is that the machine must be
accessible from the servers you want to monitor. As you add more Beats and
your traffic grows, you'll want to replace the single Elasticsearch instance with
a cluster. You'll probably also want to automate the installation process.
[[elasticsearch-installation]]
=== Install Elasticsearch
https://www.elastic.co/products/elasticsearch[Elasticsearch] is a real-time,
distributed storage, search, and analytics engine. It can be used for many
purposes, but one context where it excels is indexing streams of semi-structured
data, such as logs or decoded network packets.
The binary packages of Elasticsearch have only one dependency: Java. The minimum
supported version is Java 8. To download and install
Elasticsearch, use the commands that work with your system
(<<deb, deb>> for Debian/Ubuntu, <<rpm, rpm>> for Redhat/Centos/Fedora, <<mac,
mac>> for OS X, and <<win, win>> for Windows):
[[deb]]*deb:*
ifeval::["{release-state}"=="unreleased"]
Version {stack-version} of Elasticsearch has not yet been released.
endif::[]
ifeval::["{release-state}"!="unreleased"]
["source","sh",subs="attributes,callouts"]
----------------------------------------------------------------------
sudo apt-get install openjdk-8-jre
curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-{ES-version}.deb
sudo dpkg -i elasticsearch-{ES-version}.deb
sudo /etc/init.d/elasticsearch start
----------------------------------------------------------------------
endif::[]
[[rpm]]*rpm:*
ifeval::["{release-state}"=="unreleased"]
Version {stack-version} of Elasticsearch has not yet been released.
endif::[]
ifeval::["{release-state}"!="unreleased"]
["source","sh",subs="attributes,callouts"]
----------------------------------------------------------------------
sudo yum install java-1.8.0-openjdk
curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-{ES-version}.rpm
sudo rpm -i elasticsearch-{ES-version}.rpm
sudo service elasticsearch start
----------------------------------------------------------------------
endif::[]
[[mac]]*mac:*
ifeval::["{release-state}"=="unreleased"]
Version {stack-version} of Elasticsearch has not yet been released.
endif::[]
ifeval::["{release-state}"!="unreleased"]
["source","sh",subs="attributes,callouts"]
----------------------------------------------------------------------
# install Java, e.g. from: https://www.java.com/en/download/manual.jsp
curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-{ES-version}.zip
unzip elasticsearch-{ES-version}.zip
cd elasticsearch-{ES-version}
./bin/elasticsearch
----------------------------------------------------------------------
endif::[]
[[win]]*win:*
ifeval::["{release-state}"=="unreleased"]
Version {stack-version} of Elasticsearch has not yet been released.
endif::[]
ifeval::["{release-state}"!="unreleased"]
. If necessary, download and install the latest version of the Java from https://www.java.com[www.java.com].
. Download the Elasticsearch {ES-version} Windows zip file from the
https://www.elastic.co/downloads/elasticsearch[downloads page].
. Extract the contents of the zip file to a directory on your computer, for example, `C:\Program Files`.
. Open a command prompt as an Administrator and navigate to the directory that contains the extracted files, for example:
+
["source","sh",subs="attributes,callouts"]
----------------------------------------------------------------------
cd C:\Program Files\elasticsearch-{ES-version}
----------------------------------------------------------------------
. Run the following command to start Elasticsearch:
+
["source","sh",subs="attributes,callouts"]
----------------------------------------------------------------------
bin\elasticsearch.bat
----------------------------------------------------------------------
endif::[]
You can learn more about installing, configuring, and running Elasticsearch in the
https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html[Elasticsearch Reference].
==== Make sure Elasticsearch is up and running
To test that the Elasticsearch daemon is up and running, try sending an HTTP GET
request on port 9200.
[source,shell]
----------------------------------------------------------------------
curl http://127.0.0.1:9200
----------------------------------------------------------------------
On Windows, if you don't have cURL installed, simply point your browser to the URL.
You should see a response similar to this:
["source","sh",subs="attributes"]
----------------------------------------------------------------------
{
"name" : "ofgAtrJ",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "3h9xSrVlRJmDHgQ8FLnByA",
"version" : {
"number" : "{ES-version}",
"build_hash" : "db0d481",
"build_date" : "2017-02-09T22:05:32.386Z",
"build_snapshot" : false,
"lucene_version" : "6.4.1"
},
"tagline" : "You Know, for Search"
}
----------------------------------------------------------------------
[[logstash-installation]]
=== Install Logstash (Optional)
The simplest architecture for the Beats platform setup consists of one or more Beats,
Elasticsearch, and Kibana. This architecture is easy to get started
with and sufficient for networks with low traffic. It also uses the minimum amount of
servers: a single machine running Elasticsearch and Kibana. The Beats
insert the transactions directly into the Elasticsearch instance.
If you want to perform additional processing or buffering on the data, however,
you'll want to install Logstash.
An important advantage to this approach is that you can
use Logstash to modify the data captured by Beats in any way you like. You can also
use Logstash's many output plugins to integrate with other systems.
To download and install Logstash, use the commands that work
with your system:
*deb:*
ifeval::["{release-state}"=="unreleased"]
Version {stack-version} of Logstash has not yet been released.
endif::[]
ifeval::["{release-state}"!="unreleased"]
["source","sh",subs="attributes,callouts"]
----------------------------------------------------------------------
sudo apt-get install openjdk-8-jre
curl -L -O https://artifacts.elastic.co/downloads/logstash/logstash-{LS-version}.deb
sudo dpkg -i logstash-{LS-version}.deb
----------------------------------------------------------------------
endif::[]
*rpm:*
ifeval::["{release-state}"=="unreleased"]
Version {stack-version} of Logstash has not yet been released.
endif::[]
ifeval::["{release-state}"!="unreleased"]
["source","sh",subs="attributes,callouts"]
----------------------------------------------------------------------
sudo yum install java-1.8.0-openjdk
curl -L -O https://artifacts.elastic.co/downloads/logstash/logstash-{LS-version}.rpm
sudo rpm -i logstash-{LS-version}.rpm
----------------------------------------------------------------------
endif::[]
*mac:*
ifeval::["{release-state}"=="unreleased"]
Version {stack-version} of Logstash has not yet been released.
endif::[]
ifeval::["{release-state}"!="unreleased"]
["source","sh",subs="attributes,callouts"]
----------------------------------------------------------------------
# install Java, e.g. from: https://www.java.com/en/download/manual.jsp
curl -L -O https://artifacts.elastic.co/downloads/logstash/logstash-{LS-version}.zip
unzip logstash-{LS-version}.zip
----------------------------------------------------------------------
endif::[]
*win:*
ifeval::["{release-state}"=="unreleased"]
Version {stack-version} of Logstash has not yet been released.
endif::[]
ifeval::["{release-state}"!="unreleased"]
. If necessary, download and install the latest version of the Java from https://www.java.com[www.java.com].
. Download the Logstash {LS-version} Windows zip file from the
https://www.elastic.co/downloads/logstash[downloads page].
. Extract the contents of the zip file to a directory on your computer, for example, `C:\Program Files`.
Don't start Logstash yet. You need to set a couple of configuration options first.
endif::[]
[[logstash-setup]]
==== Set up Logstash
In this setup, the Beat sends events to Logstash. Logstash receives
these events by using the
{logstashdoc}/plugins-inputs-beats.html[Beats input plugin for Logstash]
and then sends the transaction to Elasticsearch by using the
{logstashdoc}/plugins-outputs-elasticsearch.html[Elasticsearch output plugin for Logstash].
The Elasticsearch output plugin uses the bulk API, making indexing very efficient.
To set up Logstash, you create a Logstash pipeline configuration file that
configures Logstash to listen on port 5044 for incoming Beats connections
and to index into Elasticsearch. For example, you can save the following
example configuration to a file called `logstash.conf`:
[source,ruby]
------------------------------------------------------------------------------
input {
beats {
port => 5044
}
}
# The filter part of this file is commented out to indicate that it is
# optional.
# filter {
#
# }
output {
elasticsearch {
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" <1>
document_type => "%{[@metadata][type]}" <2>
}
}
------------------------------------------------------------------------------
<1> `%{[@metadata][beat]}` sets the first part of the index name to the value
of the `beat` metadata field, `%{[@metadata][version]}` sets the second part of
the name to the beat's version, and `%{+YYYY.MM.dd}` sets the third part of the
name to a date based on the Logstash `@timestamp` field. For example:
+{beatname_lc}-2017.03.29+.
<2> `%{[@metadata][type]}` sets the document type based on the value of the `type`
metadata field. For Beats, this value resolves to `doc`.
NOTE: Starting with Logstash 6.0, the `document_type` option is deprecated due to the
https://www.elastic.co/guide/en/elasticsearch/reference/6.0/removal-of-types.html[removal of types in Logstash 6.0].
It will be removed in the next major version of Logstash. If you are running
Logstash 6.0 or later, you do not need to set `document_type` in your
configuration because Logstash sets the type to `doc` by default.
When you run Logstash with this configuration, it indexes events into
Elasticsearch in the same way that the Beat would, but you get access to other
capabilities provided by Logstash for collecting, enriching, and transforming
data. See the {logstashdoc}/introduction.html[Logstash introduction] for more
information about these capabilities.
To use this setup, you'll also need to configure your Beat to use Logstash.
For more information, see the documentation for the Beat.
[[logstash-input-update]]
===== Update the Beats input plugin for Logstash
Plugins have their own release cycle and are often released independent of
Logstash’s core release cycle. To ensure that you have the latest version of
the https://www.elastic.co/guide/en/logstash/current/plugins-inputs-beats.html[Beats input plugin for Logstash],
run the following command from your Logstash installation:
*deb, rpm, and mac:*
["source","sh",subs="attributes,callouts"]
----------------------------------------------------------------------
./bin/logstash-plugin update logstash-input-beats
----------------------------------------------------------------------
*win:*
["source","sh",subs="attributes,callouts"]
----------------------------------------------------------------------
bin\logstash-plugin update logstash-input-beats
----------------------------------------------------------------------
Keep in mind that you can update to the latest version of the plugin without
having to upgrade to a newer version of Logstash. More details about working
with input plugins in Logstash are available
https://www.elastic.co/guide/en/logstash/current/working-with-plugins.html[here].
==== Start Logstash
Now you can start Logstash. Use the command that works with your system. If you
installed Logstash as a deb or rpm package, make sure the config file is in the
expected directory.
*deb:*
["source","sh",subs="attributes,callouts"]
----------------------------------------------------------------------
sudo /etc/init.d/logstash start
----------------------------------------------------------------------
*rpm:*
["source","sh",subs="attributes,callouts"]
----------------------------------------------------------------------
sudo service logstash start
----------------------------------------------------------------------
*mac:*
["source","sh",subs="attributes,callouts"]
----------------------------------------------------------------------
./bin/logstash -f logstash.conf
----------------------------------------------------------------------
*win:*
["source","sh",subs="attributes,callouts"]
----------------------------------------------------------------------
bin\logstash.bat -f logstash.conf
----------------------------------------------------------------------
NOTE: The default configuration for Beats and Logstash uses plain TCP. For
encryption you must explicitly enable SSL when you configure Beats and Logstash.
You can learn more about installing, configuring, and running Logstash
https://www.elastic.co/guide/en/logstash/current/getting-started-with-logstash.html[here].
[[kibana-installation]]
=== Install Kibana
https://www.elastic.co/products/kibana[Kibana] is a visualization application
that gets its data from Elasticsearch. It provides a customizable and
user-friendly UI in which you can combine various widget types to create your
own dashboards. The dashboards can be easily saved, shared, and linked.
For getting started, we recommend installing Kibana on the same server as
Elasticsearch, but it is not required. If you install the products on different servers,
you'll need to change the URL (IP:PORT) of the Elasticsearch server in the
Kibana configuration file, `config/kibana.yml`, before starting Kibana.
Use the following commands to download and run Kibana.
*deb or rpm:*
ifeval::["{release-state}"=="unreleased"]
Version {stack-version} of Kibana has not yet been released.
endif::[]
ifeval::["{release-state}"!="unreleased"]
["source","sh",subs="attributes,callouts"]
----------------------------------------------------------------------
curl -L -O https://artifacts.elastic.co/downloads/kibana/kibana-{Kibana-version}-linux-x86_64.tar.gz
tar xzvf kibana-{Kibana-version}-linux-x86_64.tar.gz
cd kibana-{Kibana-version}-linux-x86_64/
./bin/kibana
----------------------------------------------------------------------
endif::[]
*mac:*
ifeval::["{release-state}"=="unreleased"]
Version {stack-version} of Kibana has not yet been released.
endif::[]
ifeval::["{release-state}"!="unreleased"]
["source","sh",subs="attributes,callouts"]
----------------------------------------------------------------------
curl -L -O https://artifacts.elastic.co/downloads/kibana/kibana-{Kibana-version}-darwin-x86_64.tar.gz
tar xzvf kibana-{Kibana-version}-darwin-x86_64.tar.gz
cd kibana-{Kibana-version}-darwin-x86_64/
./bin/kibana
----------------------------------------------------------------------
endif::[]
*win:*
ifeval::["{release-state}"=="unreleased"]
Version {stack-version} of Kibana has not yet been released.
endif::[]
ifeval::["{release-state}"!="unreleased"]
. Download the Kibana {Kibana-version} Windows zip file from the
https://www.elastic.co/downloads/kibana[downloads page].
. Extract the contents of the zip file to a directory on your computer, for example, `C:\Program Files`.
. Open a command prompt as an Administrator and navigate to the directory that
contains the extracted files, for example:
+
["source","sh",subs="attributes,callouts"]
----------------------------------------------------------------------
cd C:\Program Files\kibana-{Kibana-version}-windows
----------------------------------------------------------------------
. Run the following command to start Kibana:
+
["source","sh",subs="attributes,callouts"]
----------------------------------------------------------------------
bin\kibana.bat
----------------------------------------------------------------------
endif::[]
You can find Kibana binaries for other operating systems on the
https://www.elastic.co/downloads/kibana[Kibana downloads page].
==== Launch the Kibana web interface
To launch the Kibana web interface, point your browser to port 5601. For example,
http://127.0.0.1:5601[http://127.0.0.1:5601].
You can learn more about Kibana in the
http://www.elastic.co/guide/en/kibana/current/index.html[Kibana User Guide].
Reference in New Issue View Git Blame Copy Permalink