[[getting-started]] == Getting started with Beats and the Elastic Stack Looking for an "ELK tutorial" that shows how to set up the Elastic stack for Beats? You've come to the right place. The topics in this section describe how to install and configure the Elastic stack for Beats. A regular _Beats setup_ consists of: * Elasticsearch for storage and indexing. See <>. * Logstash (optional) for inserting data into Elasticsearch. See <>. * Kibana for the UI. See <>. * One or more Beats. You install the Beats on your servers to capture operational data. See <>. * Kibana dashboards for visualizing the data. See the https://www.elastic.co/support/matrix[Elastic Support Matrix] for information about supported operating systems and product compatibility. NOTE: To get started, you can install Elasticsearch and Kibana on a single VM or even on your laptop. The only condition is that the machine must be accessible from the servers you want to monitor. As you add more Beats and your traffic grows, you'll want to replace the single Elasticsearch instance with a cluster. You'll probably also want to automate the installation process. [[elasticsearch-installation]] === Install Elasticsearch https://www.elastic.co/products/elasticsearch[Elasticsearch] is a real-time, distributed storage, search, and analytics engine. It can be used for many purposes, but one context where it excels is indexing streams of semi-structured data, such as logs or decoded network packets. The binary packages of Elasticsearch have only one dependency: Java. The minimum supported version is Java 8. To download and install Elasticsearch, use the commands that work with your system (<> for Debian/Ubuntu, <> for Redhat/Centos/Fedora, <> for OS X, and <> for Windows): [[deb]]*deb:* ifeval::["{release-state}"=="unreleased"] Version {stack-version} of Elasticsearch has not yet been released. endif::[] ifeval::["{release-state}"!="unreleased"] ["source","sh",subs="attributes,callouts"] ---------------------------------------------------------------------- sudo apt-get install openjdk-8-jre curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-{ES-version}.deb sudo dpkg -i elasticsearch-{ES-version}.deb sudo /etc/init.d/elasticsearch start ---------------------------------------------------------------------- endif::[] [[rpm]]*rpm:* ifeval::["{release-state}"=="unreleased"] Version {stack-version} of Elasticsearch has not yet been released. endif::[] ifeval::["{release-state}"!="unreleased"] ["source","sh",subs="attributes,callouts"] ---------------------------------------------------------------------- sudo yum install java-1.8.0-openjdk curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-{ES-version}.rpm sudo rpm -i elasticsearch-{ES-version}.rpm sudo service elasticsearch start ---------------------------------------------------------------------- endif::[] [[mac]]*mac:* ifeval::["{release-state}"=="unreleased"] Version {stack-version} of Elasticsearch has not yet been released. endif::[] ifeval::["{release-state}"!="unreleased"] ["source","sh",subs="attributes,callouts"] ---------------------------------------------------------------------- # install Java, e.g. from: https://www.java.com/en/download/manual.jsp curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-{ES-version}.zip unzip elasticsearch-{ES-version}.zip cd elasticsearch-{ES-version} ./bin/elasticsearch ---------------------------------------------------------------------- endif::[] [[win]]*win:* ifeval::["{release-state}"=="unreleased"] Version {stack-version} of Elasticsearch has not yet been released. endif::[] ifeval::["{release-state}"!="unreleased"] . If necessary, download and install the latest version of the Java from https://www.java.com[www.java.com]. . Download the Elasticsearch {ES-version} Windows zip file from the https://www.elastic.co/downloads/elasticsearch[downloads page]. . Extract the contents of the zip file to a directory on your computer, for example, `C:\Program Files`. . Open a command prompt as an Administrator and navigate to the directory that contains the extracted files, for example: + ["source","sh",subs="attributes,callouts"] ---------------------------------------------------------------------- cd C:\Program Files\elasticsearch-{ES-version} ---------------------------------------------------------------------- . Run the following command to start Elasticsearch: + ["source","sh",subs="attributes,callouts"] ---------------------------------------------------------------------- bin\elasticsearch.bat ---------------------------------------------------------------------- endif::[] You can learn more about installing, configuring, and running Elasticsearch in the https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html[Elasticsearch Reference]. ==== Make sure Elasticsearch is up and running To test that the Elasticsearch daemon is up and running, try sending an HTTP GET request on port 9200. [source,shell] ---------------------------------------------------------------------- curl http://127.0.0.1:9200 ---------------------------------------------------------------------- On Windows, if you don't have cURL installed, simply point your browser to the URL. You should see a response similar to this: ["source","sh",subs="attributes"] ---------------------------------------------------------------------- { "name" : "ofgAtrJ", "cluster_name" : "elasticsearch", "cluster_uuid" : "3h9xSrVlRJmDHgQ8FLnByA", "version" : { "number" : "{ES-version}", "build_hash" : "db0d481", "build_date" : "2017-02-09T22:05:32.386Z", "build_snapshot" : false, "lucene_version" : "6.4.1" }, "tagline" : "You Know, for Search" } ---------------------------------------------------------------------- [[logstash-installation]] === Install Logstash (Optional) The simplest architecture for the Beats platform setup consists of one or more Beats, Elasticsearch, and Kibana. This architecture is easy to get started with and sufficient for networks with low traffic. It also uses the minimum amount of servers: a single machine running Elasticsearch and Kibana. The Beats insert the transactions directly into the Elasticsearch instance. If you want to perform additional processing or buffering on the data, however, you'll want to install Logstash. An important advantage to this approach is that you can use Logstash to modify the data captured by Beats in any way you like. You can also use Logstash's many output plugins to integrate with other systems. To download and install Logstash, use the commands that work with your system: *deb:* ifeval::["{release-state}"=="unreleased"] Version {stack-version} of Logstash has not yet been released. endif::[] ifeval::["{release-state}"!="unreleased"] ["source","sh",subs="attributes,callouts"] ---------------------------------------------------------------------- sudo apt-get install openjdk-8-jre curl -L -O https://artifacts.elastic.co/downloads/logstash/logstash-{LS-version}.deb sudo dpkg -i logstash-{LS-version}.deb ---------------------------------------------------------------------- endif::[] *rpm:* ifeval::["{release-state}"=="unreleased"] Version {stack-version} of Logstash has not yet been released. endif::[] ifeval::["{release-state}"!="unreleased"] ["source","sh",subs="attributes,callouts"] ---------------------------------------------------------------------- sudo yum install java-1.8.0-openjdk curl -L -O https://artifacts.elastic.co/downloads/logstash/logstash-{LS-version}.rpm sudo rpm -i logstash-{LS-version}.rpm ---------------------------------------------------------------------- endif::[] *mac:* ifeval::["{release-state}"=="unreleased"] Version {stack-version} of Logstash has not yet been released. endif::[] ifeval::["{release-state}"!="unreleased"] ["source","sh",subs="attributes,callouts"] ---------------------------------------------------------------------- # install Java, e.g. from: https://www.java.com/en/download/manual.jsp curl -L -O https://artifacts.elastic.co/downloads/logstash/logstash-{LS-version}.zip unzip logstash-{LS-version}.zip ---------------------------------------------------------------------- endif::[] *win:* ifeval::["{release-state}"=="unreleased"] Version {stack-version} of Logstash has not yet been released. endif::[] ifeval::["{release-state}"!="unreleased"] . If necessary, download and install the latest version of the Java from https://www.java.com[www.java.com]. . Download the Logstash {LS-version} Windows zip file from the https://www.elastic.co/downloads/logstash[downloads page]. . Extract the contents of the zip file to a directory on your computer, for example, `C:\Program Files`. Don't start Logstash yet. You need to set a couple of configuration options first. endif::[] [[logstash-setup]] ==== Set up Logstash In this setup, the Beat sends events to Logstash. Logstash receives these events by using the {logstashdoc}/plugins-inputs-beats.html[Beats input plugin for Logstash] and then sends the transaction to Elasticsearch by using the {logstashdoc}/plugins-outputs-elasticsearch.html[Elasticsearch output plugin for Logstash]. The Elasticsearch output plugin uses the bulk API, making indexing very efficient. To set up Logstash, you create a Logstash pipeline configuration file that configures Logstash to listen on port 5044 for incoming Beats connections and to index into Elasticsearch. For example, you can save the following example configuration to a file called `logstash.conf`: [source,ruby] ------------------------------------------------------------------------------ input { beats { port => 5044 } } # The filter part of this file is commented out to indicate that it is # optional. # filter { # # } output { elasticsearch { hosts => "localhost:9200" manage_template => false index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" <1> document_type => "%{[@metadata][type]}" <2> } } ------------------------------------------------------------------------------ <1> `%{[@metadata][beat]}` sets the first part of the index name to the value of the `beat` metadata field, `%{[@metadata][version]}` sets the second part of the name to the beat's version, and `%{+YYYY.MM.dd}` sets the third part of the name to a date based on the Logstash `@timestamp` field. For example: +{beatname_lc}-2017.03.29+. <2> `%{[@metadata][type]}` sets the document type based on the value of the `type` metadata field. For Beats, this value resolves to `doc`. NOTE: Starting with Logstash 6.0, the `document_type` option is deprecated due to the https://www.elastic.co/guide/en/elasticsearch/reference/6.0/removal-of-types.html[removal of types in Logstash 6.0]. It will be removed in the next major version of Logstash. If you are running Logstash 6.0 or later, you do not need to set `document_type` in your configuration because Logstash sets the type to `doc` by default. When you run Logstash with this configuration, it indexes events into Elasticsearch in the same way that the Beat would, but you get access to other capabilities provided by Logstash for collecting, enriching, and transforming data. See the {logstashdoc}/introduction.html[Logstash introduction] for more information about these capabilities. To use this setup, you'll also need to configure your Beat to use Logstash. For more information, see the documentation for the Beat. [[logstash-input-update]] ===== Update the Beats input plugin for Logstash Plugins have their own release cycle and are often released independent of Logstash’s core release cycle. To ensure that you have the latest version of the https://www.elastic.co/guide/en/logstash/current/plugins-inputs-beats.html[Beats input plugin for Logstash], run the following command from your Logstash installation: *deb, rpm, and mac:* ["source","sh",subs="attributes,callouts"] ---------------------------------------------------------------------- ./bin/logstash-plugin update logstash-input-beats ---------------------------------------------------------------------- *win:* ["source","sh",subs="attributes,callouts"] ---------------------------------------------------------------------- bin\logstash-plugin update logstash-input-beats ---------------------------------------------------------------------- Keep in mind that you can update to the latest version of the plugin without having to upgrade to a newer version of Logstash. More details about working with input plugins in Logstash are available https://www.elastic.co/guide/en/logstash/current/working-with-plugins.html[here]. ==== Start Logstash Now you can start Logstash. Use the command that works with your system. If you installed Logstash as a deb or rpm package, make sure the config file is in the expected directory. *deb:* ["source","sh",subs="attributes,callouts"] ---------------------------------------------------------------------- sudo /etc/init.d/logstash start ---------------------------------------------------------------------- *rpm:* ["source","sh",subs="attributes,callouts"] ---------------------------------------------------------------------- sudo service logstash start ---------------------------------------------------------------------- *mac:* ["source","sh",subs="attributes,callouts"] ---------------------------------------------------------------------- ./bin/logstash -f logstash.conf ---------------------------------------------------------------------- *win:* ["source","sh",subs="attributes,callouts"] ---------------------------------------------------------------------- bin\logstash.bat -f logstash.conf ---------------------------------------------------------------------- NOTE: The default configuration for Beats and Logstash uses plain TCP. For encryption you must explicitly enable SSL when you configure Beats and Logstash. You can learn more about installing, configuring, and running Logstash https://www.elastic.co/guide/en/logstash/current/getting-started-with-logstash.html[here]. [[kibana-installation]] === Install Kibana https://www.elastic.co/products/kibana[Kibana] is a visualization application that gets its data from Elasticsearch. It provides a customizable and user-friendly UI in which you can combine various widget types to create your own dashboards. The dashboards can be easily saved, shared, and linked. For getting started, we recommend installing Kibana on the same server as Elasticsearch, but it is not required. If you install the products on different servers, you'll need to change the URL (IP:PORT) of the Elasticsearch server in the Kibana configuration file, `config/kibana.yml`, before starting Kibana. Use the following commands to download and run Kibana. *deb or rpm:* ifeval::["{release-state}"=="unreleased"] Version {stack-version} of Kibana has not yet been released. endif::[] ifeval::["{release-state}"!="unreleased"] ["source","sh",subs="attributes,callouts"] ---------------------------------------------------------------------- curl -L -O https://artifacts.elastic.co/downloads/kibana/kibana-{Kibana-version}-linux-x86_64.tar.gz tar xzvf kibana-{Kibana-version}-linux-x86_64.tar.gz cd kibana-{Kibana-version}-linux-x86_64/ ./bin/kibana ---------------------------------------------------------------------- endif::[] *mac:* ifeval::["{release-state}"=="unreleased"] Version {stack-version} of Kibana has not yet been released. endif::[] ifeval::["{release-state}"!="unreleased"] ["source","sh",subs="attributes,callouts"] ---------------------------------------------------------------------- curl -L -O https://artifacts.elastic.co/downloads/kibana/kibana-{Kibana-version}-darwin-x86_64.tar.gz tar xzvf kibana-{Kibana-version}-darwin-x86_64.tar.gz cd kibana-{Kibana-version}-darwin-x86_64/ ./bin/kibana ---------------------------------------------------------------------- endif::[] *win:* ifeval::["{release-state}"=="unreleased"] Version {stack-version} of Kibana has not yet been released. endif::[] ifeval::["{release-state}"!="unreleased"] . Download the Kibana {Kibana-version} Windows zip file from the https://www.elastic.co/downloads/kibana[downloads page]. . Extract the contents of the zip file to a directory on your computer, for example, `C:\Program Files`. . Open a command prompt as an Administrator and navigate to the directory that contains the extracted files, for example: + ["source","sh",subs="attributes,callouts"] ---------------------------------------------------------------------- cd C:\Program Files\kibana-{Kibana-version}-windows ---------------------------------------------------------------------- . Run the following command to start Kibana: + ["source","sh",subs="attributes,callouts"] ---------------------------------------------------------------------- bin\kibana.bat ---------------------------------------------------------------------- endif::[] You can find Kibana binaries for other operating systems on the https://www.elastic.co/downloads/kibana[Kibana downloads page]. ==== Launch the Kibana web interface To launch the Kibana web interface, point your browser to port 5601. For example, http://127.0.0.1:5601[http://127.0.0.1:5601]. You can learn more about Kibana in the http://www.elastic.co/guide/en/kibana/current/index.html[Kibana User Guide].