https://www.elastic.co/blog/introducing-logstash-input-http-plugin
How do I use this plugin?
By default it will bind the webserver to all hosts ("0.0.0.0") and open the TCP port 8080 but it's
possible configure these settings:
input {
http {
host => "127.0.0.1" # default: 0.0.0.0
port => 31311 # default: 8080
}
}
That's all you need!
What about security?
You can configure basic authentication by setting a username and password. All requests done to Logstash
will then have to set the right credentials or receive a 401 response. Only correctly authenticated requests
will produce an event inside of Logstash. For SSL, it is necessary to specify the path to a Java Keystore
that contains the certificate that clients use to validate the server. Here's an example:
input {
port => 3332
user => myuser
password => "$tr0ngP4ssWD!"
ssl => on
keystore => "/tmp/mykeystore.jks"
keystore_password => "keystore_pass"
}
OK, now show me this plugin in action!
Step 1 - starting Logstash with http input:
bin/logstash -e "input { http { } } output { stdout { codec => rubydebug} }"
Step 2 - That's it!
To test it, let's issue two requests:
% curl -XPUT 'http://127.0.0.1:8080/twitter/tweet/1' -d 'hello'
% curl -H "content-type: application/json" -XPUT 'http://127.0.0.1:8080/twitter/tweet/1' -d '{
"user" : "kimchy",
"post_date" : "2009-11-15T14:12:12",
"message" : "trying out Elasticsearch"
}'
Result in Logstash:
{
"message" => "hello",
"@version" => "1",
"@timestamp" => "2015-05-29T14:49:00.392Z",
"headers" => {
"content_type" => "application/x-www-form-urlencoded",
"request_method" => "PUT",
"request_path" => "/twitter/tweet/1",
"request_uri" => "/twitter/tweet/1",
"http_version" => "HTTP/1.1",
"http_user_agent" => "curl/7.37.1",
"http_host" => "127.0.0.1:8080",
"http_accept" => "*/*",
"content_length" => "5"
}
}
{
"user" => "kimchy",
"post_date" => "2009-11-15T14:12:12",
"message" => "trying out Elasticsearch",
"@version" => "1",
"@timestamp" => "2015-05-29T14:49:04.105Z",
"headers" => {
"content_type" => "application/json",
"request_method" => "PUT",
"request_path" => "/twitter/tweet/1",
"request_uri" => "/twitter/tweet/1",
"http_version" => "HTTP/1.1",
"http_user_agent" => "curl/7.37.1",
"http_host" => "127.0.0.1:8080",
"http_accept" => "*/*",
"content_length" => "110"
}
}
You can see that in the second request, since the content-type was application/json, the body was deserialized
and expanded to the event root (notice the fields "user", "post_date" and "message").
Show me more concrete examples of how to use it!
Because, real world examples make everything clearer!
Elastic Watcher Integration
In this section, we’ll show you how to integrate Elastic Watcher -- the new Elasticsearch plugin for alerting
and notification -- with Logstash. Sending notifications to Logstash via this input provides you a powerful
toolset to further transform notifications and use Logstash’s rich collection of outputs.
Imagine that you have indices with Apache logs, and now we want to get a periodic update of how many requests
are resulting in a 404 (Not Found) response.
The required steps for this are:
Installing Watcher
Creating a new notification on Watcher that every minute reports the number of events that have a 404 response
status
Start Logstash with the HTTP input
Send data to Elasticsearch and watch updates on Logstash
Here we go!
1. Installing Watcher
cd elasticsearch-1.5.2
bin/plugin -i elasticsearch/watcher/latest
bin/plugin -i elasticsearch/license/latest
bin/elasticsearch # restart the server
2. Creating a watch
The Watcher plugin for elasticsearch provides an API to create and manipulate scheduled tasks, or "watches".
A Watch will query the data in the elasticsearch cluster according to its schedule, look for certain scenarios
(like the presence of an error event) and execute actions. Examples of actions are sending an email, writing a
document to an index, calling an outside HTTP endpoint, and more..
For this test, I created a simple watch that:
every minute
counts number of HTTP requests that resulted in a 404
posts result to http://localhost:8080
This is the resulting JSON document I need to send to Watcher:
{
"trigger" : {
"schedule" : { "cron" : "0 0/1 * * * ?" }
},
"input" : {
"search" : {
"request" : {
"indices" : [
"logstash*"
],
"body" : {
"query" : {
"term": { "response": 404 }
}
}
}
}
},
"actions" : {
"my_webhook" : {
"webhook" : {
"auth" : {
"basic" : {
"username" : "guest",
"password" : "guest"
}
},
"method" : "POST",
"host" : "127.0.0.1",
"port" : 8080,
"path": "/{{ctx.watch_id}}",
"body" : "{{ctx.payload.hits.total}}"
}
}
}
}
To install this watch you need to create it in Elasticsearch by executing a PUT request:
curl -XPUT 'http://localhost:9200/_watcher/watch/my-watch' -d @create_webhook.json
3. Logstash setup
wget http://download.elastic.co/logstash/logstash/logstash-1.5.2.tar.gz
tar -zxf logstash-1.5.2.tar.gz
cd logstash-1.5.2
bin/logstash -e "input { http { } } output { stdout { codec => rubydebug} }"
4. Results
After launching an ingestion process in another terminal, Logstash starts receiving 1 notification per
minute in the form of a HTTP POST:
% bin/logstash -e "input { http { } } output { stdout { codec => rubydebug} }"
Logstash startup completed
{
"message" => "330",
"@version" => "1",
"@timestamp" => "2015-06-02T12:53:00.037Z",
"headers" => {
"content_type" => "application/x-www-form-urlencoded",
"request_method" => "POST",
"request_path" => "/my-watch",
"request_uri" => "/my-watch?",
"http_version" => "HTTP/1.1",
"http_authorization" => "Basic Z3Vlc3Q6Z3Vlc3Q=",
"http_accept_charset" => "UTF-8",
"http_cache_control" => "no-cache",
"http_pragma" => "no-cache",
"http_user_agent" => "Java/1.8.0_20",
"http_host" => "127.0.0.1:8080",
"http_accept" => "text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2",
"http_connection" => "keep-alive",
"content_length" => "12"
}
}
{
"message" => "3103",
"@version" => "1",
"@timestamp" => "2015-06-02T12:54:00.030Z",
"headers" => {
"content_type" => "application/x-www-form-urlencoded",
"request_method" => "POST",
"request_path" => "/my-watch",
"request_uri" => "/my-watch?",
"http_version" => "HTTP/1.1",
"http_authorization" => "Basic Z3Vlc3Q6Z3Vlc3Q=",
"http_accept_charset" => "UTF-8",
"http_cache_control" => "no-cache",
"http_pragma" => "no-cache",
"http_user_agent" => "Java/1.8.0_20",
"http_host" => "127.0.0.1:8080",
"http_accept" => "text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2",
"http_connection" => "keep-alive",
"content_length" => "13"
}
}
{
"message" => "6071",
"@version" => "1",
"@timestamp" => "2015-06-02T12:55:00.031Z",
"headers" => {
"content_type" => "application/x-www-form-urlencoded",
"request_method" => "POST",
"request_path" => "/my-watch",
"request_uri" => "/my-watch?",
"http_version" => "HTTP/1.1",
"http_authorization" => "Basic Z3Vlc3Q6Z3Vlc3Q=",
"http_accept_charset" => "UTF-8",
"http_cache_control" => "no-cache",
"http_pragma" => "no-cache",
"http_user_agent" => "Java/1.8.0_20",
"http_host" => "127.0.0.1:8080",
"http_accept" => "text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2",
"http_connection" => "keep-alive",
"content_length" => "13"
}
}
A more complex example
Now that we know how to trigger notification events from Watcher, we can leverage the plugin ecosystem in
Logstash to escalate notifications depending in a certain criteria. This following config will:
continuously update the number of 404 requests in statsd
if the count reaches 10000 then send a message to HipChat, or
if reaches 40000, notify PagerDuty.
input {
http { }
}
filter {
if [headers][request_path] == "/my-watch" {
mutate { convert => ["message", "integer" ] }
}
}
output {
if [headers][request_path] == "/my-watch" {
if [message] > 40000 { # way too many, notify pagerduty
pagerduty {
description => "%{host} - Apache: Very high number of 404"
details => {
"timestamp" => "%{@timestamp}"
"message" => "%{message}"
}
service_key => "apikeyforlogstashservice"
incident_key => "logstash/apacheservice"
}
} else if [message] > 10000 { # unusual amount, notify devs in hipchat
hipchat {
from => "logstash"
room_id => "dev"
token => "[api key]"
format => "Very high number of 404 requests: %{message}"
}
}
# always update count of 404 in statsd
statsd { gauge => [ "http.status.404", "%{message}" ] }
}
}