For example there are only a dozen of container names and not hundreds.Ĭreating dashboard from visualizations in Kibana Pie Charts are pretty good for counting the amount of data when your data set has limited variability. For instance when texts on X-Axis are too long I prefer the Data Table.Ĭreating a Pie Chart visualization to show TOP 10 HTTP Response codes
The Data Table visualization usually more readable than Vertical Bars.
The free GeoIP database is enough to narrow down to counties or bigger areas.Ĭreating a Data Table visualization to show TOP 10 HTTP User Agents You can use it to see where your HTTP clients are coming from. I used this type of visualization for Fail2ban logs before. It can also present you the amount of logs each container produces.Ĭreating a Coordinate Map visualization for GeoIP data of HTTP clients You can use this type of visualization for HTTP Response codes. This chart is useful to show the size of traffic changes in time. Creating a Vertical Bar visualization for NGINX average bytes I created short videos about how you can make use of them. Visualizing NGINX access logs in Kibana can be done by using visualizations like Data Table, Vertical Bar, Pie Chart and Coordinate Map. Creating NGINX and Docker visualizations in Kibana This is already useful but we are going to create interesting visualizations in the next chapter. Scroll down a bit and you shall see the added GeoIP and http (NGINX) metadata as well. The following screenshot have been updated to Elasticsearch 7.2 and show all fields complying to ECS. Try to browse the log messages in Kibana→Discover menu. If you succeeded to follow the steps, you will have an index pattern called nginx-*. Although you may want to reconsider this as the official Filebeat Dashboards for NGINX would expect filebeat-* instead. You may want to follow that with index patterns. Please note that index template will create indexes matching nginx-*. After a couple of logs reached Elastic, make sure you create index patterns for the rest of the data, not described in the explicit mapping before.
Or go to Dev Tools→Console in Kibana and paste – again because of its length – the API command you can download here. To create the mapping either follow the steps I described in setting data type mappings for Fail2ban. For visualizing NGINX access logs in Kibana we need explicit data type mapping for some records. Most of the data which are put into the indexes are mapped as keyword. We need to create an index and an explicit data type mapping for some attributes before Elasticsearch will accept logs.Ĭreating index and data type mapping in Elasticsearch
Just do not forget to install the GeoIP updater script I wrote for Visualizing Fail2ban logs in Kibana to periodically update the GeoIP databases. The configuration I provided earlier will do GeoIP resolving for you. Again, Logstash or even an Elasticsearch ingest node would do this for you, but I use syslog-ng instead of Logstash or ingest pipelines. The other big thing this config gives you is to enrich the logs with GeoIP metadata of HTTP clients. The only thing which needs to be changed in the configuration is a global variable called elastic_host which should point to your Elasticsearch Ingest node. Use it with the either the guide I wrote in logging Docker to Elasticsearch or add it to any standalone web servers. Because of the length of the configuration I uploaded it to GitHub and not added here. The configuration sample I provide gives you ECS support, so you can even use the official Dashboards. Although it provides Logstash compatible field names, but they are not updated to comply with Elastic Common Schema (ECS) which is the new standard since Elasticsearch version 7.0.
Using it will provide you more insights about HTTP usage. It works on logs complying either to Common Log Format (Apache default) or to Combined Log Format (NGINX default). There is a specific parser in syslog-ng called apache-accesslog-parser which can further parse access logs of NGINX or Apache web servers. Parse NGINX/Apache access logs to provide insights about HTTP usage Creating dashboard from visualizations in Kibana.Creating a Pie Chart visualization to show TOP 10 HTTP Response codes.Creating a Data Table visualization to show TOP 10 HTTP User Agents.Creating a Coordinate Map visualization for GeoIP data of HTTP clients.Creating a Vertical Bar visualization for NGINX average bytes.Creating NGINX and Docker visualizations in Kibana.Creating index and data type mapping in Elasticsearch.Parse NGINX/Apache access logs to provide insights about HTTP usage.