Top 9 Log Analysis Tools - Making Data-Driven Decisions
We live in a time where data is recorded and logged for just about everything. Whether it is for web server uptime, JavaScript errors, database queries, CDN traffic, Nginx errors, etc. You name it, and you can probably log it. In a previous post, we discussed the importance of integrating a log management service into your DevOps workflow. Having centralized logs can be great, but you also need to be able to analyze them efficiently. So today we want to cover the top 9 log analysis tools which you can use to better parse your logs, run live tail searches, and query the specific log data you need.
Log analysis tools
As more and more companies move to the cloud, log analytics, log analysis, and log management tools and services are becoming more critical. DevOps engineers, system administrators, site reliability engineers, and web developers can all use logs to make better data-driven decisions.
There are a lot of different log analysis tools out there, below are 10 of the most popular ones. Each of them offers their own unique search features, live tail queries, etc. Some of the tools listed are more log management services, but they also offer more efficient ways to analyze your logs compared to looking at the raw data. These are in no particular order and include both free and paid tools.
Funnel all logs, metrics and machine events into a single hub. Get a clear overview of what is happening across your distributed environments, and spot the needle in the haystack in no time.
- Logmatic
1. GoAccess
GoAccess is designed to be a fast, terminal-based log analyzer. Its core idea is to quickly analyze and view web server statistics in real time without needing to use your browser. It is open source and available on GitHub with over 2,900 stars and 200+ forks.
GoAccess features
We love GoAccess because it is open source, and because of that, it is completely free to use. This is a tool for those who love using terminal and SSH and want quick access to their data. You can easily generate reports with this tool on the fly by generating them to real-time HTML, JSON or CSV. You can view a live demo here.
Other features include the ability to process logs incrementally, picking your log format, real-time stats, and predefined custom log format strings.
2. logz.io
logz.io offers you real-time, actionable insights into your log analytics data with hosted ELK as a service. ELK is a simple but robust log analysis platform that costs a fraction of the price. Some of their customers include Netflix, Facebook, LinkedIn, Cisco, and Microsoft. logz.io is probably one of the biggest competitors to Splunk, which is mentioned further below.
logz.io features
logz.io has both free and paid plans. Troubleshooting production issues is perhaps the most common use-case of log analytics. Their interface, which is powered by Kibana, lets you search through millions of records to investigate and pinpoint potential issues. You can filter results by server, application, or any custom parameter until you reach the source of the problem. If you are looking for a Splunk alternative you might want to give logz.io a try.
3. Graylog
Graylog is an open source log management platform which allows you to search, analyze, and alert you across all your log files. Some of their customers include BCBS, eBay, SAP, Cisco, LinkedIn, and Twilio. It is available on GitHub with over 2,000 stars and 300+ forks.
Graylog features
Because it is open source, Graylog is completely free to use. They also have an enterprise version where they charge you per license on a server. Some of its features include a REST API and Graylog's flexible processing engine which makes it easy to parse and enrich logs from any data source. You can search through terabytes of data instantly and even save search queries to be shared later with your colleagues. Their powerful drill-down analysis and charts make it easy to pinpoint issues in your logs.
4. Splunk
Splunk is a big name in the log and application management space. They have been around since 2003 are no newcomers when it comes to analyzing and monitoring data. They offer great solutions for larger enterprise customers.
Splunk features
Splunk has both free and paid plans. Their free plan, Spunk light, allows you up to log up to 500 MB data per day. There pricing for paid plans depends on the volume of data you are processing. Splunk offers a great way to collect, store, index, search, correlate, visualize, analyze and report on any machine-generated data to identify and resolve operational and security issues in a faster, repeatable, and more affordable way. They offer powerful drill-downs which allow you to easily go back in time using ad-hoc queries. Their dashboard and trend charts also provide a great way to spot and visualize possible trends.
5. Logmatic.io
Logmatic.io is a log analysis tool designed specifically to help improve software and business performance. The founders have more than 10 years experience in real-time and big data software. Their emphasis is on analyzing your "machine data."
Logmatic features
Logmatic.io has paid plans starting at $49 per month. Some of their features include the ability to create your own custom parsing rules which allows the software to automatically recognize patterns. This is supported in Apache, Nginx, syslogs, JSON events, etc. You have faceted and full-text granular searches down to the log level as well as real-time logs and real-time searches. You can use complex queries such as AND, OR, wildcards. etc. They provide you with all sorts of graphs to spot trends, everything from geo maps, flow charts, and pivot tables!
6. Logstash
Logstash is a free open source tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use. This tool goes hand in hand with both Elasticsearch and Kibana. Using these together can be a powerful combination for a log analysis tool. They are all three available on GitHub.
Logstash features
Because it is open source, Logstash is completely free to use. They also have a paid license though available for those needing additional features. You can use elasticsearch, kibana, and logstash together. Kibana allows you to more easily explore and visualize the log data you bring in with logstash, and elasticsearch gives you the powerful real-time search and analytics capabilities.
7. Sumo Logic
Sumo Logic focuses on machine learning for unified logs and metrics to uncover real-time insights into application needs and new customer opportunities. They were founded in 2010 and their cloud-native service analyzes more than 100 petabytes of data per day.
Sumo Logic features
Sumo Logic has both free and paid plans starting at $90 per month. They offer a unique feature that they call machine learning. This allows you to analyze petabytes of data and learn from this to uncover patterns more quickly. Their tool uses advanced analytics which helps make sense of large amounts of data using indexing and filtering. Their intuitive dashboard allows you to spot anomalies by setting up customer predefined metric filters.
8. Papertrail
Papertrail is more of a log management service, but they also offer some great features which make analyzing your logs fast! Some of their customers include GitHub, Instacart, Product Hunt, and DNSimple.
Papertrail features
Papertrail has both free and paid plans starting at $7 per month. Some of their features include an intuitive web-based log viewer and powerful command-line tools. They have a REST API and long-term archives with S3. If you are a developer, you will probably like what Papertrail is doing! You can run live tail searches, seek by time, content, elegant searches, save your searches, and even colorize your logs differently. They even have an OS X dashboard widget and integrations with your favorite tools such as Slack and Zendesk.
9. Fluentd
Fluentd is another open source data analysis tool that allows you to unify logs from multiple sources in order to better and more easily analyze them. A few of Fluentd's most notable users include Microsoft, Amazon AWS, and Atlassian.
Fluentd features
Fluentd allows you to decouple multiple data sources such as your access logs, app logs, system logs, etc and unify them into one logging layer. You can then filter, buffer, and route those logs to the to appropriate systems (e.g. Hadoop, Elasticsearch, AWS, etc). Fluentd users also have 300+ plugins at their disposable to connect to a multitude of data sources. It can do so all while keeping a small memory footprint of only 30-40 MB.
Summary
There are plenty of log analysis tools to help you better understand your log data and parse it in a more efficient manner. This can help streamline your DevOps workflow and save you time the next time a problem pops up. The last thing you want to do is spend hours digging through unorganized log data trying to find what you need.
Did we miss any really important log analysis tools? If so, feel free to comment below. We especially love new open source tools!