Warm tip: This article is reproduced from serverfault.com, please click

Best way of monitoring multiple log files

发布于 2020-05-16 20:53:13

I need to know what is the best tools to monitor multiple log files suitable to my needs.

What I need :

  • Send and monitor multiple log files at once
  • Support real-time viewing
  • Better graphical User Interface whenever possible
  • Capable of searching or filtering the logs
  • Setup with minimum effort whenever possible

I have gathered some tools, they are :

  1. multitail, a simple multiple log files viewer, but I prefer a better graphical UI
  2. lnav, it's like multitail, but I don't know what's the pros and cons between the two
  3. FrontTail, I think this is much better than the previous two
  4. GrayLog, I have used this once, it has great searching or filtering features, but it was setup by someone, not sure if the setup would be complex or not
  5. LogStash, I never use it, but many people say it's great, but is it easy to setup?

the source of logs coming from these :

  • gridpane.com log
  • nginx access log
  • nginx error log
  • PHP error log
  • MySQL query log
  • MySQL error log
Questioner
Budianto IP
Viewed
0
Budianto IP 2020-05-20 01:12:54

I finally found the one that suits my needs.

I'm sharing this in case anyone who wants to use the same solution.

Thanks to sourav19, I followed your advice, even though it took me 8-10 hours to install and configure everything, but it's really what I want.

I had to buy a Digital Ocean droplet, cost me $20 to get a 4 GB of RAM, but I think it's much cheaper than buying the other log monitoring applications which are way too expensive.

Before installing docker, we have to enable Virtual Private Cloud (VPC), we will use the provided IP Address for our docker containers, so they can communicate between each other, by following this article.

I used a dockerized ELK, link is here

All we need to do is to clone the dockerized ELK to our server, and then go inside the cloned folder, and build the Dockerfile

docker run -p 5601:5601 -p 9200:9200  -p 5044:5044 \
-v /var/log:/var/lib/elasticsearch --name elk sebp/elk

Then, open kibana, in the website, HTTP://your_site:5601

after that, install the Filebeat into the other server which having the log files you want to monitor, this Filebeat will send the logs to Kibana, by following this instructions, and then configure it here.

if everything is okay, we will see the logs in the Kibana.