Which resources should one monitor on a Linux server running a web-server or database

Question

When running any kind of server under load there are several resources that one would like to monitor to make sure that the server is healthy. This is specifically true when testing the system under load.

Some examples for this would be CPU utilization, memory usage, and perhaps disk space. What other resource should I be monitoring, and what tools are available to do so?

Answer 1

As many as you can afford to, and can then graph/understand/look at the results. Monitoring resources is useful for not only capacity planning, but anomaly detection, and anomaly detection significantly helps your ability to detect security events.

You have a decent start with your basic graphs. I'd want to also monitor the number of threads, number of connections, network I/O, disk I/O, page faults (arguably this is related to memory usage), context switches.

I really like munin for graphing things related to hosts.

Answer 2

I use Zabbix extensively in production, which comes with a stack of useful defaults. Some examples of the sorts of things we've configured it to monitor:

• Network usage
• CPU usage (% user,system,nice times)
• Load averages (1m, 5m, 15m)
• RAM usage (real, swap, shm)
• Disc throughput
• Active connections (by port number)
• Number of processes (by process type)
• Ping time from remote location
• Time to SSL certificate expiry
• MySQL internals (query cache usage, num temporary tables in RAM and on disc, etc)

Anything you can monitor with Zabbix, you can also attach triggers to - so it can restart failed services; or page you to alert about problems.

Collect the data now, before performance becomes an issue. When it does, you'll be glad of the historical baselines, and the fact you'll be able to show what date and time problems started happening for when you need to hunt down and punish exactly which developer made bad changes :)