Differences

This shows you the differences between two versions of the page.

--- pages:howtos:diagnose:linux-diagnosis-and-monitoring-tools [2022/02/07 13:20] – mischerh
+++ pages:howtos:diagnose:linux-diagnosis-and-monitoring-tools [2023/06/15 16:42] (current) – [mytop] mischerh
@@ Line 1: / Line 1: @@
-{{tag>howto linux windows aix diagnosis monitoring tools top htop atop netatop apache apachetop modstatus proftp ftptop mytop mysql mariadb powertop iotop ntopng iftop jnettop bandwidthd etherape ethtool nethogs iptraf ngrep mrtg bmon traceroute traceroute6 tracert6 iptstate darkstat vnstat netstat ss nmap mtr tcpdump justniffer serverdensity opennms sysusage pcp ksysguard munin nagios icinga zenoss cacti zabbix nmon conky glances squid sarg saidar libstatgrab rrdtool procexp linuxprocessexplorer df dstat netsnmp incron}}
+{{tag>howto linux windows aix diagnosis monitoring tools top htop atop netatop apache apachetop modstatus proftp ftptop mytop mysql mariadb powertop iotop ntopng iftop jnettop bandwidthd etherape ethtool nethogs iptraf ngrep mrtg bmon traceroute traceroute6 tracert6 iptstate darkstat vnstat netstat ss nmap mtr tcpdump justniffer serverdensity opennms sysusage pcp ksysguard munin nagios icinga zenoss cacti zabbix nmon conky glances squid sarg saidar libstatgrab rrdtool procexp linuxprocessexplorer df dstat netsnmp incron monitorix vmstat mpstat pmap uptime ps sar sysstat iostat collectl free goaccess logwatch swatchdog multitail}}
 FIXME
@@ Line 31: / Line 31: @@
   * https://github.com/JeremyJones/Apachetop
   * https://github.com/JeremyJones/Apachetop/wiki
-apachetop is a console-based (non-gui) tool for monitoring the threads and overall performance of a set of Apache web servers. It runs on Unix systems which have Perl, LWP, and Term::ReadKey installed. It is based largely on the excellent mytop tool written by Jeremy Zawodny.
+apachetop is a console-based (non-gui) tool for monitoring the threads and overall performance of a set of Apache web servers. It runs on Unix systems which have Perl, LWP, and <nowiki>Term::ReadKey</nowiki> installed. It is based largely on the excellent mytop tool written by Jeremy Zawodny.
 ===== Apache mod_status =====
@@ Line 58: / Line 58: @@
   * https://jeremy.zawodny.com/mysql/mytop/
   * https://jeremy.zawodny.com/mysql/mytop/mytop.html
-mytop is a console-based (non-gui) tool for monitoring the threads and overall performance of a MySQL 3.22.x, 3.23.x, and 4.x server. It runs on most Unix systems (including Mac OS X) which have Perl, DBI, and Term::ReadKey installed. And with Term::ANSIColor installed you even get color. If you install Time::HiRes, you'll get good real-time queries/second stats. As of version 0.7, it even runs on Windows (somewhat).
+mytop is a console-based (non-gui) tool for monitoring the threads and overall performance of a MySQL 3.22.x, 3.23.x, and 4.x server. It runs on most Unix systems (including Mac OS X) which have Perl, DBI, and <nowiki>Term::ReadKey</nowiki> installed. And with <nowiki>Term::ANSIColor</nowiki> installed you even get color. If you install Time::HiRes, you'll get good real-time queries/second stats. As of version 0.7, it even runs on Windows (somewhat).
 ===== powertop =====
@@ Line 367: / Line 367: @@
 ===== incron =====
+  * https://github.com/ar-/incron
+  * http://inotify.aiken.cz/?section=incron&page=about&lang=en
+This program is the "inotify cron" system. It consist of a daemon and a table manipulator. You can use it a similar way as the regular cron. The difference is that the inotify cron handles filesystem events rather than time periods.
+This project was kicked off by Lukas Jelinek in 2006 and then unfortunatally abandoned in 2012. Upstream development and bug-tracking/fixing continued in 2014 on GitHub: https://github.com/ar-/incron .
-Incron allows you to monitor a directory tree and then take action on those changes. If you wanted to copy files to directory ‘b’ once new files appeared in directory ‘a’ that’s exactly what incron does.
+ inotify is an inode-based filesystem notification technology. It provides possibility to simply monitor various events on files in filesystems. It is a very much powerful replacement of (obsolete) dnotify. inotify brings a comfortable way how to manage files used in your applications.
-* http://inotify.aiken.cz/?section=incron&page=about&lang=en
+inotify can be used for such tasks:
+  * detecting changes in files and directories (e.g. configuration files, mail directories)
+  * guarding critical files and their eventual automatic recovery
+  * file usage statistics and similar purposes
+  * automatic upload handling
+  * monitoring installations outside of packaging systems
+  * automatic on-change backup and/or versioning
+  * reflecting changes to search databases
-monitorix – System and Network Monitoring
+There is much more that can be done using inotify. Switch your fantasy on and incorporate inotify to your application the best way.
-Monitorix is a free lightweight utility that is designed to run and monitor system and network resources as many as possible in Linux/Unix servers. It has a built in HTTP web server that regularly collects system and network information and display them in graphs. It Monitors system load average and usage, memory allocation, disk driver health, system services, network ports, mail statistics (Sendmail, Postfix, Dovecot, etc), MySQL statistics and many more. It designed to monitor overall system performance and helps in detecting failures, bottlenecks, abnormal activities etc.
+===== monitorix =====
+  * https://github.com/mikaku/Monitorix
+  * https://www.monitorix.org/
+  * https://www.monitorix.org/documentation.html
+  * https://wiki.archlinux.org/title/monitorix
+Monitorix is a free, open source, lightweight system monitoring tool designed to monitor as many services and system resources as possible. It has been created to be used under production Linux/UNIX servers, but due to its simplicity and small size can be used on embedded devices as well.
-~\ownCloud\zim\Bilder\DEF\artikel-linux-monitoring-tools\Monitorix-Monitoring-592x450.png
+It consists mainly of two programs: a collector, called monitorix, which is a Perl daemon that is started automatically like any other system service, and a CGI script called monitorix.cgi. Monitorix includes its own HTTP server built in (which is listening by default on port 8080/TCP) to see the statistics graphs, so you aren't forced to install a third-party web server to use it. Just point your browser at http://localhost:8080/monitorix.
-Monitorix is lightweight system monitoring tool. It helps you monitor a single machine and gives you a wealth of metrics. It also has a built-in HTTP server to view graphs and a reporting mechanism of all metrics.
+All of its development was initially created for monitoring Red Hat, Fedora and CentOS Linux systems, so this project was made keeping in mind these type of distributions. Today it runs on different GNU/Linux distributions and even in other UNIX systems like FreeBSD, OpenBSD and NetBSD.
-* http://www.monitorix.org/
-* Read More : Monitorix a System and Network Monitoring Tool for Linux
+It is currently in active development adding new features, new graphs and correcting bugs in the attempt to offer a great tool for daily systems administration.
-vmstat
+===== vmstat =====
+  * https://linux.die.net/man/8/vmstat
+  * https://en.wikipedia.org/wiki/Vmstat
+vmstat (virtual memory statistics) is a computer system monitoring tool that collects and displays summary information about operating system memory, processes, interrupts, paging and block I/O. Users of vmstat can specify a sampling interval which permits observing system activity in near-real time.
-Linux VmStat command used to display statistics of virtual memory, kernerl threads, disks, system processes, I/O blocks, interrupts, CPU activity and much more. By default vmstat command is not available under Linux systems you need to install a package called sysstat that includes a vmstat program. The common usage of command format is.
+The vmstat tool is available on most Unix and Unix-like operating systems, such as FreeBSD, Linux or Solaris.
-procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
+===== Web VMStat =====
- r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
+  * https://github.com/joewalnes/web-vmstats
-  0      0 444512  17068 281888    0    0     1     2   27   24  1  0 99  0  0
+Display live Linux system stats (memory, CPU, IO, etc) in a pretty web-page, with charts and everything.
+===== uptime =====
+  * https://man7.org/linux/man-pages/man1/uptime.1.html
+  * https://en.wikipedia.org/wiki/Uptime#Using_uptime
+uptime gives a one line display of the following information. The current time, how long the system has been running, how many users are currently logged on, and the system load averages for the past 1, 5, and 15 minutes.
-vmstat or virtual memory statistics is a small built-in tool that monitors and displays a summary about the memory in the machine.
+===== mpstat =====
+  * https://linux.die.net/man/1/mpstat
+  * http://sebastien.godard.pagesperso-orange.fr/
+mpstat - Report processors related statistics. The mpstat command writes to standard output activities for each available processor, processor 0 being the first one. Global average activities among all processors are also reported. The mpstat command can be used both on SMP and UP machines, but in the latter, only global average activities will be printed. If no activity has been selected, then the default report is the CPU utilization report.
-For more Vmstat examples read : 6 Vmstat Command Examples in Linux
+The interval parameter specifies the amount of time in seconds between each report. A value of 0 (or no parameters at all) indicates that processors statistics are to be reported for the time since system startup (boot). The count parameter can be specified in conjunction with the interval parameter if this one is not set to zero. The value of count determines the number of reports generated at interval seconds apart. If the interval parameter is specified without the count parameter, the mpstat command generates reports continuously.
+===== pmap =====
+  * https://linux.die.net/man/1/pmap
+pmap - report memory map of a process. The pmap command reports the memory map of a process or processes.
-Web VMStat – System Statistics Monitoring
+===== ps =====
+  * https://man7.org/linux/man-pages/man1/ps.1.html
+  * https://en.wikipedia.org/wiki/Ps_(Unix)
+ps displays information about a selection of the active processes. If you want a repetitive update of the selection and the displayed information, use top instead. In most Unix and Unix-like operating systems, the ps program (short for "process status") displays the currently-running processes. A related Unix utility named top provides a real-time view of the running processes.
-Web VMStat is a very simple web application programmer, that provides a real time system information usage, from CPU to RAM, Swap and input/output information in html format.
+===== sar =====
+  * https://linux.die.net/man/1/sar
+  * http://sebastien.godard.pagesperso-orange.fr/
+  * http://sebastien.godard.pagesperso-orange.fr/man_sar.html
+sar - Collect, report, or save system activity information. The sar command writes to standard output the contents of selected cumulative activity counters in the operating system. The accounting system, based on the values in the count and interval parameters, writes information the specified number of times spaced at the specified intervals in seconds. If the interval parameter is set to zero, the sar command displays the average statistics for the time since the system was started. If the interval parameter is specified without the count parameter, then reports are generated continuously. The collected data can also be saved in the file specified by the -o filename flag, in addition to being displayed onto the screen. If filename is omitted, sar uses the standard system activity daily data file, the /var/log/sa/sadd file, where the dd parameter indicates the current day. By default all the data available from the kernel are saved in the data file.
-~\ownCloud\zim\Bilder\DEF\artikel-linux-monitoring-tools\Web-VMStat.png
+===== Sysstat =====
+  * http://sebastien.godard.pagesperso-orange.fr/
+The sysstat utilities are a collection of performance monitoring tools for Linux. These include sar, sadf, mpstat, iostat, tapestat, pidstat, cifsiostat  and sa tools. Go to the Features page to display a list of sysstat's features, or see the Documentation page to learn some more about them.
+Sysstat is Open Source / Free Software, and is freely available under the GNU General Public License, version 2.
-Read More: Web VMStat: A Real Time System Statistics Tool for Linux
+Sysstat's main features:
+  * Includes four groups of monitoring tools (sar / sadc / sadf, iostat / tapestat / cifsiostat, mpstat, pidstat) for global system performance analysis.
+  * Can monitor a huge number of different metrics:
+    * Input / Output and transfer rate statistics (global, per device, per partition, per network filesystem and per Linux task / PID).
+    * CPU statistics (global, per CPU, per NUMA nodes and per Linux task / PID), including support for virtualization architectures.
+    * Memory, hugepages and swap space utilization statistics.
+    * Virtual memory, paging and fault statistics.
+    * Per-task (per-PID) memory and page fault statistics.
+    * Global CPU and page fault statistics for tasks and all their children.
+    * Process creation activity.
+    * Interrupt statistics (global, per CPU and per interrupt, including potential APIC interrupt sources, hardware and software interrupts).
+    * Extensive network statistics: network interface activity (number of packets and kB received and transmitted per second, etc.) including failures from network devices; network traffic statistics for IP, TCP, ICMP and UDP protocols based on SNMPv2 standards; support for IPv6-related protocols.
+    * Fibre Channel traffic statistics.
+    * Software-based network processing (softnet) statistics.
+    * NFS server and client activity.
+    * Sockets statistics.
+    * Run queue and system load statistics.
+    * Kernel internal tables utilization statistics.
+    * System and per Linux task switching activity.
+    * Swapping statistics.
+    * TTY devices activity.
+    * Power management statistics (instantaneous and average CPU clock frequency, fans speed, devices temperature, voltage inputs)
+    * USB devices plugged into the system.
+    * Filesystems utilization (inodes and blocks).
+    * Tape drives statistics.
+    * Pressure-Stall Information statistics.
+  * Can generate graphs (SVG format - Scalable Vector Graphics) that can be displayed in your favorite web browser!
+  * Most system statistics can be saved in a file for future inspection.
+  * Allows to configure the length of data history to keep.
+  * On the fly detection of new devices (disks, network interfaces, etc.) that are created or registered dynamically.
+  * Support for UP and SMP machines, including machines with hyperthreaded or multi-core processors.
+  * Support for hotplug CPUs (it detects automagically processors that are disabled or enabled on the fly) and tickless CPUs.
+  * Works on many different architectures, whether 32- or 64-bit.
+  * Needs very little CPU time to run (written in C).
+  * System statistics collected by sar/sadc can be exported in various different formats (CSV, XML, JSON, SVG, etc.). DTD and XML Schema documents are included in sysstat package. JSON output format is also available for mpstat and iostat commands.
+  * sar data can also be exported by sadf to PCP (Performance Co-Pilot) archive.
+  * Smart color output for easier statistics reading.
+  * Internationalization support (sysstat has been translated into numerous different languages). Sysstat is now part of the Translation Project.
+  * Sysstat commands can automatically select the unit used to display sizes for easier reading (see option --human).
+  * Many programs available on the internet to use sysstat's data to make graphs (one of them, isag, is included in sysstat).
+  * iostat has support for devices managed by drivers in userspace like spdk.
-uptime
+===== iostat =====
+  * http://sebastien.godard.pagesperso-orange.fr/
-This small command that quickly gives you information about how long the machine has been running, how many users currently are logged on and the system load average for the past 1, 5 and 15 minutes.
+  * http://sebastien.godard.pagesperso-orange.fr/man_iostat.html
-mpstat
-~\ownCloud\zim\Bilder\DEF\artikel-linux-monitoring-tools\mpstat.jpg
-mpstat is a built-in tool that monitors cpu usage. The most common command is using mpstat -P ALL which gives you the usage of all the cores. You can also get an interval update of the CPU usage.
-pmap
-~\ownCloud\zim\Bilder\DEF\artikel-linux-monitoring-tools\pmap.jpg
-pmap is a built-in tool that reports the memory map of a process. You can use this command to find out causes of memory bottlenecks.
-ps
-~\ownCloud\zim\Bilder\DEF\artikel-linux-monitoring-tools\ps.jpg
-The ps command will give you an overview of all the current processes. You can easily select all processes using the command ps -A
-sar
-~\ownCloud\zim\Bilder\DEF\artikel-linux-monitoring-tools\sar.jpg
-sar is a part of the sysstat package and helps you to collect, report and save different system metrics. With different commands it will give you CPU, memory and I/O usage among other things.
-* http://sebastien.godard.pagesperso-orange.fr/
-Sysstat – All-in-One System Performance Monitoring
-Another monitoring tool for your Linux system. Sysstat is not a real command in fact, it’s just the name of the project, Sysstat in fact is a package that includes many performance monitoring tools like iostat, sadf, pidstat beside many other tools which shows you many statistics about your Linux OS.
-~\ownCloud\zim\Bilder\DEF\artikel-linux-monitoring-tools\Sysstat-620x381.png
-Features of Sysstat
-* Available in many Linux distributions repositories by default.
-* Ability to create statistics about RAM, CPU, SWAP usage. Beside the ability to monitor Linux kernel activity, NFS server, Sockets, TTY and filesystems.
-* Ability to monitor input & output statistics for devices, tasks.. etc.
-* Ability to output reports about network interfaces and devices, with support for IPv6.
-* Sysstat can show you the power statistics (usage, devices, the fans speed.. etc) as well.
-* Many other features..
-Read More: Install Sysstat in Linux and 20 Useful Commands of Sysstat
-collectl- All-in-One Performance Monitoring Tool
-Collectl is a yet another powerful and feature rich command line based utility, that can be used to gather information about Linux system resources such as CPU usage, memory, network, inodes, processes, nfs, tcp, sockets and much more.
-~\ownCloud\zim\Bilder\DEF\artikel-linux-monitoring-tools\collectl.jpg
-~\ownCloud\zim\Bilder\DEF\artikel-linux-monitoring-tools\Collectl-620x410.jpg
-Similar to sar collectl collects performance metrics for your machine. By default it shows cpu, network and disk stats but it collects a lot more. The difference to sar is collectl is able to deal with times below 1 second, it can be fed into a plotting tool directly and collectl monitors processes more extensively.
-* http://collectl.sourceforge.net/
-* Read More: Install Collectl (All-in-One Performance Monitoring) Tool in Linux
-iostat
-~\ownCloud\zim\Bilder\DEF\artikel-linux-monitoring-tools\iostat.jpg
 iostat is also part of the sysstat package. This command is used for monitoring system input/output. The reports themselves can be used to change system configurations to better balance input/output load between hard drives in your machine.
-* http://sebastien.godard.pagesperso-orange.fr/
-free
-~\ownCloud\zim\Bilder\DEF\artikel-linux-monitoring-tools\free.jpg
-This is a built-in command that displays the total amount of free and used physical memory on your machine. It also displays the buffers used by the kernel at that given moment.
-/Proc file system
-~\ownCloud\zim\Bilder\DEF\artikel-linux-monitoring-tools\procfile.jpg
-The proc file system gives you a peek into kernel statistics. From these statistics you can get detailed information about the different hardware devices on your machine. Take a look at the full list of the proc file statistics
-GKrellM
-GKrellm is a gui application that monitor the status of your hardware such CPU, main memory, hard disks, network interfaces and many other things. It can also monitor and launch a mail reader of your choice.
-* http://members.dslextreme.com/users/billw/gkrellm/gkrellm.html
-Gnome system monitor
-~\ownCloud\zim\Bilder\DEF\artikel-linux-monitoring-tools\gnome-system-monitor.jpg
-Gnome system monitor is a basic system monitoring tool that has features looking at process dependencies from a tree view, kill or renice processes and graphs of all server metrics.
-* http://freecode.com/projects/gnome-system-monitor
-GoAccess
-~\ownCloud\zim\Bilder\DEF\artikel-linux-monitoring-tools\goaccess.jpg
-GoAccess is a real-time web log analyzer which analyzes the access log from either apache, nginx or amazon cloudfront. It’s also possible to output the data into HTML, JSON or CSV. It will give you general statistics, top visitors, 404s, geolocation and many other things.
-* http://goaccess.io/
-Logwatch
+===== collectl =====
+  * https://linux.die.net/man/1/collectl
+  * http://collectl.sourceforge.net/
+  * http://collectl.sourceforge.net/Documentation.html
+ There are a number of times in which you find yourself needing performance data. These can include benchmarking, monitoring a system's general heath or trying to determine what your system was doing at some time in the past. Sometimes you just want to know what the system is doing right now. Depending on what you're doing, you often end up using different tools, each designed to for that specific situation.
-Logwatch is a log analysis system. It parses through your system’s logs and creates a report analyzing the areas that you specify. It can give you daily reports with short digests of the activities taking place on your machine.
+Unlike most monitoring tools that either focus on a small set of statistics, format their output in only one way, run either interatively or as a daemon but not both, collectl tries to do it all. You can choose to monitor any of a broad set of subsystems which currently include buddyinfo, cpu, disk, inodes, infiniband, lustre, memory, network, nfs, processes, quadrics, slabs, sockets and tcp.
-* http://sourceforge.net/projects/logwatch/
+===== free =====
+  * https://man7.org/linux/man-pages/man1/free.1.html
+free displays the total amount of free and used physical and swap memory in the system, as well as the buffers and caches used by the kernel. The information is gathered by parsing /proc/meminfo.
-Swatch
+===== /proc file system =====
+  * https://www.kernel.org/doc/html/latest/filesystems/proc.html
+  * https://tldp.org/LDP/Linux-Filesystem-Hierarchy/html/proc.html
+/proc is very special in that it is also a virtual filesystem. It's sometimes referred to as a process information pseudo-file system. It doesn't contain 'real' files but runtime system information (e.g. system memory, devices mounted, hardware configuration, etc). For this reason it can be regarded as a control and information centre for the kernel. In fact, quite a lot of system utilities are simply calls to files in this directory. For example, 'lsmod' is the same as 'cat /proc/modules' while 'lspci' is a synonym for 'cat /proc/pci'. By altering files located in this directory you can even read/change kernel parameters (sysctl) while the system is running.
-~\ownCloud\zim\Bilder\DEF\artikel-linux-monitoring-tools\swatch.jpg
+===== GKrellM =====
+  * http://gkrellm.srcbox.net/
+  * https://git.srcbox.net/gkrellm/gkrellm
+GKrellM is a single process stack of system monitors which supports applying themes to match its appearance to your window manager, Gtk, or any other theme.
-Much like Logwatch Swatch also monitors your logs, but instead of giving reports it watches for regular expression and notifies you via mail or the console when there is a match. It could be used for intruder detection for example.
+===== Gnome system monitor =====
-* http://sourceforge.net/projects/swatch/
+  * https://wiki.gnome.org/Apps/SystemMonitor
+  * https://help.gnome.org/users/gnome-system-monitor/stable/
+System Monitor is a tool to manage running processes and monitor system resources.
+===== GoAccess =====
+  * https://github.com/allinurl/goaccess
+  * https://goaccess.io/
+  * https://goaccess.io/man
+GoAccess is an open-source web analytics application for Unix-like operating systems. It has both a text-based and a web application user interface. It can provide real-time analytics by continuously monitoring web server logs. GoAccess is an open source real-time web log analyzer and interactive viewer that runs in a terminal in *nix systems or through your browser.
-MultiTail
+It provides fast and valuable HTTP statistics for system administrators that require a visual server report on the fly.
-~\ownCloud\zim\Bilder\DEF\artikel-linux-monitoring-tools\multitail.jpg
+===== Logwatch =====
+  * https://linux.die.net/man/8/logwatch
+  * https://sourceforge.net/projects/logwatch/
+Logwatch is a customizable, pluggable log-monitoring system. It will go through your logs for a given period of time and make a report in the areas that you wish with the detail that you wish. Logwatch is being used for Linux and many types of UNIX.
-MultiTail helps you monitor logfiles in multiple windows. You can merge two or more of these logfiles into one. It will also use colors to display the logfiles for easier reading with the help of regular expressions.
+===== Swatchdog =====
-* http://www.vanheusden.com/multitail/
+  * http://sourceforge.net/projects/swatch/
+  * http://swatch.sourceforge.net/
+Swatch: the active log file monitoring tool. Swatch started out as the "simple watchdog" for activly monitoring log files produced by UNIX's syslog facility. It has since been evolving into a utility that can monitor just about any type of log.
+===== MultiTail =====
+  * https://github.com/halturin/multitail
+  * http://www.vanheusden.com/multitail/
+  * https://www.vanheusden.com/multitail/examples.html
+  * https://linux.die.net/man/1/multitail
+MultiTail allows you to monitor logfiles and command output in multiple windows in a terminal, colorize, filter, and merge.
 acct or psacct – Monitor User Activity