nanoscopic wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
{{tag>howto diagnose}}
====== Diagnose 101 ======

===== Memory =====
This will show you top 10 process that using the most memory:
<sxh bash; gutter: false>
ps aux --sort=-%mem | awk 'NR<=10{print $0}'
</sxh>
using top: when you opened top press m will short process based on memory usage.

But this will not solve your problem, in Linux everything is either file or process. so the files you opened will eating the memory too. so this will not help.

//lsof// will give you all opened files with the size of the file or the file offset in bytes.


===== Filesysteme =====
mit //df -h// nachschauen ob die NFS/SAMBA-remote-filesysteme eingehängt sind (wenn es lange dauert bis df Ausgabe produziert, dann stimmt an dieser Stelle etwas nicht)
<sxh bash; gutter: false>
df -h
</sxh>
  df: /orabase/ACFS/DG1/V1/OGG: Input/output error
  Filesystem                           Size  Used Avail Use% Mounted on
  devtmpfs                             125G  8.0K  125G   1% /dev
  tmpfs                                125G  1.6G  124G   2% /dev/shm
  tmpfs                                125G  6.9M  125G   1% /run
  tmpfs                                125G     0  125G   0% /sys/fs/cgroup
  /dev/sda3                            2.0G  1.1G  981M  53% /
  /dev/mapper/system-usr               8.0G  4.1G  4.0G  51% /usr
  tmpfs                                8.0G  120M  7.9G   2% /tmp
  /dev/mapper/system-nosql             100G  180M  100G   1% /nosql
  /dev/mapper/system-var                10G  1.8G  8.2G  18% /var
  /dev/mapper/system-opt               4.0G  809M  3.3G  20% /opt
  /dev/mapper/system-oemcc              10G   33M   10G   1% /opt/oemcc
  /dev/mapper/system-zel                20G  9.7G   11G  49% /zel
  /dev/mapper/system-home              8.0G  3.2G  4.9G  40% /home
  /dev/mapper/system-var--crash         32G   25G  7.3G  78% /var/crash
  /dev/mapper/system-var--log          8.0G  1.3G  6.8G  16% /var/log
  /dev/mapper/system-orasw              80G   52G   29G  65% /orasw
  /dev/mapper/system-var_log_sa        5.0G  407M  4.6G   8% /var/log/sa
  /dev/mapper/system-vl_audit          749M  516M  234M  69% /var/log/audit
  /dev/mapper/system-sudolog           2.0G   40M  2.0G   2% /var/log/sudolog
  /dev/mapper/system-orabase            80G   21G   60G  26% /orabase
  tmpfs                                 25G     0   25G   0% /run/user/2000
  /dev/mapper/filevg_fs9709            639G  345G  295G  54% /zed9709
  tmpfs                                 25G     0   25G   0% /run/user/1845


===== CPU =====

load average
<sxh bash; gutter: false>
sar -q
</sxh>

  Linux 5.15.5-1-default (dellirio)       12/09/21        _x86_64_        (12 CPU)
  
  02:41:45     LINUX RESTART      (12 CPU)
  
  02:50:11      runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
  03:00:11            1      1701      1.25      0.98      0.99         0
  03:10:11            3      1692      2.20      1.47      1.22         0
  03:20:11            4      1706      1.50      1.29      1.20         0
  03:30:00            1      1755      0.70      1.30      1.35         0
  03:40:11            1      1796      1.17      2.09      1.80         0
  03:50:11            0      1787      1.51      1.81      1.86         0
  04:00:11            3      1806      0.88      1.17      1.48         0
  04:10:11            6      1753      0.77      0.78      1.13         0
  04:20:11            3      1754      0.51      0.72      0.93         0
  04:30:11            1      1757      0.62      0.65      0.80         0
  04:40:11            0      1750      0.71      0.70      0.78         0
  04:50:11            1      1753      0.69      0.72      0.74         0
  05:00:11            1      1751      0.52      0.75      0.77         0
  05:10:11            0      1751      0.61      0.71      0.73         0
  05:20:11            0      1759      1.03      1.04      0.89         0
  05:30:11            1      1749      0.75      0.92      0.89         0
  05:40:11            0      1753      0.88      0.83      0.83         0
  05:50:11            2      1756      0.91      0.88      0.84         0
  06:00:11            3      1759      1.01      0.83      0.83         0
  06:10:11            0      1754      1.09      0.86      0.81         0
  06:20:11            1      1753      1.14      1.03      0.90         0
  06:30:11            0      1758      0.78      0.81      0.84         0
  06:40:11            0      1753      0.67      0.76      0.81         0
  06:50:11            0      1745      0.77      0.78      0.81         0
  07:00:11            1      1755      1.00      0.88      0.83         0
  07:10:11            3      1753      0.84      0.77      0.80         0
  07:20:11            0      1749      0.86      0.77      0.79         0
  07:30:11            1      1749      0.67      0.66      0.73         0
  07:40:11            0      1754      0.99      0.89      0.82         0
  07:50:11            1      1746      0.75      0.76      0.80         0
  08:00:11            1      1754      0.86      0.73      0.78         0
  08:10:11            2      1750      0.70      0.74      0.77         0
  08:20:11            0      1754      0.70      0.71      0.77         0
  08:30:11            0      1752      0.67      0.74      0.82         0
  08:40:11            1      1750      0.91      0.83      0.84         0
  08:50:11            1      1749      0.71      0.75      0.81         0
  09:00:11            3      1755      0.43      0.68      0.78         0
  09:10:11            0      1754      0.68      0.75      0.76         0
  09:20:11            2      1759      1.44      1.04      0.86         0
  09:30:11            0      1754      0.64      0.80      0.85         0
  09:40:11            0      1753      0.76      0.79      0.82         0
  09:50:11            0      1751      0.80      0.82      0.81         0
  10:00:11            2      1754      0.79      0.78      0.80         0
  10:10:11            1      1756      0.82      0.90      0.86         0
  10:20:11            0      1754      1.10      0.88      0.85         0
  10:30:11            1      1751      0.60      0.70      0.76         0
  10:40:11            0      1754      0.83      0.83      0.81         0
  10:50:11            1      1753      1.04      0.91      0.86         0
  11:00:11            1      1779      1.05      0.92      0.88         0
  11:10:11            0      1763      0.76      0.76      0.81         0
  11:20:11            1      1757      0.56      0.75      0.80         0
  11:30:00            0      1762      1.00      0.84      0.80         0
  11:40:01            1      1758      0.77      0.69      0.73         0
  11:50:02            0      1753      1.11      0.96      0.82         0
  12:00:03            2      1760      0.82      0.85      0.83         0
  12:10:03            0      1759      0.80      0.78      0.82         0
  12:20:04            2      1763      1.12      0.86      0.83         0
  
  12:20:04      runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
  12:30:00            3      1771      0.55      0.66      0.77         0
  12:40:01            3      1767      0.54      0.71      0.77         0
  12:50:02            7      1766      0.74      0.72      0.77         0
  13:00:03            6      1765      0.69      0.88      0.87         0
  13:10:05            5      1760      0.77      0.80      0.82         0
  13:20:05            4      1756      0.77      0.86      0.84         0
  13:30:06            5      1754      0.59      0.70      0.80         0
  13:40:07            0      1756      0.73      0.86      0.85         0
  Average:            1      1755      0.86      0.87      0.88         0


===== user/system/idle/wait =====

<sxh bash; gutter: false>
sar -u
</sxh>

<code>
Linux 5.15.5-1-default (dellirio)       12/09/21        _x86_64_        (12 CPU)

02:41:45     LINUX RESTART      (12 CPU)

02:50:11        CPU     %user     %nice   %system   %iowait    %steal     %idle
03:00:11        all      8.33      0.00      0.93      0.02      0.00     90.72
03:10:11        all      9.00      0.00      1.13      0.02      0.00     89.85
03:20:11        all     10.42      0.00      1.42      0.01      0.00     88.14
03:30:00        all     12.78      0.00      1.00      0.02      0.00     86.20
03:40:11        all     19.48      0.00      1.01      0.02      0.00     79.49
03:50:11        all     17.14      0.01      1.15      0.01      0.00     81.69
04:00:11        all      9.82      0.00      1.25      0.01      0.00     88.91
04:10:11        all      8.07      0.00      0.89      0.02      0.00     91.02
04:20:11        all      6.65      0.00      0.50      0.01      0.00     92.84
04:30:11        all      6.45      0.00      0.48      0.01      0.00     93.06
04:40:11        all      6.53      0.00      0.47      0.01      0.00     92.98
04:50:11        all      6.44      0.00      0.46      0.01      0.00     93.08
05:00:11        all      6.55      0.00      0.49      0.01      0.00     92.96
05:10:11        all      6.54      0.00      0.47      0.01      0.00     92.98
05:20:11        all      6.54      0.00      0.50      0.01      0.00     92.95
05:30:11        all      6.36      0.00      0.46      0.01      0.00     93.17
05:40:11        all      6.62      0.00      0.49      0.01      0.00     92.88
05:50:11        all      6.54      0.00      0.48      0.01      0.00     92.97
06:00:11        all      6.47      0.00      0.47      0.01      0.00     93.05
06:10:11        all      6.64      0.00      0.51      0.01      0.00     92.83
06:20:11        all      6.55      0.00      0.49      0.01      0.00     92.95
06:30:11        all      6.49      0.00      0.48      0.01      0.00     93.02
06:40:11        all      6.61      0.00      0.48      0.01      0.00     92.90
06:50:11        all      6.52      0.00      0.48      0.01      0.00     92.99
07:00:11        all      6.46      0.00      0.48      0.01      0.00     93.04
07:10:11        all      6.66      0.00      0.50      0.01      0.00     92.83
07:20:11        all      6.48      0.00      0.49      0.01      0.00     93.02
07:30:11        all      6.43      0.00      0.47      0.01      0.00     93.08
07:40:11        all      6.62      0.00      0.49      0.01      0.00     92.87
07:50:11        all      6.50      0.00      0.47      0.01      0.00     93.02
08:00:11        all      6.51      0.00      0.48      0.01      0.00     93.00
08:10:11        all      6.46      0.00      0.48      0.01      0.00     93.04
08:20:11        all      6.50      0.00      0.47      0.01      0.00     93.01
08:30:11        all      6.44      0.00      0.47      0.01      0.00     93.08
08:40:11        all      6.58      0.00      0.48      0.01      0.00     92.92
08:50:11        all      6.44      0.00      0.48      0.01      0.00     93.07
09:00:11        all      6.51      0.00      0.47      0.01      0.00     93.01
09:10:11        all      6.60      0.00      0.49      0.02      0.00     92.89
09:20:11        all      6.44      0.00      0.48      0.01      0.00     93.07
09:30:11        all      6.53      0.00      0.48      0.01      0.00     92.98
09:40:11        all      6.61      0.00      0.50      0.01      0.00     92.88
09:50:11        all      6.40      0.00      0.47      0.01      0.00     93.11
10:00:11        all      6.49      0.00      0.46      0.01      0.00     93.04
10:10:11        all      6.53      0.00      0.49      0.01      0.00     92.97
10:20:11        all      6.55      0.00      0.48      0.01      0.00     92.95
10:30:11        all      6.44      0.00      0.48      0.01      0.00     93.07
10:40:11        all      6.58      0.00      0.48      0.01      0.00     92.93
10:50:11        all      6.45      0.00      0.48      0.01      0.00     93.06
11:00:11        all      6.64      0.00      0.54      0.01      0.00     92.80
11:10:11        all      6.60      0.00      0.50      0.01      0.00     92.89
11:20:11        all      6.54      0.00      0.49      0.01      0.00     92.96
11:30:00        all      6.52      0.00      0.48      0.01      0.00     92.99
11:40:01        all      6.63      0.00      0.51      0.01      0.00     92.85
11:50:02        all      6.60      0.00      0.49      0.01      0.00     92.90
12:00:03        all      6.42      0.00      0.49      0.01      0.00     93.08
12:10:03        all      6.63      0.00      0.48      0.01      0.00     92.88
12:20:04        all      6.52      0.00      0.49      0.01      0.00     92.98

12:20:04        CPU     %user     %nice   %system   %iowait    %steal     %idle
12:30:00        all      6.56      0.00      0.51      0.01      0.00     92.92
12:40:01        all      6.57      0.00      0.49      0.01      0.00     92.93
12:50:02        all      6.52      0.00      0.48      0.01      0.00     92.98
13:00:03        all      6.52      0.00      0.50      0.01      0.00     92.97
13:10:05        all      6.61      0.00      0.49      0.01      0.00     92.89
13:20:05        all      6.48      0.00      0.48      0.01      0.00     93.02
13:30:06        all      6.45      0.00      0.48      0.01      0.00     93.05
13:40:07        all      6.69      0.00      0.53      0.01      0.00     92.76
13:50:08        all      9.78      0.00      1.37      0.04      0.00     88.81
14:00:05        all     10.89      0.00      3.08      0.39      0.00     85.63
14:10:09        all     10.17      0.01      5.68      0.37      0.00     83.77
14:20:11        all      7.78      0.06      1.45      0.10      0.00     90.62
14:30:11        all      6.33      0.00      0.48      0.01      0.00     93.18
14:40:11        all      6.43      0.00      0.48      0.01      0.00     93.08
14:50:11        all      6.39      0.00      0.46      0.01      0.00     93.14
15:00:11        all      6.51      0.00      0.47      0.01      0.00     93.01
15:10:11        all      6.59      0.00      0.49      0.01      0.00     92.92
15:20:11        all      6.55      0.00      0.48      0.00      0.00     92.97
15:30:11        all      6.46      0.00      0.47      0.01      0.00     93.06
15:40:11        all      6.63      0.00      0.53      0.01      0.00     92.84
15:50:11        all      6.42      0.00      0.47      0.01      0.00     93.10
16:00:11        all      6.42      0.00      0.49      0.00      0.00     93.09
16:10:11        all      6.54      0.00      0.49      0.01      0.00     92.97
16:20:11        all      6.57      0.00      0.48      0.01      0.00     92.94
16:30:11        all      6.43      0.00      0.47      0.00      0.00     93.10
16:40:11        all      6.55      0.00      0.51      0.01      0.00     92.94
16:50:11        all      6.47      0.00      0.48      0.01      0.00     93.05
17:00:11        all      6.59      0.00      0.52      0.01      0.00     92.89
17:10:11        all      6.59      0.00      0.49      0.01      0.00     92.91
17:20:11        all      6.94      0.00      0.64      0.01      0.00     92.42
17:30:11        all      6.49      0.00      0.49      0.00      0.00     93.02
17:40:11        all      6.57      0.00      0.49      0.01      0.00     92.92
17:50:11        all      6.43      0.00      0.46      0.00      0.00     93.10
18:00:11        all      6.41      0.00      0.47      0.00      0.00     93.11
18:10:11        all      6.59      0.00      0.49      0.01      0.00     92.92
18:20:11        all      6.52      0.00      0.50      0.01      0.00     92.97
18:30:11        all      5.75      0.00      3.30      0.02      0.00     90.93
18:40:11        all      6.59      0.00      0.52      0.01      0.00     92.89
18:50:11        all      6.54      0.00      0.50      0.01      0.00     92.95
19:00:11        all      6.44      0.00      0.48      0.01      0.00     93.07
19:10:11        all      6.47      0.00      0.49      0.01      0.00     93.04
19:20:11        all      6.52      0.00      0.50      0.00      0.00     92.98
19:30:11        all      6.38      0.00      0.49      0.00      0.00     93.13
19:40:11        all      6.56      0.00      0.49      0.01      0.00     92.95
19:50:11        all      6.53      0.00      0.48      0.00      0.00     92.99
20:00:11        all      6.45      0.00      0.48      0.01      0.00     93.07
20:10:11        all      6.55      0.00      0.50      0.01      0.00     92.95
20:20:11        all      6.56      0.00      0.49      0.01      0.00     92.94
20:30:11        all      6.50      0.00      0.47      0.00      0.00     93.02
20:40:11        all      6.51      0.00      0.48      0.01      0.00     93.01
20:50:11        all      6.46      0.00      0.52      0.00      0.00     93.01
21:00:11        all      6.45      0.00      0.47      0.00      0.00     93.07
21:10:11        all      6.50      0.00      0.49      0.01      0.00     93.00
21:20:11        all      6.54      0.00      0.49      0.01      0.00     92.97
21:30:11        all      6.73      0.00      0.52      0.01      0.00     92.75
21:40:11        all      6.48      0.00      0.50      0.01      0.00     93.02
21:50:11        all      6.56      0.00      0.47      0.00      0.00     92.97

21:50:11        CPU     %user     %nice   %system   %iowait    %steal     %idle
22:00:11        all      6.45      0.00      0.49      0.00      0.00     93.05
22:10:09        all      6.61      0.00      0.53      0.01      0.00     92.85
22:20:10        all      6.89      0.00      0.57      0.01      0.00     92.53
22:30:11        all      6.46      0.00      0.48      0.01      0.00     93.06
22:40:11        all      6.48      0.00      0.48      0.01      0.00     93.04
22:50:11        all      8.18      0.01      0.94      0.02      0.00     90.86
23:00:11        all     10.03      0.00      1.32      0.01      0.00     88.64
Average:        all      7.02      0.00      0.64      0.02      0.00     92.32
</code>

===== Page scans =====
# pgscand/s
<sxh bash; gutter: false>
sar -B
</sxh>

<code>
Linux 5.15.5-1-default (dellirio)       12/09/21        _x86_64_        (12 CPU)

02:41:45     LINUX RESTART      (12 CPU)

02:50:11     pgpgin/s pgpgout/s   fault/s  majflt/s  pgfree/s pgscank/s pgscand/s pgsteal/s    %vmeff
03:00:11      2159.79    474.13   2044.61      0.43   2694.87    684.83      3.15    969.08    140.86
03:10:11        14.27    291.08   2818.88      0.46   3140.79     42.18      8.94     64.69    126.55
03:20:11        22.18    322.18   4364.56      0.09   4605.91    228.94     10.70    186.91     77.99
03:30:00       211.07    435.64   6575.73      0.51   4641.67    574.13      0.00    484.18     84.33
03:40:11       227.27    575.39   8227.81      0.68   7136.55    932.59      7.78   1169.61    124.38
03:50:11       133.24    274.62   9761.47      0.77   9079.62    426.59      1.83    752.30    175.60
04:00:11        68.96    196.59   3620.86      0.43   3839.63     10.41      0.97     22.01    193.50
04:10:11       108.41    343.67   2161.80      1.69   2952.70      6.90      3.54     19.28    184.71
04:20:11        31.92    170.10    937.49      0.05    975.63      0.00      0.00      0.00      0.00
04:30:11         0.57    103.90    589.12      0.00    697.15      0.00      0.00      0.00      0.00
04:40:11         2.06    235.75    649.01      0.00    744.08      0.00      0.00      0.00      0.00
04:50:11         0.20    118.81    556.53      0.00    600.60      0.00      0.00      0.00      0.00
05:00:11         2.40    114.81    545.01      0.05    595.20      0.00      0.00      0.00      0.00
05:10:11         0.45    246.77    689.25      0.03    819.35      0.00      0.00      0.00      0.00
05:20:11         0.40    129.98    661.70      0.01   1109.42      0.00      0.00      0.00      0.00
05:30:11         0.08    117.62    521.24      0.00    582.06      0.00      0.00      0.00      0.00
05:40:11         2.76    249.75    643.12      0.01    780.15      0.00      0.00      0.00      0.00
05:50:11         6.73    126.30    596.49      0.00    671.95      0.00      0.00      0.00      0.00
06:00:11         0.01     96.68    530.67      0.00    618.66      0.00      0.00      0.00      0.00
06:10:11         2.57    266.10    720.01      0.03    858.60      0.00      0.00      0.00      0.00
06:20:11         0.12    121.25    635.55      0.00    717.99      0.00      0.00      0.00      0.00
06:30:11         0.26    113.68    574.20      0.00    767.13      0.00      0.00      0.00      0.00
06:40:11         0.17    261.27    665.27      0.00    784.80      0.00      0.00      0.00      0.00
06:50:11         0.04    118.45    551.86      0.00    629.77      0.00      0.00      0.00      0.00
07:00:11         2.53    112.97    565.24      0.00    613.99      0.00      0.00      0.00      0.00
07:10:11         0.23    285.62    686.18      0.00    835.89      0.00      0.00      0.00      0.00
07:20:11         0.05    138.78    628.65      0.00    682.12      0.00      0.00      0.00      0.00
07:30:11         0.00    122.99    537.63      0.00    599.31      0.00      0.00      0.00      0.00
07:40:11         0.10    274.37    652.92      0.00    818.27      0.00      0.00      0.00      0.00
07:50:11         0.00    118.21    556.10      0.00    616.73      0.00      0.00      0.00      0.00
08:00:11         0.05    120.01    572.02      0.00    637.66      0.00      0.00      0.00      0.00
08:10:11        14.22    266.43    687.34      0.01    825.56      0.00      0.00      0.00      0.00
08:20:11         2.17    139.62    666.28      0.00    793.85      0.00      0.00      0.00      0.00
08:30:11         7.75    131.67    561.53      0.02    716.39      0.00      0.00      0.00      0.00
08:40:11         0.00    256.77    662.32      0.00    859.19      0.00      0.00      0.00      0.00
08:50:11         0.03    123.38    563.36      0.00    674.44      0.00      0.00      0.00      0.00
09:00:11         0.00    114.11    547.27      0.00    705.90      0.00      0.00      0.00      0.00
09:10:11        12.79    252.19    702.99      0.26    920.45      0.00      0.00      0.00      0.00
09:20:11         0.39    147.49    638.85      0.05    760.01      0.00      0.00      0.00      0.00
09:30:11         0.04    136.47    564.92      0.00    708.38      0.00      0.00      0.00      0.00
09:40:11         0.03    255.35    652.99      0.00    878.96      0.00      0.00      0.00      0.00
09:50:11         0.06    132.95    581.55      0.00    698.58      0.00      0.00      0.00      0.00
10:00:11         0.03    139.92    576.65      0.00    706.77      0.00      0.00      0.00      0.00
10:10:11         0.11    255.14    684.26      0.00    872.31      0.00      0.00      0.00      0.00
10:20:11         0.03    143.43    634.54      0.00    817.30      0.00      0.00      0.00      0.00
10:30:11         0.05    141.67    570.88      0.00    703.73      0.00      0.00      0.00      0.00
10:40:11         0.21    260.65    691.39      0.01    879.73      0.00      0.00      0.00      0.00
10:50:11         0.03    126.31    569.04      0.00    692.55      0.00      0.00      0.00      0.00
11:00:11        13.99    225.77    752.19      0.45    992.62      0.00      0.00      0.00      0.00
11:10:11         0.07    252.40    741.12      0.01   1017.54      0.00      0.00      0.00      0.00
11:20:11         4.35    140.42    667.81      0.02    802.82      0.00      0.00      0.00      0.00
11:30:00         0.37    122.16    563.25      0.00    744.91      0.00      0.00      0.00      0.00
11:40:01         0.11    271.16    680.93      0.00    892.33      0.00      0.00      0.00      0.00
11:50:02         5.03    151.65    614.87      0.00    791.47      0.00      0.00      0.00      0.00
12:00:03         0.04    132.86    575.88      0.00    712.23      0.00      0.00      0.00      0.00
12:10:03         0.03    252.91    676.79      0.00    908.54      0.00      0.00      0.00      0.00
12:20:04         0.39    147.20    688.58      0.01    848.35      0.00      1.81      1.65     91.53

12:20:04     pgpgin/s pgpgout/s   fault/s  majflt/s  pgfree/s pgscank/s pgscand/s pgsteal/s    %vmeff
12:30:00         3.05    195.14    724.28      0.01    930.38      0.00      0.00      0.00      0.00
12:40:01         0.05    265.53    725.85      0.00    935.13      0.00      0.28      0.27     97.62
12:50:02         0.08    136.79    598.39      0.00    743.05      0.00      0.00      0.00      0.00
13:00:03         0.15    164.60    585.57      0.00    750.99      0.00      0.46      0.65    140.29
13:10:05        10.86    270.11    743.23      0.32   1012.90      0.00      0.00      0.00      0.00
13:20:05         0.25    133.39    714.48      0.01    831.17      0.00      1.43      2.24    157.06
13:30:06         3.19    163.83    595.92      0.00    765.62      0.00      0.00      0.00      0.00
13:40:07        17.53    295.47    753.81      0.04   1006.05      0.00      0.74      0.92    124.89
13:50:08      1500.23   2825.21   4220.89      2.36   4988.38   1102.40      0.53    913.63     82.84
14:00:05      3497.65  19664.66  17760.30     21.32  19182.88   4749.71      0.00   4638.51     97.66
14:10:09      7508.38  26189.41  28168.63     12.67  37140.56   3818.99      0.42   4797.61    125.61
14:20:11      7462.33   1301.18   1929.30      2.10  51368.51   2506.62      9.02   3611.18    143.55
14:30:11       106.62    275.94    624.41      0.52    760.10     74.37      0.00     73.46     98.77
14:40:11        35.00    461.89    674.80      0.13    867.32     81.00      0.00    108.68    134.17
14:50:11         4.60    272.93    559.00      0.03    690.39      0.00      0.00      0.00      0.00
15:00:11         9.38    283.60    563.90      0.15    682.50     15.57      0.00     27.36    175.78
15:10:11         1.03    395.51    655.12      0.03    908.90     25.17      0.00     25.16     99.97
15:20:11         2.63    259.14    633.32      0.02    787.45      0.00      0.00      0.00      0.00
15:30:11        23.03    243.15    568.04      0.27    751.19      0.00      0.00      0.00      0.00
15:40:11       249.32    478.42    839.47      3.06   1220.78    102.94      2.02    177.56    169.18
15:50:11         0.30    263.99    579.86      0.01    738.63      0.00      0.00      0.00      0.00
16:00:11         6.16    304.27    566.86      0.03    727.45      0.00      0.00      0.00      0.00
16:10:11        22.66    461.11    797.63      0.11   1114.12     42.73      1.76     78.06    175.47
16:20:11        20.59    289.50    686.36      0.13    892.29      0.00     12.96     25.27    194.96
16:30:11         6.28    260.99    585.05      0.20    824.54      0.00      4.31      8.32    193.04
16:40:11        63.72    464.46    788.90      1.87   1000.21      0.00      0.00      0.00      0.00
16:50:11         0.17    312.16    595.96      0.01    798.00      0.00      0.00      0.00      0.00
17:00:11        15.58    353.77    703.13      0.50    928.24      0.00      0.00      0.00      0.00
17:10:11         6.35    463.17    683.95      0.14    965.99     50.44      0.00     87.16    172.80
17:20:11         2.24    404.11    852.33      0.12   1148.76      0.00      0.00      0.00      0.00
17:30:11         0.79    329.23    600.76      0.07    775.42      0.00      2.64      5.13    194.32
17:40:11         0.97    462.83    734.01      0.03    985.74      7.37      3.28     20.87    195.99
17:50:11         0.06    269.99    572.75      0.01    744.20      0.00      0.43      0.78    184.31
18:00:11         2.95    282.43    559.24      0.02    735.91      0.00      7.18     12.81    178.33
18:10:11         7.49    426.80    754.54      0.02   1029.93     14.22      9.14     39.07    167.23
18:20:11        38.10    343.63    722.21      0.73    923.66      0.00      7.30     13.22    181.06
18:30:11      1890.43    475.87   1852.15      0.38   3877.73   1477.39      7.27   1530.73    103.10
18:40:11        12.11    362.11    758.64      0.09    972.39      0.00      0.00      0.00      0.00
18:50:11         5.01    185.02    622.25      0.04    792.26      0.00      0.00      0.00      0.00
19:00:11         1.17    161.32    569.85      0.06    697.54      0.00      0.69      1.20    173.49
19:10:11         3.57    248.52    707.11      0.10    883.61      3.27      3.24     12.73    195.80
19:20:11         3.03    145.74    707.86      0.03    838.90      0.00      0.00      0.00      0.00
19:30:11         0.53    163.73    613.13      0.07    729.05     10.95      0.34     22.07    195.45
19:40:11        17.60    262.27    737.98      0.19    945.50      8.70      1.79     20.73    197.55
19:50:11         0.03    129.05    579.51      0.01    715.93      0.00      0.00      0.00      0.00
20:00:11         0.04    178.64    574.52      0.01    692.40      1.09      0.00      2.14    196.64
20:10:11        52.93    267.99    849.40      0.22   1146.37     39.39      0.00     78.38    198.98
20:20:11        33.46    140.97    666.55      0.45    841.07      0.00      0.00      0.00      0.00
20:30:11         6.99    108.92    564.66      0.06    707.29      0.00      1.06      1.94    183.60
20:40:11         0.03    228.57    683.28      0.01    873.53      0.00      5.95     10.87    182.75
20:50:11         1.78    142.83    615.92      0.02    849.07      0.00      6.25     12.14    194.24
21:00:11         0.02    110.33    536.53      0.01    697.26      0.00      5.52      9.34    169.10
21:10:11        26.75    253.67    718.77      1.10    926.06      0.00      5.15      8.09    156.92
21:20:11         0.78    108.53    664.89      0.18    828.10      0.00      6.73      9.97    148.16
21:30:11         0.36    107.63    634.68      0.03    864.87      0.00     14.52     27.44    188.91
21:40:11         0.16    265.75    704.65      0.01    907.50      0.00      0.34      0.53    157.21
21:50:11         0.17    107.22    607.32      0.00    801.65      0.00      0.00      0.00      0.00

21:50:11     pgpgin/s pgpgout/s   fault/s  majflt/s  pgfree/s pgscank/s pgscand/s pgsteal/s    %vmeff
22:00:11         0.13    147.05    570.40      0.02    735.08      0.00      4.46      8.37    187.45
22:10:09       190.58    506.01   1000.52      0.50   1301.50     23.45      0.00     46.35    197.63
22:20:10       117.12    584.55   1828.52      0.10   2143.24    111.97      0.64    223.33    198.32
22:30:11        19.58    125.50    589.88      0.04    761.71      0.00      0.00      0.00      0.00
22:40:11         4.71    273.35    790.63      0.12    982.89      0.00      1.96      3.35    170.75
22:50:11       346.74    188.52   2135.91     14.42   2501.70      0.00      0.00      0.00      0.00
23:00:11        33.03    253.85   5366.47      0.88   5466.65     43.77      0.00     86.02    196.54
23:10:11         9.29    300.25   4808.97      0.18   4991.85     56.38      2.89    116.08    195.84
Average:       217.45    639.20   1442.44      0.59   2100.62    141.75      1.41    168.89    117.98
</code>


===== free =====
Speicher/SWAP-Auslastung in GB
<sxh bash; gutter: false>
free -g
</sxh>

<code>
               total        used        free      shared  buff/cache   available
Mem:        16318416     7295948      451604      524528     8570864     8159504
Swap:       16322880     4127488    12195392
</code>

===== Netzwerk =====
# RX-ERR
<sxh bash; gutter: false>
netstat -i
</sxh>

<code>
Kernel Interface table
Iface   MTU Met    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
bond0    1500   0 500615690      0 680848      0 495289774      0      0      0 BMmRU
bond1    1500   0  7118264      0 387171      0  5325736      0      0      0 BMmRU
bond1:1  1500   0     - no statistics available -                        BMmRU
bond1:2  1500   0     - no statistics available -                        BMmRU
bond1:4  1500   0     - no statistics available -                        BMmRU
bond1:5  1500   0     - no statistics available -                        BMmRU
bond1:6  1500   0     - no statistics available -                        BMmRU
bond2    1500   0   730890      0 227226      0     4492      0      0      0 BMmRU
eth0     1500   0   680901      0 680846      0        0      0      0      0 BMsRU
eth1     1500   0   386965      0 386965      0        0      0      0      0 BMsRU
eth2     1500   0   227224      0 227224      0        0      0      0      0 BMsRU
eth3     1500   0   503666      0      0      0     4492      0      0      0 BMsRU
eth4     1500   0 499934789      0      0      0 495289774      0      0      0 BMsRU
eth5     1500   0  6731299      0      0      0  5325736      0      0      0 BMsRU
eth6     9000   0 656029363      0    214      0 703127744      0      0      0 BMRU
eth6:1   9000   0     - no statistics available -                        BMRU
eth7     9000   0 1173507454      0    216      0 1037376088      0      0      0 BMRU
eth7:1   9000   0     - no statistics available -                        BMRU
lo      65536   0 736520436      0      0      0 736520436      0      0      0 LRU
</code>


Im Block "Tcp" das Verhältnis von segments retransmitted zu segments send out (Diagnose wie viele Pakete retransmitted werden mussten)
<sxh bash; gutter: false>
netstat -st
</sxh>

<code>
IcmpMsg:
    InType0: 622204
    InType3: 6052
    InType8: 746391
    InType11: 23
    InType13: 3
    InType15: 3
    InType17: 3
    OutType0: 745713
    OutType3: 5401
    OutType8: 622447
    OutType11: 57
    OutType14: 3
Tcp:
    3519494 active connections openings
    761050 passive connection openings
    2016541 failed connection attempts
    451582 connection resets received
    291 connections established
    565428806 segments received
    582593439 segments send out
    
2846212       segments retransmited
    10 bad segments received.
    1231155 resets sent
    InCsumErrors: 0
UdpLite:
    InDatagrams: 0
    NoPorts: 0
    InErrors: 0
    OutDatagrams: 0
    RcvbufErrors: 0
    SndbufErrors: 0
    InCsumErrors: 0
    IgnoredMulti: 0
TcpExt:
    1 packets pruned from receive queue because of socket buffer overrun
    35 ICMP packets dropped because they were out-of-window
    ArpFilter: 0
    217878 TCP sockets finished time wait in fast timer
    4115511 delayed acks sent
    1135 delayed acks further delayed because of locked socket
    Quick ack mode was activated 157740 times
    4919622 packets directly queued to recvmsg prequeue.
    2106470 packets directly received from backlog
    728738337 packets directly received from prequeue
    260056903 packets header predicted
    1756921 packets header predicted and directly queued to user
    TCPPureAcks: 12917450
    TCPHPAcks: 256486617
    TCPRenoRecovery: 0
    TCPSackRecovery: 21917
    TCPSACKReneging: 0
    TCPFACKReorder: 36
    TCPSACKReorder: 0
    TCPRenoReorder: 0
    TCPTSReorder: 1
    TCPFullUndo: 4
    TCPPartialUndo: 0
    TCPDSACKUndo: 1056
    TCPLossUndo: 978
    TCPLostRetransmit: 1401
    TCPRenoFailures: 0
    TCPSackFailures: 421
    TCPLossFailures: 7
    TCPFastRetrans: 46262
    TCPForwardRetrans: 1889
    TCPSlowStartRetrans: 1309
    TCPTimeouts: 1915
    TCPLossProbes: 2790807
    TCPLossProbeRecovery: 3514
    TCPRenoRecoveryFail: 0
    TCPSackRecoveryFail: 258
    TCPSchedulerFailed: 0
    TCPRcvCollapsed: 43
    TCPDSACKOldSent: 157742
    TCPDSACKOfoSent: 0
    TCPDSACKRecv: 2783642
    TCPDSACKOfoRecv: 0
    TCPAbortOnData: 166874
    TCPAbortOnClose: 4068
    TCPAbortOnMemory: 0
    TCPAbortOnTimeout: 237
    TCPAbortOnLinger: 0
    TCPAbortFailed: 0
    TCPMemoryPressures: 0
    TCPSACKDiscard: 0
    TCPDSACKIgnoredOld: 4
    TCPDSACKIgnoredNoUndo: 2550554
    TCPSpuriousRTOs: 0
    TCPMD5NotFound: 0
    TCPMD5Unexpected: 0
    TCPSackShifted: 14373
    TCPSackMerged: 25768
    TCPSackShiftFallback: 263888
    TCPBacklogDrop: 0
    TCPMinTTLDrop: 0
    TCPDeferAcceptDrop: 0
    IPReversePathFilter: 285686
    TCPTimeWaitOverflow: 0
    TCPReqQFullDoCookies: 0
    TCPReqQFullDrop: 0
    TCPRetransFail: 0
    TCPRcvCoalesce: 1677684
    TCPOFOQueue: 5972
    TCPOFODrop: 0
    TCPOFOMerge: 0
    TCPChallengeACK: 11751
    TCPSYNChallenge: 10
    TCPFastOpenActive: 0
    TCPFastOpenActiveFail: 0
    TCPFastOpenPassive: 0
    TCPFastOpenPassiveFail: 0
    TCPFastOpenListenOverflow: 0
    TCPFastOpenCookieReqd: 0
    TCPSpuriousRtxHostQueues: 57
    BusyPollRxPackets: 0
    TCPAutoCorking: 229104566
    TCPFromZeroWindowAdv: 263
    TCPToZeroWindowAdv: 263
    TCPWantZeroWindowAdv: 185
    TCPSynRetrans: 2858
    TCPOrigDataSent: 542987167
    TCPHystartTrainDetect: 432795
    TCPHystartTrainCwnd: 8644006
    TCPHystartDelayDetect: 12
    TCPHystartDelayCwnd: 225
    TCPACKSkippedSynRecv: 0
    TCPACKSkippedPAWS: 0
    TCPACKSkippedSeq: 7
    TCPACKSkippedFinWait2: 0
    TCPACKSkippedTimeWait: 0
    TCPACKSkippedChallenge: 0
    TCPWinProbe: 81
    TCPKeepAlive: 533011
    TCPMTUPFail: 0
    TCPMTUPSuccess: 0
IpExt:
    InNoRoutes: 0
    InTruncatedPkts: 0
    InMcastPkts: 3618291
    OutMcastPkts: 811102
    InBcastPkts: 13248
    OutBcastPkts: 31
    InOctets: 3464926929670
    OutOctets: 2631875144172
    InMcastOctets: 580452056
    OutMcastOctets: 42202479
    InBcastOctets: 4579903
    OutBcastOctets: 12340
    InCsumErrors: 0
    InNoECTPkts: 3072086529
    InECT1Pkts: 0
    InECT0Pkts: 0
    InCEPkts: 0
</code>

===== AD =====
==== winbind ====
<sxh bash; gutter: false>
wbinfo -p
</sxh>
  Ping to winbindd succeeded

<sxh bash; gutter: false>
wbinfo -P
</sxh>
  checking the NETLOGON for domain[IDST] dc connection to "N4363022.idst.ibaintern.de" succeeded

==== AD-abfrage id gegen alle user ====
<sxh bash; gutter: false>
cd /home/BADA/
for X in *; do id BADA\\$X; done;
</sxh>

<code>
uid=409768(BADA\ax) gid=04402(BADA\do) groups=409768(BADA\ax),04402(BADA\do),06900(BADA\z9),387321(BADA\z9),50061(BADA\z9),71422(BADA\dsl),42766(BADA\z9),06508(BADA\z9),387329(BADA\z9),28064(BADA\z9),06596(BADA\z9),78981(BADA\z9),76053(BADA\l9),50060(BADA\l9),78980(BADA\l9),06901(BADA\l9),28253(BADA\l9),019318(BADA\l9),06597(BADA\l9),387320(BADA\l9),019314(BADA\l9),011016(BADA\l9),28063(BADA\l9),50070(BADA\l9),065087(BADA\l9),387318(BADA\l9),42765(BADA\l9)
uid=85757(BADA\ba) gid=04402(BADA\do) groups=85757(BADA\ba),04402(BADA\do),78915(BADA\z9),06596(BADA\z9),71422(BADA\dsl),76053(BADA\l9),28253(BADA\l9),06597(BADA\l9),78914(BADA\l9)
uid=461093(BADA\ba) gid=04402(BADA\do) groups=461093(BADA\ba),04402(BADA\do),497671(BADA\z9),78915(BADA\z9),06596(BADA\z9),71422(BADA\dsl),76053(BADA\l9),28253(BADA\l9),06597(BADA\l9),78914(BADA\l9)
uid=471050(BADA\ba) gid=04402(BADA\do) groups=471050(BADA\ba),04402(BADA\do),06900(BADA\z9),50061(BADA\z9),42766(BADA\z9),497827(BADA\z9),06508(BADA\z9),497742(BADA\z9),497768(BADA\z9),28064(BADA\z9),06596(BADA\z9),497702(BADA\z9),78981(BADA\z9),497746(BADA\z9),71422(BADA\dsl),76053(BADA\l9),50060(BADA\l9),78980(BADA\l9),06901(BADA\l9),28253(BADA\l9),019318(BADA\l9),06597(BADA\l9),019314(BADA\l9),011016(BADA\l9),28063(BADA\l9),50070(BADA\l9),065087(BADA\l9),42765(BADA\l9)
uid=226129(BADA\bo) gid=04402(BADA\do) groups=226129(BADA\bo),04402(BADA\do),06900(BADA\z9),387321(BADA\z9),50061(BADA\z9),42766(BADA\z9),387329(BADA\z9),28064(BADA\z9),06596(BADA\z9),78981(BADA\z9),71422(BADA\dsl),76053(BADA\l9),50060(BADA\l9),78980(BADA\l9),06901(BADA\l9),28253(BADA\l9),019318(BADA\l9),06597(BADA\l9),387320(BADA\l9),019314(BADA\l9),28063(BADA\l9),50070(BADA\l9),387318(BADA\l9),42765(BADA\l9)
uid=461970(BADA\bo) gid=04402(BADA\do) groups=461970(BADA\bo),04402(BADA\do),06900(BADA\z9),387321(BADA\z9),413438(BADA\z9),50061(BADA\z9),42766(BADA\z9),413441(BADA\z9),497742(BADA\z9),497768(BADA\z9),387329(BADA\z9),28064(BADA\z9),06596(BADA\z9),497702(BADA\z9),78981(BADA\z9),497746(BADA\z9),71422(BADA\dsl),76053(BADA\l9),50060(BADA\l9),78980(BADA\l9),06901(BADA\l9),28253(BADA\l9),019318(BADA\l9),06597(BADA\l9),387320(BADA\l9),019314(BADA\l9),28063(BADA\l9),50070(BADA\l9),387318(BADA\l9),42765(BADA\l9)
uid=461276(BADA\br) gid=04402(BADA\do) groups=461276(BADA\br),04402(BADA\do),06900(BADA\z9),387321(BADA\z9),413438(BADA\z9),50061(BADA\z9),42766(BADA\z9),497827(BADA\z9),413441(BADA\z9),06508(BADA\z9),497742(BADA\z9),497768(BADA\z9),387329(BADA\z9),28064(BADA\z9),06596(BADA\z9),497702(BADA\z9),78981(BADA\z9),497746(BADA\z9),71422(BADA\dsl),76053(BADA\l9),50060(BADA\l9),78980(BADA\l9),06901(BADA\l9),28253(BADA\l9),019318(BADA\l9),06597(BADA\l9),387320(BADA\l9),019314(BADA\l9),011016(BADA\l9),28063(BADA\l9),50070(BADA\l9),065087(BADA\l9),387318(BADA\l9),42765(BADA\l9)
uid=468484(BADA\ch) gid=04402(BADA\do) groups=468484(BADA\ch),04402(BADA\do),497893(BADA\z9),77856(BADA\z9),06596(BADA\z9),71422(BADA\dsl),76053(BADA\l9),28253(BADA\l9),06597(BADA\l9),77855(BADA\l9)
uid=28157(BADA\co) gid=04402(BADA\do) groups=28157(BADA\co),04402(BADA\do),387321(BADA\z9),06500(BADA\z9),78915(BADA\z9),387329(BADA\z9),06596(BADA\z9),71422(BADA\dsl),76053(BADA\l9),06501(BADA\l9),28253(BADA\l9),06597(BADA\l9),387320(BADA\l9),78914(BADA\l9),387318(BADA\l9)
uid=461042(BADA\co) gid=04402(BADA\do) groups=461042(BADA\co),04402(BADA\do),497844(BADA\z9),497671(BADA\z9),06500(BADA\z9),78915(BADA\z9),06596(BADA\z9),71422(BADA\dsl),76053(BADA\l9),06501(BADA\l9),28253(BADA\l9),06597(BADA\l9),78914(BADA\l9)
uid=460539(BADA\da) gid=04402(BADA\do) groups=460539(BADA\da),04402(BADA\do),06900(BADA\z9),387321(BADA\z9),413438(BADA\z9),50061(BADA\z9),42766(BADA\z9),497827(BADA\z9),413441(BADA\z9),06508(BADA\z9),497742(BADA\z9),497768(BADA\z9),387329(BADA\z9),28064(BADA\z9),06596(BADA\z9),497702(BADA\z9),78981(BADA\z9),497746(BADA\z9),71422(BADA\dsl),76053(BADA\l9),50060(BADA\l9),78980(BADA\l9),06901(BADA\l9),28253(BADA\l9),019318(BADA\l9),06597(BADA\l9),387320(BADA\l9),019314(BADA\l9),011016(BADA\l9),28063(BADA\l9),50070(BADA\l9),065087(BADA\l9),387318(BADA\l9),42765(BADA\l9)
uid=89047(BADA\di) gid=04402(BADA\do) groups=89047(BADA\di),04402(BADA\do),06900(BADA\z9),387321(BADA\z9),50061(BADA\z9),387329(BADA\z9),28064(BADA\z9),06596(BADA\z9),78981(BADA\z9),71422(BADA\dsl),76053(BADA\l9),50060(BADA\l9),78980(BADA\l9),06901(BADA\l9),28253(BADA\l9),06597(BADA\l9),387320(BADA\l9),019314(BADA\l9),28063(BADA\l9),50070(BADA\l9),387318(BADA\l9)
uid=006856(BADA\en) gid=04402(BADA\do) groups=006856(BADA\en),04402(BADA\do),78915(BADA\z9),06596(BADA\z9),71422(BADA\dsl),76053(BADA\l9),28253(BADA\l9),06597(BADA\l9),78914(BADA\l9)
uid=461280(BADA\en) gid=04402(BADA\do) groups=461280(BADA\en),04402(BADA\do),497671(BADA\z9),78915(BADA\z9),06596(BADA\z9),71422(BADA\dsl),76053(BADA\l9),28253(BADA\l9),06597(BADA\l9),78914(BADA\l9)
uid=85723(BADA\ga) gid=04402(BADA\do) groups=85723(BADA\ga),04402(BADA\do),06900(BADA\z9),387321(BADA\z9),50061(BADA\z9),387329(BADA\z9),28064(BADA\z9),06596(BADA\z9),78981(BADA\z9),71422(BADA\dsl),76053(BADA\l9),50060(BADA\l9),78980(BADA\l9),06901(BADA\l9),28253(BADA\l9),06597(BADA\l9),387320(BADA\l9),019314(BADA\l9),28063(BADA\l9),50070(BADA\l9),387318(BADA\l9)
uid=461942(BADA\ga) gid=04402(BADA\do) groups=461942(BADA\ga),04402(BADA\do),06900(BADA\z9),387321(BADA\z9),413438(BADA\z9),50061(BADA\z9),413441(BADA\z9),497742(BADA\z9),387329(BADA\z9),28064(BADA\z9),06596(BADA\z9),497702(BADA\z9),78981(BADA\z9),497746(BADA\z9),71422(BADA\dsl),76053(BADA\l9),50060(BADA\l9),78980(BADA\l9),06901(BADA\l9),28253(BADA\l9),06597(BADA\l9),387320(BADA\l9),019314(BADA\l9),28063(BADA\l9),50070(BADA\l9),387318(BADA\l9)
uid=461131(BADA\he) gid=04402(BADA\do) groups=461131(BADA\he),04402(BADA\do),497778(BADA\z),06900(BADA\z9),065357(BADA\z9),50061(BADA\z9),42766(BADA\z9),497686(BADA\z9),497827(BADA\z9),06508(BADA\z9),497742(BADA\z9),497768(BADA\z9),28064(BADA\z9),06596(BADA\z9),497702(BADA\z9),78981(BADA\z9),78907(BADA\z9),498933(BADA\z9),497746(BADA\z9),47231(BADA\z9),71422(BADA\dsl),76053(BADA\l9),50060(BADA\l9),78980(BADA\l9),06901(BADA\l9),28253(BADA\l9),019318(BADA\l9),06597(BADA\l9),019314(BADA\l9),011016(BADA\l9),065356(BADA\l9),78906(BADA\l9),28063(BADA\l9),50070(BADA\l9),065087(BADA\l9),47230(BADA\l9),42765(BADA\l9)
uid=063520(BADA\he) gid=04402(BADA\do) groups=063520(BADA\he),04402(BADA\do),41663(BADA\l9),019271(BADA\l9),77855(BADA\l9),011016(BADA\l9),06597(BADA\l9),28253(BADA\l9),019273(BADA\l9),76053(BADA\l9),71422(BADA\dsl),41664(BADA\z9),019272(BADA\z9),06596(BADA\z9),77856(BADA\z9),019274(BADA\z9)
uid=006852(BADA\ho) gid=04402(BADA\do) groups=006852(BADA\ho),04402(BADA\do),78914(BADA\l9),06597(BADA\l9),28253(BADA\l9),76053(BADA\l9),71422(BADA\dsl),06596(BADA\z9),78915(BADA\z9)
uid=468680(BADA\kr) gid=04402(BADA\do) groups=468680(BADA\kr),04402(BADA\do),06900(BADA\z9),387321(BADA\z9),413438(BADA\z9),50061(BADA\z9),42766(BADA\z9),497827(BADA\z9),413441(BADA\z9),06508(BADA\z9),497742(BADA\z9),497768(BADA\z9),387329(BADA\z9),28064(BADA\z9),06596(BADA\z9),497702(BADA\z9),78981(BADA\z9),497746(BADA\z9),71422(BADA\dsl),76053(BADA\l9),50060(BADA\l9),78980(BADA\l9),06901(BADA\l9),28253(BADA\l9),019318(BADA\l9),06597(BADA\l9),387320(BADA\l9),019314(BADA\l9),011016(BADA\l9),28063(BADA\l9),50070(BADA\l9),065087(BADA\l9),387318(BADA\l9),42765(BADA\l9)
uid=06680(BADA\lu) gid=04402(BADA\do) groups=06680(BADA\lu),04402(BADA\do),387318(BADA\l9),50070(BADA\l9),28063(BADA\l9),019314(BADA\l9),387320(BADA\l9),06597(BADA\l9),28253(BADA\l9),06901(BADA\l9),78980(BADA\l9),011487(BADA\l9),50060(BADA\l9),76053(BADA\l9),71422(BADA\dsl),78981(BADA\z9),06596(BADA\z9),28064(BADA\z9),387329(BADA\z9),01148(BADA\z9),50061(BADA\z9),387321(BADA\z9),06900(BADA\z9)
uid=76375(BADA\mi) gid=04402(BADA\do) groups=76375(BADA\mi),04402(BADA\do),06512(BADA\z9),065049(BADA\z9),71422(BADA\dsl),019274(BADA\z9),01148(BADA\z9),011015(BADA\z9),77856(BADA\z9),06596(BADA\z9),065041(BADA\z9),019272(BADA\z9),76053(BADA\l9),11339,19597(IB\vi),019273(BADA\l9),011487(BADA\l9),065040(BADA\l9),28253(BADA\l9),011014(BADA\l9),06597(BADA\l9),065038(BADA\l9),011016(BADA\l9),06513(BADA\l9),77855(BADA\l9),019271(BADA\l9)
id: ‘BADA\\mo’: no such user
uid=298526(BADA\ni) gid=04402(BADA\do) groups=298526(BADA\ni),04402(BADA\do),06900(BADA\z9),387321(BADA\z9),50061(BADA\z9),42766(BADA\z9),387329(BADA\z9),28064(BADA\z9),06596(BADA\z9),78981(BADA\z9),78907(BADA\z9),71422(BADA\dsl),76053(BADA\l9),50060(BADA\l9),78980(BADA\l9),06901(BADA\l9),28253(BADA\l9),019318(BADA\l9),06597(BADA\l9),387320(BADA\l9),019314(BADA\l9),78906(BADA\l9),28063(BADA\l9),50070(BADA\l9),387318(BADA\l9),42765(BADA\l9)
uid=461234(BADA\ni) gid=04402(BADA\do) groups=461234(BADA\ni),04402(BADA\do),06900(BADA\z9),387321(BADA\z9),413438(BADA\z9),50061(BADA\z9),42766(BADA\z9),497686(BADA\z9),413441(BADA\z9),497742(BADA\z9),497768(BADA\z9),387329(BADA\z9),28064(BADA\z9),06596(BADA\z9),497702(BADA\z9),78981(BADA\z9),78907(BADA\z9),497746(BADA\z9),71422(BADA\dsl),76053(BADA\l9),50060(BADA\l9),78980(BADA\l9),06901(BADA\l9),28253(BADA\l9),019318(BADA\l9),06597(BADA\l9),387320(BADA\l9),019314(BADA\l9),78906(BADA\l9),28063(BADA\l9),50070(BADA\l9),387318(BADA\l9),42765(BADA\l9)
uid=468663(BADA\pa) gid=04402(BADA\do) groups=468663(BADA\pa),04402(BADA\do),50061(BADA\z9),497827(BADA\z9),06508(BADA\z9),06596(BADA\z9),497702(BADA\z9),78981(BADA\z9),497746(BADA\z9),71422(BADA\dsl),76053(BADA\l9),50060(BADA\l9),78980(BADA\l9),28253(BADA\l9),06597(BADA\l9),019314(BADA\l9),011016(BADA\l9),50070(BADA\l9),065087(BADA\l9)
id: ‘BADA\\pa’: no such user
uid=231922(BADA\pa) gid=04402(BADA\do) groups=231922(BADA\pa),04402(BADA\do),06900(BADA\z9),50061(BADA\z9),42766(BADA\z9),28064(BADA\z9),06596(BADA\z9),78981(BADA\z9),71422(BADA\dsl),76053(BADA\l9),50060(BADA\l9),78980(BADA\l9),06901(BADA\l9),28253(BADA\l9),019318(BADA\l9),06597(BADA\l9),019314(BADA\l9),28063(BADA\l9),50070(BADA\l9),42765(BADA\l9)
uid=461078(BADA\pa) gid=04402(BADA\do) groups=461078(BADA\pa),04402(BADA\do),06900(BADA\z9),387321(BADA\z9),413438(BADA\z9),50061(BADA\z9),42766(BADA\z9),413441(BADA\z9),497742(BADA\z9),497768(BADA\z9),387329(BADA\z9),28064(BADA\z9),06596(BADA\z9),497702(BADA\z9),78981(BADA\z9),497746(BADA\z9),71422(BADA\dsl),76053(BADA\l9),50060(BADA\l9),78980(BADA\l9),06901(BADA\l9),28253(BADA\l9),019318(BADA\l9),06597(BADA\l9),387320(BADA\l9),019314(BADA\l9),28063(BADA\l9),50070(BADA\l9),387318(BADA\l9),42765(BADA\l9)
uid=06681(BADA\re) gid=04402(BADA\do) groups=06681(BADA\re),04402(BADA\do),06900(BADA\z9),50061(BADA\z9),71422(BADA\dsl),387329(BADA\z9),28064(BADA\z9),06596(BADA\z9),78981(BADA\z9),76053(BADA\l9),50060(BADA\l9),78980(BADA\l9),06901(BADA\l9),28253(BADA\l9),06597(BADA\l9),019314(BADA\l9),28063(BADA\l9),50070(BADA\l9),387318(BADA\l9)
uid=461086(BADA\re) gid=04402(BADA\do) groups=461086(BADA\re),04402(BADA\do),06900(BADA\z9),50061(BADA\z9),413441(BADA\z9),497742(BADA\z9),387329(BADA\z9),28064(BADA\z9),06596(BADA\z9),497702(BADA\z9),78981(BADA\z9),497746(BADA\z9),71422(BADA\dsl),76053(BADA\l9),50060(BADA\l9),78980(BADA\l9),06901(BADA\l9),28253(BADA\l9),06597(BADA\l9),019314(BADA\l9),28063(BADA\l9),50070(BADA\l9),387318(BADA\l9)
uid=434520(BADA\sc) gid=04402(BADA\do) groups=434520(BADA\sc),04402(BADA\do),77480(BADA\z9),434596(BADA\z9),71422(BADA\dsl),76053(BADA\l9),28253(BADA\l9),442071(BADA\l9),442073(BADA\l9)
uid=006853(BADA\sc) gid=04402(BADA\do) groups=006853(BADA\sc),04402(BADA\do),78914(BADA\l9),06597(BADA\l9),28253(BADA\l9),76053(BADA\l9),71422(BADA\dsl),06596(BADA\z9),78915(BADA\z9)
uid=006855(BADA\sc) gid=04402(BADA\do) groups=006855(BADA\sc),04402(BADA\do),78914(BADA\l9),06597(BADA\l9),28253(BADA\l9),76053(BADA\l9),71422(BADA\dsl),06596(BADA\z9),78915(BADA\z9)
uid=461308(BADA\si) gid=04402(BADA\do) groups=461308(BADA\si),04402(BADA\do),06900(BADA\z9),50061(BADA\z9),413441(BADA\z9),497742(BADA\z9),387329(BADA\z9),28064(BADA\z9),06596(BADA\z9),497702(BADA\z9),78981(BADA\z9),497746(BADA\z9),71422(BADA\dsl),76053(BADA\l9),50060(BADA\l9),78980(BADA\l9),06901(BADA\l9),28253(BADA\l9),06597(BADA\l9),019314(BADA\l9),28063(BADA\l9),50070(BADA\l9),387318(BADA\l9)
uid=06684(BADA\si) gid=04402(BADA\do) groups=06684(BADA\si),04402(BADA\do),06900(BADA\z9),50061(BADA\z9),42766(BADA\z9),387329(BADA\z9),28064(BADA\z9),06596(BADA\z9),78981(BADA\z9),71422(BADA\dsl),76053(BADA\l9),50060(BADA\l9),78980(BADA\l9),06901(BADA\l9),28253(BADA\l9),019318(BADA\l9),06597(BADA\l9),019314(BADA\l9),28063(BADA\l9),50070(BADA\l9),387318(BADA\l9),42765(BADA\l9)
uid=89048(BADA\si) gid=04402(BADA\do) groups=89048(BADA\si),04402(BADA\do),06900(BADA\z9),06512(BADA\z9),50061(BADA\z9),71422(BADA\dsl),28064(BADA\z9),06596(BADA\z9),78981(BADA\z9),76053(BADA\l9),11339,19597(IB\vi),50060(BADA\l9),78980(BADA\l9),06901(BADA\l9),28253(BADA\l9),06597(BADA\l9),019314(BADA\l9),011016(BADA\l9),06513(BADA\l9),28063(BADA\l9),50070(BADA\l9)
uid=006857(BADA\so) gid=04402(BADA\do) groups=006857(BADA\so),04402(BADA\do),387318(BADA\l9),78914(BADA\l9),387320(BADA\l9),06597(BADA\l9),28253(BADA\l9),76053(BADA\l9),71422(BADA\dsl),06596(BADA\z9),387329(BADA\z9),78915(BADA\z9),387321(BADA\z9)
uid=461460(BADA\so) gid=04402(BADA\do) groups=461460(BADA\so),04402(BADA\do),387321(BADA\z9),413438(BADA\z9),413441(BADA\z9),497671(BADA\z9),78915(BADA\z9),387329(BADA\z9),06596(BADA\z9),71422(BADA\dsl),76053(BADA\l9),28253(BADA\l9),06597(BADA\l9),387320(BADA\l9),78914(BADA\l9),387318(BADA\l9)
uid=404600(BADA\uf) gid=04402(BADA\do) groups=404600(BADA\uf),04402(BADA\do),011800(BADA\z0),71422(BADA\dsl),380450(BADA\z9)
uid=434521(BADA\vo) gid=04402(BADA\do) groups=434521(BADA\vo),04402(BADA\do),77480(BADA\z9),434596(BADA\z9),71422(BADA\dsl),76053(BADA\l9),28253(BADA\l9),442071(BADA\l9),442073(BADA\l9)
uid=461334(BADA\za) gid=04402(BADA\do) groups=461334(BADA\za),04402(BADA\do),434639(BADA\z9),06900(BADA\z9),387321(BADA\z9),413438(BADA\z9),50061(BADA\z9),42766(BADA\z9),413441(BADA\z9),497742(BADA\z9),497768(BADA\z9),387329(BADA\z9),28064(BADA\z9),06596(BADA\z9),497702(BADA\z9),78981(BADA\z9),497746(BADA\z9),71422(BADA\dsl),76053(BADA\l9),50060(BADA\l9),78980(BADA\l9),06901(BADA\l9),28253(BADA\l9),019318(BADA\l9),06597(BADA\l9),387320(BADA\l9),019314(BADA\l9),442190(BADA\l9),28063(BADA\l9),50070(BADA\l9),387318(BADA\l9),42765(BADA\l9)
</code>

===== Hardware =====
==== hplog ====
<sxh bash; gutter: false>
hplog -v
</sxh>

<code>
ID   Severity       Initial Time      Update Time       Count
-------------------------------------------------------------
0000 Information    14:37  12/11/2012 14:37  12/11/2012 0001
LOG: Maintenance note: IML Cleared (IML Log cleared by the 

0001 Caution        09:08  02/08/2013 09:08  02/08/2013 0001
LOG: POST Error: 1704-Slot X Drive Array - Array Accelerator Super-Cap is charging. The Array Accelerator Cache will be enabled once Super-Cap has been charged. No action is required.

0002 Repaired       08:13  02/11/2013 05:14  04/15/2018 0015
LOG: Network Adapter Link Down (Slot 3, Port 2)

0003 Repaired       00:12  03/08/2013 05:14  04/15/2018 0012
LOG: Network Adapter Link Down (Slot 3, Port 1)

0004 Repaired       12:46  06/20/2013 20:59  12/27/2016 0006
LOG: Network Adapters Redundancy Reduced (Slot 0, Port 4)

0005 Repaired       12:46  06/20/2013 05:17  04/15/2018 0008
LOG: Network Adapter Link Down (Slot 2, Port 2)

0006 Repaired       12:46  06/20/2013 05:17  04/15/2018 0008
LOG: Network Adapter Link Down (Slot 2, Port 1)

0007 Repaired       13:41  06/21/2013 05:19  04/15/2018 0006
LOG: Network Adapter Link Down (Slot 0, Port 2)

0008 Repaired       13:42  06/21/2013 20:40  12/27/2016 0004
LOG: Network Adapters Redundancy Reduced (Slot 0, Port 3)

0009 Repaired       13:42  06/21/2013 05:19  04/15/2018 0006
LOG: Network Adapters Redundancy Reduced (Slot 0, Port 1)

0010 Caution        06:09  06/17/2014 06:09  06/17/2014 0001
LOG: POST Error: 1792-Drive Array Reports Valid Data Found in Array Accelerator

0011 Repaired       14:38  11/08/2016 15:20  11/08/2016 0001
LOG: System Power Supply: General Failure (Power Supply 1)

0012 Repaired       14:38  11/08/2016 15:20  11/08/2016 0001
LOG: System Power Supplies Not Redundant

0013 Critical       11:35  03/21/2017 11:35  03/21/2017 0001
LOG: ASR Detected by System ROM

0014 Repaired       05:43  10/22/2017 13:57  04/15/2018 0002
LOG: Internal Storage Enclosure Device Failure (Bay 2, Box 1, Port 1I, Slot 0)

0015 Caution        18:59  11/30/2017 18:59  11/30/2017 0001
LOG: POST Error: 1786-Drive Array Recovery Needed

0016 Critical       10:50  01/31/2018 10:50  01/31/2018 0001
LOG: Uncorrectable PCI Express Error (Embedded device, Bus 0, Device 0, Function 0, Error status 0x00000000)

0017 Caution        11:12  01/31/2018 11:12  01/31/2018 0001
LOG: POST Error: 1720-S.M.A.R.T. Hard Drive Detects Imminent Failure

0018 Critical       09:46  02/01/2018 09:46  02/01/2018 0001
LOG: Uncorrectable PCI Express Error (Embedded device, Bus 0, Device 0, Function 0, Error status 0x00000000)

0019 Caution        10:01  02/01/2018 10:01  02/01/2018 0001
LOG: POST Error: 1720-S.M.A.R.T. Hard Drive Detects Imminent Failure

0020 Critical       15:51  04/13/2018 15:51  04/13/2018 0001
LOG: Uncorrectable PCI Express Error (Embedded device, Bus 0, Device 0, Function 0, Error status 0x00000000)

0021 Caution        00:00  00/00/0000 00:00  00/00/0000 0001
LOG: System Power Supply: General Failure (Power Supply 1)

0022 Caution        05:09  04/15/2018 05:09  04/15/2018 0001
LOG: POST Error: 1615-Power Supply Failure or Power Supply Unplugged in Bay 1

0023 Caution        05:10  04/15/2018 05:10  04/15/2018 0001
LOG: POST Error: 1720-S.M.A.R.T. Hard Drive Detects Imminent Failure

0024 Critical       12:37  07/18/2018 12:37  07/18/2018 0001
	  LOG: Uncorrectable PCI Express Error (Embedded device, Bus 0, Device 0, Function 0, Error status 0x00000000)
</code>

==== hpasmcli ====
<sxh bash; gutter: false>
hpasmcli
</sxh>

<code>
HPE management CLI for Linux (v2.0)
Copyright 2015 Hewlett Packard Enterprise Development LP.

--------------------------------------------------------------------------
NOTE: Some hpasmcli commands may not be supported on all Proliant servers.
      Type 'help' to get a list of all top level commands.
--------------------------------------------------------------------------

hpasmcli> show dimm

DIMM Configuration
------------------
Processor #:                     1
Module #:                     1
Present:                      Yes
Form Factor:                  9h
Memory Type:                  DDR3(18h)
Size:                         16384 MB
Speed:                        1333 MHz
Supports Lock Step:           No
Configured for Lock Step:     No
Status:                       Ok

Processor #:                     1
Module #:                     2
Present:                      Yes
Form Factor:                  9h
Memory Type:                  DDR3(18h)
Size:                         16384 MB
Speed:                        1333 MHz
Supports Lock Step:           No
Configured for Lock Step:     No
Status:                       Ok

Processor #:                     1
Module #:                     3
Present:                      Yes
Form Factor:                  9h
Memory Type:                  DDR3(18h)
Size:                         16384 MB
Speed:                        1333 MHz
Supports Lock Step:           No
Configured for Lock Step:     No
Status:                       Ok

Processor #:                     1
Module #:                     4
Present:                      Yes
Form Factor:                  9h
Memory Type:                  DDR3(18h)
Size:                         16384 MB
Speed:                        1333 MHz
Supports Lock Step:           No
Configured for Lock Step:     No
Status:                       Ok

Processor #:                     1
Module #:                     5
Present:                      Yes
Form Factor:                  9h
Memory Type:                  DDR3(18h)
Size:                         16384 MB
Speed:                        1333 MHz
Supports Lock Step:           No
Configured for Lock Step:     No
Status:                       Ok

Processor #:                     1
Module #:                     6
Present:                      Yes
Form Factor:                  9h
Memory Type:                  DDR3(18h)
Size:                         16384 MB
Speed:                        1333 MHz
Supports Lock Step:           No
Configured for Lock Step:     No
Status:                       Ok

Processor #:                     1
Module #:                     7
Present:                      Yes
Form Factor:                  9h
Memory Type:                  DDR3(18h)
Size:                         16384 MB
Speed:                        1333 MHz
Supports Lock Step:           No
Configured for Lock Step:     No
Status:                       Ok

Processor #:                     1
Module #:                     8
Present:                      Yes
Form Factor:                  9h
Memory Type:                  DDR3(18h)
Size:                         16384 MB
Speed:                        1333 MHz
Supports Lock Step:           No
Configured for Lock Step:     No
Status:                       Ok

Processor #:                     2
Module #:                     1
Present:                      Yes
Form Factor:                  9h
Memory Type:                  DDR3(18h)
Size:                         16384 MB
Speed:                        1333 MHz
Supports Lock Step:           No
Configured for Lock Step:     No
Status:                       Ok

Processor #:                     2
Module #:                     2
Present:                      Yes
Form Factor:                  9h
Memory Type:                  DDR3(18h)
Size:                         16384 MB
Speed:                        1333 MHz
Supports Lock Step:           No
Configured for Lock Step:     No
Status:                       Ok

Processor #:                     2
Module #:                     3
Present:                      Yes
Form Factor:                  9h
Memory Type:                  DDR3(18h)
Size:                         16384 MB
Speed:                        1333 MHz
Supports Lock Step:           No
Configured for Lock Step:     No
Status:                       Ok

Processor #:                     2
Module #:                     4
Present:                      Yes
Form Factor:                  9h
Memory Type:                  DDR3(18h)
Size:                         16384 MB
Speed:                        1333 MHz
Supports Lock Step:           No
Configured for Lock Step:     No
Status:                       Ok

Processor #:                     2
Module #:                     5
Present:                      Yes
Form Factor:                  9h
Memory Type:                  DDR3(18h)
Size:                         16384 MB
Speed:                        1333 MHz
Supports Lock Step:           No
Configured for Lock Step:     No
Status:                       Ok

Processor #:                     2
Module #:                     6
Present:                      Yes
Form Factor:                  9h
Memory Type:                  DDR3(18h)
Size:                         16384 MB
Speed:                        1333 MHz
Supports Lock Step:           No
Configured for Lock Step:     No
Status:                       Ok

Processor #:                     2
Module #:                     7
Present:                      Yes
Form Factor:                  9h
Memory Type:                  DDR3(18h)
Size:                         16384 MB
Speed:                        1333 MHz
Supports Lock Step:           No
Configured for Lock Step:     No
Status:                       Ok

Processor #:                     2
Module #:                     8
Present:                      Yes
Form Factor:                  9h
Memory Type:                  DDR3(18h)
Size:                         16384 MB
Speed:                        1333 MHz
Supports Lock Step:           No
Configured for Lock Step:     No
Status:                       Ok

hpasmcli> quit
</code>

----
~~DISCUSSION~~