Majority of the applications these days are deployed on (Debian / Redhat) Linux Operating System as the Base OS.
I Would like to share some generic tuning that can be done before deploying any application on it.
|Index||Component||Question / Test / Reason|
|These are some checks to validate the network setup.|
|[||Network||Are the switches redundant?
Unplug one switch.
|Network||Is the cabling redundant?
|Network||Is the network full-duplex?
Double check setup.
|Network adapter (NIC) Tuning|
|It is recommended to consult with the network adapter provider on recommended Linux TCP/IP settings for optimal performance and stability on Linux.
There are also quite a few TCP/IP tuning source on the Internet such as http://fasterdata.es.net/TCP-tuning/linux.html
|NIC||Are the NIC fault-tolerant (aka. auto-port negotiation)?
Pull cables and/or disable network adapter.
|NIC||Set the transmission queue depth to at least 1000.
Performance and stability (packet drops).
|NIC||Enable TCP/IP offloading (aka. Generic Segment Offloading (GSO)) which was added in kernel 2.6.18
Note: I recommend enabling all supported TCP/IP offloading capabilities on and EMS host to free CPU resources.
|NIC||Enable Interrupt Coalescence (aka. Interrupt Moderation or Interrupt Blanking).
Note: The configuration is system dependant but the goal is to reduce the number of interrupts per second at the ‘cost’ of slightly increased latency.
|TCP/IP Buffer Tuning|
|For a low latency or high throughput messaging system TCP/IP buffer tuning is important.
Thus instead of tuning the defaults values one should rather check if the settings (sysctl –a) provide large enough buffer The values can be changed via the command sysctrl –w <name> <value>.
The below values and comments were taken from TIBCO support FAQ1-6YOAA) and serve as a guideline towards “large enough” buffers, i.e. if your system configuration has lower values it is suggested to raise them to below values.
|TCP/IP||Maximum OS receive buffer size for all connection types.
|TCP/IP||Default OS receive buffer size for all connection types.
|TCP/IP||Maximum OS send buffer size for all connection types.
|TCP/IP||Default OS send buffer size for all types of connections.
|TCP/IP||Enable/Disable TCP/IP window scaling enabled?
|TCP/IP||TCP auto-tuning setting:
Default: 1966087 262144 393216
The tcp_mem variable defines how the TCP stack should behave when it comes to memory usage:
– The first value specified in the tcp_mem variable tells the kernel the low threshold. Below this point, the TCP stack does not bother at all about putting any pressure on the memory usage by different TCP sockets.
– The second value tells the kernel at which point to start pressuring memory usage down.
– The final value tells the kernel how many memory pages it may use maximally. If this value is reached, TCP streams and packets start getting dropped until we reach a lower memory usage again. This value includes all TCP sockets currently in use.
|TCP/IP||TCP auto-tuning (receive) setting:
Default: 4096 87380 4194304
The tcp_rmem variable defines how the TCP stack should behave when it comes to memory usage:
– The first value tells the kernel the minimum receive buffer for each TCP connection, and this buffer is always allocated to a TCP socket, even under high pressure on the system.
– The second value specified tells the kernel the default receive buffer allocated for each TCP socket. This value overrides the /proc/sys/net/core/rmem_default value used by other protocols.
– The third and last value specified in this variable specifies the maximum receive buffer that can be allocated for a TCP socket.”
|TCP/IP||TCP auto-tuning (send) setting:
Default: 4096 87380 4194304
This variable takes three different values which hold information on how much TCP send buffer memory space each TCP socket has to use. Every TCP socket has this much buffer space to use before the buffer is filled up. Each of the three values are used under different conditions:
– The first value in this variable tells the minimum TCP send buffer space available for a single TCP socket.
– The second value in the variable tells us the default buffer space allowed for a single TCP socket to use.
– The third value tells the kernel the maximum TCP send buffer space.”
|TCP/IP||This will ensure that immediately subsequent connections use these values.
|TCP Keep Alive|
|In order to detect ungracefully closed sockets either the TCP keep-alive comes into play or the EMS client-server heartbeat. Which setup or which combination of parameters works better depends on the requirements and test scenarios.
As the EMS daemon does not explicitly enables TCP keep-alive on sockets the TCP keep-alive setting (net.ipv4.tcp_keepalive_intvl, net.ipv4.tcp_keepalive_probes, net.ipv4.tcp_keepalive_time) do not play a role.
|TCP||How may times to retry before killing alive TCP connection. RFC1122 says that the limit should be longer than 100 sec. It is too small number. Default value 15 corresponds to 13-30 minutes depending on retransmission timeout (RTO).
Fault-Tolerance (EMS failover)
The default (15) is often considered too high and a value of 3 is often felt as too ‘edgy’ thus customer testing should establish a good value in the range between 4 and 10.
|Linux System Settings|
|System limits (ulimit) are used to establish boundaries for resource utilization by individual processes and thus protect the system and other processes. A too high or unlimited value provides zero protection but a too low value could hinder growth or cause premature errors.|
|Linux||Is the number of file descriptor at least 4096
Note: It is expected that the number of connected clients and thus the number of connections is going to increase over time and this setting allows for greater growth and also provides a greater safety room should some application have a connection leak. Also note that the number of open connection can decrease system performance due to the way the OS handles the select() API. Thus care should be taken if the number of connected clients increases over time that all SLA are still met.
|Linux||Limit maximum file size for EMS to 2/5 of the disk space if the disk space is shared between EMS servers.
Robustness: Contain the damage of a very large backlog.
|Linux||Consider limiting the maximum data segment size for EMS daemons in order to avoid one EMS monopolizing all available memory.
Robustness: Contain the damage of a very large backlog.
Note: It should be tested if such a limit operates well with (triggers) the EMS reserved memory mode.
|Linux||Limit number of child processes to X to contain rouge application (shell bomb)
Robustness: Contain the damage a rogue application can do.
This is just an example of a Linux system setting that is unrelated to TIBCO products. It is recommended to consult with Linux experts for recommended settings.
|Linux Virtual Memory Management|
|There are a couple of virtual memory related setting that play a role on how likely Linux swaps out memory pages and how Linux reacts to out-of-memory conditions. Both aspects are not important under “normal” operation conditions but are very important under memory pressure and thus the system’s stability under stress.
A server running EAI software and even more a server running a messaging server like EMS should rarely have to resort to swap space for obvious performance reasons. However considerations due to malloc/sbrk high-water-mark behavior, the behavior of the different over-commit strategies and the price of storage lead to above recommendation: Even with below tuning of EMS server towards larger malloc regions the reality is that the EMS daemon is still subject to the sbrk() high-water-mark and is potentially allocation a lot of memory pages that could be swapped out without impacting performance. Of course the EMS server instance must eventually be bounced but the recommendation in this section aim to provide operations with a larger window to schedule the maintenance.
As theses values operate as a bundle they must be changed together or any variation must be well understood.
|Linux||Swap-Space: 1.5 to 2x the physical RAM (24-32 GB )
Logical-Partition: One of the first ones but after the EMS disk storage and application checkpoint files.
Physical-Partition: Use a different physical partition than the one used for storage files, logging or application checkpoints to avoid competing disk IO.
|Linux||Committing virtual memory:
$ cat /proc/sys/vm/overcommit_memory
Note: The recommended setting uses a new heuristic that only commits as much memory as available, where available is defined as swap-space plus a portion of RAM. The portion of RAM is defined in the overcommit_ratio. See also: http://www.mjmwired.net/kernel/Documentation/vm/overcommit-accounting and http://www.centos.org/docs/5/html/5.2/Deployment_Guide/s3-proc-sys-vm.html
|Linux||Committing virtual memory II:
$ cat /proc/sys/vm/overcommit_ratio
Note: This value specifies how much percent of the RAM Linux will add to the swap space in order to calculate the “available” memory. The more the swap space exceeds the physical RAM the lower values might be chosen. See also: http://www.linuxinsight.com/proc_sys_vm_overcommit_ratio.html
$ cat /proc/sys/vm/swappiness
Note: The swappiness defines how likely memory pages will be swapped in order to make room for the file buffer cache.
Generally speaking an enterprise server should not need to swap out pages in order to make room for the file buffer cache or other processes which would favor a setting of 0.
On the other hand it is likely that applications have at least some memory pages that almost never get referenced again and swapping them out is a good thing.
|Linux||Exclude essential processes (Application) from being killed by the out-of-memory (OOM) daemon.
Echo “-17: > /proc/<pid>/oom_adj
Note: With any configuration but overcommit_memory=2 and overcommit_ratio=0 the Linux Virtual Memory Management can commit more memory than available. If then the memory must be provided Linux engages the out-of-memory kill daemon to kill process based on “badness”. In order to exclude essential processes from being killed one can set their oom_adj to -17.
|Linux 32bit||Low memory area – 32bit Linux only
|Linux CPU Tuning (Processor Binding & Priorities)|
|This level of tuning is seldom required for Any Application solution. The tuning options are mentioned in case there is a need to go an extra mile.|
Recommendation: Leave default
The default on Linux is IRQ balancing across multiple CPU and Linux offers two solutions in that real (kernel and daemon) of which only one should be enabled at most.
|Linux||Process Base Priority
Recommendation: Leave default
Note: The process base priority is determined by the user running the process instance and thus running processes as root (chown and set sticky bit) increases the processes base priority.
And a root user can further increase the priority of Application to real-time scheduling which can further improve performance particularly in terms of jitter. However in 2008 we observed that doing so actually decreased the performance of EMS in terms of number of messages per second. That issue was researched with Novell at that time but I am not sure of its outcome.
|Linux||Foreground and Background Processes
Note: Linux assigns foreground processes a better base priority than background processes but if it really matters and if so then how to change start-up scripts is a to-be-determined.
Recommendation: Don’t bother
Note: Linux allows defining a processor set and limiting a process to only use cores from that processor set. This can be used to increase cache hits and cap the CPU resource for a particular process instance.
If larger memory regions are allocated the malloc() in the Linux glibc library uses mmap() instead of sbrk() to provide the memory pages to the process.
The memory mapped files (mmap()) are better in the way how they release memory back to the OS and thus the high-water-mark effect is avoided for these regions.
Good Morning To All My TECH Ghettos,
Today ima show ya’ll a fuckin command to delete all files except a pattern,
ya’ll can use it in a script or even commandline ……. Life gets easy as Fuck !!!!!!!!
find . -type f ! -name ‘<pattern>’ -delete
A Live Example
After the following Command
find . -type f ! -name ‘*.gz’ -delete
Avery serious security problem has been found in the Linux kernel called “The Stack Clash.” It can be exploited by attackers to corrupt memory and execute arbitrary code. An attacker could leverage this with another vulnerability to execute arbitrary code and gain administrative/root account privileges. How do I fix this problem on Linux?
The Qualys Research Labs discovered various problems in the dynamic linker of the GNU C Library (CVE-2017-1000366) which allow local privilege escalation by clashing the stack including Linux kernel. This bug affects Linux, OpenBSD, NetBSD, FreeBSD and Solaris, on i386 and amd64. It can be exploited by attackers to corrupt memory and execute arbitrary code.
What is CVE-2017-1000364 bug?
A flaw was found in the way memory was being allocated on the stack for user space binaries. If heap (or different memory region) and stack memory regions were adjacent to each other, an attacker could use this flaw to jump over the stack guard gap, cause controlled memory corruption on process stack or the adjacent memory region, and thus increase their privileges on the system. This is a kernel-side mitigation which increases the stack guard gap size from one page to 1 MiB to make successful exploitation of this issue more difficult.
Each program running on a computer uses a special memory region called the stack. This memory region is special because it grows automatically when the program needs more stack memory. But if it grows too much and gets too close to another memory region, the program may confuse the stack with the other memory region. An attacker can exploit this confusion to overwrite the stack with the other memory region, or the other way around.
A list of affected Linux distros
- Red Hat Enterprise Linux Server 5.x
- Red Hat Enterprise Linux Server 6.x
- Red Hat Enterprise Linux Server 7.x
- CentOS Linux Server 5.x
- CentOS Linux Server 6.x
- CentOS Linux Server 7.x
- Oracle Enterprise Linux Server 5.x
- Oracle Enterprise Linux Server 6.x
- Oracle Enterprise Linux Server 7.x
- Ubuntu 17.10
- Ubuntu 17.04
- Ubuntu 16.10
- Ubuntu 16.04 LTS
- Ubuntu 12.04 ESM (Precise Pangolin)
- Debian 9 stretch
- Debian 8 jessie
- Debian 7 wheezy
- Debian unstable
- SUSE Linux Enterprise Desktop 12 SP2
- SUSE Linux Enterprise High Availability 12 SP2
- SUSE Linux Enterprise Live Patching 12
- SUSE Linux Enterprise Module for Public Cloud 12
- SUSE Linux Enterprise Build System Kit 12 SP2
- SUSE Openstack Cloud Magnum Orchestration 7
- SUSE Linux Enterprise Server 11 SP3-LTSS
- SUSE Linux Enterprise Server 11 SP4
- SUSE Linux Enterprise Server 12 SP1-LTSS
- SUSE Linux Enterprise Server 12 SP2
- SUSE Linux Enterprise Server for Raspberry Pi 12 SP2
Do I need to reboot my box?
Yes, as most services depends upon the dynamic linker of the GNU C Library and kernel itself needs to be reloaded in memory.
How do I fix CVE-2017-1000364 on Linux?
Type the commands as per your Linux distro. You need to reboot the box. Before you apply patch, note down your current kernel version:
$ uname -a
$ uname -mrs
Linux 4.4.0-78-generic x86_64
Debian or Ubuntu Linux
Type the following apt command/apt-get command to apply updates:
$ sudo apt-get update && sudo apt-get upgrade && sudo apt-get dist-upgrade
Reading package lists... Done Building dependency tree Reading state information... Done Calculating upgrade... Done The following packages will be upgraded: libc-bin libc-dev-bin libc-l10n libc6 libc6-dev libc6-i386 linux-compiler-gcc-6-x86 linux-headers-4.9.0-3-amd64 linux-headers-4.9.0-3-common linux-image-4.9.0-3-amd64 linux-kbuild-4.9 linux-libc-dev locales multiarch-support 14 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. Need to get 0 B/62.0 MB of archives. After this operation, 4,096 B of additional disk space will be used. Do you want to continue? [Y/n] y Reading changelogs... Done Preconfiguring packages ... (Reading database ... 115123 files and directories currently installed.) Preparing to unpack .../libc6-i386_2.24-11+deb9u1_amd64.deb ... Unpacking libc6-i386 (2.24-11+deb9u1) over (2.24-11) ... Preparing to unpack .../libc6-dev_2.24-11+deb9u1_amd64.deb ... Unpacking libc6-dev:amd64 (2.24-11+deb9u1) over (2.24-11) ... Preparing to unpack .../libc-dev-bin_2.24-11+deb9u1_amd64.deb ... Unpacking libc-dev-bin (2.24-11+deb9u1) over (2.24-11) ... Preparing to unpack .../linux-libc-dev_4.9.30-2+deb9u1_amd64.deb ... Unpacking linux-libc-dev:amd64 (4.9.30-2+deb9u1) over (4.9.30-2) ... Preparing to unpack .../libc6_2.24-11+deb9u1_amd64.deb ... Unpacking libc6:amd64 (2.24-11+deb9u1) over (2.24-11) ... Setting up libc6:amd64 (2.24-11+deb9u1) ... (Reading database ... 115123 files and directories currently installed.) Preparing to unpack .../libc-bin_2.24-11+deb9u1_amd64.deb ... Unpacking libc-bin (2.24-11+deb9u1) over (2.24-11) ... Setting up libc-bin (2.24-11+deb9u1) ... (Reading database ... 115123 files and directories currently installed.) Preparing to unpack .../multiarch-support_2.24-11+deb9u1_amd64.deb ... Unpacking multiarch-support (2.24-11+deb9u1) over (2.24-11) ... Setting up multiarch-support (2.24-11+deb9u1) ... (Reading database ... 115123 files and directories currently installed.) Preparing to unpack .../0-libc-l10n_2.24-11+deb9u1_all.deb ... Unpacking libc-l10n (2.24-11+deb9u1) over (2.24-11) ... Preparing to unpack .../1-locales_2.24-11+deb9u1_all.deb ... Unpacking locales (2.24-11+deb9u1) over (2.24-11) ... Preparing to unpack .../2-linux-compiler-gcc-6-x86_4.9.30-2+deb9u1_amd64.deb ... Unpacking linux-compiler-gcc-6-x86 (4.9.30-2+deb9u1) over (4.9.30-2) ... Preparing to unpack .../3-linux-headers-4.9.0-3-amd64_4.9.30-2+deb9u1_amd64.deb ... Unpacking linux-headers-4.9.0-3-amd64 (4.9.30-2+deb9u1) over (4.9.30-2) ... Preparing to unpack .../4-linux-headers-4.9.0-3-common_4.9.30-2+deb9u1_all.deb ... Unpacking linux-headers-4.9.0-3-common (4.9.30-2+deb9u1) over (4.9.30-2) ... Preparing to unpack .../5-linux-kbuild-4.9_4.9.30-2+deb9u1_amd64.deb ... Unpacking linux-kbuild-4.9 (4.9.30-2+deb9u1) over (4.9.30-2) ... Preparing to unpack .../6-linux-image-4.9.0-3-amd64_4.9.30-2+deb9u1_amd64.deb ... Unpacking linux-image-4.9.0-3-amd64 (4.9.30-2+deb9u1) over (4.9.30-2) ... Setting up linux-libc-dev:amd64 (4.9.30-2+deb9u1) ... Setting up linux-headers-4.9.0-3-common (4.9.30-2+deb9u1) ... Setting up libc6-i386 (2.24-11+deb9u1) ... Setting up linux-compiler-gcc-6-x86 (4.9.30-2+deb9u1) ... Setting up linux-kbuild-4.9 (4.9.30-2+deb9u1) ... Setting up libc-l10n (2.24-11+deb9u1) ... Processing triggers for man-db (184.108.40.206-2) ... Setting up libc-dev-bin (2.24-11+deb9u1) ... Setting up linux-image-4.9.0-3-amd64 (4.9.30-2+deb9u1) ... /etc/kernel/postinst.d/initramfs-tools: update-initramfs: Generating /boot/initrd.img-4.9.0-3-amd64 cryptsetup: WARNING: failed to detect canonical device of /dev/md0 cryptsetup: WARNING: could not determine root device from /etc/fstab W: initramfs-tools configuration sets RESUME=UUID=054b217a-306b-4c18-b0bf-0ed85af6c6e1 W: but no matching swap device is available. I: The initramfs will attempt to resume from /dev/md1p1 I: (UUID=bf72f3d4-3be4-4f68-8aae-4edfe5431670) I: Set the RESUME variable to override this. /etc/kernel/postinst.d/zz-update-grub: Searching for GRUB installation directory ... found: /boot/grub Searching for default file ... found: /boot/grub/default Testing for an existing GRUB menu.lst file ... found: /boot/grub/menu.lst Searching for splash image ... none found, skipping ... Found kernel: /boot/vmlinuz-4.9.0-3-amd64 Found kernel: /boot/vmlinuz-3.16.0-4-amd64 Updating /boot/grub/menu.lst ... done Setting up libc6-dev:amd64 (2.24-11+deb9u1) ... Setting up locales (2.24-11+deb9u1) ... Generating locales (this might take a while)... en_IN.UTF-8... done Generation complete. Setting up linux-headers-4.9.0-3-amd64 (4.9.30-2+deb9u1) ... Processing triggers for libc-bin (2.24-11+deb9u1) ...
Reboot your server/desktop using reboot command:
$ sudo reboot
Type the following yum command:
$ sudo yum update
$ sudo reboot
Type the following dnf command:
$ sudo dnf update
$ sudo reboot
Suse Enterprise Linux or Opensuse Linux
Type the following zypper command:
$ sudo zypper patch
$ sudo reboot
SUSE OpenStack Cloud 6
$ sudo zypper in -t patch SUSE-OpenStack-Cloud-6-2017-996=1
$ sudo reboot
SUSE Linux Enterprise Server for SAP 12-SP1
$ sudo zypper in -t patch SUSE-SLE-SAP-12-SP1-2017-996=1
$ sudo reboot
SUSE Linux Enterprise Server 12-SP1-LTSS
$ sudo zypper in -t patch SUSE-SLE-SERVER-12-SP1-2017-996=1
$ sudo reboot
SUSE Linux Enterprise Module for Public Cloud 12
$ sudo zypper in -t patch SUSE-SLE-Module-Public-Cloud-12-2017-996=1
$ sudo reboot
You need to make sure your version number changed after issuing reboot command
$ uname -a
$ uname -r
$ uname -mrs
Linux 4.4.0-81-generic x86_64
Cpustat is a powerful system performance measure program for Linux, written using Go programming language. It attempts to reveal CPU utilization and saturation in an effective way, using The Utilization Saturation and Errors (USE) Method (a methodology for analyzing the performance of any system).
It extracts higher frequency samples of every process being executed on the system and then summarizes these samples at a lower frequency. For instance, it can measure every process every 200ms and summarize these samples every 5 seconds, including min/average/max values for certain metrics.
Cpustat outputs data in two possible ways: a pure text list of the summary interval and a colorful scrolling dashboard of each sample.
How to Install Cpustat in Linux
You must have Go (GoLang) installed on your Linux system in order to use cpustat, click on the link below to follow the GoLang installation steps that is if you do not have it installed:
- Install GoLang (Go Programming Language) in Linux
Once you have installed Go, type the go get command below to install it, this command will install the cpustat binary in your GOBIN variable:
# go get github.com/uber-common/cpustat
How to Use Cpustat in Linux
When the installation process completes, run cpustat as follows with root privileges using the sudo command that is if your controlling the system as a non-root user, otherwise you’ll get the error as shown:
$ $GOBIN/cpustat This program uses the netlink taskstats interface, so it must be run as root.
Note: To run cpustat as well as all other Go programs you have installed on your system like any other commands, include GOBIN variable in your PATH environment variable. Open the link below to learn how to set the PATH variable in Linux.
This is how cpustat works; the
/proc directory is queried to get the current list of process IDs for every interval, and:
- for each PID, read /proc/pid/stat, then compute difference from previous sample.
- in case it’s a new PID, read /proc/pid/cmdline.
- for each PID, send a netlink message to fetch the taskstats, compute difference from previous sample.
- fetch /proc/stat to get the overall system stats.
Again, each sleep interval is adjusted to account for the amount of time consumed fetching all of these stats. Furthermore, each sample also records the time it took to scale each measurement by the actual elapsed time between samples. This attempts to account for delays in cpustat itself.
When run without any arguments, cpustat will display the following by default: sampling interval: 200ms, summary interval: 2s (10 samples), showing top 10 procs, user filter: all, pid filter: all as shown in the screenshot below:
$ sudo $GOBIN/cpustat
From the output above, the following are the meanings of the system-wide summary metrics displayed before the fields:
- usr – min/avg/max user mode run time as a percentage of a CPU.
- sys – min/avg/max system mode run time as a percentage of a CPU.
- nice – min/avg/max user mode low priority run time as a percentage of a CPU.
- idle – min/avg/max user mode run time as a percentage of a CPU.
- iowait – min/avg/max delay time waiting for disk IO.
- prun – min/avg/max count of processes in a runnable state (same as load average).
- pblock – min/avg/max count of processes blocked on disk IO.
- pstart – number of processes/threads started in this summary interval.
Still from the output above, for a given process, the different columns mean:
- name – common process name from /proc/pid/stat or /proc/pid/cmdline.
- pid – process id, also referred to as “tgid”.
- min – lowest sample of user+system time for the pid, measured from /proc/pid/stat. Scale is a percentage of a CPU.
- max – highest sample of user+system time for this pid, also measured from /proc/pid/stat.
- usr – average user time for the pid over the summary period, measured from /proc/pid/stat.
- sys – average system time for the pid over the summary period, measured from /proc/pid/stat.
- nice – indicates current “nice” value for the process, measured from /proc/pid/stat. Higher means “nicer”.
- runq – time the process and all of its threads spent runnable but waiting to run, measured from taskstats via netlink. Scale is a percentage of a CPU.
- iow – time the process and all of its threads spent blocked by disk IO, measured from taskstats via netlink. Scale is a percentage of a CPU, averaged over the summary interval.
- swap – time the process and all of its threads spent waiting to be swapped in, measured from taskstats via netlink. Scale is a percentage of a CPU, averaged over the summary interval.
- vcx and icx – total number of voluntary context switches by the process and all of its threads over the summary interval, measured from taskstats via netlink.
- rss – current RSS value fetched from /proc/pid/stat. It is the amount of memory this process is using.
- ctime – sum of user+sys CPU time consumed by waited for children that exited during this summary interval, measured from /proc/pid/stat.
Note that long running child processes can often confuse this measurement, because the time is reported only when the child process exits. However, this is useful for measuring the impact of frequent cron jobs and health checks where the CPU time is often consumed by many child processes.
- thrd – number of threads at the end of the summary interval, measured from /proc/pid/stat.
- sam – number of samples for this process included in the summary interval. Processes that have recently started or exited may have been visible for fewer samples than the summary interval.
The following command displays the top 10 root user processes running on the system:
$ sudo $GOBIN/cpustat -u root
To display output in a fancy terminal mode, use the
-t flag as follows:
$ sudo $GOBIN/cpustat -u roo -t
To view the top x number of processes (the default is 10), you can use the
-n flag, the following command shows the top 20 Linux processes running on the system:
$ sudo $GOBIN/cpustat -n 20
You can also write CPU profile to a file using the
-cpuprofile option as follows and then use the cat command to view the file:
$ sudo $GOBIN/cpustat -cpuprofile cpuprof.txt $ cat cpuprof.txt
To display help info, use the
-h flag as follows:
$ sudo $GOBIN/cpustat -h
Find additional info from the cpustat Github Repository: https://github.com/uber-common/cpustat
There is a serious vulnerability in sudo command that grants root access to anyone with a shell account. It works on SELinux enabled systems such as CentOS/RHEL and others too. A local user with privileges to execute commands via sudo could use this flaw to escalate their privileges to root. Patch your system as soon as possible. Run yum command: Run dnf command: Run zypper command: Run pacman command: Run apk command: Run upgradepkg command: Run emerge command:
It was discovered that Sudo did not properly parse the contents of /proc/[pid]/stat when attempting to determine its controlling tty. A local attacker in some configurations could possibly use this to overwrite any file on the filesystem, bypassing intended permissions or gain root shell.
From the description
A list of affected Linux distro
How do I patch sudo on Debian/Ubuntu Linux server?
How do I patch sudo on CentOS/RHEL/Scientific/Oracle Linux server?
$ sudo yum update
How do I patch sudo on Fedora Linux server?
$ sudo dnf update
How do I patch sudo on Suse/OpenSUSE Linux server?
$ sudo zypper update
How do I patch sudo on Arch Linux server?
$ sudo pacman -Syu
How do I patch sudo on Alpine Linux server?
# apk update && apk upgrade
How do I patch sudo on Slackware Linux server?
# upgradepkg sudo-1.8.20p1-i586-1_slack14.2.txz
How do I patch sudo on Gentoo Linux server?
# emerge --sync
# emerge --ask --oneshot --verbose ">=app-admin/sudo-1.8.20_p1"
There is a serious vulnerability in sudo command that grants root access to anyone with a shell account. It works on SELinux enabled systems such as CentOS/RHEL and others too. A local user with privileges to execute commands via sudo could use this flaw to escalate their privileges to root. Patch your system as soon as possible.
Run yum command:
Run dnf command:
Run zypper command:
Run pacman command:
Run apk command:
Run upgradepkg command:
Run emerge command:
Impermanence, also called Anicca or Anitya, is one of the essential doctrines and a part of three marks of existence in Buddhism The doctrine asserts that all of conditioned existence, without exception, is “transient, evanescent, inconstant”
On Linux, the root of all randomness is something called the kernel entropy pool. This is a large (4,096 bit) number kept privately in the kernel’s memory. There are 24096 possibilities for this number so it can contain up to 4,096 bits of entropy. There is one caveat – the kernel needs to be able to fill that memory from a source with 4,096 bits of entropy. And that’s the hard part: finding that much randomness.
The entropy pool is used in two ways: random numbers are generated from it and it is replenished with entropy by the kernel. When random numbers are generated from the pool the entropy of the pool is diminished (because the person receiving the random number has some information about the pool itself). So as the pool’s entropy diminishes as random numbers are handed out, the pool must be replenished.
Replenishing the pool is called stirring: new sources of entropy are stirred into the mix of bits in the pool.
This is the key to how random number generation works on Linux. If randomness is needed, it’s derived from the entropy pool. When available, other sources of randomness are used to stir the entropy pool and make it less predictable. The details are a little mathematical, but it’s interesting to understand how the Linux random number generator works as the principles and techniques apply to random number generation in other software and systems.
The kernel keeps a rough estimate of the number of bits of entropy in the pool. You can check the value of this estimate through the following command:
A healthy Linux system with a lot of entropy available will have return close to the full 4,096 bits of entropy. If the value returned is less than 200, the system is running low on entropy.
The kernel is watching you
I mentioned that the system takes other sources of randomness and uses this to stir the entropy pool. This is achieved using something called a timestamp.
Most systems have precise internal clocks. Every time that a user interacts with a system, the value of the clock at that time is recorded as a timestamp. Even though the year, month, day and hour are generally guessable, the millisecond and microsecond are not and therefore the timestamp contains some entropy. Timestamps obtained from the user’s mouse and keyboard along with timing information from the network and disk each have different amount of entropy.
How does the entropy found in a timestamp get transferred to the entropy pool? Simple, use math to mix it in. Well, simple if you like math.
Just mix it in
A fundamental property of entropy is that it mixes well. If you take two unrelated random streams and combine them, the new stream cannot have less entropy. Taking a number of low entropy sources and combining them results in a high entropy source.
All that’s needed is the right combination function: a function that can be used to combine two sources of entropy. One of the simplest such functions is the logical exclusive or (XOR). This truth table shows how bits x and y coming from different random streams are combined by the XOR function.
Even if one source of bits does not have much entropy, there is no harm in XORing it into another source. Entropy always increases. In the Linux kernel, a combination of XORs is used to mix timestamps into the main entropy pool.
Generating random numbers
Cryptographic applications require very high entropy. If a 128 bit key is generated with only 64 bits of entropy then it can be guessed in 264 attempts instead of 2128 attempts. That is the difference between needing a thousand computers running for a few years to brute force the key versus needing all the computers ever created running for longer than the history of the universe to do so.
Cryptographic applications require close to one bit of entropy per bit. If the system’s pool has fewer than 4,096 bits of entropy, how does the system return a fully random number? One way to do this is to use a cryptographic hash function.
A cryptographic hash function takes an input of any size and outputs a fixed size number. Changing one bit of the input will change the output completely. Hash functions are good at mixing things together. This mixing property spreads the entropy from the input evenly through the output. If the input has more bits of entropy than the size of the output, the output will be highly random. This is how highly entropic random numbers are derived from the entropy pool.
The hash function used by the Linux kernel is the standard SHA-1 cryptographic hash. By hashing the entire pool and and some additional arithmetic, 160 random bits are created for use by the system. When this happens, the system lowers its estimate of the entropy in the pool accordingly.
Above I said that applying a hash like SHA-1 could be dangerous if there wasn’t enough entropy in the pool. That’s why it’s critical to keep an eye on the available system entropy: if it drops too low the output of the random number generator could have less entropy that it appears to have.
Running out of entropy
One of the dangers of a system is running out of entropy. When the system’s entropy estimate drops to around the 160 bit level, the length of a SHA-1 hash, things get tricky, and how they effect programs and performance depends on which of two Linux random number generators are used.
Linux exposes two interfaces for random data that behave differently when the entropy level is low. They are /dev/random and /dev/urandom. When the entropy pool becomes predictable, both interfaces for requesting random numbers become problematic.
When the entropy level is too low, /dev/random blocks and does not return until the level of entropy in the system is high enough. This guarantees high entropy random numbers. If /dev/random is used in a time-critical service and the system runs low on entropy, the delays could be detrimental to the quality of service.
On the other hand, /dev/urandom does not block. It continues to return the hashed value of its entropy pool even though there is little to no entropy in it. This low-entropy data is not suited for cryptographic use.
The solution to the problem is to simply add more entropy into the system.
Hardware random number generation to the rescue?
Intel’s Ivy Bridge family of processors have an interesting feature called “secure key.” These processors contain a special piece of hardware inside that generates random numbers. The single assembly instruction RDRAND returns allegedly high entropy random data derived on the chip.
It has been suggested that Intel’s hardware number generator may not be fully random. Since it is baked into the silicon, that assertion is hard to audit and verify. As it turns out, even if the numbers generated have some bias, it can still help as long as this is not the only source of randomness in the system. Even if the random number generator itself had a back door, the mixing property of randomness means that it cannot lower the amount of entropy in the pool.
On Linux, if a hardware random number generator is present, the Linux kernel will use the XOR function to mix the output of RDRAND into the hash of the entropy pool. This happens here in the Linux source code (the XOR operator is ^ in C).
Third party entropy generators
Hardware number generation is not available everywhere, and the sources of randomness polled by the Linux kernel itself are somewhat limited. For this situation, a number of third party random number generation tools exist. Examples of these are haveged, which relies on processor cache timing, audio-entropyd and video-entropyd which work by sampling the noise from an external audio or video input device. By mixing these additional sources of locally collected entropy into the Linux entropy pool, the entropy can only go up.