How To Patch and Protect Linux Kernel Stack Clash Vulnerability CVE-2017-1000364 [ 19/June/2017 ]

Avery serious security problem has been found in the Linux kernel called “The Stack Clash.” It can be exploited by attackers to corrupt memory and execute arbitrary code. An attacker could leverage this with another vulnerability to execute arbitrary code and gain administrative/root account privileges. How do I fix this problem on Linux?

the-stack-clash-on-linux-openbsd-netbsd-freebsd-solaris
The Qualys Research Labs discovered various problems in the dynamic linker of the GNU C Library (CVE-2017-1000366) which allow local privilege escalation by clashing the stack including Linux kernel. This bug affects Linux, OpenBSD, NetBSD, FreeBSD and Solaris, on i386 and amd64. It can be exploited by attackers to corrupt memory and execute arbitrary code.

What is CVE-2017-1000364 bug?

From RHN:

A flaw was found in the way memory was being allocated on the stack for user space binaries. If heap (or different memory region) and stack memory regions were adjacent to each other, an attacker could use this flaw to jump over the stack guard gap, cause controlled memory corruption on process stack or the adjacent memory region, and thus increase their privileges on the system. This is a kernel-side mitigation which increases the stack guard gap size from one page to 1 MiB to make successful exploitation of this issue more difficult.

As per the original research post:

Each program running on a computer uses a special memory region called the stack. This memory region is special because it grows automatically when the program needs more stack memory. But if it grows too much and gets too close to another memory region, the program may confuse the stack with the other memory region. An attacker can exploit this confusion to overwrite the stack with the other memory region, or the other way around.

A list of affected Linux distros

  1. Red Hat Enterprise Linux Server 5.x
  2. Red Hat Enterprise Linux Server 6.x
  3. Red Hat Enterprise Linux Server 7.x
  4. CentOS Linux Server 5.x
  5. CentOS Linux Server 6.x
  6. CentOS Linux Server 7.x
  7. Oracle Enterprise Linux Server 5.x
  8. Oracle Enterprise Linux Server 6.x
  9. Oracle Enterprise Linux Server 7.x
  10. Ubuntu 17.10
  11. Ubuntu 17.04
  12. Ubuntu 16.10
  13. Ubuntu 16.04 LTS
  14. Ubuntu 12.04 ESM (Precise Pangolin)
  15. Debian 9 stretch
  16. Debian 8 jessie
  17. Debian 7 wheezy
  18. Debian unstable
  19. SUSE Linux Enterprise Desktop 12 SP2
  20. SUSE Linux Enterprise High Availability 12 SP2
  21. SUSE Linux Enterprise Live Patching 12
  22. SUSE Linux Enterprise Module for Public Cloud 12
  23. SUSE Linux Enterprise Build System Kit 12 SP2
  24. SUSE Openstack Cloud Magnum Orchestration 7
  25. SUSE Linux Enterprise Server 11 SP3-LTSS
  26. SUSE Linux Enterprise Server 11 SP4
  27. SUSE Linux Enterprise Server 12 SP1-LTSS
  28. SUSE Linux Enterprise Server 12 SP2
  29. SUSE Linux Enterprise Server for Raspberry Pi 12 SP2

Do I need to reboot my box?

Yes, as most services depends upon the dynamic linker of the GNU C Library and kernel itself needs to be reloaded in memory.

How do I fix CVE-2017-1000364 on Linux?

Type the commands as per your Linux distro. You need to reboot the box. Before you apply patch, note down your current kernel version:
$ uname -a
$ uname -mrs

Sample outputs:

Linux 4.4.0-78-generic x86_64

Debian or Ubuntu Linux

Type the following apt command/apt-get command to apply updates:
$ sudo apt-get update && sudo apt-get upgrade && sudo apt-get dist-upgrade
Sample outputs:

Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  libc-bin libc-dev-bin libc-l10n libc6 libc6-dev libc6-i386 linux-compiler-gcc-6-x86 linux-headers-4.9.0-3-amd64 linux-headers-4.9.0-3-common linux-image-4.9.0-3-amd64
  linux-kbuild-4.9 linux-libc-dev locales multiarch-support
14 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/62.0 MB of archives.
After this operation, 4,096 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Reading changelogs... Done
Preconfiguring packages ...
(Reading database ... 115123 files and directories currently installed.)
Preparing to unpack .../libc6-i386_2.24-11+deb9u1_amd64.deb ...
Unpacking libc6-i386 (2.24-11+deb9u1) over (2.24-11) ...
Preparing to unpack .../libc6-dev_2.24-11+deb9u1_amd64.deb ...
Unpacking libc6-dev:amd64 (2.24-11+deb9u1) over (2.24-11) ...
Preparing to unpack .../libc-dev-bin_2.24-11+deb9u1_amd64.deb ...
Unpacking libc-dev-bin (2.24-11+deb9u1) over (2.24-11) ...
Preparing to unpack .../linux-libc-dev_4.9.30-2+deb9u1_amd64.deb ...
Unpacking linux-libc-dev:amd64 (4.9.30-2+deb9u1) over (4.9.30-2) ...
Preparing to unpack .../libc6_2.24-11+deb9u1_amd64.deb ...
Unpacking libc6:amd64 (2.24-11+deb9u1) over (2.24-11) ...
Setting up libc6:amd64 (2.24-11+deb9u1) ...
(Reading database ... 115123 files and directories currently installed.)
Preparing to unpack .../libc-bin_2.24-11+deb9u1_amd64.deb ...
Unpacking libc-bin (2.24-11+deb9u1) over (2.24-11) ...
Setting up libc-bin (2.24-11+deb9u1) ...
(Reading database ... 115123 files and directories currently installed.)
Preparing to unpack .../multiarch-support_2.24-11+deb9u1_amd64.deb ...
Unpacking multiarch-support (2.24-11+deb9u1) over (2.24-11) ...
Setting up multiarch-support (2.24-11+deb9u1) ...
(Reading database ... 115123 files and directories currently installed.)
Preparing to unpack .../0-libc-l10n_2.24-11+deb9u1_all.deb ...
Unpacking libc-l10n (2.24-11+deb9u1) over (2.24-11) ...
Preparing to unpack .../1-locales_2.24-11+deb9u1_all.deb ...
Unpacking locales (2.24-11+deb9u1) over (2.24-11) ...
Preparing to unpack .../2-linux-compiler-gcc-6-x86_4.9.30-2+deb9u1_amd64.deb ...
Unpacking linux-compiler-gcc-6-x86 (4.9.30-2+deb9u1) over (4.9.30-2) ...
Preparing to unpack .../3-linux-headers-4.9.0-3-amd64_4.9.30-2+deb9u1_amd64.deb ...
Unpacking linux-headers-4.9.0-3-amd64 (4.9.30-2+deb9u1) over (4.9.30-2) ...
Preparing to unpack .../4-linux-headers-4.9.0-3-common_4.9.30-2+deb9u1_all.deb ...
Unpacking linux-headers-4.9.0-3-common (4.9.30-2+deb9u1) over (4.9.30-2) ...
Preparing to unpack .../5-linux-kbuild-4.9_4.9.30-2+deb9u1_amd64.deb ...
Unpacking linux-kbuild-4.9 (4.9.30-2+deb9u1) over (4.9.30-2) ...
Preparing to unpack .../6-linux-image-4.9.0-3-amd64_4.9.30-2+deb9u1_amd64.deb ...
Unpacking linux-image-4.9.0-3-amd64 (4.9.30-2+deb9u1) over (4.9.30-2) ...
Setting up linux-libc-dev:amd64 (4.9.30-2+deb9u1) ...
Setting up linux-headers-4.9.0-3-common (4.9.30-2+deb9u1) ...
Setting up libc6-i386 (2.24-11+deb9u1) ...
Setting up linux-compiler-gcc-6-x86 (4.9.30-2+deb9u1) ...
Setting up linux-kbuild-4.9 (4.9.30-2+deb9u1) ...
Setting up libc-l10n (2.24-11+deb9u1) ...
Processing triggers for man-db (2.7.6.1-2) ...
Setting up libc-dev-bin (2.24-11+deb9u1) ...
Setting up linux-image-4.9.0-3-amd64 (4.9.30-2+deb9u1) ...
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-4.9.0-3-amd64
cryptsetup: WARNING: failed to detect canonical device of /dev/md0
cryptsetup: WARNING: could not determine root device from /etc/fstab
W: initramfs-tools configuration sets RESUME=UUID=054b217a-306b-4c18-b0bf-0ed85af6c6e1
W: but no matching swap device is available.
I: The initramfs will attempt to resume from /dev/md1p1
I: (UUID=bf72f3d4-3be4-4f68-8aae-4edfe5431670)
I: Set the RESUME variable to override this.
/etc/kernel/postinst.d/zz-update-grub:
Searching for GRUB installation directory ... found: /boot/grub
Searching for default file ... found: /boot/grub/default
Testing for an existing GRUB menu.lst file ... found: /boot/grub/menu.lst
Searching for splash image ... none found, skipping ...
Found kernel: /boot/vmlinuz-4.9.0-3-amd64
Found kernel: /boot/vmlinuz-3.16.0-4-amd64
Updating /boot/grub/menu.lst ... done

Setting up libc6-dev:amd64 (2.24-11+deb9u1) ...
Setting up locales (2.24-11+deb9u1) ...
Generating locales (this might take a while)...
  en_IN.UTF-8... done
Generation complete.
Setting up linux-headers-4.9.0-3-amd64 (4.9.30-2+deb9u1) ...
Processing triggers for libc-bin (2.24-11+deb9u1) ...

Reboot your server/desktop using reboot command:
$ sudo reboot

Oracle/RHEL/CentOS/Scientific Linux

Type the following yum command:
$ sudo yum update
$ sudo reboot

Fedora Linux

Type the following dnf command:
$ sudo dnf update
$ sudo reboot

Suse Enterprise Linux or Opensuse Linux

Type the following zypper command:
$ sudo zypper patch
$ sudo reboot

SUSE OpenStack Cloud 6

$ sudo zypper in -t patch SUSE-OpenStack-Cloud-6-2017-996=1
$ sudo reboot

SUSE Linux Enterprise Server for SAP 12-SP1

$ sudo zypper in -t patch SUSE-SLE-SAP-12-SP1-2017-996=1
$ sudo reboot

SUSE Linux Enterprise Server 12-SP1-LTSS

$ sudo zypper in -t patch SUSE-SLE-SERVER-12-SP1-2017-996=1
$ sudo reboot

SUSE Linux Enterprise Module for Public Cloud 12

$ sudo zypper in -t patch SUSE-SLE-Module-Public-Cloud-12-2017-996=1
$ sudo reboot

Verification

You need to make sure your version number changed after issuing reboot command
$ uname -a
$ uname -r
$ uname -mrs

Sample outputs:

Linux 4.4.0-81-generic x86_64
Advertisements

Cpustat – Monitors CPU Utilization by Running Processes in Linux

Linux security alert: Bug in sudo’s get_process_ttyname() [ CVE-2017-1000367 ]

There is a serious vulnerability in sudo command that grants root access to anyone with a shell account. It works on SELinux enabled systems such as CentOS/RHEL and others too. A local user with privileges to execute commands via sudo could use this flaw to escalate their privileges to root. Patch your system as soon as possible.

It was discovered that Sudo did not properly parse the contents of /proc/[pid]/stat when attempting to determine its controlling tty. A local attacker in some configurations could possibly use this to overwrite any file on the filesystem, bypassing intended permissions or gain root shell.
From the description

We discovered a vulnerability in Sudo’s get_process_ttyname() for Linux:
this function opens “/proc/[pid]/stat” (man proc) and reads the device number of the tty from field 7 (tty_nr). Unfortunately, these fields are space-separated and field 2 (comm, the filename of the command) can
contain spaces (CVE-2017-1000367).

For example, if we execute Sudo through the symlink “./ 1 “, get_process_ttyname() calls sudo_ttyname_dev() to search for the non-existent tty device number “1” in the built-in search_devs[].

Next, sudo_ttyname_dev() calls the function sudo_ttyname_scan() to search for this non-existent tty device number “1” in a breadth-first traversal of “/dev”.

Last, we exploit this function during its traversal of the world-writable “/dev/shm”: through this vulnerability, a local user can pretend that his tty is any character device on the filesystem, and
after two race conditions, he can pretend that his tty is any file on the filesystem.

On an SELinux-enabled system, if a user is Sudoer for a command that does not grant him full root privileges, he can overwrite any file on the filesystem (including root-owned files) with his command’s output,
because relabel_tty() (in src/selinux.c) calls open(O_RDWR|O_NONBLOCK) on his tty and dup2()s it to the command’s stdin, stdout, and stderr. This allows any Sudoer user to obtain full root privileges.

A list of affected Linux distro

  1. Red Hat Enterprise Linux 6 (sudo)
  2. Red Hat Enterprise Linux 7 (sudo)
  3. Red Hat Enterprise Linux Server (v. 5 ELS) (sudo)
  4. Oracle Enterprise Linux 6
  5. Oracle Enterprise Linux 7
  6. Oracle Enterprise Linux Server 5
  7. CentOS Linux 6 (sudo)
  8. CentOS Linux 7 (sudo)
  9. Debian wheezy
  10. Debian jessie
  11. Debian stretch
  12. Debian sid
  13. Ubuntu 17.04
  14. Ubuntu 16.10
  15. Ubuntu 16.04 LTS
  16. Ubuntu 14.04 LTS
  17. SUSE Linux Enterprise Software Development Kit 12-SP2
  18. SUSE Linux Enterprise Server for Raspberry Pi 12-SP2
  19. SUSE Linux Enterprise Server 12-SP2
  20. SUSE Linux Enterprise Desktop 12-SP2
  21. OpenSuse, Slackware, and Gentoo Linux

How do I patch sudo on Debian/Ubuntu Linux server?

To patch Ubuntu/Debian Linux apt-get command or apt command:
$ sudo apt update
$ sudo apt upgrade

How do I patch sudo on CentOS/RHEL/Scientific/Oracle Linux server?

Run yum command:
$ sudo yum update

How do I patch sudo on Fedora Linux server?

Run dnf command:
$ sudo dnf update

How do I patch sudo on Suse/OpenSUSE Linux server?

Run zypper command:
$ sudo zypper update

How do I patch sudo on Arch Linux server?

Run pacman command:
$ sudo pacman -Syu

How do I patch sudo on Alpine Linux server?

Run apk command:
# apk update && apk upgrade

How do I patch sudo on Slackware Linux server?

Run upgradepkg command:
# upgradepkg sudo-1.8.20p1-i586-1_slack14.2.txz

How do I patch sudo on Gentoo Linux server?

Run emerge command:
# emerge --sync
# emerge --ask --oneshot --verbose ">=app-admin/sudo-1.8.20_p1"

Impermanence in Linux – Exclusive (By Hari Iyer)

Impermanence, also called Anicca or Anitya, is one of the essential doctrines and a part of three marks of existence in Buddhism The doctrine asserts that all of conditioned existence, without exception, is “transient, evanescent, inconstant”

On Linux, the root of all randomness is something called the kernel entropy pool. This is a large (4,096 bit) number kept privately in the kernel’s memory. There are 24096 possibilities for this number so it can contain up to 4,096 bits of entropy. There is one caveat – the kernel needs to be able to fill that memory from a source with 4,096 bits of entropy. And that’s the hard part: finding that much randomness.

The entropy pool is used in two ways: random numbers are generated from it and it is replenished with entropy by the kernel. When random numbers are generated from the pool the entropy of the pool is diminished (because the person receiving the random number has some information about the pool itself). So as the pool’s entropy diminishes as random numbers are handed out, the pool must be replenished.

Replenishing the pool is called stirring: new sources of entropy are stirred into the mix of bits in the pool.

This is the key to how random number generation works on Linux. If randomness is needed, it’s derived from the entropy pool. When available, other sources of randomness are used to stir the entropy pool and make it less predictable. The details are a little mathematical, but it’s interesting to understand how the Linux random number generator works as the principles and techniques apply to random number generation in other software and systems.

The kernel keeps a rough estimate of the number of bits of entropy in the pool. You can check the value of this estimate through the following command:

cat /proc/sys/kernel/random/entropy_avail

A healthy Linux system with a lot of entropy available will have return close to the full 4,096 bits of entropy. If the value returned is less than 200, the system is running low on entropy.

The kernel is watching you

I mentioned that the system takes other sources of randomness and uses this to stir the entropy pool. This is achieved using something called a timestamp.

Most systems have precise internal clocks. Every time that a user interacts with a system, the value of the clock at that time is recorded as a timestamp. Even though the year, month, day and hour are generally guessable, the millisecond and microsecond are not and therefore the timestamp contains some entropy. Timestamps obtained from the user’s mouse and keyboard along with timing information from the network and disk each have different amount of entropy.

How does the entropy found in a timestamp get transferred to the entropy pool? Simple, use math to mix it in. Well, simple if you like math.

Just mix it in

A fundamental property of entropy is that it mixes well. If you take two unrelated random streams and combine them, the new stream cannot have less entropy. Taking a number of low entropy sources and combining them results in a high entropy source.

All that’s needed is the right combination function: a function that can be used to combine two sources of entropy. One of the simplest such functions is the logical exclusive or (XOR). This truth table shows how bits x and y coming from different random streams are combined by the XOR function.

Even if one source of bits does not have much entropy, there is no harm in XORing it into another source. Entropy always increases. In the Linux kernel, a combination of XORs is used to mix timestamps into the main entropy pool.

Generating random numbers

Cryptographic applications require very high entropy. If a 128 bit key is generated with only 64 bits of entropy then it can be guessed in 264 attempts instead of 2128 attempts. That is the difference between needing a thousand computers running for a few years to brute force the key versus needing all the computers ever created running for longer than the history of the universe to do so.

Cryptographic applications require close to one bit of entropy per bit. If the system’s pool has fewer than 4,096 bits of entropy, how does the system return a fully random number? One way to do this is to use a cryptographic hash function.

A cryptographic hash function takes an input of any size and outputs a fixed size number. Changing one bit of the input will change the output completely. Hash functions are good at mixing things together. This mixing property spreads the entropy from the input evenly through the output. If the input has more bits of entropy than the size of the output, the output will be highly random. This is how highly entropic random numbers are derived from the entropy pool.

The hash function used by the Linux kernel is the standard SHA-1 cryptographic hash. By hashing the entire pool and and some additional arithmetic, 160 random bits are created for use by the system. When this happens, the system lowers its estimate of the entropy in the pool accordingly.

Above I said that applying a hash like SHA-1 could be dangerous if there wasn’t enough entropy in the pool. That’s why it’s critical to keep an eye on the available system entropy: if it drops too low the output of the random number generator could have less entropy that it appears to have.

Running out of entropy

One of the dangers of a system is running out of entropy. When the system’s entropy estimate drops to around the 160 bit level, the length of a SHA-1 hash, things get tricky, and how they effect programs and performance depends on which of two Linux random number generators are used.

Linux exposes two interfaces for random data that behave differently when the entropy level is low. They are /dev/random and /dev/urandom. When the entropy pool becomes predictable, both interfaces for requesting random numbers become problematic.

When the entropy level is too low, /dev/random blocks and does not return until the level of entropy in the system is high enough. This guarantees high entropy random numbers. If /dev/random is used in a time-critical service and the system runs low on entropy, the delays could be detrimental to the quality of service.

On the other hand, /dev/urandom does not block. It continues to return the hashed value of its entropy pool even though there is little to no entropy in it. This low-entropy data is not suited for cryptographic use.

The solution to the problem is to simply add more entropy into the system.

Hardware random number generation to the rescue?

Intel’s Ivy Bridge family of processors have an interesting feature called “secure key.” These processors contain a special piece of hardware inside that generates random numbers. The single assembly instruction RDRAND returns allegedly high entropy random data derived on the chip.

It has been suggested that Intel’s hardware number generator may not be fully random. Since it is baked into the silicon, that assertion is hard to audit and verify. As it turns out, even if the numbers generated have some bias, it can still help as long as this is not the only source of randomness in the system. Even if the random number generator itself had a back door, the mixing property of randomness means that it cannot lower the amount of entropy in the pool.

On Linux, if a hardware random number generator is present, the Linux kernel will use the XOR function to mix the output of RDRAND into the hash of the entropy pool. This happens here in the Linux source code (the XOR operator is ^ in C).

Third party entropy generators

Hardware number generation is not available everywhere, and the sources of randomness polled by the Linux kernel itself are somewhat limited. For this situation, a number of third party random number generation tools exist. Examples of these are haveged, which relies on processor cache timing, audio-entropyd and video-entropyd which work by sampling the noise from an external audio or video input device. By mixing these additional sources of locally collected entropy into the Linux entropy pool, the entropy can only go up.

/etc/security/limits.conf file – In A Nutshell

The /etc/security/limits.conf file contains a list line where each line describes a limit for a user in the form of:

<Domain> <type> <item> <shell limit value>

Where:

  • <domain> can be:
    • an user name
    • a group name, with @group syntax
    • the wildcard *, for default entry
    • the wildcard %, can be also used with %group syntax, for maxlogin limit
  • <type> can have the two values:
    • “soft” for enforcing the soft limits (soft is like warning)
    • “hard” for enforcing hard limits (hard is a real max limit)
  • <item> can be one of the following:
    • core – limits the core file size (KB)
  • <shell limit value> can be one of the following:
    • core – limits the core file size (KB)
    • data – max data size (KB)
    • fsize – maximum file size (KB)
    • memlock – max locked-in-memory address space (KB)
    • nofile – Maximum number of open file descriptors
    • rss – max resident set size (KB)
    • stack – max stack size (KB) – Maximum size of the stack segment of the process
    • cpu – max CPU time (MIN)
    • nproc – Maximum number of processes available to a single user
    • as – address space limit
    • maxlogins – max number of logins for this user
    • maxsyslogins – max number of logins on the system
    • priority – the priority to run user process with
    • locks – max number of file locks the user can hold
    • sigpending – max number of pending signals
    • msgqueue – max memory used by POSIX message queues (bytes)
    • nice – max nice priority allowed to raise to
    • rtprio – max realtime priority
    • chroot – change root to directory (Debian-specific)

 

  • Sigpending – examine pending signals.

sigpending () returns the set of signals that are pending for delivery to the calling thread (i.e., the signals which have been raised while blocked). The mask of pending signals is returned in set.

sigpending() returns 0 on success and -1 on error

 

credits :- Sagar Salunkhe

Linux KVM: Disable virbr0 NAT Interface

The virtual network (virbr0) used for Network address translation (NAT) which allows guests to access to network services. However, NAT slows down things and only recommended for desktop installations. To disable Network address translation (NAT) forwarding type the following commands:

Display Current Setup

Type the following command:
# ifconfig
Sample outputs:

virbr0    Link encap:Ethernet  HWaddr 00:00:00:00:00:00  
          inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
          inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:39 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 b)  TX bytes:7921 (7.7 KiB)

Or use the following command:
# virsh net-list
Sample outputs:

Name                 State      Autostart
-----------------------------------------
default              active     yes       

To disable virbr0, enter:
# virsh net-destroy default
# virsh net-undefine default
# service libvirtd restart
# ifconfig

TIBCO Administrator – Error (Core Dump Error)

Sometimes the administrator process in UNIX Platform Stops intermittently and then in the following location,

$TIBCO_HOME/administrator/<version>/tomcat/hs_err_pid<pid_of_admin>.log

file you will see a core dump error something like this

#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007efcdb723df8, pid=12496, tid=139624169486080
#
# JRE version: Java(TM) SE Runtime Environment (8.0_51-b16) (build 1.8.0_51-b16)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.51-b03 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V [libjvm.so+0x404df8] PhaseChaitin::gather_lrg_masks(bool)+0x208
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try “ulimit -c unlimited” before starting Java again
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#

————— T H R E A D —————

Current thread (0x00000000023a8800): JavaThread “C2 CompilerThread1” daemon [_thread_in_native, id=12508, stack(0x00007efcc8f63000,0x00007efcc9064000)]

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000000

Registers:
RAX=0x0000000000000000, RBX=0x00007efc901723e0, RCX=0x00007efc9016e890, RDX=0x0000000000000041
RSP=0x00007efcc905f650, RBP=0x00007efcc905f6c0, RSI=0x00007efcc9060f50, RDI=0x00007efc90b937a0
R8 =0x000000000000009a, R9 =0x0000000000000003, R10=0x0000000000000003, R11=0x0000000000000000
R12=0x0000000000000004, R13=0x0000000000000000, R14=0x0000000000000002, R15=0x00007efc90b937a0
RIP=0x00007efcdb723df8, EFLAGS=0x0000000000010246, CSGSFS=0x0000000000000033, ERR=0x0000000000000004
TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007efcc905f650)
0x00007efcc905f650: 01007efcc905f6c0 00007efcc9060f50
0x00007efcc905f660: 0000003dc905f870 00007efc9012d900
0x00007efcc905f670: 0000000100000002 ffffffff00000002
0x00007efcc905f680: 00007efc98009f40 00007efc90171d30
0x00007efcc905f690: 0000023ac9061038 00007efcc9060f50
0x00007efcc905f6a0: 0000000000000222 0000000000000090
0x00007efcc905f6b0: 00007efcc9061038 0000000000000222
0x00007efcc905f6c0: 00007efcc905f930 00007efcdb72705a
0x00007efcc905f6d0: 00007efcc905f750 00007efcc905f870
0x00007efcc905f6e0: 00007efcc905f830 00007efcc905f710
0x00007efcc905f6f0: 00007efcc905f7d0 00007efcc905f8a0
0x00007efcc905f700: 00007efcc9060f50 0000001200000117
0x00007efcc905f710: 00007efcdc2544b0 00007efc0000000c
0x00007efcc905f720: 00007efcc9061dd0 00007efcc9060f50
0x00007efcc905f730: 0000000000000807 00007efc9044a2e0
0x00007efcc905f740: 00007efc9012c610 0000000000000002
0x00007efcc905f750: 00007efcc905f820 00007efcdbae2ea3
0x00007efcc905f760: 000007c000000010 00007efcc90610d8
0x00007efcc905f770: 0000000000000028 ffffffe80000000e
0x00007efcc905f780: 00007efc9076dd60 00007efcc9061080
0x00007efcc905f790: 0000001100001f00 0000001a00000011
0x00007efcc905f7a0: 0000000100000001 00007efc903a1c78
0x00007efcc905f7b0: 00007efc908213e0 00007efcdbd7cd46
0x00007efcc905f7c0: 0000000000000008 00007efcdbd7cc97
0x00007efcc905f7d0: 00007efc00000009 00007efcc9061dd0
0x00007efcc905f7e0: 00007efc909059f0 00007efc900537a0
0x00007efcc905f7f0: 00007efc9040b7c0 00007efc9040c130
0x00007efcc905f800: 00007efc9081d280 00007efcc9061080
0x00007efcc905f810: 00007efcc9061060 0000000000000222
0x00007efcc905f820: 00007efcc905f870 00007efcdb8f8831
0x00007efcc905f830: 00007efc0000000b 00007efcc9061dd0
0x00007efcc905f840: 00007efc906cbd70 00007efcc9060f00

Instructions: (pc=0x00007efcdb723df8)
0x00007efcdb723dd8: 18 00 48 c7 c0 ff ff ff ff 4c 89 ff 49 0f 44 c7
0x00007efcdb723de8: 48 89 43 18 49 8b 07 ff 90 80 00 00 00 49 89 c5
0x00007efcdb723df8: 8b 00 21 43 38 41 8b 45 04 21 43 3c 4c 89 ff 41
0x00007efcdb723e08: 8b 45 08 21 43 40 41 8b 45 0c 21 43 44 41 8b 45

Register to memory mapping:

RAX=0x0000000000000000 is an unknown value
RBX=0x00007efc901723e0 is an unknown value
RCX=0x00007efc9016e890 is an unknown value
RDX=0x0000000000000041 is an unknown value
RSP=0x00007efcc905f650 is pointing into the stack for thread: 0x00000000023a8800
RBP=0x00007efcc905f6c0 is pointing into the stack for thread: 0x00000000023a8800
RSI=0x00007efcc9060f50 is pointing into the stack for thread: 0x00000000023a8800
RDI=0x00007efc90b937a0 is an unknown value
R8 =0x000000000000009a is an unknown value
R9 =0x0000000000000003 is an unknown value
R10=0x0000000000000003 is an unknown value
R11=0x0000000000000000 is an unknown value
R12=0x0000000000000004 is an unknown value
R13=0x0000000000000000 is an unknown value
R14=0x0000000000000002 is an unknown value
R15=0x00007efc90b937a0 is an unknown value
Stack: [0x00007efcc8f63000,0x00007efcc9064000], sp=0x00007efcc905f650, free space=1009k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x404df8] PhaseChaitin::gather_lrg_masks(bool)+0x208
V [libjvm.so+0x40805a] PhaseChaitin::Register_Allocate()+0x71a
V [libjvm.so+0x49abe0] Compile::Code_Gen()+0x260
V [libjvm.so+0x49e032] Compile::Compile(ciEnv*, C2Compiler*, ciMethod*, int, bool, bool, bool)+0x14b2
V [libjvm.so+0x3ebeb8] C2Compiler::compile_method(ciEnv*, ciMethod*, int)+0x198
V [libjvm.so+0x4a843a] CompileBroker::invoke_compiler_on_method(CompileTask*)+0xc9a
V [libjvm.so+0x4a93e6] CompileBroker::compiler_thread_loop()+0x5d6
V [libjvm.so+0xa5cbcf] JavaThread::thread_main_inner()+0xdf
V [libjvm.so+0xa5ccfc] JavaThread::run()+0x11c
V [libjvm.so+0x911048] java_start(Thread*)+0x108
C [libpthread.so.0+0x7aa1]
Current CompileTask:
C2:77124624 8967 ! 4 com.tibco.repo.RVRepoProcessBridge::handleServerHeartbeat (1075 bytes)
————— P R O C E S S —————

Java Threads: ( => current thread )
0x00007efcb8024000 JavaThread “Thread-41” daemon [_thread_blocked, id=13990, stack(0x00007efc372f5000,0x00007efc373f6000)]
0x00007efcac372000 JavaThread “http-bio-8989-exec-68” daemon [_thread_blocked, id=13853, stack(0x00007efc3ca41000,0x00007efc3cb42000)]
0x00007efc4002d000 JavaThread “http-bio-8989-exec-67” daemon [_thread_blocked, id=13837, stack(0x00007efc376f9000,0x00007efc377fa000)]
0x00007efc945b6800 JavaThread “http-bio-8989-exec-66” daemon [_thread_blocked, id=13828, stack(0x00007efc37cfd000,0x00007efc37dfe000)]
0x00007efc40028000 JavaThread “http-bio-8989-exec-65” daemon [_thread_blocked, id=13228, stack(0x00007efc374f7000,0x00007efc375f8000)]
0x00007efc993ba800 JavaThread “http-bio-8989-exec-64” daemon [_thread_blocked, id=13227, stack(0x00007efc3ce45000,0x00007efc3cf46000)]
0x00007efcc4012000 JavaThread “http-bio-8989-exec-63” daemon [_thread_blocked, id=13218, stack(0x00007efc3c63f000,0x00007efc3c740000)]
0x00007efc50006000 JavaThread “http-bio-8989-exec-62” daemon [_thread_blocked, id=13217, stack(0x00007efc373f6000,0x00007efc374f7000)]
0x00007efca800c000 JavaThread “http-bio-8989-exec-61” daemon [_thread_blocked, id=13216, stack(0x00007efc3c13a000,0x00007efc3c23b000)]
0x00007efc68004000 JavaThread “http-bio-8989-exec-60” daemon [_thread_blocked, id=13215, stack(0x00007efc3d34a000,0x00007efc3d44b000)]
0x00007efcb0006800 JavaThread “http-bio-8989-exec-59” daemon [_thread_blocked, id=13214, stack(0x00007efc3d54c000,0x00007efc3d64d000)]
0x00007efca8044800 JavaThread “http-bio-8989-exec-58” daemon [_thread_blocked, id=13213, stack(0x00007efc375f8000,0x00007efc376f9000)]
0x00007efc902c5800 JavaThread “http-bio-8989-exec-57” daemon [_thread_blocked, id=13212, stack(0x00007efc36ef1000,0x00007efc36ff2000)]
0x00007efcb4010800 JavaThread “http-bio-8989-exec-56” daemon [_thread_blocked, id=13211, stack(0x00007efc3d148000,0x00007efc3d249000)]
0x00007efc4408c800 JavaThread “http-bio-8989-exec-55” daemon [_thread_blocked, id=13210, stack(0x00007efc3e053000,0x00007efc3e154000)]
0x00007efcb4036800 JavaThread “http-bio-8989-exec-54” daemon [_thread_blocked, id=13201, stack(0x00007efc371f4000,0x00007efc372f5000)]
0x00007efcb0018800 JavaThread “http-bio-8989-exec-53” daemon [_thread_blocked, id=13200, stack(0x00007efc3c23b000,0x00007efc3c33c000)]
0x00007efc6c1e1000 JavaThread “http-bio-8989-exec-52” daemon [_thread_blocked, id=13199, stack(0x00007efc3c43d000,0x00007efc3c53e000)]
0x00007efc58005000 JavaThread “http-bio-8989-exec-51” daemon [_thread_blocked, id=13198, stack(0x00007efc3c039000,0x00007efc3c13a000)]
0x00007efc74006800 JavaThread “http-bio-8989-exec-50” daemon [_thread_blocked, id=13197, stack(0x00007efc9da1e000,0x00007efc9db1f000)]
0x00007efc54005800 JavaThread “AMI Worker 2” daemon [_thread_blocked, id=13120, stack(0x00007efcc2253000,0x00007efcc2354000)]
0x00007efc54003800 JavaThread “AMI Worker 1” daemon [_thread_blocked, id=13119, stack(0x00007efc36df0000,0x00007efc36ef1000)]
0x00007efcbc02f800 JavaThread “http-bio-8989-exec-49” daemon [_thread_blocked, id=13091, stack(0x00007efc37afb000,0x00007efc37bfc000)]
0x00007efc80083000 JavaThread “http-bio-8989-exec-48” daemon [_thread_blocked, id=13090, stack(0x00007efc3cd44000,0x00007efc3ce45000)]
0x00007efc4c01b000 JavaThread “http-bio-8989-exec-47” daemon [_thread_blocked, id=13089, stack(0x00007efc3d44b000,0x00007efc3d54c000)]
0x00007efca403a800 JavaThread “http-bio-8989-exec-46” daemon [_thread_blocked, id=13088, stack(0x00007efc36cef000,0x00007efc36df0000)]
0x00007efcc4023000 JavaThread “http-bio-8989-exec-45” daemon [_thread_blocked, id=13087, stack(0x00007efc3d249000,0x00007efc3d34a000)]
0x00007efc8468a000 JavaThread “http-bio-8989-exec-44” daemon [_thread_blocked, id=13086, stack(0x00007efc3d64d000,0x00007efc3d74e000)]
0x000000000252f800 JavaThread “http-bio-8989-AsyncTimeout” daemon [_thread_blocked, id=13032, stack(0x00007efc3d74e000,0x00007efc3d84f000)]
0x000000000252e800 JavaThread “http-bio-8989-Acceptor-0” daemon [_thread_in_native, id=13031, stack(0x00007efc3d84f000,0x00007efc3d950000)]
0x000000000252d000 JavaThread “ContainerBackgroundProcessor[StandardEngine[Catalina]]” daemon [_thread_blocked, id=13030, stack(0x00007efc3d950000,0x00007efc3da51000)]
0x00007efc44085800 JavaThread “Thread-37” daemon [_thread_blocked, id=13029, stack(0x00007efc3f0f5000,0x00007efc3f1f6000)]
0x00007efc78971000 JavaThread “Thread-36” daemon [_thread_blocked, id=13028, stack(0x00007efc3de51000,0x00007efc3df52000)]
0x00007efc78967000 JavaThread “Thread-33” daemon [_thread_blocked, id=13025, stack(0x00007efc3e154000,0x00007efc3e255000)]
0x00007efc788fe000 JavaThread “Tibrv Dispatcher” daemon [_thread_in_native, id=13024, stack(0x00007efc3e255000,0x00007efc3e356000)]
0x00007efc788fa800 JavaThread “RVAgentManagerTransport dispatch thread” daemon [_thread_in_native, id=13023, stack(0x00007efc3e356000,0x00007efc3e457000)]
0x00007efc788f8000 JavaThread “RacSubManager” daemon [_thread_blocked, id=13022, stack(0x00007efc3e457000,0x00007efc3e558000)]
0x00007efc788d8800 JavaThread “InitialListTimer” daemon [_thread_blocked, id=13021, stack(0x00007efc3e558000,0x00007efc3e659000)]
0x00007efc788d7000 JavaThread “RvHeartBeatTimer” daemon [_thread_blocked, id=13020, stack(0x00007efc3e659000,0x00007efc3e75a000)]
0x00007efc788d4000 JavaThread “AgentAliveMonitor dispatch thread” daemon [_thread_in_native, id=13019, stack(0x00007efc3ebf2000,0x00007efc3ecf3000)]
0x00007efc788aa000 JavaThread “AgentEventMonitor dispatch thread” daemon [_thread_in_native, id=13018, stack(0x00007efc3ecf3000,0x00007efc3edf4000)]
0x00007efc787c2800 JavaThread “Thread-30” daemon [_thread_blocked, id=13017, stack(0x00007efc3ea2b000,0x00007efc3eb2c000)]
0x00007efc4c009800 JavaThread “Thread-29(HawkConfig)” daemon [_thread_blocked, id=13016, stack(0x00007efc3eff4000,0x00007efc3f0f5000)]
0x00007efc4c007800 JavaThread “Thread-28” daemon [_thread_in_native, id=13015, stack(0x00007efc3f1f6000,0x00007efc3f2f7000)]
0x00007efc7827b800 JavaThread “Thread-27” daemon [_thread_blocked, id=13012, stack(0x00007efc3f2f7000,0x00007efc3f3f8000)]
0x00007efc780fb800 JavaThread “Thread-26(HawkConfig)” daemon [_thread_blocked, id=13011, stack(0x00007efc3f5f8000,0x00007efc3f6f9000)]
0x00007efc780f9800 JavaThread “Thread-25” daemon [_thread_in_native, id=13010, stack(0x00007efc3f6f9000,0x00007efc3f7fa000)]
0x00007efc78060000 JavaThread “Thread-24(MonitoringManagement)” daemon [_thread_blocked, id=13009, stack(0x00007efc3f7fa000,0x00007efc3f8fb000)]
0x00007efc7805e000 JavaThread “Thread-23” daemon [_thread_in_native, id=13008, stack(0x00007efc3f8fb000,0x00007efc3f9fc000)]
0x00007efc78210000 JavaThread “Thread-21(quality)” daemon [_thread_blocked, id=13007, stack(0x00007efc3f9fc000,0x00007efc3fafd000)]
0x00007efc7820f800 JavaThread “Thread-20” daemon [_thread_in_native, id=13006, stack(0x00007efc3fafd000,0x00007efc3fbfe000)]
0x00007efc6c1d6800 JavaThread “Thread-17” daemon [_thread_blocked, id=13003, stack(0x00007efc5d6fb000,0x00007efc5d7fc000)]
0x00007efc6c1ff800 JavaThread “CommitQueue4_0” daemon [_thread_in_native, id=13002, stack(0x00007efc3fbfe000,0x00007efc3fcff000)]
0x00007efc6c1fd000 JavaThread “NormalQueue4_2” daemon [_thread_in_native, id=13001, stack(0x00007efc3feff000,0x00007efc40000000)]
0x00007efc6c1fb000 JavaThread “NormalQueue4_1” daemon [_thread_in_native, id=13000, stack(0x00007efc5c0e9000,0x00007efc5c1ea000)]
0x00007efc6c1f9000 JavaThread “NormalQueue4_0” daemon [_thread_in_native, id=12999, stack(0x00007efc5c1ea000,0x00007efc5c2eb000)]
0x00007efc6c1f5000 JavaThread “CommitQueue3_0” daemon [_thread_in_native, id=12998, stack(0x00007efc5c2eb000,0x00007efc5c3ec000)]
0x00007efc6c1f3000 JavaThread “NormalQueue3_2” daemon [_thread_in_native, id=12997, stack(0x00007efc5c3ec000,0x00007efc5c4ed000)]
0x00007efc6c1f1800 JavaThread “NormalQueue3_1” daemon [_thread_in_native, id=12996, stack(0x00007efc5c4ed000,0x00007efc5c5ee000)]
0x00007efc6c1ef800 JavaThread “NormalQueue3_0” daemon [_thread_in_native, id=12995, stack(0x00007efc5c5ee000,0x00007efc5c6ef000)]
0x00007efc6c1eb800 JavaThread “CommitQueue2_0” daemon [_thread_in_native, id=12994, stack(0x00007efc5c6ef000,0x00007efc5c7f0000)]
0x00007efc6c1ea000 JavaThread “NormalQueue2_2” daemon [_thread_in_native, id=12993, stack(0x00007efc5c7f0000,0x00007efc5c8f1000)]
0x00007efc6c1e9800 JavaThread “NormalQueue2_1” daemon [_thread_in_native, id=12992, stack(0x00007efc5c8f1000,0x00007efc5c9f2000)]
0x00007efc6c1de800 JavaThread “NormalQueue2_0” daemon [_thread_in_native, id=12991, stack(0x00007efc5c9f2000,0x00007efc5caf3000)]
0x00007efc6c124000 JavaThread “Thread-16(AUTH_quality)” daemon [_thread_blocked, id=12989, stack(0x00007efc5ccf3000,0x00007efc5cdf4000)]
0x00007efc6c116800 JavaThread “Thread-15” daemon [_thread_in_native, id=12988, stack(0x00007efc5cdf4000,0x00007efc5cef5000)]
0x00007efc6c101000 JavaThread “Timer-0” daemon [_thread_blocked, id=12987, stack(0x00007efc5cef5000,0x00007efc5cff6000)]
0x00007efc6c0f7800 JavaThread “net.sf.ehcache.CacheManager@5a6a4d5b” daemon [_thread_blocked, id=12986, stack(0x00007efc5cff6000,0x00007efc5d0f7000)]
0x00007efc6c0c6000 JavaThread “CommitQueue1_0” daemon [_thread_in_native, id=12985, stack(0x00007efc5d0f7000,0x00007efc5d1f8000)]
0x00007efc6c0c4000 JavaThread “NormalQueue1_2” daemon [_thread_in_native, id=12984, stack(0x00007efc5d1f8000,0x00007efc5d2f9000)]
0x00007efc6c0c2800 JavaThread “NormalQueue1_1” daemon [_thread_in_native, id=12983, stack(0x00007efc5d2f9000,0x00007efc5d3fa000)]
0x00007efc6c0bd800 JavaThread “NormalQueue1_0” daemon [_thread_in_native, id=12982, stack(0x00007efc5d3fa000,0x00007efc5d4fb000)]
0x00007efc6c09e800 JavaThread “HB-1” daemon [_thread_in_native, id=12980, stack(0x00007efc9c013000,0x00007efc9c114000)]
0x00007efc6c09b800 JavaThread “HB-0” daemon [_thread_in_native, id=12979, stack(0x00007efc9c114000,0x00007efc9c215000)]
0x00007efc6c09a000 JavaThread “SYNC-0” daemon [_thread_in_native, id=12978, stack(0x00007efc9c215000,0x00007efc9c316000)]
0x00007efc6c067800 JavaThread “ImstMgmt” daemon [_thread_in_native, id=12973, stack(0x00007efc9c316000,0x00007efc9c417000)]
0x00007efc6c028800 JavaThread “HawkImplantDisp” daemon [_thread_in_native, id=12972, stack(0x00007efc9c417000,0x00007efc9c518000)]
0x00007efc781a1800 JavaThread “Thread-11” daemon [_thread_blocked, id=12967, stack(0x00007efc9cf19000,0x00007efc9d01a000)]
0x00000000030fe000 JavaThread “GC Daemon” daemon [_thread_blocked, id=12540, stack(0x00007efcc80bc000,0x00007efcc81bd000)]
0x00000000023bf800 JavaThread “Service Thread” daemon [_thread_blocked, id=12510, stack(0x00007efcc8d61000,0x00007efcc8e62000)]
0x00000000023aa800 JavaThread “C1 CompilerThread2” daemon [_thread_blocked, id=12509, stack(0x00007efcc8e62000,0x00007efcc8f63000)]
=>0x00000000023a8800 JavaThread “C2 CompilerThread1” daemon [_thread_in_native, id=12508, stack(0x00007efcc8f63000,0x00007efcc9064000)]
0x00000000023a5800 JavaThread “C2 CompilerThread0” daemon [_thread_blocked, id=12507, stack(0x00007efcc9064000,0x00007efcc9165000)]
0x00000000023a4000 JavaThread “Signal Dispatcher” daemon [_thread_blocked, id=12506, stack(0x00007efcc9165000,0x00007efcc9266000)]
0x000000000236c000 JavaThread “Finalizer” daemon [_thread_blocked, id=12505, stack(0x00007efcc9266000,0x00007efcc9367000)]
0x000000000236a000 JavaThread “Reference Handler” daemon [_thread_blocked, id=12504, stack(0x00007efcc9367000,0x00007efcc9468000)]
0x00000000022f6800 JavaThread “main” [_thread_in_native, id=12496, stack(0x00007ffec1ae5000,0x00007ffec1be5000)]

Other Threads:
0x0000000002364800 VMThread [stack: 0x00007efcc9468000,0x00007efcc9569000] [id=12503]
0x00000000023c2800 WatcherThread [stack: 0x00007efcc8c60000,0x00007efcc8d61000] [id=12511]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap:
PSYoungGen total 46080K, used 23811K [0x00000000f5580000, 0x00000000f8600000, 0x0000000100000000)
eden space 45568K, 51% used [0x00000000f5580000,0x00000000f6ca0dc0,0x00000000f8200000)
from space 512K, 25% used [0x00000000f8280000,0x00000000f82a0000,0x00000000f8300000)
to space 2048K, 0% used [0x00000000f8400000,0x00000000f8400000,0x00000000f8600000)
ParOldGen total 122368K, used 45598K [0x00000000e0000000, 0x00000000e7780000, 0x00000000f5580000)
object space 122368K, 37% used [0x00000000e0000000,0x00000000e2c87800,0x00000000e7780000)
Metaspace used 47578K, capacity 50064K, committed 50432K, reserved 1093632K
class space used 5434K, capacity 5762K, committed 5888K, reserved 1048576K

Card table byte_map: [0x00007efccb5c4000,0x00007efccb6c5000] byte_map_base: 0x00007efccaec4000

Marking Bits: (ParMarkBitMap*) 0x00007efcdc2bd660
Begin Bits: [0x00007efcc3000000, 0x00007efcc3800000)
End Bits: [0x00007efcc3800000, 0x00007efcc4000000)

Polling page: 0x00007efcdc323000

CodeCache: size=245760Kb used=27975Kb max_used=28034Kb free=217784Kb
bounds [0x00007efccba85000, 0x00007efccd625000, 0x00007efcdaa85000]
total_blobs=7417 nmethods=6888 adapters=442
compilation: enabled

Compilation events (10 events):
Event: 75538.391 Thread 0x00000000023a5800 8963 4 org.hsqldb.Expression::collectInGroupByExpressions (61 bytes)
Event: 75538.393 Thread 0x00000000023a8800 nmethod 8962 0x00007efcccac8fd0 code [0x00007efcccac9140, 0x00007efcccac92b8]
Event: 75538.394 Thread 0x00000000023a8800 8964 4 org.hsqldb.Expression::isConstant (118 bytes)
Event: 75538.397 Thread 0x00000000023a8800 nmethod 8964 0x00007efcccb62110 code [0x00007efcccb62320, 0x00007efcccb62468]
Event: 75538.398 Thread 0x00000000023a5800 nmethod 8963 0x00007efccbf127d0 code [0x00007efccbf12ae0, 0x00007efccbf12e70]
Event: 76104.554 Thread 0x00000000023a8800 8965 4 com.tibco.tibrv.TibrvMsg::writeBool (164 bytes)
Event: 76104.565 Thread 0x00000000023a8800 nmethod 8965 0x00007efccca4e190 code [0x00007efccca4e420, 0x00007efccca4e7f0]
Event: 76857.543 Thread 0x00000000023aa800 8966 1 com.tibco.uac.monitor.server.MonitorServer::access$000 (4 bytes)
Event: 76857.544 Thread 0x00000000023aa800 nmethod 8966 0x00007efccc3c07d0 code [0x00007efccc3c0920, 0x00007efccc3c0a10]
Event: 77124.580 Thread 0x00000000023a8800 8967 ! 4 com.tibco.repo.RVRepoProcessBridge::handleServerHeartbeat (1075 bytes)

GC Heap History (10 events):
Event: 70999.039 GC heap before
{Heap before GC invocations=77 (full 8):
PSYoungGen total 50688K, used 48256K [0x00000000f5580000, 0x00000000f8a80000, 0x0000000100000000)
eden space 48128K, 100% used [0x00000000f5580000,0x00000000f8480000,0x00000000f8480000)
from space 2560K, 5% used [0x00000000f8800000,0x00000000f8820000,0x00000000f8a80000)
to space 3072K, 0% used [0x00000000f8480000,0x00000000f8480000,0x00000000f8780000)
ParOldGen total 122368K, used 45526K [0x00000000e0000000, 0x00000000e7780000, 0x00000000f5580000)
object space 122368K, 37% used [0x00000000e0000000,0x00000000e2c75800,0x00000000e7780000)
Metaspace used 47577K, capacity 50064K, committed 50432K, reserved 1093632K
class space used 5434K, capacity 5762K, committed 5888K, reserved 1048576K
Event: 70999.045 GC heap after
Heap after GC invocations=77 (full 8):
PSYoungGen total 48128K, used 160K [0x00000000f5580000, 0x00000000f8a00000, 0x0000000100000000)
eden space 47616K, 0% used [0x00000000f5580000,0x00000000f5580000,0x00000000f8400000)
from space 512K, 31% used [0x00000000f8480000,0x00000000f84a8000,0x00000000f8500000)
to space 3072K, 0% used [0x00000000f8700000,0x00000000f8700000,0x00000000f8a00000)
ParOldGen total 122368K, used 45550K [0x00000000e0000000, 0x00000000e7780000, 0x00000000f5580000)
object space 122368K, 37% used [0x00000000e0000000,0x00000000e2c7b800,0x00000000e7780000)
Metaspace used 47577K, capacity 50064K, committed 50432K, reserved 1093632K
class space used 5434K, capacity 5762K, committed 5888K, reserved 1048576K
}
Event: 72379.077 GC heap before
{Heap before GC invocations=78 (full 8):
PSYoungGen total 48128K, used 47776K [0x00000000f5580000, 0x00000000f8a00000, 0x0000000100000000)
eden space 47616K, 100% used [0x00000000f5580000,0x00000000f8400000,0x00000000f8400000)
from space 512K, 31% used [0x00000000f8480000,0x00000000f84a8000,0x00000000f8500000)
to space 3072K, 0% used [0x00000000f8700000,0x00000000f8700000,0x00000000f8a00000)
ParOldGen total 122368K, used 45550K [0x00000000e0000000, 0x00000000e7780000, 0x00000000f5580000)
object space 122368K, 37% used [0x00000000e0000000,0x00000000e2c7b800,0x00000000e7780000)
Metaspace used 47577K, capacity 50064K, committed 50432K, reserved 1093632K
class space used 5434K, capacity 5762K, committed 5888K, reserved 1048576K
Event: 72379.082 GC heap after
Heap after GC invocations=78 (full 8):
PSYoungGen total 48640K, used 128K [0x00000000f5580000, 0x00000000f8880000, 0x0000000100000000)
eden space 47104K, 0% used [0x00000000f5580000,0x00000000f5580000,0x00000000f8380000)
from space 1536K, 8% used [0x00000000f8700000,0x00000000f8720000,0x00000000f8880000)
to space 2560K, 0% used [0x00000000f8380000,0x00000000f8380000,0x00000000f8600000)
ParOldGen total 122368K, used 45566K [0x00000000e0000000, 0x00000000e7780000, 0x00000000f5580000)
object space 122368K, 37% used [0x00000000e0000000,0x00000000e2c7f800,0x00000000e7780000)
Metaspace used 47577K, capacity 50064K, committed 50432K, reserved 1093632K
class space used 5434K, capacity 5762K, committed 5888K, reserved 1048576K
}
Event: 73744.273 GC heap before
{Heap before GC invocations=79 (full 8):
PSYoungGen total 48640K, used 47232K [0x00000000f5580000, 0x00000000f8880000, 0x0000000100000000)
eden space 47104K, 100% used [0x00000000f5580000,0x00000000f8380000,0x00000000f8380000)
from space 1536K, 8% used [0x00000000f8700000,0x00000000f8720000,0x00000000f8880000)
to space 2560K, 0% used [0x00000000f8380000,0x00000000f8380000,0x00000000f8600000)
ParOldGen total 122368K, used 45566K [0x00000000e0000000, 0x00000000e7780000, 0x00000000f5580000)
object space 122368K, 37% used [0x00000000e0000000,0x00000000e2c7f800,0x00000000e7780000)
Metaspace used 47578K, capacity 50064K, committed 50432K, reserved 1093632K
class space used 5434K, capacity 5762K, committed 5888K, reserved 1048576K
Event: 73744.279 GC heap after
Heap after GC invocations=79 (full 8):
PSYoungGen total 47104K, used 96K [0x00000000f5580000, 0x00000000f8800000, 0x0000000100000000)
eden space 46592K, 0% used [0x00000000f5580000,0x00000000f5580000,0x00000000f8300000)
from space 512K, 18% used [0x00000000f8380000,0x00000000f8398000,0x00000000f8400000)
to space 2560K, 0% used [0x00000000f8580000,0x00000000f8580000,0x00000000f8800000)
ParOldGen total 122368K, used 45582K [0x00000000e0000000, 0x00000000e7780000, 0x00000000f5580000)
object space 122368K, 37% used [0x00000000e0000000,0x00000000e2c83800,0x00000000e7780000)
Metaspace used 47578K, capacity 50064K, committed 50432K, reserved 1093632K
class space used 5434K, capacity 5762K, committed 5888K, reserved 1048576K
}
Event: 75098.826 GC heap before
{Heap before GC invocations=80 (full 8):
PSYoungGen total 47104K, used 46688K [0x00000000f5580000, 0x00000000f8800000, 0x0000000100000000)
eden space 46592K, 100% used [0x00000000f5580000,0x00000000f8300000,0x00000000f8300000)
from space 512K, 18% used [0x00000000f8380000,0x00000000f8398000,0x00000000f8400000)
to space 2560K, 0% used [0x00000000f8580000,0x00000000f8580000,0x00000000f8800000)
ParOldGen total 122368K, used 45582K [0x00000000e0000000, 0x00000000e7780000, 0x00000000f5580000)
object space 122368K, 37% used [0x00000000e0000000,0x00000000e2c83800,0x00000000e7780000)
Metaspace used 47578K, capacity 50064K, committed 50432K, reserved 1093632K
class space used 5434K, capacity 5762K, committed 5888K, reserved 1048576K
Event: 75098.831 GC heap after
Heap after GC invocations=80 (full 8):
PSYoungGen total 48128K, used 160K [0x00000000f5580000, 0x00000000f8780000, 0x0000000100000000)
eden space 46080K, 0% used [0x00000000f5580000,0x00000000f5580000,0x00000000f8280000)
from space 2048K, 7% used [0x00000000f8580000,0x00000000f85a8000,0x00000000f8780000)
to space 2560K, 0% used [0x00000000f8280000,0x00000000f8280000,0x00000000f8500000)
ParOldGen total 122368K, used 45590K [0x00000000e0000000, 0x00000000e7780000, 0x00000000f5580000)
object space 122368K, 37% used [0x00000000e0000000,0x00000000e2c85800,0x00000000e7780000)
Metaspace used 47578K, capacity 50064K, committed 50432K, reserved 1093632K
class space used 5434K, capacity 5762K, committed 5888K, reserved 1048576K
}
Event: 76440.356 GC heap before
{Heap before GC invocations=81 (full 8):
PSYoungGen total 48128K, used 46240K [0x00000000f5580000, 0x00000000f8780000, 0x0000000100000000)
eden space 46080K, 100% used [0x00000000f5580000,0x00000000f8280000,0x00000000f8280000)
from space 2048K, 7% used [0x00000000f8580000,0x00000000f85a8000,0x00000000f8780000)
to space 2560K, 0% used [0x00000000f8280000,0x00000000f8280000,0x00000000f8500000)
ParOldGen total 122368K, used 45590K [0x00000000e0000000, 0x00000000e7780000, 0x00000000f5580000)
object space 122368K, 37% used [0x00000000e0000000,0x00000000e2c85800,0x00000000e7780000)
Metaspace used 47578K, capacity 50064K, committed 50432K, reserved 1093632K
class space used 5434K, capacity 5762K, committed 5888K, reserved 1048576K
Event: 76440.360 GC heap after
Heap after GC invocations=81 (full 8):
PSYoungGen total 46080K, used 128K [0x00000000f5580000, 0x00000000f8600000, 0x0000000100000000)
eden space 45568K, 0% used [0x00000000f5580000,0x00000000f5580000,0x00000000f8200000)
from space 512K, 25% used [0x00000000f8280000,0x00000000f82a0000,0x00000000f8300000)
to space 2048K, 0% used [0x00000000f8400000,0x00000000f8400000,0x00000000f8600000)
ParOldGen total 122368K, used 45598K [0x00000000e0000000, 0x00000000e7780000, 0x00000000f5580000)
object space 122368K, 37% used [0x00000000e0000000,0x00000000e2c87800,0x00000000e7780000)
Metaspace used 47578K, capacity 50064K, committed 50432K, reserved 1093632K
class space used 5434K, capacity 5762K, committed 5888K, reserved 1048576K

In Such cases the following are the checks we need to do

  • Check the ulimit
    • It is supposed to be unlimited
  • Check the limit for no of open files
    • You can use the following command to check the number of open files
    • lsof | grep <user>|grep -v grep|wc -l

    • Then Further check the limits.conf for the value set from your end for the same
    • <user> soft nofile 350000
      <user> hard nofile 350000
      <user> soft nproc 65536
      <user> hard nproc 65536
      <user> soft stack 10240
      <user> hard stack 10240
      <user> soft sigpending 1548380
      <user> hard sigpending 1548380
  • Hopefully the problem will be resolved.

Happy Troubleshooting Guys 🙂