CDH Services crash with Java Virtual Machine error after application of Linux Security Kernel Patch for CVE-2017-1000364
Cloudera has received a number of customer cases where CDH services using the JSVC set of libraries crash with a Java Virtual Machine error in CDH clusters after applying a recent Kernel Security Patch. The list below contains the announcements about the patch for a number of distributions.
- RHSA-2017-1484 (RHEL 7). RedHat has fixed this problem for RHEL 7, marked this patch as outdated and superseded by RHSA-2017-1674.
- RHSA-2017-1486 (RHEL 6). RedHat has fixed this problem for RHEL 6, marked this as outdated and superseded by RHSA-2017-1723.
- ELSA-2017-1484 (Oracle Linux 7). Oracle has also added the fix for Oracle Linux 7 in ELBA-2017-1674.
- ELSA-2017-1486 (Oracle Linux 6). Oracle has fixed this problem in ELSA-2017-1723.
- USN-3335-1 (Ubuntu 14.04). Fixed in USN-3338-2.
- CVE-2017-1000364 (SUSE). SUSE has also fixed this problem and the patch names are included in this same advisory.
We have been able to reproduce Cloudera services not starting after the application of the original patch.
Products affected: Impala, HDFS DataNodes, HDFS NFS Gateway
Releases affected: Any CDH release that runs on RHEL, CentOS, Oracle Linux, SUSE Linux and Ubuntu where this Security Kernel Patch is applied.
Users affected: All Impala & HDFS users
Severity (Low/Medium/High): High
Impact:
After applying the patch the service's daemon crashes with an error similar to the following:
----
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGBUS (0x7) at pc=0x00007fe91ef6cebc, pid=30321, tid=0x00007fe930c67700
> #
> ----
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGBUS (0x7) at pc=0x00007fe91ef6cebc, pid=30321, tid=0x00007fe930c67700
> #
> ----
Immediate action required:
Apply a kernel update containing the fix for your Operating System.
If you are unable to apply a kernel update, the Java thread stack size may be set for the affected service(s). The Java Configuration Options will need to include the following value: "-Xss1280k". For roles that do not have a parameter for Java Configuration Options in Cloudera Manager, the value will need to be set in the Environment Advanced Configuration Snippet (Safety Valve).
Example for adding the value in the Java Configuration Options:
- In Cloudera Manager, Navigate to Impala > Configuration > Category (Advanced)
- Type “Java Configuration Options” in the configuration search box and locate the required field
- Add “-Xss1280k” to the field
- Save Changes and restart the affected roles
Example for adding the value in the Advanced Configuration Snippet (Safety Valve):
- In Cloudera Manager, Navigate to Impala > Configuration > Category (Advanced)
- Type “Advanced Configuration Snippet” in the configuration search box and locate the required field
- Add “JAVA_TOOL_OPTIONS=-Xss1280k” to the field.
- Save Changes and restart the affected roles
The parameters to change for all affected services are shown in the table below.
| Service | Parameter |
|---|---|
| HDFS DataNodes | Java Configuration Options for DataNode |
| HDFS NFS Gateway | Java Configuration Options for NFS Gateway |
| Impala | Impala Service Environment Advanced Configuration Snippet (Safety Valve) |
| Impala | Java Configuration Options for Catalog Server |
| Impala | Impala Daemon Environment Advanced Configuration Snippet |
For the latest update on this issue see the corresponding Knowledge article:
TSB 2017-242: CDH Services crash with Java Virtual Machine error after application of Linux Security Kernel Patch for CVE-2017-1000364
TSB 2017-242: CDH Services crash with Java Virtual Machine error after application of Linux Security Kernel Patch for CVE-2017-1000364
All rights belong to: Cloudera, Inc. 395 Page Mill Rd, 3rd Floor · Palo Alto, CA 94306 USA
