You are not logged in.

#1 2012-10-26 08:19:34

rmetzger
Member
Registered: 2012-10-04
Posts: 13

JVM fails to allocate 1 GB of byte-arrays

Hello,

I'm running Arch linux, and I'm having troubles with a larger application that uses byte-arrays internally to manage memory. But the relevant code is a quite short loop:

public class MemAlloc {
	public static void main(String[] args) {
		List<byte[]> m = new ArrayList<byte[]>();
		for (int i = 0; i < 36000; i++) {
			m.add(new byte[32768]);
		}
	}
}

So this code just allocates around 1 GB of memory.
When running it, the JVM becomes unresponsive, all CPUs go up to 100% and the JVM never finishes. It does not seem that it allocates any memory.

The problem occurs only on my computer, that has the following configuration:
16 GB main memory, Intel Core i7
$ java -version
java version "1.7.0_09"
OpenJDK Runtime Environment (IcedTea7 2.3.3) (ArchLinux build 7.u9_2.3.3-1-x86_64)
OpenJDK 64-Bit Server VM (build 23.2-b09, mixed mode

The problem also occurs with Oracles JVM


My laptop has the same versions but the program works. The laptop has only 6 GB of main memory, but it is also a 64 bit machine.

Offline

#2 2012-10-26 09:23:54

yungtrizzle
Member
Registered: 2011-04-24
Posts: 139

Re: JVM fails to allocate 1 GB of byte-arrays

Have you considered the fact that your code probably leaks memory?  Why are you allocating so much memory to a class, the JVM is not immune to memory leaks?

Offline

#3 2012-10-26 10:42:38

rmetzger
Member
Registered: 2012-10-04
Posts: 13

Re: JVM fails to allocate 1 GB of byte-arrays

Thanks for your comment!

As I said, this code is part of a larger application (100.00 LOC), so I have no choice. Additionally, the code runs well on my other computer and it did run well before using Arch (I was using Ubuntu).
So I'm guessing that there is something wrong with the JVM, glibc's malloc() or the kernel itself. The problem probably occurs because Java is using multiple threads to allocate memory (at least I expect this because all 8 cores are utilized). So it could be the case that something is failing when allocating memory from multiple threads in parallel.

Why is the application doing this? The app is a Hadoop-like "Big Data" processing engine, that streams through multi giga-or terabyte of data. When doing so, the garbage collector becomes confused wink so the're using byte arrays to manage their memory.
I know that java might not be the best choice for implementing a database, but I'm not in the position to decide this wink

Additionally, I expect the JVM to be able to allocate one gig of memory.

Offline

#4 2012-10-26 11:04:54

Ramses de Norre
Member
From: Leuven - Belgium
Registered: 2007-03-27
Posts: 1,289

Re: JVM fails to allocate 1 GB of byte-arrays

Runs fine here if I run it with

java -Xmx1500M MemAlloc

Without the -Xmx option I run out of heapspace.

By the way, by preallocating the ArrayList size, the program runs in less then three seconds instead of the almost fifteen seconds for your original example:

public class MemAlloc {
    public static void main(String[] args) {
        final List<byte[]> m = new ArrayList<byte[]>(36000);    % Prealloc the backing array
        for (int i = 0; i < 36000; i++) {
            m.add(new byte[32768]);
        }
    }
}

JRE version:

jre7-openjdk: 7.u9_2.3.3-1

Offline

#5 2012-10-26 11:13:56

rmetzger
Member
Registered: 2012-10-04
Posts: 13

Re: JVM fails to allocate 1 GB of byte-arrays

Thanks for trying out!

I forgot to mention that you need to set an appropriate heap-space. I'm running it with

-Xms3000M -Xmx8000M

The preallocation of the List makes things faster, right.
I'm running the same JRE version.

How many cores do you have?

UPDATE: It really seems to be a multi-core issue, because this works:

taskset 1 java -jar javamem.jar

Last edited by rmetzger (2012-10-26 11:20:15)

Offline

#6 2012-10-26 11:20:23

Ramses de Norre
Member
From: Leuven - Belgium
Registered: 2007-03-27
Posts: 1,289

Re: JVM fails to allocate 1 GB of byte-arrays

That was on a core2duo, 2 cores, 1.8GHz, no hyperthreading.
Did you try simply starting the jvm with those memory settings? (Just a System.out.println in main for example.)

Offline

#7 2012-10-26 11:24:46

rmetzger
Member
Registered: 2012-10-04
Posts: 13

Re: JVM fails to allocate 1 GB of byte-arrays

Ramses de Norre wrote:

Did you try simply starting the jvm with those memory settings? (Just a System.out.println in main for example.)

I did print out the value of i, and interestingly, the loop works until "i = 23798", e.g. 23 798 * 32 768 bytes = 743.6875 megabytes.

The program works well when only using one CPU (see update on previous post), so it seems to be a multicore-issue.

Offline

#8 2012-10-26 12:34:27

rmetzger
Member
Registered: 2012-10-04
Posts: 13

Re: JVM fails to allocate 1 GB of byte-arrays

Okay, I did some more experiments. The problem is not JVM specific.

I'm using the following c-program to allocate memory:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

int main(int argc, char** argv) {
	long mem = atol(argv[1])*1024l*1024l;

	printf("Allocating %ld byte.\n", mem);
	void* data = malloc(mem);
	memset(data,0,mem);
	printf("Done allocating %ld byte.\n", mem);
}

and the following bash script to invoke the resulting program multiple times:

for i in `seq 5`; do
	./a.out 200 &
done

The the troubles occur when starting at least 5 "memory allocators". So no process terminates and they all use one cpu for 100%.
The output looks like this:

% ./mult.sh
Allocating 209715200 byte.
Allocating 209715200 byte.
Allocating 209715200 byte.                                                                        
Allocating 209715200 byte.
Allocating 209715200 byte.

Offline

#9 2012-10-26 12:59:16

Spacenick
Member
From: Germany
Registered: 2010-04-02
Posts: 168

Re: JVM fails to allocate 1 GB of byte-arrays

The C example works for me, even with 16 parallel allocators.

Offline

#10 2012-11-02 15:05:23

eaglexro
Member
Registered: 2011-03-18
Posts: 11

Re: JVM fails to allocate 1 GB of byte-arrays

rmetzger wrote:

Hello,

I'm running Arch linux, and I'm having troubles with a larger application that uses byte-arrays internally to manage memory. But the relevant code is a quite short loop:

public class MemAlloc {
	public static void main(String[] args) {
		List<byte[]> m = new ArrayList<byte[]>();
		for (int i = 0; i < 36000; i++) {
			m.add(new byte[32768]);
		}
	}
}

So this code just allocates around 1 GB of memory.
When running it, the JVM becomes unresponsive, all CPUs go up to 100% and the JVM never finishes. It does not seem that it allocates any memory.

The problem occurs only on my computer, that has the following configuration:
16 GB main memory, Intel Core i7
$ java -version
java version "1.7.0_09"
OpenJDK Runtime Environment (IcedTea7 2.3.3) (ArchLinux build 7.u9_2.3.3-1-x86_64)
OpenJDK 64-Bit Server VM (build 23.2-b09, mixed mode

The problem also occurs with Oracles JVM


My laptop has the same versions but the program works. The laptop has only 6 GB of main memory, but it is also a 64 bit machine.

Finally someone else experienced this issue, I was starting to go crazy since I couldn't find any other references.

Thanks to your (java) test, I started mixing configurations. The only one that worked is:

java version 1.6.0_27 and kernel version: 3.6.3-1 (both available from ARM)

All other configurations gave varying degrees of success, but none worked 100%.

Last edited by eaglexro (2012-11-02 15:05:50)

Offline

#11 2012-11-02 15:24:03

rmetzger
Member
Registered: 2012-10-04
Posts: 13

Re: JVM fails to allocate 1 GB of byte-arrays

eaglexro wrote:

Finally someone else experienced this issue, I was starting to go crazy since I couldn't find any other references.

Thanks to your (java) test, I started mixing configurations. The only one that worked is:

java version 1.6.0_27 and kernel version: 3.6.3-1 (both available from ARM)

All other configurations gave varying degrees of success, but none worked 100%.

Hi,

cool! I'm very happy too. I stopped updating this thread since I found no reliable way to reproduce the behavior.
I noticed that it does not occur right after booting. So there must be some weird conditions that trigger the behavior.

On what hardware are you running your stuff?
I've the following relevant components:
Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz
GeForce GTX 560 Ti
ASRock Z68 Extreme4 Gen3 Mainboard

Offline

#12 2012-11-02 15:30:10

eaglexro
Member
Registered: 2011-03-18
Posts: 11

Re: JVM fails to allocate 1 GB of byte-arrays

rmetzger wrote:

Hi,

cool! I'm very happy too. I stopped updating this thread since I found no reliable way to reproduce the behavior.
I noticed that it does not occur right after booting. So there must be some weird conditions that trigger the behavior.

On what hardware are you running your stuff?
I've the following relevant components:
Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz
GeForce GTX 560 Ti
ASRock Z68 Extreme4 Gen3 Mainboard

I was able to reproduce it always, like this:

public class MemAlloc {
        public static void main(String[] args) {
                while (true) {
                        List<byte[]> m = new ArrayList<byte[]>(36000);
                        for (int i = 0; i < 36000; i++) {
                                System.out.println("i: " + i);
                                m.add(new byte[32768]);
                        }
                }
        }
}

Depending on the configuration, sometimes the first iteration would finish ok without any issues, thats why I added the infinite loop.

After a few iterations, it would stop at a certain "i" value, cpu usage would be 100%, and even running "java -version" in another terminal would seem to wait for something on the looping instance.

A bit surprising, because I was expecting JVM's to be fully independent; and no, it wasn't due to system load, because everything else ran ok.

Anyway, hardware:

Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz
GeForce GT 430
Gigabyte P67A-UD3-B3

Offline

#13 2012-11-02 16:14:58

Spacenick
Member
From: Germany
Registered: 2010-04-02
Posts: 168

Re: JVM fails to allocate 1 GB of byte-arrays

Ok so you guys have the exact same CPU and you're both using an Nvidia graphics card. Could you try reproducing with the Noveau graphics driver or even Vesa because nvidia might introduce bugs in the kernel. Also try with/without HyperThreading

Offline

#14 2012-11-02 17:17:12

rmetzger
Member
Registered: 2012-10-04
Posts: 13

Re: JVM fails to allocate 1 GB of byte-arrays

Okay, I uninstalled the nvidia driver and used nouveau. The problem still occurs!

UPDATE: It seems that disabling Hyper Threading for my CPU solves the issue. But that's not really a solution.
Another feasible workaround is assigning the jvm to only one CPU (using taskset) (even with HT on). That underlines that the problem is related to multicore memory allocation.

UPDATE 2: The newest 3.6.5 kernel does not solve the issue.

Last edited by rmetzger (2012-11-02 20:18:16)

Offline

#15 2012-11-02 20:23:30

Spacenick
Member
From: Germany
Registered: 2010-04-02
Posts: 168

Re: JVM fails to allocate 1 GB of byte-arrays

Ok I think we're looking at a serious kernel bug here. I think it's time for a mail to the lkml, as the kernel bug zilla is, well, problematic.
Might as well be glibc though

Last edited by Spacenick (2012-11-02 20:24:39)

Offline

#16 2012-11-05 16:44:03

rmetzger
Member
Registered: 2012-10-04
Posts: 13

Re: JVM fails to allocate 1 GB of byte-arrays

Do you know if the JVM uses glibc for memory allocation?
If not, the problem is independent of glibc.

I should mention that I can see from htop, that the JVM (or my little test program that uses glibc) is spending 100% of the cpu time in kernel mode (the cpu bars in htop are red for all cpus)

The problem occurs as well, if I assign only four cpu's to the JVM, then the 4 cpu's are at 100%.

I'd like to have a reliable way to reproduce the bug prior to posting to the lkml. On the other hand, I don't know the culture of the list, so I can not decide if it's okay to post there.

Offline

#17 2012-11-05 17:21:03

rmetzger
Member
Registered: 2012-10-04
Posts: 13

Re: JVM fails to allocate 1 GB of byte-arrays

Okay .. Spacenick is probably right. It seems to be a kernel bug.

After playing around a bit, I found out, that sometimes (when using not all my CPUs), khugepaged uses 100% CPU as well. This lead me to https://bbs.archlinux.org/viewtopic.php?id=119644, and then to https://lkml.org/lkml/2011/4/20/435.
I disabled khugepaged with

echo never >/sys/kernel/mm/transparent_hugepage/enabled

and the problem does not occur anymore. Neither with java, nor with my c code

http://bugs.centos.org/view.php?id=5716
http://marc.info/?l=linux-mm&m=135179005510688

Last edited by rmetzger (2012-11-27 09:48:32)

Offline

Board footer

Powered by FluxBB