Heap size limit in java

[Paul Dlug]

I originally posed to my question to freebsd-java. My issue was that I
was unable to set a java heap size of more than approx 2000m using java
(-Xmx2000m), it would fail with the error "COuld not reserve enough
space for object heap.

Panagiotis Astithas looked into this a bit further and found that it
was due to the JVM being unable to mmap 3gb or more of memory. His
explanation is below along with some of the background information.

Any help would be greatly appreciated. I imagine others have hit this
issue or will shortly. It's somewhat critical to getting large scale
java applications running on FreeBSD that the JVM be able to access
enough memory.

Thanks,
Paul


From: Panagiotis Astithas <***@noc.ntua.gr>
Date: May 24, 2004 3:57:27 AM EDT
To: Greg Lewis <***@eyesbeyond.com>
Cc: Paul Dlug <***@aps.org>, freebsd-***@freebsd.org
Subject: Re: Heap size limit

I'm trying to start a JVM with a large heap size and getting an error
java -Xms256m -Xmx2024m
Error occurred during initialization of VM
Could not reserve enough space for object heap
I tried a bunch of different sizes and anything under 2024m works
just fine so this appears to be a hard limit. I've tried changing a
few system parameters but can't seem to stumble on the right one. Can
someone help me out with this?

Obviously I'm a long way behind on freebsd-java, but I'm catching up
now...
My strong suggestion is that you're hitting the maximum amount of
memory
that your user can mmap. I'm not exactly sure what setting dictate
that,
but its almost certainly what you're running into.

Definitely:
ktrace java -Xms256m -Xmx2024m

[...]
15034 java CALL mmap(0,0x82800000,0,0x1042,0xffffffff,0,0,0)
15034 java RET mmap -1 errno 22 Invalid argument
15034 java CALL write(0x1,0xbfbfd670,0x2b)
15034 java GIO fd 1 wrote 43 bytes
"Error occurred during initialization of VM
"
15034 java RET write 43/0x2b
15034 java CALL write(0x1,0xbfbfd670,0x2e)
15034 java GIO fd 1 wrote 46 bytes
"Could not reserve enough space for object heap"
[...]


% grep 22 /usr/include/errno.h
#define EINVAL 22 /* Invalid argument */

%man mmap
[...]

[EINVAL] MAP_FIXED was specified and the addr argument was not
page aligned, or part of the desired address space
resides out of the valid address space for a user
process.
[...]


Now try changing vm.max_proc_mmap to an insanely large value:

# sysctl vm.max_proc_mmap=10000000000000
vm.max_proc_mmap: 26960 -> 2147483647

We see the requested size gets truncated to 2 GB.
This still doesn't help us, though:

# /usr/local/jdk1.4.2/bin/java -Xms256m -Xmx2024m
Error occurred during initialization of VM
Could not reserve enough space for object heap


The weird thing is that in a GENERIC kernel, the kernel/userland
address space split is 1 GB / 3 GB. As it can bee sen from the
following excerpt from /sys/i386/conf/NOTES:

#
# Change the size of the kernel virtual address space. Due to
# constraints in loader(8) on i386, this must be a multiple of 4.
# 256 = 1 GB of kernel address space. Increasing this also causes
# a reduction of the address space in user processes. 512 splits
# the 4GB cpu address space in half (2GB user, 2GB kernel).
#
options KVA_PAGES=260


Why you can't mmap up to 3 GB of memory, I don't know. Perhaps we
should take it to -current or -hackers.

Cheers,

[Andrew Milton]

+-------[ Paul Dlug ]----------------------
| I originally posed to my question to freebsd-java. My issue was that I
| was unable to set a java heap size of more than approx 2000m using java
| (-Xmx2000m), it would fail with the error "COuld not reserve enough
| space for object heap.
|
| Panagiotis Astithas looked into this a bit further and found that it
| was due to the JVM being unable to mmap 3gb or more of memory. His
| explanation is below along with some of the background information.
|
| Any help would be greatly appreciated. I imagine others have hit this
| issue or will shortly. It's somewhat critical to getting large scale
| java applications running on FreeBSD that the JVM be able to access
| enough memory.

I think it's pretty well summed up in the mmap manpage (if this is still
currently correct).

BUGS
The len argument is limited to 2GB. Mmapping slightly more than 2GB does
not work, but it is possible to map a window of size (filesize % 2GB) for
file sizes of slightly less than 2G, 4GB, 6GB and 8GB.

The limit is imposed for a variety of reasons. Most of them have to do
with FreeBSD not wanting to use 64 bit offsets in the VM system due to
the extreme performance penalty. So FreeBSD uses 32bit page indexes and
this gives FreeBSD a maximum of 8TB filesizes. It is actually bugs in
the file system code that causes the limit to be further restricted to
1TB (loss of precision when doing blockno calculations).

Another reason for the 2GB limit is that file system metadata can reside
at negative offsets.

[Charles Swiger]

On Jun 18, 2004, at 12:05 PM, Paul Dlug wrote:
[ ...not being able to mmap() more than 2 GB of memory... ]

Post by Paul Dlug
Any help would be greatly appreciated. I imagine others have hit this
issue or will shortly. It's somewhat critical to getting large scale
java applications running on FreeBSD that the JVM be able to access
enough memory.

One approach would be to split up the workload into two (or more)
JVM's, each of which using less than the current 2GB limit.

If that approach is not workable for your problem domain, then please
consider the fact that no matter what changes are made, a 32-bit
platform is never going to be able to give your JVM more than 4 GB. If
you really have a need to exceed 2GB JVM heap size, you should
seriously consider switching to a 64-bit platform. I don't know
whether Java+FreeBSD+AMD64 is stable enough for production, so Sun
SPARC or even a PowerPC-based Mac might be a better choice.

[Doug White]

Post by Paul Dlug
I originally posed to my question to freebsd-java. My issue was that I
was unable to set a java heap size of more than approx 2000m using java
(-Xmx2000m), it would fail with the error "COuld not reserve enough
space for object heap.

What do you have MAXDSIZ set to?

You're running into the limits of 32 bit memory addressing, probably.
FreeBSD reserves ~1.5GB of the 4GB address space for the kernel's use, and
gives the rest to userland. You can adjust this boundary with the
KVA_PAGES kernel compile option, but you have to do some mean tuning to
avoid running out of KVM. I have nameservers set up this way, but named
is the only thing running on the system.

On 64 bit platforms (sparc64, amd64), the limit is far enough away that
you won't encounter this specific problem.

The command line in question is:
# /usr/local/jdk1.4.2/bin/java -Xms256m -Xmx2024m