Insufficient memory in swisscom cloudfoundry springboot app

Question

I have a SpringBoot app which only needs max. 284 MB memory. But I can only start the app with max. 768 MB memory. Even if I reduce the memory later I always get the following error:

[APP/PROC/WEB/0] ERR Cannot calculate JVM memory configuration: There is insufficient memory remaining for heap. Memory limit 384M is
less than allocated memory 672509K (-XX:ReservedCodeCacheSize=240M, -XX:MaxDirectMemorySize=10M, -XX:MaxMetaspaceSize=109309K, -Xss1M * 300 threads)

I am already using the same app in cf without this problem. What has changed in cf? Before 2 or 3 weeks it still worked.

Thank you in advance for your quick answer!

Answer 1

I am already using the same app in cf without this problem. What has changed in cf? Before 2 or 3 weeks it still worked.

You must have upgraded to Java Build Pack v4. With JBP v3, you could push apps with smaller amounts of memory, but app that were pushed with that version of the buildpack and small memory limits were also very prone to crashing. The reason for this is because JBP v3 did not take into account all of the regions of memory used by the JVM when the JBP did it's calculations, so while it would let your app start, the app could still go over it's memory limit later. The way JBP v4 calculates memory is more accurate now, and take all of the JVM's memory regions into account.

The net result is that with JBP v4 apps that run with less than 1G of RAM can see the following error at staging, and apps that run with 512M or less will almost certainly see this error at staging:

ERR Cannot calculate JVM memory configuration: There is insufficient memory remaining for XXXX

However, Java apps that to do successfully stage are significantly less likely to crash.

https://discuss.pivotal.io/hc/en-us/articles/115011717548-Insufficient-memory-when-using-Java-Buildpack-4-0-

So does this mean you can't run a Java app with less than 1G of RAM? No. You can, but it's going to require some tuning. Here is a list of some things you can tune to save some memory.

• Thread count. The JBP assumes your application will run with 250 threads. This is pretty common for a Web application, which will have about 50 for Tomcat's internal stuff and 200 for request handling. You can lower this value, but you need to make sure your app does not exceed whatever number of threads you set, otherwise your app can crash. For you, this would adjusting Tomcat's configuration through Spring Boot.

  Example of lower thread count for the JBP:

  cf set-env my-application JBP_CONFIG_OPEN_JDK_JRE '{ memory_calculator: { stack_threads: 50 } }'

  Exaple of lower thread count in Spring Boot application.properties:

  server.tomcat.max-threads=25

  Note: this is just the request processing threads. Tomcat itself has threads and your app may create threads too. You should really look in a thread dump or JVisualVM to see how many threads your app requires (i.e. measure, don't guess)

• Thread stack size (-Xss). This is a value passed through to the JVM which controls how much memory is required per thread. It defaults to 1M, which is very high. In most cases, you can safely lower it to 160K, which is the minimum allowed with Java 8. If you have 250 threads and make this change, you'll save (1M * 250) - (160K - 250) = 211M of RAM.

  Note: If you lower the thread stack size too much, you'll see StackOverflow Exceptions in your app. If that happens, keep calm and raise the value up until they go away.

• Lower the Reserved Code Cache. This is where the JVM caches JIT'd code. It defaults to 240M. You can lower this value, but do so with caution as this can absolutely impact performance. You can also see errors at runtime if this value is not high enough.

  Again, it's better to measure what your app requires rather than guess. You should be able to measure by using the JVM's NMT.

• You can manually adjust the other areas of memory used by the JVM like Heap & Metaspace. With JBP v4, it's as easy as setting the JAVA_OPTS environment variable with cf set-env or in your manifest.yml file. Obviously, if you lower these values you need to be careful that you don't go too low, in which case you'll end up with OutOfMemoryErrors.

  Note: If you leave one region unset, like heap or metaspace the JBP will smartly calculate it with whatever is remaining after your manual customizations.

Hope that helps!

Answer 2

I am not an expert in the memory handling of java apps in cloud foundry, but from what I have figured out one needs to know the following:

• Java apps are normally deployed via the Java Buildpack. If you don't define a Version in your manifest.yml it will take the master branch. Since the master branch changes all the time, I assume deployment parameters might change as well.
• The Java Build pack uses the Java Buildpack Memory Calculator to evaluate the JVM memory parameters like -Xmx. You could configure those parameters yourself in your manifest.yml, but it is not recommended if you want to use the scaling features of cloud foundry.
• So - when you define that your cloud foundry app is allowed to use 500MB memory and the build pack calculator evaluates more, it might not work and lead to some errors.

What you can do in your manifest is to play around with all those parameters mentioned in the docs of the Memory calculator. For example you can give the calculator a hint, hat you need less threads (e.g. memory_calculator: { stack_threads: 100})

When you are fine with "hardcoding" some memory parameters: here an example manifest.yml for my Spring Boot 2 / Java 9 app which I use with a limit of 400 MB. It works for me, but is probably not something to copy right away. It depends on your app.

applications:
- name: demo-app
  path: target/demo-app-1.0.0.jar
  instances: 1
  buildpack: https://github.com/cloudfoundry/java-buildpack.git
  memory: 400m
  env:
    JAVA_OPTS: '-XX:MaxMetaspaceSize=80780K -Xss512k -Xmx200M -XX:ReservedCodeCacheSize=16M -XX:MaxDirectMemorySize=16M'
    JBP_CONFIG_OPEN_JDK_JRE: '{ jre: { version: 9.+ } }'