Working as a freelancer, I spend my time on different projects. During the last one, like many others, I had to design and build a CRUD application. My usual approach is to use the multi-layered architecture described here.
I’ve never had problems, but this time it was different. Due to performance issues, I had to push my back-end to the limit. Extremely complex queries, multiple levels of parallelization, bigger data, and more requests than usual. Although I don’t call myself a Spring Boot expert, I’m pretty confident about how to use it. However, every 3 or 4 days, an OutOfMemoryError occurred.
This is the story of how I fixed it, improving performance as a side effect, and what this taught me.
Specs and Architecture
The Spring Boot web application is running on a Docker container handled by Dokku. I have been using this deployment system for years and nothing wrong has ever happened, so I am pretty sure my problem does not stand here.
The 4GB 2 CPU VPS I am using is enough for such a project, especially considering that we do not expect so many users.
My Procfile is very simple and scaled on available resources:
web: java -Xms1024M -Xmx4096M -jar build/libs/my-app-1.0.jar
- 1GB as initial heap size [
-Xms1024M] - 4GB as maximum heap size [
-Xmx4096M]
The database is a MySql server running on another VPS, so they are not sharing the same resources.
The Recurring Error
The back-end runs flawlessly for about two weeks. Day by day, data becomes bigger, and users making concurrent requests increase. During the third week, I get the first OutOfMemoryError. Logs are pretty clear: the VPS failed while trying to create a new thread, running out of memory. I immediately ask for advice from Dario Pellegrini, a colleague and developer with 5+ years of experience. Together we decide to double the RAM of the VPS and the maximum heap size limit consequently.
Problem fixed, right?
Nope!
3 days later, the same OutOfMemoryError exception strikes again! This time I want to tackle the problem for real and look for a real solution.
These are my attempts to fix it…
1. Trying a Practical Solution
Running some tests I find out that a query returning 20k rows can easily become the bottleneck of an application. This is especially true if the @Entity mapping class consists of more than 130 attributes (just like mine). Instead of retrieving 10/20k rows each time, mapping them into Hibernate objects, iterate through them to retrieve a single computed value, I replace it all with a very advanced query. By defining a custom mapping class and writing a highly optimized query, I am able to achieve the same result with a much more efficient single-row query. With this (not so) simple trick, I increase the performance by 70/80% and reduce significantly the heap size.
Not to brag, but this time I’m pretty sure I’ve solved the problem…
Not yet…
2. Looking for Esoteric Reasons
Since the story is still going on, I’m not proud to say that it didn’t work out. 6 days later my back-end fails for the same OutOfMemoryError.
I decide to take it seriously this time. I look for someone with a similar issue online and come across some interesting, but somewhat esoteric, topics.
Firstly, a couple of APIs are really complex and require hundreds of queries. Then, the query plan cache provided by Hibernate may be the culprit. By default, the maximum number of entries in the plan cache is 2048. An HQLQueryPlan object occupies approximately 3MB. ~3 * 2048 = ~6GB, and the heap size is limited to 4GB…
Finally! That must be the cause!
The solution is simple: decrease the query plan cache size by setting the following properties:
-
spring.jpa.properties.hibernate.query.plan_cache_max_size: controls the maximum number of entries in the plan cache (defaults to 2048) -
spring.jpa.properties.hibernate.query.plan_parameter_metadata_max_size: manages the number ofParameterMetadatainstances in the cache (defaults to 128)
I set them to 1024 and 64 respectively.
Secondly, I find out that IN queries with a variable number of parameters are a bit tricky for the Hibernate query plan cache (further reading here and here).
The solution is easy again. You can improve the statement caching efficiency with IN clause parameter padding by setting the following property to true:
spring.jpa.properties.hibernate.query.in_clause_parameter_padding
Thirdly, my application is multi-thread. Since the error seems to occur while it tries to create a new thread, why not reduce thread stack size from 1024KB to 512KB by adding -Xss512k to my Procfile?
Each of these properties should decrease the amount of heap used and therefore fix the OutOfMemoryError problem, once and for all. Right? Nope!
3. Surrendering to Reality
Last but not least, I decide to do what I should have done immediately: review my code! The JVM Garbage Collector gathers all the unreachable objects to try to delete them. In the case of too many unreachable objects, the heap will fill up before the GC intervenes. In this case, an OutOfMemoryError exception is thrown as soon as the application tries to allocate a new object. Reviewing my code I find nothing suspicious.
My application is multi-thread and I used thread pools more than once, especially ForkJoinPool. I have not used it for ages, so the problem could be there. I started looking online for well-known memory leaks and during my researches, I come across this interesting GitHub page. I immediately realize what a beginner mistake I made. I forgot to shut down each of the ForkJoinPool objects I used!
Oops!
Conclusion
This experience taught me how complex problems can have very simple solutions. Instead of stubbornly looking for any esoteric reason or insufficient resources, you should consider trivial errors first. I hope this story helps you not to make the same mistake I did.
