The software that we are likely to explore in this article was managing on Elastic Beanstalk (EBS) support in Amazon Web Providers (AWS). Intermittently this application was throwing an HTTP 502 Bad Gateway error. In this publish, let’s examine how we troubleshoot and take care of this HTTP 502 undesirable gateway error in the Elastic Beanstalk service.
AWS Elastic Beanstalk Architecture
This application was operating on AWS Elastic Load Balancer, Nginx 1.18., Java 8, Tomcat 8, and Amazon Linux in AWS Elastic BeanStalk support (EBS). For the people who are not that acquainted with EBS, underneath is its higher-degree architecture.
Fig: AWS Elastic Beanstalk architecture
There is an AWS elastic load balancer in the forefront. This Load Balancer will distribute the website traffic to a set of EC2 cases (which can be vehicle-scaled). Every single EC2 occasion will have an Nginx website server and a Tomcat software server. Requests sent by the Elastic Load Balancer are first managed by the Nginx server. Then the Nginx server forwards the ask for to the tomcat server.
HTTP 502 Lousy Gateway Mistake
Intermittently (not often), this application was throwing HTTP 502 bad gateway problems. A number of seconds later the moment all over again support will resume and items will start to perform generally. It wasn’t clear what was triggering this HTTP 502 poor gateway mistake in the AWS Elastic Beanstalk atmosphere.
We initially have to have to fully grasp what this HTTP 502 negative gateway error usually means. This error is thrown by a website server/gateway/proxy server when it gets an invalid reaction from the backend stop server to which it is conversing.
HTTP 502 Thrown by Nginx in AWS EBS
Now the issue is: There are three key elements in the EBS stack:
- Elastic Load Balancer
- Nginx net server
- Tomcat Application server
In these 3 parts which 1 is throwing HTTP 502 undesirable gateway error?
Fig: Screenshot of the HTTP 502 Bad gateway mistake thrown by Nginx server
Higher than is the screenshot of the HTTP 502 terrible gateway error that we had been receiving. There is a clue in this screenshot to point out who is throwing this HTTP 502 mistake. If you observe the highlighted element of the display, you will see this HTTP 502 bad gateway mistake to be thrown by the Nginx server.
As for every the HTTP 502 error definition, Nginx should really be throwing this mistake only if it would have obtained an invalid response from the tomcat server. Hence, this clue helped to slender down that the Tomcat server is the resource of the problem.
Out of Memory: Destroy System or Sacrifice Youngster
In get to discover the resource of the challenge, we executed the open-resource yCrash script on the EC2 occasion in which the tomcat server was functioning. yCrash script captures 16 different artifacts from the technological innovation stack, which features: Rubbish Assortment log, thread dump, heap dump, ps, top, best -H, vmstat, netstat, ….. We uploaded the captured artifacts into the yCrash server for analysis.
One of the artifacts that yCrash script captures is a kernel log file. In this log file, all the events that transpired in the Linux kernel can be located. yCrash pulls out essential glitches and warnings from the log file & offers them. Under is the evaluation report of the kernel log produced by the yCrash.
Fig: yCrash’s Kernel log examination reporting ‘Out of memory: get rid of the method or sacrifice child’
Please see the highlighted error message in the kernel log:
[Sat May 21 17:31:00 2022] Out of memory: Get rid of method 24339 (java) rating 874 or sacrifice boy or girl
It indicates that the Tomcat server which is a Java system was terminated. Linux kernel will terminate procedures if their memory consumption exceeds the device’s RAM ability limit. This is the specific situation going on in this software as well. When the application’s memory use goes beyond the ability limitations, the Linux kernel was terminating the tomcat server.
Root Induce – Lack of RAM
Now the query is: How is Linux terminating the tomcat server that can result in intermittent HTTP 502 terrible gateway mistake? Shouldn’t a finish outage need to have to happen? It is a truthful issue.
If you recall, this application is managing on AWS Elastic Beanstalk (EBS) support. EBS services will quickly restart the tomcat server any time it will get terminated. Consequently it’s hilarious. Linux is terminating and EBS is restarting the tomcat server. In the course of this intermittent interval, shoppers have been experiencing HTTP 502 negative gateway glitches.
Solution – Upgrading EC2 Occasion RAM potential
Evidently, the software was jogging on EC2 cases which had only 1GB RAM capacity. It wasn’t enough memory to operate the tomcat server, Nginx server, and other kernel procedures. Consequently, when the application was upgraded to run on a 2GB RAM capacity EC2 instance, the difficulty obtained settled.
Observe: Listed here is a comparable challenge confronted by one more software that was operating in a non-AWS surroundings. It may possibly be an appealing go through as nicely.