How to get heapdump for any microservice running in Docker container in Google App Engine

Getting heapdump for our services is very important for doing any memory profiling to debug any leaks or performance issues. Unfortunately it is not that straight forward. Fortunately, it is not impossible either!

We need to SSH in the container in which our service is running. Go to App Engine → Instances page and select your service from the 'services' dropdown.

SSH into any of the instances:

Ignore this and SSH into VM.

Once you SSH, you should be able to see cloud shell as below. If you are logging in for the first time, there may be some questions asking for SSH key. You can leave the key blank and continue.

This shell is the Linux VM on which our docker image is running. We need to find out more details about it before we can do anything with it. 

There are a bunch of images running. We are interested in the first one ('us.gcr.io/.....') running in a container named 'gaeapp'. Now we need to go inside this docker container running our service. Below command executes '/bin/bash' command in the container and we get a bash shell inside container. (Call it VMception!)

Create a directory here to be able to store the dump in and give it all access.

mkdir dump_dir.

chmod 777 dump_dir

We will use it later.

Now in this VM we can check the processes running and the users running those. Give 'top' command on the shell and you see this:

As we can see 'jetty' user is running a java process. This is what we are interested in. We would now try to get heapdump of this process. For this we need to connect as the user 'jetty'. Currently we are logged in as user 'root' and we can't take the dump. Press Ctrl+C to exit this screen and get back to the shell. 

su - jetty

/bin/bash

We get to the shell now as 'jetty'.

Goto the directory we created earlier.

cd jda/scripts/dump_dir

All set to take the dump! Here we need to give the PID of the java process we found above (50 in this example)

jmap -dump:live,file=dump_dir/ 50

If all is well, you should be able to get the hprof file created in this directory. Stage 1 is now over. Stage 2 is getting this file to your VM.

Write 'exit' on the bash shell to exit from the bash shell of jetty user.

Write 'exit' again on the prompt to exit from the jetty user.

Write 'exit' again on the bash shell to exit from the root user and from the docker.

Now you are on the VM where you logged first. Copy the file to this vm file system from the docker file system and give it all access.

docker cp gaeapp:/jda/scripts/dump_dir/ .

chmod 777 filename.prof

Stage 2 is now over. Stage 3 is now getting this file to your local machine. (I know, this seems like a never ending process but please stay with me!)

On a shell on your local machine give following commands. Second command is optional if you have already a default project configured.

gcloud auth login

gcloud init

gcloud app instances scp --service  --version  :/<filename> 

For example, in my case above command looks like this:

gcloud app instances scp --service metadata-service --version v171 aef-metadata--service-v171-qhl0:/home/abhishek_asthana/dump_hprof.bin dump.hprof

And with this Stage 3 is also over. You have finally got the hprof file on your machine and can now use any memory profiler for analysis.