Skip to main content

How to get heapdump for any microservice running in Docker container in Google App Engine

Getting heapdump for our services is very important for doing any memory profiling to debug any leaks or performance issues. Unfortunately it is not that straight forward. Fortunately, it is not impossible either!
We need to SSH in the container in which our service is running. Go to App Engine → Instances page and select your service from the 'services' dropdown.


SSH into any of the instances:

Ignore this and SSH into VM.

Once you SSH, you should be able to see cloud shell as below. If you are logging in for the first time, there may be some questions asking for SSH key. You can leave the key blank and continue.

This shell is the Linux VM on which our docker image is running. We need to find out more details about it before we can do anything with it. 

There are a bunch of images running. We are interested in the first one ('us.gcr.io/.....') running in a container named 'gaeapp'. Now we need to go inside this docker container running our service. Below command executes '/bin/bash' command in the container and we get a bash shell inside container. (Call it VMception!)
Create a directory here to be able to store the dump in and give it all access.
mkdir dump_dir.
chmod 777 dump_dir
We will use it later.
Now in this VM we can check the processes running and the users running those. Give 'top' command on the shell and you see this:

As we can see 'jetty' user is running a java process. This is what we are interested in. We would now try to get heapdump of this process. For this we need to connect as the user 'jetty'. Currently we are logged in as user 'root' and we can't take the dump. Press Ctrl+C to exit this screen and get back to the shell. 
su - jetty
/bin/bash
We get to the shell now as 'jetty'.
Goto the directory we created earlier.
cd jda/scripts/dump_dir
All set to take the dump! Here we need to give the PID of the java process we found above (50 in this example)
jmap -dump:live,file=dump_dir/ 50
If all is well, you should be able to get the hprof file created in this directory. Stage 1 is now over. Stage 2 is getting this file to your VM.
Write 'exit' on the bash shell to exit from the bash shell of jetty user.
Write 'exit' again on the prompt to exit from the jetty user.
Write 'exit' again on the bash shell to exit from the root user and from the docker.
Now you are on the VM where you logged first. Copy the file to this vm file system from the docker file system and give it all access.
docker cp gaeapp:/jda/scripts/dump_dir/ .
chmod 777 filename.prof
Stage 2 is now over. Stage 3 is now getting this file to your local machine. (I know, this seems like a never ending process but please stay with me!)
On a shell on your local machine give following commands. Second command is optional if you have already a default project configured.
gcloud auth login
gcloud init
gcloud app instances scp --service  --version  :/<filename>
For example, in my case above command looks like this:
gcloud app instances scp --service metadata-service --version v171 aef-metadata--service-v171-qhl0:/home/abhishek_asthana/dump_hprof.bin dump.hprof
And with this Stage 3 is also over. You have finally got the hprof file on your machine and can now use any memory profiler for analysis.

Labels:

Comments

Post a Comment

Popular posts from this blog

How I converted a table in an email to a graph on a website? Hint: Azure Functions and Logic App

This mini-project started as a necessity.  I get an email like this everyday, which gives daily numbers for different metrics. The problem with such an email is that it is difficult to keep track of or gauge the rate of growth of these metrics. If you want to know how what the previous day's numbers were you need to go to the previous day's email.  The better option any day is to have a graph that shows the values of the metrics every day. Something like this: With this goal I set out to make this system but with as less coding and expenses as possible.  First obvious step in the process was to get hold of the email and extract its text. The easiest way to do this is to use Azure Logic App.  Here is what my logic app looks like: When any new email arrives in my Outlook 365 account, with a subject that contains "System usage Stats as on", it passes the body of the email to an Azure function named 'EmailTextCleanerFunction' (code below), whose job is to clean an...
Post a Comment
Read more

Changing Eclipse Workspace Directory

Recently I moved my entire Eclipse installation directory but the workspace was still getting created in the older location only. And worst there was no option to select the Workspace directory in the Window->Options->Workspace menu. To change the workspace location in Eclipse do this. Goto ECLIPSE_HOME\configuration\.settings directory, edit the org.eclipse.ui.ide.prefs file and change the RECENT_WORKSPACES value to the desired location. If you want that Eclipse prompts you to select workspace when you start it, change the SHOW_WORKSPACE_SELECTION_DIALOG value to true. And you are done!
10 comments
Read more

Hands on review: Yahoo! Mail app for Windows 8

Before you start mocking me for using Yahoo! Mail, let me explain that this is NOT my primary mail account and I use Gmail and Outlook.com for my daily use. Yes, this Yahoo! account was once my daily haunt but that was during college days and those days are long gone. I don't use Yahoo! Mail much but still there are some group subscriptions and I just log in once in a while to see what's happening in those forgotten groups. And due to this reason, I have always kept track of changes in Yahoo! Mail and the fact is that you can blame them for everything but not really lack of trying. Though most of those tries just ended up cluttering the UI and adding somewhat needless features. However, now we hear that new CEO Marissa Mayer is focusing once again on the mail and first major product to come out is Yahoo! Mail app for Windows 8. So how is it? The Login screen is pretty neat and purple background with an envelope watermark is very pleasing, very Yahoo!. However why ...
7 comments
Read more