Setup

You’ll need a couple of things to get started. First, make sure you’ve enabled RDP access to your Azure roles. It’s pretty easy to set up through Visual Studio, and there are plenty of walkthroughs already, so I’m not going to cover it. If you didn’t have RDP set up already, then re-deploy your application with RDP enabled.

Second, you need WinDbg which is part of the Debugging Tools for Windows. WinDbg is not as easy to use as some of the commercial products, but if you’ve seen a command line before then you’ll be fine. It also has the notable advantage of being free. You can get the installer here. It does take a little one-time configuration – I recommend the Basic Configuration section of this post.

Collecting a Memory Dump

Once the setup is out of the way, you need to collect a memory dump. To do that, you need to RDP into your machine, but with a slight settings change that’ll help you download the dump file.

Go to the Azure portal, select an instance of your role, and hit the Connect button. That should download a .RDP file to your machine. Don’t open it immediately, we want to change settings to share your desktop’s drive to the Azure VM. Go find the .RDP file on your disk, right click it and select “Edit”. Go to the “Local Resources” tab, click the “More” button, and share one of your drives. It’ll look something like this:

Changing settings for RDP

Now connect to the VM as normal. Once you’re logged in, open Windows Explorer and you verify that your desktop drive shows up in the list of drives. Then open Task Manager and find the leaking process. If you’re debugging a worker role, it’ll probably be WaWorkerHost.exe. For web roles, it will probably be w3wp.exe. (Or you could just look for the one that’s using a whole lot of memory!) When you’ve found the misbehaving process right click on it and select “Create Dump File”.

This will take a while. When it’s done you’ll see a dialog telling you where to find the dump file. Locate the dump file, then copy it to the drive you shared from your desktop. Dump files tend to be very large, so this will probably take a few minutes.

Analyzing the Dump File

Now you need crack open the dump file and see what’s going on. The dump file contains a snapshot of all the memory in your program. We’re going to look through it to figure out what’s taking up all that memory. You’re basically looking for anything suspicious, such as a class that has thousands more instances than it should. Open WinDbg and use File -> Open Crash Dump to open the dump file you just collected. Then type “.load sos” to load the .NET debugger module.

Now you need to get out your magnifying glass and detective hat, and try to figure out what’s going on in your program. I’m certainly no expert on this topic, but this approach worked for me.

1. Run !dumpheap -stat. This will dump out a summary of what classes are using the most memory, with the biggest ones at the bottom. Look for anything that seems suspicious.
2. When you find something suspicious, copy the MT value (the first column), then run !dumpheap -mt <copied mt value>. This will dump out a list of all the instances of that object.
3. Copy the Address (first column, again) for an object, and run !gcroot <copied address>. This should show you what objects are referring to that object instance. It should give you an idea of who created it, and why it hasn’t been reclaimed. Repeat the process for several objects, and look for patterns.

Hopefully the above gives you some ideas. If not, try taking two or three dumps and look at what’s been allocated in the time between them. If you’re still stuck, Tess Ferrandez wrote a very good and still relevant blog post series about this topic. I particularly recommend part 6 and this followup post for ASP.NET problems.

To demonstrate the problem I have set up my Glassboard API instance to do no caching whatsoever. I then ran a unit test which simulates a user getting his Newsfeed, then posting a status. I highlighted each set of repeated calls in a different color.

Just look at all those colorful, repeated requests! There are 8 requests there, slowing down our app and sucking up money.

Depending on your application there are many possible ways to get rid of repeated requests. You may be able to design them out of the system, share data between layers, or implement local or distributed caches. We used a combination of all of these to fix our problem in Glassboard. With our caching re-enabled, the same test results in this HTTP trace.

Good morning. We’re Sepia Labs. Nice to meet you too.

You don’t know us because we’re new. Well, sort of new. Sepia Labs is a software company comprised of a few folks who have some history in the software business, notably Brent Simmons, the creator of NetNewsWire, and Nick Bradbury, creator of FeedDemon and TopStyle. Together with Nick Harris, Jenny Blumberg, Brian Reischl and myself (Walker Fenton) – we have started this new venture called Sepia Labs.

We are spinning off from NewsGator Technologies, Inc, we are the team that was largely responsible for NewsGator’s consumer facing business.

Our mission at Sepia Labs is simple. First and foremost, we want to make great software. We want our software to provide instant utility, to be good looking, fast, and fun to use. Our first product, Glassboard, reflects this mission, and we hope you’ll agree.

Glassboard is an app focused on sharing. Yes, a bit like Facebook or Twitter, but we take privacy seriously. “Know who you’re sharing with” is the tagline. And we mean it.

Glassboard is launching in July. If you would like to keep posted, head over to Glassboard.com and give us your email. We won’t spam you or share your email with anyone else. We’ll just let you know when the app is ready.

Thanks for visiting.