One of the things I enjoy most is diagnosing storage latency. I honestly couldn’t tell you why I enjoy it so much; but, I just do. When it comes to storage in Windows one of the best things you can do is capture an ETL trace for the STORPORT driver.
STORPORT is the driver that handles incoming and outgoing IO (IRPs). PERFMON is a great tool as well but it’s higher in the driver stack and doesn’t track as much as STORPORT.
The benefit with PERFMON is that the data is presented in a nice, easy to interpret graph and STORPORT only gives gobs and gobs of data viewable in Event Viewer. This is OK for 100 events but these traces often contain hundreds of thousands.
To read a STORPORT trace you’ll want to open it in Event Viewer. Here we can see that there are 715,615 IO events that were greater than 75ms. How do I know 75ms? It’s what I set the minimum threshold to.
So how do you quickly filter through 715k events to see what’s going on?
You could thumb through the data and get a general idea; but, this approach isn’t very accurate. Despite this it’s what I’ve typically done in the past.
I’ve been working with a client recently that has some fairly high latency affecting SQL Server workloads and over the weekend I decided to have a little programming fun in C# WinForms.
The idea is that I can’t possibly thumb through all 715k records in any reasonable time and Event Viewer isn’t helping out this dilemma. To assist my efforts and again just to have some good ole fashion weekend programming fun, I created a tiny C# WinForms app to summarize a STORPORT ETL trace into a datagridview and .NET chart.
Using this tool will help me and hopefully you get to root cause quicker for your SQL Server / Windows storage latency woes.
Here’s a quick peek at the tool:
It’s been tested with files up to 200 MB. Keep in mind that larger files will take much longer and there’s probably some optimization that could happen in the C# code itself. I’m a SQL consultant not a .NET developer.
You can find the entire C# project for the tool on Microsoft’s TechNet here:
and for reference here’s a link to STORPORT tracing:
With any luck, I’ll be blogging over the next week or two about storage for both OnPrem and Azure. Until then, have a great day!