Rootkit Debugging (runtime2 postmortem) - SwishDbgExt, SysecLabs script, etc.

Today we're going to be doing some rootkit debugging, specifically regarding runtime2, with a bit of a twist! I have a ton of rootkit debugging posts coming in the next few weeks, as I've decided to break them up rather than throwing them together in one giant mess of a post.

I've shown various scenarios in which I've debugged a rootkit before (0x7A, etc), but this time we're going to use various extensions to help us, other methods, and overall go a lot more in-depth. The postmortem runtime2 rootkit KMD that will be used in this post was generated by niemiro, a good friend from Sysnative forums. He aimed to make it a good example of some things a rootkit/malware developer can do to make things not as obvious when you resort to methods such as hooking the SSDT, which is rather old and very detectable these days.

 CRITICAL_OBJECT_TERMINATION (f4)  
 A process or thread crucial to system operation has unexpectedly exited or been  
 terminated.  
 Several processes and threads are necessary for the operation of the  
 system; when they are terminated (for any reason), the system can no  
 longer function.  
 Arguments:  
 Arg1: 00000003, Process  
 Arg2: 86664d90, Terminating object  
 Arg3: 86664edc, Process image file name  
 Arg4: 819e91f0, Explanatory message (ascii)  

Right, so here's our bug check. Most if not all of these 'older' rootkits will use Direct Kernel Object Manipulation (DKOM) to hook low-level routines/functions within the System Service Dispatch Table (SSDT). When this is occurring, assuming the developer of the rootkit didn't do a very good job in writing their rootkit, a lot can go wrong when disabling write protection, carelessly swapping memory, and inserting hooks. Malware scans from many AV programs can also cause crashes when they detect that the SSDT is hooked under certain circumstances. The rootkit can also intentionally call a bug check if written this way when it has detected a scan has initiated.

However, if the driver is well written, you may not crash at all. What do you do if you're suspicious of a rootkit infection/hooking, yet a bug check isn't occurring naturally due to proper programming of the rootkit? This is when in some cases (like in this specific example here), you may need to take matters into your own hands and either:

• Force a bug check.
• Live kernel debugging session.
• Run an ARK (Anti-Rootkit) tool. This is no fun... we want to have fun and learn : )

The first option is much less effective than the second, and that's due to the fact that the rootkit may not be doing any obvious hooking of the SSDT, etc, at the time of you forcing the crash. If you're investigating during a live session, it's much more effective and there are a lot more commands you can use. However, for learning purposes, I will show both.

1st argument - The value in the 2nd argument (0x3) implies it was a process as opposed to a thread that unexpectedly terminated. If it was a thread however, it would have instead been 0x6.

2nd argument - The value in the 2nd argument (86664d90) is a pointer to the _EPROCESS object that unexpectedly terminated. We can dump it using !object, which I will show below:

 kd> !object 86664d90  
 Object: 86664d90 Type: (841537e8) Process  
   ObjectHeader: 86664d78 (old version)  
   HandleCount: 4 PointerCount: 127  

3rd argument - The value in the 3rd argument (86664edc) is the process image file name. Essentially, it's the process name that unexpectedly terminated. We can dump it by using dc, which I will show below:

 kd> dc 86664edc  
 86664edc 73727363 78652e73 00000065 00000000 csrss.exe.......  
 86664eec 00000000 00000000 00000000 8605a278 ............x...  
 86664efc 869356d8 00000000 00000000 0000000a .V..............  
 86664f0c 8c04d631 00000000 00000000 7ffdc000 1...............  
 86664f1c 00000000 00000099 00000000 00000000 ................  
 86664f2c 00000000 000005c6 00000000 0003cc50 ............P...  
 86664f3c 00000000 00000000 00000000 00006d77 ............wm..  
 86664f4c 00000000 00000000 00000162 00000000 ........b.......  

The process that unexpectedly terminated was csrss.exe, which is the Client/Server Runtime Subsystem.

For some extra food for thought, we can get a lot of the information above manually if we'd like to. If we take the 2nd argument (_EPROCESS object) and run the following:

 dt _EPROCESS 86664d90 imageFileName  

dt will display the type, which will show us the offset, etc. More specifically it displays information about a local variable, global variable or data type:

 kd> dt _EPROCESS 86664d90 imageFileName  
 nt!_EPROCESS  
   +0x14c ImageFileName : [16] "csrss.exe"  

If we take the _EPROCESS object and add it to the offset (+0x14c), we get the following:

 kd> ? 86664d90+0x14c  
 Evaluate expression: -2040115492 = 86664edc  

Look familiar? It's our 3rd argument, the process image name (csrss.exe).

4th argument - The value in the 3rd argument (819e91f0) is the explanatory message regarding the reason for the bug check, specifically in ASCII. To dump this, we'll need to use the dc command as we used earlier on the 3rd argument:

 kd> dc 819e91f0  
 819e91f0 6d726554 74616e69 20676e69 74697263 Terminating crit  
 819e9200 6c616369 6f727020 73736563 25783020 ical process 0x%   

As we can see, this is reiterating the 1st argument.

--------------------

So far all we know is that this bug check occurred because csrss.exe, a critical Windows process, unexpectedly terminated. Why? If you weren't able to tell whilst reading through by now, we purposely terminated it via Task Manager to force the bug check.With that said, for this post we're choosing the first method (forcing a bug check).

Pretending we ourselves didn't purposely force the bug check for a moment, what would we at this point do if this was a crash dump we were looking at from a system that isn't ours, and/or from our doing? Among many things, one of the first few things I do in 'not so obvious' 0xF4's is I check the summary and stats regarding virtual memory on the system at the time of the crash:

 kd> !vm  
 *** Virtual Memory Usage ***  
      Physical Memory:   917370 (  3669480 Kb)  
      Page File: \??\C:\pagefile.sys  
       Current:  3976680 Kb Free Space:  3976676 Kb  
       Minimum:  3976680 Kb Maximum:   4193280 Kb  
      Available Pages:   769568 (  3078272 Kb)  
      ResAvail Pages:    869504 (  3478016 Kb)  
      Locked IO Pages:      0 (     0 Kb)  
      Free System PTEs:   348283 (  1393132 Kb)  
      Modified Pages:    11884 (   47536 Kb)  
      Modified PF Pages:   11830 (   47320 Kb)  
      NonPagedPool Usage:   8265 (   33060 Kb)  
      NonPagedPool Max:   522998 (  2091992 Kb)  
      PagedPool 0 Usage:   5501 (   22004 Kb)  
      PagedPool 1 Usage:   10013 (   40052 Kb)  
      PagedPool 2 Usage:    623 (   2492 Kb)  
      PagedPool 3 Usage:    631 (   2524 Kb)  
      PagedPool 4 Usage:    726 (   2904 Kb)  
      PagedPool Usage:    17494 (   69976 Kb)  
      PagedPool Maximum:  523264 (  2093056 Kb)  
      Session Commit:     3758 (   15032 Kb)  
      Shared Commit:     9951 (   39804 Kb)  
      Special Pool:       0 (     0 Kb)  
      Shared Process:     1260 (   5040 Kb)  
      Pages For MDLs:      2 (     8 Kb)  
      PagedPool Commit:   17550 (   70200 Kb)  
      Driver Commit:     2335 (   9340 Kb)  
      Committed pages:   115256 (  461024 Kb)  
      Commit limit:    1882649 (  7530596 Kb)  

We can see right away that we don't have insufficient non-paged pool, which is a pretty popular cause of most 0xF4's as it at that point cannot handle I/O operations, etc. It's generally due to buggy drivers causing pool related memory leaks, etc.

With the above said, assuming we are looking at a crash dump that wasn't ours, we can almost entirely rule out this specific 0xF4 being caused by a buggy driver (MSFT or 3rd party). To be extra sure, double-check the modules list and see if there's anything that jumps out as problematic.

--------------------

At this point I would start becoming suspicious of a rootkit, as I would not right away suggest the hard disk (whether HDD or SSD) is the immediate problem given we'd probably see obvious NT_STATUS codes for that. For example, possibly 0xc0000006. Either that, or an entirely different bug check (perhaps 0x7A). With that said, now we get to have some fun!

There are many ways to go about detecting a rootkit hooking the SSDT, and we will discuss extensions/scripts for the moment:

1st Method - SwishDbgExt

The first method we will be using in this postmortem debugging example is the wonderful SwishDbgExt, which was developed/created by a friend of mine (Matt Suiche). I've made various contributions to the help file considering the love I have gathered for this extension. It makes a lot of our lives as debuggers much easier.

Once you have the extension loaded, we're going to be using the !ms_ssdt command. This command displays the System Service Dispatch Table, which is extremely helpful in the investigation of suspected rootkit hooks through using what is known as Direct Kernel Object Manipulation (DKOM).

-- Chopping some of the SSDT output as it's fairly large, let's skip to what's important:

   |-------|--------------------|--------------------------------------------------------|---------|--------|  
   | Index | Address      | Name                          | Patched | Hooked |  
   |-------|--------------------|--------------------------------------------------------|---------|--------|  
  *** ERROR: Module load completed but symbols could not be loaded for Gjglly.sys  
   |  126 | 0xFFFFFFFF91F054C2 | Gjglly                         |     |    |  
   |  127 | 0xFFFFFFFF81A34F80 | nt!NtDeviceIoControlFile                |     |    |  
   |  128 | 0xFFFFFFFF81949B44 | nt!NtDisplayString                   |     |    |  
   |  129 | 0xFFFFFFFF81A2117F | nt!NtDuplicateObject                  |     |    |  
   |  130 | 0xFFFFFFFF81A18134 | nt!NtDuplicateToken                  |     |    |  
   |  131 | 0xFFFFFFFF81AB14E8 | nt!NtEnumerateBootEntries               |     |    |  
   |  132 | 0xFFFFFFFF81AB278A | nt!NtEnumerateDriverEntries              |     |    |  
   |  133 | 0xFFFFFFFF91F04FFA | Gjglly                         |     |    |  
   |  134 | 0xFFFFFFFF81AB10B7 | nt!NtEnumerateSystemEnvironmentValuesEx        |     |    |  
   |  135 | 0xFFFFFFFF81A9F073 | nt!NtEnumerateTransactionObject            |     |    |  
   |  136 | 0xFFFFFFFF91F051B6 | Gjglly                         |     |    |  
   |  137 | 0xFFFFFFFF81A802D5 | nt!NtExtendSection                   |     |    |  
   |  138 | 0xFFFFFFFF819A113A | nt!NtFilterToken                    |     |    |  
   |  139 | 0xFFFFFFFF819B39FC | nt!NtFindAtom                     |     |    |  
   |  140 | 0xFFFFFFFF819DCA86 | nt!NtFlushBuffersFile                 |     |    |  
   |  141 | 0xFFFFFFFF819AD0F6 | nt!NtFlushInstructionCache               |     |    |  
   |  142 | 0xFFFFFFFF819781EB | nt!NtFlushKey                     |     |    |  
   |  143 | 0xFFFFFFFF818B11C1 | nt!NtFlushProcessWriteBuffers             |     |    |  
   |  144 | 0xFFFFFFFF819C175B | nt!NtFlushVirtualMemory                |     |    |  
   |  145 | 0xFFFFFFFF81A82D64 | nt!NtFlushWriteBuffer                 |     | Yes  |  

We can see a module (Gjglly.sys), and nt!NtFlushWriteBuffer is hooked. Let's not jump to conclusions just yet as this could be a completely legitimate hook.

First of all, what's Gjglly.sys? This is a driver in relation to AntiDebugLIB.

An advanced software encryption tool.

AntiDebugLIB is a useful tool that was designed in order to assist software developers protect their applications against advanced reverse engineering and software cracking. It offers a powerful advanced license control which allow developers to distribute trial versions of their applications securely.

This is a legitimate driver, but we can also at the same time not jump to conclusions on it being safe as said above (hint: it's not - I will discuss later).

2nd Method - Script

The second method we will be using in this postmortem debugging example is an older script developed by Lionel d'Hauenens of Laboskopia. This script will only work with the x86 WinDbg client. Regardless, as we know, it's always best to debug a crash dump in the client based off of the system's architecture. Given the system that generated this crash dump was 32-bit, we'll be debugging it with the x86 WinDbg client.

This script is incredibly helpful in not only checking the SSDT for hooking, but also detecting what is known as a Shadow SSDT hook. You can find a great article on Shadow SSDT hooking from my very good friend Harry - Shadow SSDT Hooking with Windbg.

Nevertheless, if you don't understand French instructions, to install the script, simply drag and drop the script folder into your x86 Debuggers folder.

Once it's installed, type the following command into WinDbg with your postmortem rootkit crash dump:

 kd> $$><script\@@init_cmd.wdbg  

So long as you installed the script properly, you should see the following:

 SysecLabs Windbg Script : Ok :)  
 ('al' for display all commands)  

Once you see that which has verified the script is loaded properly, type the following command:

 !!display_system_call  

Here's a few excerpts:

 007D  0003  OK  nt!NtDeleteObjectAuditAlarm (81a46500)  
 007E  0002 HOOK-> *** ERROR: Module load completed but symbols could not be loaded for Gjglly.sys  
 007F  000A  OK  nt!NtDeviceIoControlFile (81a34f80)  
 0080  0001  OK  nt!NtDisplayString (81949b44)  
 0081  0007  OK  nt!NtDuplicateObject (81a2117f)  
 0082  0006  OK  nt!NtDuplicateToken (81a18134)  
 0083  0002  OK  nt!NtEnumerateBootEntries (81ab14e8)  
 0084  0002  OK  nt!NtEnumerateDriverEntries (81ab278a)  
 0085  0006 HOOK-> Gjglly+0x1ffa (91f04ffa)  
 0086  0003  OK  nt!NtEnumerateSystemEnvironmentValuesEx (81ab10b7)  
 0087  0005  OK  nt!NtEnumerateTransactionObject (81a9f073)  
 0088  0006 HOOK-> Gjglly+0x21b6 (91f051b6)  

 00BC  0003  OK  nt!NtOpenJobObject (81a912df)  
 00BD  0003 HOOK-> Gjglly+0x1f4c (91f04f4c)  
 00BE  0004  OK  nt!NtOpenKeyTransacted (819727e1)  

 0143  0001  OK  nt!NtSetUuidSeed (8195977b)  
 0144  0006 HOOK-> Gjglly+0x2372 (91f05372)  
 0145  0005  OK  nt!NtSetVolumeInformationFile (81a6c5de)  

Here's Gjglly.sys again, and we can see it's hooking. At this point, we would be suspicious enough to run an ARK tool such as GMER. Of course, if we ran GMER at this point, it would show that the system is in fact infected with runtime2, and Gjglly.sys is our rootkit driver. Normally, by itself and by default, the runtime2 driver is runtime2.sys, not Gjglly.sys. In this specific scenario, niemiro used a different loader to inject the driver than the original, and renamed runtime2.sys to Gjglly.sys (a legitimate module name). Although GMER if run would still say the system is infected with a rootkit, and it would label Gjglly.sys as the rootkit driver, it's done not to trick various ARK tools, but the user.

If you've ever been infected with runtime2, you know it additionally drops startdrv.exe in the Temp directory of the current Windows install. startdrv.exe is the part that infuriates those who become infected with runtime2, because it's the part that immediately makes you suspicious of an infection as it's a trojan dropper. This is the specific part of the rootkit that will cause slowness, garbage popups, etc.

In this specific case, niemiro intentionally corrupted startdrv.exe which stopped it from executing entirely. You may be saying to yourself, what's the point of a rootkit's protection driver with no execution of payload, etc, by its dropped trojan? Well, there really is no point (unless your goal was to use something else that's malicious and not as obvious as a trojan dropper)! This was just an example to show how you can better hide a rootkit (protection driver, really) if you wanted to. Although ARK tools would still pick it up, it's not anywhere near as obvious to the user.

Thanks for reading, and I will get around to doing a live debugging of runtime2 as soon as I can.

--------------------

References

Hunting rootkits with Windbg.
Some notes about System Service Dispatch Table hook.