Showing posts with label bug. Show all posts
Showing posts with label bug. Show all posts

Wednesday, May 20, 2015

FwpsStreamInjectAsync0 bug/leak - Bitdefender (0x4A)

Today I'll be investigating an issue involving Bitdefender, which is turned out to be a Windows bug/issue more than Bitdefender, although there are developmental changes that could be made aside from a hotfix to stop this issue. Bitdefender's 0x4A bug check issue has been prevalent for quite awhile now, but there's little to no documentation on solving it or what's causing it, just a few things to try like updating Bitdefender, uninstalling it, etc
 
First off, taking a look at a non-verifier enabled kernel dump, here's our bug check as discussed:

 10: kd> .bugcheck  
 Bugcheck code 0000004A  
 Arguments 00000000`77a1dc2a 00000000`00000002 00000000`00000000 fffff880`13695b60  

0x4A bug check, essentially implying that the thread which was previously involved in a system call  attempted to return to user mode at an IRQL higher than PASSIVE_LEVEL (zero [0] on x86 and x64).

 10: kd> !irql  
 Debugger saved IRQL for processor 0xa -- 2 (DISPATCH_LEVEL)  

In this case, at the time of the crash, the IRQL was DISPATCH_LEVEL (Two [2] on x86 and x64).

 X64_RAISED_IRQL_FAULT_vsserv.exe_nt!KiSystemServiceExit+245  

The process involved in the IRQL raise was vsserv.exe, Bitdefender's main active protection process.

Let's also go further and dump the address of the system function involved:

 10: kd> !address 0000000077a1dc2a  
 Usage:         VAD  
 Base Address:      00000000`779d0000  
 End Address:      00000000`77b79000  
 Region Size:      00000000`001a9000  
 VA Type:        UserRange  
 VAD Address:      0xfffffa8020d1f830  
 Commit Charge:     0xd  
 Protection:       0x7 [ReadWriteCopyExecute]  
 Memory Usage:      Section [\Windows\System32\ntdll.dll]  
 No Change:       no  
 More info:       !vad 0x779d0000  

We can see its VA type is UserRange, its protection is 0x7 which implies it's R/W/X (or E).

If we run !vad on the VAD Address field, we can see frequent mention of Bitdefender:

 10: kd> !vad 0xfffffa8020d1f830  
 ...  
 fffffa8018c95a70 ( 2)  7fee7980 7fee79c8     6 Mapped Exe EXECUTE_WRITECOPY \Program Files\Bitdefender\Bitdefender 2015\asengines_00015_008\mimepack.dll  
 fffffa802278cb70 ( 3)  7fee79d0 7fee7aa2    69 Mapped Exe EXECUTE_WRITECOPY \Program Files\Bitdefender\Bitdefender 2015\asengines_00015_008\asregex.dll  
 fffffa8018c944e0 ( 0)  7fee7ab0 7fee7bb9     9 Mapped Exe EXECUTE_WRITECOPY \Program Files\Bitdefender\Bitdefender 2015\asengines_00015_008\asmcocr.dll  
 fffffa804d4baa50 ( 2)  7fee7bc0 7fee7dea    259 Mapped Exe EXECUTE_WRITECOPY \Program Files\Bitdefender\Bitdefender 2015\asengines_00015_008\asunicode.dll  
 fffffa8022f25450 ( 3)  7fee7df0 7fee810b    89 Mapped Exe EXECUTE_WRITECOPY \Program Files\Bitdefender\Bitdefender 2015\asengines_00015_008\asemlthin.mdl  
 fffffa8050690e60 ( 1)  7fee8110 7fee8333    86 Mapped Exe EXECUTE_WRITECOPY \Program Files\Bitdefender\Bitdefender 2015\asengines_00015_008\asemlrtr.mdl  
 fffffa8018c0b1e0 ( 2)  7fee8340 7fee8442    81 Mapped Exe EXECUTE_WRITECOPY \Program Files\Bitdefender\Bitdefender 2015\asengines_00015_008\ascore.dll  
 fffffa804f4f7970 ( 3)  7fee8550 7fee8622    69 Mapped Exe EXECUTE_WRITECOPY \Program Files\Bitdefender\Bitdefender 2015\otengines_00350_006\asregex.dll  
 fffffa804e14ff80 (-1)  7fee8630 7fee885a    259 Mapped Exe EXECUTE_WRITECOPY \Program Files\Bitdefender\Bitdefender 2015\otengines_00350_006\asunicode.dll  
 fffffa804f509640 ( 2)  7fee8860 7fee89ef    82 Mapped Exe EXECUTE_WRITECOPY \Program Files\Bitdefender\Bitdefender 2015\otengines_00350_006\ashttprbl.mdl  
 fffffa804f4bfc80 ( 3)  7fee89f0 7fee8cc9    88 Mapped Exe EXECUTE_WRITECOPY \Program Files\Bitdefender\Bitdefender 2015\otengines_00350_006\ashttpph.mdl  
 fffffa80232c0460 ( 1)  7fee8cd0 7fee8dca    82 Mapped Exe EXECUTE_WRITECOPY \Program Files\Bitdefender\Bitdefender 2015\otengines_00350_006\ashttpdsp.mdl  
 fffffa804ce69120 ( 3)  7fee8dd0 7fee8edb    81 Mapped Exe EXECUTE_WRITECOPY \Program Files\Bitdefender\Bitdefender 2015\otengines_00350_006\ashttpbr.mdl  
 fffffa804ce52bb0 ( 2)  7fee8ee0 7fee8fe2    81 Mapped Exe EXECUTE_WRITECOPY \Program Files\Bitdefender\Bitdefender 2015\otengines_00350_006\otcore.dll  
 ...  

Let's use !address and -v together to get nice verbose PTE/PFN/VAD information:

 10: kd> !address -v -map 0x779d0000  
 PXE:  fffff6fb7dbed000 [contains 02e0000763d62867]  
     Page Frame Number: 763d62, at address: fffffa80162b8260  
     Page Location:   6 (ActiveAndValid)  
     PTE Frame:     0000000000763e3c  
     Attributes:     M:Modified,Cached  
     Usage:       PPEs Process fffffa8020d96b10 [vsserv.exe], Entries:5  
 PPE:  fffff6fb7da00008 [contains 18700007641ad867]  
     Page Frame Number: 7641ad, at address: fffffa80162c5070  
     Page Location:   6 (ActiveAndValid)  
     PTE Frame:     0000000000763d62  
     Attributes:     M:Modified,Cached  
     Usage:       PDEs Process fffffa8020d96b10 [vsserv.exe], Entries:31  
 PDE:  fffff6fb40001de0 [contains 0370000764ab6867]  
     Page Frame Number: 764ab6, at address: fffffa80162e0220  
     Page Location:   6 (ActiveAndValid)  
     PTE Frame:     00000000007641ad  
     Attributes:     M:Modified,Cached  
     Usage:       PTEs Process fffffa8020d96b10 [vsserv.exe], Entries:159  
 PTE:  fffff680003bce80 [contains 82a000079e2d4025]  
     Page Frame Number: 79e2d4, at address: fffffa8016da87c0  
     Page Location:   6 (ActiveAndValid)  
     PTE Frame:     000000000079ec0c  
     Attributes:     P:Prototype,Cached  
     Usage:       MappedFile CA:fffffa801f3a3010 [\Windows\System32\ntdll.dll]  
 Type:  Valid  
 Attrs: Private,NormalPage,NotDirty,NotDirty1,Accessed,User,NotWritable,NotWriteThrough  
 PFN:  79e2d4  

Overall, we can see vsserv.exe is listed as active and valid within the page regarding its location, as well as ntdll being involved with memory usage:

 10: kd> !vad 0x779d0000  
 VAD       level   start   end  commit  
 fffffa8020d1f830 (-1)    779d0   77b78    13 Mapped Exe EXECUTE_WRITECOPY \Windows\System32\ntdll.dll  

Throughout all of the 0x4A Bitdefender related crashes, the NT kernel was labeled as the fault:

 Probably caused by : ntkrnlmp.exe  

Given we're seeing ntdll, we can likely imagine the reason for the NT kernel being blamed as being the fault of the crash is because most of the API from ntdll is implemented in the NT kernel variants, with this being ntkrnlmp.exe because this system has a multi-processor without physical address extension configuration.

 10: kd> vertarget  
 Windows 7 Kernel Version 7601 (Service Pack 1) MP (12 procs) Free x64  
 Product: WinNt, suite: TerminalServer SingleUserTS  
 Built by: 7601.18839.amd64fre.win7sp1_gdr.150427-0707  
 Machine Name:  
 Kernel base = 0xfffff800`03064000 PsLoadedModuleList = 0xfffff800`032ab730  
 Debug session time: Fri May 15 09:00:27.644 2015 (UTC - 4:00)  
 System Uptime: 0 days 16:20:00.892  

So regarding processor #10, that's probably as far as we're going to go considering it's the bug check thread and there's no information really whatsoever.

 10: kd> k  
 Child-SP     RetAddr      Call Site  
 fffff880`13695928 fffff800`030d7e69 nt!KeBugCheckEx  
 fffff880`13695930 fffff800`030d7da0 nt!KiBugCheckDispatch+0x69  
 fffff880`13695a70 00000000`77a1dc2a nt!KiSystemServiceExit+0x245  
 00000000`2798f908 00000000`00000000 0x77a1dc2a  

All we can see if we're exiting user-mode code using the KiSystemServiceExit function, and we go off the rails right there - KiSystemServiceExit+0x245. This function is in charge of handling the various call-styles used to enter kernel-mode, and then returning to user-mode.

With that said, let's switch to the other processor within the system that was involved and see what's going on at the time of the crash. To find out the active processors on the specific system, we'll use !running:

 10: kd> !running  
 System Processors: (0000000000000fff)  
  Idle Processors: (00000000000003ff) (0000000000000000) (0000000000000000) (0000000000000000)  
     Prcbs       Current     (pri) Next      (pri) Idle  
  10  fffff880038c9180 fffffa8021c85b50 ( 9)            fffff880038d41c0 ................  
  11  fffff8800393b180 fffffa8021c91060 ( 9)            fffff880039461c0 ................  

We can see our processors are #10 and #11. We've explored #10, so let's check #11. The reason 0-9 aren't listed is because they're idle.

 11: kd> knL  
  # Child-SP     RetAddr      Call Site  
 00 fffff880`03969e20 fffff880`0584f75e e1c62x64+0x558e  
 01 fffff880`03969e50 fffff880`0584ff1f e1c62x64+0x2075e  
 02 fffff880`03969ec0 fffff880`0584fb43 e1c62x64+0x20f1f  
 03 fffff880`03969f70 fffff880`0584fa49 e1c62x64+0x20b43  
 04 fffff880`03969fa0 fffff800`0301f62f e1c62x64+0x20a49  
 05 fffff880`03969fe0 fffff880`01a0c600 hal!HalBuildScatterGatherList+0x203  
 06 fffff880`0396a050 fffff880`0584ffb2 ndis!NdisMAllocateNetBufferSGList+0x110  
 07 fffff880`0396a0f0 fffff880`05850649 e1c62x64+0x20fb2  
 08 fffff880`0396a150 fffff880`0585028e e1c62x64+0x21649  
 09 fffff880`0396a1b0 fffff880`01ac84f1 e1c62x64+0x2128e  
 0a fffff880`0396a1f0 fffff880`01a0c4d4 ndis!ndisMSendNBLToMiniport+0xb1  
 0b fffff880`0396a250 fffff880`05c6d6b8 ndis!NdisFSendNetBufferLists+0x64  
 0c fffff880`0396a290 fffff880`05c6d92c bdfndisf6+0x16b8  
 0d fffff880`0396a2f0 fffff880`05c6df4b bdfndisf6+0x192c  
 0e fffff880`0396a380 fffff880`01a0c4d4 bdfndisf6+0x1f4b  
 0f fffff880`0396a480 fffff880`00c16199 ndis!NdisFSendNetBufferLists+0x64  
 10 fffff880`0396a4c0 fffff880`01a0c419 pacer!PcFilterSendNetBufferLists+0x29  
 11 fffff880`0396a5c0 fffff880`01ac85d5 ndis!ndisSendNBLToFilter+0x69  
 12 fffff880`0396a620 fffff880`01c60eb6 ndis!NdisSendNetBufferLists+0x85  
 13 fffff880`0396a680 fffff880`01c67895 tcpip!IpNlpFastSendDatagram+0x496  
 14 fffff880`0396aa30 fffff880`01c68450 tcpip!TcpTcbSend+0x495  
 15 fffff880`0396acb0 fffff880`01c671a8 tcpip!TcpEnqueueTcbSendOlmNotifySendComplete+0xa0  
 16 fffff880`0396ace0 fffff880`01b30267 tcpip!TcpEnqueueTcbSend+0x258  
 17 fffff880`0396ad90 fffff880`01b35f5d NETIO!StreamInjectRequestsToStack+0x287  
 18 fffff880`0396ae60 fffff880`01b376b4 NETIO!StreamPermitDataHelper+0x5d  
 19 fffff880`0396ae90 fffff800`030e41dc NETIO!StreamPermitRemoveDataDpc+0x84  
 1a fffff880`0396af00 fffff800`030db335 nt!KiRetireDpcList+0x1bc  
 1b fffff880`0396afb0 fffff800`030db14c nt!KyRetireDpcList+0x5  
 1c fffff880`13abf190 fffff800`0312371c nt!KiDispatchInterruptContinue  
 1d fffff880`13abf1c0 fffff800`030c2aec nt!KiDpcInterrupt+0xcc  
 1e fffff880`13abf350 fffff880`01b383aa nt!KeInsertQueueDpc+0x1dc  
 1f fffff880`13abf3e0 fffff880`01b3b468 NETIO!StreamPermitData+0x13a  
 20 fffff880`13abf450 fffff880`01b3b99a NETIO!StreamInternalClassify+0x1a8  
 21 fffff880`13abf520 fffff880`01b3bd8e NETIO!StreamInject+0x1ca  
 22 fffff880`13abf5f0 fffff880`01b91df3 NETIO!FwppStreamInject+0x12e  
 23 fffff880`13abf680 fffff880`05c9aaf1 fwpkclnt!FwpsStreamInjectAsync0+0xcf  
 24 fffff880`13abf6e0 fffff880`05c9bce3 bdfwfpf+0x2af1  
 25 fffff880`13abf780 fffff880`05ca469c bdfwfpf+0x3ce3  
 26 fffff880`13abf7c0 fffff880`05ca4d0a bdfwfpf+0xc69c  
 27 fffff880`13abf840 fffff880`05c9ebb3 bdfwfpf+0xcd0a  
 28 fffff880`13abf8a0 fffff800`033f3e47 bdfwfpf+0x6bb3  
 29 fffff880`13abf8d0 fffff800`033f46a6 nt!IopXxxControlFile+0x607  
 2a fffff880`13abfa00 fffff800`030d7b53 nt!NtDeviceIoControlFile+0x56  
 2b fffff880`13abfa70 00000000`77a1dc2a nt!KiSystemServiceCopyEnd+0x13  
 2c 00000000`27a9f928 00000000`00000000 0x77a1dc2a  

I used knL as opposed to the other stack dump commands as I wanted to get the frame # feature for reference reasons.

Starting at frame # 2a, we can see the NtDeviceIoControlFile function calls IopXxxControlFile. The latter function appears to be undocumented, so I'm unsure as to what it does. What I do know is, the NtDeviceIoControlFile function is ultimately used to build descriptors for a driver. I imagine it's using the IopXxxControlFile function to aid in passing such to the driver.

Also, for what it's worth, although NtDeviceIoControlFile has since been superseded by DeviceIoControl, the former native function provides more information that may be beneficial to the caller (especially for debugging purposes). This is likely why Bitdefender chose to use the former function instead.

 11: kd> ln nt!IopXxxControlFile  
 (fffff800`033f3840)  nt!IopXxxControlFile   
 (fffff800`033f4650)  nt!NtDeviceIoControlFile  
 Exact matches:  
   nt!IopXxxControlFile (<no parameter info>)  

If we disassemble this function, we can wade through some of the stuff and find some of the interesting tidbits:

 11: kd> u fffff800`033f3840 fffff800`033f4650  
 fffff800`033f3956 e845c5ffff   call  nt!ProbeForWrite (fffff800`033efea0)  
 fffff800`033f39b7 e81498fdff   call  nt!ObReferenceObjectByHandleWithTag (fffff800`033cd1d0)  
 fffff800`033f3b05 e84688cfff   call  nt!IoGetRelatedDeviceObject (fffff800`030ec350)  
 fffff800`033f402a e8d130cdff   call  nt!IoGetAttachedDevice (fffff800`030c7100)  
 fffff800`033f3c01 e88a7bcfff   call  nt!IoAllocateIrp (fffff800`030eb790)  
 fffff800`033f40d1 e82af9cfff   call  nt!IoAllocateMdl (fffff800`030f3a00)  

So after neatly putting together this disassembly of sorts, we can see that this is indeed how the NtDeviceIoControlFile function is passing on the buffer and such to the driver.

The IoAllocateMdl function in this specific case is used to ultimately associate the MDL with an IRP, which is why we call into the IoAllocateIrp function, to of course assign the IRP. IoGetAttachedDevice is called likely to return a pointer to the devobj, with help from the IoGetRelatedDeviceObject function to probably obtain the devobj from the file system driver stack.

ObReferenceObjectByHandleWithTag is called to increment the reference count of the object, and to write a four-byte value known as a "tag" so it can support object reference tracing for debugging purposes. Finally, the ProbeForWrite function is called to ensure that a user-mode buffer meets the following:

  • Resides in the user-mode portion of the address space.
  • Is writeable.
  • Is correctly aligned.

As all appears to have went well, we can see the driver we were ultimately building and passing descriptors to/for was bdfwfpf.sys, which is Bitdefender's firewall filter driver. As it's a driver in charge of a firewall, it of course uses the WFP API (Windows Filtering Platform) to achieve its goals (not just filtering and monitoring).

We can confirm this easily by looking at the very first driver/function call after Bitdefender's firewall, which is fwpkclnt.sys. Specifically, Bitdefender's firewall driver called it to inject new/cloned data to the data stream. Directly afterwords we have calls from the Network I/O Subsystem to continue the injecting, which is because fwpkclnt.sys exports kernel-mode functions, as opposed to fwpuclnt.dll which exports and handles the user-mode side.

To handle and/or continue the injection into the data stream, it looks like DPC(s) are used to handle it by calling KeInsertQueueDpc to create a queued DPC for execution.

 11: kd> !dpcs  
 CPU Type   KDPC    Function  
 10: Normal : 0xfffffa806a7b7cb0 0xfffff88001b37630 NETIO!StreamPermitRemoveDataDpc  

-- After discussion with a fellow kernel-debugger friend of mine, we also thought that the IRQL was possibly DISPATCH_LEVEL due to the multiple injections, etc, therefore Windows deferred it to a DPC. Given this possibly being the case, when the DPC was to be worked on, the system service finished but the IRQL is still DISPATCH_LEVEL. Since that was the case, we get a bug check.

We continue through netio.sys' functions regarding the data stream injection, ultimately injecting the request to the stack and going through a few tcpip.sys functions.

To continue sending the data along, NDIS' NdisSendNetBufferLists function is called, and NDIS' filter driver (which I believe is pacer.sys), called NdisFSendNetBufferLists to send the list of network data buffers back to Bitdefender's firewall driver.

Bitdefender's firewall driver then calls into NDIS' network data buffer sending functions to send the list to the user's network miniport driver, e1c62x64.sys (Intel(R) 82579V Gigabit Network Connection). The network miniport driver then calls NDIS' NdisMAllocateNetBufferSGList function to obtain a scatter/gather list for the network data for the associated NET_BUFFER structure.

In order to do so, NDIS needs to call the HAL, which we can see through the function HalBuildScatterGatherList. What is supposed to happen next is, the HAL builds the scatter/gather list, and we go on through various registered miniport functions. However, this did not happen, and we go off the rails on frame #00 with a call to the miniport driver.

So, where's our problem? Frame #23:

 23 fffff880`13abf680 fffff880`05c9aaf1 fwpkclnt!FwpsStreamInjectAsync0+0xcf  

FwpsStreamInjectAsync0, the function in charge of injecting TCP data segments into a TCP data stream, is the issue. How so? Well, let's get dirty once again.

Using the NDIS debugging extension (!ndiskd), we can get a lot of information to help us here. On its lonesome, !ndiskd isn't too special. However, when we use !ndiskd.miniport, it gets fun.

 11: kd> !ndiskd.miniport  
   MiniDriver     Miniport      Name                   
   fffffa8020c71cd0  fffffa8018c281a0  RAS Async Adapter  
   fffffa801f844cd0  fffffa801f8771a0  SonicWALL NetExtender Adapter  
   fffffa801f862840  fffffa801f86b1a0  WAN Miniport (SSTP)  
   fffffa801f84bb70  fffffa801f8671a0  WAN Miniport (PPTP)  
   fffffa801f837c30  fffffa801f8631a0  WAN Miniport (PPPOE)  
   fffffa801f8409b0  fffffa801f85e1a0  WAN Miniport (IPv6)  
   fffffa801f8409b0  fffffa801f85a1a0  WAN Miniport (IP)  
   fffffa801f8409b0  fffffa801f8561a0  WAN Miniport (Network Monitor)  
   fffffa801f835cd0  fffffa801f8411a0  WAN Miniport (L2TP)  
   fffffa801f82f820  fffffa801f83d1a0  WAN Miniport (IKEv2)  
   fffffa801f664020  fffffa801f7b81a0  Intel(R) 82579V Gigabit Network Connection  
   fffffa801f5cb9e0  fffffa801f5e61a0  Teredo Tunneling Pseudo-Interface  
   fffffa801f5cb9e0  fffffa801f5e21a0  Microsoft ISATAP Adapter #2  
   fffffa801f5cb9e0  fffffa801f5de1a0  Microsoft ISATAP Adapter  
   fffffa801f5cb9e0  fffffa801f5d61a0  Microsoft 6to4 Adapter  

So we know that our miniport involved in all of this was the Intel Gigabit, so let's look at that one:

 11: kd> !ndiskd.minidriver fffffa801f664020  
 MINIPORT DRIVER  
   e1cexpress  
   Ndis handle    fffffa801f664020  
   Driver Context   NULL  
   DRIVER_OBJECT   fffffa801f7b6e70  
   Driver image    e1c62x64.sys  
   Registry path   \REGISTRY\MACHINE\SYSTEM\ControlSet001\services\e1cexpress  
   Reference Count  2  
   Flags       [No flags set]  
 MINIPORTS  
   Miniport                                    
   fffffa801f7b81a0 - Intel(R) 82579V Gigabit Network Connection  

If we take a look at the miniport address:

 11: kd> !ndiskd.miniport fffffa801f7b81a0  
 MINIPORT  
   Intel(R) 82579V Gigabit Network Connection  
   Ndis handle    fffffa801f7b81a0  
   Ndis API version  v6.20  
   Adapter context  fffffa801f990000  
   Miniport driver  fffffa801f664020 - e1cexpress v12.6  
   Network interface fffffa8019c8c870  
   Media type     802.3  
   Device instance  PCI\VEN_8086&DEV_1503&SUBSYS_849C1043&REV_06\3&11583659&0&C8  
   Device object   fffffa801f7b8050  More information  
   MAC address    e0-3f-49-78-a1-dd  

 STATE  
   Miniport      Running  
   Device PnP     Started  
   Datapath      Normal  
   Interface     Up  
   Media       Connected  
   Power       D0  
   References     0n10  
   Total resets    0  
   Pending OID    None  
   Flags       BUS_MASTER, 64BIT_DMA, SG_DMA, DEFAULT_PORT_ACTIVATED,  
             SUPPORTS_MEDIA_SENSE, DOES_NOT_DO_LOOPBACK,  
             MEDIA_CONNECTED  
   PnP flags     PM_SUPPORTED, DEVICE_POWER_ENABLED,  
             DEVICE_POWER_WAKE_ENABLE, RECEIVED_START,  
             HARDWARE_DEVICE  

 BINDINGS  
   Protocol list   Driver       Open        Context        
   RSPNDR       fffffa8021b39cf0  fffffa8021b608d0  fffffa8021b62010  
   LLTDIO       fffffa8021b1a8f0  fffffa8021b528d0  fffffa8021b361b0  
   TCPIP6       fffffa801d05c2c0  fffffa801fb13010  fffffa801fb0b010  
   TCPIP       fffffa8019c7b890  fffffa801fb08580  fffffa801fb03ba0  
   Filter list    Driver       Module       Context        
   WFP LightWeight Filter-0000  
             fffffa801f59f010  fffffa801faff660  fffffa801faff400  
   QoS Packet Scheduler-0000  
             fffffa801f5ab930  fffffa801fb00780  fffffa801f9d3010  
   BitDefender Firewall NDIS6 Filter Driver-0000  
             fffffa801f574d40  fffffa801fb04c80  fffffa801fb04850  

We get a lot of good information, and can see that Bitdefender's firewall filter driver is/was involved with this miniport. We know this, because we saw it all happening in the stack, but this just confirms it.

Anyway, what's next? Well, let's check for any pending NBLs (NET_BUFFER_LISTS):

 11: kd> !ndiskd.pendingnbls fffffa801f7b81a0  
 PHASE 1/3: Found 23 NBL pool(s).           
 PHASE 2/3: Found 512 freed NBL(s).                    
   Pending Nbl    Currently held by                      
   fffffa80593c82c0  fffffa801f7b81a0 - Intel(R) 82579V Gigabit Network Connection [Miniport]            
 PHASE 3/3: Found 1 pending NBL(s) of 789 total NBL(s).             
 Search complete.  

Ah ha, we have one held by the miniport driver that was involved in passing data to Bitdefender's firewall filter driver. Let's look at the pending NBL:

 11: kd> !ndiskd.nbl fffffa80593c82c0  
   NBL        fffffa80593c82c0  Next NBL      NULL  
   First NB      fffffa80593c83f0  Source       fffffa801fb08580 - TCPIP   

From here we can take a direct look at the NBL:

 11: kd> dt _NET_BUFFER_LIST fffffa80593c82c0  
 ndis!_NET_BUFFER_LIST  
  _NET_BUFFER_LIST  
   +0x000 Next       : (null)   
   +0x008 FirstNetBuffer  : 0xfffffa80`593c83f0 _NET_BUFFER  
   +0x000 Link       : _SLIST_HEADER  
   +0x010 Context     : 0xfffffa80`593c84a0 _NET_BUFFER_LIST_CONTEXT  
   +0x018 ParentNetBufferList : (null)   
   +0x020 NdisPoolHandle  : 0xfffffa80`1cfe6080 Void  
   +0x030 NdisReserved   : [2] (null)   
   +0x040 ProtocolReserved : [4] 0x746c6100`00000001 Void  
   +0x060 MiniportReserved : [2] 0xfffffa80`1f990000 Void  
   +0x070 Scratch     : (null)   
   +0x078 SourceHandle   : 0xfffffa80`1fb08580 Void  
   +0x080 NblFlags     : 0  
   +0x084 ChildRefCount  : 0n0  
   +0x088 Flags      : 0x100  
   +0x08c Status      : 0n0  
   +0x090 NetBufferListInfo : [19] 0x00000000`00220015 Void  

What appears to be happening here is multiple NBLs in a chain are being passed, the FwpsStreamInjectAsync0 function is called to pass Bitdefender's data, and then the chain is broken as the call goes on (see the NBL next member is zeroed out/null).

Possibly a fix (in Bitdefender's case) is to avoid multiple injections inside the stream callout routine, possibly taking NBLs in a chain and calling the FwpsStreamInjectAsync0 function just ONCE for each callout routine execution. Unsure, kernel development isn't my strong point : )

A fix for user's is to install this hotfix and hope it works, as it should. Overall, maybe Bitdefender instead of making any developmental changes could just raise awareness for this issue, like creating a well explained documentation page with a link to the hotfix. I think developmental changes would be a better workaround.

Friday, April 25, 2014

HP Envy 700-074 ACPI BSOD BIOS update released + story!

*CLICK HERE FOR BIOS UPDATE*

---------------------------

Way back in January I made this blog post (here), which also made me realize truly how fast times flies by, because it's already almost May!

In that post, I mentioned I wasn't sure whether or not I was the only person who had reported the issue in-depth, and that it was a model widespread bug, and not just a system-specific issue. Well, it appears that I was, because for a few weeks I was in contact with various 'higher-ups' within HP. Supposedly they really had no idea the bug even existed as I can only assume what was happening was owners of the 700-074 model were contacting HP's basic customer support, and ultimately getting either refunds or the same model replaced, which would eventually have the exact same problem. I imagine it didn't appear 'alarming' enough yet for HP's customer/replacement support to bring the attention to the higher-ups within HP, but I digress.

First off, I wasn't sure at all where to go to actually report this as a somewhat serious issue. I looked all over and all I could find regarding contact was basic technical and/or customer support. I myself did not own an HP device, so I couldn't contact either of those because unless you have a service tag, you are charged. I didn't want to risk being charged money I didn't have to possibly either be hung up on, or not get anywhere. So, with that said, I wandered over to the HP Support forums, created a thread, and hoped I would get something. I checked the thread a few hours later, and it received a response notifying me that there is no official HP presence on the forums, but the person was nice enough to go ahead and escalate my issue so that it could possibly get the attention I was hoping.

Sure enough, around 24 hours later I received an email from someone in HP's Executive Customer Relations. I was asked for any info about the problem I was trying to bring to attention, at which time I then explained thoroughly what was going on with the Envy 700-074. I made sure I stressed that I'm just a young guy that isn't doing this for any reason other than to help HP possibly push a solution to this problem faster, so their customers didn't have to constantly return the same model 5+ times until they said screw it and got a full refund. I then shortly after my reply received a reply back stating that I would be updated in at least 24/48 hours with additional information on their findings. I never ended up receiving a reply back. At this point, I wasn't really sure what to do. I didn't want to email again and be annoying, so I let that go.

About a week or so later, thanks to a good friend of mine (MS MVP - John Griffith/Jcgriff2), I got into contact with an HP Social Media Ambassador, who was very kind. We exchanged emails back-and-forth for a few weeks, but I eventually stopped receiving responses from them as well. At the last time we spoke, I was told that the information had successfully reached HP's Escalation Engineer team (very cool). I can only imagine someone in HP said 'stop contacting this person' or something related, because I'm just a regular 20 year old guy interested in computers trying to help fix something any way I can, and they're a major business. I suppose it could have been considered sensitive information once it reached a certain point. I do appreciate both people I was in contact with, though.

Fast forward to almost three months later (now), and I receive an email notification that my thread on the HP forums received a reply. I figured it was just another Envy 700-074 customer saying they are having this problem, etc, however, it was a person notifying me a BIOS update had been released very few days ago, and it addressed this problem. Sure enough, I checked HP's website for the Envy 700-074, and a BIOS update was posted on 4/22!

Rejoice, another bug solved that I had a little to do with! :o)

Friday, April 11, 2014

0x44 btfilter.sys Bug (?)

-- For any users reading this experiencing this problem, my current advice to you is to go ahead and ensure all of your USB/Bluetooth related drivers (especially Atheros) are up to date via your manufacturers website.

Worst case, rollback any recent Windows Updates.

--------------------------------

Hello everyone!

Over the past short few days, many 0x44 BSOD's have been popping up. I can only help but smile as it reminds me of this - LogMeIn Hamachi Windows 8 and 8.1 Bug. When you have such a frequency of crashes with consistency, it's generally a bug in a driver regarding a specific piece of software, update rolled out, etc.

Anyway, enough rambling, let's get on with the analysis!

--------------------------------

First off, the bug check seems to consistently be MULTIPLE_IRP_COMPLETE_REQUESTS (44).

-- A driver has called IoCompleteRequest to ask that an IRP be completed, but the packet has already been completed.
3: kd> k
Child-SP          RetAddr           Call Site
ffffd000`207bb8e8 fffff803`43c08b4b nt!KeBugCheckEx
ffffd000`207bb8f0 fffff800`02cd4389 nt! ?? ::FNODOBFM::`string'+0x2a9ab
ffffd000`207bba00 fffff800`02cd391d USBPORT!USBPORT_Core_iCompleteDoneTransfer+0x979
ffffd000`207bbba0 fffff800`02cd33b8 USBPORT!USBPORT_Core_iIrpCsqCompleteDoneTransfer+0x21d
ffffd000`207bbc00 fffff800`02cd312e USBPORT!USBPORT_Core_UsbIocDpc_Worker+0x238
ffffd000`207bbc70 fffff803`43adad10 USBPORT!USBPORT_Xdpc_Worker_IocDpc+0x1fe
ffffd000`207bbcf0 fffff803`43ada9f0 nt!KiExecuteAllDpcs+0x1b0
ffffd000`207bbe40 fffff803`43bd0dd5 nt!KiRetireDpcList+0xd0
ffffd000`207bbfb0 fffff803`43bd0bd9 nt!KxRetireDpcList+0x5
ffffd000`25051a40 fffff803`43bd2e45 nt!KiDispatchInterruptContinue
ffffd000`25051a70 fffff803`43bcead3 nt!KiDpcInterruptBypass+0x25
ffffd000`25051a80 00000018`c76f0399 nt!KiChainedDispatch+0x173
00000018`c5cccfc0 00000000`00000000 0x00000018`c76f0399
Pretty interesting call stack! As we move up, we can see that it calls to retire a DPC list, and then it makes a call to execute all DPCs. We then call into what appears to be the USBPORT.sys driver and have a few worker routines going on, which inevitably call into the bug check.

-- FAILURE_BUCKET_ID:  0x44_IMAGE_ACPI
We can see that by default, WinDbg notes that the cause of the crash was the Advanced Configuration and Power Interface (ACPI). Essentially, in its simplest terms, it brings power management under the control of the operating system, as opposed to the previous BIOS-central system which relied on platform-specific firmware to determine power management and configuration policy.

-- ADDITIONAL_DEBUG_TEXT:  USB\VID_0CF3&PID_3004
WinDbg provides us with a Vendor/Product ID, which we can sift through a database to find out which USB-based device this specifically is. 0CF3 regarding a Vendor ID falls under Atheros Communications, Inc. Specifically, regarding its Product ID, 3004 is not listed, but 3000, 30002, and 3005 are all related to Bluetooth (from Atheros).

Interestingly enough, in the crash dump itself, this is also mentioned:

-- OVERLAPPED_MODULE: Address regions for 'bthport' and 'bthport.sys' overlap.
bthport.sys is the Bluetooth Bus driver.

Right, so the summary so far is that we appear to possibly be having a Bluetooth related driver (possibly related to Atheros) causing issues. Let's look forward more to confirm or deny this.

--------------------------------

3: kd> dt nt!_IO_STATUS_BLOCK ffffe00001587ac0
   +0x000 Status           : 0n20223232
   +0x000 Pointer          : 0xffffe000`01349500 Void
   +0x008 Information      : 0
Now that we have this, let's check the STATUS field layout:


3: kd> ? 0n20223232
Evaluate expression: 20223232 = 00000000`01349500
3: kd> .formats 0n20223232
Evaluate expression:
  Hex:     00000000`01349500
  Decimal: 20223232
  Octal:   0000000000000115112400
  Binary:  00000000 00000000 00000000 00000000 00000001 00110100 10010101 00000000
  Chars:   .....4..
  Time:    Sat Aug 22 21:33:52 1970
  Float:   low 3.31677e-038 high 0
  Double:  9.9916e-317
Assuming I am correct, '00000000`01349500' falls under:

NT_SUCCESS(Status) Evaluates to TRUE if the return value specified by Status is a success type (0 − 0x3FFFFFFF) or an informational type (0x40000000 − 0x7FFFFFFF).
Moving forward, let's take a closer look at the IRP:

3: kd> !irp ffffe00001587ac0 1
Irp is active with 8 stacks 7 is current (= 0xffffe00001587d40)
 No Mdl: No System Buffer: Thread 00000000:  Irp stack trace. 
Flags = 00000000
ThreadListEntry.Flink = ffffe00001587ae0
ThreadListEntry.Blink = ffffe00001587ae0
IoStatus.Status = 00000000
IoStatus.Information = 00000000
RequestorMode = 00000000
Cancel = 00
CancelIrql = 0
ApcEnvironment = 00
UserIosb = 00000000
UserEvent = 00000000
Overlay.AsynchronousParameters.UserApcRoutine = 00000000
Overlay.AsynchronousParameters.UserApcContext = 00000000
Overlay.AllocationSize = 00000000 - 00000000
CancelRoutine = 00000000  
UserBuffer = 00000000
&Tail.Overlay.DeviceQueueEntry = ffffe00001587b38
Tail.Overlay.Thread = 00000000
Tail.Overlay.AuxiliaryBuffer = 00000000
Tail.Overlay.ListEntry.Flink = 00000000
Tail.Overlay.ListEntry.Blink = 00000000
Tail.Overlay.CurrentStackLocation = ffffe00001587d40
Tail.Overlay.OriginalFileObject = 00000000
Tail.Apc = 00000000
Tail.CompletionKey = 00000000
     cmd  flg cl Device   File     Completion-Context
 [  0, 0]   0  0 00000000 00000000 00000000-00000000   

            Args: 00000000 00000000 00000000 00000000
 [  0, 0]   0  0 00000000 00000000 00000000-00000000   

            Args: 00000000 00000000 00000000 00000000
 [  0, 0]   0  0 00000000 00000000 00000000-00000000   

            Args: 00000000 00000000 00000000 00000000
 [  0, 0]   0  0 00000000 00000000 00000000-00000000   

            Args: 00000000 00000000 00000000 00000000
 [  0, 0]   0  0 00000000 00000000 00000000-00000000   

            Args: 00000000 00000000 00000000 00000000
 [  0, 0]   0  0 00000000 00000000 00000000-00000000   

            Args: 00000000 00000000 00000000 00000000
*** ERROR: Module load completed but symbols could not be loaded for btfilter.sys
>[  f, 0]   0 e1 ffffe000033f1050 00000000 fffff8000415d460-ffffe00001660180 Success Error Cancel pending
           \Driver\usbohci    btfilter
            Args: ffffe00003831910 00000000 00220003 00000000
 [  f, 0]   0  0 00000000 00000000 00000000-00000000   

            Args: ffffe00003831910 00000000 00220003 00000000
This is a very interesting IRP output, as a lot of the values are zeroed out. Also, it's not noting that the IRP has completed, or that the PENDING has returned.

In any case, we can see that btfilter.sys is mentioned. btfilter.sys is the Atheros Bluetooth driver.


3: kd> !devstack ffffe000033f1050
  !DevObj   !DrvObj            !DevExt   ObjectName
  ffffe000031b4b80 *** ERROR: Module load completed but symbols could not be loaded for usbfilter.sys
 \Driver\usbfilter  ffffe000031b4cd0 
  ffffe000026ab050  \Driver\usbhub     ffffe000026ab1a0  0000003a
  ffffe000031b5bf0  \Driver\usbfilter  ffffe000031b5d40 
  ffffe000031cfe50  \Driver\ACPI       ffffe00000f94b90 
> ffffe000033f1050  \Driver\usbohci    ffffe000033f11a0  USBPDO-2
!DevNode ffffe0000336a130 :
  DeviceInst is "USB\ROOT_HUB\4&301d4ea9&0"
  ServiceName is "usbhub"
Here's where we can see mention of ACPI, and other USB related drivers (even the root hub of USB).

--------------------------------

Overall, with all of this said, I can only guess that with the zeroed out values + no completed notice, etc, that it's well beyond actual completion, PENDING return, etc, OR despite the bug check, the IRP is not completing (or in an IRP completion state), but rather remaining PENDING, etc. Also, it's entirely possible that usbfilter.sys and btfilter.sys are both battling for completion believing that they own the packet. In any case, I believe this has been brought upon by a recent update being bugged, but I will have to see as time goes on. I also need to look into a few more things regarding debugging this.

More info soon, hopefully.


Wednesday, January 29, 2014

LogMeIn Hamachi Windows 8 and 8.1 Bug

(Last updated 2/5/2014 - Update #7) - Updates are scattered throughout the blog post, so I'd recommend using CTRL+F "Update #numberhere" without the quotations of course to get to the update you'd like to read.

To those reading currently experiencing the 0x44 BSOD caused by LogMeIn Hamachi, the only current workaround until LogMeIn implements a fix and/or releases a stable version is to fully uninstall LogMeIn Hamachi.

As of 2/5/2014, a LogMeIn update has been released that addresses this bug. Be sure to go ahead and download and/or update to the latest version if yours doesn't automatically:


Thanks to timbadia @ Sysnative for the screenshot!

--------------------------------

At this point, Harry (x BlueRobot) and I have analyzed this to the point where we can do no more as we are only volunteer analysts with an interest in helping people. We've found consistency thanks to everyone who submitted kernel-dumps. If you would like to do your part in making sure this gets solved, please go ahead and submit a support ticket (HERE) and in the 'Please describe your issue or question' field, if you could include a link to this blog post so the LogMeIn team can see what the problem is, etc, so this can be solved as fast as possible for its users, that would be fantastic.

Keep reading below if you'd like to learn more about the specifics and internals of this crash! : )

--------------------------------

So, what's the problem?

Well, first off, the bug check seems to consistently be MULTIPLE_IRP_COMPLETE_REQUESTS (44).

-- A driver has called IoCompleteRequest to ask that an IRP be completed, but the packet has already been completed.
When dealing with the first thread, the first thing I did was straight away take a look at the call stack and saw this:




Very simple call stack, really.

1. We have Hamdrv.sys (LogMeIn Hamachi Virtual Miniport Driver).

2.  nt! ?? ::FNODOBFM::`string'+0x2a9ab - TO MY KNOWLEDGE, the debugger (WingDbg) is slightly confused about symbol names in NTDLL due to the binary being reorganized into function chunks. The functions are no longer contiguous in memory. Hot code paths are clustered together with hot code paths of other functions. “Cold” code paths are moved elsewhere. That way you save on paging I/O by maximizing the amount of relative data on each code page.

Essentially, to my understanding, when a sequence of code is compiled, it will occupy a single contiguous chunk of memory. With this said however, the optimizer can spread the executable code all over the place, replacing the inline code with a jump to some other memory location. As said above, this is done for two main reasons (maybe more, but I only understand two):

A. Code size - If there's a common code sequence, make a copy and have everything jump to that copy.

B. Locality of reference - If there's a code path that is taken quite often, putting that code together into a single location can possibly improve performance. As far as how this improves performance, to my knowledge it's mostly cache purposed. For example, a TLB (Translation lookaside buffer).

With this said, if the optimizer moves the code to a different (random) location, what happens to the information in the PDB (Program Database)?

When this happens, to my knowledge, FunctionName+Offset no longer equals FunctionAddress+Offset, therefore the output of information in the debugger isn't correct. In these specific cases, the code is moved to a location (which is random, to my knowledge) and the closest symbolic name is a string in the image. When this happens, the debugger (WinDbg) uses the string as a best guess for the return address on the stack.

To further debug this, I believe I would need to set breakpoints and then disassemble a specific routine and take a look at the control flaws (which since this is not happening to me, I cannot do). Also, as much as I am interested in doing something like this, requesting the average computer user on a forum such as MS Answers to set a breakpoint(s) for knowledge related purposes is going a little far. It's just another reason for me to set up a guinea pig system.

3. Bug check itself.

-------------

At first I thought it may have been the same user just making a new thread under a separate account possibly because they didn't want to follow my advice, but:



 












^^ Different models entirely, however the OS environment remained consistent (Windows 8.1). With this said, possibly a bug exclusive to 8.1.

-------------

At first glance of the original dump I saw, I figured there may have been security software installed on the system in question that was causing conflicts with remote access, etc. I took a look at the loaded modules list and noted McAfee, nothing more. I informed the user to remove McAfee as it may be causing conflicts, but the same exact issue and bug check persisted after McAfee's removal. With this said, it does not appear to be caused by anything network and/or security related, although I could be wrong.

As per usual, there doesn't appear to be a simple way to contact LogMeIn if you don't have an account, and their customer support seems to have vanished entirely since Jan 21st (their move away from being FREE). With this said, not sure how to bring this up. I'll keep my eyes peeled for any other of the same bug check.

-------------

Update #1 - A few thread examples:

1. Blue Screen Error - Bugcheck - Microsoft Community

2. Windows 8.1 Error : MULTIPLE IRP COMPLETE REQUESTS. - Microsoft Community

3. BSOD Win 8.1 Multiple Complete IRP Request

4.
BLUE SCREEN - MULTIPLE IRP COMPLETE REQUESTS

-- All are the same OS environment (Windows 8.1). I was thinking of providing a link for each 0x44 BSOD thread I come across, but that would be quite a bit. I may do it eventually, just not now.

I've also received notice from my good friend and Microsoft MVP zigzag3143 that he has seen quite a number of these popping up very recently. At this point, it's very evident there is a bug with the software itself and Windows 8.1 as an OS, and that this is very likely NOT a user specific or system specific issue.

Update #2 - I discovered that LogMeIn has a ticket system (here), so I've gone ahead and reported this issue. I hope to hear from them soon!

Update #3 - I have not received a response from LogMeIn since submitting a ticket, and I probably won't. In other news, I received a kernel-dump from a very generous user after I requested if it's possible they allow their system to crash with the system set to generate kernel dumps as opposed to minidumps. This way, I would be able to take a closer look and get more information. Here is what I was able to see:

First off, the call stack is interesting:

7: kd> kv
Child-SP          RetAddr           : Args to Child                                                           : Call Site
fffff880`031a9d48 fffff803`d802e6f2 : 00000000`00000044 fffffa80`0ea3f910 00000000`00000f7a 00000000`00000000 : nt!KeBugCheckEx
fffff880`031a9d50 fffff880`1a801f0a : 00000000`00000000 fffff880`1ca7ac00 fffffa80`0c7852c0 fffffa80`0c7852c0 : nt! ?? ::FNODOBFM::`string'+0xb318
fffff880`031a9e30 fffff880`1a803059 : fffffa80`0c7852c0 fffff880`031a9f00 fffff880`031a9f80 fffffa80`0000004a : Hamdrv+0x1f0a
fffff880`031a9e80 fffff880`1a802a56 : fffff880`031aa680 fffffa80`00000001 fffffa80`09686b90 fffffa80`09686cf0 : Hamdrv+0x3059
fffff880`031aa540 fffff880`01e7dcc9 : fffffa80`0b3f51a0 fffffa80`09686b90 00000000`00000000 00000000`00000000 : Hamdrv+0x2a56
fffff880`031aa570 fffff880`01e7c166 : fffffa80`09686b90 00000000`00000000 fffffa80`0c4dd7f0 fffffa80`0c4dd7f0 : ndis!ndisMSendNBLToMiniport+0xc9
fffff880`031aa5f0 fffff880`02305230 : fffffa80`09686b90 00000000`00000001 fffffa80`09686b90 fffffa80`0c4dd7f0 : ndis!ndisFilterSendNetBufferLists+0xd6
fffff880`031aa630 fffff880`0230540f : 00000000`00000000 fffffa80`09686b90 00000000`00000000 fffffa80`0c4dd7f0 : wfplwfs!L2NdisFSendNetBufferLists+0x80
fffff880`031aa670 fffff880`01e7dd8d : 00000000`00000000 fffffa80`09686b90 00000000`00000000 00000000`00000000 : wfplwfs!LwfLowerSendNetBufferLists+0x9f
fffff880`031aa6e0 fffff803`d80cb9a6 : 00000000`00000202 fffff803`d877abb9 fffff880`04714970 00000000`000007ff : ndis!ndisDataPathExpandStackCallback+0x31
fffff880`031aa720 fffff803`d80ce405 : fffff880`01e7dd5c fffff880`031aa910 00000000`00000000 fffff803`d82a8a08 : nt!KeExpandKernelStackAndCalloutInternal+0xe6
fffff880`031aa820 fffff880`01e7c05f : fffff8a0`046587f0 fffff803`d828d409 00000000`00000001 fffff803`d828c200 : nt!KeExpandKernelStackAndCalloutEx+0x25
fffff880`031aa860 fffff880`01e7c166 : 00000000`00000000 fffffa80`094f5bf0 fffffa80`08ff9970 00000000`00000000 : ndis!ndisInvokeNextSendHandler+0x2de
fffff880`031aa970 fffff880`04796997 : fffffa80`09686b90 00000000`00000022 fffffa80`0c5aebf0 fffff880`031aaa39 : ndis!ndisFilterSendNetBufferLists+0xd6
fffff880`031aa9b0 fffff880`0479bbc5 : fffffa80`0e44950e fffffa80`0e449500 fffffa80`0000004a fffffa80`00000014 : bdfndisf6+0x3997
fffff880`031aaaa0 fffff803`d80c4b77 : fffffa80`095061a0 00000000`0020b553 00000000`00000000 fffff880`0479b934 : bdfndisf6+0x8bc5
fffff880`031aab10 fffff803`d80b42a1 : fffff803`d82a2110 fffffa80`0850a040 fffff803`d80c4b18 fffff803`d807fc00 : nt!IopProcessWorkItem+0x5f
fffff880`031aab80 fffff803`d8048fd9 : 9212ecd9`41a9baff 00000000`00000080 fffff803`d80b4160 fffffa80`0850a040 : nt!ExpWorkerThread+0x142
fffff880`031aac10 fffff803`d80fd7e6 : fffff880`009cb180 fffffa80`0850a040 fffff880`009d6f40 fffffa80`0849f740 : nt!PspSystemThreadStartup+0x59
fffff880`031aac60 00000000`00000000 : fffff880`031ab000 fffff880`031a5000 00000000`00000000 00000000`00000000 : nt!KiStartSystemThread+0x16
Aside from the various Network Driver Interface Specification and Windows Filtering Platform routines, we can see two BitDefender Firewall calls. As we ascend up the stack, we of course eventually see our three LogMeIn Hamachi Virtual Miniport Driver calls, and then the bug check.

What's really interesting to me is that BitDefender's firewall turned up in the stack. Normally, if I saw this first as opposed to a minidump, I would assume that BitDefender's firewall was causing conflicts, but it has shown in various scenarios that the user would remove their security software and or firewall and the issue continued, or they didn't have any installed whatsoever to cause conflict in the first place.

So, was it called maybe for reasons other than potential conflict? Possibly just noting that it had to deal with BitDefender before beginning the Network Driver Interface Specification routines?

Also, there's definitely a trend in the packet being marked Pending as opposed to Completed:

7: kd> dt nt!_IRP fffffa800ea3f910
   +0x000 Type             : 0n-12272
   +0x002 Size             : 0xc5a
   +0x004 AllocationProcessorNumber : 0xfa80
   +0x006 Reserved         : 0xffff
   +0x008 MdlAddress       : (null)
   +0x010 Flags            : 0x60900
   +0x018 AssociatedIrp    : <unnamed-tag>
   +0x020 ThreadListEntry  : _LIST_ENTRY [ 0xfffffa80`0ea3f930 - 0xfffffa80`0ea3f930 ]
   +0x030 IoStatus         : _IO_STATUS_BLOCK
   +0x040 RequestorMode    : 1 ''
   +0x041 PendingReturned  : 0x1 ''
   +0x042 StackCount       : 1 ''
   +0x043 CurrentLocation  : 3 ''
   +0x044 Cancel           : 0 ''
   +0x045 CancelIrql       : 0 ''
   +0x046 ApcEnvironment   : 0 ''
   +0x047 AllocationFlags  : 0x4 ''
   +0x048 UserIosb         : 0x00000000`01b12c30 _IO_STATUS_BLOCK
   +0x050 UserEvent        : 0xfffffa80`09017860 _KEVENT
   +0x058 Overlay          : <unnamed-tag>
   +0x068 CancelRoutine    : 0xfffff880`1a80188c     void  +0
   +0x070 UserBuffer       : (null)
   +0x078 Tail             : <unnamed-tag>

The IoStatus also consistently appears to be IO_STATUS_BLOCK, which the driver sets the IRP's I/O status block to indicate the final status of an I/O request before calling IoCompleteRequest for the IRP. An I/O status block serves two purposes:


  • It provides a higher-level driver's IoCompletion routine a way of determining whether the service worked when the IRP is completed.
  • It provides more information about why the service either worked or did not work.

As far as I know, this is the completion status, either STATUS_SUCCESS if the requested operation was completed successfully as you mentioned earlier, or an informational, warning, or error STATUS_XXX value. This is where I am stuck, I cannot find where the error, warning, etc, is?

I'm having a fellow BSOD and Windows Internals Expert have a look at the kernel-dump I was provided. I will add more updates regarding it when I get them.


 -------------

Update #4 - Fellow BSOD, Windows Internals Expert, and good friend Harry has taken a look at the kernel-dump I provided him and some more interesting information was discovered. What's below likely won't make sense to the average user, but that's okay because it's more-so for knowledge related purposes and just some extra poking around. The initial temporary workaround still stands, which is to simply uninstall LogMeIn Hamachi until the LogMeIn developers issue a fix or patch.

7: kd> dt nt!_IO_STATUS_BLOCK fffffa800ea3f910
   +0x000 Status           : 0n207278096
   +0x000 Pointer          : 0xfffffa80`0c5ad010 Void
   +0x008 Information      : 0

Status field layout can be found below.

7: kd> ? 0n207278096
Evaluate expression: 207278096 = 00000000`0c5ad010

I've evaluated the value to hexiadecimal value, to try and decipher what it could mean. It seems that it may have returned NT_ERROR. I managed to find this within the Ntdef.h Header file, I apologise if anyone has difficulty reading my dark theme for Visual Studio.

7: kd> .formats 0n207278096
Evaluate expression:
  Hex:     00000000`0c5ad010
  Decimal: 207278096
  Octal:   0000000000001426550020
  Binary:  00000000 00000000 00000000 00000000 00001100 01011010 11010000 00010000 <-- 0 starts here
  Chars:   .....Z..
  Time:    Tue Jul 27 02:14:56 1976
  Float:   low 1.68567e-031 high 0
  Double:  1.02409e-315

Additional Information from the WDK -

When testing a return value, you should use one of the following system-supplied macros (defined in Ntdef.h):

NT_SUCCESS(Status) Evaluates to TRUE if the return value specified by Status is a success type (0 − 0x3FFFFFFF) or an informational type (0x40000000 − 0x7FFFFFFF).

NT_INFORMATION(Status) Evaluates to TRUE if the return value specified by Status is an informational type (0x40000000 − 0x7FFFFFFF).

NT_WARNING(Status) Evaluates to TRUE if the return value specified by Status is a warning type (0x80000000 − 0xBFFFFFFF).

NT_ERROR(Status) Evaluates to TRUE if the return value specified by Status is an error type (0xC0000000 - 0xFFFFFFFF).

7: kd> !irp fffffa800ea3f910 1
Irp is active with 1 stacks 3 is current (= 0xfffffa800c440c10)
 No Mdl: No System Buffer: Thread fffffa800c716900:  Irp is completed.  Pending has been returned
Flags = 00060900
ThreadListEntry.Flink = fffffa800ea3f930
ThreadListEntry.Blink = fffffa800ea3f930
IoStatus.Status = 00000000
IoStatus.Information = 00000118
RequestorMode = 00000001
Cancel = 00
CancelIrql = 0
ApcEnvironment = 00
UserIosb = 01b12c30
UserEvent = fffffa8009017860
Overlay.AsynchronousParameters.UserApcRoutine = 00000000
Overlay.AsynchronousParameters.UserApcContext = 01b12c30
Overlay.AllocationSize = 00000000 - 00000000
CancelRoutine = fffff8801a80188c   Hamdrv
UserBuffer = 00000000
&Tail.Overlay.DeviceQueueEntry = fffffa800ea3f988
Tail.Overlay.Thread = fffffa800c716900
Tail.Overlay.AuxiliaryBuffer = fffff803d8567998
Tail.Overlay.ListEntry.Flink = 00000000
Tail.Overlay.ListEntry.Blink = 00000000
Tail.Overlay.CurrentStackLocation = fffffa800c440c10
Tail.Overlay.OriginalFileObject = 00000000
Tail.Apc = 00580012
Tail.CompletionKey = 00580012
     cmd  flg cl Device   File     Completion-Context
 [  3, 0]   0  0 fffffa8009ff6060 00000000 00000000-00000000  
           \Driver\hamachi
            Args: 00000000 00000000 00000000 00000000

The 3 is the Major Function Code for IRP_MJ_READ and the Minor Function Code is IRP_MN_NORMAL from my understanding.

7: kd> !devstack fffffa8009ff6060
  !DevObj   !DrvObj            !DevExt   ObjectName
> fffffa8009ff6060  \Driver\hamachi    fffffa8009ff61b0  HamachiTap.C6AE2999-2529-481A-A4BA-C3ED805229CC

7: kd> dt nt!_IO_STACK_LOCATION 0xfffffa800c440c10
   +0x000 MajorFunction    : 0x5 ''
   +0x001 MinorFunction    : 0 ''
   +0x002 Flags            : 0xd8 ''
   +0x003 Control          : 0 ''
   +0x008 Parameters       : <unnamed-tag>
   +0x028 DeviceObject     : (null)
   +0x030 FileObject       : (null)
   +0x038 CompletionRoutine : (null)
   +0x040 Context          : (null)

The data structure shows the current stack location of the IRP. Since IRPs can be handled by different Driver Objects through Driver Stacks, and therefore may be a indication of the different driver handling a different IRP.

7: kd> !drvobj fffffa800968ee60
Driver object (fffffa800968ee60) is for:
 \Driver\hamachi
Driver Extension List: (id , addr)
(4e4d4944 fffffa800ef88bb0)
Device Object list:
fffffa8009ff6060  fffffa800b3f5050 

7: kd> !devobj fffffa800b3f5050
Device object (fffffa800b3f5050) is for:
 NDMP24 \Driver\hamachi DriverObject fffffa800968ee60
Current Irp 00000000 RefCount 0 Type 00000017 Flags 00002050
Dacl fffff9a100265f51 DevExt fffffa800b3f51a0 DevObjExt fffffa800b3f6688
ExtensionFlags (0x00000800)  DOE_DEFAULT_SD_PRESENT
Characteristics (0x00000100)  FILE_DEVICE_SECURE_OPEN
AttachedTo (Lower) fffffa800e6bc7f0 \Driver\PnpManager
Device queue is not busy.

7: kd> !devstack fffffa800b3f5050
  !DevObj   !DrvObj            !DevExt   ObjectName
> fffffa800b3f5050  \Driver\hamachi    fffffa800b3f51a0  NDMP24 <-- Network Data Management Protocol?
  fffffa800e6bc7f0  \Driver\PnpManager 00000000  0000006a
!DevNode fffffa8008e37010 :
  DeviceInst is "ROOT\NET\0000"
  ServiceName is "Hamachi"

The NDMP24 may explain the network related calls in the Call Stack for the thread which was responsible for the IRP.

7: kd> !thread fffffa800c716900
THREAD fffffa800c716900  Cid 1c40.1da0  Teb: 000007f7fc3b4000 Win32Thread: fffff901007ffb90 RUNNING on processor 1
IRP List:
    fffffa800f678980: (0006,0118) Flags: 00060000  Mdl: 00000000 <-- Npfs
    fffffa800ea04ee0: (0006,0118) Flags: 00060000  Mdl: 00000000 <-- Npfs
    fffffa800f5f7520: (0006,01f0) Flags: 00060900  Mdl: 00000000 <-- nsiproxy
    fffffa800c52a520: (0006,01f0) Flags: 00060800  Mdl: 00000000 <-- nsiproxy
Not impersonating
DeviceMap                 fffff8a00000c280
Owning Process            fffffa800c4c8940       Image:         hamachi-2.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      864481         Ticks: 0
Context Switch Count      43034          IdealProcessor: 0            
UserTime                  00:00:00.125
KernelTime                00:00:00.140
Win32 Start Address 0x000007fd00a44aa0
Stack Init fffff8801ce29c90 Current fffff8801ce29050
Base fffff8801ce2a000 Limit fffff8801ce24000 Call 0
Priority 8 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           : Args to Child                                                           : Call Site
fffff880`1ce29438 fffff880`00b77115 : fffffa80`0cc3b840 fffffa80`093c9270 00000000`00000000 fffffa80`08531e00 : nt!ObReferenceObjectByHandle
fffff880`1ce29440 fffff880`00b590a7 : fffff880`1ce29918 00000000`00000320 00000000`00000000 fffffa80`0cb9e620 : afd!AfdEnumNetworkEvents+0x75
fffff880`1ce294e0 fffff803`d844b47a : fffffa80`0c549610 00000000`00000000 00000000`000003a8 00000000`00000114 : afd!AfdFastIoDeviceControl+0x1097
fffff880`1ce29890 fffff803`d8460a76 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!IopXxxControlFile+0x3d9
fffff880`1ce29a20 fffff803`d8075453 : fffffa80`0c716900 00000000`01f9fb18 fffff880`1ce29aa8 00000000`01f9fb40 : nt!NtDeviceIoControlFile+0x56
fffff880`1ce29a90 000007fd`03242c5a : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`1ce29b00)
00000000`01f9fcb8 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x000007fd`03242c5a

7: kd> !irp fffffa800f5f7520
Irp is active with 2 stacks 2 is current (= 0xfffffa800f5f7638)
 No Mdl: No System Buffer: Thread fffffa800c716900:  Irp stack trace.
     cmd  flg cl Device   File     Completion-Context
 [  0, 0]   0  0 00000000 00000000 00000000-00000000  

            Args: 00000000 00000000 00000000 00000000
>[  3, 0]   0  1 fffffa800caf9260 fffffa8008c53d10 00000000-00000000    pending
           \FileSystem\Npfs
            Args: 00002000 00000000 00000000 00000000

Npfs is a Windows driver, but I'm correct, isn't the driver attempting to complete the IRP packet again? I don't think I may be correct though, if the IRP is targeted at a different Device Object.

-------------

So, that was Harry's analysis of the 0x44 kernel-dump. As discovered above, Npfs.sys appears to be sitting at status PENDING (not complete). Npfs.sys is the Named Pipe File System service. It enables the exchange of data among multiple paths, whether the exchange is for the same application or for two or more different applications at a time.

Is this the reason the crash is occurring? Maybe, we cannot say because one kernel-dump is not enough as there needs to be a check for consistency. If we receive more kernel-dumps, we can check and see whether or not Npfs.sys is consistent in its status PENDING. If it is, it appears the developers will need to iron that out and everything will be solid.

-------------

Update #5 - Wow, today I woke up to multiple emails from various people either providing their kernel-dumps, or asking if I'd like theirs. This is absolutely incredible, I am ecstatic. I'd like to extend a HUGE thank you to every single one of you. At the end of it all, I will be absolutely sure to give credit to every single one of you by your name, etc. Let's get started!

David Laverdiere II's kernel-dump:

This is the first kernel-dump I received, so it's only fair I use it in the first major analysis to check for consistency : )

First off, let's take a look at the call stack just to see what's going on:

1: kd> kv
Child-SP          RetAddr           : Args to Child                                                           : Call Site
ffffd000`3a7bb818 fffff802`ef991b4b : 00000000`00000044 ffffe000`07e7e2a0 00000000`00000f8a 00000000`00000000 : nt!KeBugCheckEx
ffffd000`3a7bb820 fffff800`045c1f0a : 00000000`00000000 ffffd000`3ac72f00 ffffe000`088a1340 ffffe000`088a1340 : nt! ?? ::FNODOBFM::`string'+0x2a9ab
ffffd000`3a7bb930 fffff800`045c3059 : ffffe000`088a1340 ffffd000`3a7bba02 ffffd000`3a7bba80 ffffe000`00000054 : Hamdrv+0x1f0a
ffffd000`3a7bb980 fffff800`045c2a56 : 00000000`00000000 ffffe000`04b02bb0 ffffe000`04b02bb0 ffffe000`04b02d10 : Hamdrv+0x3059
ffffd000`3a7bc040 fffff800`010b7b14 : 00000000`00000000 ffffd000`3a7bc170 00000000`00000000 00000000`00000002 : Hamdrv+0x2a56
ffffd000`3a7bc070 fffff800`012e6115 : ffffe000`062ed430 ffffe000`04b02bb0 ffffe000`00000000 00000000`00000017 : ndis!NdisSendNetBufferLists+0x554
ffffd000`3a7bc250 fffff800`012e2c54 : fffff800`0144a370 00000000`00000000 ffffd000`3a7bc400 ffffe000`000086dd : tcpip!IppFragmentPackets+0x4e5
ffffd000`3a7bc3a0 fffff800`012a2475 : fffff800`0144a370 ffffe000`0657b000 ffffd000`00000000 00000000`00000000 : tcpip!IppDispatchSendPacketHelper+0x94
ffffd000`3a7bc530 fffff800`012e1b02 : 00000000`00000000 00000000`00000017 ffffd000`3a7bca40 ffffe000`02839498 : tcpip!IppPacketizeDatagrams+0x2d5
ffffd000`3a7bc6d0 fffff800`0129c5f2 : 00000000`00000000 ffffe000`00cf1204 fffff800`0144a370 ffffe000`06cd7630 : tcpip!IppSendDatagramsCommon+0x4a2
ffffd000`3a7bc8b0 fffff800`0128ed2e : 00000000`00000000 00000000`000035ce 00000000`0000eb14 00000000`0000080a : tcpip!IpNlpSendDatagrams+0x42
ffffd000`3a7bc8f0 fffff800`012ea33a : ffffe000`0808c100 00000000`00000002 ffffe000`0576a2e8 fffff800`0144a370 : tcpip!UdpSendMessagesOnPathCreation+0x42e
ffffd000`3a7bcd10 fffff800`0128f154 : 00000000`00000000 ffffe000`06214010 ffffe000`05a1a480 00000000`00000000 : tcpip!UdpSendMessages+0x1da
ffffd000`3a7bd140 fffff802`ef8c13f9 : fffff800`00c45b74 fffff800`012edef5 fffff800`00c1dfd8 ffffe000`0169de40 : tcpip!UdpTlProviderSendMessagesCalloutRoutine+0x15
ffffd000`3a7bd170 fffff800`0128e7ac : fffff800`0128f140 ffffd000`3a7bd2f0 ffffe000`04d18d00 00000000`00000000 : nt!KeExpandKernelStackAndCalloutInternal+0xe9
ffffd000`3a7bd2c0 fffff800`00c47629 : ffffe000`0576a280 ffffd000`3a7bdb80 ffffe000`0576a280 ffffe000`0576a280 : tcpip!UdpTlProviderSendMessages+0x6c
ffffd000`3a7bd340 fffff800`00c2f0ca : ffffe000`007d5950 ffffd000`3a7bd908 ffffe000`01467d00 00000000`00000000 : afd!AfdFastDatagramSend+0x579
ffffd000`3a7bd500 fffff802`efbb3f97 : ffffe000`01467d00 00000000`00000000 ffffe000`01467d00 ffffd000`20607180 : afd!AfdFastIoDeviceControl+0x10bb
ffffd000`3a7bd870 fffff802`efbb4d7a : e0000131`daf054d3 0000000c`001f0003 00000000`00000000 00000000`00000000 : nt!IopXxxControlFile+0x3f7
ffffd000`3a7bda20 fffff802`ef9624b3 : 00000000`00000000 ffffd000`3a7bdad8 00000000`00000001 000000a4`ef92f600 : nt!NtDeviceIoControlFile+0x56
ffffd000`3a7bda90 00007ffb`c0dc65ea : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ ffffd000`3a7bdb00)
000000a4`ef92eff8 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x00007ffb`c0dc65ea
From this call stack, we can see various networking related calls and routines.

1. For example, very early on in the stack we have two afd.sys routine calls, which is the Ancillary Function Driver. Ancillary Function Driver (AFD) is the pilot auxiliary function, a Microsoft Windows service, responsible to the BSD socket function. Windows (see Winsock), the most famous application that uses sockets is Apache.

In the Windows GUI, setting AFD is via the Device Manager (devmgmt.msc) and not in services and applications (services.msc). It is displayed in the list of Non-Plug and Play drivers, as the Environment network support (AFD AFD Networking Support Environment).


2. As we move up the call stack, we go into many tcpip.sys (TCP/IP Protocol driver) routines. This is self explanatory.

3. We then have one single ndis.sys (Network Driver Interface Specification driver) routine call. The Network Driver Interface Specification (NDIS) is an application programming interface (API) for network interface cards (NICs). The NDIS forms the Logical Link Control (LLC) sublayer, which is the upper sublayer of the OSI data link layer (layer 2). Therefore, the NDIS acts as the interface between the Media Access Control (MAC) sublayer, which is the lower sublayer of the data link layer, and the network layer (layer 3). 

The NDIS is a library of functions often referred to as a "wrapper" that hides the underlying complexity of the NIC hardware and serves as a standard interface for level 3 network protocol drivers and hardware level MAC drivers. Another common LLC is the Open Data-Link Interface (ODI).

4. We then of course have our three HamDrv.sys (LogMeIn Hamachi Virtual Miniport) driver calls.

5. Finally, we then have our inevitable bug check itself.

Moving on, something very interesting in David's kernel-dump is this: 

ADDITIONAL_DEBUG_TEXT:  USB\VID_04F2&PID_B374
What does this mean? Well, this is a USB device. VID = Vendor ID, and PID =  Product ID. As we can see here in this case, the VID = 04F2 and the PID = B374. As of right now, this doesn't really tell us anything other than a bunch of numbers which isn't very helpful. However, these numbers are actually very important!

If we run them against a VID/PID database, we see that the 04F2 VID = Chicony Electronics Co., Ltd. Great, so we have a vendor! Now, what about the product? Unfortunately, there doesn't seem to be anything in a database for PID B374. With this said, I went to Google and hoped for the best. With very little searches available, the best guess would be to assume that this is a Chicony webcam and/or camera based device. I will have to ask David. Moving on...

1: kd> dt nt!_IO_STATUS_BLOCK ffffe00007e7e2a0
   +0x000 Status           : 0n18350086
   +0x000 Pointer          : 0x00000001`01180006 Void
   +0x008 Information      : 0xffffe000`00b69b10
 Now that we have this, let's check the STATUS field layout:

1: kd> ? 0n18350086
Evaluate expression: 18350086 = 00000000`01180006
 Let's go forward more:

1: kd> .formats 0n18350086
Evaluate expression:
  Hex:     00000000`01180006
  Decimal: 18350086
  Octal:   0000000000000106000006
  Binary:  00000000 00000000 00000000 00000000 00000001 00011000 00000000 00000110
  Chars:   ........
  Time:    Sat Aug 01 05:14:46 1970
  Float:   low 2.7918e-038 high 0
  Double:  9.06615e-317
 Assuming I am correct, '00000000`01180006' falls under:

NT_SUCCESS(Status) Evaluates to TRUE if the return value specified by Status is a success type (0 − 0x3FFFFFFF) or an informational type (0x40000000 − 0x7FFFFFFF). This is different than our first kernel as the first kernel returned NT_ERROR as opposed to NT_SUCCESS).
Moving forward, let's take a closer look at the IRP:

1: kd> !irp ffffe00007e7e2a0 1
Irp is active with 1 stacks 3 is current (= 0xffffe00000d39070)
 Mdl=ffffe00000b69b10: No System Buffer: Thread fffff802ef8af540:  Irp is completed.  Pending has been returned
Flags = 00060900
ThreadListEntry.Flink = ffffe000016fb590
ThreadListEntry.Blink = ffffe000076866d8
IoStatus.Status = 00000000
IoStatus.Information = 00000054
RequestorMode = 00000001
Cancel = 00
CancelIrql = 0
ApcEnvironment = 00
UserIosb = 01471f40
UserEvent = ffffe00001136790
Overlay.AsynchronousParameters.UserApcRoutine = 00000000
Overlay.AsynchronousParameters.UserApcContext = 01471f40
Overlay.AllocationSize = 00000000 - 00000000
CancelRoutine = fffff800045c188c   Hamdrv
UserBuffer = 00000000
&Tail.Overlay.DeviceQueueEntry = ffffe00007e7e318
Tail.Overlay.Thread = fffff802ef8af540
Tail.Overlay.AuxiliaryBuffer = fffff802efdd78dc
Tail.Overlay.ListEntry.Flink = 00000000
Tail.Overlay.ListEntry.Blink = 00000000
Tail.Overlay.CurrentStackLocation = ffffe00000d39070
Tail.Overlay.OriginalFileObject = 00000000
Tail.Apc = 00580012
Tail.CompletionKey = 00580012
     cmd  flg cl Device   File     Completion-Context
 [  3, 0]   0  0 ffffe000029bf900 00000000 00000000-00000000   
           \Driver\hamachi
            Args: 00000000 00000000 00000000 00000000
Great, we're consistent here. The IRP is completed and the pending has returned. We also appear to have the same # (3) which appears to be the Major Function Code for IRP_MJ_READ and the same for the Minor Function Code, which is IRP_MN_NORMAL. Let's go ahead and take 'ffffe000029bf900' and look into it:

1: kd> !devstack ffffe000029bf900
  !DevObj   !DrvObj            !DevExt   ObjectName
> ffffe000029bf900  \Driver\hamachi    ffffe000029bfa50  HamachiTap.FBA77CC0-00D0-459A-AEA1-63052984BC34
Very nice, more Hamachi. Let's now go ahead and take 'ffffe00000d39070' (CurrentStackLocation) and look into it:

1: kd> dt nt!_IO_STACK_LOCATION 0xffffe00000d39070
   +0x000 MajorFunction    : 0x5 ''
   +0x001 MinorFunction    : 0 ''
   +0x002 Flags            : 0xd8 ''
   +0x003 Control          : 0 ''
   +0x008 Parameters       : <unnamed-tag>
   +0x028 DeviceObject     : (null)
   +0x030 FileObject       : (null)
   +0x038 CompletionRoutine : (null)
   +0x040 Context          : (null)
Yes, more consistency! Remember, the data structure shows the current stack location of the IRP. Since IRPs can be handled by different Driver Objects through Driver Stacks, and therefore may be a indication of the different driver handling a different IRP. Moving on...

1: kd> !drvobj ffffe00005c66230 f
Driver object (ffffe00005c66230) is for:
 \Driver\hamachi
Driver Extension List: (id , addr)
(4e4d4944 ffffe000010b5810) 
Device Object list:
ffffe000029bf900  ffffe000075cf050
Let's take  'ffffe000075cf050' (part of the Device Object list) and look into it:

1: kd> !devobj ffffe000075cf050
Device object (ffffe000075cf050) is for:
 NDMP18 \Driver\hamachi DriverObject ffffe00005c66230
Current Irp 00000000 RefCount 0 Type 00000017 Flags 00002050
Dacl ffffc10100d9bf51 DevExt ffffe000075cf1a0 DevObjExt ffffe000075d0720
ExtensionFlags (0x00000800)  DOE_DEFAULT_SD_PRESENT
Characteristics (0x00000100)  FILE_DEVICE_SECURE_OPEN
AttachedTo (Lower) ffffe00007098e50 \Driver\PnpManager
Device queue is not busy.
More consistency! Let's keep going...

1: kd> !devstack ffffe000075cf050
  !DevObj   !DrvObj            !DevExt   ObjectName
> ffffe000075cf050  \Driver\hamachi    ffffe000075cf1a0  NDMP18
  ffffe00007098e50  \Driver\PnpManager 00000000  000000a4
!DevNode ffffe00000c15d30 :
  DeviceInst is "ROOT\NET\0000"
  ServiceName is "Hamachi"
More and more consistency! Also, if you've been noticing, I have been highlighting NDMP(n) as we've been going through this. The reason for this is NDMP(n) = Network Data Management Protocol, which is a protocol invented by the NetApp and Legato companies, meant to transport data between network attached storage (NAS) devices and backup devices. This removes the need for transporting the data through the backup server itself, thus enhancing speed and removing load from the backup server.

This is likely why we see so many network related routines in the call stack which was inevitably responsible for the IRP.

Let's now go ahead and run a !thread on the thread in which the IRP completed:


1: kd> !thread fffff802ef8af540
fffff802ef8af540 is not a thread object, interpreting as stack value...
TYPE mismatch for thread object at fffff802ef8af540
Oh no! We've reached a very unfortunate dead end in David's kernel-dump. After debugging for over an hour, I was not about to give up. I went ahead and opened up another kernel-dump (provided by Maximilian Kless), and quickly got to where we are. The thread extension worked this time!

2: kd> !thread ffffe00004edd880
THREAD ffffe00004edd880  Cid 08a0.0968  Teb: 00007ff63f615000 Win32Thread: fffff901400f86d0 RUNNING on processor 3
IRP List:
    ffffe00005a625b0: (0006,0118) Flags: 00060000  Mdl: 00000000
    ffffe00005a2c1e0: (0006,0118) Flags: 00060000  Mdl: 00000000
    ffffe00002904cc0: (0006,01f0) Flags: 00060900  Mdl: 00000000
    ffffe000025cc270: (0006,01f0) Flags: 00060800  Mdl: 00000000
Not impersonating
DeviceMap                 ffffc0000000c250
Owning Process            ffffe00004bd0900       Image:         hamachi-2.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      227813         Ticks: 0
Context Switch Count      8289           IdealProcessor: 0            
UserTime                  00:00:00.281
KernelTime                00:00:00.406
Win32 Start Address 0x00007ffe21c281b0
Stack Init ffffd00035a46c90 Current ffffd00035a45fe0
Base ffffd00035a47000 Limit ffffd00035a41000 Call 0
Priority 8 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           : Args to Child                                                           : Call Site
00000000`01f8f7e0 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x00007ff6`40361752
At this point, I am honestly jumping up and down in my chair. We see a huge consistency here, and we have one more step to go. Let's finally go ahead and run an !irp on the 3rd value in the IRP list:

2: kd> !irp ffffe00002904cc0
Irp is active with 2 stacks 2 is current (= 0xffffe00002904dd8)
 No Mdl: No System Buffer: Thread ffffe00004edd880:  Irp stack trace. 
     cmd  flg cl Device   File     Completion-Context
 [  0, 0]   0  0 00000000 00000000 00000000-00000000   

            Args: 00000000 00000000 00000000 00000000
>[  3, 0]   0  1 ffffe00001f37060 ffffe00005de5f20 00000000-00000000    pending
           \FileSystem\Npfs
            Args: 00002000 00000000 00000000 00000000
And that's it, we have consistency! Npfs.sys marked as status PENDING in TWO kernel dumps on TWO ENTIRELY different systems. Also, even though David's kernel-dump didn't show us a !thread, it showed alarming consistency in regards to many other things seen in the dump itself.

Now, with all of this found, I will bring it up to Harry and my fellow analysts, and see if we can explain what role Npfs.sys has in these crashes. More updates to come! I will of course also take a look at the other kernel-dumps I was provided. Busy busy day.

Update #6 - This is not an analysis update, mostly because the analysis is finished. From this point on, it's of course up to LogMeIn to examine their driver code and see why things like Npfs.sys remain PENDING, etc.

Aside from receiving any new kernel-dumps from any kind individuals to quickly check for consistency, Harry (x BlueRobot) and I have analyzed this to the point where we can do no more as we are only volunteer analysts with an interest in helping people. This update is instead directed more towards how much of a huge thank you I want to extend.

First off, I simply couldn't have done a large amount of this without Harry's help. As much as I appreciate all of the recognition I am receiving for this, Harry deserves it just as much as I do. It would have taken me a very long time and it would have been very stressful. Harry provided tons of documentation and debugging related information, and both of us have learned a very large amount from this. So far I have worked on two bugs with him (HP-Envy-700-074 BIOS bug) and now this. So far, both have been extremely successful.

I'd also like to thank ZigZag3143 for not only submitting tickets and remaining in contact with LogMeIn, and providing up to date threads across various communities displaying the sheer number of people having this problem, but for being a good friend as always and lending a hand.

Next, I'd like to of course extend an extremely large thanks to everyone around the world who has sent me their kernel-dumps simply because they wanted to be a part of this and help contribute to solving the issue. I am going to be making a list of the people who contributed their kernel-dumps, by name. As I am still receiving (or waiting to receive) kernel-dumps from various people who've contacted me, it will likely be an ongoing list.

Essentially, what I am trying to convey here is that this was not just me, it was a huge team effort. I had tons of help from Harry, and extremely kind people from all around the world. I truly appreciate situations like this more than anything in life, because they are very important. As an analyst with a huge passion in debugging, this was not only a beautiful learning experience, but I got to work with a lot of extremely great people. It truly goes to show that when people work together, you can essentially solve any problem.

Update #7 - It appears as I stated at the very top that this bug has been SOLVED! Again, a huge thank you to everyone involved, and to LogMeIn for fixing it promptly for its customers.