When 0xFC is not entirely a driver issue...

I very recently received a crash dump from a user in which they stated their system either rebooted randomly or bug checked during streamed content, etc. The user had their system built by a 3rd party, specifically Power4PC in Belgium. Given it was relatively new and was under their warranty type guarantee of course, the user shipped it back to them and they ran "diagnostics". During these "diagnostics", they reported to the user that there were, and I quote, "faulty drivers", so it doesn't fall under their warranty guarantee, and the user had to ultimately pay money to get the system back.

I contemplated whether or not I wanted to name drop Power4PC, but given they pretty much robbed the user of money and went about it extremely poorly, I decided I was going to. Interestingly enough the user gets their system back and it's still crashing, what a surprise! Let's take a look at the crash dump.

 2: kd> .bugcheck  
 Bugcheck code 000000FC  
 Arguments ffffc001`b4bf7010 d6b00001`05e5b963 ffffd000`bfe2adb0 00000000`00000002  

So here's our bug check - ATTEMPTED_EXECUTE_OF_NOEXECUTE_MEMORY (0xFC). I won't go into the specifics regarding executable/non-exectuable memory, but know that it mostly has to do with security, such as attempting to prevent things such as buffer overflows. Windows (and other OS' of course, although all implemented a bit differently) defines certain pages of memory as non-executable, which in turn tells the processor not to execute the data stored in those pages.

When an attempt is made to execute memory that is defined as non-executable, Windows throws a bug check for security (and disaster prevention as usual) purposes. Do note that there's of course exploits and ways around non-executable memory, such as the well known return to libc (function RET pointed to library/popular function or system API).

Before going into the various parameters, let's check and see if PAE is enabled so we know what to expect. To know what register to look for to see if PAE is enabled/in use, the best way to go about that is to check your corresponding processor manual. For example, on this system:

 2: kd> !cpuinfo  
 CP F/M/S Manufacturer MHz PRCB Signature  MSR 8B Signature Features  
  0 6,60,3 GenuineIntel 3399 0000000900000000          3d193fff  
  1 6,60,3 GenuineIntel 3399 0000000900000000          3d193fff  
  2 6,60,3 GenuineIntel 3399 0000000900000000          3d193fff  
  3 6,60,3 GenuineIntel 3399 0000000900000000          3d193fff  
            Cached Update Signature 0000000900000000  
            Initial Update Signature 0000000900000000  

We can see that's it a quad core Intel processor, therefore we'd check the Intel manual. According to the Intel manual (2.5 CONTROL REGISTERS), PAE is stored on x64 (and x86 I believe as well) in CR4.

CR4 — Contains a group of flags that enable several arch itectural extensions, and indicate operating system or executive support for specific processor capabilities. The control registers can be read and loaded (or modified) using the move-to-or-from-control-registers forms of the MOV instruction. In protected mode, the MOV instructions allow the control registers to be read or loaded (at privilege level 0 only). This restriction means that application programs or operating-system procedures (running at privilege levels 1, 2, or 3) are prevented from reading or loading the control registers.

If we use the r command (show registers) along with cr4:

 2: kd> r cr4  
 Last set context:  
 cr4=00000000001506f8  

We have our cr4 address now, and from this point can use .formats to check the bits:

 2: kd> .formats 00000000001506f8  
 Evaluate expression:  
  Hex:   00000000`001506f8  
  Decimal: 1378040  
  Octal:  0000000000000005203370  
  Binary: 00000000 00000000 00000000 00000000 00000000 00010101 00000110 11111000  

Starting from the right most zero, it's the fifth bit. We can see it's enabled, which helps. The first parameter of the bug check contains the virtual address whose execution was attempted:

 2: kd> !pte ffffc001b4bf7010  
                       VA ffffc001b4bf7010  
 PXE at FFFFF6FB7DBEDC00  PPE at FFFFF6FB7DB80030  PDE at FFFFF6FB70006D28  PTE at FFFFF6E000DA5FB8  
 contains 0000000000A75863 contains 0000000000A74863 contains 00000001087E8863 contains D6B0000105E5B963  
 pfn a75    ---DA--KWEV pfn a74    ---DA--KWEV pfn 1087e8  ---DA--KWEV pfn 105e5b  -G-DA--KW-V  

 2: kd> dt nt!_MMPTE u.Hard  
   +0x000 u   :   
    +0x000 Hard  : _MMPTE_HARDWARE  

Here we can see the data type regarding the PDE. Let's go further and dump the PDE from the virtual address:

 2: kd> dt _MMPTE_HARDWARE FFFFF6FB70006D28  
 nt!_MMPTE_HARDWARE  
   +0x000 Valid      : 0y1  
   +0x000 Dirty1      : 0y1  
   +0x000 Owner      : 0y0  
   +0x000 WriteThrough   : 0y0  
   +0x000 CacheDisable   : 0y0  
   +0x000 Accessed     : 0y1  
   +0x000 Dirty      : 0y1  
   +0x000 LargePage    : 0y0  
   +0x000 Global      : 0y0  
   +0x000 CopyOnWrite   : 0y0  
   +0x000 Unused      : 0y0  
   +0x000 Write      : 0y1  
   +0x000 PageFrameNumber : 0y000000000000000100001000011111101000 (0x1087e8)  
   +0x000 reserved1    : 0y0000  
   +0x000 SoftwareWsIndex : 0y00000000000 (0)  
   +0x000 NoExecute    : 0y0  

If we pay attention to the various flags from PDE section of the VA dump, we can see it's:

• D - Dirty, as in the page was previously written to.

• A - Accessed, as in the page (or table) has been previously accessed (read).

• K - This page is owned by kernel-mode, not user-mode.

• W - Writable, as in the page is able to be written to (not just read).

• E - Executable, as in the page is executable.

• V - Valid, as in the page is located in physical memory. 
•  

However if we take a look at the PTE section of the VA dump, we can see it's:

• G - Global, as in the TLB won't be flushed upon a context switch.

• D - Dirty, as in the page was previously written to.

• A - Accessed, as in the page (or table) has been previously accessed (read).

• K - This page is owned by kernel-mode, not user-mode.

• W - Writable, as in the page is able to be written to (not just read).

• V - Valid, as in the page is located in physical memory.

Notice anything missing? The executable flag. In addition however, there's the global flag. The second parameter of the bug check contains the contents of the page table entry:

 2: kd> !pte d6b0000105e5b963  
                       VA d6b0000105e5b963  
 PXE at FFFFF6FB7DBED000  PPE at FFFFF6FB7DA00020  PDE at FFFFF6FB40004178  PTE at FFFFF6800082F2D8  
 contains 00C00001364F5867 contains 0000000000000000  
 pfn 1364f5  ---DA--UWEV not valid  
 WARNING: noncanonical VA, accesses will fault !  

We can see it's a noncanonical address, therefore it's of course going to fail. Let's dump the stack so we can find the pagefault:

 2: kd> kv  
 Child-SP     RetAddr      : Args to Child                              : Call Site  
 ffffd000`bfe2ab18 fffff803`ac420096 : 00000000`000000fc ffffc001`b4bf7010 d6b00001`05e5b963 ffffd000`bfe2adb0 : nt!KeBugCheckEx  
 ffffd000`bfe2ab20 fffff803`ac30b7ce : d6b00001`05e5b963 00000980`00000000 ffffd000`bfe2ad40 fffff801`16e343d8 : nt! ?? ::FNODOBFM::`string'+0x4b9e6  
 ffffd000`bfe2ab60 fffff803`ac2d9e78 : 00000000`00000008 ffffe000`f1ee5900 ffffd000`bfe2adb0 ffffe000`f0618420 : nt!MiSystemFault+0xb5e  
 ffffd000`bfe2ac00 fffff803`ac3ce42f : ffffc001`b4bf7010 00000000`00000000 00000000`00000000 ffffd000`bfe2adb0 : nt!MmAccessFault+0x758  
 ffffd000`bfe2adb0 ffffc001`b4bf7010 : ffffc001`b4bf7010 ffffc001`b4bf7010 ffffd000`bfe2b060 ffffc001`00000000 : nt!KiPageFault+0x12f (TrapFrame @ ffffd000`bfe2adb0)  
 ffffd000`bfe2af48 ffffc001`b4bf7010 : ffffc001`b4bf7010 ffffd000`bfe2b060 ffffc001`00000000 00000000`00000000 : 0xffffc001`b4bf7010  
 ffffd000`bfe2af50 ffffc001`b4bf7010 : ffffd000`bfe2b060 ffffc001`00000000 00000000`00000000 00000000`00000000 : 0xffffc001`b4bf7010  
 ffffd000`bfe2af58 ffffd000`bfe2b060 : ffffc001`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0xffffc001`b4bf7010  
 ffffd000`bfe2af60 ffffc001`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`0000000f : 0xffffd000`bfe2b060  
 ffffd000`bfe2af68 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`0000000f fffff801`17ae846e : 0xffffc001`00000000  

 2: kd> .trap ffffd000`bfe2adb0  
 NOTE: The trap frame does not contain all registers.  
 Some register values may be zeroed or incorrect.  
 rax=0000000000000001 rbx=0000000000000000 rcx=ffffd000bfe2ae20  
 rdx=0000000000000000 rsi=0000000000000000 rdi=0000000000000000  
 rip=ffffc001b4bf7010 rsp=ffffd000bfe2af48 rbp=ffffd000bfe2b060  
  r8=fffff80116e1bcd0 r9=0000000000000000 r10=0000000000000000  
 r11=ffffd000bfe2aee0 r12=0000000000000000 r13=0000000000000000  
 r14=0000000000000000 r15=0000000000000000  
 iopl=0     nv up ei ng nz na pe nc  
 ffffc001`b4bf7010 400200     add   al,byte ptr [rax] ds:00000000`00000001=??  

We failed on a very basic arithmetic instruction, which was to add the single byte stored at rax to the 8 least significant bits of AX (whereas AX itself is the 16 least significant bits of EAX).

 2: kd> !pte rax  // Or 0000000000000001
                       VA 0000000000000001  
 PXE at FFFFF6FB7DBED000  PPE at FFFFF6FB7DA00000  PDE at FFFFF6FB40000000  PTE at FFFFF68000000000  
 contains 00C00001364F5867 contains 0610000136476867 contains 0000000000000000  
 pfn 1364f5  ---DA--UWEV pfn 136476  ---DA--UWEV not valid  

It's not uncommon whatsoever that buggy and malicious drivers cause 0xFC bug checks, but in this case I think the company that built the system couldn't figure it out, took a look at the bug check, Googled it, saw the MSDN article/a few forum posts that it was a generally a driver related bug check, said it's being caused by "faulty drivers" so they don't have to look into it, and then charged money to do absolutely nothing. Also, not only is the thread in regards to the thread completely bare and showing no signs of "faulty drivers", let's take a look and see what actual 3rd party drivers there are on the system:

 3: kd> lm  
 start       end         module name  
 fffff801`15828000 fffff801`1588e000  mcupdate_GenuineIntel  (deferred)         
 fffff801`1588e000 fffff801`1589c000  werkernel  (deferred)         
 fffff801`1589c000 fffff801`158fd000  CLFS    (deferred)         
 fffff801`158fd000 fffff801`1591f000  tm     (deferred)         
 fffff801`1591f000 fffff801`15934000  PSHED   (deferred)         
 fffff801`15934000 fffff801`1593e000  BOOTVID  (deferred)         
 fffff801`1593e000 fffff801`159c6000  CI     (deferred)         
 fffff801`15a00000 fffff801`15a69000  spaceport  (deferred)         
 fffff801`15a83000 fffff801`15ae0000  msrpc   (deferred)         
 fffff801`15ae0000 fffff801`15baf000  Wdf01000  (deferred)         
 fffff801`15baf000 fffff801`15bc0000  WDFLDR   (deferred)         
 fffff801`15bc0000 fffff801`15bd8000  acpiex   (deferred)         
 fffff801`15bd8000 fffff801`15be3000  WppRecorder  (deferred)         
 fffff801`15c00000 fffff801`15c8b000  cng    (deferred)         
 fffff801`15c96000 fffff801`15ca0000  msisadrv  (deferred)         
 fffff801`15ca0000 fffff801`15ce8000  pci    (deferred)         
 fffff801`15cf4000 fffff801`15d7e000  ACPI    (deferred)         
 fffff801`15d7e000 fffff801`15d88000  WMILIB   (deferred)         
 fffff801`15d88000 fffff801`15d95000  vdrvroot  (deferred)         
 fffff801`15d95000 fffff801`15db1000  pdc    (deferred)         
 fffff801`15db1000 fffff801`15dc9000  partmgr  (deferred)         
 fffff801`15dc9000 fffff801`15dde000  volmgr   (deferred)         
 fffff801`15e00000 fffff801`15e55000  CLASSPNP  (deferred)         
 fffff801`15e72000 fffff801`15ed1000  volmgrx  (deferred)         
 fffff801`15ed1000 fffff801`15eec000  mountmgr  (deferred)         
 fffff801`15eec000 fffff801`15f17000  Wof    (deferred)         
 fffff801`15f17000 fffff801`15f59000  WdFilter  (deferred)         
 fffff801`15f59000 fffff801`15f89000  ksecpkg  (deferred)         
 fffff801`15f89000 fffff801`15fcf000  rdyboost  (deferred)         
 fffff801`1601d000 fffff801`162ee000  iaStorA  (deferred)         
 fffff801`162ee000 fffff801`1634d000  storport  (deferred)         
 fffff801`1634d000 fffff801`16367000  EhStorClass  (deferred)         
 fffff801`16367000 fffff801`163c3000  fltmgr   (deferred)         
 fffff801`163c3000 fffff801`163d9000  fileinfo  (deferred)         
 fffff801`163d9000 fffff801`163f5000  disk    (deferred)         
 fffff801`16400000 fffff801`16415000  crashdmp  (deferred)         
 fffff801`16418000 fffff801`1660e000  Ntfs    (deferred)         
 fffff801`1660e000 fffff801`1662a000  ksecdd   (deferred)         
 fffff801`1662a000 fffff801`1663a000  pcw    (deferred)         
 fffff801`1663a000 fffff801`16645000  Fs_Rec   (deferred)         
 fffff801`16645000 fffff801`1675d000  ndis    (deferred)         
 fffff801`1675d000 fffff801`167d5000  NETIO   (deferred)         
 fffff801`167d5000 fffff801`167e4000  intelpep  (deferred)         
 fffff801`16800000 fffff801`16895000  fvevol   (deferred)         
 fffff801`16895000 fffff801`168e4000  volsnap  (deferred)         
 fffff801`168e5000 fffff801`16b56000  tcpip   (deferred)         
 fffff801`16b56000 fffff801`16bc2000  fwpkclnt  (deferred)         
 fffff801`16bc2000 fffff801`16be7000  wfplwfs  (deferred)         
 fffff801`16be7000 fffff801`16bfe000  mup    (deferred)         
 fffff801`16e00000 fffff801`16e61000  dxgmms1  (pdb symbols)     c:\symbols\dxgmms1.pdb\55D4ABFFE5B6411898E90F5E6E72B1071\dxgmms1.pdb  
 fffff801`16e61000 fffff801`17132000  dump_iaStorA  (deferred)         
 fffff801`17132000 fffff801`17175000  srvnet   (deferred)         
 fffff801`1717f000 fffff801`171ad000  cdrom   (deferred)         
 fffff801`171ad000 fffff801`171b6000  Null    (deferred)         
 fffff801`171b6000 fffff801`171be000  Beep    (deferred)         
 fffff801`171be000 fffff801`171cc000  BasicRender  (deferred)         
 fffff801`17200000 fffff801`17220000  tdx    (deferred)         
 fffff801`1722c000 fffff801`173ad000  dxgkrnl  (pdb symbols)     c:\symbols\dxgkrnl.pdb\9D44B2A5938E41528DA86348D6D1F5C21\dxgkrnl.pdb  
 fffff801`173ad000 fffff801`173bf000  watchdog  (deferred)         
 fffff801`173bf000 fffff801`173d1000  BasicDisplay  (deferred)         
 fffff801`173d1000 fffff801`173e5000  Npfs    (deferred)         
 fffff801`173e5000 fffff801`173f1000  Msfs    (deferred)         
 fffff801`173f1000 fffff801`173ff000  TDI    (deferred)         
 fffff801`17411000 fffff801`1745d000  netbt   (deferred)         
 fffff801`1745d000 fffff801`174ef000  afd    (deferred)         
 fffff801`174ef000 fffff801`17519000  pacer   (deferred)         
 fffff801`17519000 fffff801`1752a000  netbios  (deferred)         
 fffff801`1752a000 fffff801`1759a000  rdbss   (deferred)         
 fffff801`1759a000 fffff801`175b6000  drmk    (deferred)         
 fffff801`175b6000 fffff801`175dd000  AMDACPKSL  (deferred)         
 fffff801`17600000 fffff801`1766a000  usbhub   (deferred)         
 fffff801`17679000 fffff801`17707000  csc    (deferred)         
 fffff801`17707000 fffff801`17715000  nsiproxy  (deferred)         
 fffff801`17715000 fffff801`17721000  npsvctrig  (deferred)         
 fffff801`17721000 fffff801`1772d000  mssmbios  (deferred)         
 fffff801`1772d000 fffff801`17753000  dfsc    (deferred)         
 fffff801`17763000 fffff801`1777a000  ahcache  (deferred)         
 fffff801`1777a000 fffff801`17789000  CompositeBus  (deferred)         
 fffff801`17789000 fffff801`17794000  kdnic   (deferred)         
 fffff801`17794000 fffff801`177a5000  umbus   (deferred)         
 fffff801`177a5000 fffff801`177ec000  portcls  (deferred)         
 fffff801`17800000 fffff801`1784e000  ks     (deferred)         
 fffff801`1784e000 fffff801`17859000  rdpbus   (deferred)         
 fffff801`17859000 fffff801`17865000  USBD    (deferred)         
 fffff801`17874000 fffff801`1790a000  atikmpag  (no symbols)        
 fffff801`1790a000 fffff801`17979000  USBPORT  (deferred)         
 fffff801`17979000 fffff801`17997000  intelppm  (deferred)         
 fffff801`17997000 fffff801`179a8000  ISCTD64  (deferred)         
 fffff801`179a8000 fffff801`179b3000  NdisVirtualBus  (deferred)         
 fffff801`179b3000 fffff801`179ee000  AtihdWB6  (deferred)         
 fffff801`17a00000 fffff801`17a18000  usbehci  (deferred)         
 fffff801`17a18000 fffff801`17a32000  serial   (deferred)         
 fffff801`17a32000 fffff801`17a3c000  wmiacpi  (deferred)         
 fffff801`17a3c000 fffff801`18cc0000  atikmdag  (no symbols)        
 fffff801`18cc0000 fffff801`18cc1600  swenum   (deferred)         
 fffff801`18cc2000 fffff801`18cc7300  ksthunk  (deferred)         
 fffff801`18cca000 fffff801`18ce3000  HDAudBus  (deferred)         
 fffff801`18ce3000 fffff801`18d38000  USBXHCI  (deferred)         
 fffff801`18d38000 fffff801`18d6a000  ucx01000  (deferred)         
 fffff801`18d6a000 fffff801`18d7d000  HECIx64  (deferred)         
 fffff801`18d7d000 fffff801`18df1000  e1d64x64  (deferred)         
 fffff801`18df1000 fffff801`18dfe000  serenum  (deferred)         
 fffff801`18e00000 fffff801`18ea9000  peauth   (deferred)         
 fffff801`18eb7000 fffff801`18f2f000  UsbHub3  (deferred)         
 fffff801`18f2f000 fffff801`18f3d000  monitor  (deferred)         
 fffff801`18f3d000 fffff801`18f61000  luafv   (deferred)         
 fffff801`18f61000 fffff801`18f75000  lltdio   (deferred)         
 fffff801`18f75000 fffff801`18f8d000  rspndr   (deferred)         
 fffff801`18f8d000 fffff801`18fd8000  mrxsmb10  (deferred)         
 fffff801`18fd8000 fffff801`18ff5000  Ndu    (deferred)         
 fffff801`19000000 fffff801`1900c000  dump_diskdump  (deferred)         
 fffff801`1900c000 fffff801`19022000  dump_dumpfve  (deferred)         
 fffff801`1902b000 fffff801`19362400  RTKVHD64  (deferred)         
 fffff801`19363000 fffff801`1938a000  usbccgp  (deferred)         
 fffff801`1938a000 fffff801`19398000  hidusb   (deferred)         
 fffff801`19398000 fffff801`193b7000  HIDCLASS  (deferred)         
 fffff801`193b7000 fffff801`193bef00  HIDPARSE  (deferred)         
 fffff801`193bf000 fffff801`193cd000  kbdhid   (deferred)         
 fffff801`193cd000 fffff801`193dd000  kbdclass  (deferred)         
 fffff801`193dd000 fffff801`193ea000  mouhid   (deferred)         
 fffff801`193ea000 fffff801`193fa000  mouclass  (deferred)         
 fffff801`19600000 fffff801`19639000  mrxsmb20  (deferred)         
 fffff801`19639000 fffff801`19644000  secdrv   (deferred)         
 fffff801`1964e000 fffff801`19748000  HTTP    (deferred)         
 fffff801`19748000 fffff801`19768000  bowser   (deferred)         
 fffff801`19768000 fffff801`1977f000  mpsdrv   (deferred)         
 fffff801`1977f000 fffff801`197eb000  mrxsmb   (deferred)         
 fffff801`197eb000 fffff801`197fd000  tcpipreg  (deferred)         
 fffff801`19a12000 fffff801`19abe000  srv2    (deferred)         
 fffff801`19abe000 fffff801`19b4c000  srv    (deferred)         
 fffff801`19b4c000 fffff801`19b79000  tunnel   (deferred)         
 fffff801`19b79000 fffff801`19b8f000  mslldp   (deferred)         
 fffff801`19b8f000 fffff801`19bae000  WdNisDrv  (deferred)         
 fffff801`19bae000 fffff801`19bbe000  condrv   (deferred)         
 fffff803`ab4f4000 fffff803`ab4fd000  kd     (deferred)         
 fffff803`ac205000 fffff803`ac275000  hal    (deferred)         
 fffff803`ac275000 fffff803`aca0b000  nt     (pdb symbols)     c:\symbols\ntkrnlmp.pdb\E018720B8C414EF799EACB23114763921\ntkrnlmp.pdb  
 fffff960`00018000 fffff960`00430000  win32k   (pdb symbols)     c:\symbols\win32k.pdb\F8BA0C5F0A844DC199DAC8CAB88C653B2\win32k.pdb  
 fffff960`00746000 fffff960`0074f000  TSDDD   (deferred)         
 fffff960`0091c000 fffff960`00957000  cdd    (deferred)  

I give or take count ~7 3rd party drivers, none of which show any apparent or obvious evidence to being faulty and are essentially necessary to the hardware to properly communicate with the OS (minus Intel Rapid Storage, which unless the user was running a RAID config could easily uninstall and replace with default MSFT AHCI). Instead to me, this right away looked like a RAM issue as opposed to a driver issue.

I had the user run Memtest right away as opposed to verifier given the signs in the dumps and the mention of random reboots, and within a few minutes there were over 700+ errors. Now the user is hopefully getting their money back, free shipping to and from, and new RAM.