Mark's Sysinternals Blog

Last week when I was testing the latest version of RootkitRevealer (RKR) I ran a scan on one of my systems and was shocked to see evidence of a rootkit. Rootkits are cloaking technologies that hide files, Registry keys, and other system objects from diagnostic and security software, and they are usually employed by malware attempting to keep their implementation hidden (see my “Unearthing Rootkits” article from thre June issue of Windows IT Pro Magazine for more information on rootkits). The RKR results window reported a hidden directory, several hidden device drivers, and a hidden application:



Given the fact that I’m careful in my surfing habits and only install software from reputable sources I had no idea how I’d picked up a real rootkit, and if it were not for the suspicious names of the listed files I would have suspected RKR to have a bug. I immediately ran Process Explorer and Autoruns to look for evidence of code that would activate the rootkit each boot, but I came up empty with both tools. I next turned to LiveKd, a tool I wrote for Inside Windows 2000 and that lets you explore the internals of a live system using the Microsoft kernel debugger, to determine what component was responsible for the cloaking.

Rootkits that hide files, directories and Registry keys can either execute in user mode by patching Windows APIs in each process that applications use to access those objects, or in kernel mode by intercepting the associated kernel-mode APIs. A common way to intercept kernel-mode application APIs is to patch the kernel’s system service table, a technique that I pioneered with Bryce for Windows back in 1996 when we wrote the first version of Regmon. Every kernel service that’s exported for use by Windows applications has a pointer in a table that’s indexed with the internal service number Windows assigns to the API. If a driver replaces an entry in the table with a pointer to its own function then the kernel invokes the driver function any time an application executes the API and the driver can control the behavior of the API.

It’s relatively easy to spot system call hooking simply by dumping the contents of the service table: all entries should point at addresses that lie within the Windows kernel; any that don’t are patched functions. Dumping the table in Livekd revealed several patched functions:



I listed one of the intercepting functions and saw that it was part of the Aries.sys device driver, which was one of the images I had seen cloaked in the $sys$filesystem directory:



Armed with the knowledge of what driver implemented the cloaking I set off to see if I could disable the cloak and expose the hidden processes, files, directories, and Registry data. Although RKR indicated that the \Windows\System32\$sys$filesystem directory was hidden from the Windows API, it’s common for rootkits to hide directories from a directory listing, but not to prevent a hidden directory from being opened directly. I therefore checked to see if I could examine the files within the hidden directory by opening a command prompt and changing into the hidden directory. Sure enough, I was able to enter and access most of the hidden files:



Perhaps renaming the driver and rebooting would remove the cloak, but I also wanted to see if Aries.sys was doing more than cloaking so I copied it to an uncloaked directory and loaded it into IDA Pro, a powerful disassembler I use in my exploration of Windows internals. Here’s a screenshot of IDA Pro’s disassembly of the code that calculates the entries in the system service table that correspond to the functions it wants to manipulate:



I studied the driver’s initialization function, confirmed that it patches several functions via the system call table and saw that its cloaking code hides any file, directory, Registry key or process whose name begins with “$sys$”. To verify that I made a copy of Notepad.exe named $sys$notepad.exe and it disappeared from view. Besides being indiscriminate about the objects it cloaks, other parts of the Aries code show a lack of sophistication on the part of the programmer. It’s never safe to unload a driver that patches the system call table since some thread might be just about to execute the first instruction of a hooked function when the driver unloads; if that happens the thread will jump into invalid memory. There’s no way for a driver to protect against this occurrence, but the Aries driver supports unloading and tries to keep track of whether any threads are executing its code. The programmer failed to consider the race condition I’ve described. They’ll have to come up with a new approach to their rootkit sooner or later anyway, since system call hooking does not work at all on x64 64-bit versions of Windows.

After I finished studying the driver's code I rebooted the system. The cloak was gone as I expected and I could see all the previously hidden files in Explorer and Registry keys in Regedit. I doubted that the files had any version information, but ran my Sigcheck utility on them anyway. To my surprise, the majority did have identifying product, file and company strings. I had already recognized Dbghelp.dll and Unicows.dll as Microsoft Windows DLLs by their names. The other files claimed to be part of the “Essential System Tools” product from a company called “First 4 Internet”:



I entered the company name into my Internet browser’s address bar and went to http://www.first4internet.com/. I searched for both the product name and Aries.sys, but came up empty. However, the fact that the company sells a technology called XCP made me think that maybe the files I’d found were part of some content protection scheme. I Googled the company name and came across this article, confirming the fact that they have deals with several record companies, including Sony, to implement Digital Rights Management (DRM) software for CDs.

The DRM reference made me recall having purchased a CD recently that can only be played using the media player that ships on the CD itself and that limits you to at most 3 copies. I scrounged through my CD’s and found it, Sony BMG’s Get Right with the Man (the name is ironic under the circumstances) CD by the Van Zant brothers. I hadn’t noticed when I purchased the CD from Amazon.com that it’s protected with DRM software, but if I had looked more closely at the text on the Amazon.com web page I would have known:



The next phase of my investigation would be to verify that the rootkit and its hidden files were related to that CD’s copy protection, so I inserted the CD into the drive and double-clicked on the icon to launch the player software, which has icons for making up to three copy-protected backup CDs:



Process Explorer showed the player as being from Macromedia, but I noticed an increase in CPU usage by $sys$DRMServer.exe, one of the previously cloaked images, when I pressed the play button. A look at the Services tab of its process properties dialog showed it contains a service named “Plug and Play Device Manager”, which is obviously an attempt to mislead the casual user that stumbles across it in the Services MMC snapin (services.msc) into thinking that it’s a core part of Windows:



I closed the player and expected $sys$DRMServer’s CPU usage to drop to zero, but was dismayed to see that it was still consuming between one and two percent. It appears I was paying an unknown CPU penalty for just having the process active on my system. I launched Filemon and Regmon to see what it might be doing and the Filemon trace showed that it scans the executables corresponding to the running processes on the system every two seconds, querying basic information about the files, including their size, eight times each scan. I was quickly losing respect for the developers of the software:



I still had to confirm the connection between the process and the CD’s player so I took a closer look at each process. Based on the named pipe handles I saw they each had opened when I looked in Process Explorer’s handle view I suspected that the player and $sys$DRMServer communicated via named pipes and so I launched Filemon, checked Named Pipes in the Volumes menu, and confirmed my theory:



At that point I knew conclusively that the rootkit and its associated files were related to the First 4 Internet DRM software Sony ships on its CDs. Not happy having underhanded and sloppily written software on my system I looked for a way to uninstall it. However, I didn’t find any reference to it in the Control Panel’s Add or Remove Programs list, nor did I find any uninstall utility or directions on the CD or on First 4 Internet’s site. I checked the EULA and saw no mention of the fact that I was agreeing to have software put on my system that I couldn't uninstall. Now I was mad.

I deleted the driver files and their Registry keys, stopped the $sys$DRMServer service and deleted its image, and rebooted. As I was deleting the driver Registry keys under HKLM\System\CurrentControlSet\Services I noted that they were either configured as boot-start drivers or members of groups listed by name in the HKLM\System\CurrentControlSet\Control\SafeBoot subkeys, which means that they load even in Safe Mode, making system recovery extremely difficult if any of them have a bug that prevents the system from booting.

When I logged in again I discovered that the CD drive was missing from Explorer. Deleting the drivers had disabled the CD. Now I was really mad. Windows supports device “filtering”, which allows a driver to insert itself below or above another one so that it can see and modify the I/O requests targeted at the one it wants to filter. I know from my past work with device driver filter drivers that if you delete a filter driver’s image, Windows fails to start the target driver. I opened Device Manager, displayed the properties for my CD-ROM device, and saw one of the cloaked drivers, Crater.sys (another ironic name, since it had ‘cratered’ my CD), registered as a lower filter:



Unfortunately, although you can view the names of registered filter drivers in the “Upper filters” and “Lower filters” entries of a device’s Details tab in Device Manager, there’s no administrative interface for deleting filters. Filter registrations are stored in the Registry under HKLM\System\CurrentControlSet\Enum so I opened Regedit and searched for $sys$ in that key. I found the entry configuring the CD’s lower filter:



I deleted the entry, but got an access-denied error. Those keys have security permissions that only allow the Local System account to modify them, so I relaunched Regedit in the Local System account using PsExec: psexec –s –i –d regedit.exe. I retried the delete, succeeded, and searched for $sys$ again. Next I found an entry configuring another one of the drivers, Cor.sys (internally named Corvus), as an upper filter for the IDE channel device and also deleted it. I rebooted and my CD was back.

The entire experience was frustrating and irritating. Not only had Sony put software on my system that uses techniques commonly used by malware to mask its presence, the software is poorly written and provides no means for uninstall. Worse, most users that stumble across the cloaked files with a RKR scan will cripple their computer if they attempt the obvious step of deleting the cloaked files.

While I believe in the media industry’s right to use copy protection mechanisms to prevent illegal copying, I don’t think that we’ve found the right balance of fair use and copy protection, yet. This is a clear case of Sony taking DRM too far.

For an update on the story, read More on Sony: Dangerous Decloaking Patch, EULAs and Phoning Home.

Privileges are special security powers that you assign to accounts in Local Policies->User Rights Assignment node of the Local Security Policy editor, secpol.msc. When a user logs in, the Local Security Authority Subsystem process - Lsass.exe - creates a kernel-mode data structure called a token that contains the list of groups the user belongs to and the privileges assigned to their account. Lsass attaches the token to the first process of the logon session and a user’s security identity propagates through their logon session because child processes inherit a copy of their parent’s token. A process must enable a privilege in its token before it can call an API that requires the privilege, so Windows makes copies in order that a process enabling a privilege won’t affect the privilege state of other processes.

You can view the contents of a process’ token in the Security page of the process’ properties dialog box in Process Explorer, as seen below. The grey background seen on some privileges is just Process Explorer’s visual indicator that a privilege is currently disabled.



One privilege you’ll see by default in every token with the “Default Enabled” flag is SeChangeNotifyPrivilege (the second one in the bottom listing of the screenshot). As its name implies, the privilege is required by the Windows APIs that processes can call to request notification of changes to parts of the file system or Registry that they want to monitor. One such process is Explorer, which monitors directories you open for changes so that it can automatically update the display if a directory’s contents change. You can witness this behavior and the underlying file system request by first creating a test directory, running Filemon, and setting an include filter to the full path of the directory:



Then open an Explorer window to the directory and you’ll see Explorer post a change notification request. The result field of the request will remain blank until the process cancels the request or the directory changes:



Scroll Filemon so that the change notification line is visible and then disable autoscroll in the Options menu. Now open a command prompt, set the current directory to the one you created, and enter “echo hello > out.txt”. When you hit the Enter key, Filemon will fill in the change notification request result and Explorer will refresh its window to show the new file. Take away the SeChangeNotifyPrivilege privilege and Explorer would not be able to automatically update.

Now launch the Local Security Policy editor (LSPE) and navigate to the User Rights Assignment node to see the full list of privileges (Note that LSPE shows both privileges and computer access rights in this node). You won’t find SeChangeNotifyPrivilege in the list, or any of other privilege you see in Process Explorer’s security page for that matter. Process Explorer displays internal privilege names whereas LSPE shows more descriptive and user-friendly aliases. Some of the translations are obvious, like the SeShutdownPrivilege, which LSPE represents as “Shut down the system”, but try to find the entry that maps to SeChangeNotifyPrivilege. Never mind, don’t bother. The user right that LSPE maps to SeChangeNotifyPrivilege is “Bypass traverse checking”:



It’s perfectly natural if you’re having trouble fathoming the connection between SeChangeNotifyPrivilege and “Bypass traverse checking”. The two have nothing to do with each other, other than the fact that Windows overloads the same privilege with two unrelated powers. The security system internally names the privilege according to one if its uses, change notification, and LSPE names it by the other, bypass traverse checking.

Put succinctly, bypass traverse checking gives a process the ability to skip security checks on intermediate directories or Registry keys when it opens a file, directory or Registry key. The following experiment, which requires a Windows XP system that has System Restore enabled, graphically demonstrates the privilege. To begin, log on to the system with an account that’s a member of the local Administrators group.

The Windows XP System Restore facility stores restore points in subdirectories of the System Volume Information directory on each volume that have path names formatted as \System Volume Information\_restore{XXXX}\RPn, where XXXX is the Globally Unique Identifier (GUID) of the Windows installation and n is the internal System Restore identifier of the restore point. When you try to navigate into the restore point directory you should be denied opening System Volume Information. That’s because by default the directory only allows access to the Local System account. If you are able to open it then your computer’s OEM modified permissions on the directory after installing Windows and you will need to open the permissions editor and remove access to the Administrators group for the experiment.

Next run Filemon and configure the include filter as “\rp”. Launch the System Restore wizard by opening the Help and Support center in the Start menu and selecting “Undo changes to your computer with System Restore”. Filemon should record accesses to files stored in the System Restore subdirectory as the wizard scans the existing restore points:



Select one of the lines that shows SUCCESS in the result column, double-click, and an Explorer window will open to the corresponding restore point subdirectory. How is that possible when you don’t have access to System Volume Information? Because you have the Bypass Traverse Checking privilege (SeChangeNotify), NTFS only checks the permissions on the directory or file you open, not on any parent directories. You can see further proof of this by running Notepad and entering “c:\system volume information\tracking.log” in the File Open dialog. Even though you have no acess to the parent directory NTFS still lets you open files within it that you do have access to.

At this point you might believe that Bypass Traverse Checking is a security hole, but most of the time security settings on a directory are inherited by files and subdirectories within it. Thus, if a user doesn’t have access to a directory they won’t have access to items within it, even if they happen to know the names of those items, unless someone explicitly grants them access.

So why does Windows assign Bypass Traverse Checking by default? The primary reason is performance. Instead of having to perform access checks potentially multiple times each time a process opens a file, directory or Registry key, it executes only one check per open. Why does Windows overload a single privilege with two uses? My guess is that only someone on the original Windows NT security team would know.

You can learn more about privileges and user rights, including the precise meaning of each one, in the Security chapter of my book, Windows Internals.

Registry cleaners have always been popular, but I never paid much attention to them. I originally thought that there might be valid reasons for their existence, but over time changed my mind, only to recently recognize that even today they can help maintain Registry hygiene.

It used to be common for developers to write their own application installation routines in order to avoid paying hundreds of dollars for commercial setup toolkits. Their focus in coding installers was of course the install part of setup, because coding uninstallers is in some sense an admission that the software you’ve developed might not be useful or robust enough to become a permanent fixture on end-user systems. As a result, software uninstalls were often incomplete, leaving behind Registry and file system detritus. A few hundred kilobytes of unused keys and values causes no noticeable performance impact on system operation, but I figured it was natural for a Registry cleaner to be an essential part of running a tight ship for the anal retentive systems administrator.

Installer technology has come a long way and today there are literally dozens of reliable freeware and low-cost installation toolkits available both for old-style and Windows Installer Package (.MSI)-based setup. I previously believed that meant the end of Registry scrap and any reason for the existence of Registry cleaners. However, one of the Regmon troubleshooting examples Dave and I present in our Windows internals seminars made me realize that it’s not only possible, but common, for even best-of-breed uninstallers that have earned the Windows logo from Microsoft to leave our Registries littered with traces of applications deleted long ago.

The troubleshooting example that triggered my realization is one where a user discovers that Internet Explorer (IE) hangs upon startup unless they’ve used their ISP’s dialer to establish an Internet connection before opening IE. The user captured a Regmon trace of the hang and found references to their previous ISP’s dialer in the “RAS Phonebook” key under HKEY_CURRENT_USER (HKCU, the personal settings area of the Registry). After deleting the stale values their new ISP’s dialer began launching automatically when IE opened like it was supposed to.

Some research on my part suggests that the user had manually configured a RAS phonebook entry and when they changed ISPs forgotten to update the entry, but the fact that the problematic key in the example was in the per-user part of the Registry got me thinking. If the user’s old dialer had created the phonebook entry automatically and then been uninstalled from a different account the user would have been left with a broken IE configuration. That’s because uninstallers typically delete their application’s system-wide settings from the HKEY_LOCAL_MACHINE part of the Registry and any per-user settings of the user running the uninstaller from HKEY_CURRENT_USER. But what happens to the per-user settings of the other users that used the application? You guessed it, Registry junk gets created - and possibly file system junk in the application's Application Data folder in the \Documents and Settings directories of other users. An uninstall is only thorough if the user performing it is the only one that used the software.

You can verify my assertion by first installing and using any application that creates personal settings in the Registry (try Winzip if you want one to experiment with). Then log into a different account and use the software. Uninstall from either account then log into the other and you’ll find the application’s per-user settings left behind (Winzip stores its settings in HKCU\Software\Nico Mak Computing\WinZip).

Uninstallers that want to be as meticulous as possible could use the LoadUserProfile Windows API to load each profile stored on the system, as listed in HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList, and look for their application settings in other profiles. The problem with that approach is that Registry permissions would likely prevent an uninstalling user from deleting keys out of other user profiles. And even if permissions weren’t an issue domain users with roaming profiles carry application settings with them to other computers, so profiles that need to be updated might not even be locally accessible.

So it seems that Registry junk is a Windows fact of life and that Registry cleaners will continue to have a place in the anal-sysadmin’s tool chest, at least until we’re all running .NET applications that store their per-user settings in XML files – and then of course we’ll need XML cleaners.