Mark's Sysinternals Blog

I’ve been a DVR user since around the year 2000 when I purchased a ReplayTV (then sold by SonicBlue). A couple of years later its disk was too small for the programs all I was recording and it lacked some of the features available newer DVRs, like broadband, so I compared the TiVo and ReplayTV's current versio(which was being sold by Panasonic after it acquired SonicBlue). I decided to go with ReplayTV again, because it matched most of TiVo’s features, but primarily because I was familiar with it and its ability to skip forward 7 or 30 seconds makes it extremely easy to avoid commercials.

Unfortunately, the unit I purchased wouldn’t contact the ReplayTV service. In a manner that reminded all too much of many experiences with Windows software it reported an error that was something like “Error: 80004030. Contact Technical Support” (which was strange because that looks like a Windows COM error message, but ReplayTV ran on Linux). After an hour on the phone with a less-than-helpful customer service representative, who was probably talking to me from India, I gave up and returned the unit.

Just before I had purchased the ReplayTV I’d received a mailer from my cable company, Time Warner, announcing that they were offering a DVR service for a mere $20 per month (in addition to the digital cable fee). I decided to sign up because I figured that since I didn’t own the DVR, I could avoid the expense of upgrading every couple of years like I would end up doing if I went with a TiVo. Plus, the DVR doubles as the cable set top box, has two receivers allowing you to tape two shows at the same time, and I believed it would have an authoritative schedule downloaded directly from the cable company.

Two years later I can report that the DVR’s number of deficiencies borders on the absurd. Here are some highlights:

• The schedule search requires you to know the day of the show you’re interested in
• The schedule the DVR downloads often does not reflect the schedule that’s broadcast
• The DVR sometimes just “forgets” a recording schedule you’ve configured for a particular show
• If you’re watching in delay a show as its being recorded the DVR resets you to the start of the show when the show ends
• The DVR doesn’t consistently remember where you were watching when you pause a recorded show you and later resume
• There are only three forwarding speeds and no way to skip forward, making it painfully slow to get to a particular point in a show
• When you’re watching a show the DVR announces, for no good reason, when it’s about to start recording a show on another station
• There’s no audible feedback on menu navigation or selection and the lag when navigating the menu can result in you entering input that, when processed, causes you to miss what you’re after

I could go on. I discovered most of these problems within a few days of usage, but gave Time Warner the benefit of the doubt, expecting rough edges on new software and looking forward to automatic updates. The updates have never come and I long ago became convinced that they never will.

Time Warner is delivering consumers the bare minimum solution, which is probably the cheapest one they could find. They’re able to get away with it because most cable subscribers have never used another DVR and are therefore ignorant of what DVR software should provide them.

This is a great example of what happens to consumers when there’s a lack of real competition in a marketplace. Go TiVo!

One thing I do regularly is run Filemon and Regmon to identify poorly written software. In the context of these tools a poorly written application is one that polls (repeatedly queries) the file system or Registry for changes. Polling is undesirable because it degrades performance. A thread that polls sleeps for the polling intervals, wakes up to perform its queries, and then goes back to sleep.

When the thread wakes up and the kernel schedules it the system must context-switch away from potentially useful work to the polling thread. The context switch results in an invalidation of the translation lookaside buffer (TLB – a high-speed cache of virtual to physical memory translations) of the processor and consumption of CPU resources as the thread executes. In addition, the thread references code and data as it executes, which the memory manager must make resident in physical memory, thus consuming memory resources.

If polling is so bad, what’s the alternative? If an application wants to respond in a timely manner to Registry changes it can use the RegNotifyChangeKeyValue API, and for file system changes it can use ReadDirectoryChangesW or FindFirstChangeNotification.

All that said, there are still lots of applications out there, including Microsoft applications, that poll. I suggest you run Filemon and Regmon on your own systems to see what kinds of badly-behaved software you have. Some of the most egregious offenders are the Windows services associated with major antivirus products.

When I ran Regmon yesterday the trace showed that MSN Desktop Search polls. This Regmon output shows msnlAdmin.exe, the MSN Desktop Search administrator process, repeatedly querying the current computer name value (about 4 times a second):



When I saw this I suspected that msnlAdmin.exe wasn’t polling the value directly, but instead repeatedly executing an API that reads the value. I attached to the process using Windbg (from Microsoft’s Debugging Tools for Windows), set a breakpoint on NtQueryValueKey (the native API invoked by Windows API Registry queries), and when it hit looked at the stack:



So msnlAdmin.exe is actually querying the name of the computer, which underneath results in a Registry API. The developer probably assumed that the value was cached, but since its not they should cache it. Why msnlAdmin.exe is constantly running I can’t say, however.

Yesterday I discovered a bug in MSN Desktop Search (MDS) and its Outlook integration. I would enter a search term in the MDS toolbar in Outlook and the resulting Explorer window (someone pointed out that MDS uses an Explorer shell extension to display results instead of IE) would reflect a search for a term that I had entered earlier in the day. The behavior was totally reproducible - regardless of what I entered in Outlook the search was for the previous term. A search from the taskbar search window worked as expected.

Figuring that a restart of Outlook might clear up the problem I exited. When I relaunched Outlook I was confronted with a "The Operation has Failed" dialog that I've seen so many times before when exiting and restarting Outlook. You get this dialog when you start a second instance of Outlook before the previous one has completely exited. Its just one of the many examples of lame Outlook error reporting.

You can cause Outlook to generate this same lame error message a different way:

1. Use winzip to create a zip file, add a large directory to the zip file, and press Esc during the add operation to get the 'cancel?' dialog
2. Create a new outlook message and attach the zip file. Here's the error message:



Here’s what Filemon shows:



The problem is a sharing violation on the zip file that occurs because Winzip has opened it without allowing shared access. It’s amazing to me that in the year 2003 a program like Outlook, one of the most widely used pieces of add-on Windows software on the planet, behaves like this. I consider meaningless error messages a bug, since they do absolutely nothing to help you resolve whatever problem was encountered.

Yesterday we released RootkitRevealer v1.30. This release is in direct response to Microsoft Product Support Services (PSS) discovering actual installations of the Hacker Defender rootkit on customer systems that target RootkitRevealer.

RootkitRevealer works by comparing a high-level scan of the system via the Windows API with a low-level direct scan of file system and Registry on-disk structures. Rootkits that cloak by modifying a system view at any level above the on-disk structures will be visible as discrepancies between the two scans - that is, if their cloaking is active.

Hacker Defender's installation includes a configuration file where a malware author specifies the files, drivers, services, and other items that should be cloaked. The configuration file also includes a section where ‘root’ processes are specified. A Hacker Defender root process is one that Hacker Defender allows to see an unmodified system view. Microsoft PSS found that RootkitRevealer wasn’t detecting discrepancies on some customer that they verified had Hacker Defender installations. Their investigation revealed that the RootkitRevealer had been added to the ‘root’ process section of the configuration file. Thus, RootkitRevealer’s two scans showed no differences.

To defeat this Microsoft started renaming RootkitRevealer’s executable before a scan. Bryce and I decided that many users would likely not know to do this and requiring a manual rename is inconvenient, so we modified RootkitRevealer to perform the rename automatically. When you execute RootkitRevealer it makes a copy of itself in \Windows\System32 with a randomly-generated name. It then installs that copy as a Windows service that displays its UI on the console desktop and cleans up the service installation when the scan is complete. Since this approach doesn’t work well with a command-line executable we added command-line options for automatic scanning and logging to a file.

Is this the last modification we’ll have to make? Not likely. This was an easy attack since it required no modification of Hacker Defender, but more sophisticated attacks are possible where a rootkit can detect a scan of RootkitRevealer in other ways, like version information or behavior analysis, and disable cloaking so as not to be detected. That’s why a better approach to detection is to combine the detector with a virus scanner: if the rootkit cloaks it will be detected by the rootkit detector and if it doesn’t virus scanning can pick it up. The bottom line is that there can never be a universal rootkit detector – only ones that work against certain types of rootkit technology.

People have asked us to add a rootkit removal feature like the “rename” functionality of F-Secure’s Blacklight product. We’ve decided not to for several reasons. First, renaming of rootkit files is easily defeated by a rootkit that activates before the rename operation and blocks the rename. Second, if the rootkit detector hasn’t detected all of the components of the malware the rename can result in only a partial removal of the rootkit. We’ve already seen spyware and adware that automatically repairs its own broken installations and its so its also possible for rootkits to reinstall disabled components.

Rootkits are a very scary thing. The focus of the security community and IT professional should be on preventing their installation. Detection is a last resort with uncertain results.

I've been using MSN Desktop Search (MDS) for several days now and overall am happy with it. Since I'm using Outlook and Office I don't have a problem with its lack of support for some non-MS technologies. Its search is fast, though the results aren't presented as cleanly as Google's.

One thing that has annoyed me, though, is the fact that when I initiate a search from the MDS taskbar window it launches a new IE window to display search results, instead of displaying in an existing IE instance. I run Avant Browser and it would be more convenient if it opened a new tab in Avant Browser instead, especially since IE doesn't seem to remember its screen position or size from execution to execution.

I've also been told to look at Copernic. I haven't tried it myself, but a colleague has and reports that, while its much more configurable than MDS, its search is slow.

Since I'm blogging now I thought I'd share with you some of the other blogs to which I subscribe:

http://www.schneier.com/blog/
http://blogs.msdn.com/robert_hensing
http://blogs.msdn.com/oldnewthing/
http://blogs.msdn.com/larryosterman
http://blog.ziffdavis.com/seltzer/
http://silverstr.ufies.org/blog/
http://blogs.msdn.com/aaron_margosis/

I haven't blogged the last couple of days because I both wanted my desktop search testing to progress before reporting back, and because I've just finished an article on rootkits for Windows IT Pro Magazine's June issue. If you’re not familiar with rootkits check out RootkitRevealer, a tool that Bryce and I recently released that tries to detect them. Its home page gives a brief description of rootkits.

The June issue of Windows IT Pro Magazine is going to result in a nice synergy for me because it will be released at TechEd US in Orlando, where I'll be delivering a breakout session that discusses rootkits, "Understanding and Fighting Malware: Viruses, Spyware, and Rootkits". If you're going to TechEd you might want to check out Dave Solomon (my coauthor on Windows Internals and Inside Windows 2000) and me copresenting a preconference tutorial on advanced Windows troubleshooting. The precon is a lot of fun with a lot of practical information and we highlight some of the advanced uses of a number of Sysinternals tools, including Process Explorer, Filemon and Regmon.

Picking up on my desktop search testing, shortly after I learned that Google Desktop Search (GDS) only does partial document indexing I uninstalled it and decided to give Microsoft a chance by installing MSN Desktop Search, which is also in beta. My experiences with it have been a little better, though Google did a much better job of telling me where it was during the initial indexing process. Even finding MSN Desktop Search's status window is difficult. I expected to find it on the context menu associated with the deskbar search window that you can have it display in the task bar, but it’s only accessible through the context menu of its tray icon. Well, I was going to tell you the current status of the indexing on my laptop, where I installed MSN Search this morning, but it looks like they've got a bug in their context menu: it appears, but is only partially drawn and I can't interact with it. Anyway, on my desktop system it said for hours that it had indexed 16575 items with 0 left to index - when a search for text I knew to be in e-mails failed, indicating that it hadn't indexed Outlook, yet.

Desktop searches with MSN Search are working great, though, and its finding the text within the power point documents that GDS couldn't. MSN Search's documentation states that it will index only the first 1 MB of documents, which is annoying, but most (not all) of mine are smaller than that. There's no reason for Microsoft and Google to exclude advanced options like that. Speaking of advanced options, MSN Search also has an easily accessible index for selecting what you want to index (someone posted a comment to my previous blog posting explaining that GDS has a way - that in my short usage I didn't come across - to specify what you want to exclude, which would require more work for me).

Another reader pointed out that this month's issue of PC Magazine has a comparative review of desktop search engines that you can read on-line at http://www.pcmag.com/article2/0,1759,1771684,00.asp. They choose Yahoo Search as their winner, but then again, they don't mention either the GDS or MSN Search document size limitations, so I'll see if MSN Search works for me before trying another engine that might have other gotchas.

As a side note, as I was trying to open MSN Search's tray icon context menu I was reminded of just how annoying it is when the “hide inactive icons” task bar configuration option is selected. I’ve already switched the option off on my desktop system and now have done so on my laptop. When I’ve tried to interact with a hidden icon by clicking the open button on the tray I’ve usually had to race to click on the icon before the tray closes up on me. Hiding the inactive icons is nice visually, but the implementation is flawed.

Someone wrote in to tell me that Google has a FAQ that addresses my incomplete search issue:

http://desktop.google.com/support/bin/answer.py?answer=12764&query=5000&topic=0&type=f

What good is a partial search of my documents, especially when I have no control over the portion that's searched? Does Google's web search engine only index parts of web pages?

Its a little amazing to me that, besides not clearly telling you what it indexes, Google Desktop doesn't have options that allow you to specify the following options for the indexer:

• File types
• Directories
• Date ranges
• Volumes

I don't ever see myself interested in a partial documents indexing.

People are complaining that the index storage is taking up a signficant percentage of their disk space (its 3 GB on my system). These options would help the size problem as well as give people confidence in the search results.

Time to uninstall and evaluate other desktop search solutions...

Welcome to my first blog posting!

I installed Google desktop a few days ago and have had a pretty lackluster experience with it so far. After it finished indexing my system (which has over 250,000 files), I decided to try sample searches. I was encouraged when I entered terms that I knew to be in e-mails and power point documents and results pages appeared virtually instantly.

However, my initial optimism turned to distrust of the output when when I entered a term that I knew was in several presentations, but the search came up empty. A little experimenting with a particular power point revealed that the search could find any term or phrase in the first 10 pages of the 50-page document, but nothing located deeper. I then tried a few e-mail searches, looking for phrases that were in older e-mails. Again, the searches turned up nothing.

I haven't spent much time looking into the problem so I don't know if Google Desktop has a size limit, perhaps based on disk free space, on its index files, if there's an age limit on documents it indexes, or if it will only go 10 pages into Office documents, but a quick read through the help file didn't turn up anything. In any case, I'm waiting a few weeks to investigate and to see if they release an update before I give up and uninstall. In the meantime I'm certainly not relying on it for searches.

Another thing I noticed when I opened Process Explorer to see what processes Google Desktop creates is that none of the Google Desktop executable images have version information (like company name or description), let alone digital signatures authenticating them as coming from Google. In light of Google's recent run in with malware I would expect them to be more security conscious. I've reported both these issues to Google and they've responded with a note saying they'll look into them.