Maybe it's a bug in PrinterSettings.get_InstalledPrinters, or maybe it's a bug in the underlying Win32 EnumPrinters function. One could probably argue that it's both -- we'll get to that later. To start with, the bug is this: when the 'spooler' service is stopped, some users were occasionally experiencing an OutOfMemoryException, thrown from the Marshal.AllocCoTaskMem call inside get_InstalledPrinters. Here's a snip of the stack trace:
System.OutOfMemoryException: Insufficient memory to continue the execution of the program. at System.Runtime.InteropServices.Marshal.AllocCoTaskMem(Int32 cb) at System.Drawing.Printing.PrinterSettings.get_InstalledPrinters() at ...my code, logic attempting to count the number of installed printers, to enable/disable the printing menu items...
When I first received this stack trace from the field (our product is, thankfully, still in beta) I didn't know what to think. The product in question *does* need to allocate large amounts of memory (upwards of 10MB) during the ordinary course of its usage -- I felt chances were good that this was a real out-of-memory condition. (Who knows what other memory-hungry apps the user had running?)
Consider that OutOfMemoryExceptions, when they happen, can happen almost anywhere -- even the smallest of allocations might be the straw that breaks a heap's back -- so I didn't pay much attention to the exact context of the stack frame... however, something didn't quite seem right. While it's true that a small straw can break a camel's back, heavier straws carry a much greater chance of injury! The get_InstalledPrinters method seemed, to me, like an extraordinarily unlikely little straw -- especially when the second report of this bug came in.
And especially after both beta testers reported that they had no printers installed. In fact, one claimed not to have the spooler service running at all -- that would sound like a smoking gun, if not for the fact that that's how I run my own dev machine at home. I squashed this bug, long long ago -- under ordinary circumstances, get_InstalledPrinters will throw a Win32Exception complaining that its RPC server is unavailable -- I had a catch block in place, to deal with that. So, what could the problem be? Why out-of-memory?
Finally, late last night, I hit the elusive OutOfMemoryException myself! Better yet: I was able to reproduce it in a debugger -- rigged to break on the "first chance" at the moment any OutOfMemoryException was thrown. When I peered inside the watch window, the value of 'cb' passed to the AllocCoTaskMem call was an astonishing 2,125,875,824 (0x7EB64A70). Out of memory, indeed!
At this point, I began suspecting a bug in the FCL -- there's no chance that correct code should be trying to allocate almost 2GB of memory, in order to enumerate printers -- that value is almost certainly pure garbage, considering the scarcity with which this bug was reproduced.
Here's a snip of the relevant get_InstalledPrinters code, courtesy of Reflector. Can you spot the bug?
int num2; int num3; SafeNativeMethods.EnumPrinters(6, null, num4, IntPtr.Zero, 0, out num2, out num3); IntPtr ptr1 = Marshal.AllocCoTaskMem(num2); int num1 = SafeNativeMethods.EnumPrinters(6, null, num4, ptr1, num2, out num2, out num3); textArray1 = new string[num3]; if (num1 == 0) { Marshal.FreeCoTaskMem(ptr1); throw new Win32Exception(); }
The first call to EnumPrinters is made with a null buffer -- the required size is passed out via the num2 argument. This is a standard pattern in Win32 programming. (Note the first call doesn't bother to check the return value -- without any buffer for the results, it's expected to fail.)
Now, clearly it's this 'num2' value that's holding garbage at the time of the call to AllocCoTaskMem. How did that happen? For starters, apparently the EnumPrinters function isn't zero-initializing its [out] parameters before all other possible return-paths... But that shouldn't matter, right? Isn't the value of 'num2' pre-initialized to zero, ahead of time, by the CLR?
Hint: you'll have to close Reflector and open up ILDasm in order to spot the more proximate cause of this bug... any takers?
