Just wondering if I could pick your collective brains, I have some questions about how you schedule your Windows patches and handle offline computers.
I'm looking to revamp our patch management system and wondering how other MSPs handle patches in general - how strict you are about patching, time table for patching, deadlines... etch. I'm also super curious about how you handle offline computers. Currently we are providing service for 42 laptops who are hit or miss if they're left on.
Any suggestions or insights would be awesome, thanks again!
For offsite laptops, we do tick wake on LAN and don't tick the 'skip if offline' box in the schedule. If they're not able to be woken up, then they're patched the next time they are turned on.
No point trying to patch a powered off machine. And if they're not internet-connected, then no remote tool in the world can patch them .....
We patch workstations daily( i.e. ASAP) and servers monthly, the weekend after patch tuesday. I spend a few minutes each month fixing Kaseya's broken patch auto-approval process for new & updated patches, otherwise it just runs itself.
I may occasionally pull a patch if I hear something really bad about it or its proven to screw something up. But the need is rare.
We see little benefit in delaying patching ... its easier just to fix a busted patch when/if it becomes an issue, rather than wasting time testing checking and manually approving patches all the damn time. cost/benefit analysis and all that. This is what system restore is for.
We've got a very similar approach as Craig does. Very rarely do patches get denied, well apart from SharePoint patches which ALWAYS break the intended servers.
For laptops that aren't on during the patch schedule make sure they're not set to skip if offline and in my experience awkward to reach machines often also get WUA enabled for a belt and braces approach.
Don't give clients the opportunity to specify when and how they want patching done as this ends up defeating the purpose of having a system like Kaseya where you can manage a large number of machines in more or less exactly the same way. I've seen this in the past and it creates so much extra work that's really unnecessary.
Thanks for the replies for the guys! Good stuff!
Every month though it seems we're spending hours narrowing down offline machines, do you guys bill out for that kind of work? If a machine is offline during your time table to install patches how much time do you set aside to connect individually with machines to patch them?
Nill. As stated, the setting 'dont skip if offline' means they are patched immediately, the next time they DO come online.
Who ares if an offline machine isn't patched? It's OFFLINE - so out of harm's way.
What TIME do you all run patches? We scan on Wednesday morning and patch on Wednesday afternoon, spread over a three hour window each. I'd prefer to patch at night, but so many machines are off that I worry about bogging their computers down in the morning when they fire them up. We also can't trust WOL in every office.
I have users complain that their computers run slow during the scan period. I trust them on this because they complain about it without knowing the scan schedule. Logs show that the scan only takes a minute or so.
1. Fix WOL so it works - Not that hard....but I agree some generic desktops just won't WOL reliably thanks to faulty NICs or whatever.This should be a very low % of machines; a new NIC is <$10. Consider if it's worth your while to do upgrades.
2. Train users. You have to patch *some* time, so either leave the machine on for the patch-window, fix WOL, or get them to suck-it-up - your reply being you didn't leave your machine on last night, as instructed.
3. The patch *scan* shouldn't be intensive (same load a running windows update manually). the patch *install* can be (especially for .NET etc. with mscorsvc etc. tasks to run). If their machines are that touchy, the machine probably needs an upgrade, or at least a defrag, temp files cleanout, browser cache flush etc....check you're maintaining the machine properly.
How do you handle machines that cannot have dot net 4.x or IE10/11 installed? I've created two patch policies that I use for these kinda machines, but keeping them up to date is a chore. All in all we have 5 patch policies in use:
1. "All important patches approved" that approves almost every patch automatically. This is used only in our own machines, not for customer use.
2. "High Priority Patches" that is a copy of the above. I copy polices from the above one when I'm sure nothing broke in our machines. It has some Default approval status for certain products set to approve, like the security updates. This policy is used by default for all customer machines via policies.
3. "Deny dot net 4.x" that has all dot net 4.x updates set to denied. Our customers have software that does work when dot net 4.x is installed. For this one I approve new patches mostly manually.
4. "Deny IE10/11 updates" that has all IE10 & 11 updates set to denied. Same reason as above.
5. "Cash registers" that are Win XP & 7 machines. Only Automatic update approval for XP & 7, but still I'm seeing that other product's patches are being approved for some weird reason.
We are using Kaseya 6.3 at the moment, but moving to 7.0 soon.
Managing patch policies should not be a burdensome chore. There is some setup work involved, as well as some regular 'maintenance', but the regular maintenance really shouldn't be a huge amount of work. That isn't to say it is a lot of fun - I find laundry and dishes to be a chore, but if i keep up on them, neither takes much time. If I let either (or both) go too long, it's miserable. I'm sure you can relate.
Here's what I recommend to keep patch policies manageable, drawing from your scenario above.
1. Create a policy for workstation and configure the default approval status such that ALL workstations in ALL cases (for ALL clients/machine groups across the board) could use this policy. This is probably going to be a fairly liberal policy that allows patches that you may often, but not always, want blocked. If possible, choose Approve or Deny - avoid using Pending Approval. This will make management of this policy close to hands-off going forward since every patch would be EITHER approved OR denied automatically.
1a. You might have a secondary workstation policy for your customers (what you're calling "High Priority Patches" in your scenario) in addition to the base workstation policy so you can "test" patches in your enterprise environment before deploying to your customers. If you do this, try to follow the same logic as above - the customer-facing policy should be usable for ALL customers and try to avoid using "Pending" as a default status, just to minimize maintenance on this policy each month.
2. Create a policy for servers just as you did for workstations
3. Manually deny any patches in the two above policies that should be denied. Approve everything else.
4. Create a patch policy called "deny .net" (or whatever variation you would like). On the Approval by Policy page, select the deny .net policy. Microsoft does not have a specific product or classification only for .net patches. Therefore, determine which products or classifications might include a .net patch (consult MS documentation for this). Set all of these classifications with a default approval status of "Pending Approval". Set ALL others to "Approved".
5. In the Deny .net policy, you will need to now process all of the patches already known to your VSA (since these will have the approval status that was in place at the time the patch was first discovered - probably "Pending Approval" if you just created this policy. Within this policy, click a hyperlink that will show you all Pending patches (the TOTAL number link for Pending Approval). Use the patch filter to find all patches where the Title includes "*.net*" (no quotes, but do include the * as wildcards). Deny these patches. Clear the filter. This should cause the remaining patches to appear. Select all and approve these patches. Returning to the matrix, you should have NO patches left in the Pending Approval column. All patches should be approved except the .net patches, which should be marked as Denied.
6. Create a patch policy to Deny IE. On the Approval by Policy page, select the Deny IE policy. Set the default approval status on the Classification view to Approved for ALL classification types. Change the view to Product (instead of Classification). IE does not have its own product. Microsoft includes IE in the OS level products. Therefore, set all OSs to Pending Approval.
7. Select a hyperlink to view all Pending patches. Using the patch filter, filter to all patches that include "*explore*" in the title. Deny all of these patches. Approve all remaining patches.
8. Create a patch policy for your cash registers. On the approval by policy page, set ALL classifications to APPROVED. Switch to the Product view. Set all products to Denied EXCEPT XP and Win7 patches. Manually approve all patches in the XP and Win7 products. You should be left with all patches denied EXCEPT XP and Win7. Going forward, this patch policy should need very little to no work. Any XP or Win7 patch in ANY classification will be approved, but any patch that is not 7/XP will be denied.
9. Each month or so, go into the "Deny" policies. Filter to the Pending patches. Deny the .net or IE patches (respective to their policies). Approve ALL others. Do NOT use the IE or .net specific policies to block ANYTHING other than what the policy is intended for.
10. Assign the policies to the machines based on their needs. Every workstation should get the "workstation" policy created in step 1. Every server should get the "server" policy. Then apply the others to those that need to block those specific types of patches. The policies will work together such that only those patches explicitly approved in ALL policies assigned to the machine will be considered approved for the endpoint.
Using this method, you do have to do some setup work (some of which you might have already done). But on a weekly/monthly basis, you only need to go into a couple of policies, list the Pending patches, set your patch filter, deny the "bad" patches, then approve all others. This should take less than 2 minutes per policy. Still a chore (with a little "c") that has to be done, but not a Chore (with a capital "C") like having three weeks' worth of laundry and dishes piled up.
I hope this is helpful.
Maybe I was a bit harsh by using the word chore :) I usually try to check all the polices at least once week and check for pending approvals. Does R7 bring any new features to patch management?