Kaseya Community

Microsoft is pulling the rug out from under us on patching... what are you doing about it?

  • I'm sharing the bullet points from my internal e-mail to Engineering & Customer Success on patching of Windows 7, 8.1, Server 2008 R2, Server 2012 & 2012 R2 and Windows 10.  I'd love to know what other Kaseya users are doing and how they're messaging this to their Customers.

    • Microsoft is severely limiting our ability to control what patches are installed and when
    • Our ability to control patching varies based on the operating system
    • The limitations in our ability to control patching in the granular fashion we’ve grown used to is not a function of Kaseya—all RMM platforms are currently suffering from the same issues by virtue of the fact that they’re all dependent on the Windows Update mechanisms
    • For Windows 7, 8.1 & Server 2008 R2, 2012 and 2012 R2, starting in October 2016:
      • Microsoft is no longer pushing individual patches—they’re instead pushing Monthly Rollups
      • These Monthly Rollup patches include all patches for that month, regardless of criticality–everything including:
        • Security Updates
        • Reliability Updates
        • Recommended Updates
        • Optional Updates
    • They do offer a “Security-Only Update” model; however, no RMM platform (including Kaseya) will be able to see these updates as they are not listed in the Windows Update catalog
    • Patches will be distributed weekly (for Customers on our standard patching schedules); however, effectively only one patch per month will get installed (the patching window will be skipped if the Monthly Rollup isn’t queued up & approved)
    • TL;DR: Patching for Windows 7, 8.1 & Server 2008 R2, 2012 and 2012 R2 is “all or nothing” starting in October 2016, and if an individual component of a Monthly Rollup causes an issue, the entire Monthly Rollup must be removed.  We’ll continue to schedule patches weekly, but effectively only one patch per month will go out for these operating systems.
    • For Windows 10:
      • Windows 10 is an Operating System as a Service—Microsoft believes it knows best as far as which updates should be pushed and when
      • Our options were to either:
        • A) Let Microsoft push patches on a continual basis -or-
        • B) Wait four months and leave users unpatched until four months after the patch is released -or-
        • C) Deploy only security updates (no usability updates, OS fixes or anything else)
    • We chose option A
    • We’ll still try to “nudge” Windows 10 via Kaseya to kick off patching on [time redacted]; however, this doesn’t really seem to have any impact—Windows installs patches whenever it pleases
    • Users should be prompted to reboot by Windows -or- to “Update and Shut Down” when they leave for the day—reboots are not forced—the user will always have an option to reboot now or defer
    • I’m continuing to monitor developments on all RMM platforms (Kaseya, Autotask Endpoint Management, N-Able, etc.) to see how they’re going to address this—I know that Kaseya has an entirely new patching framework in beta starting next month—in any case, updates will follow if this situation changes

    Care to weigh in?

  • I have thought of this future limitation concerning and affecting the patch management on our systems.  My ultimate concern is if/when microsoft releases these roll-ups, will Microsoft give us the ability through cmd or powershell to rollback problematic updates and if so in what scope?  

    While I consider us at the mercy of Microsoft in this regard; you did pose a potential business challenge and opportunity for all RMM vendors.

    Can and will Kaseya be the first to leverage this new approach in their favor?  Personally, I hope so.

  • Kaseya can't and wont be the first to leverage anything with a third party, have you ever seen how they handle issues with AV modules? it takes them months to fix issues with it, because they don't really have a flexible or open relationship with third party developers. And when these third party developers make a change the farthest thing from their mind is "Oh will this screw up our friends at Kaseya" They don't think that way because Kaseya doesn't pay them enough to make the slightest considerations on there behalf

    The long and the short of it if is an issue within core will be resolved quickly anything else that relies on another company gets the long drawn out wait .

  • I can understand Microsoft's desire to remove control of system patching from the public but it's completely unacceptable, in my opinion, to remove it from the Enterprise. I would also be very surprised and amazed to hear that Kaseya had wrested control of patch management back from Microsoft and I don't expect that to happen in my lifetime.

    I've been a Microsoft supporter for most all of the last 20 years but, the more I see of what Microsoft has done with Windows 10 and how frequently they are thumbing their nose at Enterprise management in so many ways, the farther I get from wanting to stay with them. Were retirement not looming I would be taking a very hard and serious look at moving the computer systems, of the company that employs me to manage their systems, away from Microsoft and off to some other solution...

  • oromero
    Can and will Kaseya be the first to leverage this new approach in their favor?  Personally, I hope so.

    I hope so too.  I'm looking forward to seeing how the new patching & third-party software update module behaves in the 9.4 beta (though I obviously won't be able to share my impressions beyond the beta group).  The granular & process-oriented manner in which we handle patch distribution has always been one of our key differentiators, and I hope that whichever third-party solution Kaseya integrates into the VSA to replace the largely reliable but dated Patch Management module can address these challenges--particularly rollback if we identify a "bum rollup" that borks a bunch of machines.

    The long and the short of it if is an issue within core will be resolved quickly anything else that relies on another company gets the long drawn out wait.

    I'd argue that, regardless of the third-party integrator, patching functionality is still part of the core functionality of the product.  Kaseya Support had better be ready to not "punt the football" to the integrator when they hit the ground running with 9.4... 

    the more I see of what Microsoft has done with Windows 10 and how frequently they are thumbing their nose at Enterprise management in so many ways, the farther I get from wanting to stay with them

    Way of the future, my friend  Wink Office 365 / Azure are gaining significant traction, Windows is becoming an operating system as-a-service, on-prem servers are an endangered species, on-prem phone systems are being replaced with cloud-hosted PBX at nearly every refresh, and Customers are insisting that all of their capital expenditures transition to operational expenditures as soon as possible.  Anyway, a bit off-topic... I'm not saying traditional enterprise management is dead, but it's definitely undergoing a full & rigorous health assessment with vendors lined up to replace the worn-out legacy bits with bionic appendages.

    / I haven't yet had enough coffee to make good analogies Stick out tongue

  • This is certainly a 'hot topic' among MSPs and Enterprise IT departments at the moment, and worthy of extended discussion.

    I'm in favour of moving to a cumulative update model for several reasons.

    1. Patch fatigue set in for me years ago - having to try on a monthly basis to sufficiently review, test and approve dozens of patches each month for a half-dozen supported Windows operating systems across nearly 3000 endpoints (plus the same again for Microsoft Office) is time consuming, error prone, frustrating and thankless. Having to then follow up on why some seemingly random selection of patches failed to deploy on potentially hundreds of machines is just brutal. I want Microsoft to take back a large part of that responsibility - it won't be perfect but it will be better than what we've got now.

    2. I like the idea that ISVs are forced into (shock horror) *maintaining their software* and can no longer hide behind ridiculous statements like 'oh we can't work with XYZ patch, don't deploy that'. Their malfeasance has gone on too long and I'm sick of having to come up with workarounds and put special exceptions in place to make up for their shortcomings.

    3. Getting and keeping machines up to date will be significantly simplified. If you're on-boarding a customer who has previously had poor patch management, or if something goes wrong and machines miss a patch cycle or two you're still only one patch away from fully up to date.

    Now - what's Kaseya's patch management role right now and going forward?

    1. Scheduling. This is key - customers need to know when their machines will be patched and rebooted. Right now I think Kaseya's control over patch scheduling for 7/8.1 is very good but things seem to come a bit unstuck for Windows 10. There are good examples out there of products that can reliably scan and trigger patch installation on Windows 10 (see for example www.powershellgallery.com/.../PSWindowsUpdate) so there should be no reason Kaseya can't achieve this. Tight scheduling and control of update-related reboots with Windows 10 looks to be somewhat of a bugbear but there are 'active hours' settings that Kaseya could centrally control, and ultimately we need to make sure we have clearly communicated with the customer what to expect, even if the message is that we have less control than previous.

    2. Approval. Right now Kaseya offers very granular approval pushing the work back on us to 'pick and choose' the patches we need, but with the increasing move of products to a cumulative update / click-to-run /  'as a service' model this becomes less and less relevant. What I'd like to see is Kaseya taking advantage of the well-documented WUB registry settings for Windows 10 to allow us to centrally control the upgrade and update release 'rings' - setting customer machines to CBB with a 6-month upgrade delay and 2 week update delay, for example. With regards to the 'security only' cumulative update if you're leaning that way - although it won't be offered when scanning against 'Windows Update' it's a safe bet there will be a new release each month for each supported OS and it *will be* downloadable from the Microsoft Update Catalog - it's not a stretch to expect that Kaseya should handle scheduled downloading and installing of this update if we nominate this as a preferred approach. Presumably you could even switch streams when you felt the need - sticking with security updates only until you saw a compelling fix released in the latest CU, you test and deploy this, then move back to the security updates only stream until some later CU takes your fancy.

    3. Distribution. I wish I could say I liked the whole 'LAN cache' implementation for patch deployment but I don't - it's finicky and annoying and I've run into lots of problems with it over the years. Additionally, with the move to cumulative updates the LAN cache will get increasingly full of huge outdated patches that we'll probably need to clear out manually. Windows 10 offers WUDO (Windows Update Delivery Optimization) and I would like to see Kaseya offer us central control of the registry settings for delivery optimization rather than try to use the oldey-styley LAN cache approach. Microsoft - if you're listening - if you're going to force a CU model for 7/8.1 you really should be back-porting WUDO to 7/8.1 too!

    There's plenty more to delve into but that will probably do for now :-) I welcome everyone's thoughts and any constructive feedback.

    David Svirskis

  • I truly enjoy these high level conversations in the community!  

    I agree with in regards to the opportunity Kaseya has here.  They can set themselves apart from the 'rest' as technology trends are shifting it is important to have products agile enough to maintain their bloodlines through most changes.

  • I'll share my notes as well.  We haven't started the communication process with our clients just yet.

    1. Patch types NOT subject to getting moved to the MAIN rollup
      1. Servicing stack  (windows update agent, etc)
      2. Flash
      3. .NET
        1. BUT .NET gets its OWN rollup
      4. Office
      5. GWX (The old windows 10 upgrade notifications)
      6. Windows Defender definition updates
    2. Prereq’s
      1. Win 7 SP1, Server 2008 R2 SP1, Win 8.1, Server 2012, Server 2012 R2
      2. April 2015 servicing stack update

    3. Important quotes from Nathan Mercer
      1. “Office has its own patches which are separate from Windows and not part of these servicing changes moving to rollups”
      2. “announcements only relate to Windows, not other Microsoft products”
      3. “Security-only will use the security category, Monthly Rollup will use the rollup category.”
      4. “available to everyone and every Windows SKU not just business versions”
      5. “IE version upgrades will not happen with Monthly rollup, but we plan to eventually include patches for which ever version of IE you currently have installed in the Monthly rollup, similar to the .NET rollup”
      6. “announcement does not effect POSReady 2009”
      7. “We are working to get IE included in the monthly rollup and security-only update but do not have a confirmed schedule yet”
      8. “don’t currently have plans to extend the Rollup servicing model to Windows Vista or Windows Server 2008”
      9. “this announcement does not effect driver updates. Driver updates are not included in Monthly Rollup or the Security-only rollup”
      10. “In the new servicing model we will still be able to release security updates out of band if needed and then they would also be included in the next monthly rollup and security-only update that is released”
      11. “After installing monthly rollups, we recommend running diskclean to clean up older superseded updates”
      12. “Diskclean.exe is built into Windows 7 and can be used to clean up superseded updates. Windows 8.1 and Windows 10 automatically run clean up”
      13. “rollups will start out small, but we expect that these will grow over time to something close to the convenience rollup size”
      14. “Monthly Rollup will be classified as Critical or Security depending on the highest level of security fix in the Monthly Rollup”
      15. “Monthly Rollup will be categorized as Update rollups”
      16. “Security-only update will be classified as Critical or Security depending on highest level of security fix in the update”
      17. “Security-only will be categorized as Security updates”
      18. “If you only install security and critical updates, then you should use the security-only update rather than Monthly Rollup”
      19. “Windows Defender definition updates are completely separate from this announcement and not impacted by this change”
      20. “some ISVs also receive pre-release access to these updates to perform their own validation”
      21. “These CU are improving the overall quality of the OS while also significantly reducing the rate of support calls. So we consider the changes to be very successful and that’s why we are making similar changes with Windows 7 and Windows 8.1.”
      22. “you can still uninstall a rollup patch, its the entire rollup patch, not individual fixes included in the patch”
      23. “GWX is not included in these rollups”
      24. “for Windows 7, once the Monthly Rollup goes cumulative, the baseline will be SP1”
      25. “Monthly rollup will be available thru all the same distribution methods, Security-only rollup the same except not available thru WU”

    [edited by: ChrisB at 11:16 AM (GMT -7) on Sep 6, 2016]
  • has anyone heard/seen an update on this? October is next week and Microsoft seems very quiet for such a big change