Hey Team, In preparation for patch Tuesday April 2020, I really just wanted to take a moment and share with the community some of my recommended resources to research and properly plan your patch deployments to your endpoints from my experience.
During these unprecedented times with a substantial large spike in remote users, VPN utilization, and network saturation I find it critical to share some best practices on patch deployment during these scenarios.
1 - Prepare and Plan
- During my time as a Lead Engineer for a Service Provider, the research and preparation was the most important step. This would ensure the patches I was getting ready to approve would not hinder our managed endpoints and servers (of course there are exceptions).
I recommend this blog personally.
2 - Scan and Deployment Schedules
Execute your scans throughout this week (Patch Tuesday week) to ensure you have the latest information available on your endpoints. Distribute your scans extensively. This is important as users are mostly working from home networks, we want to conduct software management/patch management related tasks during non-peak hours to ensure the tasks are executed.
Distribute your deployments. It is no secret Windows patches are beginning to get larger in size (some over 1GB). This can strain not only your server but also your remote user’s network. I highly recommend staggering deployments with 6+ hour distribution windows if you are deploying during ‘business hours’.
3 - Utilize different delivery methods
- For our Software Management users, we just released a new native control patching feature. You can learn more about it here. We have given our users the ability to manage endpoint patching at a GPO-level with specific settings available within your VSA.
- I recommend to create these new Native-OS profiles for any endpoints that may have limited bandwidth/availability. There are key settings to help support the patching of these endpoints.
- Our users also have the ability to enable remote power features and more to support your patch strategy
4 – Revisit your entire patch strategy
As a large majority of users are working remote, this presents an opportunity to revisit and redefine patch strategies within Kaseya.
Successful Kaseya users have a planned and phased approach to patching (utilizing Classic Patch Management or Software Management). I recommend to review the following:
- Reviewing new patches as they are released and create a plan to test deployments of these newly available patches to a farm or select group of endpoints.
- Review scheduling of Scanning times with expanded distribution windows. Can you scan the endpoints during non-business hours? Daily scanning should only be utilized for key scenarios of endpoints, can you revise frequency of scanning?
- Review scheduling Deployment times with expanded distribution windows. Can you deploy the endpoints during non-business hours? Would your remote endpoints benefit by configuring the windows update settings via software management?
I hope you found some of these suggestions useful. If you have any questions, please post on the community so we can help each other be successful during these challenging times.
Here is a detailed list of Security updates from April 2020 Patch cycle.This is particularly useful to understand specifics around deployment approvals and rejections.
Recent update on a particular April patch which seems to be causing issues.
A few things we are doing on the Kaseya side I wish to share with you:
Oscar Romero if I click on the word documentation in your post immediately above, i receive a 'group not found' error.
Thank you Craig Hart - This happens more frequently with me (need to get better at that).
SO ... doesn't make a difference what we do, every time we enable Software Management it just completely destroys our bandwidth ... All these new things are NICE, but it hasn't;t fixed the initial problem of killing the bandwidth of the KSERVER network .... When will this be fixed?
Thanks for your feedback. We are working towards elevating our patching and third party applications.
The feature I specifically mentioned above does utilize the GPO Windows Update Control functions which reduce the dependency of VSA being the source of file and bandwidth.
It still slammed my bandwidth just setting it up for less than 250 machines...
Here is a detailed guide on the new Windows Update Bandwidth control.
(Available within Software Management Native Controls today).
1 - Is it possible to both apply the GPO controls and still choose which updates to approve/deny, or are we limited to one "strategy" per agent? We want to still be able to choose what updates are approved and when they are installed.
2 - We have started trying using SM to deploy OS updates but so far the results seem unreliable. We attempted deploying updates from the 'Machines' tab inside SM, and for days the status of the updates was stuck on "Installing" even after reboot+rescan. Then today the agents started showing errors that the updates failed. We have a ticket open for this but looking for more feedback about what we should be expecting from SM at this time.
1 - Currently you can assign one strategy per agent.
2 - What is the ticket number on this?
Just posting here as an update on the new feature ... We decided to give it another go, and deploy slowly (a few machines at a time) ... While it no longer slams our bandwidth this way, it now however totally screws the endpoint in a very inconsistent manner ... Hundreds and Thousands of GPO not able to be applied hit a lot of machines, every second 4 or 5 are logged in the Application Event Log ... and this causes system slowdowns, and other apps to crash (most notable QuickBooks, Outlook, WordPerfect, and a few others) Some are complete crashes as WordPerfect just exits for no reason spontaneously in the middle of typing, and QuickBooks becomes unuseable from flickering and the drop down menus are unable to be selected because it flickers away before you can click on anything ... Once we remove the profile for setting up the new feature in SM all of this stops immediately - the warnings in the log files and the weird crashes, etc ...
We have logged a support ticket to report this issue - the ticket # is 744774 in case you would like to follow up on it ...
Hey Tom Fehlberg
We will have our support team look into this. I use this feature personally and have not noticed anything performance-wise and I have event logs from the GPO category three days ago (they are warnings).
Yes they are warnings stating that the GPO can't be processed for some reason or another, and it happens every second (quite a few every second actually in our environments) and it causes the machines to eventually have issues with other software ...