After having things setup and only seeing occasional patch failures, suddenly this month I'm seeing a *ton* of failed patches across multiple clients. I have well over 200 machines with failed patches at this point in time with an installed and configured base of just over 1000 machines configured for patch management.
I'm just curious to know if others are seeing issues with this or not. I've been working with a couple of example cases and have narrowed down my overall issue at this point, but I'm wondering if anyone else is seeing this behavior. (and yes I'm getting ready to open a ticket with support right this moment as soon as I finish typing this.). In the investigation process that I'm working through, what I'm finding is that the patches are failing to download. In most cases we are using the LANCache configuration, and when I look after the patches have failed the patch will not be in the LANCache folder. So I dug into this with one specific case just now.
I scheduled 1 patch to run on a 1 system to trace the process all the way through. kb2817478 an update for Publisher 2010. So I watched and saw where the system scheduled a script on the machine to verify that it could access the lan cache, which succeeded. It then scheduled a script *on* the LANCache server to download the patch. I used the hidden functionality in Kaseya to look at these system scripts from the live connect screen to see exactly what it does to download the patch.
It first writes out curl-nossl.exe to the system in the lancache\vsasharedfiles\PATCH folder. Then it runs that file with the commandline options to download the patch. IN this case:
curl-nossl.exe https://download.microsoft.com/download/5/C/A/5CAB8098-2AD3-4BDC-A0BF-7BAD2BC70750/publisher2010-kb2817478-fullfile-x86-glb.exe -C - -k -L -o "C:\LANCache\VSAFileShare\PATCH\publisher2010-kb2817478-fullfile-x86-glb.exe"
So to test I connected to the system and ran the same commandline option myself... It basically just silently fails... No file is downloaded, and it gives no error messages, it just doesn't do anything and returns to the command line. I tried varying options including just specifying the URL with the -S option to show any errors, and they all simply just return to the command line silently failing with no errors.
I tried the same commands directly on the Kaseya server, and my own system with the same results..
On a system where the install is failing (preferrably the same one where you ran the curl-nossl.exe command locally), what happens if you open a web browser and navigate to the URL provided for the download (download.microsoft.com/.../publisher2010-kb2817478-fullfile-x86-glb.exe)?
It's possible the machine can't access the URL - and if the machine can't browse to it, curl won't be able to snag the results of the URL (the file download) either. I was able to download the patch from the URL you provided, so the location appears valid.
Kaseya Community The download link downloads just fine in a web browser. I tested using the command line curl-nossl.exe from Kaseya from multiple places and it fails on *all* of them.
In some additional testing that I did I confirmed that *some* patches are working fine. The difference between the patches that fail and the patches that succeed is that the ones that are failing to download are specifically ones where the file source is httpS://download..... and the ones where it's working are http://download.... Offhand I'm going to guess the cause is that Kaseya's compiled version of curl doesn't support https URL's (Thus the name curl-nossl.exe), and suddenly microsoft has started supplying the patch downloads via SSL secure sites...
Same problem for me mate. Had 20 clients lined up for Patches over the weekend and everyone failed with 0 patches installing.
Same here. All sorts of issues with patches failing for no apparent reason, since R9.2 upgrade.
Ticket 111447 open on the matter. No response to a level 2 ticket after 3 days. Sigh.
All sorts of issues here as well....both with 6.5 (prior to Friday) and 9.2 (over the weekend)
Yeah, I'm 99% certain that the issue is with Microsoft starting to issue patches with SSL required for downloading, and Kaseya was caught off-guard by this, but I've yet to get a response on my open ticket though, so at this point it's all supposition.
For anyone from Kaseya looking at this thread my currently opened ticket that still in "assigned" status is #111539
Did get a response from a tech this afternoon just letting me know that they have escalated it to Product Support Engineering to investigate further and that it may be another 24-48 hours before I get an additional update.
The support tech *did* specifically recommend that for anyone running into this that you do *NOT* manually update the patch location, as if you change anything in the patch location for a patch, then it actually sets a flag in the database so that Kaseya knows you've manually overridden the patch location and from that point forward it will not update that record so that it doesn't overwrite what you manually put in place. Overall it makes sense to me they would do that, but I guess to be safe I won't test by replacing any of the patch locations with a "Non-SSL" URL.
Thanks for the update Jonathan. Please keep us up to date....I haven't opened a ticket since I am monitoring yours here
Thanks for this info, we didn't notice this issue yet.
Will be looking into this today.
We are seeing a lot of patches with HTTPS:// download locations - these patches never make it to the VSA. If any patch download fails, it seems to abort the whole patch cycle and ALL patches fail.
If we install patches one at a time, those with a HTTP:// patch location do download to the agent, however we then get a second issue - we get the error code back "Windows update could not be installed because of error 2359302". 2359302 is 0x240006 in hex which is "0x00240006 WU_S_ALREADY_INSTALLED The update to be installed is already installed on the system".
This implies WU or Kaseya has gone completely nuts and now thinks patches that are already installed, still need installing.
Check this screengrab - the WU GUI thinks KB2647753 *IS* required (and will successfully load it, if asked), but at the SAME TIME the Kaseya-downloaded .MUI version of the same patch indicates that it is ALREADY INSTALLED. W.T.F. ???
I'm guessing either Microsoft has messed up, or, this whole HTTP vs HTTPS download situation is totally messed up in some way that I can't get my head around.
...and where is Kaseya's response in this? Patch management is one of the pillars of an RMM and all our clients are currently out of compliance with what we have promised to provide!
The issue has been reported to Kaseya Support and they are working on the issue. As with all technical issues, to keep apprised of the most current status, if you are experiencing the issue, please open a ticket with support. The Community is a great place to get and share information, but for up-to-date status of an ongoing issue, a ticket is the best way to ensure you receive regular status updates.
I do have same issue where hundreds of machine fails patches. I feel it pertinent to mention that if there are 20 patches lined up for a machine few out of them fails from same file source and few fails.
I have blocked installation of IE-11 and windows 10 Upgrade patch in KB override but still I see these installed on most of the machines. I am not sure if Microsoft is fooling us by changes KB article number of these patches.