Kaseya Community

Way more than normal Patch Failures this month

  • I am seeing a *way* more than normal instance of patch failures across all of our agents from the September patches, and I'm wondering if I'm the only one?

    I'm in the process of starting a support ticket on this (provided I can get in to the helpdesk site my password is being a pain right now.), but thought I'd bring it up out here just to see if anyone else is seeing this.

    From what I'm seeing right now while there seem to be several different patches having issues, there is one that I'm seeing in the list of failed patches a *lot* more than others.  That is KB3172605.  I started trying to diagnose this specific patch on several failed machines and the only information shown in the agent procedure log from the patch scan is simply "<agentname> does not have a patch log file for KB3172605".  I went to try and manually install this patch on one of the failed machines and when I do so, it actually states "Update for windows (KB3172605) is already installed on this computer.).

    My overall guess at what's going on here is that this was a patch that was originally released in July 2016, but then was "re-released" with the same KB article in September..   I'll update this thread with what I hear back from Kaseya if/when I hear it. 

    <edit>Finally got into the helpdesk site and was able to create a ticket, my ticket # for reference is:#156042</edit>



    Added ticket Number.
    [edited by: Jonathan at 7:28 AM (GMT -7) on Oct 5, 2016]
  • KB3172605 is actually a September update to an earlier patch and the KB number stayed the same (IIRC).  I removed the patch source so that one would download directly and that seems to have helped.  This is located in Patch Management - Patch Parameters - Patch location; filter on KB Article 3172605 and remove the location so it is Internet-based Install Only



    expanded on explanation
    [edited by: karoded at 8:07 AM (GMT -7) on Oct 5, 2016]
  • ,

    If you are seeing the issue where installing the patch manually reports that the patch is already installed, the failure(s) may be due to faulty detection logic.  Here's the deal:  Microsoft scans machines (whether through Kaseya or when running a local scan via Control Panel) to determine what patches are appropriate for a machine, as well as their status (installed, not installed).  If a patch is not detected, it is reported as not installed (listed as "Missing" in the VSA).  If a patch tried to install via Kaseya and a post-reboot rescan reports the patch is still not installed, the patch is considered failed.  However, if MS is not detecting the patch as installed, even though it is installed, this can lead to a 'false' report of a failed patch.  It all goes back to the detection logic.

    An easy way to determine if this is likely the case:  use one machine for testing.  Check the local machine for the installed updates and see whether the patch in question is listed.  Next, run a scan on the machine via the Control Panel > Windows Update > Check for Updates function.  Make note whether the patch in question is listed as missing.  You can try to install directly via Windows Update and then rerunning a scan to see if the patch successfully installed via Windows Update is still being reported as missing.  However, if MS has already re-released a corrected version of the patch, then the patch would install and there'd be no further indication of it missing on the local machine.  Timing can be important here - a successful patch install via Windows Update does not necessarily mean that detection logic is not the cause of the behavior you're seeing.

    If the patch is showing as installed in the Installed Updates section BUT is showing as missing from the Check for Updates list, then the issue is the detection logic.  This is a common-enough occurrence.  Microsoft will generally re-release a corrected patch that CAN appropriately be detected, but that could be several days or weeks after the initial release of the patch.  It's possible that MS already has released a corrected version of the patch (assuming all of the above is the cause of the issues for this specific patch).  If you are using a LAN Share, LAN Cache, or KServer as your file source, I recommend you delete the patch file stored on that file source, then rerun this update.  That will force the patch file to be redownloaded from MS.  If they've released a corrected version, then the patch should install AND be reported as installed.  However, if MS has not yet rereleased the patch with one having corrected detection logic, you'll be in the same boat as you are right now.  The patch IS installed but ISN'T being detected by the MS Windows Update Agent.

    If you determine this is likely due to detection logic and you want to make the VSA as 'clean' as possible until the patch is rereleased, you can always set the patch as Denied and then use the "Clear Failed" option (Patch Status > click the hyperlinked "failed" number for the endpoint, and click "Clear Failed" button).  This will prevent the patch from attempting to reinstall with each patch cycle.  You can re-approve the patch once MS rereleases it.

    Regarding the AP Log entry about no KB Log file available:  When a patch is determined to be failed after an install attempt, Kaseya attempts to gather an associated installation log.  MS does not automatically create logs for all patch installs, but because they do for some, the system will attempt to locate and upload that file.  If no file is found, you will see that entry in the AP Log history.  This entry itself is not indicative of a problem - rather just that the log file doesn't exist because the patch wasn't coded to automatically generate the log.  The failure of the install is, of course, an issue, but the presence of the AP log entry is not.  If you want to generate a log file for a patch install, run the patch locally on a machine via command line and add the /? switch.  This will display the help dialogue.  if a log switch is supported, you can either add that to the patch under Patch Management > Command Line (On Prem only) or you can run the patch via a custom agent procedure or locally on the machine using the appropriate log switch.

    All of the above is assuming the issue you're seeing is related to detection logic.  If this is not the case, Support will be able to work with you to access your system and investigate further.  If these particular failures are not due to detection logic, I hope this info is useful at some point in the future.

    Thanks,

    Brande

  • Well at this point the test machine that I can interrupt during the middle of the is showing me this.  

    The "original" patch KB3172605 does show as being installed back in July when it first came out.

    The "new" patch that release 9/13 also named KB3172605 does in fact show up as "missing" when you a search with windows update.  However *manually* trying to download and install the update from Microsoft's site, it won't install as it thinks it's already installed.

    I tried installing it from within Windows update on the machine, and it *then* it installs properly...  

    Is there a way to clear failed on *all* of the machines with this failed update right now, if I want to go ahead and deny it for now, just to clear things up for the moment until I come up with a better fix?  Right now I have over 350 machines showing with this patch failed, so doing them one at a time is going to be very repetitive and time consuming.

  • We were able to get right by this problem using the steps I provided.  Since you are making the change for the specific patch, you only have to click 4 items

  • We are in KB3172605 hell also. I'll try karoded's advice - I recall this 'fix' working for other broken MS updates in the past.

  • , ,

    To clarify what this 'fix' is doing, it is just forcing the update to use the Windows Update Agent to complete the install (rather than use the .exe, .msi, or .msu to execute the distributable version of the patch via command line).  Once the patch URL is removed, if MS releases a new download URL for the same patch, your server will not receive the updated URL.  Editing or removing the URL will set a flag instructing the VSA to ignore any further URL updates to that specific patch.  I'm not suggesting you can't move forward with this approach, but it isn't a solution for everyone.  In fact, this specific option of Patch Location is only available for On Premise servers.

    The same end-result can be achieved by configuring the File Source as "download from internet" and running Automatic Update.  This specific install method + file source will result in the local Windows Update Agent (WUA) to install the patch.  This may not be a solution for everyone, but it does prevent creating a permanent flag against the patch location record.  If you are using Policy Management, you could define the preferred file source (LAN Cache, LAN Share, etc.) via a policy, then create an override by setting the file source in the patch module to "download from internet."  With the file source as Internet, run an Automatic Update cycle.  Once the cycle is complete, force the machines back into Policy compliance by running the Policy Management > Machines > Clear Overrides / Reprocess Policies function.  Note that you must use the Automatic Update function to install (not Patch Update or Machine Update).  Patch and Machine Update functions will download the .exe, .msi, or .msu from the internet directly to the machine, then execute via command line.  Automatic Update will always leverage WUA when the file source is "download from internet."

    This solution might seem a bit more involved than simply removing the patch location, but it is a viable option for Cloud/SaaS customers and is generally preferred over creating that flag on the patch location in the db.

    If there is a specific patch creating issues like this and MS does not address the issue quickly, you can open a ticket with Support and request the patch location be updated (in this case, removed) to allow for the successful install and detection.  This option may be reserved for situations where we have several consistent reports of issues with the same patch or we can identify the root cause of the installation failures as a detection logic issue.  It is important that any changes made to the master URL location file (which all VSAs received as part of a regular, backend process) are applicable to the majority of customers.  Allowing Kaseya to update the URL, rather than doing so on your individual server, will allow Kaseya to control the patch location, prevent the URL flag in your db, and if MS does correct the patch location, will ensure you get the corrected location once it is released.  All that said, if you choose to adjust the patch location for an On Prem server, you are of course welcome to do so - I just want to ensure what's actually occurring and the potential impact is fully understood.

  • Brande, your solution to let Kaseya update the URL with the new version would work great if Kaseya actually did that. It is now February 21 and I never received an updated URL for that patch. The URL still referred to the version originally released in July, not the updated one from September. Therefore, I had no choice but to remove all the URLs and it's now installing cleanly.