Kaseya Community

IMPORTANT -- [Unexpected Issue Identified with Kaseya Patch 9.3.0.16]

  • We have identified an unexpected issue with our 9.3.0.16 patch that was released last night that potentially could impact your Kserver after the install. We have pulled this patch to prevent any possible impact and are working on correcting the issue and will re-post an updated patch as soon as this has been corrected. We sincerely apologize for any inconvenience this may have caused.

    Michael Duncan
    Vice President | Global Customer Support
    Kaseya



    removing sticky
    [edited by: Nicolas Ponce at 1:31 PM (GMT -7) on Aug 23, 2016]
  • For those of us who did install 9.3.0.16, what is the unexpected issue?

  •  After installing patch 9.3.0.16, a re-apply schema may give an error. If this does occur, customers are advised to contact Kaseya Support immediately by submitting a Severity 1 ticket, to assist with the remediation.

  • it looks like this has officially be re-released?

  • 9.3.0.16 showing as available again in our VSA.

    - How did QA not catch the fact that a patch broke Reapply Schema?

  • Hmm.. we installed 9.3.0.16 last week but have not needed to reapply schema / not noticed any problems so far.

    Should we run the update process again and tick the box to re-apply 9.3.0.16, or should we just wait for the next patch release (presumably 9.3.0.17)?

  • Hi

    As of this moment I would not recommend running the update and re-applying .16. I would go by Michael Duncan's post on the 19th and wait until an official word is released on this. I strongly recommend you open a ticket with your concern and hold off until you can get a confirmation.

  • You should be able to check again and only see .15 available.

  • Thank you Oscar.❤

  • FYI for those following this thread:  9.3.0.16 has been rereleased.

  • The unexpected issue with our 9.3.0.16 patch has been corrected and an updated 9.3.0.16 patch has just been released and is now available. Thank you for your patience through this.

    Michael Duncan

    Vice President | Global Customer Support

    Kaseya

  • Hey Brian,

    I think it might be more appropriate if I answer that question.

    Patch installers are designed in a way that they are NOT supposed to run "Reapply Schema". As such, we never test that part of the code on a patch rollout as it should be irrelevant to the behavior of the patch. However, an unexpected technical failure in our build automation process during a recent change caused an edge case condition to occur that caused Reapply Scheme to break as the wrong mapping file was picked up during the construction of the installer. This issue only is seen if Reapply is run.

    Ultimately this was a failure in our engineering processes. We have been doing so well with the release management process lately that we decided to start running two patches live in parallel in an effort to deliver more value quicker. In doing so, we stumbled upon a defect in our build automation software that cached files for a longer period of time than we expected. As such, a beta build of the v9.3.0.17 patch left behind a file in memory that the v9.3.0.16 picked up. We have worked with the vendor of the build automation software and found a way to prevent that from ever happening again.  

    At the same time, when we conducted a root cause analysis (RCA) for the failure we determined that going forward we will avoid this fault condition by always doing a Reapply Schema test, even on a patch. This way we will prevent that condition from ever leaking out again. We have already documented this in our runbook and created a template in the test cycle management software to ensure it will always be done going forward.

    Our apologies for this failure. As we continue to become more efficient in our processes and we entertain parallel processing options we will be a bit more careful in validating things across the build cluster. We literally were building v9.3.0.16, v9.3.0.17, 9.4.0.1beta1 and the new cloud backup module within seconds of each other and found technical failures in our tools we never encountered before. We believe that is behind us, but that is no excuse.  

  • Thanks Dana,

    It's nice to know that these sorts of issues are being addressed systemically on the back end. Good work in catching the bug early and preventing as many people as possible from suffering issues.

    Now, I have 4 KAV tickets open regarding KAV problems in patch 9.3.0.16 that are still in waiting for assignment after 2 days. Any ideas on when they may be looked at?

  • Well Dana, kudos to you for explaining what was going on. Software will always surprise you, no matter what you catch, there's always more but I do like to see updates from you and Michael on these issues.

    We installed 9.3.0.16 last night and the process was nice and smooth....

  •  Hi Craig - I will follow-up internally on those 4 KAV tickets you mentioned regarding KAV problems in patch 9.3.0.16, and will have someone from my end reach out to you ASAP.

    Regards,

    Michael Duncan

    Vice President | Global Customer Support

    Kaseya