We have identified an unexpected issue with our 188.8.131.52 patch that was released last night that potentially could impact your Kserver after the install. We have pulled this patch to prevent any possible impact and are working on correcting the issue and will re-post an updated patch as soon as this has been corrected. We sincerely apologize for any inconvenience this may have caused.
Michael DuncanVice President | Global Customer SupportKaseya
For those of us who did install 184.108.40.206, what is the unexpected issue?
Craig Hart After installing patch 220.127.116.11, a re-apply schema may give an error. If this does occur, customers are advised to contact Kaseya Support immediately by submitting a Severity 1 ticket, to assist with the remediation.
it looks like this has officially be re-released?
18.104.22.168 showing as available again in our VSA.
Michael Duncan - How did QA not catch the fact that a patch broke Reapply Schema?
Hmm.. we installed 22.214.171.124 last week but have not needed to reapply schema / not noticed any problems so far.
Should we run the update process again and tick the box to re-apply 126.96.36.199, or should we just wait for the next patch release (presumably 188.8.131.52)?
As of this moment I would not recommend running the update and re-applying .16. I would go by Michael Duncan's post on the 19th and wait until an official word is released on this. I strongly recommend you open a ticket with your concern and hold off until you can get a confirmation.
You should be able to check again and only see .15 available.
Thank you Oscar.❤
FYI for those following this thread: 184.108.40.206 has been rereleased.
The unexpected issue with our 220.127.116.11 patch has been corrected and an updated 18.104.22.168 patch has just been released and is now available. Thank you for your patience through this.
Vice President | Global Customer Support
I think it might be more appropriate if I answer that question.
Patch installers are designed in a way that they are NOT supposed to run "Reapply Schema". As such, we never test that part of the code on a patch rollout as it should be irrelevant to the behavior of the patch. However, an unexpected technical failure in our build automation process during a recent change caused an edge case condition to occur that caused Reapply Scheme to break as the wrong mapping file was picked up during the construction of the installer. This issue only is seen if Reapply is run.
Ultimately this was a failure in our engineering processes. We have been doing so well with the release management process lately that we decided to start running two patches live in parallel in an effort to deliver more value quicker. In doing so, we stumbled upon a defect in our build automation software that cached files for a longer period of time than we expected. As such, a beta build of the v22.214.171.124 patch left behind a file in memory that the v126.96.36.199 picked up. We have worked with the vendor of the build automation software and found a way to prevent that from ever happening again.
At the same time, when we conducted a root cause analysis (RCA) for the failure we determined that going forward we will avoid this fault condition by always doing a Reapply Schema test, even on a patch. This way we will prevent that condition from ever leaking out again. We have already documented this in our runbook and created a template in the test cycle management software to ensure it will always be done going forward.
Our apologies for this failure. As we continue to become more efficient in our processes and we entertain parallel processing options we will be a bit more careful in validating things across the build cluster. We literally were building v188.8.131.52, v184.108.40.206, 220.127.116.11beta1 and the new cloud backup module within seconds of each other and found technical failures in our tools we never encountered before. We believe that is behind us, but that is no excuse.
It's nice to know that these sorts of issues are being addressed systemically on the back end. Good work in catching the bug early and preventing as many people as possible from suffering issues.
Now, I have 4 KAV tickets open regarding KAV problems in patch 18.104.22.168 that are still in waiting for assignment after 2 days. Any ideas on when they may be looked at?
Well Dana, kudos to you for explaining what was going on. Software will always surprise you, no matter what you catch, there's always more but I do like to see updates from you and Michael on these issues.
We installed 22.214.171.124 last night and the process was nice and smooth....
Craig Hart Hi Craig - I will follow-up internally on those 4 KAV tickets you mentioned regarding KAV problems in patch 126.96.36.199, and will have someone from my end reach out to you ASAP.