We've had this issue before and it's been back after our upgrade to 9.2 we did last Thursday - but that seems unrelated.It seems the act of running kinstall.exe with a full patch install was enough to trigger this condition for us.This also happened, but with lower numbers, on our 220.127.116.11 installation.
So, what is our issue?We're seeing the SQL table kav.UnprocessedClientEvent filling up with entries numbering over 200,000 in a day.Normally they would be processed by Kaseya and turned over to the kav.ProcessedClientEvent table.We managed to get ourselves heard by the new Kaseya Management, Michael Duncan (VP support) called us yesterday to take this up.Since then a developer from Miami looked at this, and someone from India, having previous experience with the issue, is now involved.Currently the issue seems to be caused by an unknown Windows Update and we're not the only one seeing this now.We're told it has top priority to fix this and some SQL magic is being tried to find the culprit for this issue.
A workaround for this is to delete KAV installations on 14 machines (out of 3,000 total KAV clients) that are blocking the table from being processed.This is a temporary workaround, but then machines would need an alternative and since we don't have KES licenses, that's being debated now.They were found with the following SQL query (could be usefull if you know what you're doing):
select * from machnametab where agentguid in (select distinct (agentguid) from kav.ClientEventDeadLetter where eventtype in (163852, 163842, 163843))
I'm thinking about using Kinstall.exe, choosing option 3 and only choose the KAV module. This trick fixed it for us last time, but this time we're told it will not work, so I'm wondering if that's true.Let me see how the day ends and if I'm able to run Kinstall without bothering my colleagues.....To be continued....
Thanks for sharing this. I've been running in circles with this issue for about 2 weeks now. We clear the backed up entries, reinstall KAV through kinstall and if we're lucky we can get 1-2 installs to work. Support has just had us repeating that process over and over and it's not good when you have about 200 installs/repairs/removals waiting to be done.
If it's being caused by an unknown update I'm wondering if rolling back any recent updates might help.
Thanks for this!!! We've had a ticket open with Kaseya since September 9th for this issue.
FINALLY status are updating correctly again!!
Seeing this exact same issue only the affected machines - 13 of them - are all at a single client.
Have put in a ticket yesterday, waiting on a response.
Time for an update on this issue, wich is not yet solved for us, unfortunately.
Have been talking to Michael Duncan for the past three days and will do that again this afternoon (time zone difference is costing us some extra time to fix this).Did take the action of reinstalling the KAV module on our VSA last Tuesday, despite assurances this couldn't fix this issue.This had a very interesting effect on the issue. Instead of 14 machines blocking our KAV queue, we're down to just 2 machines!
So, this must mean Windows patches are not really the issue here, it must be something in the Kaseya database.It's interesting that of those 2 machines only 1 machine is one of the original 14 that were causing our issue.And 1 new machine was added to the mix.
Yesterday development wrote and tested a workaround fix, to get the KAV queue processing again, but it didn't work.Unfortunately there was no more time after that to discuss what's coming next.Development is going back to the drawing board and Kaseya will be testing an installation workaround for us with agent procedures.Did I already mention installations of KAV are no longer possible... - that's what's causing us the biggest headache at this time.
I hope to have positive news about this in 7 or 8 hours time.To be continued (again)
So, as promised an update and fortunately with some better news to tell.
First of all the bad news, the KAV module is running into problems that are really difficult to fix.They made it the priority of developers in India to get this sorted out, but that will take a few days, maybe even longer.
Michael Duncan came up with a nice alternative with some agent procedures to take over installations of KAV and removal.I just witnessed my notebook uninstall KAV, without rebooting, did a manual reboot, then was able to do a fresh installation.So, these agent procedures will be posted on Zendesk, all credits go to Gonzalo Carillo for a good post about this.Anyone that experiences similar problems with the kav.UnprocessedClientEvent table filling up, should make us of this.
I'll be testing this in more detail tomorrow, it's now a bit late over here, so will go for a bite and a snooze before tackling that.He's still working on some additional magic to assign a profile, that could be real important and is not handled now.
But in summary, hats off to Michael Duncan pulling the strings and Kaseya support for their great help on this issue.Maybe one last update on this tomorrow to confirm everything works without a hitch and it's back to the order the day....
OudjesEric - Thanks for confirming, and for the benefit of others, this is now posted on our help-desk as a Knowledge Base (KB) article here: kaseya.zendesk.com/.../97957368-Manually-Uninstall-Install-KAV-Both-6-and-10-
Vice President | Global Customer Support
Today we've started to use this workaround and have seen two challenges pop up.
The first is that the new Custom Field sometimes isn't filled with a Yes. You can do this manually, so just check it and don't worry about it.It's just implemented to keep track of machines you need to Connect at a later date in the KAV module.
The second is a bit more troublesome and thats applying the KAV license key is not always working. I've tried a few tricks, but the Agent Procedure for this step tells me KAV is not installed on the client. We installed a few 64-bit machines and some work, others don't at this time.So will look into this and take it up in a few hours when Miami is up and about...
The workaround has been modified just a bit to take in account some variations with the license key.That has been tested and seems to work, the post has been updated and has some extra agent procedures now.
The issue with KAV not installing at all is under investigation.We have a workstation and server, both 64 bit, that had this issue today.Windows installer seems to start and sort of hangs taking up some 30% CPU for hours on end.Disk activity is low but not zero, so something is going on.... will have to wait for an update on that.....
It's been a little while, so what's been happening on the KAV issue...Besides the Agent Procedures workaround a 'real' fix has been in the works.Hope to get an update today so we can test this.
The fix should start processing KAV events again so the KAV module can be used again with normal functionality.If the fix arrives today, we'll start testing and if all goes well I can update this one last time and forget about it.....
After two weeks of troubles we seem to have a stable situation once again.Kaseya support has put in the hours and the effort to find and fix the issue, so full credits to them.It's no surprise we wanted this found and fixed faster, but just happy we're there.
Just a very short explanation what actually was happening.KAV clients deliver all sorts of status updates and messages to a Message Queue on our VSA server.This Message Queue communicates with some default Microsoft Queue Storage mechanism.Up to this point this is happening on our frontend VSA server.The Microsoft Queue storage transfers it's content to the kav.UnprocessedClientEvent table in the Kaseya database.And the last step, unprocessed events are handled and related tables get a status update and those messages move to the kav.ProcessedClientEvent table.
Kaseya has the knowledge about almost every step and the logic needed to understand what's going wrong.They don't know much about the Microsoft Queue storage, can't look in the queue or reset it, so that's the weakest link.Due to an issue in one of the newest patches in version 9 the communication between KAV clients, the Kaseya and Microsoft queue was broken.The Microsoft Queue storage then fills up with a lot of messages, in our case we estimate some 3 million were not handled correctly.
Even after clearing all Queues and related tables it was strange to see time stamps dating back to August making it to the kav.UnprocessedClientEvent table.With a three step fixing and testing situation, the Microsoft Queue storage now seems emptied, the VSA Message Queue is stable at around 200 messages and even the kav.UnprocessedClientEvent table is able to keep up with the current pace, averaging around 50 events.
Testing confirms that new installations are carried out normally from the KAV menu option and Repair was applied to a manually installed KAV client restoring it's state to a very dull but very functional normal, but ideal Kaseya situation.
To end this long update, let me add our volume of KAV agents (over 3,000) was the main cause of our problems.Having hundreds of agents creates a lot less traffic and then the mechanism was able to keep up by brute force and took much longer for this issue to kick in.The next patch should have the fixes now tested to prevent this entire chain of events happening again.
I didn't expect to update this, maybe only to add things are looking bright and sunny, except they're not!
After a day or 2 of happiness and cheer and fixing KAV, things went to a standstill again.The queues were filling up again and not getting processed, so a reset was done (again).After the reset KAV actions were not added to Pending Procedures directly, but mostly after 15 to 20 minutes.So actions were processed, but with a delay and screen updates were lagging even more.
I don't think the KAV patch that was developed to fix issues is in the latest 18.104.22.168 patch, that just came out.I'll verify that and keep this post updated.....
What is the MS Msg Queue name exactly that contains the KAV related stuff? We are having KAV problems too, thank god for the manual install procedures.
edit: I take it is kes.service? We had over 11k messages in it so I purged it. KAV mgmt does not do anything, repair/install commands are not going to the agents.
On your VSA server, go ot Computer Management, open Services and Applications, Message Queing and then Private Queues.
You can sort this on Number of messages and then you'll probably see kes.service being a high number.Any number over 10,000 means you're in trouble and the queue is not being handled anymore.If you go down in the list of Private Queues you can open "kes.service" (only that one, not the other ones with # in the name) .
Queue messages you can purge, which will throw away the contents, meaning things start moving again.With us the issue was a Microsoft managed queue, that feeds the kes.service queue, that needed to get rid of all it's messages.Problem is, Kaseya doesn't control this queue or have tools to look into it or clean it.So, you just have to wait and repeat the purge action a few times a day, hoping the MS queue gets cleared out.
With this you should also clear out the SQL table kes.UnprocessedClientEvent for the ksubscribers database.This afternoon I'll get an update for ongoing research into the root of this issue, which is still a mystery...So, hope you can get this stable, if not, please create a ticket, support should be able to assist with actions if you are not comfortable doing this....
OudjesEric thanks for the info! Do you have the exact sql commands that clears out the kes.UnprocessedClientEvent table? I'm novice when it comes to SQL :)
Is this it:
select * from machnametab where agentguid in (select distinct (agentguid) from kav.ClientEventDeadLetter where eventtype in (163852, 163842, 163843)) ?