Eric Nemchiksupport knew about the issue [with Agent Procedures not running correctly] and was able to resolve it fairly quickly but you'll want to keep an eye out for procedures to make sure they are all running correctly
This would be devastating for us if I upgraded our production server to 9.1 and our Agent Procedures weren't running. Do you have a ticket number you can share so that I can preemptively ping KSupport to know what to watch out for and how to fix it?
Or perhaps it's fixed in 220.127.116.11 (already running on Preview, but not yet GA or listed in Release Notes). Perhaps someone from KSupport can weigh in?
I do not have a ticket number to reference because the issue was handled as part of a a web session while discussing another ticket. The tech mentioned in the notes that loadsubagent.sql was reloaded (i dont think that's the exact name). I know we ended up opening an SQL file from the kserver and running the SQL and that resolved it.
Lucky you! When I reported the same problem they told me I migrated improperly and needed to start over. We were moving to a new server. Support told me I restored the database wrong. Disappointing to see it was (is now?) a known issue.
Mike_Judd We did an upgrade, not a migration, so that MIGHT have been the reason for their willingness to assist with my issue.
No, I believe that not everyone is on the same page.
I spoke to 3 different techs before settling on the process to migrate to a new server. Even then, I was told it was wrong. Funny, the techs that helped me define that process weren't employed at Kaseya afterwards.
Sounds like you got a tech that knew of something that others did not.
Hi Eric Nemchik Mike Judd Brian Dagan
The issue Eric is reporting was caused by performance of the server.
The technician who was assigned to the case and worked it, requested performance tuning from a higher level of support.
Our senior support engineers were able to provide certain steps that would assist and alleviate the high performance usage on the server.
To give a high level overview of the case, high consumption of resources on the environment caused the procedures to not run.
With all this said, Eric's support case was not a code issue but more of a performance tuning case.
If you are experiencing any of these symptoms and want to verify if you may be affected, please submit a support ticket.
I've got a few names of the internal staff that I definitely feel do a good job, and I've also got a few names of people I won't deal with and I request another tech when they get my tickets.
If you're still having trouble, ask to speak with someone who is specifically in the agent procedure department and then mention reloading the SQL file. I was told the issue was recognized internally and caused by upgrading kaseya (including upgrading patch releases on the same major release).
I am actually referring to a ticket where I was discussing remote control issues (which is still open) and during that ticket the technician I worked with asked for assistance from another technician who is more familiar with Agent Procedures, since I brought up all the issues I had with R9.1 on a single ticket before separating the issues into individual tickets. The technician who was asked to assist spoke with me on the phone and guided me through reloading an SQL file from my KServer. This fixed the issues I was having with agent procedures failing. He said this was a known issue with upgrading and even happened when doing patch upgrades. Most of this was not noted in my ticket because the original technician assigned is a KRC specialist and not familiar with agent procedure troubleshooting.
Regardless the issue was resolved, and this fix should be easy for others to perform. Hopefully Kaseya support can facilitate this for those who need it.
Hi Eric Nemchik
You are correct, I was able to discuss with the technicians who assisted you on the case and they have clarified the exact steps taken.
There was two unique issues that were present after your upgrade that caused procedures to run but fail. (Separate from the performance issue which causes procedures to not run at all or hang)
In this case the procedures are running, or trying to run and failing due to the following - which will be visible in the Agent Procedure Log:
This was resolved running specific steps by Kaseya Support. This involves running a handful of SQL queries to detect if any procedures are not approved then, deleting certain data, and reprocessing the procedure approval task.
Anyone who has recently upgraded and wants to find out if they have unsigned procedures trying to execute - please create a support ticket.
This issue has been identified and reported to engineering whom is working on resolving this in a future release.
This was resolved in Eric's case by what he mentioned - reloading the loadsubagent.sql file.
Another alternative to correct this is run reapply schema.
We are performing a RCA for this to identify what me be causing this for certain customers.
If you do see these symptoms, please create a support ticket as well so we can ensure it is corrected with the steps provided.
From discussing with support not all customers will experience these issues, however some may. We strongly encourage you to create a ticket if you do have these symptoms.
Support was able to identify these issues from the failures in the Agent Procedure Log.
So if you are interested in proactively monitoring this issue, you can setup the "Agent Procedure Failure" Alert on your machines to notify you.
Please let me know if I can clarify further, provide any other information, or help in any other way on these issues.
Nicolas Ponce For reference, in my case, reapply schema did NOT resolve the issue, nor did performing a patch upgrade, or fully reinstalling the VSA. Loading the SQL file manually was required.
Thank you for posting the information publicly.
I had the same problem with the Agent Procedure signing. To re-execute the agent procedure signing process I performed the steps below against the ksubscribers database to resolve this issue:
truncate table [dbo].[ScriptHash]
truncate table [dbo].[ScriptApproval]
delete from tempdata where tempname = 'AllProceduresSignedAndApproved'
Note: Afterwards, you can monitor the progress of the approval process by running the query below and watch the "ScriptsRemaining" column decrease to 0:
SELECT (SELECT Count (*) FROM dbo.scriptidtab) AS TotalScripts,
(SELECT Count (*) FROM dbo.scriptapproval) AS ApprovedScripts,
(SELECT Count (*) FROM dbo.scripthash) AS HashedScripts,
(SELECT Count (*) FROM dbo.scriptidtab
WHERE scriptid NOT IN (SELECT scriptid FROM dbo .scriptapproval)) AS ScriptsRemaining,
(SELECT Count (*) FROM dbo.scriptapproval
WHERE scriptid NOT IN (SELECT scriptid FROM dbo .scriptidtab)) AS OrphanedScriptApprovals,
(SELECT Count (*) FROM dbo.scripthash
WHERE scriptid NOT IN (SELECT scriptid FROM dbo .scriptidtab)) AS OrphanedScriptHashes,
(SELECT tempname FROM dbo.tempdata WHERE tempname = 'AllProceduresSignedAndApproved') AS Outcome
BlueRockTech YES! that's exactly what I had to do.
I'm sure you found this out already, but for anyone just tuning in, be aware that re-hashing of all of the scripts can take a long time. The last time we had to do this, we hashed 201,084 scripts in 1 hour and 40 minutes (2,011 scripts hashed/minute).
In our case, the re-hashing was needed because KSupport had recommended a query to "clean up" the ScriptThenElse table of all of the no-longer-needed dynamically-generated "legacy" patching & monitoring scripts, but when you do that, you break the existing hashes. In other words, a SQL query they would otherwise routinely run to clear out the ScriptThenElse table of stale data caused script approvals to break entirely once Script Approvals were introduced.
I've had to trigger this due to an upgrade as well. The frustrating part is the unforeseen consequences of having your KServer up and running while it's doing script approvals. For example, we (nightly) populated a comma-delimited list of detected services/apps that need monitored into the Machine Custom Fields. The machines then fall into Views, which activate Policies, which activate Monitors, Agent Procedures, etc...
The problem comes when the KServer starts accepting check-ins after a version upgrade (which you can't stop anymore by changing the check-in port due to the new back-channel communication/relay servers that KRC uses) in the midst of running this re-hashing re-approving process. The chain of procedures will get halfway through and bork when it stumbles on a subscript that's not yet hashed/approved (and fail silently except for entries like this in the Agent Procedure Log):
Sat Nov 08 13:24:08 2014: Agent procedure Daily Agent Check* (1289944764) is not approved! Sat Nov 08 13:24:10 2014: Agent procedure Check Anti-Virus Status* (1991663930) is not approved! Sat Nov 08 13:24:11 2014: Agent procedure Automatic Monitoring Detection* (1053449658) is not approved! Sat Nov 08 13:24:12 2014: Agent procedure Check Anti-Virus Status* (1991663930) is not approved! Sat Nov 08 13:24:14 2014: Agent procedure Daily Agent Check* (1289944764) is not approved!
So, in our case, we ended up with a half of a comma-delimited list, which meant that half of our machines weren't falling into the correct Views, and thus picking up the correct Policies, and were therefore not being monitored correctly until the next iteration of the Automatic Monitoring scripts & Policy re-process.
Anyway, this is just an example of how Kaseya implemented something virtually nobody wanted (Agent Procedure Approvals) without fully considering the consequences (long re-hash timing), and KSupport not all being on the "same page" regarding some SQL queries which used to be okay but break things on the new version.
TL;DR: Watch your KServer Log after running a Framework upgrade, because sometimes your Agent Procedures won't get re-hashed and re-approved, and the only way you'll know is either A) Noticing something wonky / not working & digging deeper -or- B) Checking your KServer Log. I can confirm that the procedures & queries above will re-force a hashing of the Agent Procedures (approving all existing scripts), but be aware that it takes a loooong time.
I was coming here just to gripe about that; I had to dig in the CSS to find the actual width & height values for the logo element. Sloppy, Kaseya guys, just sloppy!
Of course, I kinda miss being able to have our custom image off to the side of the login, too. I'll get over it...
Having this issue with ticket #90070
The unique problem we seem to be having is that certain procedures (when run during the day) and/ or around 1AM on its own, any non system script fails to run and has the "Error: This agent procedure has not been approved yet" in the AP logs.
We have that re approval query scheduled to run twice a day to try and put a band aid on the issue until they can figure out why it continues to happen.
Ticket has been open over a month now, getting very frustrated with this new procedure approval thing...