I have a client that recently asked this question. I looked into a few servers and I found that there is quite a high "I/O Other" stat when looking at the AgentMon.exe process for all servers running the Kaseya agent.
My client's example was on an SQL server, and the AgentMon.exe process had an "I/O Other" of 109,344,016, whereas the sqlservr.exe process only had an "I/O Other" of 68,119,447 which stuck them and I as a bit odd.
I did some research into the issue and found a quick definition of "I/O Other":
"Shows the rate at which the process is issuing I/O operations that are neither a read or a write operation. An example of this type of operation would be a control function. This counter counts all I/O activity generated by the process to include file, network and device I/O's."
So I then broke out ProcessMonitor and began investigating the operations being performed by the AgentMon.exe process. I found that about twice every second, the following set of operations happen:
RegOpenKey HKLM\System\CurrentControlSet\Services\VxD\KaseyaRegOpenKey HKLM\System\CurrentControlSet\Services\VxD\Kaseya RegQueryValue HKLM\System\CurrentControlSet\Services\VxD\Kaseya\ConfigurationFileRegQueryValue HKLM\System\CurrentControlSet\Services\VxD\Kaseya\ConfigurationFileRegCloseKey HKLM\System\CurrentControlSet\Services\VxD\Kaseya CreateFile C:\Program Files\Kaseya\Agent\KaseyaD.iniQueryBasicInformation C:\Program Files\Kaseya\Agent\KaseyaD.iniCloseFile C:\Program Files\Kaseya\Agent\KaseyaD.iniRegOpenKey HKLM\System\CurrentControlSet\Services\VxD\KaseyaRegOpenKey HKLM\System\CurrentControlSet\Services\VxD\Kaseya RegQueryValue HKLM\System\CurrentControlSet\Services\VxD\Kaseya\FirewallSettingsFileRegQueryValue HKLM\System\CurrentControlSet\Services\VxD\Kaseya\FirewallSettingsFileRegCloseKey HKLM\System\CurrentControlSet\Services\VxD\Kaseya CreateFile C:\Program Files\Kaseya\Agent\KaseyaFW.iniQueryBasicInformation C:\Program Files\Kaseya\Agent\KaseyaFW.iniCloseFile C:\Program Files\Kaseya\Agent\KaseyaFW.ini
I believe that this is the reason that the "I/O Other" stat is quite high on all of our managed machines. I've confirmed that this is occuring with both our 5.1 agents and our 6.2 agents.
This leads me to a few questions:
1. Why is each registry key opened twice?2. Why is each registry value queried twice?3. Why can't the Configuration value and the FirewallSettingsFile value be queried at the same time (reducing the number of times you need to open the Kaseya key)?4. Can I reduce the frequency of these calls at all?5. Is there anything else that can be done to reduce this counter?
Or maybe these operations are not affecting system performance? If that is the case, I'll need to be able to explain why to the client as well.
Any insight into this would be greatly appreciated.
I'd be interested in knowing the answer to these questions too. Whenever our consultants are troubleshooting a performance issue, they invariably come to me querying the number of I/Os they see from AgentMon, and it'd be nice to throw a KB article at them and state "not a problem".
I'm working with Kaseya support to understand this issue.
Their initial response was asking if I was running two agents on the affected machine. They also mentioned that they would forward my issue to the dev team.
I responded by taking a fresh server, installing a K6.2 agent on it, and running process monitor to see what operations were being performed by AgentMon.exe (sending everything along to Kaseya support for review). Here's a screenshot:
So in less than 1 second, AgentMon.exe opened and queried the ConfigurationFile and FirewallSettingsFile registry value 8 times, plus a number of other operations in between.
To see what the "Other I/O" counter looks like for AgentMon.exe, I took another screenshot from the same machine:
This is showing that that AgentMon.exe is calling about as many other I/O commands as VMWare tools, so not so crazy, but it still is noticeably high. On some client machines, the I/O Other counter was far above the SQLServer process which is why one of my clients began asking questions.
Of course, this may be standard operating procedure, and it may have a negligible performance impact, but I'd like to know the actual reasons behind this behaviour so I can then intelligently inform my clients (especially the ones that are asking these questions).
I will update here as I hear back from Kaseya support.
If others could do a bit of research to see if they see the same results as me, that would be great. http://technet.microsoft.com/en-us/sysinternals/bb896645 <-- Process Monitor
Here's the latest update from Kaseya support:
"These are good questions and I will have to escalate this to agent dev to help answer your questions fully. Running this process monitor I have noticed the same behavior on my test machines as well."
Hopefully we can get some answers now.
Another update on this issue.
Kaseya support has sent this back to me:
"We have collaborated with Kaseya Engineering on this and a lot of this type of activity deals with fundamental changes in how the Kaseya Windows Agent is programmed. They are going to record these items and change request them in a future software release. As such, from here, would you reply back with the exact items you would like to see changed in the Kaseya Windows Agent dealing with this?"
I replied with the following:
Current Operation - occurs about twice a second
RegQueryKey HKLMRegOpenKey HKLM\SOFTWARE\Kaseya\Agent\GUIDGetSetInfoKey HKLM\SOFTWARE\Kaseya\Agent\GUIDRegQueryValue HKLM\SOFTWARE\Kaseay\Agent\GUID\ConfigurationFileRegQueryValue HKLM\SOFTWARE\Kaseay\Agent\GUID\ConfigurationFileRegCloseKey HKLM\SOFTWARE\Kaseya\Agent\GUIDCreateFile HKLM\SOFTWARE\Kaseya\Agent\GUID\KaseyaD.iniQueryBasicInfo HKLM\SOFTWARE\Kaseya\Agent\GUID\KaseyaD.iniCloseFile HKLM\SOFTWARE\Kaseya\Agent\GUID\KaseyaD.iniRegQueryKey HKLMRegOpenKey HKLM\SOFTWARE\Kaseya\Agent\GUIDGetSetInfoKey HKLM\SOFTWARE\Kaseya\Agent\GUIDRegQueryValue HKLM\SOFTWARE\Kaseay\Agent\GUID\FirewallSettingsFileRegQueryValue HKLM\SOFTWARE\Kaseay\Agent\GUID\FirewallSettingsFileRegCloseKey HKLM\SOFTWARE\Kaseya\Agent\GUIDCreateFile HKLM\SOFTWARE\Kaseya\Agent\GUID\KaseyaFW.iniQueryBasicInfo HKLM\SOFTWARE\Kaseya\Agent\GUID\KaseyaFW.iniCloseFile HKLM\SOFTWARE\Kaseya\Agent\GUID\KaseyaFW.ini
RegQueryKey HKLMRegOpenKey HKLM\SOFTWARE\Kaseya\Agent\GUIDGetSetInfoKey HKLM\SOFTWARE\Kaseya\Agent\GUIDRegQueryValue HKLM\SOFTWARE\Kaseay\Agent\GUID\ConfigurationFileRegQueryValue HKLM\SOFTWARE\Kaseay\Agent\GUID\FirewallSettingsFileRegCloseKey HKLM\SOFTWARE\Kaseya\Agent\GUIDCreateFile HKLM\SOFTWARE\Kaseya\Agent\GUID\KaseyaD.iniQueryBasicInfo HKLM\SOFTWARE\Kaseya\Agent\GUID\KaseyaD.iniCloseFile HKLM\SOFTWARE\Kaseya\Agent\GUID\KaseyaD.iniCreateFile HKLM\SOFTWARE\Kaseya\Agent\GUID\KaseyaFW.iniQueryBasicInfo HKLM\SOFTWARE\Kaseya\Agent\GUID\KaseyaFW.iniCloseFile HKLM\SOFTWARE\Kaseya\Agent\GUID\KaseyaFW.ini
Even if the proposed operations ran twice a second, that would reduce the operations per second from 36 to 24 a 33% reduction.
Even better would be to reduce the frequency of these calls. If these operations ran once every 10 seconds instead of twice a second, the average operations per second would be reduced from 36 to 1 or 2.
It's hard for me to tell you exactly what I'd like to see changed though because I don't understand and I've still yet to get answers to the following questions:
1. What is the purpose of these operations? Why are they called? What do they do? Are they required?
2. Why is each Registry Value queried twice?
RegQueryValue HKLM\SOFTWARE\Kaseay\Agent\GUID\ConfigurationFileRegQueryValue HKLM\SOFTWARE\Kaseay\Agent\GUID\ConfigurationFileRegQueryValue HKLM\SOFTWARE\Kaseay\Agent\GUID\FirewallSettingsFileRegQueryValue HKLM\SOFTWARE\Kaseay\Agent\GUID\FirewallSettingsFile
3. Why are these operations called so frequently?
4. Why is the registry key closed after querying each value? Why not just keep the registry key open until both values have been queried?
Here's the latest from Kaseya (their words highlighted in yellow, mine are plain):
1. What is the purpose of these operations? Why are they called? What do they do? Are they required?
These are part of the overall operations of the Kaseya Agent. They keep the operation of the Kaseya Agent in check and steady (connections, configurations, etc...). They are required.
This is the present method in which this operates (they are part of its routine activities). This is something that Kaseya Engineering especially would like to address in a future release (and subsequently in your feature request).
This is also the present method in which this operates. This is something that Kaseya Engineering especially would like to address in a future release (and subsequently in your feature request).
This is how the design of the Agent presently operates. This is what is being considered to be changed.
I have confirmed that all of these activities are changed more towards what you would like in the next version of the Kaseya Agent & VSA softwre. As such, would a feature/change request such as this for v6.2 suffice for you?:
FEATURE REQUEST for v6.2 VSA Software and the Kaseya Win Agent:
- Query the HKLM\SOFTWARE\Kaseya\Agent\GUID\ConfigurationFile once and/or as less as possible.
- Query the HKLM\SOFTWARE\Kaseya\Agent\GUID\FirewallSettingsFile once and/or as less as possible.
- Keep the registry key values open until both values have been queried.
Thoughts? I will set this ticket into an awaiting status from here as such.
I haven't responded yet, although I will soon. I feel like the answers are so vague and I still haven't really been given any hard information on what these operations actually do and why they operate the way they do. I've basically been told that this is the way it is, and log a change request if you don't like it.
I know that some Kaseya staff watch these forums, so would any of them be able to answer my questions with more specificity?
If not, then this thread will probably die here, and all of our Kaseya agents will continue to thrash away at the registry for eternity.
Q: Why does it happen?
A: Because it does.
I guess I can't really argue with the accuracy, but like you say some more detail would be nice. Thanks for sharing your response though, I'm interested to see how it all pans out.
It may be that it's more of a case of good practice than a real performance issue - but given that these agents run with high privileges on many important machines, removing any 'questionable' behaviour can only increase confidence in the agents and be a Good Thing(tm).
I'm still concerned with how I relate this answer I've been given to my client that is still looking for a solid answer.
I have a feeling that I won't be able to get away with a "Because it does" answer with them as our relationship is much closer and requires more substance in response than Kaseya's apparent relationship with it's user-base.
BTW - Feature request has been logged with Kaseya - CS092372