This is a long shot, but any chance it's a power issue. In particular, a UPS compatibility issue with newer servers that have "green" power supplies that use Power Factor Correcting technology. The fact that you saw some log entries related to RAID seems to rule out a power issue, but just thought I'd mention it...

Legacy Forum Name: KES,
Legacy Posted By Username: ReedMikel

ReedMikel

This is a long shot, but any chance it's a power issue. In particular, a UPS compatibility issue with newer servers that have "green" power supplies that use Power Factor Correcting technology. The fact that you saw some log entries related to RAID seems to rule out a power issue, but just thought I'd mention it...

I have been following the other post on the power issues. In this case, I am certain that power is not the cause of the issue. They have APC SmartUPS 1500VA units.

Legacy Forum Name: KES,
Legacy Posted By Username: lwolf

Very odd. Want kind of events are logged from the raid cards when the servers lock up? Was anything else running at the same time (audit, backup, VSS, kes scan, etc)?

Have you tried uninstalling KES temporarily?

Legacy Forum Name: KES,
Legacy Posted By Username: Coldfirex

Thansk for the reply. here are the answers to your questions.

Also let me note one fact change. I had originally reported that one server had a PERD5 RAID controller and the other had a PERC6 controller. Both actually have PERC6 controllers.

And to re-state some of the other hardware facts (unchanged), they are both DELL PowerEdge 2900 servers - one is 13 months old, one is 11 months old. Bother are running Windows Server 21008 64-bit, one is an SBS2008 server, and one is a remote office member server. They are for different clients, at totally different offices.

Coldfirex

Want kind of events are logged from the raid cards when the servers lock up?

The events are in the Windows System log, from the DELL Server Administrator software, Source: Server Administrator, Event ID: 2336, include descriptions like the following:

Controller event log: Fatal firmware error: Driver detected possible FW hang, halting FW.
: Controller 1 (PERC 6/i Integrated)

Controller event log: Fatal firmware error: Line 3622 in ../../raid/1078dma.c
: Controller 1 (PERC 6/i Integrated)

And of course, as a result of the lockups, there is always the dreded event 6008 The previous system shutdown was unexpected.

Coldfirex

Was anything else running at the same time (audit, backup, VSS, kes scan, etc)?

The events occur randomly at various throughout the day or night - sometimes 2-3 times in one day, somethimes it will go problem free for 3-4 days. Nothing else seems to be running at the exact time - either from Kaseya, or other software in the local machine.

Although I suppose it is possibel that the local AVG software may have been doing somethign, outside of the integration with Kaseya. I am not exactly sure how to tell that.

Coldfirex

Have you tried uninstalling KES temporarily?

We worked with DELL on both servers (one with PERC5 card, and one with PERC6 card). So far, over 2-3 weeks, we have done the following: We updated the RAID drivers, we updated the RAID firmware, we applied a RAID hotfix from DELL, and then we even replaced both RAID cards. The problem continues to randomly occured.

Mid-week last week, we unisntalled KES from the servers. So far, both servers have been problem free, although it has only been a few days. If we go a whole week without the problem,

It could be unrelated to to KES/AVG, but I am leaning towards KES/AVG being the cause at this point, as that was the only major change (other than basic Windows patching) that was made to both servers, prior to the problem starting.

Legacy Forum Name: KES,
Legacy Posted By Username: lwolf

Uuuugggghhhh. We just had a crash of a 3rd server. The server is a DELL PowerEdge 2900 with a PERC6 card, running Windows Server 2008 32-bit (the others were 64-bit). It is a branch office server (and domain controller).

The server had been running Symantec Endpoint Protection. Last week, Symantec was uninstalled and AVG was installed on all computers on their network.

Event log entries match the errors on the other servers also experiencing the problem.
===================================
Controller event log: Fatal firmware error: Driver detected possible FW hang, halting FW.: Controller 0 (PERC 6/i Adapter)

Controller event log: Fatal firmware error: Line 3622 in ../../raid/1078dma.c: Controller 0 (PERC 6/i Adapter)
===================================

The server HAD been fine since it was setup months ago. The only change made in the last week was Symantec uninstall and AVG install, and now we are having crashes, just like the other two servers that started crashing after AVG was installed.

I am now even more confident that AVG is the source of the problem. I just opened Ticket ID: 209533 with K support, although, it will be hard to troubelshoot now that we have uninstalled AVG from the servers.

Legacy Forum Name: KES,
Legacy Posted By Username: lwolf

I am glad I do not install AVG (or any AVAS) on servers. It's too risky IMO, as AV programs and definitions change hourly sometimes. That's too much room for bugs... And we know how few bugs AVG manufactures

Legacy Forum Name: KES,
Legacy Posted By Username: ReedMikel

Allw e use is Dell servers as well with Perc5 and Perc6 cards as well. We havn't had any issues. We use KES, but i don't install any of the additional componenets like Link Scanner, Web shield, ID Protection. Do you use any of the componenets?

Legacy Forum Name: KES,
Legacy Posted By Username: scootrz32

scootrz32

we use is Dell servers as well with Perc5 and Perc6 cards as well. We havn't had any issues. We use KES, but i don't install any of the additional componenets like Link Scanner, Web shield, ID Protection. Do you use any of the componenets?

Good question. Sorry, I should have stated this before. We do NOT install Web shield, or Email Scanner, or ID Protection, or Firewall, or Exchange Server plug in.

We have been installing Link Scanner - both Active safe search and Search Shield - which I beleive only affects web surfing/browsing. So I did not suspect those components to be the cause of the random lockup problems - as the lockups occur when no one is workign on the server or surfing.

Legacy Forum Name: KES,
Legacy Posted By Username: lwolf

Same here we use primarly all dell servers with Perc6 cards all over with no issues. We don't use any of the "extra's" as well and would recommend not using link scanner to see if that helps....

We don't allow Web Surfing on server's anymore as this isn't the machine we need to be pulling down files from the internet on....

Hope that helps...

Joe Axne
IT-Guru, LLC

Legacy Forum Name: KES,
Legacy Posted By Username: itgurullc

I have not yet got a response from K support to my ticket. Rather frustrating.

I was just looking at some Views in by VSA, looking at some stats:
- I found that we have KES installed on 47 servers running Windows Server 2003 - with 22 of them being AVG9, and the balance being AVG8.5 (too chicken to upgrade at this time)
- I found that we have KES installed on 23 servers running Windows Server 2008 - with 8 of them being AVG9, and the balance being AVG8.5 (again, too chicken to upgrade at this time)
- Additionally,there were the 3 servers running AVG9 that we uninstalled, due to the crashes.
- Almost all of the above servers are DELL PowerEdge 2800's and 2900's.

Based on the above stats, part of me says that the PERC6 errors and lockups cannot be because of AVG9, because there are 8 remaining DELL PowerEdge 2800/2900 servers running AVG9 and Windows Server 2008.

But on the other hand, I have 3 separate servers, at 3 separate offices, in 3 separate domains, that had a solid record of running fine. Each had AVG installed, one at a time, over a period of several weeks. Each server started to experience the lockup/crash problem within a week of installing KES/AVG9, and each server stopped having the problem after uninstalling KES/AVG9.

I suppose I could wait an additional week or two to confirm that the problem stays away, and then reinstall and see if the problem returns - but I am too scared to reinstall at this point.

Legacy Forum Name: KES,
Legacy Posted By Username: lwolf

Long shot, but is it possible there are remnants of the old AV install?

Legacy Forum Name: KES,
Legacy Posted By Username: Coldfirex

Coldfirex

Long shot, but is it possible there are remnants of the old AV install?

That did cross my mind, although I was not sure how to tell or determine or troubelshoot that possibility.

Legacy Forum Name: KES,
Legacy Posted By Username: lwolf

OK, I may have spoke too soon as we just saw a Dell 2900 reboot in the middle of the day today for no reason at all. So something similar is going on here as there are no event logs prior to or after indicating anythign out of the norm so this has got me very concerned.

The real last change to the server was an KES Update at 12:34pm and the crash occured at 3:15pm.

I may have spoke too soo on my previous post about not seeing any issues.

Have to ask, how did you lead to AVG possibly causing the issue?

But the most important question, how do we prevent this from occuring!!! Do we need to esclude specific Dell hardware files or drivers?

Regards,

Joe Axne
IT-Guru, LLC

Legacy Forum Name: KES,
Legacy Posted By Username: itgurullc

I said reboot, however it was a hard crash 6008 in event log. And I see that you actually uninstalled and found things ran better. If we see another issue with this server, I woudl have to agree it will be time to uninstall AVG off of this server ASAP as this is a main database for a client of mine and this caused big time work stoppage.

I will escalate to Kaseya IT Services NOC and see if they have any insight into this and how to resolve as well. Will advise what I find.

Joe Axne
IT-Guru, LLC

Legacy Forum Name: KES,
Legacy Posted By Username: itgurullc