Kaseya Community

PERC RAID card issues with DELL PowerEdge 2900 servers with AVG installed

  • itgurullc
    I said reboot, however it was a hard crash 6008 in event log. And I see that you actually uninstalled and found things ran better. If we see another issue with this server, I would have to agree it will be time to uninstall AVG off of this server ASAP as this is a main database for a client of mine and this caused big time work stoppage.



    Do you have the DELL Server Administrator software installed? In our case, we did have it installed, the following events were in the Windows System log, from the DELL Server Administrator software, Source: Server Administrator, Event ID: 2336, include descriptions like the following:

    Controller event log: Fatal firmware error: Driver detected possible FW hang, halting FW.
    : Controller 1 (PERC 6/i Integrated)

    Controller event log: Fatal firmware error: Line 3622 in ../../raid/1078dma.c
    : Controller 1 (PERC 6/i Integrated)

    Followed by the Event ID 6008 for the unexpected restart.



    itgurullc
    Have to ask, how did you lead to AVG possibly causing the issue?

    As stated in the prior posts, uninstalling Symantec and installing AVG was the only major shange to all 3 servers that started experiencing the PERC RAID errors and server crashes. We worked with DELL to update RAID drivers and RAID firmware, and then even rplaced 2 of the RAID cards, but the problem continued to randomly occur - so I don;t see how it could possibly be the same hardware issue, occuring on 3 totally different servers.


    itgurullc
    I will escalate to Kaseya IT Services NOC and see if they have any insight into this and how to resolve as well. Will advise what I find.


    You may want to also mention this thread, and refer them to my open support ticket ID 209533. I opened it 2.5 days ago, and sadly, so far all I received was "Our Specialist is investigating your issue who would update you shortly."

    Legacy Forum Name: KES,
    Legacy Posted By Username: lwolf
  • I talked with someone at K Support today. They are going to conatct AVG. We'll wee what happens.

    Legacy Forum Name: KES,
    Legacy Posted By Username: lwolf
  • This is an interesting issue. For starters, there is nothing at all relating to that firmware error. Anywhere. So far, I can only find one other person (Warning...Experts Exchange link...last comment..bottom of the page) that has had that error. That was 4 months ago. It is possible that he had AVG 9 on there, as the business version was released right around then (Free version ~4 days after that post). You will notice that the guy was experiencing the server locking up and that the post was abandoned before he posted his resolution. Maybe someone wants to contact him to ask if that server had AVG 9 installed on it? His website is www.lakeviewcomputer.com

    There has to be some other variable (combination of things?) that is missing from this. If AVG 9 alone was causing Dell servers with the PERC raid cards to crash, thousands of angry admins would be cursing the death of AVG all over the place.

    I wish I had something to add that would help the problem, but to be honest...I got nothing. ConfusedConfusedConfused The only thing I can think of is that there is something that is common among the servers that are crashing. Powersupply, ram, PERC card mfg. date, etc.

    Good luck.

    Legacy Forum Name: KES,
    Legacy Posted By Username: CeruleanBlue
  • CeruleanBlue,

    Thanks for the reply and comments. I checked out the Experrts-Exchange article that you mentioned.

    Like you said, and as I had posted earlier, I too have AVG8.5 and AVG9 installed on a bunch of DELL PowerEdge serves with PERC5 and PERC6 cards. So if AVG was a 100% direct conflict with DELL servers or PERC cards, all of my other servers, along with everyone else's servers, would be crashing.

    But... can report to everyone that all 3 of these servers have now been probelm free for more than a week now, after uninstalling KES/AVG9. So the problem must be, at least somehow, somehow related to AVG.

    So to summarize everything, I have: 3 totally separate servers, all being DELL PowerEdge 2900 models; all running Windows Server 2008; 2 of them being 64-bit and one of them being 32bit; at 3 totally different locations; on 3 totally different domains; all were all running just fine for many 6-12 months (since original setups); all which all started to experience the same random crash/lockup problem with PERC errors in the event long within a few days of installing AVG9; which we worked with DELL Tech Support on 2 of the 3 servers to update RAID drivers, update RAID firmware, and even replace the RAID cards, yet the crash/lockup problem continued; and then all 3 servers stopped having the crash/lockup problem once KES/AVG9 was uninstalled.

    So I feel pretty strongly that the problem must be somehow related to AVG. But, not that I have uninstalled AVG, I suppose it is going to be difficult to troubleshoot any further.

    Legacy Forum Name: KES,
    Legacy Posted By Username: lwolf
  • We've had similar issues on some servers, however ours have been HPs, not Dells. I believe we've had to remove AVG from at least 5-6 servers now due to random hanging. We put Avast's free trial on and every single one of the servers stopped locking up as soon as AVG was gone. On a couple of the servers we spent a few weeks going back and forth with Kaseya and AVG's tech support, but didn't find a resolution, and it reached a point where we were risking losing customers. I'm anxiously awaiting KES 3.0 so we have another option aside from AVG, because I just don't trust it anymore.

    Legacy Forum Name: KES,
    Legacy Posted By Username: kcears
  • lwolf
    CeruleanBlue,

    So to summarize everything, I have: 3 totally separate servers, all being DELL PowerEdge 2900 models; all running Windows Server 2008; 2 of them being 64-bit and one of them being 32bit; at 3 totally different locations; on 3 totally different domains; all were all running just fine for many 6-12 months (since original setups); all which all started to experience the same random crash/lockup problem with PERC errors in the event long within a few days of installing AVG9; which we worked with DELL Tech Support on 2 of the 3 servers to update RAID drivers, update RAID firmware, and even replace the RAID cards, yet the crash/lockup problem continued; and then all 3 servers stopped having the crash/lockup problem once KES/AVG9 was uninstalled.

    So I feel pretty strongly that the problem must be somehow related to AVG. But, not that I have uninstalled AVG, I suppose it is going to be difficult to troubleshoot any further.


    I have seen the same thing on 10-15 Dell servers with Windows Server 2008, we are since uninstalled AVG from those machines and no longer experience any lock-ups.

    Legacy Forum Name: KES,
    Legacy Posted By Username: cnwicsurrett
  • I still don't understand why anyone puts AV software on a server. In my experience, there were no gains from it and there were LOT's of problems from it. The Exchange plug-in is one thing, but who the heck thinks they're getting more "protection" from having active file scanning on a server?

    Legacy Forum Name: KES,
    Legacy Posted By Username: mastermindpro
  • I can't wait to ditch AVG too. AVG's link scanner has been giving me a lot of false positives in the last few days. I really have 0 confidence in AVG these days Sad

    kcears
    I'm anxiously awaiting KES 3.0 so we have another option aside from AVG, because I just don't trust it anymore.


    Legacy Forum Name: KES,
    Legacy Posted By Username: ReedMikel
  • cnwicsurrett
    I have seen the same thing on 10-15 Dell servers with Windows Server 2008, we are since uninstalled AVG from those machines and no longer experience any lock-ups.


    cnwicsurrett
    We've had similar issues on some servers, however ours have been HPs, not Dells. I believe we've had to remove AVG from at least 5-6 servers now due to random hanging. We put Avast's free trial on and every single one of the servers stopped locking up as soon as AVG was gone. On a couple of the servers we spent a few weeks going back and forth with Kaseya and AVG's tech support, but didn't find a resolution, and it reached a point where we were risking losing customers. I'm anxiously awaiting KES 3.0 so we have another option aside from AVG, because I just don't trust it anymore.


    kcears and cnwicsurrett ,

    Wow, I am so glad to hear that I am not alone here (although sorry to hear that you have had to go thru what I am going thru). K Support told me, that AVG Support told them, that they have no knowledge of such problems with AVG.

    Do either of you happen to have any Kaseya ticket #'s that I could reference when I am working with K support on my ticket - to help prove that others have also had issues, and I am not alone?

    Legacy Forum Name: KES,
    Legacy Posted By Username: lwolf
  • Do they create dumps in the C:\windows\minidump folder when it reboots? Maybe that will provide some insight.

    Legacy Forum Name: KES,
    Legacy Posted By Username: scootrz32
  • scootrz32
    Do they create dumps in the C:\windows\minidump folder when it reboots? Maybe that will provide some insight.


    I checked, there is no such folder. Would it be somewhere else for Windows Server 2008 64-bit?

    Legacy Forum Name: KES,
    Legacy Posted By Username: lwolf
  • I am somewhat scared, but for troubelshooting purposes in workign with K Support and AVG Support, I have reinstalled KES/AVG9 on one of my 3 problem servers. We'll see what happens.

    Legacy Forum Name: KES,
    Legacy Posted By Username: lwolf
  • Same issue here -- when upgraded to AVG 9 - two server froze on reboot -- I have two 2950 with PERC5i and PERC6i cards. Worked with Dell -- changed PERC card -- only fix was to remove AVG 9 ..and now no issue. Same errors you got. AT time i could boot into windows but most of the time it just froze at Applying Computer Settings. I sent email to support for fix.. i needed to move to another AV product on the servers until this is fixed.

    Legacy Forum Name: KES,
    Legacy Posted By Username: Light Speed
  • Light Speed
    Same issue here -- when upgraded to AVG 9 - two server froze on reboot -- I have two 2950 with PERC5i and PERC6i cards. Worked with Dell -- changed PERC card -- only fix was to remove AVG 9 ..and now no issue. Same errors you got. AT time i could boot into windows but most of the time it just froze at Applying Computer Settings. I sent email to support for fix.. i needed to move to another AV product on the servers until this is fixed.


    Light Speed and all -> Please post a ticket to ksupport with this issue for the two servers, so they know the problem is more than just the few servers they already know about (so far they only have 2 tickets, mine and someone elses).

    All -> The server crashed on Sunday, on whcih I had reinstalled AVG on Friday. It had been problemf ree for over a week, since uninstalling AVG. So I feel very confident that AVG is definately related to (somehow) causing the crashes.

    All -> I am actively working with K Support on my issue. They were very responsive today. We ran some AVG Gather programs and sent the files to AVG Support, along with log files and MSINFO.

    Legacy Forum Name: KES,
    Legacy Posted By Username: lwolf
  • We have a dell 2900/perc6 windows 2003 64bit with SQL2005. No problems with AVG 9. Could it be a 2008 driver issue/conflict ?Confused

    Legacy Forum Name: KES,
    Legacy Posted By Username: southshore