Jump to content

Some system builder nightmares and troubles


Recommended Posts

You might have read my MSI board thread and I cant hide that I love Asus for a reason and avoid others.. for the same reason. Reliability, stability, support, available software and other soft factors that may turn your day grey or sunny.

 

Well, here is a mixed cloud story, sun & grey same day I have to admit:

 

Board of choice ( for customer ): Asus X99-WS/IPMI - i7 5820k and some 4x4GB DDR4-2400 in a nice Chieftec tower.

 

I installed 2 850Pro's, defined Raid-1, installed WInSRV2008R2 without any errors or glitches.

It just went SMOOTH. Drivers installed throughtout the embedded functions, USB3, Intel Dual Network Adapters, some Asus GUI tools etc... the usual stuff. Nothing to complain.

 

Well, it started to become STRANGE after I decided to update the Bios for the board and later on the Firmware for the IPMI function.

 

IPMI, Base Board Manager, integrated Remote Access Module, Out-of-Band-Service Module..they carry many names but usually serve the same function. Service & monitor a server over a dedicated network interface with its own IP, have full Power Control for shitdown, reboot, coldstart, go to Bios with it and watch it booting from remote, capture BSODs for review etc...

very very handy and they usually cost as much as this board costs alone. A Dell iDRAC-8 goes for no less than 300€, tho it has some more features than this Asus ASBM8 module..but the Asus ASBM8 is good enough to handle most any trouble you could have and fix it...

 

...IF...

 

it wouldnt have tilted itself in a firmware flash SIGHHHHHHHHH

 

 

Shit happens, some firmware/Bios flashings just go wrong and if you do them daily you encounter one from time to time tho they get more seldom as systems advance and often you have a simple way back to a functioning Bios for example with USB-Flashback, that even works with no CPU and RAM put in.

 

Well, not so with the ASBM8, it prompts me with "Please Update Bios First" message"...I have it updated..stupid ASBM8 flash routine..look closely..I am running 3202..just flashed it...GRHH

 

I spend 1 and a half day booting that thing trying to convince the ASBM8 to accept another firmware..it wouldnt.

 

I edited the flash batch file..to no luck..same outcome.

 

I contacted Asus, which I have to say way no hassle. After less than 5 minutes I had a server techie on the phone who gave me 1st instructions...also no luck with that.

 

He connected me to specialists who wrote me an email same day, even called me but I missed the call ( Very very cool, at Dell this option is a 3k€ thing to have and called Premium Support, I have that for many servers so I know what that costs ).

He said, downgrade to 3101 ( it came with 3101 ) and flash again using FreeDOS and the Firmware on DL site.

 

Well, I did that too, all day long, no luck...the ASBM8 refuses to flash itself due to an too old Bios it says, which isnt the case.

 

The board takes 15min to bypass Bios routine until it boots a medium, so imagine how time consuming that was..and still is.

 

 

The good thing is that the board has a switch to turn OFF the ASBM8 so it boots as normal in a few seconds, just no ASBM8 function ( a reason I choose that board ).

 

 

I now finish installing the actual Server OS and applications and await feedback from Asus.

 

 

I cant say Asus CC is bad, they responded very fast, professional and polite. they dont try to put the error in my shoes as many low-level support crews try to do regularly.

 

 

I will try some more as I have the time, maybe I can convince the ASBM8 to accept the firmware finally. It still has a login page up, you just cant get in hehe..a door with nothing behind.

 

 

Asus says there is no way to kill the ASBM8, it MUST reflash itself. OK !! Then tell me HOW :)

 

 

I did a Dell T330 1-Socket CPU this week with iDRAC-8 Enterprise edition...almost same scenario

as dell redid their website and there was no firmware to be found near & far for it...

I took the phone a called them up in US of A and asked for help...the guy who helped me couldnt find it either..ehhhh..LOL....he finally found the FW on their internal repository servers as those get updated before the website does ( reason we found none ! ).

 

Jordan, the Dell guy, stayed on the phone for 2h+ until we remotely flashed the whole damn T330

from LifeCycleController image down to the Bios. Old Bios had a malfunction known by dell that kept many T330's from booting..it didnt recognize the Memory and needed somebody AT THE SITE to pull the 230Volt cables for 10 secs and then boot again. Nothing an iDRAC can do and if you service at night you cant call in and say " can you pull the plug for me for 10 sec ! PLEASE ?? .

 

 

Problems arise everywhere and everytime, the difference is how much help you get as a standard admin that just cant know it all.

 

From my experience, Dell and Asus wont let you down if you get stuck. Dell wants money for it but will stay with you on the phone or at site until solved and Asus has been very quick too, tho this incidence is still going on and I am afraid this will be the 2nd board this month that will go back to manufacturer for exchange..if the ASBM8 stays stubborn hehe.

 

 

I just hate to reattach the CPU Socket cover...if you do any misfit there it was YOU who killed the board..no matter what. That I want to avoid if I can fix it with software.

 

 

The Asus X99-WS/IPMI itself rocks.....compared to my MSI I gave back..LOL miles apart.

 

Hit the XMP switch and it goes XMP-2667MHz without raising any stupid Voltages sky high right away.

Clear Bios..no fuss, no fancy colours, very nice straight forward.

 

 

Need to slide another SSD in and make it y game rig over Xmas time as my Asus board is still in mail and I doubt I will get it before Xmas..SIGH :(

 

 

Bit

 

* the board killer* waahhaaaaa


Edited by BitMaster

Gigabyte Aorus X570S Master - Ryzen 5900X - Gskill 64GB 3200/CL14@3600/CL14 - Asus 1080ti EK-waterblock - 4x Samsung 980Pro 1TB - 1x Samsung 870 Evo 1TB - 1x SanDisc 120GB SSD - Heatkiller IV - MoRa3-360LT@9x120mm Noctua F12 - Corsair AXi-1200 - TiR5-Pro - Warthog Hotas - Saitek Combat Pedals - Asus PG278Q 27" QHD Gsync 144Hz - Corsair K70 RGB Pro - Win11 Pro/Linux - Phanteks Evolv-X 

Link to comment
Share on other sites

This is why I often don't miss my days as the office IT guy. Stuff happens sometimes and the folks I supported assumed that I could fix anything in a short time.

 

I once spent almost an hour, in front of an audience of about 50 customers, trying to get one of our demo machines to behave. It is even less fun when you are wearing a suit and tie and trying to look professional.

 

It can always be worse, though. In 1988-90, I worked at the United Press International (some of you might remember the UPI tag on news stories) main computer center. When the winter weather got nasty I would have to go up on the roof and knock ice off of the satellite dishes and the only time the DEC folks were allowed to touch our machines was after 2:00am and before 5:00am.

 

When there was a bad storm over Guam, I would get calls from half of Asia asking me why the data feeds were down. In those days most of our data moved over 56k leased lines and when three different telecom companies are involved I could spend hours on the phone just trying to figure out where, or which company, had the problem.

 

Nothing I have ever seen, or will be likely to see again, can top the server rack that was returned with a hole in the bottom caused by a forklift blade, and lots of other damage. It took more time to pry the main chassis out of the rack than it did to get the machine running again. It was a real shame too. This was a machine with three Xeons, 2GB of memory (in 1995 that was incredible) nine hot swap hard drives in a RAID setup, and three hot swap power supplies. It was intended to continue to function even if the building around it was destroyed. It was also all black, which, to me, made it extra awesome.

 

The only good part was this was a recovery for a demo machine that one of our salesmen loaned out. One of my jobs, at the time, was recovering this equipment. When this server came home, it was mine, sort of. I named it Monolith (Space Odyssey) and ran some amazing database stuff from it. It was the size of a telephone booth, if any of you remember those and was nothing short of awesome for it's time.

 

I really love being retired and only worrying about my own machines, and my brother's, and the neighbors, and anyone else I know that realizes that I actually know what is inside of the magic box. :D


Edited by cichlidfan

ASUS ROG Maximus VIII Hero, i7-6700K, Noctua NH-D14 Cooler, Crucial 32GB DDR4 2133, Samsung 950 Pro NVMe 256GB, Samsung EVO 250GB & 500GB SSD, 2TB Caviar Black, Zotac GTX 1080 AMP! Extreme 8GB, Corsair HX1000i, Phillips BDM4065UC 40" 4k monitor, VX2258 TouchScreen, TIR 5 w/ProClip, TM Warthog, VKB Gladiator Pro, Saitek X56, et. al., MFG Crosswind Pedals #1199, VolairSim Pit, Rift CV1 :thumbup:

Link to comment
Share on other sites

"I just hate to reattach the CPU Socket cover...if you do any misfit there it was YOU who killed the board..no matter what." Boy can I identify with that. Had an issue with Newegg about an ASUS board and when I sent the board back to the "point of purchse" of course they found issues with the CPU Socket and refused an exchange. I contacted ASUS, sent the board to them and THEY replaced it. Have not done business with Newegg since, but am a dedicated user of ASUS MB's.

Intel i5-4690K Devil's Canyon, GForce TitanX, ASUS Z-97A MB, 16GB GDDR3 GSkill mem, Samsung SSD X3,Track IR, TM Warthog, MFG Crosswind pedals, Acer XB280HK monitor,GAMETRIX KW-908 JETSEAT

Link to comment
Share on other sites

I can't argue with that. I have three ASUS laptops (one is dead in the closet), and ASUS MBs in every desktop machine that is currently running.,

ASUS ROG Maximus VIII Hero, i7-6700K, Noctua NH-D14 Cooler, Crucial 32GB DDR4 2133, Samsung 950 Pro NVMe 256GB, Samsung EVO 250GB & 500GB SSD, 2TB Caviar Black, Zotac GTX 1080 AMP! Extreme 8GB, Corsair HX1000i, Phillips BDM4065UC 40" 4k monitor, VX2258 TouchScreen, TIR 5 w/ProClip, TM Warthog, VKB Gladiator Pro, Saitek X56, et. al., MFG Crosswind Pedals #1199, VolairSim Pit, Rift CV1 :thumbup:

Link to comment
Share on other sites

cichlidfan, I hear ya :)

 

Most of my money I dont make with building PC's anymore, that time is long gone, for my sake !

 

I put this server together for a long term customer and friend as well, I wouldnt do it any other way anymore. I only build anything for people I know well and they know the value of what I do.

 

As you never know what comes along until you have done it till the end I have come across a myriad of problems with devices most wouldnt know what they are for, you would know, but many would just go ehhhhh???.

 

Most of the time I am called for troubleshooting Internet connections, same as you did with the dish up on the roof. I dont have to go on a roof but need to check where the problem originates from, pinpoint it and solve it, no matter what. Time is more critical than money so I have the benefit of support contracts and people I can call once I traced it down. Dell and Gateprotect ( rohde & schwarz owned meanwhile ) are my usual phonenumbers I dial. Asus just added itself to my list as-of-today LoL.

 

Things like broken switches in the streetbox of your static IP line, malfunctioning routers, broken cables, broken DSL-modems, broken NIC's..most of the stuff is network related.

 

As windows got more reliable since Win95 with every step they made the workload shifted towards

configuring and managing, monitoring and readjusting. Troubleshooting BSODs is very very seldom these days, and it used to be one of the main things to fight 10-15 years back..LoL..you remember "You have moved your mouse, please reboot to make the change effective !" era ??

 

Ransomware protection from the entrance point down to the deepest Backup you have is high priority these days...and it aint easy, believe me. It could be easy..IF...and that voids it all.

 

Problems have shifted considerably in the prof. IT segment.

 

This stupid ASBM8 is one of the "old style" problems. Can easily be fixed with swopping the board.

 

A packet loss/discard in NW-America on its way to Wells Fargo Bank can take 3 month to fix..until you find the idiot who doesnt properly maintain his DNS...had that once ! Was the hardest ever to fix and shut most SW-German users off using that Bank ( mostly military people use that here, germans dont, how lucky they were ! ).

 

I am waiting on Asus call back...so I just keep on typing some old stories for your enjoyment.

 

 

I had a server installed at a Newspaper publisher, SuSE as usual at that time. Linux Kernel Raid driven with ReiserFS and multiple disks..many of them...it ran like a charm.

One day one employee had an argue with that company I serviced and while I was at site with my apprentice my SMB file share dropped of my screen...aka one folder was unavailable suddenly.

Usually this just DONT happen !! I clicked again, click click click...NADA...

So I told my young friend to go next door and look at the diodes if the server was talking to us....

...he went next door and SCREAMED....."THE SERVER IS MISSING 2 DRIVES...SOMEBODY MUST'VE RIPPED THEM OUT...BOSS"..

 

I couldnt believe my ears...went next door and immedeatly told him to NOT TOUCH ANYTHING...when I saw he was right....this has become a crime scene now !

 

With newspapers and contracts...time is crucial, time is everything. If you cant send to press at that time window...you will not get 100k copies next morning 3am to load on trucks and deliver.

You miss that window..you are DEAD.

 

The good thing was, that guy stole the wrong harddrives !!! Hehe, he stole the ones with old graphics folder any my "utilities" folder with drivers and tools.

 

That beast of server, Linux Raid and ReiserFS kept running strong, the paper was able to processed and we were able to move all the data and functions on to my ASUS laptop, including a PCMCIA ISDN card for HylaFAX. All done is less than a few hours ( I had SuSE on my laptop, lucky that was, compiled Hylafax for PCMCIA, moved the Samba config over and the most critical data )

and by the time the Police showed up the Laptop had taken over seemlessly the core functions and crucial data.

 

The Police had to take the whole damn thing to the lab for fingerprints. OMG !!!

2 weeks later I was called in to put the returned server back into operation, new drives etc at hand. Well, when I saw the beast..I said NOOOOO WHAT HAVE YA DONE TO MY BABY ???!!!"

 

The whole damn thing was covered in BLACK FORENSIC FINGER PRINT POWDER, all over, just everywhere. It took me 2 days to clean that thing so I dared to hit the power button !

It all went fine, the server went back into production and they guy in question, mil personal btw, had a nice interrogation by german police and military police...not so nice I guess.

The drives stolen remained stolen, they couldnt charge him despite we all knew he did it, all others had a meeting and only me and my apprentice were in the floor...and he had a key and time...the apple never falls far from the tree we say, if you can proof it or not.

 

That taught me many things:

 

LOCK your server away, have entrance control: Some employees leave without a smile and want to hurt you.

 

Use RAID-5 or better Raid-6 if possible !

 

Use Linux if possible ( windows would have crashed when the drives were pulled )

 

Have a 2nd server that can take over in an hour, usually I size Backup Servers so that in WCS they can mimic the big daddy server until problem solved.

 

Expect the worst and you will still have suprises.

 

 

Asus just called....the lady said she saw my number and wanted to make sure I am taken care of !

How nice !

 

Still waiting on the tech call..or RMA number ( NOOOOO ! )


Edited by BitMaster

Gigabyte Aorus X570S Master - Ryzen 5900X - Gskill 64GB 3200/CL14@3600/CL14 - Asus 1080ti EK-waterblock - 4x Samsung 980Pro 1TB - 1x Samsung 870 Evo 1TB - 1x SanDisc 120GB SSD - Heatkiller IV - MoRa3-360LT@9x120mm Noctua F12 - Corsair AXi-1200 - TiR5-Pro - Warthog Hotas - Saitek Combat Pedals - Asus PG278Q 27" QHD Gsync 144Hz - Corsair K70 RGB Pro - Win11 Pro/Linux - Phanteks Evolv-X 

Link to comment
Share on other sites

Some tips for Ransomware protection:

 

DEATTACH your Backup drive after having done a backup

 

REATTACH only when you do a backup, then see above

 

This sounds simple but will save your behind.

 

 

If you backup over a network, DONT use file shares which are permanently mounted in your OS.

The Ransomware will look for it, find it & destroy it, as it will with ANY other mounted share or disk.

If you use network to move backups, use ssh tunnel in your tool/script and ONLY let it establish to move the data, cut it afterwards so a Ransomware wont find and destroy your backup.

 

USE CLOUD SERVICE if applicable and affordable for your extra Backup, stored safely somewhere else deep down in a swiss mountain :)

 

 

WATCH what you click in emails. I had 2 other incidents meanwhile with companies that work with a client I have. my client had 45k+ attacks with ransomware and it held tight, the other two are DESTROYED...from clients to servers----ALL ENCRYPTED...NO JOKE..THIS IS REAL

 

REAL DAMAGE REAL FRUSTRATION

 

NOTHING YOU CAN DO ONCE IT HAPPENED !!!!! PREPARE WHILE YOU CAN

 

 

If you have websites with emails up, expect TAILORED attacks that will be VERY HARD to tell apart from an valid email...so bad it got meanwhile.

 

Those attacks are very clever, precise and mimic real email like no other I have ever seen.

 

 

be safe


Edited by BitMaster

Gigabyte Aorus X570S Master - Ryzen 5900X - Gskill 64GB 3200/CL14@3600/CL14 - Asus 1080ti EK-waterblock - 4x Samsung 980Pro 1TB - 1x Samsung 870 Evo 1TB - 1x SanDisc 120GB SSD - Heatkiller IV - MoRa3-360LT@9x120mm Noctua F12 - Corsair AXi-1200 - TiR5-Pro - Warthog Hotas - Saitek Combat Pedals - Asus PG278Q 27" QHD Gsync 144Hz - Corsair K70 RGB Pro - Win11 Pro/Linux - Phanteks Evolv-X 

Link to comment
Share on other sites

cichlidfan, I hear ya :)

 

:)

ASUS ROG Maximus VIII Hero, i7-6700K, Noctua NH-D14 Cooler, Crucial 32GB DDR4 2133, Samsung 950 Pro NVMe 256GB, Samsung EVO 250GB & 500GB SSD, 2TB Caviar Black, Zotac GTX 1080 AMP! Extreme 8GB, Corsair HX1000i, Phillips BDM4065UC 40" 4k monitor, VX2258 TouchScreen, TIR 5 w/ProClip, TM Warthog, VKB Gladiator Pro, Saitek X56, et. al., MFG Crosswind Pedals #1199, VolairSim Pit, Rift CV1 :thumbup:

Link to comment
Share on other sites

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...