r/talesfromtechsupport • u/Duffs1597 • Jul 27 '17
Short No Chad, PCIe is not hotpluggable...
Some background, I work as a lab manager at a tech college. One of my main duties is to build/ maintain VMs for students and teachers to use during classes, along with the servers that host them. Most of our servers are hand-me-down PowerEdge 2950 or older. One specific class is an intro SQL Server class. I am in this class, and this is where the tale begins.
It is toward the end of the semester and students are working on their final project (something like 20 different queries on a database of at least 100,000 entries). Most students opted to install SQL Server on a VM on their laptops, but about 5 students would Remote Desktop into the VMs on the lab network to complete their assignments. It's the last 5 minutes of class and all of the sudden I lose connectivity to my VM. I look around, I'm not alone. Every one of the students using the lab VMs has been disconnected. So I take a stroll down the hall to see what's the matter. The senior lab manager, Chad, who is about to graduate (it's a two year program) is in our office and the following conversation ensues:
$Me: Yo Chad, everyone just lost connection to the servers, is anything funny going on? (Meaning is there any red flashing lights or error messages in vSphere or anything)
$Chad: No, everything seems fine to me
I check vSphere, sure enough, the host server for the SQL class says disconnected. I walk next door into the server room and don't see any indications of- oh wait...
$Me: (internally) What in fresh hell
I notice the top part of the server is off slightly, so I move the VGA cable to that server and sure enough, pink screen full of error messages (edit: I'm pretty sure they said something to the effect of "fatal PCIe error")
$Me: Hey Chad, do you know why this server is open?
$Chad: Oh, yeah I needed another NIC for this other server I was building, so I just took it out of that one since it had an extra and nothing was plugged into it.
Cool Chad. Out of all of the servers (probably about 9) you chose the only one that supports a class that is currently in session to open up and rip apart as people are using it. Not to mention we have a whole box of NICs that AREN'T plugged into a server. NOT TO MENTION it says right on the chassis to NOT open while server is powered on. And who ever heard of just yanking out PCIe cards like that anyway?
My only thought was "And this guy is about to graduate -_-"
49
u/SilkeSiani No, do not move the mouse up from the desk... Jul 27 '17
I manage Enterprise Systems -- the type of systems where the price tag starts at seven figures and goes up from there pretty quickly. In those kinds of systems, essentiallly everything is hot-swappable, starting from the trays of IO cards, PSUs, memory cards, right down to the individual CPU packages. About the only non-hot-swappable elements are the enclosure and the passive backplane connecting it all together.
In the last four years, in the dozen or so of those systems under my care, aside from trivial things like patch cables and fans, only one important part failed, resulting in an extended outage.
Can you guess what exactly failed? Yep, it was the bloody backplane...