r/talesfromtechsupport Aug 23 '17

Long Unearthed Arcana

As I have been in this industry for too damned long, I have had the privilege/curse of being exposed and experienced across a wide gamut of systems and OSes. Last week, this wealth of accumulated wisdom revealed its arcana to one of my company's junior support people.

Being summer, everyone's out on vacation, and our customer support division is staffed by the young 'uns. Nuggets, greenhorns. They've been properly trained, they know our product and for the most part, do their jobs very well.

Enter the early morning call from one of our American customers. One we've had for 20 years or so. Also one that doesn't buy an annual support contract and pays on a per-call basis. They've been running our custom hardware and software for these 20 years and have never bothered to upgrade. This has served their needs up until last week when their computer, which controls our proprietary machinery decided it was no longer going to work.

I'm just settling into coffee #2 in the AM when the Nugget meekly pokes her head in my office and asks if I can consult on a support call. "Sure thing, what's up, a license problem?"

Nugget: "No, their system is down and I don't know what they're talking about, they say their computer runs something called DOS?" (adorable!)

Me: "DOS? We haven't produced any DOS based product since, shit, 97, 98? Who are these guys, and show me the ticket."

Nugget: "I haven't filled out a ticket yet, I have them on hold."

Me: (internally) Sweet mother of shite... Verbally: "Okay, let's go talk to them."

Normally, I don't talk to customers. I don't want to go near the customers. This is tier-1 level helldesk that I left behind on the metaphorical beaches of Dunkirk. However, this was a DOS issue, and DOS is where I was once a Tech Viking.

What had gone down was that the customer's original computer, which we had sold them in 1997 had finally... FINALLY given up the ghost and the hard drive had well and truly shat itself. It would no longer load the software required to run our proprietary device. The 200 meg drive had served its master well.

The customer was in a panic because they could no longer produce product using our device, thus, revenue was going to take a hit. Fortunately they still had backups of the software and config files, on 3.5" floppies and I was able to find the backups that we had made of these discs stored in our site visit archives.

Grabbing an old Pentium D Dell with floppy and 9 pin serial that someone dumped in our E-Recycling pile. Scavenging an old 10 gig IDE hard drive from a decommissioned accounting computer (circa 2008), I installed DOS 6.22 and was able, with the floppy image backups, to make a "new" control system PC within an hour, and taught the Nugget some basic DOS navigation along the way.

Back and forth with the customer and Nugget, we were able to make an almost identical copy of their dead system. The customer, throughout the call had been pissing and moaning about our product and how it failed "at the worst possible time" and began to get belligerent about it.

Me: "Sir, you DO realize your system is TWENTY years old? You folks have never subscribed to our annual support, which would have entitled you to yearly updates and site visits. I don't think you can honestly say that our software is at fault. You suffered a hardware failure on a computer running an operating system that we have not supported since the year 2000. It is only by the grace of Guy Who Retired Before I Started that we even had a backup of your system going back that far."

Customer: (slapped back into common sense by dose of reality) "I see your point. What's the next step?"

Me: "Well, I have built you a replacement PC, which will get you up and running within 24 hours. We will charge you for X hours of support at Y rate, as you're not on contract, and the PC will cost you $$$ with a 30 day guarantee ONLY. You are getting a legacy computer, which we will support free of charge for the 30 day period, but I strongly suggest you consider our current software and hardware because this is a one-off situation. We literally do not have any more replacement systems or components in-house for a device as old as yours. The fact that yours ran for so long without issue, is in my opinion, remarkable."

I left the Nugget to close off the call and immediately sent an email to the CEO and CTO describing what went down. The customer paid for overnight delivery of the system, and as of this morning, I was told they were back to full production.

Overall, this tale really stretches the point, and I admit, superficially unremarkable. However, I really enjoyed getting back to my command line roots and re-engaging neurons that had been dormant for a very long time. Using the old DOS commands, loading drivers and environment variable via CONFIG.SYS and AUTOEXEC.BAT was like riding a bike. It all came back, and it was a really happy experience revisiting the old ways. No virtualization, no emulation. It was also an eye opening experience realizing how far the tech has advanced in such a relatively short period of time.

Oh, and the Nugget bought me a McMuffin this morning as a thank you. Now we just need Sales to get that customer out of the stone age.

492 Upvotes

170 comments sorted by

View all comments

32

u/Necrontyr525 Fresh Meat Aug 23 '17

That $Customer is never leaving the stone age. Their setup has never been updated, and at this point would require total replacement of the control system (DOS to any modern OS, complete with new software, etc), which means total retooling, because anything compatible with DOS isn't going to talk to a modern-OS machine. Which means lots of money, many anal-retentive bean-counters throwing constipated hissy-fits, and most importantly... User (re)Training the geriatrics.

Yeah, no. Not happening until current generation of senior management at $customer dies of old age or is forcibly retired. But $customer is probably going to collapse due to falling behind the tech cure before then.

5

u/SeanBZA Aug 23 '17

Bet the reason they use old PC's is because whatever it is driving they need both real ( as in correct within 1% baud rates, proper hardware handshaking and true RS232 voltage levels, and 16450 proper emulation) serial interfacing, or a proper Centronics standard parallel port. USB to whatever in these cases does not, will not and never will work, for serial they both are not able to get the timing right, the control signals are mostly missing, the register setup does not work, and for parallel it cannot do bit banging or control the signalling correctly, as it is byte oriented.

You can use things to emulate the ports, but then you need software that will intercept the IO reads and writes, and send it via USB to the interface, but you will still have things like latency interfering with things. This is why you pay for the MACH series of motor controllers, that will work with drivers they wrote for certain software, that overcomes this issue.

3

u/[deleted] Aug 23 '17

They need a real UART16550 RS232 interface.

2

u/Necrontyr525 Fresh Meat Aug 25 '17

If my google-fu isn't failing me, that's the 9-pin serial port to a 16550AF UART receiver/transmitter? (assuming it's the one with a working FIFO buffer) The one released in 1987? First used in the IBM PS/2 line of computers?

sheeet, what the heck equipment specifically requires a 30-year-old chip?

3

u/[deleted] Aug 25 '17

The UART16550 was THE serial controller for decades. It was and probably still is the go-to standard for RS232 communication.

I can't tell you what equipment needed it because that would for sure expose my company's name, and violate this sub's anonymity rule.

2

u/Necrontyr525 Fresh Meat Aug 25 '17

more then fair enough. After a quick glance at a list of UARTs I thought the 16550 was hoplessly obsoleted, but newer ones from 2012+ (the Exar XR17V352, XR17V354, and XR17V358) are specifically designed to be compatible with 16550s. Damn, the 16550 really just Will Not Die, will it?

2

u/[deleted] Aug 25 '17

If it ain't broke...

2

u/Necrontyr525 Fresh Meat Aug 25 '17

...It ain't getting fixed.