Four minutes past eleven on a quiet Saturday morning, red alarms went blaring out at Uniregistry’s NOC in the Caymans.
There was an unexpected emergency, something persistent and resembling an orchestrated attack against the registry’s network.
Jonas Right, junior sysadmin, caught the incident first; he had to alert Frank Schilling.
“Frank, we have a problem, the Domain Name Sales forms are breaking, repeated user input data is generating multiple core dumps!” exclaimed Jonas Right, showing Frank Schilling a diagram full of little red dots that blinked.
JUDAS 5 alert at Uniregistry.
Frank scratched his fashionably unruly, dirty blonde hair, then raised his right eyebrow, looking perplexed.
“Well then, shift the load balancing server in tandem with the backup servers, plug in two more boxes and reroute the database server off the nginx setup, that should do it,” he said, pointing at the network diagram’s outer edges.
Jonas blinked twice, then tried to mutter something, but Frank cut him off.
“Just do it!”
Thirty seconds later, Jonas was punching new IP ranges for the servers, assigning the overloaded box to a new subnet. After two minutes the load had dropped to acceptable levels.
“Frank, we are doing fine, will monitor this for the next hour and a half, looking good man, thanks for the suggestion!” said Jonas, perspiring heavily.
Six minutes of network stability soon ended, when the network’s load shot past 80, once again sending the Joint Uniregistry Deployment Alert System (JUDAS) into red territory.
It was a JUDAS 5, the highest level of emergency at Uniregistry. Frank Schilling shot out of his office and ran to the NOC quarters.
“What the hell is this Jonas, are we under attack? Drop the secondary firewall down and call Bobby to help out!” said Frank Schilling, looking rather antsy; he went back to his office, and sat into his Aeron chair, muttering the “F” word. Twice.
That wasn’t looking good.
Jonas ran half way across the building, past the kitchen and the meeting rooms overlooking the blue ocean waters, looking for Bobby Stardom, the senior administrator of Uniregistry. He found him reading the Cayman Islands Journal, while sitting on the toilet.
“What the hell dude, you have to go that bad? Knock on the door maybe, hello?” exclaimed Bobby Stardom, pulling his pants up in one go.
Jonas had no time to explain, he pulled Bobby off the toilet seat and out of the bathroom, then rushed him back out to the NOC area.
“No time to wash your hands Bobby, we are having a JUDAS 5 incident right now, Frank needs this fixed or we are royally fucked!” exclaimed Jonas.
Bobby Stardom took his station spot, cracked his knuckles, then wiped his hands with a disinfecting towelette and handed it to Jonas.
Eight wide screen monitors tracked every aspect of Uniregistry’s network. A few key-strokes later, he was in control of the situation.
“Re-routing the nodes, with full round robin telemetry. See that dot here? Not that one dude, right there. It’s a single core crashing the kernel every time you trigger a query from DNS. I’m isolating this bitch and we’ll deal with it later. Now then, tell Frank we are good to go in two minutes. Go, dude!”
Dripping hot sweat, Jonas ran to Frank Schilling’s office. From the door, he gave the two thumbs up signal, and wiped his forehead with his sleeve.
“Frank, it’s up, we are good, Bobby did his magic again. There should not be any more JUDAS alerts today,” said Jonas, half out of breath.
Frank Schilling checked his iPhone.
The Uniregistry App blinked green, indicating an “all clear” status. The incident had been dealt with in under 15 minutes.
“Great. This should not happen again Jonas, better put some processes in place ok? Tell Bobby thanks, I’ll come check with you guys later.”
Jonas gave another two-thumbs up sign, a bit hesitant this time; he looked at Frank and attempted to smile, then left the office, closing the door behind him.
In the network operations room, Bobby Stardom was standing by his array of monitors, having a sip from a tall glass of iced coffee.
“Dude, so you want to know what happened?” he asked Jonas, smirking. “This little bitch here started it all up,” he added, pointing at a Domain Name Sales contact form dump.
Following his finger across the screen, Jonas Right scrolled down the database output of Domain Name Sales inquiries, stopping at a really long line.
“What the fuck? Josias Feliz Macedo Santos DelaCruz Fuerte Jimenez Alamierda? Who has a name like that?” exclaimed Jonas.
That was fresh! A person’s rather long name had crashed the database field processor, triggering a chain reaction of unfortunate incidents and a JUDAS 5 alert across Uniregistry; it wasn’t an attack after all.
“You know, things in Mexico are different, these guys give their kids some really long names. Better re-program the input handler just in case, you never know,” said Bobby Stardom, taking a long sip from his coffee glass. “Make sure you double pad the output on a fixed 16k container to trim out excessive data, and we’ll be fine,” added Bobby.
Jonas drew in a long breath.
That was a close call, but easy to fix in a couple of hours, then they could all go for cocktails by the bay, before sundown.
“Thanks man, I appreciate it. You rock!’ he exclaimed.
Bobby Stardom shrugged his shoulders, then headed out back towards the Uniregistry bathroom. He had a personal business to finish.
Hey, I’m Bobby.
Just here to clarify the aftermath. We wrote a special unit test called `signup_form_can_handle_spanish_names()` to catch any future regressions. We’re also migrating all of our DB tables from fixed-length VARCHARs to PostgreSQLs TEXT field as I’m writing this.
Please don’t persuade Frank to build any “processes” around our work, that would leave less time for Mojitos on the beach.
Thanks
Bobby – No need to be modest, it’s understandable that you’re well worth your salary’s figures. Mas mojitos!