My Three Worst Experiences in Software Development

December 10, 2008
By

Eric Spiegel

• Submit Feedback »
• More by Author »

“Gone?”

I almost dropped the cell phone and almost hit the car in front of me. The phone call was the result of something bad happening on a high profile data conversion project. Our team had to convert three legacy payroll and HR systems into a brand-spanking new ERP system. And we had to do it in a weekend or potentially impact the payroll of 100,000 employees.

Now, everything was going down the toilet.

I’m sure everyone in software has moments in their careers where they heard that same flushing sound and hoped it was just a project and not their career going down the commode. The fact is, it’s just part of working in IT. Like any job, things go wrong.

Although in IT, it’s kind of like working for the power company because it’s only when things go wrong that anyone pays any attention to you.

When I reminisce over my twenty year, mostly non-eventful career in IT, there are three instances I’d like to share with you, which will hopefully help you realize you aren’t alone the next time you have a really bad day.

Getting back to my conversion project, frankly, I thought it was going to be a breeze. We had run multiple simulations, which not only instilled confidence that the data would be accurate, but more important, we knew exactly how long the conversion would take to run.

But first the pre-conversion scripts had to run to scrub and prep the data for the actual conversion. Those scripts take two days to execute and verify the results. Two days.

I left work about 3AM the night (morning) before after successfully running the scripts. Everything had gone peachy. As I drove back into work at 9 AM (not sure why I even bothered going home), I receive that unwelcome cell phone call.

It was my boss and he said there was a little problem with the conversion data. “Actually”, he said, “there was a big problem because the conversion data has been wiped out. It’s gone.”

I thought “Oh crap” what did I do wrong? I went over and over the script progressions in my head, but everything had checked out.

“What do you mean, gone? How?”

Turns out our DBA had accidentally pointed the nightly backups to the directory we used for the conversion. He over wrote the data.

It was an honest (but very painful) mistake. The backups were supposed to backup the conversion data, not overwrite them. In retrospect, should have done this backup manually. We never even tested the automated backup process – which turned out to be the primary weakness in our conversion plan.

Now I had to find a way to rerun the scripts in eight hours instead of two days. I quickly identified which scripts were not dependent on each other and created a multi-threaded approach that enabled us to rerun everything just in time.

Unfortunately, haste makes waste and sure enough there were some errors in the converted data as a result of this last minute tinkering.

Ultimately, it didn’t matter. In the process of devouring an entire bottle of antacids, we were able to manually clean up the data and the first payroll run that next week went off without a hitch.

My second bad IT experience was at my first job out of college and it didn’t end so well. It involved another data conversion project, but this one was for a mammoth financial management system for one of the largest US government agencies. I knew this project wasn’t going to be any fun when I realized the conversion would happen in August, the hottest time of the year in Texas.