Small Business Fulfillment: Take 2

Could you imagine being given a second, third or fourth chance to get it right?  TTake 2hink about it:  You have just made what seems like the biggest mistake in the history of your business and rather than have to dig your way out of; rather than having to make sacrifices to fix it; rather than dealing with the headaches and hassle – you are given a mulligan, a do-over,  or as we call it a “Take 2!”

 

I often write about my companies journey to get to where we are today, but it wouldn’t be possible without our own “Take 2.”  To set the scene.  It was the day before U.S. Thanksgiving.  We had begun preparing for the busy holiday weekend and Cyber Monday by pre-assembling boxes, staging inventory and organizing the anticipated shipments.  Everything went really well and then it all fell apart.  Emails began flooding in from clients:

 

“Is there something wrong with the Order Management System?  I cannot access it.”

 

“Patrick, the system appears to be down.  Can you advise?”

 

I tried to log into our Order Management System and then our Warehouse Management System.  Could it really be possible – our software was down?  We called the company in Erie, PA that was hosting our server.  The initial response – ”You have a server here?”  After a few minutes it became apparent that something terrible had gone wrong.  The company had a change in management and the Chief Information Officer had performed a system audit.  Our servers were not properly identified and were removed from the racks.  They were plugged into a wall outlet that was only protected by a department store surge protector.  It just so happened that construction was taken place on a street adjacent to the hosting company.  Earlier that day – there was a power surge that destroyed the surge protector.  Ten minutes before the emails started arriving in my inbox there was a second power surge that destroyed our main server.  The mirrored drives then attempted to copy the main drives – it was successful, but its success meant that it copied a destroyed drive.  The backup server then attempted to copy the mirrored drive – again it copied data that was not there.  The result was that our software was useless.  For a warehouse that is as advanced technologically as ours – we were blind.  The inventory in our building, its location, quantities and disposition were lost – not physically, but technologically.  Our Order Management System – which our clients are reliant on was offline.  The technicians and engineers that we spoke to explained that it was the perfect storm – we actually had a better chance of winning PowerBall than we did of this happening.  So what did we do?

 

No matter what your disaster plan is, nothing, and I mean nothing, prepares you for an incident like this.  The first step was to communicate to our clients regarding the incident and provide as much information as possible.  We knew that the drives were physically available to us.  We knew that the inventory was physically within our building.  We knew that our staff was willing to do what it takes to minimize the impact on our clients.  We quickly learned that we had a backup from three days prior that was clean and available.  We loaded that backup which gave us a bit of insight as to the inventory location.  But the inventory counts were still off.  Our clients were kind enough to provide access to their shopping carts.  We used this access to print orders.  This combined with the backup allowed us to process orders.  We celebrated Thanksgiving within our warehouse with our families, but we worked non-stop over the long weekend.  I did not leave the building for days.  The destroyed drives were sent to a data recovery company.  Within five days we had restored the drives.  We then manually entered every order into our system and loaded the tracking and shipping information as well.  The result – of all the orders – we failed to meet our guarantee on only three.  Our clients were impressed and sung our praises:

 

“Patrick, you folks are awesome!  I’m not sure how you did it, but there was no impact to my business.”

 

“I’m a customer for life.  You took what could have been a very bad incident and made it a positive one.  You did not miss a beat.  Fantastic!”

 

“Thank you for your efforts. Your dedication to my business is appreciated.”

 

When you operate a small business – you deal in real time situations.  Without the resources that a large corporation can bring to bear on an issue – the efforts fall squarely on you and your staff.  In our case, we went to extreme measures to prevent an incident like this from happening again.  Our main server was moved to a server farm.  The backup was moved to a separate location and a data backup to a third location.  The result is that the most we will ever lose again is four hours of data.  Our recovery plan was modified and the services of a team capable of handling data loss was engaged.  We went on to add additional infrastructure and capacity.

 

We were able to survive what could have been a devastating situation, but would you have?  Have you made plans for a disaster?  Remember – business is not like the movies where a Director can replay a scene until he gets the angle, the actors, the lighting and other parameters just right.  Business lives – it evolves – it breaths.  The measure of your success and your companies future is contingent not on what goes right, but how you handle what goes wrong.

 

take 2

Take 2 – Our appropriately named company mascot!

Leave a Reply

Your email address will not be published. Required fields are marked *