Blackballed For Doing The Right Thing

Ripples In a Still Pool of Water

Have you ever suffered negative consequences for doing the right thing? I am sure we all have. Today’s story involves a time when I knew that I had to do the right thing, but at a huge personal and professional risk.

As mentioned in an earlier blog, I once was the lead development manager for a small team of developers responsible for developing a centralized self-healing network restoration system for my telecom company. We had inherited this application from another development manager who was brilliant but had designed the system in a way that it would never be successful. There were some design flaws that could not be overcome. He had chosen network efficiency over technical obstacles.

I am probably much older than most of the readers of my blog. The older readers may remember as children / young adults, when there was a cable cut on the telecommunications network, that we often got messages on our TV’s that looked like a bull’s eye target and a corresponding message that due to a network outage, the program was interrupted.  In those days, the broadcast TV networks used telecommunications networks for the terrestrial facilities that carried the signals to their radio towers where the signals were broadcast. For things like sports events or special news events, the terrestrial land lines could be long distances. Sometimes these terrestrial lines sustained a cable cut, usually due to an incident such as a backhoe breaking the line or a natural disaster perhaps a flood, an avalanche, a wildfire, etc. When the terrestrial lines signals were disrupted, it impacted service not only to the broadcast networks but also for many consumer and business customers.

Before this centralized self-healing restoration system was implemented, it often took up to 15 hours to restore service to all the customers. If a fiber cable splice was required, then a 1.5 mile cable span had to be laid to repair the cable. This new self-healing restoration system eliminated the broadcast network lengthy outages and allowed the service associated with the cable cut to be restored in < 5 minutes. The restoration event restored Digital Services circuits based on an assigned priority. A simple analogy makes the visualization of the restoration easy. The restoration events looked like someone dropping a pebble onto a still body of water with the ripples in the pool getting longer and longer as the ripple --- visualization of the restored circuits --- traversed the still pool.

The inefficiency of the original design was that the two-way circuit was being restored one direction at a time, often resulting in the two directions being across different path lengths. and creating an echo on the signal. It would be similar to watching a video where the sound and the picture do not exactly match up.

My team was on a pretty tight schedule to get the original design working properly when we visited the Digital Services Lab in Holmdel, NJ just before Thanksgiving one year. We met with the DSL testers and received lots of feedback on all the problems with the current design that made it extremely difficult and time-consuming to remediate to work properly.

Coincidently, there was another NJ team that was working on an unfunded project (we generally called them skunk works projects) to replace the existing system with a different design. That effort had transferred three members from my small team to help this effort, but they kept missing initial design dates which had changed from weeks to months. The ultimate goal was to move the work entirely from GA to NJ. Their design had fatal flaws too, but they were not interested in hearing that input. So, delays kept occurring as they tried to produce a design that would work.

I attended the Holmdel meeting with my top two developers. Our conversation on the way back to Newark Airport was that we had a small window to get the current platform working and prevent the migration of the project from GA to NJ. Our window of opportunity was between Thanksgiving and the end of the Christmas / New Year holiday period when many activities were winding down for the year. We agreed to engage only three other developers and to tell no one that we were embarking on this effort. I decided not to tell my GA based boss because I wanted to protect him from the likely fallout.

We were not just patch-working fixes to the existing system, we were doing a total re-write to make the system faster by utilizing an in-memory database of the core network and fixing the other design flaws.

When I got back into my office, I called Roberto (not his real name) one of my peers who was the Program Manager of the self-healing network restoration effort. Just to keep things in perspective, this program was in the top three of the entire telecom company’s initiative for a couple of years running. Think of it as a high priority and extremely high visibility program.

My conversation with Roberto was that I was about to commit political suicide and would most likely end my career and employment. That might happen even if we succeeded. I asked him whether he and his Vice President JoAnne (not her real name) would provide me some political coverage against the leadership in my Labs development organization. My Labs Vice President and several his Associate Vice Presidents were going to be really pissed if we succeeded. If we failed, it would surely be a termination for me, but hopefully not for my team. If successful, my organization’s political structure was likely not to forgive easily, if ever.

It was the right thing to do. However, I knew I would be blackballed for doing the right thing.

Roberto and JoAnne agreed to provide the political backing I requested and agreed to send a note to my organization’s political structure when we were ready to announce the completion of the re-design and were ready to begin System Test and Digital Services Lab testing.  I could not afford any advance notice before we were ready to initiate the testing.

The first week of January, after a Herculean all hands on-deck effort by my five developers over the holidays, we were a go. My boss knew nothing of the effort.

During that first week of January, a letter went out from JoAnne to my organization's Vice President and his two main Associate Vice Presidents thanking them for letting me and my team initiate the skunk works re-design and JoAnne added that she was thrilled that we were ready to begin the re-design testing.

My organization’s political organization was blind-sided. Their skunk works project had made little progress in 7 months and were nowhere ready to announce or test their efforts. When they called to read my boss the riot act, he was completely clueless --- by design. You cannot beat someone up for something that they were completely unaware was occurring.

My plan worked. However, I will never forget the call I received from the two Assistant Vice Presidents in my organization. They were not happy and promised retribution.

Net / net though was that the effort was successful. I was blackballed for doing the right thing. That took about 2 years. I could forget good appraisals or good raises or good bonuses during that time period. It did not happen. Eventually they forgave me though.

The self-healing network restoration effort was uber successful. Most of us have never seen the network outage bull’s eye on our TVs since. The program was patented and was nominated to the Commerce Department for "the best patent for the decade of the 1990’s". It came in second place. First place was a cancer drug nominated by one of the pharmaceutical companies.

The leaders in my Labs political structure all got promotions ---- one, two, three, four, or five levels over time ---- into new areas of responsibility. My Assistant Vice President eventually became a Senior Vice President due to the effort. He became my mentor and helped me along in my career journey.

Sometimes we must take risks and do things that we know may have unpleasant consequences. However, at the end of the day, we really need to hold true to ourselves and to the things we believe or know to be right despite the consequences.

I often told my teams through the years that it is easier to ask forgiveness for doing what you know to be right than to ask permission. Sometimes it works out. Even if it does not, you have the satisfaction of knowing you did the right thing..