Rescuing Victory from the Jaws of Defeat: Lessons from a High-Risk Project

The Journey of a Doomed Project

I think we have all worked on one of THOSE projects before. You know the kind that seems doomed from the start. I found myself in a meeting yesterday with a former co-worker who was a partner on a high-profile, highly risky project. It’s definitely a case study in one of THOSE projects. In this instance, victory was snatched from the jaws of defeat. The project led to significant growth for everyone involved. Trusted, lifelong relationships were formed.

The story is quite interesting. If any of us had to do it over again, we certainly would have approached it 180 degrees differently. Yet, the lessons learned and the stories generated have lasted a lifetime.

Launching a New Digital Platform

In the early 2000s, we were launching a new digital platform for business customers. One of the central applications was an eBill portal. This application allowed business customers to perform typical billing functions—view a bill, dispute a charge, pay a bill, and run analytical reports. My legacy telecom had multiple billers for various products. Thus, this new eBill application needed to show all charges across all billers in a combined format and allow those billing-related functions. At the time, these were leading technological capabilities.

My telecom company issued an RFP (request for proposal) to gather multiple vendor proposals for third-party software as the foundation for this capability. The original plan was to issue the RFP in February, evaluate vendors and select one in March, sign a contract immediately, and start the project in April. The goal was to implement the first phase by June. It was an aggressive timeline. To complicate matters, the initial phase aimed to integrate three of the five billers into the first eBill release.

The Challenges Begin

As I mentioned earlier, this project was one of THOSE projects that seemed doomed from the start. The RFP didn’t get issued until March. The vendor selection came down to two options. Procurement was hesitant about the first choice because they were a small company, and this contract would make them overly dependent on contract revenue. The second vendor, highly regarded in the industry for billing capabilities, was chosen.

However, there was a little-known fact about this vendor: their industry-leading eBill platform had only been implemented where they also handled the billing. This meant their eBill platform relied on billing data produced by their native billers. For my telecom, the vendor would consume legacy biller data that was not native to either their billing or eBill platform. Our legacy billers had various bill cycle dates, different technologies, and their data was formatted differently across each product they supported.

The primary business biller, known as the Universal Biller, had multiple products, multiple bill cycles, and the data produced for each bill cycle was stored in twenty-four different output files. These files were created over a five-day period for each bill cycle processed, and they were not produced in a sequential order. This meant the eBill platform had to wait for the 24th file to start reassembling the data—like Humpty Dumpty. If one of the files was corrupted or missing, the entire file recreation process had to start over.

The contract was a fixed-price $5 million agreement with a three-month term. It was definitely one of THOSE high-profile, highly risky projects.

Delays and Compounding Complexities

In reality, vendor selection and contract negotiations did not complete until June. Contractually, the first phase for the three billers was supposed to be completed by September. However, the promise of later phases was contingent on the first phase being delivered on time with full functionality.

As the complexities began to compound, the September date slipped to October, then December, and finally to a mid-January start for User Acceptance Testing.

Due to the slipping dates and the complexity caused by the non-native biller inputs, the vendor began doing what most vendors do in such situations: they started throwing resources at the problem. The vendor was based in Israel and began sending hundreds of on-site developers to New Jersey to try to get the project back on track. Remember, it was a fixed-bid contract, so there was no monetary incentive for flooding my New Jersey location or the Israel core product team with additional developers.

My overall delivery manager from Israel visited me in New Jersey each month to discuss our project status and the plan moving forward. After the September missed date, we met in early October. I expressed my belief that throwing more resources at this project wouldn’t rescue it. I felt we didn’t have the “A” team or even the “B” team working on this project. In a nod to a sitcom from the 1960s and 1970s, I joked that we had the “F” troop doing the development. This reference was to a group of soldiers in Hogan’s Heroes who continually outsmarted their bumbling captors. The point was clear: we didn’t need more developers; we needed more skilled developers to get us back on track.

A Shift in Strategy

My development lead took this advice to heart. He immediately began culling the team, releasing a substantial number of developers and importing more skilled professionals, including some of the burned-out experts already on the team. We went from two hundred resources down to seventy-five.

As mentioned earlier, this was a high-profile project. Every month, my boss—the company CTO in the LABS—met with the vendor’s North American Vice President and me. Just the three of us. During our October meeting, we had an interesting conversation. My boss, John, reaffirmed to the vendor VP that our costs were not changing despite the shifting dates and resources. He was concerned about how much this project must be costing the vendor and wanted reassurance that they wouldn’t cut their losses and walk away.

We were both flabbergasted by the vendor’s response. He said, “John, this is the first contract my firm has with (the legacy) AT&T. We believe we need to deliver on this project to establish credibility for future work with AT&T. So, we are committed to delivering and covering whatever costs are necessary to make the project successful.” John reiterated that the contract was for $5 million and asked how much they had already spent on the project. The answer was $15 million and still growing. The vendor VP continued, “We could choose to spend marketing dollars on wining and dining or providing tickets to events like the Super Bowl or World Series. Instead, we’d rather invest in the relationship and deliver the project so that our relationship continues and grows. We believe it’s a better investment and builds our credibility. We certainly will not walk away from the project.” By the way, procurement rules were much looser in those days.

John and I were deeply impressed.

The Commitment to Success

The bottom line was that there were long days, nights, and weekends. The vendor CEO traveled from Israel to New Jersey in the middle of a Northeastern snowstorm to visit the New Jersey-based resources and encourage them to complete the task at hand. One of my personal recollections was driving back to New Jersey from Atlanta after Christmas that year, stopping at multiple service centers along I-95 to get status updates and work through issues with the team. Those moments add to the memories of this project.

The project finally delivered to User Acceptance Testing in mid-January of that year. There were later phases to consider. Twenty-something years later, that eBill platform still supports business customers. Numerous new features, products, technology upgrades, and capabilities have been added over the years.

So, one of THOSE projects also became one of THOSE platforms that endure over time.

Building Relationships in IT

I have retold this story many times. Over the years, I’ve partnered with many vendors, including some of the biggest names—Amdocs, Accenture, IBM, Deloitte, Tech Mahindra, Cognizant, Boston Consulting Group, Infosys, and many others. I’ve shared this story with them as the vendor gold standard from my perspective. In the Information Technology space, it’s all about relationships, credibility, and trust. Partners work together to overcome problems and find solutions. Hopefully, those solutions are win-win. Sometimes, they may not be, but a trusted partner manages tough times as an aberration to the relationship and works through circumstances with a long-term view in mind.

Sometimes my vendors had to go to the mat for me, and other times I had to go to the mat for my vendors, even when that might not have been politically expedient. It didn’t matter. Partners are partners, and trust is trust.

It is indeed possible to rescue victory from the jaws of defeat, but it takes the right people, trust, and partnership to do it. I often say, “Give me ten highly skilled development resources, and I can change the world. Give me three hundred average skilled development resources, and I may not be able to fight my way out of a wet paper bag.” The point is clear: throwing additional resources at a problem rarely solves it. Of course, adding resources to understaffed work or bringing in specialty skilled resources as needed is essential. However, trying to solve a problem by simply adding team members usually guarantees failure due to the increased complexity in collaboration, communication, and change control.

In the end, the journey through that challenging project taught me invaluable lessons about resilience, teamwork, and the power of trust. It’s a story I carry with me, a reminder that even in the face of adversity, success is possible when we work together with purpose and integrity.