Vendor Gold Standard Story

Federal Reserve Gold Bar

I think we have all worked on one of THOSE projects before. You know the kind that was doomed from the start. You were responsible for retrieving victory from the jaws of defeat. I was in a meeting yesterday with a former co-worker who was a partner on a high profile, highly risky, doomed from the start project. It is definitely a case study in one of THOSE projects. In this case, victory was rescued from the jaws of defeat. The project led to a huge growth experience for all involved. Trusted, lifetime relationships were formed.

The story is pretty interesting. If any of us had to do it over again, we certainly would have done it 180 degrees differently, but the lessons learned and the stories generated have lasted a lifetime.

In the early 2000’s, we were launching a new digital platform for business customers. One of the central new applications was an eBill portal application to allow business customers to perform the normal billing related functions --- view a bill, dispute a charge, pay a bill, run analytical reports, etc. My legacy telecom had multiple billers for various products, so this new eBill application had a requirement to allow a business customer to see all their charges across all the billers in a combined fashion and perform the aforementioned billing related functions. For the time, this was leading technology capabilities.

My telecom company was issuing an RFP (request for proposal) to get multiple vendor proposals on third-party software as the basis for this capability. The original plan was to get the RFP issued in February, evaluated / vendor selected in March, contract signed immediately with a project start in April and first phase implemented by June. Pretty aggressive. Adding to the complexity was that the initial phase was to have three of the five billers integrated into the first eBill release.

As indicated earlier, the project was one of THOSE projects that seemed doomed from the start. The RFP did not get issued until March. The vendor selection came down to two vendors. Procurement did not like the first choice because they were a small company and this contract would make them too dependent on the contract revenue. The second vendor --- highly regarded in the industry for billing capabilities --- was chosen. A little understood fact with the vendor was that their industry leading eBill platform had only been implemented in cases where this vendor also did the billing. So, their eBill platform consumed billing data that was produces by its native billers. For my telecom, the vendor would be consuming legacy biller data that was not native to either their billing platform or eBill platform. Our legacy billers had multiple bill cycle dates, different technologies, and the data they produced was formatted differently and was vastly different by each product they supported. The primary business biller, called the Universal Biller, had multiple products of its own, multiple bill cycles, and the data produced for each bill cycle was stored in twenty four different output files that were created over a five day period (for each bill cycle processed) and the files were not created in a sequential order where the eBill platform could re-assemble them as each piece part was produced.  You had to wait for the 24^th file to start reassembling (like humpty dumpty). If one of the files was corrupted or missing, the file recreation process had to start over.

The contract was a fixed price $5M contract with a fixed 3-month term. It was definitely one of THOSE high profile, highly risky, doomed from the start projects.

In reality, vendor selection and contract negotiations did not complete until June. Contractually, the first phase for the three billers was to complete in September. A promise of later phases was dependent on the first phase being delivered on time with full functionality.

As the complexities began to compound each other the September date became October, then December, and finally a mid-January start of User Acceptance Testing.

Due to the slipping dates and the complexity caused by the non-native biller inputs, the vendor started doing what most vendors do. They started throwing resources at the problem. The vendor was Israel based and started providing hundreds of on-site developers in NJ to try to get this project back on track. Remember, it was a fixed bid contract, so there was no monetary benefit for flooding my NJ location or the Israel core product team with additional developers.

My overall Israel base delivery manager was visiting me in NJ each month to discuss where we were on the project and the plan forward. After the September missed date, we were meeting in early October. My input was that throwing resources at this project was not going to rescue it. My opinion was that we did not have the “A” team or the “B” team working on this project. In a reference to a sitcom from the 1960’s and 1970’s (Hogan’s Heroes), I indicated that I thought we had the “F” troop doing the development. The cultural reference to the “F” troop for my younger readers referred to a group of soldiers in Hogan’s WW II POW camp who continually befuddled and outsmarted their bumbling captors. The point here was that we did not need more developers to solve the problem. We needed more skilled developers to get us back on track. My Development lead took the advice to heart and immediately started culling the team by releasing a substantial number of developers and importing more skilled people in addition to some of the burn out skilled professionals who were on the team. We went from two hundred resources down to seventy-five.

As mentioned earlier, this was a high-profile project. Every month, my boss --- the company CTO in the LABS, met with the vendor North American Vice President, and myself. Just the three of us. In this October time frame during our monthly meeting, the three of us had an interesting conversation. My boss, John, made a comment to the vendor VP reaffirming that our cost was not changing as the date and resources were changing. He was concerned about how this project must be costing the vendor several times more than they had contracted and wanted some reassurance that the vendor was not going to cut their losses and walk away from the project.

We were both flabbergasted by the vendor’s response. He said, “John, this is the first contract my firm has with (the legacy) ATT. We believe we need to deliver on this project so that we have the credibility to do other work for ATT. So, we are committed to deliver and to cover whatever the cost is to make the project successful.” John reiterated that the contract was for $5M. He asked how much they had already spent on the project. The answer was $15M and still growing. The vendor VP said, “We could choose to spend all kinds of marketing dollars by wining and dining and providing tickets to sporting events like the Super Bowl / World Series, concerts, etc. Instead, we would rather invest in the relationship and deliver the project so that our relationship continues and grows. We believe it is a better investment and builds us credibility. We certainly will not walk away from the project.” BTW the procurement rules were MUCH looser in those days.

John and I were deeply impressed.

Bottom line, there were long days, nights, and weekends. The vendor CEO travelled from Isreal to NJ in the middle of a Northeaster with three feet of snow on the ground to visit the NJ based resources and encourage them to complete the task at hand. One of my personal recollections was driving back to NJ from Atlanta after Christmas that year and stopping in multiple Service Centers along I-95 as I was getting status updates and working issues with the team. That just adds to the memories of this project.

The project did deliver to User Acceptance Testing in mid-January of that year. There were later phases. Twenty something years later, that eBill platform is still supporting business customers. Lots of new features, products, technology upgrades, and capabilities have been added over the years.

So, one of THOSE projects also became one of THOSE platforms that endure over time.

I have re-told this story many times. I have had many vendor partners over the years with some of the biggest vendors --- Amdocs, Accenture, IBM, Deloitte, Tech Mahindra, Cognizant, Boston Consulting Group, Infosys, and many others. I shared this story with them as the vendor Gold standard story from my perspective. In the Information Technology space, it is all about relationships, credibility, and trust. Partners work together to overcome problems and find solutions. Hopefully, the solutions are win-win. Sometimes, they may not be, but a trusted partner manages the tough times as an aberration to the relationship and works through circumstances with the long-term view in mind. Sometimes my vendors had to go to the mat for me and other times I had to go to the mat for my vendors even when that might not have been politically expedient. It did not matter. Partners are partners and trust is trust.

It is possible to rescue victory from the jaws of defeat, but it takes the right set of people, trust, and partnership to do it. I was often quoted as saying that “Give me ten highly skilled development resources and I can change the world. Give me three hundred average skilled development resources and I may not be able to fight my way out of a wet paper bag.”  Point is that throwing additional resources at a problem rarely solves the problem. Of course, adding resources to understaffed work or adding specialty skilled resources as needed is essential, but trying to solve a problem by just adding team members usually is a guarantee for failure because of the increased complexity for collaboration and communication and change control.