Budget-friendly Acceleration, Course Correction and Agile Coaching when a Data Lake pilot hits turbulence

Top 10 private equity firm uses a data lake transformation to also pilot agile development. Their first attempt loses steam, but Ippon takes a different approach.

Key Figures

< 40%
team velocity
consulting firm
4 months
to course correct


Four months into the project, the architecture was completed and the intent fairly well defined but data ingestion was overly complex. And there were leaking elements when reconciling Curated and Reporting layers with RAW, and inconclusive root cause.  

The previous consultancy had adequately built a foundation for the data lake leveraging Azure and Databricks, but there were recurring complications ingesting and mapping data, and consequently their contract severed. Additionally, the internal supporting data engineers were new to agile principles, and shared resources across other projects outside of this initiative. The team did not have a product owner, but had a PMP certified project manager playing the role of Scrum Master. 

Ippon was contracted for project rehabilitation. We would have until the end of the year, or just under four months, to audit the current situation, possibly design and execute a new destination state, support successful data ingestion of 7 key sources, while training the surrounding team and improving their use of agile methodology. 


Ippon ultimately succeeded in launching the data lake MVP to support its first use case reports, including automating the first seven required ingested sources aligned with all security requirements. Two engineers were retained for an additional month to triage and resolve data mapping errors. 

More than a successful product launch however, we may celebrate the organization enablement more. 

Through coaching, the team developed strength in story creation, pointing and execution. In the final sprints, team velocity eclipsed 85%, reflecting the team’s growth in efficiency. Overall time in meetings had reduced, and forums more focused on value. The project manager was operating effectively as a Product Owner, building the backlog around the next Use Case intent.


This model was an embedded coaching, hybrid team augmentation. It was also somewhat unique in comparison to many Ippon enterprise deployments. Often we may add a full stack ‘Team-in-a-box’ to accelerate work while demonstrating and upskilling other full teams on the same initiative. 

This client team had very limited experience with agile, and included 1 Project Manager (the org was traditionally waterfall) playing Scrum Master, 3 Data Engineers and 1 Sr Infrastructure Engineer, all cross-functional shared resources on other projects. The team also included 3 dedicated QA analysts working off-shore, 10.5 hours ahead. 

Ippon embedded a Sr Data Tech Lead and a Sr Product Coach, both at 50% time, 2 data engineers full time and 1 cloud engineer to support CI/CD a total of 6 weeks. 

We augmented to create a single, blended acceleration team.

The delivery

Ippon spent the first three weeks onboarding, auditing current state architecture, then proposing and aligning the team (and stakeholders behind) an improvement plan. Redesigning the Azure Data Factory (ADF) ingestion framework became necessary when too many data leaks were too difficult to resolve through overengineered rule-sets on top of overwritten default functions for no particular need. Instead, Ippon suggested leveraging open-source Airflow for orchestration, which would be easier long term for support and monitoring, knowledge transfer or further extending to adopt new ingestion sources. 

Our team began as the only full time engineering resources on the project, so momentum was slow to see. Instead of working from an infinite backlog, our Product Coach paired with the team Scrum Master to define an intent roadmap and delivery milestones. With each new sprint, Ippon also refined the team ceremonies as much as the backlog, incrementally pushing new efficiency levers including reducing meeting minutes (and function), revising the scrum board, adopting a closing ceremony to capture and assess velocity, as well as introducing retrospectives. 

Delivery was broken into intent work streams, where Ipponite engineers usually paired with associates to upskill along the way. Office Hours and demos were established to further create team synergy as much as training grounds throughout development cycles. Product coaching developed the Project Manager’s skills in product ownership, building confidence in problem definition, customer discovery, intent refinement, product roadmaps and stakeholder communication alerting for risks, dependencies and escalations. 


Our latest success stories

Success story date icon05/31/2023

Revolutionizing Operations: A Triumph on the Platform of a Major Financial Institution

The portion of the large financial institution that we interfaced with was the center for machine learning. Ippon provided the center for machine learning with a team in a box composed of several site reliability engineers (SRE). SRE engineers have a mixed toolbag comprised of software and devOps skills, there primary focus is to try drive high availability, reliability and scalability. These SRE engineers took up the challenge of overseeing the adoption of SLI/SLO metrics for users of the platform, standing up dashboards to properly reflect the aforementioned metrics, redefining the incident support model and gathering data about all critical services on the platform. As a result of the SREs work, the center for machine learning at the large financial institution has better metrics, tracking and reporting for incidents.

Read more
Capital One
Success story date icon05/31/2023

Debt Financing Fintech

Accelerating a Fintech’s Data Migration to Power Data Insights for Making Impactful Business Decisions

A debt financing fintech that offers a single point-of-sale platform with a wide range of pay-over-time products and services which allow merchants to make offers to all customer types with varying levels of credit. The company credits their ability to deliver such a positive experience for merchants and customers due to their world-class support, advanced technology, and analytics that power their product. As growth continues to increase, it is vital for their company to ensure they are able to track and report on key metrics supporting business decisions.

Read more
Success story date icon05/31/2023

Wealth Management FinTech

Ippon was engaged to help integrate a Wealth Management FinTech into one of our main customers in the finance industry. The primary goal was to increase security and automate the creation of users by integrating this new application into the company’s Single Sign-On (SSO). Ippon worked as a team of three developers closely with their architecture team to deliver this over 4 months.

Read more