Budget-friendly Acceleration, Course Correction and Agile Coaching when a Data Lake pilot hits turbulence

Four months into the project, the architecture was completed and the intent fairly well defined but data ingestion was overly complex. And there were leaking elements when reconciling Curated and Reporting layers with RAW, and inconclusive root cause.  

The previous consultancy had adequately built a foundation for the data lake leveraging Azure and Databricks, but there were recurring complications ingesting and mapping data, and consequently their contract severed. Additionally, the internal supporting data engineers were new to agile principles, and shared resources across other projects outside of this initiative. The team did not have a product owner, but had a PMP certified project manager playing the role of Scrum Master. 

Ippon was contracted for project rehabilitation. We would have until the end of the year, or just under four months, to audit the current situation, possibly design and execute a new destination state, support successful data ingestion of 7 key sources, while training the surrounding team and improving their use of agile methodology. 

Ippon ultimately succeeded in launching the data lake MVP to support its first use case reports, including automating the first seven required ingested sources aligned with all security requirements. Two engineers were retained for an additional month to triage and resolve data mapping errors. 

More than a successful product launch however, we may celebrate the organization enablement more. 

Through coaching, the team developed strength in story creation, pointing and execution. In the final sprints, team velocity eclipsed 85%, reflecting the team’s growth in efficiency. Overall time in meetings had reduced, and forums more focused on value. The project manager was operating effectively as a Product Owner, building the backlog around the next Use Case intent.

This model was an embedded coaching, hybrid team augmentation. It was also somewhat unique in comparison to many Ippon enterprise deployments. Often we may add a full stack ‘Team-in-a-box’ to accelerate work while demonstrating and upskilling other full teams on the same initiative. 

This client team had very limited experience with agile, and included 1 Project Manager (the org was traditionally waterfall) playing Scrum Master, 3 Data Engineers and 1 Sr Infrastructure Engineer, all cross-functional shared resources on other projects. The team also included 3 dedicated QA analysts working off-shore, 10.5 hours ahead. 

Ippon embedded a Sr Data Tech Lead and a Sr Product Coach, both at 50% time, 2 data engineers full time and 1 cloud engineer to support CI/CD a total of 6 weeks. 

We augmented to create a single, blended acceleration team.

Ippon spent the first three weeks onboarding, auditing current state architecture, then proposing and aligning the team (and stakeholders behind) an improvement plan. Redesigning the Azure Data Factory (ADF) ingestion framework became necessary when too many data leaks were too difficult to resolve through overengineered rule-sets on top of overwritten default functions for no particular need. Instead, Ippon suggested leveraging open-source Airflow for orchestration, which would be easier long term for support and monitoring, knowledge transfer or further extending to adopt new ingestion sources. 

Our team began as the only full time engineering resources on the project, so momentum was slow to see. Instead of working from an infinite backlog, our Product Coach paired with the team Scrum Master to define an intent roadmap and delivery milestones. With each new sprint, Ippon also refined the team ceremonies as much as the backlog, incrementally pushing new efficiency levers including reducing meeting minutes (and function), revising the scrum board, adopting a closing ceremony to capture and assess velocity, as well as introducing retrospectives. 

Delivery was broken into intent work streams, where Ipponite engineers usually paired with associates to upskill along the way. Office Hours and demos were established to further create team synergy as much as training grounds throughout development cycles. Product coaching developed the Project Manager’s skills in product ownership, building confidence in problem definition, customer discovery, intent refinement, product roadmaps and stakeholder communication alerting for risks, dependencies and escalations.