Under a federal digital modernization strategy, Oddball was tasked with improving the security and speed at which services and applications were deployed on a high-visibility, high-impact platform. Our solution? Use our DevOps expertise to create a staged continuous integration/continuous delivery (CI/CD) data pipeline, and isolate applications front-end code to deliver quick and secure Veteran-facing features and improvements.
Problem Statement / Definition
Oddball supports a central digital hub for Veterans, families, and caregivers to access their benefits and services, including healthcare, disability, education, and home loans. Before Oddball, applications to improve or add new features to these Veteran-facing products were deployed in a single unified deployment via a full build versus single applications. This meant if an application’s front-end code was not properly isolated, the pipeline would still push it into production, leading to an increase in unvetted deployments, long build/test cycles, and risk that one team’s change in code could unintentionally break another team’s application due to the conveyor belt deployment.
Proposed Solution & Architecture
To help VFS teams across the platform deliver improvements and new features to Veterans more quickly and reliably, Oddball’s engineers leveraged DevOps expertise in both the development of a staged and secure CI/CD pipeline, and the careful isolation of front-end code. This allowed our 46 Veteran-facing services (VFS) teams to quickly deliver improvements to over 350 Veteran-facing products, and securely deploy new features to millions of Veterans and their families all over the globe. Key elements of our front-end deployment improvements include:
- Switching from deploying applications on an automatic CI/CD pipeline, to building a staged CI/CD pipeline that gated production deployment, and required releases/builds to receive approval from a VFS team member during staging.
- Guiding VFS teams on how to operate within the new guardrails and isolate their front-end application code so they can have the autonomy to deploy applications on the pipeline at their own pace.
- Preventing “code stealing” or accidental dependencies between VFS teams, removing parent-level cross-application imports that cause cascading failures, and approving applications to deploy on demand or redeploy when fixes are needed.
- Implementing strict guidelines that requires each VFS team to prove their code is fully isolated or else they are restricted to daily deployments on the original pipeline.
The improvements associated with our initiative to enhance the frequency and security of deployments through a staged CI/CD pipeline has been significant, from dramatically reducing the scope of impact of each application, to shortening feedback loops, to cutting emergency work, all of these developments provided a more stable and reliable production environment for VFS teams to deliver critical services to millions of Veterans.
Outcomes & Success Metrics
This modernized CI/CD workflow enhanced reliability and performance, and enabled a robust foundation for VFS teams to provide improvements, and new features securely. Key outcomes included:
- Reliable service delivery to 20 million users with 99.8% continuous uptime
- Reduced front-end build times from 12–20 minutes to 2–4 minutes
- Refined test selection which cut 30+ minutes from multiple test runs during CI
- Zero security breaches or disruptions
- Accelerated product development across 46 VFS teams, including 600+ developers, and seamless support for 350+ products built on the platform
Total Cost of Ownership Analysis Performed
AWS delivered the lowest total cost of ownership by directly addressing the high operational and labor costs associated with the previous front-end deployment process. By refining the deployment workflow with a staged CI/CD pipeline and enforcing code isolation, front-end build times and unscheduled deployments were dramatically reduced, which translated into a lower cost per deployment as our engineer’s time was spent building and shipping new features rather than fixing emergency disruptions. This approach to improving frontend application deployments resulted in a more stable environment for not only our VFS teams to build and deploy on, but for millions of Veterans, families, and caregivers by providing them with quick, secure access to critical products, and less central hub disruptions or downtime. We minimized total lifecycle costs of the platform by making the front-end application deployment process inherently faster, more secure, and less likely to cause costly failures.
Lessons Learned
- Application Isolation: When faced with slower deployments due to size and scale, Oddball isolated applications and allowed them to deploy independently from the daily deployment. This isolation reduced continuous integration times across applications, allowing for faster, reliable releases.
- Staged Continuous Deployment: Oddball implemented staged continuous deployment to allow applications to deploy at their desired cadence and reduced “accidental” production deployments.