Under a federal digital modernization strategy, Oddball was tasked with improving a high-visibility, high-impact platform. Our solution? Shifting to AWS Elastic Kubernetes Service (EKS) and implementing other AWS services.
Problem Statement / Definition
Oddball supports a central digital hub for Veterans, families, and caregivers to access their benefits and services, including healthcare, disability, education, and home loans. As a high-visibility, high-impact, critical service serving millions of Veterans and their families, this program required a modernized, reliable technical infrastructure to support over 350 Veteran-facing products while accelerating product development for 46 Veteran-facing services (VFS) teams, which includes more than 600 developers, hundreds of endpoints, and tens of external services.
Proposed Solution & Architecture
To strengthen stability, scalability, and security of the platform, Oddball designed and built a new, cloud-based architecture that leverages AWS cloud-based infrastructure, platforms, and services. Key elements of our AWS modernization efforts include:
● Migrating vets-api from an Elastic Compute Cloud (EC2) instance to AWS Elastic Kubernetes Service (EKS), a key step taken to advance cloud-native architecture, enhancing the platform’s stability, scalability, and efficiency while enabling continuous delivery
● Designing with multi-availability zone AWS failover to seamlessly shift traffic in the event of a data center outage
● Implementing AWS container scanning during build and runtime to prevent critical operating system vulnerabilities
● Leveraging a comprehensive suite of AWS products and services (e.g., AWS CloudWatch, AWS Systems Manager Parameter Store, Relational Data Store, S3, GuardDuty, AWS SSM agents) to improve visibility, scalability, and usability, minimize operational risk, and improve platform reliability.
Oddball created the gold standard for building a cloud-based infrastructure that VFS teams and their developers now follow. We followed our proven iterative rollout strategy to 1) minimize user impact and migration issues, 2) complete the transition with zero downtime, and 3) transition to AWS ahead of schedule.
Outcomes & Success Metrics
This modernized cloud-native architecture enabled a robust foundation for secure cloud-based builds. Key outcomes included:
● Reliable service delivery to 20 million users with 99.8% continuous uptime
● Zero security breaches or disruptions
● Automated scaling supported 53.8 million successful healthcare, education, and claims transactions without experiencing significant latency
● Accelerated product development across 46 VFS teams, including 600+ developers, and seamless support for 350+ products built on the platform
● Real-time performance monitoring, continuity of operations, and ongoing Veteran access to critical services
Total Cost of Ownership Analysis Performed
AWS delivered the lowest total cost of ownership by replacing static, EC2-based infrastructure with a cloud-native, containerized architecture that automatically scales to support tens of millions of transactions. Managed services including EKS, multi-region failover, and integrated security/monitoring tools reduced operational labor, outage risk, and long-term maintenance costs, enabling zero-downtime deployments and continuous delivery to VFS teams. This approach balanced predictable infrastructure spend with elastic capacity, allowing Oddball to maximize platform reliability, security, and developer velocity while minimizing total lifecycle costs for a high-visibility, mission-critical system.
Lessons Learned
● Application Isolation: When faced with slower deployments due to size and scale, Oddball isolated applications and allowed them to deploy independently from the daily deployment. This isolation reduced continuous integration times across applications, allowing for faster, reliable releases.
● Staged Continuous Deployment: Oddball implemented staged continuous deployment to allow applications to deploy at their desired cadence and reduced “accidental” production deployments.
● Seamless EKS Upgrades: Oddball’s preparation activities included checking compatibility for all core/services for each EKS version upgrade, which resulted in zero downtime and zero triggered alerts.