Oddball modernized a federal platform using AWS infrastructure and tools to deliver secure, real-time, multichannel communication to Veterans.
Problem Statement/Definition
This platform provides critical notifications for Veteran healthcare through four data-enabled products. To keep up with increasing workloads while enabling seamless client onboarding, Oddball was tasked with migrating 100+ on-premise servers to AWS infrastructure and services.
Proposed Solution & Architecture
As part of our multi-year modernization effort for this program, Oddball implemented DevOps tools and a comprehensive AWS infrastructure/tool suite that includes:
● AWS Identify Access Management (IAM) to securely control access by managing users, groups, roles, and permissions
● AWS Relational Database Service (RDS) for scalable database management
● AWS CloudWatch for complete visibility into our technology stack
● Amazon Simple Email Service (SES) to rapidly push email reminders (e.g., overpayment, information change, form submission confirmations)
● Amazon Simple Notification Service (SNS) to push SMS notifications
● Amazon Elastic Container Service (ECS) Fargate to run containerized applications
● Amazon Elastic Container Registry (ECR) to store, manage, and deploy Docker container images and OCI artifacts
● Amazon Simple Queue Service (SQS) to send messages between components with no data loss
● Amazon Elasticache for Redis and MemcacheD
● AWS Application Load Balancers (ALB) for rapid routing and minimal latency
● Amazon PinPoint to notify customers (e.g., prescription tracking, reminders and updates, measurement-based care, document submission confirmations)
● Migrating MongoDB data store to the AWS GovCloud DocDB
In addition, Oddball architected a new AWS Python-based workflow designed to send SMS notifications to Veterans. Currently, we are leveraging several serverless Amazon Lambdas to translate opt-in data via integrations, consume payment events, and track eligible notifications in Amazon noSQL DynamoDB. We also enhanced the existing Flask application to send notifications during a defined timeframe, minimizing disruption.
We iteratively rolled out each modernization in close communication with our stakeholders and clients to avoid disruption and minimize deployment issues. Ultimately, AWS enabled us to build, operate, and maintain performant, reliable software that powers millions of notifications per year. This includes crucial Veteran outreach for appointment reminders, vaccine outreach, surgery notifications, prescription shipment tracking, emergency events, and beyond.
Outcomes & Success Metrics
● Low latency (P95 at <0.16s) and error rates below .02%, ensuring a performant application and reliable UX
● On average, we can securely send 14 million SMS notifications each month
● Modernization has resulted in high Veteran outreach success, 214 million texts, 27 million emails, and counting
● Modernized Flask app processes 6 million monthly events overnight
● Administrators can rapidly configure important healthcare notifications
Total Cost of Ownership Analysis Performed
Throughout the modernization period, AWS offered the lowest total cost of ownership by replacing fixed-capacity infrastructure with scalable, pay-as-you-go services that align costs directly to fluctuating notification volumes reaching tens of millions of messages per month. By leveraging managed and serverless services (e.g., RDS, ECS Fargate, Lambda, SQS, SES/SNS), Oddball significantly reduced long-term operational overhead, maintenance labor, and infrastructure risk. This architecture enabled high-throughput, low-latency delivery at scale while minimizing re-platforming costs, accelerating feature delivery, and ensuring sustainable operations for millions of Veteran healthcare notifications annually.
Lessons Learned
● Containerization in Cloud Environments: When deciding on a container orchestration solution, we found that using AWS services like ECS with Fargate for serverless container management simplified scaling and resource allocation, allowing us to focus on development instead of infrastructure management.
● Serverless Infrastructure: By choosing a fully serverless architecture for all AWS resources, we streamlined authority to operate (ATO) management, reducing overhead and improving compliance processes.
● Database Migration: Migrating from a legacy on-prem MongoDB system to a cloud-native, serverless database solution (Amazon DocumentDB) was crucial for eliminating maintenance overhead.
● Legacy Tool Compatibility: We faced difficulties with an old image uploading tool that was incompatible with newer environments. This required us to migrate to a new API service, enabling smoother integration and future-proofing against evolving platform requirements (e.g., RHEL 7 end-of-life).
● Single Sign-On (SSO) Integration: Integrating SSO into a serverless ECS environment required collaboration with the SSO team to facilitate proper configuration adjustments, highlighting the importance of cross-functional communication when adapting legacy services to modern architectures.