How CBG built a COVID-19 test ordering app & service, from soup to nuts, in 2 days
Many companies with software engineering teams champion values, missions, and tenets centered around helping them build meaningful software solutions for their customers rapidly. In cases I’ve seen where dev teams weren’t actually able to deliver quickly it’s often been due to factors & organizational debt outside of the technology teams control, like misalignment with business-wide value streams.
At CBG, aka Cooperative Benefits Group, we’re fortunate to have Development, Solutions, & Ops teams aligned to value streams led by leaders with common mindsets, intent on unitedly pushing forward our CBG vision, mission, and roadmap. While many of our dev team deliverables have been on a roadmap for several weeks, we came across an instance about 3 weeks ago where we had to pivot quickly to tackle and complete something that wasn’t on our roadmap.
Based on a new federal requirement we received, we needed to quickly build an online COVID-19 test ordering app & service for health plan members. Federal compliance mandated, with 5 days notice, that we had to fulfill a requirement for health plans to offer a DTC (direct to consumer) option where members could order COVID-19 tests online and then have product mailed to them for no cost. The required infrastructure to bill claims to the prescription benefit were not in place, especially for initiating an order that doesn’t require a prescription.
The remainder of this post is about the values that united us as a dev team, our high-level app architecture tenets, and the dev practices we executed, along with the tooling and stack we build this requirement on.
Our department values are what unify us
It would be insincere from one technology leader to another if this post were some magic recipe and combination of perfect technology tools our team used to deliver something like this application in two days. The most foundational ingredient of magic, the most basest secret sauce to how our team delivered so quickly is our unification by our own department values.
Our mindsets, our goals, our architecture principles and tenets, our philosophical approaches as a team align through 4 key values listed in the above screenshot from an internal deck we use to help remind ourselves what’s important. Without this foundation, delivering quickly would be more difficult.
Our key technology tenets guide our cloud strategy
Building upon our core values, next comes the key technology tenets that guide our approach and implementation angles from a tech perspective. These are high-level mindsets that we constantly remind ourselves of, discuss during staff meetings and trainings, and strive to embody in our strategic and tactical deliverables.
They make a difference, and they require tremendous discipline to live. When we’re trying to move quickly and engineer systems as simple as possible, sometimes these tenets can be perceived as slowing us down. Once they become habit, they actually help to accelerate our delivery of highly resilient, secure, and well-architected solutions.
Our architecture principles provide our rails
We’re almost to the tech stack that we built this solution around. One more key layer helps to remind us about the rails we’ve committed our team to, the technology “rules” we strive to imbue our solutions with.
We’re a little old school in that we love the “12 factor app” approach, even in this day of “microservice this” and “cloud-native that.” We use a combination of these principles to as rails to bang up against for our architectural and development methodologies. They’re not prescriptive and allow us the creativity to apply them from various directions without being locked into rigid solutions. Our values, tenets and architecture principles provide the foundation for our team to launch brightly.
What does the architecture actually look like?
Most visitors to this blog post are coming here to find out exactly what tech stack was used to build such a remarkably quick product. To set context, we did this with four full-time developers and one part-time business analyst. The developers were focused on this project for 75% of each day in the two day period. Let’s get into that now.
Building upon our values, tenants, and architecture principles we’ve been able to build a very resilient and reliable cloud infrastructure in AWS. We already had that in place with best practices in effect. We were able to quickly pivot and deliver upon that reliable infrastructure.
There are 6–7 key pieces to this tech stack. We have a public facing member portal, a core API, an ETL pipeline for checking demographics, an internal web UI for service teams to manage order state, the CI/CD dev pipeline for delivering code quickly, and production assurance dashboards for monitoring system health.
For our public facing member portal, we rely on React and Next.js with well-architected AWS infrastructure in place. We quickly spun up a new form to capture member input and trigger a call to our core API upon form submission.
For our API design we use Postman and actively design and manage our OpenAPI specs before development even begins.
For the core API, we rely on API Gateway secured by Cognito & WAF for security. This API is fronted by Route53 with a TLS certificate. Our API Gateway calls are proxied through to AWS Lambdas running Python runtimes that are deployed to a VPC in private subnets. The data store these Lambda functions persist to are Amazon RDS Aurora clusters that are in those same private subnets. We use the Serverless framework for abstracting away the complexities of managing serverless APIs in AWS.
For logic checks against our healthcare data stores, specifically around member data, we use the SnapLogic Intelligent Integration Platform against member eligibility to determine if the member submitting a COVID-19 test order is eligible.
All of our Lambdas log to CloudWatch log groups which have subscriptions to Kinesis Firehose that push to Datadog.
Once those orders are submitted by members, our internal service teams need a way to quickly manage those orders. We rely on Retool for its low-code/no-code approach of delivering complex web UIs for internal teams. We connect those UIs to the data store via built-in data connectors in Retool.
For our developer workflows using CI/CD, we use Visual Studio Code, GitHub for source code control, GitHub Actions for continuously deploying once integrated via AWS CloudFormation or the Serverless Framework CLI, depending on what artifacts are being pushed. We typically hit a cadence of 2–3 releases per day for this feature.
Last, but not least, we actively monitor for production assurance purposes the performance of this delivered system against expected traffic patterns. We have Datadog monitors, and dashboards defined to achieve these purposes.
TL;DR
That about sums this awesome project up. The specific tools chosen to build this service were not the reason we were able to complete it, but they definitely accelerated the process. Our teams aren’t beholden to any one tool, though we do value great toolchains for the efficiencies they bring our teams.
The most important digital enablers in place at CBG for developing rapidly are our values and aligned value streams with business owners working together with developer teams, without waste and without toxicity. I’m proud to work with a team so committed to fulfilling their purpose through positive, growth-first mindsets. If you get those few pieces in place, you can build tools like this in two days.
