Bringing Agile and DevOps to HUD Exchange to fuel mission outcomes
HUD Exchange (HUDX) is a training and technical assistance (T/TA) website that provides resources, information, and program support to the U.S. Department of Housing & Urban Development (HUD) community partners—which include states, nonprofits, cities, public housing agencies, and tribes. The agencies that rely on HUD Exchange seek to provide affordable housing and prevent homelessness in their communities, and they require timely information on HUD requirements, program design best practices, and policy changes to help individuals in need.
Unsurprisingly, the pandemic has led to an increase in the volume of content that HUD Exchange produces and hosts. Webinars, self-paced online trainings, and COVID-19 specific guidance that HUD and TA providers create to inform grantees of fast-changing best practices, resources, and on-the-ground insights from various cities all live on hudexchange.info.
To help HUDX perform at its best, a lot needed to happen behind the scenes. Here is the story of how and why we migrated HUDX to the AWS tech stack.
HUDX behind the scenes
The HUD Exchange began as a monolithic Adobe ColdFusion application with a MariaDB database backend, both manually installed on separate virtual machines (VMs) in an on-premises data center. It also leveraged Solr, running on a shared GlassFish VM. These VMs were long-lived, and developers conveyed configuration changes and upgrades via plain-English Software Installation Plans (SIPs), which were meticulously carried out by the Operations team. However, this left resources underutilized and despite great care in documentation and implementation, human errors caused inconsistencies and wasted time due to troubleshooting.
We are supporting the mission of HUDX by using modern DevOps techniques, moving to containers, embracing continuous integration, and bringing development and operations teams together.
Over time, our team took strategic steps to address technical debt, embracing a combined Agile and DevOps approach to move the program to a modern application delivery. First, we migrated the Adobe ColdFusion CFML application to the open-source Lucee engine, which was more container-friendly. Then we containerized the Lucee and Solr services. Later, we implemented a site search using a trio of containers—Fess, Elasticsearch, and Kibana. This containerization eliminated the need for SIPs and long-lived VMs, as the services were generated dynamically using Infrastructure as Code (IAC) each time a configuration change or service upgrade was needed. This also made efficient use of resources, as many stacks and service containers could be run on the same Docker host VMs. Finally, we implemented automated testing as a safety net for application modernization, with automated tests being added to the automated Continuous Integration/Continuous Deployment (CI/CD) pipeline. The CI/CD pipeline uses Jenkins, which is itself configured with IAC and run as a containerized service.
HUD Exchange plays a crucial role in supporting HUD’s community partners—and an AWS Migration helped it meet the increasing demands of site users.
With nearly $5 billion in funds allocated to the mission, the HUDX website needs to perform at its best to meet the various needs of its community partners. In 2019, we migrated HUD Exchange to AWS to improve the user experience while saving a significant amount on rising hosting costs. HUD and TA providers often produce important resources that need to be shared with community partners (e.g. an email blast about equal access to homelessness assistance that drives grantees to the website to download new guidebooks) and AWS provides a smooth and seamless web experience for those users, even when site traffic spikes dramatically.
Another benefit to the AWS migration, coupled with a combined Agile and DevOps approach, is that the agency is now better able to respond to multiple development efforts that are happening. We can quickly spin up another instance of the site in order to do additional testing or new development. This gives us the flexibility we need to better respond to HUD or grantee needs.
AWS migration behind the scenes
Because of its prior containerization efforts, the migration went much more smoothly than it otherwise would have if it were still using SIPs and manually configured VMs. As part of this migration, the MariaDB database was migrated to an AWS-managed service, RDS. Later, as the project pursued service clustering (multiple container replicas per service), a need for centralized logging emerged. Containerized Elastic Filebeats and Logstash, along with AWS-managed Elasticsearch and Kibana, were used to provide a central interface for log storage, querying, and visualization. This use of Elastic Stack made exception and anomaly analysis quick and easy, and therefore, those became frequent practices.
The impact of the AWS migration on the HUDX mission
By embracing Agile and DevOps, we were able to move the program to a modern application delivery approach. This has translated to mission benefits—we are better able to respond to client change requests, accommodate spikes in site traffic, and meet changing grantee needs in real time through a combination of domain expertise and digital transformation solutions.
