Towards High availability and beyond.

Someone once said, “If you can dream it, you can do it.”. Our partners dreams are our dreams, and we partner with them to reach those dreams.

Recently we were challenged by one of our partners to implement the new version of their product. From their experience of the previous version, our partner had some challenges that they wanted to overcome, avoid or minimize the impact in day-to-day activities.  

This new product must have a web, Android, and IOS mobile application, be cloud-based, scalable, and of high availability. It should be possible to deploy every new version of the application during office hours without losing service. 

I will not discuss monotonous topics like solution architecture or application design. Still, I’ll guide you through some challenges and decisions made to deliver value and, in some cases, reduce day to day costs.

Choice of an Agile Platform.

Our first main decision was which agile platform would we use, on one hand to support the development team, and on the other to allow the project continuity once it enters in maintenance. From our experience from other projects, we have chosen to work with Azure DevOps.

Azure DevOps has a pleasant free tier for small development teams and unlimited stakeholders and covers all project sections. With Azure Boards for specifying Epics and Backlog definitions, going through Azure Repo (for source code control), Azure Pipelines (for continuous integration, delivery, and deployment), and Azure Test Plans to test the applications. With another great benefit: all related activities (Epics, User Storys, Application Changes) are available on the same workspace.

Building web source code.

Given the solution complexity, several repositories were created to keep the multiple components decoupled and cohesive, supporting each other with a single responsibility. This approach leads us to our next challenge, building and testing and deploy all these projects, since these activities take time and there is a limit of build minutes on the Azure DevOps platform. 

Azure DevOps free plan comes with 1800 build minutes for a single agent. Anything more has an extra cost. To overcome this limitation, we installed some on-permisses agents and configured them on Azure DevOps. With this approach, we were able to build agents for specific needs (Eg: allow SonarQube runs, build Java, React, and other solutions) and have several projects to execute them.

Building mobile source code.

While this scenario allows most of the project solutions to be built, one of the product requirements is to develop a mobile application targeting IOS and Android. Although it is possible to create Azure DevOps built pipelines to make these applications, the necessary effort doesn’t make much sense for an out-of-the-box setup. 

There is an alternative with easy integration and with a set of excellent features (like Analytics and Diagnostics for free). Visual Studio App Center enables continuous integration (with 240 build minutes), delivery and deployment for mobile applications, as well as to integrate with significant git hosting providers. Unfortunately, it doesn’t support on-premises git hosting. 

This platform handles all the hassle of the mobile application lifecycle, and the analytics/diagnostics features consolidate helpful information about application usage.  

Environmental Variables.

Polarising’s solution design specified the decoupling of the frontend application from the backend and other services, and with this decision, different challenges emerged. 

For the frontend application, the chosen technology was React. Like any other frontend technology, the application property configurations (such as services endpoints) are defined by the build (then minified) task for each environment (Development, Users Acceptance Tests, Production), raising considerations while defining the build pipeline.

Once again, Azure DevOps proves to be the right choice by providing some actions to replace the application configurations before the build and having a user interface for setting those properties only in one location. So, for each build pipeline, for each environment, the correct configurations are applied.

Image from Dreamstime.

For the backend applications, dotnet core was the chosen technology. With this approach, we only have one build pipeline to do the build, test, and code analysis tasks and generate a single artefact deployed on the different environments. For environmental variables and configurations, there are three main possibilities:

1. On the release pipeline, define all environmental configurations and then apply and replace them on configuration files. 
2. On Azure WebApp Configuration, set up all configurations for each environment.
3. Use Azure App Configuration to store non-sensitive information, use Azure Key Vault to store sensitive information, and configure the Azure WebApp Configuration to use the data from these sources.

The last option extends the second option and provides a more secure and centralized data storage. This approach allows multiple application uses for the same data without replicate the information between all applications. If we have to update any key, we only have to edit it in just one place. 

Control application deployments.

If while deploying into a development environment, it shouldn’t provide many hazards. When deploying for more crucial environments, the challenges start to uprise. To overcome this sensitive checkpoint, within the release pipeline, we apply pre-deployment conditions to activate the pre-deployment approvals and set up the person(s) that must approve the pipeline execution.

For instance, to deploy into the User Acceptance Tests environment, the Product Owner must support this deployment, and for Production it must be the Product Owner and the IT Supervisor. It could be any other recommended person, someone related to the project or maintenance team.

The platform keeps track of the versions deployed for each environment, making the artefacts versioning, or rolling back, much more manageable.

Deploy without losing Service.

By now, we can take a few minutes from committing a source code change to deploying it into the Azure Cloud, making it available to application customers. 

Everyone remembers those days when IT teams planned to deploy new versions of the application at night, near the weekend, and had to turn down the service, and for the maintenance timeframe, the system didn’t process any requests.

There are many deployment strategies and most of them target specific complex use cases. The simplest is the “Blue-Green Deployment.”, where we deploy our application to one node (Blue), the partner users do regression tests on the significant features, and if everything is OK, we swap with the “Production” (Green) node (which has the previous version). Afterwards, if any malfunction is found, we can roll back by switching the node again.

Within Azure, we can achieve this goal by using deployment slots, where we have the “Production” slot and create another space for a “Blue” deployment and configure it. The swap action will tell the infrastructure that the current production version will not receive more requests and handle the current ones. Any new bids will be accepted by the new application. This action may take a while, but by the end, the production slot will have the most recent version, while the “blue” place will have the previous version.

Application High Availability.

Most application users will complain about the slowness, or even loss of service, of the application, and the time it takes to do some operations, especially in high demand periods when many application users interact with the applications.

One of the many features of “Azure App Service” is autoscaling, where we can auto-scale in or out according to defined rules. Those rules are defined in two factors:

1. Schedule based: specified by a time frame.
2. Metrics bases: determined by metrics of usage such as CPU.

For the solution in hand, we applied both strategies. By daylight, we configured to auto-scale based on metrics (such as CPU, Memory or Network) and once the peak goes by, apply auto-scale out rules. Don’t forget, for every scale-in control, you should set up a scale-out control.

By night, the application usage is much less, so we created other rules that do a scale-out at night and apply a scale-in at the beginning of the morning. 

This strategy increases the application’s high availability and resilience and attracts savings by reducing the night capacity and optimizing infrastructure usage.

Conclusion: dreams can come true.

To make our partners dreams into a reality is always challenging. It is more than creating a new application or following code best practices. It is also about looking ahead and build an infrastructure that allows the team to have a pleasant application maintenance experience and reduce the time and cost of building and deploying new features (and fixes) into the cloud.

It is also essential to provide the application consumers with a pleasant user experience by having service availability and optimizing the infrastructure cost by reducing the infrastructure during the low usage timeframes.

We have faced more challenges, but these were the ones with the most impact.

(This article was originaly published at Building the Future event website.)

Nuno Cancelo
Microsoft Practice Lead at Polarising