How to use Terraform in Azure Devops Pipelines

⚡ Learning Azure Devops YAML pipelines? Check out this What Ive Learned article

In this example I’ll show you how to create an Azure Function App by using Terraform in an Azure Devops CI Pipeline.

Since Microsoft is shoving their YAML model in throats lately, we shall use YAML to build our CI Pipeline.

Difficulty: 💚💚🤍🤍🤍

• Terraform: Beginner
  I’m a total noob with this tool, so you should be fine
• Azure DevOps: Beginner
  I’m quite experienced with this platform, however this example doesn’t require pre in-depth experience
  Prerequisite: You have an Azure Devops Organisation set up, and know how pipelines work in general
• Azure Resource Manager: Intermediate
  Azure Resources cost money and we are going manipulate them in an automated fashion. So it is in your wallet’s best interest to have a decent grasp of Azure fundamentals.
  Prerequisite: I’m assuming you have atleast Owner permissions on an Azure Resource Group
• Git Repo: You need to know how to use Git Repo’s

High level overview:

1. Provision Azure Backend
2. Create the Terraform Template
3. Prepare the Azure Devops Organisation
4. Create CI Pipeline
5. Troubleshooting

1 — Provision Azure Backend

First things first, we need create the required Azure Resources that won’t be created by the CI Pipeline.
Terraform needs to keep a State file to keep track what Resources are managed by Terraform.
Using this State file, Terraform knows which Resources are going to be created/updated/destroyed by looking at your Terraform plan/template (we will create this plan in the next section).

So in Azure, we need a:

• Storage Account:
  Create a Storage Account, any type will do, as long it can host Blob Containers.
• a Blob Container:
  In the Storage Account we just created, we need to create a Blob Container — not to be confused with a Docker Container, a Blob Container is more like a folder.

2 — The Terraform Template file

Create a local Folder, use the below code and save it as functionapp.tf in the root.
🚦 Use your own defined names and subscription_id

# this line is important so that backend connection is extablished in 
the pipeline
terraform {
    backend "azurerm" {}
}
# Configure the Microsoft Azure Provider
provider "azurerm" {
    subscription_id = "74d6a1ea-aaaa-bbbb-cccc-28b098c3435f"
    skip_provider_registration = "true"
    features {}
}
resource "azurerm_app_service_plan" "test" {
  name                = "azure-functions-test-service-plan"
  location            = "westeurope"
  resource_group_name = "resource_group_name"
  kind                = "FunctionApp"
  sku {
    tier = "Dynamic"
    size = "Y1"
  }
}
resource "azurerm_application_insights" "test" {
  name                = "miel-test-terraform-insights"
  location            = "westeurope"
  resource_group_name = "resource_group_name"
  application_type    = "web"
}
resource "azurerm_function_app" "test" {
  name                      = "miel-test-terraform"
  location                  = "westeurope"
  resource_group_name       = "resource_group_name"
  app_service_plan_id       = azurerm_app_service_plan.test.id
  storage_connection_string = "storage_connection_string"
  app_settings = {
    APPINSIGHTS_INSTRUMENTATIONKEY = azurerm_application_insights.test.instrumentation_key
  }
}

3 — Prepare the Azure Devops Organisation

Install the Terraform Extension (free) to your DevOps Organisation *

• Create a classic Release Pipeline
  we dont really need this pipeline, but we need it install the Terraform tasks
• Go to the Stage and edit the Tasks.
• Go to empty Agent job and add a Task
• Search the Marketplace for Terraform (by Microsoft DevLabs)

* [Update 2020-05-16] As a reddit user pointed out in this comment, using Microsofts provided Tasks is quite risky because of lack of support. Ideally you should be using the Azure CLI and perform the native Terraforms commands.

Create a Service Connection

This allows your Pipeline to have access the Azure Resources

• Go to your Azure Devops Project, hit the Cog icon, go the Service connections
• Click on the New service connection button (top right)
• Select Azure Resource Manager — Service Principal (automatic)
• Select your Subscription and Resource Group, check the Grant access permission to all pipelines, and Save it

4 — Create the CI Pipeline

Git repo

In the root of your local folder (the one you created in 1.)
Create an azure-pipelines.yml file using the below template code:

🚨 As some fellow redditors have pointed out, using a destroy step in the Pipeline is not a best practice (like don’t ever do this in production), the only reason I included it the example was to demonstrate the usage of all the types of available commands.
*Update 2020–05–16: Another improvement is to upload the tf plan as an Artifact, and create a different pipeline/stage to use this Artifact to deploy.🚨

trigger:
- master
pool:
  vmImage: 'ubuntu-latest'
steps:
- task: TerraformTaskV1@0
  displayName: Terra Init
  inputs:
    provider: 'azurerm'
    command: 'init'
    workingDirectory: $(System.DefaultWorkingDirectory)
    backendServiceArm: 'ServiceConnectionName'
    backendAzureRmResourceGroupName: 'common-services-miel'
    backendAzureRmStorageAccountName: 'mielstorage001'
    backendAzureRmContainerName: 'configman'
    backendAzureRmKey: 'tf/terraform.tfstate'
- task: TerraformTaskV1@0
  displayName: Terra Destroy
  inputs:
    provider: 'azurerm'
    command: 'destroy'
    workingDirectory: $(System.DefaultWorkingDirectory)
    environmentServiceNameAzureRM: 'ServiceConnectionName'
- task: TerraformTaskV1@0
  displayName: Terra Plan
  inputs:
    provider: 'azurerm'
    command: 'plan'
    workingDirectory: $(System.DefaultWorkingDirectory)
    environmentServiceNameAzureRM: 'ServiceConnectionName'
- task: TerraformTaskV1@0
  displayName: Terra Apply
  inputs:
    provider: 'azurerm'
    command: 'apply'
    workingDirectory: $(System.DefaultWorkingDirectory)
    environmentServiceNameAzureRM: 'ServiceConnectionName'

Now Push your local folder into your Git Repo.

Create a New Pipeline

In Azure Devops, go to your Project — Pipelines and click New Pipeline (Top right corner).
Point to the Git Repo containing your Template, and select Existing Azure Pipelines YAML file, select the .yml file you just created

At this point, you can just save and queue the Pipeline.
If all went well, you will see output that resembles something like below.

If you nailed it at the first try, kudos 👍, if not (like me), go to the troubleshooting section where I’ll give some troubleshooting advice.

Super satisfying all-green-build

5 — Troubleshooting

Here are some errors I have encountered while trying to set up this demo.

Error: ##[error]Error: Input required: backendServiceArm

Solution: in the TerraformTaskV1 task, provide all backend* inputs

Error: ##[error]Error: There was an error when attempting to execute the process ‘/usr/local/bin/terraform’. This may indicate the process failed to start.

Solution: Make sure your paths are correct. Note that in Linux you have to use front slashes ‘/’

Error: “features”: required field is not set

Solution: This happened to when I copied an existing Template from the Interwebs. Apparantly you need to specify a features {} key in the provider block (I’ve included it in my example)

Error: Error: expected application_type to be one of [web other java MobileCenter phone store ios Node.JS], got Web
on functionapp.tf line 19, in resource "azurerm_application_insights" "test":
19: resource "azurerm_application_insights" "test" {

Solution: Same as above, Terraform is apparantly case sensitive, and I had to change the application_type from Web to web

Worth mentioning:

Make sure your Init task made connection with the Azure backend. Otherwise you won’t have a state file saved in the Cloud (the Blob Container). This State file allows your next Run of the Pipeline to manage the created Resources.

Without the saved State file, you will get errors like:

‘a resource with ID “x” already exists’

This is solved by ensuring the terraform { backend “azurerm” {} } block in the beginning of the Template file.

Pipeline is succesfully connected with the backend

With the State saved in the Cloud, every destroy step removes the previous created resources and thus prevent the ‘Id already exists’ problem (unless somebody manually created a Resource with this ID, but who is creating things manually anyways right?🤠).
If you skip the destroy task, an apply will only update the Resource if the properties of the Resource are changed.