Deploying a Multi-Account Cloud WAN Infrastructure with VPC attachments using Terraform, and GitLab

Components:

For specifications and available options, the following links should be useful:

AWS Cloud WAN Build, manage, and monitor global wide area networks

AWS Resource Access Manager Simply and securely share your AWS resources across multiple accounts

HashiCorp Terraform AWS Provider Use the Amazon Web Services (AWS) provider to interact with the many resources supported by AWS. Lifecycle management of AWS resources powered by the AWS Cloud Control API.

Gitlab CI CD Infrastructure as Code with Terraform and GitLab all tiers

Setting up git with ci/cd:

Create the following enviroment variables in the repo to be acccessed by CI/CD runner.

➜  cwman git:(master) glab variable list 
KEY                    PROTECTED  MASKED  SCOPE
AWS_SECRET_ACCESS_KEY  false      false   *    
AWS_ACCESS_KEY_ID      false      false   *    
TF_STATE_NAME          false      false   *    

Create .gitlab-ci.yml that specifies pipeline configuration. Manual action would be required for deploy and destory.

include:
  - template: Terraform/Base.gitlab-ci.yml  # https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Terraform/Base.gitlab-ci.yml
  - template: Jobs/SAST-IaC.gitlab-ci.yml   # https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Jobs/SAST-IaC.gitlab-ci.yml

stages:
  - validate
  - test
  - build
  - deploy
  - cleanup

fmt:
  extends: .terraform:fmt
  needs: []

validate:
  extends: .terraform:validate
  needs: []

build:
  extends: .terraform:build

deploy:
  extends: .terraform:deploy
  dependencies:
    - build
  environment:
    name: $TF_STATE_NAME

destroy:
  extends: .terraform:destroy

Create backend.tf which defines remote terraform backend managed with Gitlab HTTP backend. TF_STATE_NAME defines the state file name.

  terraform {
  backend "http" {
  }
}

Create provider.tf to specicify AWS providers and provider specifications. Note us-east-1. While cloudwan is a global construct, resource sharing for core networks is only available at us-east-1.

terraform {
  required_providers {
    aws   = { source = "hashicorp/aws" }
    awscc = { source = "hashicorp/awscc" }
  }

}

provider "aws" {
  region = "us-east-1"
}

Set up VPCs for VPC

Create vpc.tf to include a couple of VPCs. This example intends to be limited to this poc.

  resource "aws_vpc" "cwman_vpc_1" {
  cidr_block = "172.16.0.0/24"
  tags = {
    Name = "cwman_vpc_1"
  }
}

resource "aws_vpc" "cwman_vpc_2" {
  cidr_block = "172.16.1.0/24"
  tags = {
    Name = "cwman_vpc_2"
  }
}

resource "aws_subnet" "cwman_subnet_1" {
  vpc_id            = aws_vpc.cwman_vpc_1.id
  cidr_block        = "172.16.0.0/28"
  availability_zone = "us-east-1a"
  tags = {
    Name = "cwman_subnet_1"
  }
}

resource "aws_subnet" "cwman_subnet_2" {
  vpc_id            = aws_vpc.cwman_vpc_2.id
  cidr_block        = "172.16.1.0/28"
  availability_zone = "us-east-1b"
  tags = {
    Name = "cwman_subnet_2"
  }
}

Deploy Cloud WAN

Create global_network.tf that defines the global network

resource "aws_networkmanager_global_network" "cwman" {
  description = "cwman"
  tags        = { "Name" : "cwman" }
}

Create core_network.tf that specifies core network and attaches with the global network

resource "aws_networkmanager_core_network" "cwmancore" {
  global_network_id = aws_networkmanager_global_network.cwman.id
  description       = "cwmancore"
}

Create policy_document.tf specifies a Core Network policy document. This example specifies

• all the edge locations the core network will be active in: us-east-1
• creation of segments and specifications: flatnet, pita
• attachement policy towards segments, which matches variables while creating vpc attachements and assigning respective segments.

  data "aws_networkmanager_core_network_policy_document" "cwmanpolicy" {
  core_network_configuration {
    vpn_ecmp_support = true
    asn_ranges       = ["64515-64530"]
    edge_locations {
      location = "us-east-1"
    }
  }

  segments {
    name                          = "pita"
    description                   = "unflatearth"
    require_attachment_acceptance = true
  }
  segments {
    name                          = "flatnet"
    description                   = "flatearth"
    require_attachment_acceptance = true
  }

  segment_actions {
    action     = "share"
    mode       = "attachment-route"
    segment    = "pita"
    share_with = ["*"]
  }

  attachment_policies {
    rule_number     = 100
    condition_logic = "or"

    conditions {
      type     = "tag-value"
      operator = "equals"
      key      = "segment"
      value    = "flatnet"
    }
    action {
      association_method = "constant"
      segment            = "flatnet"
    }
  }
  attachment_policies {
    rule_number     = 200
    condition_logic = "or"

    conditions {
      type     = "tag-value"
      operator = "equals"
      key      = "segment"
      value    = "pita"
    }
    action {
      association_method = "constant"
      segment            = "pita"
    }
  }
}

Create policy_document_attachment.tf that enables policy document attachment with the core network.

resource "aws_networkmanager_core_network_policy_attachment" "cwmanpolicyattach" {
  core_network_id = aws_networkmanager_core_network.cwmancore.id
  policy_document = data.aws_networkmanager_core_network_policy_document.cwmanpolicy.json
}

Set up VPC attachements:

Create vpc_attachement.tf. Note segment tag which would cause policy condition match.

resource "aws_networkmanager_vpc_attachment" "cwman_vpc_1_attach" {
  subnet_arns     = [aws_subnet.cwman_subnet_1.arn]
  core_network_id = aws_networkmanager_core_network.cwmancore.id
  vpc_arn         = aws_vpc.cwman_vpc_1.arn
  tags            = { "segment" = "flatnet" }
}
resource "aws_networkmanager_vpc_attachment" "cwman_vpc_2_attach" {
  subnet_arns     = [aws_subnet.cwman_subnet_2.arn]
  core_network_id = aws_networkmanager_core_network.cwmancore.id
  vpc_arn         = aws_vpc.cwman_vpc_2.arn
  tags            = { "segment" = "pita" }

}

Create vpc_attachment_accepter.tf which actions attachement acceptance of a resource with a segment. Here out of 2 VPCs only one is defined, so while one will be attached, the second one will be in pending.

resource "aws_networkmanager_attachment_accepter" "cwman_vpc_1_accept" {
  attachment_id   = aws_networkmanager_vpc_attachment.cwman_vpc_1_attach.id
  attachment_type = aws_networkmanager_vpc_attachment.cwman_vpc_1_attach.attachment_type
}

At this point the network is locally available within the account scope.

Set up multi account attachments using AWS RAM

Create resource_share.tf which defines a resource share which will compromise of core network & accounts.

resource "aws_ram_resource_association" "cwman_share_association" {
  resource_arn       = aws_networkmanager_core_network.cwmancore.arn
  resource_share_arn = aws_ram_resource_share.cwman_share.arn
}

Create resource_share_association which associates core network with resoure share.

resource "aws_ram_resource_association" "cwman_share_association" {
  resource_arn       = aws_networkmanager_core_network.cwmancore.arn
  resource_share_arn = aws_ram_resource_share.cwman_share.arn
}

Create resource share pricipal resource_share_principal.tf which defines sharing scope. This can contain specific account IDs or the entire organisation. The following shares the resource with an entire organisation.

resource "aws_ram_principal_association" "cwman_share_principal" {
  principal          = "arn:aws:organizations::<12-digit-org_master_account_ID>:organization/o-<org-ID>"
  resource_share_arn = aws_ram_resource_share.cwman_share.arn
}

VPC attachments can now be introduced from other accounts, by associating those VPCs with the shared core network.