Deploying Packer Images with Dynamic Secrets from HashiCorp Vault

Using Packer to automate the build process for machine images is awesome. In fact I have found great success in automating image builds across many environments including VMware, AWS, Azure & GCP, but one thing has always been a bit of a concern for me…..storing the credentials. Of course each environment requires a different set of credentials and to easily access them I typically store them as environment variables or within a local file on my computer. In fact, doing something like this for my AWS deployments has been the norm.

$ export AWS_ACCESS_KEY_ID="awsaccesskey"
$ export AWS_SECRET_ACCESS_KEY="awssecretkey"

A better way

As the number of environments and set of credentials has been growing, I have found a need for storing and accessing them in a better way. Ideally I am looking to store credentials in a central location, look them up when needed for deployment and rotate them when done with the job at hand. I would like the ability to create a set of rotating credentials that can be called and used within my Packer automation workflows.

Enter Vault

HashiCorp, the creators of Packer, also have a secrets management product called Vault. Vault secures, stores, and tightly controls access to tokens, passwords, certificates, API keys, and other secrets in modern computing. Vault supports my idea of rotating credentials or as they call them ‘dynamic secrets’ to minimize the length of time a set of credentials should exist. Vault supports a number of secrets engines to ease integration including AWS, Azure, GCP, Databases and Active Directory. (There are a ton of other engines available, but these will get me started).

In this case, we will utilize Vault’s AWS secrets engine to generate dynamic, on-demand AWS access credentials for my Packer AMI builds.

Enabling The AWS Secrets Engine in Vault

The AWS secrets engine can be enabled via the Vault command line our UI. Below is a quick video capturing the steps using the UI.

vault_packer_awsengine.gif

  1. Enable the AWS Secrets Engine

  2. Configure the AWS Secrets Engine by providing AWS credentials that provide Vault the ability to dynamically manage IAM Users.

  3. Create a ‘Packer’ role and specify the minimum set of permissions Packer needs to build AMIs. This is done by attaching the appropriate policy to the Packer role, which can be cut/past from the policy outlined in Packer’s AWS AMI builder documentation.

  4. Generate dynamic credentials in AWS that will provide access to Packer to generate an AMI.

  5. Revoke the credentials at any time.

If so inclined, these same series of steps can be performed via the Vault command line.

Build Your AMI Using PACKER With Dynamic AWS Credentials

Now that we have the Vault and AWS integration working we are ready to utilize the dynamic credentials in our Packer build. We will leverage Packer’s user variables to pass in our dynamic secrets by means of a variable file called awskeys.json.

vault_packer_build.gif


  1. Generate a set of dynamic credentials for Packer within Vault.

  2. Save credentials to a variable file called awskeys.json. Vault provides these in JSON format which is what Packer expects.

    {
      "accessKey": "AKIAIMQUVKMCSRB5NEZA",
      "secretKey": "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",
      "leaseId": "aws/creds/packer/1pf3nMh8KEEJtQQGADFtGYAE"
    }

  3. Specify the correct variable names in the packer build template. The builder stanza of my packer template for creating a vBrisket AMI looks like:

    {
  "builders": [{
    "access_key": "{{ user `accessKey`}}",
    "secret_key": "{{ user `secretKey`}}",
    "type": "amazon-ebs",
    "region": "us-east-1",
    "source_ami": "ami-0f9cf087c1f27d9b1",
    "instance_type": "t2.medium",
    "ssh_username": "ubuntu",
    "ami_name": "vbrisket-image {{timestamp}}",
    "ami_description": "vBrisket Image",
    "ami_groups": ["all"]
  }]
    }

Run a Packer build specifying the variable and template file:

packer build -var-file=awskeys.json vbrisket.json

Vault will automatically expire or rotate these credentials based on their lease time so there is no harm for me to show you my credentials as they will be revoked and become utterly useless after they have served their purpose. This workflow can be further automated utilizing consul-template and/or envconsul but that is a subject for a different post. Happy building (securely).