An Amazon Machine Image (AMI) is an AWS resource we use to deploy Amazon EC2(Elastic Compute Cloud) instances. Ideally, an AMI contains the Operating System(Windows, Linux etc.) and a list of customization made on it to meet specific requirements. It helps to quickly bootstrap one or more EC2 instances with similar configuration.
As per Amazon documentation, An AMI includes the following:
- One or more EBS snapshots, or, for instance-store-backed AMIs, a template for the root volume of the instance (for example, an operating system, an application server, and applications).
- Launch permissions that control which AWS accounts can use the AMI to launch instances.
- A block device mapping that specifies the volumes to attach to the instance when it’s launched.
In my previous few posts I went through the process of creating Azure VM images for Windows and Linux platform. You may refer to those here:
- Create Azure Linux VM Image with Packer and Powershell
- Create Azure Windows VM Image with Packer and Powershell
In this post I will share the process of creating a custom private AMI from an Amazon Linux AMI using HashiCorp Packer with set of customization that I need in my baseline image for new EC2 instances. Note, I will share only a few customization to keep it simple. Feel free to add your list of changes either as Inline script in the provisioner section of the template JSON file or create a separate bash script and use it there. You can also add script blocks in the user data section.
This process requires a set of credential(Access Key and Secret Access Key) with required role assignments to perform the Packer builder tasks. So, make sure you have those ready. Refer IAM role for EC2 document while setting up your permission.
You also, need to install and import AWSPowershell Module on your system. You can skip this if you already have it setup.
Next important task is to decide how you are going to refer those access key and secret key in your script. I am running this script from a Windows system and set up Environment variables for those secrets. I will refer them in the script. You can follow the same process or you may use a file with secrets stored or AWS profile or some alternate method. It is absolutely on your convenience and security practice that your organization allows.
So, to set those secrets as persistent environment variables run following commands on Powershell console :
[System.Environment]::SetEnvironmentVariable('AWS_ACCESS_KEY_ID', 'AKIAMYSAMPLEACCESSKEYEXAMPLE',[System.EnvironmentVariableTarget]::Machine)
[System.Environment]::SetEnvironmentVariable('AWS_SECRET_ACCESS_KEY', 'myyyyyyyyyyyysecrettttttttttttttttt',[System.EnvironmentVariableTarget]::Machine)AWS supports a list of standard Environment variables. Refer following document to learn more :
Environment Variables To Configure the AWS CLI
Next important consideration – I am using Amazon provided Linux image as my base image and the ssh user name for it is by default EC2-USER. You can use images provided by other vendors or even your own custom AMI as source image.
As, Packer will deploy a temporary EC2 instance to create image, you need to provide a VPC Id, Subnet Id and Security Group Id. If you are running behind a proxy, update your script to authenticate through it. Your Security Group must allow ssh(port 22) to the EC2 instance as Packer will use ssh to connect the temporary instance to configure it and run scripts to set your customization defined in provisioner.
With that let’s start scripting.
First, as my previous posts, I will use chocolatey to install packer on my Windows system. So, let’s set that :
try{
$checkChoco = choco --version
}
catch{
Write-Host "ERROR : $_" -Foregroundcolor Yellow
}
if([String]::IsNullOrEmpty($checkChoco)){
Set-ExecutionPolicy Bypass -Scope Process -Force
[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072
Invoke-Expression ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))
}Similarly, I will check if Packer is already there in my system. If not, will install using choco :
try{
$checkPacker = packer --version
}
catch{
Write-Host "ERROR : $_" -Foregroundcolor Yellow
}
try{
if([String]::IsNullOrEmpty($checkPacker)){
choco install packer
}
}
catch{
Write-Host "ERROR : $_" -Foregroundcolor Yellow
Exit
}Now, I am defining a name for the new AMI and then checking if an AMI with same name already exists in my AWS account. If I already have one, add some extra random characters to make it unique :
$amiName = "$($amiNamePrefix)-CentOs-$((Get-Date).ToString("dd-MMM-yyyy"))"
$awsAccount = Get-STSCallerIdentity | Select-Object Account -ExpandProperty Account
$amiDetails = Get-EC2Image -AccessKey $accessKey -SecretKey $secretKey -Region $region -Filter @{ Name="name"; Values=$amiName} -Owner $awsAccount -ErrorAction SilentlyContinue
if(!([String]::IsNullOrEmpty($amiDetails))){
$amiName = "$($amiName)-$(-join ((65..90) + (97..122) | Get-Random -Count 6 | ForEach-Object {[char]$_}))"
}
Next, we will generate the Template JSON file with all required information for Packer to create the AMI for us. It includes variables, builders and provisioners. Note, I assigned empty string to access key and secret key variables as those will be user defined variables coming from the build command parameters. You still can assign default values instead of nothing if you want. Rest of items are quite self-explanatory.
As mentioned above, update provisioner inline script with your own stuff or create a separate bash script and refer here. JSON template creation script block is there in the full script. Following script block will call Packer build process with parameters :
Try{
powershell packer build -var "access_key=$($accessKey)" -var "secret_key=$($secretKey)" $jsonTemplateFile
}
Catch
{
Write-Host "ERROR : $_" -Foregroundcolor Red
Exit
}
}Finally, I am checking if the AMI has been created successfully and then cleaning the JSON file.
Try{
Write-Host "Deleting Template JSON file"
if(Test-Path -Path $jsonTemplateFile){
Remove-Item -Path $jsonTemplateFile -Force
}
}
Catch{
Write-Host "ERROR : $_" -Foregroundcolor Red
}
==================================================
$amiDetails = Get-EC2Image -AccessKey $accessKey -SecretKey $secretKey -Region $region -Filter @{ Name="name"; Values=$amiName} -Owner $awsAccount -ErrorAction SilentlyContinue
if(!([String]::IsNullOrEmpty($amiDetails))){
Write-Host "New Image $($amiName)($($amiDetails.ImageId)) created successfully!" -ForegroundColor Green
}
else{
Write-Host "ERROR : Something went wrong! Please try again!" -ForegroundColor Red
}
Here is the complete script with all required statements and functions we discussed above:
Param(
[Parameter(Mandatory=$true)]
[String]$amiNamePrefix,
[Parameter(Mandatory=$false)]
[ValidateSet("eu-north-1","ap-south-1","eu-west-3","eu-west-2","eu-west-1","ap-northeast-2","ap-northeast-1","sa-east-1","ca-central-1","ap-southeast-1","ap-southeast-2","eu-central-1","us-east-1","us-east-2","us-west-1","us-west-2")]
[String]$region = "us-east-2",
[Parameter(Mandatory=$false)]
[String]$vpcId,
[Parameter(Mandatory=$true)]
[String]$subnetId,
[Parameter(Mandatory=$true)]
[String]$securityGroupId
)
# Function to create template definition json file for Packer
Function Create-JsonTemplate()
{
Try{
$input = @"
{
"variables": {
"access_key": "",
"secret_key": ""
},
"builders": [{
"type": "amazon-ebs",
"access_key": "{{user ``access_key``}}",
"secret_key": "{{user ``secret_key``}}",
"ami_name": "$($amiName)",
"vpc_id": "$($vpcId)",
"subnet_id": "$($subnetId)",
"security_group_id":"$($securityGroupId)",
"region": "$($region)",
"source_ami_filter": {
"filters": {
"virtualization-type": "hvm",
"name": "amzn-ami-hvm-*-x86_64-gp2",
"root-device-type": "ebs"
},
"owners": ["137112412989"],
"most_recent": true
},
"ssh_username": "ec2-user",
"instance_type": "t2.medium",
"ami_description": "Linux AMI for Avengers Team",
"disable_stop_instance": "false",
"ssh_timeout": "20m",
"communicator": "ssh",
"launch_block_device_mappings": [
{
"device_name": "/dev/xvda",
"volume_type": "gp2",
"delete_on_termination": true
}
],
"run_tags": {
"Name": "Packer Builder",
"OsType": "Linux",
"CreationDate": "$((Get-Date).ToString("dd-MMM-yyyy"))",
"Owner": "Avengers"
},
"tags": {
"CreationDate": "$((Get-Date).ToString("dd-MMM-yyyy"))",
"OsType": "Linux",
"Owner": "Avengers",
"Name": "$($amiName)"
}
}],
"provisioners": [{
"type": "shell",
"inline": [
"sudo yum update -y",
"sudo yum install -y httpd",
"sudo service httpd start"
]
}]
}
"@
$input | Out-File $jsonTemplateFile -Encoding ascii
}
Catch
{
Write-Host "ERROR : $_" -Foregroundcolor Red
Exit
}
}
# Function to create Image using Packer
Function Create-Image(){
Try{
powershell packer build -var "access_key=$($accessKey)" -var "secret_key=$($secretKey)" $jsonTemplateFile
}
Catch
{
Write-Host "ERROR : $_" -Foregroundcolor Red
Exit
}
}
# Function to remove resources created during the image creation process
Function Cleanup-Resources(){
Try{
Write-Host "Deleting Template JSON file"
if(Test-Path -Path $jsonTemplateFile){
Remove-Item -Path $jsonTemplateFile -Force
}
}
Catch{
Write-Host "ERROR : $_" -Foregroundcolor Red
}
}
# Check AWS Secrets are configured as Environment Variable. If not either set those or use altername method to pass those secrets.
# If you use a different method, like user Profile, File or any other method comment out below script block.
if((Test-Path -Path Env:\AWS_ACCESS_KEY_ID) -and (Test-Path -Path Env:\AWS_SECRET_ACCESS_KEY)){
$accessKey = $Env:AWS_ACCESS_KEY_ID
$secretKey = $Env:AWS_SECRET_ACCESS_KEY
}
else{
Write-Host "ERROR : AWS Credentials are not set as Environment Variables. Either set those or use different method to pass those values.If you use alternate method.Comment this Block" -ForegroundColor Yellow
Exit
}
# Check if passed VPCId exists in the AWS Account, if not Exit
$vpcInfo = Get-EC2Vpc -VpcId $vpcId -Region $region -AccessKey $accessKey -SecretKey $secretKey
if([String]::IsNullOrEmpty($vpcInfo)){
Write-Host "ERROR : Incorrect VPC ID Supplied. Exiting.."
Exit
}
# Check if passed SubnetId exists in the AWS Account, if not Exit
$subnetInfo = Get-EC2Subnet -SubnetId $subnetId -Region $region -AccessKey $accessKey -SecretKey $secretKey
if([String]::IsNullOrEmpty($subnetInfo)){
Write-Host "ERROR : Incorrect Subnet ID Supplied. Exiting.."
Exit
}
# Check if passed SecurityGroupId exists in the AWS Account, if not Exit
$sgInfo = Get-EC2SecurityGroup -GroupId $securityGroupId -Region $region -AccessKey $accessKey -SecretKey $secretKey
if([String]::IsNullOrEmpty($sgInfo)){
Write-Host "ERROR : Incorrect Subnet ID Supplied. Exiting.."
Exit
}
$jsonTemplateFile = "$($env:Temp)\image-template.json"
if(Test-Path $jsonTemplateFile){Remove-Item -Path $jsonTemplateFile -Force}
# Check if chocolatey is already installed on local system. If not, install.
try{
$checkChoco = choco --version
}
catch{
Write-Host "ERROR : $_" -Foregroundcolor Yellow
}
if([String]::IsNullOrEmpty($checkChoco)){
Set-ExecutionPolicy Bypass -Scope Process -Force
[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072
Invoke-Expression ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))
}
# Check if Packer is already installed.If not, install using chocolatey
try{
$checkPacker = packer --version
}
catch{
Write-Host "ERROR : $_" -Foregroundcolor Yellow
}
try{
if([String]::IsNullOrEmpty($checkPacker)){
choco install packer
}
}
catch{
Write-Host "ERROR : $_" -Foregroundcolor Yellow
Exit
}
# Generate AMI Name. Check if an AMI already exists with that name. If so, suffix name with random letters
$amiName = "$($amiNamePrefix)-CentOs-$((Get-Date).ToString("dd-MMM-yyyy"))"
$awsAccount = Get-STSCallerIdentity | Select-Object Account -ExpandProperty Account
$amiDetails = Get-EC2Image -AccessKey $accessKey -SecretKey $secretKey -Region $region -Filter @{ Name="name"; Values=$amiName} -Owner $awsAccount -ErrorAction SilentlyContinue
if(!([String]::IsNullOrEmpty($amiDetails))){
$amiName = "$($amiName)-$(-join ((65..90) + (97..122) | Get-Random -Count 6 | ForEach-Object {[char]$_}))"
}
Create-JsonTemplate
Create-Image
$amiDetails = Get-EC2Image -AccessKey $accessKey -SecretKey $secretKey -Region $region -Filter @{ Name="name"; Values=$amiName} -Owner $awsAccount -ErrorAction SilentlyContinue
if(!([String]::IsNullOrEmpty($amiDetails))){
Write-Host "New Image $($amiName)($($amiDetails.ImageId)) created successfully!" -ForegroundColor Green
}
else{
Write-Host "ERROR : Something went wrong! Please try again!" -ForegroundColor Red
}
Cleanup-Resources
Here are some screenshots from my last run:
Updating the temporary ec2 instance through yum update command :
Finally, creating the AMI from the temp ec2 :
Packer makes sure to terminate the temp ec2 after completing its process.
Here is the resultant AMI :
