Persisting Docker Volumes in ECS using EFS
Post • 4 min read
Last week we faced a new challenge to persist our Docker Volume using EFS. Sounds easy, right? Well, it turned out to be a bit more challenging than expected and we were only able to find a few tips here and there. That is why we wrote this post so others may succeed faster.
Before digging into the solution, let’s take a minute to describe our context to elaborate a bit more on the challenge.
First of all, we believe in Infrastructure as Code and thereby we use CloudFormation to be able to recreate our environments. Luckily Amazon provides a working sample and we got EFS working quite easily. The next part was to get Docker to use a volume from EFS. We got lucky a second time as Amazon provides another working sample.
We managed to combine these resources and everything looked alright, but a closer look revealed that the changes did not persist. We found one explanation for why it didn’t work. It appears that we mount EFS after the Docker daemon starts and therefore the volume mounts an empty non-existing directory. In order to fix that we did two things, first we orchestrated the setup and then we added EFS to fstab in order to auto-mount on reboot.
The solution looks a bit like the following:
EcsCluster:
Type: AWS::ECS::Cluster
Properties: {}
LaunchConfiguration:
Type: AWS::AutoScaling::LaunchConfiguration
Metadata:
AWS::CloudFormation::Init:
configSets:
MountConfig:
- setup
- mount
setup:
packages:
yum:
nfs-utils: []
files:
"/home/ec2-user/post_nfsstat":
content: !Sub |
#!/bin/bash
INPUT="$(cat)"
CW_JSON_OPEN='{ "Namespace": "EFS", "MetricData": [ '
CW_JSON_CLOSE=' ] }'
CW_JSON_METRIC=''
METRIC_COUNTER=0
for COL in 1 2 3 4 5 6; do
COUNTER=0
METRIC_FIELD=$COL
DATA_FIELD=$(($COL+($COL-1)))
while read line; do
if [[ COUNTER -gt 0 ]]; then
LINE=`echo $line | tr -s ' ' `
AWS_COMMAND="aws cloudwatch put-metric-data --region ${AWS::Region}"
MOD=$(( $COUNTER % 2))
if [ $MOD -eq 1 ]; then
METRIC_NAME=`echo $LINE | cut -d ' ' -f $METRIC_FIELD`
else
METRIC_VALUE=`echo $LINE | cut -d ' ' -f $DATA_FIELD`
fi
if [[ -n "$METRIC_NAME" && -n "$METRIC_VALUE" ]]; then
INSTANCE_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
CW_JSON_METRIC="$CW_JSON_METRIC { \"MetricName\": \"$METRIC_NAME\", \"Dimensions\": [{\"Name\": \"InstanceId\", \"Value\": \"$INSTANCE_ID\"} ], \"Value\": $METRIC_VALUE },"
unset METRIC_NAME
unset METRIC_VALUE
METRIC_COUNTER=$((METRIC_COUNTER+1))
if [ $METRIC_COUNTER -eq 20 ]; then
# 20 is max metric collection size, so we have to submit here
aws cloudwatch put-metric-data --region ${AWS::Region} --cli-input-json "`echo $CW_JSON_OPEN ${!CW_JSON_METRIC%?} $CW_JSON_CLOSE`"
# reset
METRIC_COUNTER=0
CW_JSON_METRIC=''
fi
fi
COUNTER=$((COUNTER+1))
fi
if [[ "$line" == "Client nfs v4:" ]]; then
# the next line is the good stuff
COUNTER=$((COUNTER+1))
fi
done <<< "$INPUT"
done
# submit whatever is left
aws cloudwatch put-metric-data --region ${AWS::Region} --cli-input-json "`echo $CW_JSON_OPEN ${!CW_JSON_METRIC%?} $CW_JSON_CLOSE`"
mode: '000755'
owner: ec2-user
group: ec2-user
"/home/ec2-user/crontab":
content: "* * * * * /usr/sbin/nfsstat | /home/ec2-user/post_nfsstat\n"
owner: ec2-user
group: ec2-user
commands:
01_createdir:
command: !Sub "mkdir -p /${MountPoint}"
mount:
commands:
01_mount:
command:
Fn::Join:
- ""
- - "mount -t nfs4 -o nfsvers=4.1 "
- Fn::ImportValue:
Ref: FileSystem
- ".efs."
- Ref: AWS::Region
- ".amazonaws.com:/ /"
- Ref: MountPoint
02_fstab:
command:
Fn::Join:
- ""
- - "echo \""
- Fn::ImportValue:
Ref: FileSystem
- ".efs."
- Ref: AWS::Region
- ".amazonaws.com:/ /"
- Ref: MountPoint
- " nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 0 0\" >> /etc/fstab"
03_permissions:
command: !Sub "chown -R ec2-user:ec2-user /${MountPoint}"
04_restart_docker_and_ecs:
command: !Sub "service docker restart && start ecs"
Properties:
AssociatePublicIpAddress: true
ImageId:
Fn::FindInMap:
- AWSRegionArch2AMI
- Ref: AWS::Region
- Fn::FindInMap:
- AWSInstanceType2Arch
- Ref: InstanceType
- Arch
InstanceType:
Ref: InstanceType
KeyName:
Ref: KeyName
SecurityGroups:
- Fn::ImportValue:
Ref: SecuritygrpEcsAgentPort
- Ref: InstanceSecurityGroup
IamInstanceProfile:
Ref: CloudWatchPutMetricsInstanceProfile
UserData:
Fn::Base64: !Sub |
#!/bin/bash -xe
echo ECS_CLUSTER=${EcsCluster} >> /etc/ecs/ecs.config
yum update -y
yum install -y aws-cfn-bootstrap
/opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --resource LaunchConfiguration --configsets MountConfig --region ${AWS::Region}
crontab /home/ec2-user/crontab
/opt/aws/bin/cfn-signal -e $? --stack ${AWS::StackName} --resource AutoScalingGroup --region ${AWS::Region}
DependsOn:
- EcsCluster
Here is what we did compared to the original AWS provided template:- extracted FileSystem EFS into another CF template and exported the EFS identifier so that we can use ImportValue
- added -p to the mkdir command just in case
- enhanced mount to use imported filesystem reference
- added mount to fstab so that we auto-mount on reboot
- recursive changed EFS mount ownership
- restarted Docker daemon to include mounted EFS and started ECS as it does not automatically restart when the Docker daemon restarts
- added ECS cluster info to ECS configuration
- added ECS agent security group so that port 51678 which the ECS agent uses is open
- added yum update just in case
- included launch configuration into auto scaling group for the ECS cluster and added depends on ECS cluster
Get in Touch.
Let’s discuss how we can help with your cloud journey. Our experts are standing by to talk about your migration, modernisation, development and skills challenges.
Ilja’s passion and tech knowledge help customers transform how they manage infrastructure and develop apps in cloud.
Ilja Summala
LinkedIn
Group CTO
