Extending CloudFormation with custom resource types

CATEGORIES

AWSTech Community

In this post I show how you can extend AWS CloudFormation with new resource types and use them in the same way as AWS native resources. And do this without running your own Lambda functions or EC2 instances like custom resources would require you to do.

Accessing information outside of CloudFormation stack

In the real world it is often not possible to have 100% of your cloud infrastructure defined and maintained in code. Ideally all your cloud infrastructure would be sharing the single state, and allow references to any resource needed.

In more realistic scenario your stack is one of many and some resources aren’t managed in code at all, or are created with different IaaC tool.

Making references to resources, or data, outside of the stack, has never been the strongest point of CloudFormation. You can pass information between stacks with export/import, nested stacks, SSM parameter store, stack sets or sometimes even with copy-pasting.

What has been missing is the ability to reference external resources same way as Terraform data sources do.

Extending CloudFormation

For a long time it has been possible to extend CloudFormation with custom resources. SAM templates made it possible to combine the infrastructure and logic of custom resource into single template, assuming your code was compact enough to fit into template and didn’t depend on libraries outside of standard lambda runtimes.

More modern way to expand CloudFormation is to use custom resource types. They are real 1st class citizens comparable to AWS provided resources. Major difference between these options is who is responsible of running the code.

You must have a lambda function or an EC2 instance for a custom resource but CloudFormation service runs your code for resource type. This makes resource types, combined with CloudFormation Registry, much easier to share and consume across multiple projects and AWS accounts.

I would recommend reading Managing resources using AWS CloudFormation Resource Types to understand how both models works.

Nordcloud::Dataprovider::Variable

What would be the most simple data provider to test resource type development? I came up with the idea of a pseudo resource that doesn’t do anything but allow setting the state when resource is created/updated and return the set value with GetAtt -call.

Here is a sample template using Nordcloud::Dataprovider::Variable -resource. This is to show how your resource types can be used in templates exact the same way as native AWS resources.

AWSTemplateFormatVersion: 2010-09-09
Description: Nordcloud-Dataprovider-Variable

Parameters:
  
  MyValue:
    Description: MyVar Content 
    Type: String
    Default: HelloWorld

Resources:

  MyVar:
    Type: Nordcloud::Dataprovider::Variable
    Metadata:
      Content: !Sub "Simple reference to ${MyValue}"

Outputs:
  
  Output:
    Description: Content of MyVar
    Value: !GetAtt MyVar.Content

Source code for Nordcloud::Dataprovider::Variable is available in GitHub.

Resource type development

Resource type development workflow is

  • Install CFN CLI and dependencies
  • Initialize a new project
    cfn init
  • Write the resource schema and generate handler skeletons
    cfn generate
  • Implement the logic in handler functions
  • Validate the resouce type
    cfn validate
  • Deploy the new version of resource type and set it as default
    cfn submit --set-default
  • Deploy a template using the resource type
  • Remember to cleanup old versions of resource type

Nordcloud::Dataprovider::Variable doesn’t have much code in handlers. All other handlers are really just a dummy functions returning success, except the read_handler that will return Content value of resource Metadata. Storing resource state in Metadata I didn’t need to deploy any AWS resources for storing value of variable.

What Next?

I have a long list of ideas for more serious and useful data providers. Finding an AMI, VPC or subnet ID based on given attributes, or mapping between HostedZoneName and HostedZoneId. Or maybe, instead of creating separate types for each use-case, build a generic data provider that can get attributes of any resource.

Resources

  • CloudFormation Provider Development Toolkit and repos for Java/Python/Go -plugins. Java seems to be the most mature language for resource type development and popular for AWS resources. It is more difficult to find good examples written in Python or Go.
  • Build your first AWS CloudFormation resource provider re:Invent 2020 session describes the details how resource types works. I found this helpful in understanding the callback mechanism that will be necessary for any non-trivial (de)provisioning processes.

For further insights follow Petri’s private blog in https://carriagereturn.nl

Blog

Building better SaaS products with UX Writing (Part 3)

UX writers are not omniscient, and it’s best for them to resist the temptation to work in isolation, just as...

Blog

Building better SaaS products with UX Writing (Part 2)

The main purpose of UX writing is to ensure that the people who use any software have a positive experience.

Blog

Building better SaaS products with UX Writing (Part 1)

UX writing is the process of creating all the copy and content of a digital experience.

Get in Touch

Let’s discuss how we can help with your cloud journey. Our experts are standing by to talk about your migration, modernisation, development and skills challenges.