Common CloudFormation Errors, and Common Solutions For Them

Know how to fix what breaks

In the world of tech, in any niche, it is a well known fact that things will and do break. No matter how good you are or how well you check over your work there are times where things will not go according to plan. This is the entire reason different deployment strategies exist, so that when mistakes are made it does not cause disruptions elsewhere.

One of the best practices you can do to avoid mistakes as much as you can, is to automate as many processes as possible. Computers are much less likely to commit errors than a human who didn’t get good sleep last night and doesn’t function optimally until they have had at least 2 cups of coffee.

In the world of AWS, this can be accomplished with nifty tools called CloudFormation templates. In a nutshell, CF templates allow you to define all elements of what resources you would like to provision (how much, how large, upon what conditions being met, in which region, etc.). Another nice thing is that templates, when formatted properly, are largely reusable so you don’t have to waste time creating them over and over (and subsequently, cause more errors by doing such repetitive work).

However as nice as CloudFormation and its templates are, like with anything else there are errors that you will run in to. I would like to run through a couple of them and also give you some common solutions to them so that when you do see them, you are able to resolve them much quicker.

Generic error messages you will get

At face value, upon deploying your new or updated cloud formation stack, there are a handful of errors you could potentially receive:

• UPDATE_ROLLBACK_IN_PROGRESS
• UPDATE_ROLLBACK_FAILED
• ROLLBACK_IN_PROGRESS
• ROLLBACK_FAILED
• UPDATE_FAILED
• DELETE_FAILED

The errors about being in progress are normal, but annoying. However, failures are something to be more concerned about. A Rollback or Update failed usually requires a stack deletion, and an outright Delete failed is about as bad as it gets. A Delete failed most of the time will require you to contact AWS support to request a hard delete for the stack. However once this occurs you will still need to manually clean up the associated resources. Your best bet to track them down would be to reference the templates being used in your stack.

For your information, CloudTrail can be further utilized to get more detailed information on what exactly is the problem, but that is a topic for another day. For now I would just like to give you some common solutions that you can run through and check that will hopefully save you some time.

1. Syntax and formatting

This may be the most common error, but luckily it is easily resolvable. Templates can be in either YAML or JSON format, and a format error just means that there is something syntactically off, it may be as simple as a misplaced bracket, comma, or indent. CloudFormation will generally tell you what line the error is on as well, so if you have relatively complex templates you do not have to dig forever to find the problem.

2. Missing required permissions

Templates are most of the time going to references multiple services that will work with and communicate with each other, however it is easy to forget that they require permissions to do so. The fix could be as easy as adding in a role or equivalent permission. For example you may have a Lambda function that you want to write to a database, but if that Lambda function doesn’t have access to your DB then an error will be thrown. To satisfy the principle of least privilege, it can be helpful in this case to give admin access to Lambda so it works and then systematically strip away permissions that are not needed, until you are left only with what is needed to make it work.

3. Dependency errors

With more complex architectures, there will come a time where certain resources require other resources to be created first. For example if you have some EC2 instances designated for a private subnet, you first will want to make sure that the VPC and the private subnet exist, otherwise it will fail. You can use the DependsOn attribute to specify the ID of a resource that needs to be created prior to starting this tagged resource. A tip to help you visualize what needs to be created and when is to use a visual planning tool (I personally prefer CloudCraft) before you get started deploying too many complex templates.

4. Region Specific Resources

Unfortunately some AWS resources are region specific, so it is hard to recycle templates and deploy them to another region as they currently are. Another limitation here is that perhaps some of your EC2 instances are using a particular AMI, however AMI’s are also region specific. In this case it does take a little work to copy your AMI to another region, but it can be done.

5. Resource limit reached

This can happen again in more complex architectures, where you have a large chunk of the same type of service resource being deployed in the same region. For example, EC2 has a limit of 20 instances per region, so if you hit this error you need to either start working in a different region, or you can contact AWS support to request a higher limit. You will need to provide them with information about how many you are expecting to need and within which region.

Hopefully this helps save you some time when deploying and updating CloudFormation stacks via templates. As with anything else tech related though there are a ton of other things that could potentially go wrong, so these issues are just the tip of the iceberg.

Luckily AWS has great documentation and FAQs out there so I would highly recommend plugging away at them for more solutions, as well as enabling CloudTrail (if you aren’t opposed to spending a little extra to save yourself some troubleshooting headaches) to help solve your problems.

As always, best of luck in your continued journey through Cloud Computing.