How to build a decoupled environment using SQS, SNS, and SES

What these services are and the role of each in your environment

Photo by Christian Lue on Unsplash

Recently I wrote a piece about what components are, and why they are so important in helping you to build and maintain a decoupled environment where your services and applications are not completely dependent on each other.

As a quick recap, a decoupled environment in a nutshell is where you have some compute, storage, database, or other services, that are all working together, but if one fails the others will continue to function as normal while you restore service to your failed component.

However in order to work in harmony, your services need to be able to communicate with each other, you as the admin, and your end users. This is where the AWS services of SQS, SNS, and SES come into play. Lets take a quick look at what each of these services are before we go into how they can integrate with your architecture.

SQS stands for Simple Queue Service, which is a service that handles the delivery of messages between components. It is capable of sending, storing, and receiving messages at scale without dropping message data. It works off of a polling based model, and has 3 main components:

• Producer — Responsible for sending message to your queue
• Queue — As the name says, a queue that holds messages
• Consumers — Responsible for retrieving messages from queue and marking them as consumed

One final thing to note with SQS is that there are 3 kinds of queues you can utilize:

• Standard Queue — Default type, guarantee at least once delivery, but not in a specific order. Offers best effort in trying to preserve message ordering and almost unlimited number of transactions per second
• First In First Out (FIFO) queue — Order of messages is maintained and there are no duplicates. Limited to 300 transactions per second, and up to 3000 if you utilize batching.
• Dead letter queues — A queue where messages that fail to be processed or consumed go, so you can further analyze the root problem of why it was not processed. Must be same queue type as source it is used against (Standard or FIFO)

SNS stands for Simple Notification Service, which is a publish/subscribe messaging service centered around topics. Users or endpoints can subscribe to a topic to receive a notification when a message is published, which usually occurs when a certain event you specify takes place. Supported subscribers to SNS topics can include the following:

• HTTP/S
• Email
• JSON
• SQS
• Application
• Lambda
• SMS

You can specify which protocol subscribers use, and control which users have access to which topics.

SES stands for Simple Email Service, which is a service that handles an automated email system to communicate with customers. It is commonly used in the fields of marketers and developers, for example it could be used to send confirmation emails upon account registration or order confirmations.

SES will automatically scan for spam and viruses and reject messages from untrusted sources, via recipient based control or IP Address based control. It can be used to trigger actions from other services such as S3, SNS, Lambda, or Workmail.

Now that we have an idea of what each of these 3 services are, lets take a look at an example of how they can facilitate communication between services in a decoupled environment.

Imagine you have a relatively simple web app where you sell hair products. An example architecture of this might look as follows:

To put it simply, you have a presentation layer on EC2 that hosts your website, an application layer that processes orders, user registrations, and other app logic, and a couple of database instances on RDS to keep track of site users and product inventory.

A new user comes to your site and decides they like what they see, so they decide to sign up and purchase some products to try them out and see if they like them.

When they register, SES can both send them a confirmation email with a custom HTML page you define for it to send out, as well as send a notification to a new-user-registration topic you have configured to SNS. This message sent to the topic can contain information the user put into your site when they registered such as their name and email. Lets imagine you have an SQS “users” queue subscribed to this topic. The SQS queue can take in the details of the user information, and a Lambda function can then consume the messages once they become available, and write that information to your RDS instance in a users table.

Now when this new user goes to buy some products, you can follow a similar format. While logged into your site they place an order, the details of that order go to an SNS topic, which then go to an SQS “orders” queue that is subscribed to the topic that contains metadata such as user and order information. Lambda can again consume this message from your SQS queue, and do the appropriate operations on a products table in your RDS to fulfill the customers order. We are of course abstracting some steps here that would include checking that the product is still in stock, reducing the available quantity, etc.

The great thing here is that if for some reason your EC2 or RDS services were to experience outages, existing orders and user registrations will still reside in an SQS queue. So whenever your services come back online they can resume processing these messages as normal, making this a decoupled environment.

Hopefully this was helpful for you to understand how you can better leverage SQS, SNS, and SES. Each of them have their own quirks and use cases but there is virtually limitless potential between all of them.

There are of course other fine details that I would always recommend consulting AWS documentation on, in regards to making your services work together. An example of this would be setting up the proper IAM roles so your services have permission to communicate with each other.

Let me know if you have any feedback positive or negative, and as always good luck in your continued journey of cloud computing.