- AWS manages the ongoing operations and underlying infrastructure needed to reliability run and scale your message queues.
- You eliminate the complexity and administrative overhead associated with managing dedicated message-oriented middleware (MoM) and associated infrastructure.
Simple Queue Service
- Offers a reliable, highly-scalable hosted queue for storing messages as they travel between applications or microservices.
- It moves data between distributed application components and helps you decouple these components.
- Amazon SQS provides familiar middleware constructs such as dead-letter queues and poison-pill management.
- It also provides a generic web services API.
- Use Amazon SQS when you need each unique message to be consumed only once and for cases such as the following:
○Decoupling the components of an application
○Configuring individual message delay
○Dynamically increasing concurrency or throughput at read time
- Main Features of Amazon SQS
○Multiple producers and consumers
○Configurable settings per queue
○Variable message size
Basic Architecture of Amazon SQS
- There are three main actors in the overall system:
○The components of your distributed system
○Messages in the queues
Amazon SQS Queues
- Basic Prerequisites
- Standard Queues : Amazon SQS stores copies of your messages on multiple servers for redundancy and high availability. On rare occasions, one of the servers that stores a copy of a message might be unavailable when you receive or delete a message, When you consume messages from the queue using short polling, Amazon SQS samples a subset of the servers and returns messages from just these servers.
○The following figure shows the short-polling behavior of messages returned after one of your system components makes a receive request.
Amazon SQS Queues
- FIFO (First-In-First-Out) Queues
○FIFO (First-In-First-Out) queues are designed to enhance messaging between applications when the order of operations and events is critical, or where duplicates can’t be tolerated.
○The most important features of this queue type are FIFO (First-In-First-Out) delivery and exactly-once processing.
○If multiple messages are sent in succession to a FIFO queue, each with a distinct message deduplication ID, Amazon SQS stores the messages and acknowledges the transmission.
○When receiving messages from a FIFO queue with multiple message group IDs, Amazon SQS first attempts to return as many messages with the same message group ID as possible.
○FIFO queues allow the producer or consumer to attempt multiple retries.
○The Amazon SQS Buffered Asynchronous Client doesn’t currently support FIFO queues.
Amazon SQS Queues
- Queue and Message Identifiers
○Queue Name and URL
■When you create a new queue, you must specify a queue name that is unique within the scope of all your queues.
■Each message receives a system-assigned message ID that Amazon SQS returns to you in the SendMessage response.
■This identifier is useful for identifying messages.
■Every time you receive a message from a queue, you receive a receipt handle for that message.
■This handle is associated with the action of receiving the message, not with the message itself.
○Message Deduplication ID
■If a message with a particular message deduplication ID is sent successfully, any messages sent with the same message deduplication ID are accepted successfully.
○Message Group ID
■Messages that belong to the same message group are always processed one by one, in a strict order relative to the message group.
■The large, non-consecutive number that Amazon SQS assigns to each message.
- Resources Required to Process Messages
○To help you estimate the resources you need to process queued messages, Amazon SQS can determine the approximate number of delayed, visible, and not visible messages in a queue.
- Visibility Timeout
○To prevent other consumers from processing the message again, Amazon SQS sets a visibility timeout, a period of time during which Amazon SQS prevents other consuming components from receiving and processing the message.
■A message is considered to be in flight after it’s received from a queue by a consumer, but not yet deleted from the queue.
■The following figure illustrates the visibility timeout.
Amazon SQS Queues
○Configuring the Visibility Timeout
■The visibility timeout clock starts ticking once Amazon SQS returns the message.
■During that time, the component processes and deletes the message.
■Each queue starts with a default setting of 30 seconds for the visibility timeout.
○Changing a Message’s Visibility Timeout
■You can shorten or extend a message’s visibility by specifying a new timeout value using the ChangeMessageVisibility action.
- Message Lifecycle
○The following diagram describes the lifecycle of an Amazon SQS message, from creation to deletion.
○Component 1 sends Message A to a queue, and the message is distributed across the Amazon SQS servers redundantly.
○When Component 2 is ready to process a message, it consumes messages from the queue, and Message A is returned.
○While Message A is being processed, it remains in the queue and isn’t returned to subsequent receive requests for the duration of the visibility timeout. Component 2 deletes Message A from the queue to prevent the message from being received and processed again once the visibility timeout expires.
- Amazon SQS Dead-Letter Queues
○A dead-letter queue is a queue that other (source) queues can target for messages that can’t be processed (consumed) successfully.
○Messages that can’t be processed in a timely manner are delivered to a dead-letter queue.
○Setting up a dead-letter queue allows you to do the following:
■Configure an alarm for any messages delivered to a dead-letter queue.
■Examine logs for exceptions that might have caused messages to be delivered to a dead-letter queue.
■Analyze the contents of messages delivered to a dead-letter queue to diagnose software or the producer’s or consumer’s hardware issues.
■Determine whether you have given your consumer sufficient time to process messages.
○ Use of a Dead-Letter Queue
■Dead-letter queues can help you troubleshoot incorrect message transmission operations with standard queues.
■To decrease the number of messages and to reduce the possibility of exposing your system to poison-pill messages
- Amazon SQS Message Attributes
○Message attributes allow you to provide structured metadata items about the message.
○Each message attribute consists of the name, type and value.
- Amazon SQS Long Polling
○Long polling helps reduce your cost of using Amazon SQS by reducing the number of empty responses and eliminating false empty responses.
- The Differences Between Short and Long Polling
○Amazon SQS uses short polling by default, querying only a subset of the servers to determine whether any messages are available for inclusion in the response.
- Amazon SQS Delay Queues
○Delay queues let you postpone the delivery of new messages in a queue for the specified number of seconds.
○Any message that you send to that queue is invisible to consumers for the duration of the delay period.
○Delay queues are similar to visibility timeouts because both features make messages unavailable to consumers for a specific period of time.
○The following figure illustrates the relationship between delay queues and visibility timeouts.
- Amazon SQS Message Timers
○Amazon SQS message timers allow you to specify an initial invisibility period for a message that you add to a queue.
○To set a delay period that applies to all messages in a queue, use delay queues.
○A message timer setting for an individual message overrides any Delay Seconds value that applies to the entire delay queue.
Using JMS with Amazon SQS
- The Amazon SQS Java Messaging Library is a JMS interface for Amazon SQS that lets you take advantage of Amazon SQS in applications that already use JMS.
- The interface lets you use Amazon SQS as the JMS provider with minimal code changes.
- Together with the AWS SDK for Java, the Amazon SQS Java Messaging Library lets you create JMS connections and sessions, as well as producers and consumers that send and receive messages to and from Amazon SQS queues.
Monitoring Amazon SQS using CloudWatch
- Amazon SQS and Amazon CloudWatch are integrated so you can use CloudWatch to view and analyze metrics for your Amazon SQS queues.
- CloudWatch metrics for your Amazon SQS queues are automatically collected and pushed to CloudWatch every five minutes.
- These metrics are gathered on all queues that meet the CloudWatch guidelines for being active.
Authentication and Access Control
- Amazon SQS has its own resource-based permissions system that uses policies written in the same language used for AWS Identity and Access Management (IAM) policies.
- In addition to creating IAM users with their own security credentials, IAM also allows you to grant temporary security credentials to any user, allowing the user to access your AWS services and resources.
Protecting Data Using Server-Side Encryption (SSE) and AWS KMS
- SSE lets you transmit sensitive data in encrypted queues.
- SSE protects the contents of messages in Amazon SQS queues using keys managed in the AWS Key Management Service (AWS KMS).
- The messages are stored in encrypted form and Amazon SQS decrypts messages only when they are sent to an authorized consumer.
- Encrypting a message makes its contents unavailable to unauthorized or anonymous users.
- The data encryption key (DEK) is used which is responsible for encrypting the contents of Amazon SQS messages.
- The length of time, in seconds, for which Amazon SQS can reuse a data key to encrypt or decrypt messages before calling AWS KMS again is called data key reuse period.
Amazon SQS Compliance
- PCI DSS
○Amazon SQS supports the processing, storage, and transmission of credit card data by a merchant or service provider, and has been validated as compliant with Payment Card Industry (PCI) Data Security Standard (DSS).
○AWS has expanded its HIPAA compliance program to include Amazon SQS as a HIPAA Eligible Service.