AWS SQS
A SQS event source tells maelstromd to poll a SQS queue for messages. If a message is received, the related
component is activated and the message body is sent to the component via HTTP POST. The message body is sent as the
POST body. If the component responds with a 200 status code, the SQS message is deleted from the queue. Otherwise
the message will stay in the queue and become eligible for redelivery when the visibility timeout elapses.
While SQS has no native notion of priority queueing, this event source optionally supports polling a set of queues
that share a common name prefix. For example, given an event source configured with nameasprefix: true and
a set of queues named:
resize-image-0
resize-image-1
resize-image-2
maelstromd will poll the queues in name order, draining the "0" queue completely, then "1", then "2".
While driaining lower priority queues, maelstromd will periodically reset and poll from the head of the list
in order to re-check the higher priority queues.
Concurrency
You can control the number of messages maelstromd will dequeue at a time. For example, if you want your system
to process no more than 10 messages concurrently, set maxconcurrency: 10 on the SQS event source.
This is different than the maxconcurrency setting on the component. The component setting specifies max concurrency
per instance of a component across all event sources, where the SQS setting specified the max concurrency across ALL
instances of the component that will originate from this event source.
If you are using a SQS FIFO queue you may also wish to set concurrencyperpoller: 1 to ensure that messages are
processed in the strict order they are dequeued.
The max number of pollers = ceil(sqs-maxconcurrency / concurrencyperpoller)
Back pressure
If you return a HTTP header pause-seconds, maelstromd will sleep for that number of seconds before polling SQS again.
The header value must be an integer (e.g.: pause-seconds: 30).
The poller will pause once for the specified number of seconds, then clear this value.
Properties
| Property | Description | Default |
|---|---|---|
| queuename | Name of queue (or queue prefix) to poll | None (required) |
| nameasprefix | If true, treat queuename as prefix. Poll all queues starting with that name | false |
| path | Request path to POST to on component | None (required) |
| maxconcurrency | Max concurrency messages to process via SQS across all instances | 10 |
| messagesperpoll | Messages to receive per polling attempt (1..10) | 1 |
| concurrencyperpoller | Concurrent messages to process per polling process | messagesperpoll |
| visibilitytimeout | Seconds to mark message invisible before it is eligible to dequeue again | 300 |
Examples
Example 1: Poll a single queue and process up to 15 messages concurrently
components:
mywebapp:
image: example/myapp
eventsources:
process_order_sqs:
sqs:
queuename: pending-orders
path: /queues/pendingorder
maxconcurrency: 15
visibilitytimeout: 600
Example 2: Poll a set of queues with a name prefix. Pull 10 messages at a time per poller.
components:
mywebapp:
image: example/myapp
eventsources:
process_order_sqs:
sqs:
queuename: resize-image-
nameasprefix: true
path: /queues/resizeimage
maxconcurrency: 50
messagesperpoll: 10
concurrencyperpoller: 10