FULL NOTES:https://docs.google.com/a/bondgifts.com/document/d/1Q_NR70HKny-0VdmUscXvTJo7P_QwoSlAYGD9yMSWyhc/edit?usp=sharing ONGOING ISSUES: SNS won’t publish to a FIFO Queue GENERAL NOTES: SNS: Amazon SNS allows you to push real-time notification messages to interested subscribers over multiple delivery protocols. (It uses a “push” mechanism, eliminating the need to periodically check or “poll” for updates.) Question: Messages are added to any queue subscribed to the topic, but you still have to call sqs.recieveMessage periodically to check get items off the queue. So, it seems like we’re still polling? SQS: Amazon SQS is a message queue service used by distributed applications* to exchange messages through a polling model** It’s main advantage is that it can be used to decouple sending and receiving components—without requiring each component to be concurrently available.*** USING THEM TOGETHER: By using Amazon SNS and Amazon SQS together, messages can be delivered to applications that require immediate notification of an event, and also persisted in an Amazon SQS queue for other applications to process at a later time. Fanout Pattern A message published to an SNS topic is distributed to a number of SQS queues in parallel. https://aws.amazon.com/blogs/aws/queues-and-notifications-now-best-friends/ WITH FANOUT- a message published to an SNS topic is distributed to a number of SQS queues in parallel(uploaded images could be processed as soon as a message was received so the recognition software could run simultaneously) SUPER SIMPLE DIAGRAM: SQS QUEUE ------> Subscribes to -------> SNS TOPIC THEN WHEN message published to topic: SNS -----------> sends SQS message to the subscribed queue (and all other queues) KEY CONCEPTS: SNS http://docs.aws.amazon.com/sns/latest/dg/AccessPolicyLanguage_KeyConcepts.html Topic: A topic is a communication channel to send messages and subscribe to notifications.(http://docs.aws.amazon.com/sns/latest/dg/CreateTopic.html) EndPoints A mobile app, web server, email address, or an Amazon SQS queue that can receive notification messages from Amazon SNS. HTTP or HTTPS: When you subscribe an endpoint to a topic, On notification, Amazon SNS sends an HTTP POST to the subscribed endpoints with the message as json Header and Body Parsing http://stackoverflow.com/questions/18484775/how-do-you-access-an-amazon-sns-post-body-with-express-node-js Request: JSON Subject, Message, Metadata http://docs.aws.amazon.com/sns/latest/dg/SendMessageToHttp.html Subscribing: http://docs.aws.amazon.com/sns/latest/dg/SendMessageToHttp.html#SendMessageToHttp.subscribe ***(Must be public IP) Amazon SNS will send a subscription confirmation- The message from Subscribe will contain a SubscribeURL- that URL will return an xml doc with the the ARN for that endpoint in the result (see step 1.1 - http://docs.aws.amazon.com/sns/latest/dg/SendMessageToHttp.html#SendMessageToHttp.prepare) WHY NOT JUST USE HTTP ENDPOINTS? BECAUSE... You may not want an external service to make connections to your hosts (firewall may block all incoming connections to your host from outside). Your end point may just die because of heavy volume of messages Guaranteed delivery. If you configure SNS to send messages to an http end point or email or SMS, several failures to send message may result in message being dropped. http://stackoverflow.com/questions/13681213/what-is-the-difference-between-amazon-sns-and-amazon-sqs SQS Queue: When you subscribe an endpoint to a topic, On notification, Amazon SNS sends an Amazon SQS message to the subscribed queue(looks similar to http) Messages will persist in an Amazon SQS queue for other applications to process at a later time(14 days) Permissions: http://docs.aws.amazon.com/sns/latest/api/API_AddPermission.html Private IP: Status Code: 403- fix: http://stackoverflow.com/questions/15001497/sns-publishing-to-multiple-ec2-instances Policy: Note: Give IAM user/group permission to perform the sns:Publish action on the topic MyTopic: http://docs.aws.amazon.com/sns/latest/dg/SendMessageToSQS.html Managing Access- Architectural Overview: http://docs.aws.amazon.com/sns/latest/dg/AccessPolicyLanguage_ArchitecturalOverview.html KEY CONCEPTS: SQS SQS QUEUE TYPES: FIFO: The order in which messages are sent and received is strictly preserved exactly-once processing***** No Duplicates A message is delivered once and remains available until a consumer processes and deletes i Up to 300 transactions per second STANDARD: https://aws.amazon.com/sqs/faqs/#fifo-queues Occasionally more than one copy of a message might be delivered out of order (because of highly-distributed architecture******) best-effort ordering*******- messages are generally delivered in the same order as they are sent. Nearly-unlimited number of transactions per second **IMPORTANT ISSUES: FIFO Queues can only be created where the location is Ohio, SNS doesn’t broadcast to FIFO Queues RELEVANT LINKS: Diff between standard & fifo: https://aws.amazon.com/sqs/faqs/#fifo-queues General Info: http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues.html How to Create a FIFO queue:(by default sqs queue’s are standard): http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-create-queue.html ARCHITECTURE-RECEIVING MESSAGES REDRIVE POLICY/DEAD LETTER QUEUES MESSAGE LOCK When a message is received, it becomes “locked” while being processed. https://aws.amazon.com/sqs/details/ SHARED QUEUES http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/acp-overview.html LINKS: **NOTE** You can subscribe an Amazon SQS queue to an Amazon SNS topic by using the Amazon SQS console or the API. SNS CONSOLE: https://console.aws.amazon.com/sns/v2/home?region=us-east-1#/home GETTING STARTED WITH SNS: https://aws.amazon.com/sns/getting-started/ HOW SNS AND SQS WORK TOGETHER/ARCHITECTURAL OVERVIEW http://docs.aws.amazon.com/sns/latest/dg/SendMessageToSQS.html’ GENERAL WEBHOOK ARCHITECTURE OVERVIEW AND QUESTIONS: http://blog.restcase.com/webhooks-role-in-the-api-world/ API NOTES:SNS http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/SNS.html CREATE TOPIC http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/SNS.html#createTopic-property Credentials: http://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/setting-credentials-node.html API NOTES: SQS STEPS: Create Queue: Example: http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/SQS.html MESSAGE SENDING: http://docs.aws.amazon.com/sns/latest/dg/SendMessageToSQS.html * POTENTIAL BUG- If you delete a queue, you must wait at least 60 seconds before creating a queue with the same name KEY CONCEPTS: AWS EC2, Instances Access Control Concepts: http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-creating-custom-policies.html#sqs-creating-custom-policies-key-concepts TUTORIALS/NPM MODULES: OFFICIAL: https://aws.amazon.com/sdk-for-node-js/ UNOFFICIAL: http://www.markcallen.com/node/aws-sns-and-sqs-example-using-nodejs/ https://matoski.com/article/snssqs-for-node-js/ * Distributed Applications: Fancy word for software that is executed or run on multiple computers within a network(or stored on servers/cloud computing). (vs: traditional applications that run on a single system, distributed applications run on multiple systems simultaneously for a single task or job) ** Polling Model: A system where a single server visits a set of queues in some order. A group of n queues are served by a single server, typically in a cyclic order. (Look more into how the server chooses when to progress to the next node if time allows) https://en.wikipedia.org/wiki/Polling_system *** (Note on component availability and the polling model) **** LOCAL HOST WILL POINT TO AMAZON’S SERVER- only public API’s can subscribe to a topic