Search

Showing posts with label aws cli. Show all posts
Showing posts with label aws cli. Show all posts

Tuesday, 19 March 2019

AWS Webinar: Testing and Deployment Best Practices for AWS Lambda-Based Applications

[This page contains the Q&A from the AWS webinar on “Testing and Deployment Best Practices for AWS Lambda-Based applications.

The details below are useful, but always use this information in combination with your own research.

Answers to the questions were provided by Eric Johnson of AWS during the webinar.

]


AWS SAM: AWS Serverless Application Model
The AWS Serverless Application Model (AWS SAM) is a model to define serverless applications. AWS SAM is natively supported by AWS CloudFormation and defines simplified syntax for expressing serverless resources. The specification currently covers APIs, Lambda functions and Amazon DynamoDB tables.
https://docs.aws.amazon.com/lambda/latest/dg/serverless_app.html



Q: What is your recommendation on creating and maintaining connection pools to RDS / POSTGRES databases that are needed in a Lamba Function?
A: If you are using Aurora Serverless look at the Data API. https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/data-api.html


Q: Can you use Cloud9 to debug SAM local?
A: Yes, however, C9 also offers a UI for debugging, that is based on SAM local - either work.


Q: I don't really understand the context object of a lambda function. What is it used for?
A: This object provides methods and properties that provide information about the invocation, function, and execution environment.
https://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-context.html


Q: How does debugging work with SAM?
A: You can set breakpoints and have a call stack available. It connects to the debug port provided by SAM Local. https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-using-debugging.html


Q: Is there any way to autogenerate sam template? or need to write it yourself?
A: Using "SAM init" on the SAM cli will create a base template for a project.


Q: How to do canary, an blue/green with serverless?
A: https://aws.amazon.com/blogs/compute/implementing-canary-deployments-of-aws-lambda-functions-with-alias-traffic-shifting/ You can do traffic shifting between your serverless functions. Just what Chris is explaining right now.


Q: If a lambda function is contained on a VPC, being that VPC different per stage… how can we define the VPC to be used for a certain Environment? (if using tags to deploy)
A: You can specify VPC through template parameters.


Q: How to daisy-chain lambdas together?
A: Step-functions allows you to write loosely coupled lambda functions together instead of creating dependency between them. More elegant and less glue code required;


Q: If running a serverless .net core app on a lambda function which exposes numerous endpoints does a cold start relate to the lambda function as a whole or per endpoint?
A: Cold start is specific to a Lambda function and not individual endpoints supported by the same Lambda function. For example: if one endpoint starts the Lambda execution environment, another call to an endpoint on the same lambda function will be warm and ready to go.


Q: Is there anyway to use OpenAPI to generate SAM template?
A: API Gateway resources can use OpenAPI documents to define the endpoints. You can also export your API Gateway to an OpenAPI doc from within the console.


Q: What is the best way to trigger N number of lambda functions at once? Should SQS be used?
A: Lambda functions can be triggered from SQS but it is not guaranteed they will be triggered at the same time. You can also broadcast to multiple Lambda functions via SNS.


Q: Is serverless a good idea to sue for a project where my users start using the app everyday between 8;30 to 9 A.m and shuts down at 5:30 pm?
A: As far as timing, Serverless would work well for this because you will only be charged for the invocations. There will be no resources sitting idle.


Q: is sam also a sub component of the aws cli or will it be?
A: SAM is not a sub-component of the AWS CLI, but DOES depend upon it. For example, SAM package is a wrapper for aws lambda package.


Q: will there be delay to execute the Lamda when user first start using in the morning? considering the cold start ?
A: The first time a Lambda function is invoked there will be a slight cold start delay. The length of this delay is dependant on multiple factors like memory, function size, language, and if it is in a VPC. Outside of a VPC the cold start is often in the milliseconds


Q: Do lambda layers work locally in node js? I've updated aws-cli, sam local, and pip, but this doesn't seem to work locally
A: Yes it does work. You do need the latest version of AWS CI and SAM. Updating AWS CLI requires the latest Python as well for the latest features. If you are on a MAC I have found that using Brew is the best way to install.


Q: Is the AWS SAM CLI suitable for use in automation?
A: Depends on the automation, but I would generally say it is best for development. Automation should be achieved through tools like CodePipeline, CodeBuild, CodeDeploy, etc


Q: Are there some example projects using AWS SAM?
A: The Serverless Application Repository has many examples to look at. https://aws.amazon.com/serverless/serverlessrepo/


Q: Where do the various environment configs live, in their own template files?
A: Configs can live several places. Parameter Store, Build Variables, Template Variables. If using a different account for environments I encourage the use of the parameter store


Q: It seems like when I use layers in sam templates. I need to push from a sam template first to create a new version of the layer. Then manually change the version number in the template to reference the new layer version. Is there a way to reference code that you are zipping up and sending to s3 to become a layer version in the lambda template. Then referencing that same template for zipping into s3 for your layer link?
A: For iterating purposes, if the layer is on the same template you can reference the logical name of the layer. It will grab the latest each time

Monday, 11 March 2019

AWS training links

Below are a short list of links to AWS training material that I’ve found extremely useful.
Comment Link
AWS Training / Learning library home page.
Free registration required.
Hundreds of modules to learn about the different AWS services.
An incredible resource which is amazingly free.
https://www.aws.training/

https://aws.amazon.com/events/awsome-day-2019/
AWS Cloud Practitioner Essentials course.
A great way to quickly learn the basics of the core services, through watching a series of structure videos with multiple choice questions at the end of each topic to enforce and consolidate your learning.
https://www.aws.training/learningobject/curriculum?id=27076
Qwiklabs.
A a great resource to learn cloud skills.  The courses provide sandbox environments to learn new skills.
There’s also free courses to learn to help learn some of the basics of the course services.
Qwiklabs covers both AWS and GCP.
https://www.qwiklabs.com/catalog?keywords=&cloud%5B%5D=AWS&format%5B%5D=any&level%5B%5D=any&duration%5B%5D=any&price%5B%5D=free&modality%5B%5D=any&language%5B%5D=any
AWS Monthly webinars.
These are a great resource to see what’s new, or gleam in depth insight into a particular service.\
These sessions change frequently as AWS services evolve, so also keep popping back to the page to see what’s new.
https://aws.amazon.com/about-aws/events/monthlywebinarseries/

ARCHIVE
https://aws.amazon.com/about-aws/events/monthlywebinarseries/archive/


AWS Documentation.
The trusty default go to when you need to reference something within AWS.
https://docs.aws.amazon.com/index.html#lang/en_us
AWS Youtube channelhttps://www.youtube.com/user/AmazonWebServices


AWS FREE TIER - when creating a new account -- does the free-tier time/window start as soon as the account is created, or as soon as a particular service is started?

The 12 Months starts from the account creation

https://aws.amazon.com/free/?all-free-tier.sort-by=item.additionalFields.SortRank&all-free-tier.sort-order=asc

Please see the FAQ - https://aws.amazon.com/free/free-tier-faqs/

As well as a link to an article discussing tracking free tier usage.

You are also able to leverage Cloudwatch and creating billing alarms

https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/tracking-free-tier-usage.html



Saturday, 18 February 2017

ZappySys SSIS Tasks - Importing data from SQL SERVER into Redshift


Recently, I've been using AWS (Amazon Web Services), primarily using the Redshift database as the platform for a Data Warehouse.
One of the challenges has been how to automate the data load from SQL SERVER, to Redshift.
There's various best practices - one of those is to store flatfile data in S3, prior to ingesting this into Redshift via the Redshift COPY command.
S3, or AWS S3 is a "simple storage service" offered by Amazon - the data is stored in "the cloud".
AWS offers AWS CLI (command line interface) to help achieve this task.
A rough guide would be to perform the following tasks
  1. Using SSIS (SQL SERVER INTEGRATION SERVICES) to read data from SQL SERVER, and create flatfiles.
  2. Using AWS CLI, push this data into an S3 bucket.
  3. Within Redshift - issue the COPY command, to pull the data from S3 into a Redshift table
  4. Optionally, issue another S3 command and move the files into an "Archive" folder - for better house keeping.
To achieve these 4 steps could be time consuming to automate.

This is where ZappySys comes in to help streamline this process, and cut out hours of development.
"ZappySys is a USA based software development company. 
We specialize in Data integration, Business Intelligence, 
Cloud computing and .Net Application Development."

In a nut-shell - ZappySys have developed SSIS task addon's that perform many different tasks - the one that's important for this import into Redshift is the Redshift Data Transfer task.
SSIS AWS Redshift Connector
This is equivalent to the SSIS Data Flow Task (DFT), but makes the whole process of connecting to the AWS account, Redshift cluster / database, using S3 as a flatfile staging environment a piece of cake.


SSIS Amazon AWS Redshift Data Transfer Task - Load from local file


Zappy even have recorded a straight-forward video on how to use this task to speed up the implementation.


Here's some more details (from http://zappysys.com/ to help summaries what they have to offer)

Why SSIS PowerPack?
  • Coding free, drag and drop high performance suite of Custom SSIS Components and SSIS Tasks
  • Used by many companies around the globe
  • Compatible with SQL Server 2005,2008, 2008 R2, 2012, 2014 and 2016
  • Support for 32bit and 64bit Server/Desktop OS
  • Easy to use, Familiar looks and feel and fully integrated in BIDS/SSDT
  • No license needed if you are developing/testing in BIDS/SSDT
  • Rapid update cycle and prompt support (See what’s new). We will help you over phone, web meetings and via email.
  • 30 days money back guarantee if not satisfied for whatever reason.

SSIS Integration Packs

SSIS Integration Pack offers cost effective way to accomplish your specific integration need. Click on the following link to find out more about each integration pack.

SSIS Tasks\Components by Category