AWS cloud automation with Python and Boto
Getting started with Python and boto
This is a hands on article on how to use Python with boto library to manage AWS resources in the cloud.Just to make it clear from the start, boto has two versions, the older one referred simply as "boto" and the new one, the one we'll be using referred to as boto3. We will use boto3. In this article, when we refer to boto we generally mean boto3.
Also, we are using Python 3.
Resources used through out the article
Boto3 Github repository: https://github.com/boto/boto3
Boto3 documentation: Boto3 Docs
Prerequisites and overviewYou will need:
- An AWS account and potentially spend a small amount of money if you want to test for real the creation of AWS resources
- Basic knowledge of Python
- A desire to learn cloud automation
- I recommend using Linux or MacOS just because all the commands below are tested on a Linux. Alternatively use either a Linux VM, or GitBash or Windows Subsystem for Linux to get to a Linux-like environment on Windows
Creating the Python virtual env and dependenciesWe assume you have the folder ~/venvs where you store all your Python virtual enironments. If not, modify accordingly. On a Linux terminal, create the virtual env:
We're going to create a requirements.txt file where we store the PyPi dependencies, one per line, and add our
cd ~/venvs python -m venv pyboto-getting-started
boto3library as a dependency From this point onwards, make sure you activate your virtual env and work with it activated:
To get our dependencies installed, we can simply execute:
While not required for the project, I usually install ipython within the project's virtualenv as it is an awesome interpretor and it facilitates quick experimentation. I also prefer to use pudb as a debugger, so I install that as well.
pip install -r requirements.txt
pip install ipython pudb
Create hello_boto.py and test the boto3 importNow, first thing we should do is just create a file and import the boto3 library and execute to see if it works so far. This will check that we have setup the virtualenv correctly and that the library can be imported So, create hello_boto.py with just
And execute with
python hello_boto.pyIf everything works OK you should see no output.
AWS connection optionsIn order to connect to our AWS resources we usually need three pieces of information:
- aws access key id
- aws secret access key
- aws region
For the curious, there is indeed another way to connect to the AWS resources from an EC2 machine that has an IAM role assigned, without using the access key id and secret access key credentials, but we will not explore this option for now.
There are also 3 different ways to configure the AWS credentials (access key id and secret access key) and the default region. The first two options (environment variables and config files) work for both connecting via the aws cli (command line tool) and using the Python code with boto3, the third option works with boto3 only, which is the one we're focusing on in this article.
First we will present all three options, so read on, then we will explain how the Python with boto3 code will look like in each case, so bear with me.
1. Use the standard AWS environment variables
To achieve that you need to add in your .bashrc (or .zshenv file for ZSH users) these environment variables.Make sure you replace the values of the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY with the actual credentials of your AWS IAM user. Also, do not forget to restart the terminal after adding the environment variables, or source the .bashrc (.zshenv) file.
export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY export AWS_DEFAULT_REGION=us-east-1
2. Use the aws configuration files
The credentials for connecting to AWS can be configured using configuration files also.
Be aware though, the environment variables take precedence over configuration files. So it's a bad idea to use option 1 and option 2 in the same time.The configuration files need to be situated in a folder called .aws (do not forget the starting dot) which should be inside your home directory.
The credentials should be added in a ~/.aws/credentials file, for example:Then the default region to use should be added in ~/.aws/config
3. Send the aws credentials and region as parameters to the boto3.session.Session constructor
Let's start coding
We'll continue on our file hello_boto.py and we assume you did use either option 1 (environment variables) or option 2 (configuration files) to setup your AWS credentials.
There are a few ways to connect with Python and boto3 to your AWS account, what I choose here is to explicitly create a session object, then from the session object I obtain a client object specific to each of the AWS services.
As creating the actual session does not validate the AWS credentials, in other words just by creating the session object we can't tell if the AWS credentials are good or not, so we need to call an AWS API and actually do something in our AWS account. I'll use the S3 service and create an S3 bucket to begin testing boto3 and the connection to AWS.Without further ado:
As the S3 bucket name will need to be unique, globally, please choose a different name that is less probably to have been used before.
import boto3 session = boto3.session.Session() s3_client = session.resource('s3') s3_client.create_bucket(Bucket='hello-boto')
In the example below,
boto3 is the name of the Python package that represents the boto3 library, and
session is the name of the Python module.
Session is the name of the class that is used to create session objects, which are responsible with configuring the connection to an AWS account.
When called without parameters,
Session will look for the AWS credentials either in the regular environment variables (see above option 1) or in the configuration files (see above option 2).
Sessionclass as arguments:
Going forward we want to build our connection code in such a way that we pass explicitly the credentials and the region. This way, if the project gets bigger, we are able to switch from one AWS IAM user to another, by passing different aws access key id and aws secret access key, no matter how we obtain it. For this, we want to set them as environment variables, but we might as well set them with different names from the recommended AWS environment variables so we can fetch them explicitly with python. Last, but not least, we can make these optional, so we call a session without arguments, to still leave an option for using the configured credentials as in option 1 or 2. Let's set the different environment variables as follows:
import boto3 session = boto3.session.Session( aws_access_key_id='AKIAIOSFODNN7EXAMPLE', aws_secret_access_key='wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY', region_name='us-east-1') s3_client = session.resource('s3') s3_client.create_bucket(Bucket='hello-boto2')
Now, to avoid confusion, remove any existing environment variables related with AWS. Our code becomes:
export HOCLI_AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE export HOCLI_AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY export HOCLI_AWS_REGION_NAME=us-east-1