Logo

AWS cloud automation with Python and Boto

Getting started with Python and boto

This is a hands on article on how to use Python with boto library to manage AWS resources in the cloud.

Just to make it clear from the start, boto has two versions, the older one referred simply as "boto" and the new one, the one we'll be using referred to as boto3. We will use boto3. In this article, when we refer to boto we generally mean boto3.

Also, we are using Python 3.

Resources used through out the article

Boto3 Github repository: https://github.com/boto/boto3

Boto3 documentation: Boto3 Docs

Prerequisites and overview

You will need:

Creating the Python virtual env and dependencies

We assume you have the folder ~/venvs where you store all your Python virtual enironments. If not, modify accordingly. On a Linux terminal, create the virtual env:
cd ~/venvs
python -m venv pyboto-getting-started
We're going to create a requirements.txt file where we store the PyPi dependencies, one per line, and add our boto3 library as a dependency From this point onwards, make sure you activate your virtual env and work with it activated:
source ~/venvs/pyboto-getting-started/bin/activate
To get our dependencies installed, we can simply execute:
pip install -r requirements.txt
While not required for the project, I usually install ipython within the project's virtualenv as it is an awesome interpretor and it facilitates quick experimentation. I also prefer to use pudb as a debugger, so I install that as well.
pip install ipython pudb

Create hello_boto.py and test the boto3 import

Now, first thing we should do is just create a file and import the boto3 library and execute to see if it works so far. This will check that we have setup the virtualenv correctly and that the library can be imported So, create hello_boto.py with just
import boto3
And execute with python hello_boto.py If everything works OK you should see no output.

AWS connection options

In order to connect to our AWS resources we usually need three pieces of information:

For the curious, there is indeed another way to connect to the AWS resources from an EC2 machine that has an IAM role assigned, without using the access key id and secret access key credentials, but we will not explore this option for now.

There are also 3 different ways to configure the AWS credentials (access key id and secret access key) and the default region. The first two options (environment variables and config files) work for both connecting via the aws cli (command line tool) and using the Python code with boto3, the third option works with boto3 only, which is the one we're focusing on in this article.

First we will present all three options, so read on, then we will explain how the Python with boto3 code will look like in each case, so bear with me.

1. Use the standard AWS environment variables

To achieve that you need to add in your .bashrc (or .zshenv file for ZSH users) these environment variables.

Make sure you replace the values of the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY with the actual credentials of your AWS IAM user. Also, do not forget to restart the terminal after adding the environment variables, or source the .bashrc (.zshenv) file.
export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
export AWS_DEFAULT_REGION=us-east-1

2. Use the aws configuration files

The credentials for connecting to AWS can be configured using configuration files also.

Be aware though, the environment variables take precedence over configuration files. So it's a bad idea to use option 1 and option 2 in the same time.

The configuration files need to be situated in a folder called .aws (do not forget the starting dot) which should be inside your home directory.

The credentials should be added in a ~/.aws/credentials file, for example:



Then the default region to use should be added in ~/.aws/config


3. Send the aws credentials and region as parameters to the boto3.session.Session constructor

Let's start coding

We'll continue on our file hello_boto.py and we assume you did use either option 1 (environment variables) or option 2 (configuration files) to setup your AWS credentials.

There are a few ways to connect with Python and boto3 to your AWS account, what I choose here is to explicitly create a session object, then from the session object I obtain a client object specific to each of the AWS services.

As creating the actual session does not validate the AWS credentials, in other words just by creating the session object we can't tell if the AWS credentials are good or not, so we need to call an AWS API and actually do something in our AWS account. I'll use the S3 service and create an S3 bucket to begin testing boto3 and the connection to AWS.

Without further ado:
import boto3


session = boto3.session.Session()
s3_client = session.resource('s3')
s3_client.create_bucket(Bucket='hello-boto')
As the S3 bucket name will need to be unique, globally, please choose a different name that is less probably to have been used before.

In the example below, boto3 is the name of the Python package that represents the boto3 library, and session is the name of the Python module.

Then, Session is the name of the class that is used to create session objects, which are responsible with configuring the connection to an AWS account.

When called without parameters, Session will look for the AWS credentials either in the regular environment variables (see above option 1) or in the configuration files (see above option 2).

Now, let's look how to use the credentials in the code (option 3) and pass them to the Session class as arguments:
import boto3

session = boto3.session.Session(
    aws_access_key_id='AKIAIOSFODNN7EXAMPLE',
    aws_secret_access_key='wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY',
    region_name='us-east-1')

s3_client = session.resource('s3')

s3_client.create_bucket(Bucket='hello-boto2')
Going forward we want to build our connection code in such a way that we pass explicitly the credentials and the region. This way, if the project gets bigger, we are able to switch from one AWS IAM user to another, by passing different aws access key id and aws secret access key, no matter how we obtain it. For this, we want to set them as environment variables, but we might as well set them with different names from the recommended AWS environment variables so we can fetch them explicitly with python. Last, but not least, we can make these optional, so we call a session without arguments, to still leave an option for using the configured credentials as in option 1 or 2. Let's set the different environment variables as follows:

export HOCLI_AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export HOCLI_AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
export HOCLI_AWS_REGION_NAME=us-east-1
    
Now, to avoid confusion, remove any existing environment variables related with AWS. Our code becomes: