boto3 and the Cloud Provider SDKs

If you write Python in a cloud team, you'll spend a lot of time calling cloud APIs. The three big providers ship official Python SDKs that all follow similar conventions; learn one and the others are easy.

AWS: boto3

pip install boto3

Credentials — the default chain

Never hardcode keys. boto3 looks (in order) at:

1. Constructor parameters (last resort, only for tests)
2. Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN)
3. ~/.aws/credentials + ~/.aws/config profiles
4. Container credentials (ECS task role)
5. EC2 instance profile / EKS IRSA / Lambda execution role
6. SSO / IAM Identity Center

In production, your code should rely on instance/pod identity — no static keys ever land on disk.

Two flavours of API

import boto3

# Client API
ec2 = boto3.client("ec2")
resp = ec2.describe_instances()
for reservation in resp["Reservations"]:
    for instance in reservation["Instances"]:
        print(instance["InstanceId"], instance["State"]["Name"])

# Resource API (higher-level, only for some services)
ec2_r = boto3.resource("ec2")
for instance in ec2_r.instances.all():
    print(instance.id, instance.state["Name"])

S3 examples

s3 = boto3.client("s3")

# Upload
s3.upload_file("local.txt", "my-bucket", "remote/key.txt")

# Download
s3.download_file("my-bucket", "remote/key.txt", "local.txt")

# Stream small object into memory
obj = s3.get_object(Bucket="my-bucket", Key="config.json")
content = obj["Body"].read().decode()

# List with paginator (essential — list_objects_v2 truncates at 1000)
paginator = s3.get_paginator("list_objects_v2")
for page in paginator.paginate(Bucket="my-bucket", Prefix="logs/"):
    for obj in page.get("Contents", []):
        print(obj["Key"], obj["Size"])

# Pre-signed URL — let a user upload/download without AWS credentials
url = s3.generate_presigned_url(
    "get_object",
    Params={"Bucket": "my-bucket", "Key": "report.pdf"},
    ExpiresIn=3600,
)

Paginators

Almost every "list" operation in AWS is paginated. Use get_paginator() rather than handling NextToken yourself:

paginator = boto3.client("ec2").get_paginator("describe_instances")
for page in paginator.paginate(Filters=[{"Name": "instance-state-name", "Values": ["running"]}]):
    for reservation in page["Reservations"]:
        for instance in reservation["Instances"]:
            print(instance["InstanceId"])

Waiters

ec2 = boto3.client("ec2")
resp = ec2.run_instances(ImageId="ami-...", MinCount=1, MaxCount=1, InstanceType="t3.micro")
instance_id = resp["Instances"][0]["InstanceId"]

waiter = ec2.get_waiter("instance_running")
waiter.wait(InstanceIds=[instance_id])
print("instance is running")

Waiters poll on a sensible interval and time out cleanly — much better than rolling your own loop.

Sessions and Profiles

import boto3

# Use a named profile from ~/.aws/credentials
session = boto3.Session(profile_name="staging", region_name="eu-west-1")
s3 = session.client("s3")

# Assume a role — common for cross-account access
sts = boto3.client("sts")
creds = sts.assume_role(
    RoleArn="arn:aws:iam::222222222222:role/CrossAccountRead",
    RoleSessionName="my-script",
)["Credentials"]

assumed = boto3.Session(
    aws_access_key_id=creds["AccessKeyId"],
    aws_secret_access_key=creds["SecretAccessKey"],
    aws_session_token=creds["SessionToken"],
)
s3 = assumed.client("s3")

Azure

Azure SDKs are split per service, all sharing azure-identity for auth. The pattern: pick up DefaultAzureCredential which works through env vars, managed identity, or az login automatically.

pip install azure-identity azure-storage-blob azure-mgmt-resource

from azure.identity import DefaultAzureCredential
from azure.storage.blob import BlobServiceClient
from azure.mgmt.resource import ResourceManagementClient

credential = DefaultAzureCredential()

blob = BlobServiceClient(
    account_url="https://mystorage.blob.core.windows.net",
    credential=credential,
)
for container in blob.list_containers():
    print(container.name)

# Management plane
sub_id = "00000000-0000-0000-0000-000000000000"
rm = ResourceManagementClient(credential, sub_id)
for rg in rm.resource_groups.list():
    print(rg.name, rg.location)

GCP

Google's SDKs are similarly split per product, sharing google.auth. Set GOOGLE_APPLICATION_CREDENTIALS to a service-account JSON file, or rely on workload identity in GKE / Cloud Run.

pip install google-cloud-storage google-cloud-compute

from google.cloud import storage, compute_v1

# Storage
client = storage.Client()
bucket = client.bucket("my-bucket")
blob = bucket.blob("hello.txt")
blob.upload_from_string("hello")
print(blob.download_as_text())

# Compute
instances = compute_v1.InstancesClient()
for instance in instances.list(project="my-proj", zone="us-central1-a"):
    print(instance.name, instance.status)

Common Pitfalls

• Forgetting paginators and only seeing the first 100/1000 results
• Hardcoded regions — let the SDK pick up AWS_REGION / Azure default region / GCP default project
• Silent exception swallow — wrap calls in try/except for the SDK-specific exception types (botocore.exceptions.ClientError, etc.)
• Throttling. Cloud APIs rate-limit; use the SDK's built-in retry config and exponential backoff for batch jobs.
• Mocking in tests: use moto for AWS, official mock libraries for Azure/GCP, or wrap calls behind your own interface so production code can be unit-tested without hitting the cloud.