Categories
DevOps & Infrastructure

AWS CloudFormation IaC Generator Hands-On

In this post, I have a hands-on with AWS CloudFormation‘s new IaC Generator feature and give my thoughts on the experience.

Table of Contents

Introduction

Earlier this month, AWS launched new CloudFormation features for importing existing AWS resources:

AWS CloudFormation launches a new feature that makes it easy to generate AWS CloudFormation templates and AWS CDK apps for existing AWS resources that are managed outside CloudFormation.

You can use the generated templates and apps to import resources into CloudFormation and CDK or replicate resources in a new AWS Region or account.

https://aws.amazon.com/about-aws/whats-new/2024/02/aws-cloudformation-templates-cdk-apps-minutes/

An announcement from AWS Ray Of Sunshine Matheus Guimaraes followed this:

I’ve suggested something like this to AWS before, so I was very keen to try IaC Generator with my recently created WordPress data pipeline resources!

Firstly, I’ll explain some of the fundamental concepts of CloudFormation and the IaC Generator. Next, I’ll run the generator on my AWS account and see what comes back. Finally, I’ll run some tests and give my findings and thoughts.

Services Primer

This section introduces AWS CloudFormation and the IaC Generator, and explains what they’re for.

AWS CloudFormation

AWS CloudFormation is an infrastructure as code (IaC) service that manages and provisions AWS resources using code and automation instead of manual processes. It can configure resources, handle dependencies and estimate costs as part of a reproducible version-controlled CI/CD pipeline.

CloudFormation has extensive functionality. For this post, the two concepts I’m most interested in are:

CloudFormation Templates: JSON or YAML formatted text files which serve as blueprints for building AWS resources. Templates form the basis of CloudFormation Stacks.

CloudFormation Stacks: Collections of AWS resources that are managed as a single unit. Creating, updating or deleting a stack directly influences the related resources. Stacks can represent constructs like serverless applications, web servers and event-driven architectures.

The CloudFormation User Guide has further reading with sample code.

CloudFormation IaC Generator

The IaC Generator is a CloudFormation feature designed for use with manually provisioned resources:

With the AWS CloudFormation IaC generator (infrastructure as code generator), you can generate a template using AWS resources provisioned in your account that are not already managed by CloudFormation. Use the template to import resources into CloudFormation or replicate resources in a new account or Region.

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/generate-IaC.html

The generator has three steps:

2024 02 25 IaCGeneratorGettingStarted

IaC Generator can be used from the CloudFormation console and the AWS CLI.

IaC Generator Hands-On

In this section, I have an IaC Generator hands-on using CloudFormation’s console. Let’s start with an account scan.

Account Scan

Firstly, IaC Generator needs to know about my AWS account’s resources. The generator then uses a completed scan to create a resource inventory which is valid for 30 days.

IaC Generator scans are limited per day. Accounts with under 10,000 resources can perform 3 daily scans, while larger accounts can perform a single daily scan. Like CloudFormation, IaC Generator is a regional service that scans resources in the selected region. Avoid wasted scans and unintended confusion by checking the selected region is correct before starting the scan!

In my account, IaC Generator found 209 resources in my main region in about two minutes.

2024 02 23 IaC GeneratorScan

I then ran a second scan in a different region to see what would happen. It found 71 resources, so a clear difference. Interestingly, resources from global services like S3 and IAM were included in both scans while regional services like Lambda and CloudWatch resources weren’t. So pay attention to the selected region!

With a successful scan, I can now make a template.

Template Generation

There are three steps to making an IaC Generator template:

  • Specify template details
  • Add scanned resources
  • Add related resources

Template Details

Firstly, the generator asks if it’s creating a new template or updating an existing one. Selecting the latter displays that region’s CloudFormation stacks, but I don’t have any yet so let’s move on.

Secondly, the generator asks for a template name and sets the stack DeletionPolicy and UpdateReplacePolicy attributes to Retain:

2024 02 13 CloudFormationTemplateDetails

These are important decisions, so let’s clarify what they mean and why they default to Retain.

The DeletionPolicy attribute controls what happens to resources when a stack is deleted. It has two options:

  • Delete: CloudFormation deletes the stack and the resources. Nothing is kept.
  • Retail: CloudFormation deletes the stack and keeps the resources and their data.

The UpdateReplacePolicy attribute controls what happens to a resource that must be changed during a stack update. For example, upgrading an EC2 instance or a Lambda function.

It has three options:

  • Delete: CloudFormation makes new resources and then deletes the old ones.
  • Retain: CloudFormation makes new resources and keeps the existing ones. The old resources are then removed from CloudFormation’s scope.
  • Snapshot: For resources that support snapshots like RDS and EBS, CloudFormation creates a snapshot for the resource before deleting it.

The Retain default for both policies ensures that no unintended resource deletions happen while using the generator. When using Retain, remember that CloudFormation-created resources cost the same as their manual counterparts and can get expensive if forgotten about!

The IaC Generator applies these attributes to every resource in the template. After creation, they can be changed for specific resources if needed.

Adding Resources

Next, IaC Generator shows the resources found in the most recent scan. For example:

2024 02 13 CloudFormationSelectedResources

This can be a big list, and can be filtered by:

  • Resource Type
  • Tag Key
  • Tag Value
  • Resource Identifier

So, filtering on Resource Type = bucket returns S3 buckets and S3 bucket policies, while Resource Identifier = rtb returns VPC route tables. Desired resources can then be selected, appearing in a list under the scan results.

Next, IaC Generator checks my selections and recommends related resources that enable service interactions or belong to the same workload. Here, it suggested two IAM roles for my Lambda function and EventBridge scheduler that I’d missed. This is great, as that would have caused big problems down the line!

Finally, IaC Generator summarises the selections and creates a CloudFormation template.

IaC Generator Outputs

In this section, I examine the template and notifications provided by IaC Generator following my hands-on.

Summary

After IaC Generator has created a template, three tabs are presented:

  • Template Definition: The IaC Generator template in YAML and JSON, with options to download it, copy it and import it to a CloudFormation stack.
  • Template Resources: A list of the template’s resource types, each resource’s CloudFormation Logical ID (which can be changed) and each resource’s status within the template. Resources can be added to the template and resynced if they’ve changed since the initial scan.
  • AWS CDK: A two-step process to generate a CDK application for the template using the cdk migrate command. This is out of scope for this post, but this is the generated Python command:
Bash
cdk migrate --stack-name wordpress-api-raw --from-path ./wordpress-api-raw.yaml --language python

So let’s take a look at the template!

Template Examination

The full template comes in at 500 lines! So I’ll pull out some sections and take a closer look instead of going through the whole thing. Firstly, one of my SNS topics. I’ve highlighted the FIFO setting, subscription protocol and topic name:

YAML
  SNSTopic00lambdageneralfailure00CeVCO:
    UpdateReplacePolicy: "Retain"
    Type: "AWS::SNS::Topic"
    DeletionPolicy: "Retain"
    Properties:
      FifoTopic: false
      Subscription:
      - Endpoint: "[REDACTED_EMAIL]"
        Protocol: "email"
      TracingConfig: "PassThrough"
      ArchivePolicy: {}
      TopicName: "lambda-general-failure"

Next, my data lakehouse S3 bucket. I’ve highlighted the bucket name, object ownership setting and default encryption setting:

YAML
  S3Bucket00datalakehouseraw[REDACTED_AWS_ACC_ID]00GtGFU:
    UpdateReplacePolicy: "Retain"
    Type: "AWS::S3::Bucket"
    DeletionPolicy: "Retain"
    Properties:
      PublicAccessBlockConfiguration:
        RestrictPublicBuckets: true
        IgnorePublicAcls: true
        BlockPublicPolicy: true
        BlockPublicAcls: true
      BucketName: "data-lakehouse-raw-[REDACTED]"
      OwnershipControls:
        Rules:
        - ObjectOwnership: "BucketOwnerEnforced"
      BucketEncryption:
        ServerSideEncryptionConfiguration:
        - BucketKeyEnabled: true
          ServerSideEncryptionByDefault:
            SSEAlgorithm: "AES256"

Finally, my EventBridge Schedule including the cron expression, IAM role and schedule name:

YAML
  SchedulerSchedule00daily0700006qr2E:
    UpdateReplacePolicy: "Retain"
    Type: "AWS::Scheduler::Schedule"
    DeletionPolicy: "Retain"
    Properties:
      GroupName: "default"
      ScheduleExpression: "cron(0 7 * * ? *)"
      Target:
        Arn:
          Fn::GetAtt:
          - "LambdaFunction00datawordpressapiraw006Ybdu"
          - "Arn"
        RetryPolicy:
          MaximumEventAgeInSeconds: 86400
          MaximumRetryAttempts: 3
        RoleArn:
          Fn::GetAtt:
          - "IAMRole00AmazonEventBridgeSchedulerLAMBDA35d568499400iuCzM"
          - "Arn"
      Description: ""
      State: "ENABLED"
      FlexibleTimeWindow:
        Mode: "OFF"
      ScheduleExpressionTimezone: "Europe/London"
      Name: "daily-0700"

In this example, CloudFormation will get the Lambda function and IAM role ARNs after they are created to avoid any EventBridge stack failures.

So everything’s fine, yes? Well…

Errors

Alongside the template, the Template Definition tab threw this warning:

2024 02 22 CloudFormationIaCGeneratorWarning

Uh, ok.

I was a bit thrown by this one so started investigating. View Warning Details had a lot to say, raising these Lambda issues:

2024 02 22 GeneratorResourceWarningsLambda

The warning’s Learn More link leads to IaC Generator and write-only properties. This, along with the accompanying page about updating AWS::Lambda::Function resources started to fill in the blanks:

The IaC Generator cannot determine which set of exclusive properties was applied to the resource during creation. For example, you can provide the code for a AWS::Lambda::Function using one of these sets of properties.

  • Code/S3BucketCode/S3Key, and optionally Code/S3ObjectVersion
  • Code/ImageUri
  • Code/ZipFile
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/generate-IaC-write-only-properties.html

The generator doesn’t know how the function code was provided, and can’t guess. So these warnings are included by design to stop IaC Generator from using the wrong function code:

When a generated template contains AWS::Lambda::Function resources, then warnings are generated stating that Code/S3Bucket and Code/S3Key properties are identified as MUTUALLY_EXCLUSIVE_PROPERTIES. In addition, the Code/S3ObjectVersion property receives a UNSUPPORTED_PROPERTIES warning.

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/generate-IaC-lambda-function.html

So what’s in the template’s Lambda code block exactly?

Lambda Function

This part of the template is my Lambda function:

YAML
  LambdaFunction00datawordpressapiraw006Ybdu:
    UpdateReplacePolicy: "Retain"
    Type: "AWS::Lambda::Function"
    DeletionPolicy: "Retain"
    Properties:
      MemorySize: 250
      Description: ""
      TracingConfig:
        Mode: "PassThrough"
      Timeout: 120
      RuntimeManagementConfig:
        UpdateRuntimeOn: "Auto"
      Handler: "lambda_function.lambda_handler"
      Code:
        S3Bucket:
          Ref: "LambdaFunction00datawordpressapiraw006YbduCodeS3BucketOneOfmPyvF"
        S3Key:
          Ref: "LambdaFunction00datawordpressapiraw006YbduCodeS3KeyOneOfLqj3f"
      Role:
        Fn::GetAtt:
        - "IAMRole00datawordpressapirawroleygxg4wz400zzf3d"
        - "Arn"
      FileSystemConfigs: []
      FunctionName: "data_wordpressapi_raw"
      Runtime: "python3.12"
      PackageType: "Zip"
      LoggingConfig:
        LogFormat: "Text"
        LogGroup: "/aws/lambda/data_wordpressapi_raw"
      EphemeralStorage:
        Size: 512
      Architectures:
      - "x86_64"

Some items of note:

  • Function name and CloudWatch Log Group:
YAML
      Role:
        Fn::GetAtt:
        - "IAMRole00datawordpressapirawroleygxg4wz400zzf3d"
        - "Arn"
      FileSystemConfigs: []
      FunctionName: "data_wordpressapi_raw"
      Runtime: "python3.12"
      PackageType: "Zip"
      LoggingConfig:
        LogFormat: "Text"
        LogGroup: "/aws/lambda/data_wordpressapi_raw"
      EphemeralStorage:
        Size: 512
      Architectures:
      - "x86_64"
  • Memory allocation and timeout config settings:
YAML
  LambdaFunction00datawordpressapiraw006Ybdu:
    UpdateReplacePolicy: "Retain"
    Type: "AWS::Lambda::Function"
    DeletionPolicy: "Retain"
    Properties:
      MemorySize: 250
      Description: ""
      TracingConfig:
        Mode: "PassThrough"
      Timeout: 120
      RuntimeManagementConfig:
        UpdateRuntimeOn: "Auto"
YAML
      FileSystemConfigs: []
      FunctionName: "data_wordpressapi_raw"
      Runtime: "python3.12"
      PackageType: "Zip"
      LoggingConfig:
        LogFormat: "Text"
        LogGroup: "/aws/lambda/data_wordpressapi_raw"
      EphemeralStorage:
        Size: 512
      Architectures:
      - "x86_64"

Now let’s focus on the parts causing the warnings – the function’s Code block:

YAML
      RuntimeManagementConfig:
        UpdateRuntimeOn: "Auto"
      Handler: "lambda_function.lambda_handler"
      Code:
        S3Bucket:
          Ref: "LambdaFunction00datawordpressapiraw006YbduCodeS3BucketOneOfmPyvF"
        S3Key:
          Ref: "LambdaFunction00datawordpressapiraw006YbduCodeS3KeyOneOfLqj3f"
      Role:
        Fn::GetAtt:
        - "IAMRole00datawordpressapirawroleygxg4wz400zzf3d"
        - "Arn"

Those don’t look like any S3 resources in my account, so what are they? To begin to answer that question, let’s take a look at one of Bytescale’s example CloudFormation Lambda templates:

YAML
Code: 
  S3Bucket: my-lambda-function-code
  S3Key: MyBundledLambdaFunctionCode.zip

This is simple to break down, with a clear bucket name and object key. The big difference is the lack of Ref in the sample code block. So what’s it doing in my template?

CloudFormation’s Ref is an intrinsic function that returns the value of a specified parameter or resource. Here, both Ref functions point to CloudFormation Parameters defined earlier in the template:

YAML
Parameters:
  LambdaFunction00datawordpressapiraw006YbduCodeS3KeyOneOfLqj3f:
    NoEcho: "true"
    Type: "String"
    Description: "The Amazon S3 key of the deployment package.\nThis property can\
      \ be replaced with other exclusive properties"
  LambdaFunction00datawordpressapiraw006YbduCodeS3BucketOneOfmPyvF:
    NoEcho: "true"
    Type: "String"
    Description: "An Amazon S3 bucket in the same AWS-Region as your function. The\
      \ bucket can be in a different AWS-account.\nThis property can be replaced with\
      \ other exclusive properties"

Now we’re getting somewhere! These parameters clearly reference an S3 key and an S3 bucket.

When this template is imported into a stack, CloudFormation will use these parameters to request the function code’s S3 location. The parameter values will then be passed to the Lambda code block when creating the function. The template can be manually updated with these values, but I’ll use the template as-is to see how the standard process unfolds.

This is a good time to talk about IaC Generator’s other limitations.

Limitations

CloudFormation has several operations, not all of which support every AWS resource. These operations are:

  • IaC Generator.
  • Resource Import: bringing resources into existing stacks.
  • Drift detection: finding differences between a stack’s expected and actual configurations.

The AWS CloudFormation User Guide has a table of operations supported by resource types. This is a substantial table, so let’s focus on my template’s resources.

Firstly, there’s currently no IaC Generator support for the AWS::Lambda::EventInvokeConfig resource that handles Lambda Destinations. My template will deploy the Destination SNS topic, but will not configure the destination’s parameters.

Additionally, IaC Generator doesn’t currently support the AWS::SSM::Parameter resource. This is great from a security angle (no plain text credentials!) but adds an important manual step when using the stack.

Everything else is fully supported, so let’s try using the template.

Testing

In this section, I import the template from my IaC Generator Hands-On into a CloudFormation stack and see what happens.

CloudFormation Designer

A good way to check the template’s contents is to import it to CloudFormation Designer – an AWS tool for visually creating, viewing, and modifying CloudFormation templates.

This is Designer’s first diagram using IaC Generator’s template:

new designer cloudwatchincluded

Eek!

Here, Designer is visualising all the CloudWatch resources. It looks fine on a big screen but isn’t great here, so I made a second template with no CloudWatch resources and imported that:

new designer

Much clearer! My EventBridge schedule and Lambda function are mapped to their respective IAM roles, and the SNS and S3 resources are present. As discussed, no Parameter Store resources or SNS destinations appear because these are not currently supported.

For comparison, here’s the AWS section of my pipeline’s architectural diagram:

2024 02 24 ArchitectureAWSSection

Now, I can add the missing resources to the generated template myself. However, because I want to see how the standard template behaves in testing I’ll be using it as-is.

Stack Import

A stack import is the process of transforming a template into a CloudFormation Stack. The stack is then used to provision AWS resources.

Stack imports are triggered in the IaC Generator’s Template Definition tab. After choosing the stack’s name, the Lambda parameters are requested:

2024 02 22 CloudFormationCreateStackParameters

These values are then assigned to the Lambda code block’s S3Bucket and S3Key parameters.

Next, some basic admin. The stack needs an IAM role for all operations performed. IaC Generator then checks for any resource changes resulting from the stack’s creation. This is a completely new stack so this doesn’t apply.

With these details, CloudFormation starts creating the stack, which can then be deployed as usual.

2024 02 25 CloudFormationStack

Thoughts

So what do I think after my IaC Generator Hands-On? Well, there are some quality of life improvements that I think would make IaC Generator even more useful:

When browsing the scanned resources while making a template, I’d love to have direct links to the resources in the same way that I can see the EBS volumes attached to an EC2 instance and a Lambda function’s CloudWatch Log Stream.

While some resources have fairly friendly Resource Identifier names, others appear with names like aclassoc-0295235ac2e7c7e98 and rtb-0f28bd9a05e4ea696. Having a direct link to these resources from the selection screen would be massively helpful.

Simpler Errors

If IaC Generator is intended for people with little or no CloudFormation experience, then the Lambda and API Gateway warnings could be simpler.

I had some theoretical CloudFormation knowledge and have no qualms about reading documentation, so I got by. But users completely new to CloudFormation could easily see those warnings, assume they made a mistake and forget the whole thing.

Improved Reproducibility

When I deployed the template to my preferred region everything went fine. But when I tried to deploy it in a different AWS region it failed and threw CREATE_FAILED errors:

Resource handler returned message: "Resource of type 'AWS::SNS::Topic' with identifier 'data-lakehouse-raw' already exists." (RequestToken: e71fe163-af5e-bc59-4fd3-95dce19c9494, HandlerErrorCode: AlreadyExists)
Resource handler returned message: "Resource of type 'AWS::SNS::Topic' with identifier 'lambda-general-failure' already exists." (RequestToken: f6e9722f-0f01-8c54-353a-cf62f4f266f4, HandlerErrorCode: AlreadyExists)

For now, this can be resolved by using the generator’s template as a starting point for further templates. However, generating templates that can be immediately deployed in other regions would be very helpful.

Fewer Missing Resources

Finally, it would be good to see fewer missing resource types over time. Some of them will present AWS with some challenges, but hopefully others are just a matter of dev time.

For example, Lambda Destinations are already on the automated AWS SAM templates that can be generated from the Lambda console:

YAML
#***# indicates that sections have been removed.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
  datawordpressapiraw:
    Type: AWS::Serverless::Function
    Properties:
      #***#
      EventInvokeConfig:
        DestinationConfig:
          OnFailure:
            Destination: arn:aws:sns:eu-west-1:REDACTED:lambda-general-failure
            Type: SNS
        MaximumEventAgeInSeconds: 21600
        MaximumRetryAttempts: 0
        #***#

Summary

In this post, I had a hands-on with AWS CloudFormation’s new IaC Generator feature. I think it’s really impressive! It makes CloudFormation more accessible than it was before, and I’m excited to see how it develops over time.

If this post has been useful, the button below has links for contact, socials, projects and sessions:

SharkLinkButton 1

Thanks for reading ~~^~~

Categories
Developing & Application Integration

WordPress Data Extraction Automation With AWS

In this post, I set up the automation of my WordPress API data extraction Python script with AWS managed serverless services.

Table of Contents

Introduction

In my previous post, I wrote a Python script for extracting WordPress API data. While it works fine, it relies on me logging in and pressing buttons. This isn’t convenient, and would be completely out of the question in a commercial use case. Wouldn’t it be great if something could run the script for me?

Enter some AWS managed serverless services that are very adept at automation! In this post, I’ll integrate these services into my existing architecture, test that everything works and see what my AWS costs are to date.

A gentle reminder: this is my first time setting up some of these services from scratch. This post doesn’t represent best practices, may be poorly optimised or include unexpected bugs, and may become obsolete. I expect to find better ways of doing these processes in the coming months and will link updates where appropriate.

Architectural Decisions

In this section, I examine my architectural decisions before starting work. Which AWS services will perform my WordPress data extraction automation? Note that these decisions are in addition to my previous ones.

AWS Lambda

Probably no surprises here. Whenever AWS and serverless come up, Lambda is usually the first service that comes to mind.

And with good reason! AWS Lambda deploys quickly and scales on demand. It supports several programming languages and practically every AWS service. It also has a generous free tier and requires no infrastructure management.

Lambda will provide my compute resources. This includes the runtime, execution environment and network connectivity for my Python script.

Amazon Cloudwatch

Amazon CloudWatch is a monitoring service that can collect and track performance data, generate insights and respond to resource state changes. It provides features such as metrics, alarms, and logs, letting users monitor and troubleshoot their applications and infrastructure in real time.

CloudWatch will record and store my Lambda function’s logs. I can see when my function is invoked, how long it takes to run and any errors that may occur.

So if something does go wrong, how will I know?

Amazon SNS

Amazon Simple Notification Service (SNS) is a messaging service that delivers notifications to a set of recipients or endpoints. It supports various messaging protocols like SMS, email and HTTP, making it helpful for building scalable and decoupled applications.

SNS will be the link between AWS and my email inbox. It will deliver messages from AWS about my Lambda function.

So that’s my alerting sorted. How does the function get invoked?

Amazon EventBridge

Amazon EventBridge is an event bus service that enables communication between different services using events. It offers a serverless and scalable platform with advanced event routing, integration capabilities and, crucially, scheduling and time expression functionality.

EventBridge is here to handle my automation requirements. Using a CRON expression, it’ll invoke my Lambda function regularly with no user input required.

Architectural Diagram

This is an architectural diagram of the AWS automation of my WordPress data extraction process:

  1. EventBridge invokes AWS Lambda function.
  2. AWS Lambda calls Parameter Store for WordPress, S3 and SNS parameters. Parameter Store returns these to AWS Lambda.
  3. Lambda Function calls WordPress API. WordPress API returns data.
  4. API data is written to S3 bucket.

If there’s a failure, the Lambda function publishes a message to an SNS topic. SNS then delivers this message to the user’s subscribed email address.

Meanwhile, Lambda is writing to a CloudWatch Log Group throughout its invocation.

SNS & Parameter Store

In this section, I configure Amazon SNS and update AWS Parameter Store to enable my WordPress data extraction automation alerting. This won’t take long!

SNS Configuration

SNS has two fundamental concepts:

  • Topics: communication channels for publishing messages.
  • Subscriptions: endpoints to send messages to.

Firstly, I create a new wordpress-api-raw standard SNS Topic. This topic doesn’t need encryption or delivery policies, so all the defaults are fine. An Amazon Resource Name (ARN) is assigned to the new SNS Topic, which I’ll put into Parameter Store.

Next, I create a new SNS Subscription for my SNS Topic that emails me when invoked.

There’s not much else to add here! That said, SNS can do far more than this. Check out SNS’s features and capabilities in the Developer Guide.

Parameter Store Configuration

Next, I need to add the new SNS Topic ARN to AWS Parameter Store.

I create a new string parameter, and assign the SNS Topic’s ARN as the value. That’s….it! With some changes, my Python script can now get the SNS parameter in the same way as the S3 and WordPress parameters.

Speaking of changing the Python script…

Python

In this section, I integrate SNS into my existing Python script and test the new outputs.

Function Updates

My script now has a new send_sns_message function:

It expects four arguments:

  • sns_client: the boto3 client used to contact AWS.
  • topic_arn: the SNS topic to use for the message.
  • subject: the message’s subject.
  • message: the message to send.

Everything bar sns_client has string type hints. No return value is needed.

I create a try except block that attempts to send a message using the sns_client’s publish method and the supplied values. The log is updated with publish‘s success or failure.

Separately, I’ve also added a ParamValidationError exception to my get_parameter_from_ssm function. Previously the exceptions were:

Python
    except ssm_client.exceptions.ParameterNotFound:
        logging.warning(f"Parameter {parameter_name} not found.")
        return ""

    except botocore.exceptions.ClientError as e:
        logging.error(f"Error getting parameter {parameter_name}: {e}")
        return ""

They are now:

Python
    except ssm_client.exceptions.ParameterNotFound as pnf:
        logging.warning(f"Parameter {parameter_name} not found: {pnf}")
        return ""

    except botocore.exceptions.ParamValidationError as epv:
        logging.error(f"Error getting parameter {parameter_name}: {epv}")
        return ""

    except botocore.exceptions.ClientError as ec:
        logging.error(f"Error getting parameter {parameter_name}: {ec}")
        return ""

Variable Updates

My send_sns_message function needs some new variables. Firstly, I create an SNS Client using my existing boto3 session and assign it to client_sns:

Python
    # AWS sessions and clients
    session = boto3.Session()
    client_ssm = session.client('ssm')
    client_s3 = session.client('s3')
    client_sns = session.client('sns')
    requests_session = requests.Session()

Next, I assign the new SNS parameter name to a parametername_snstopic object:

Python
    # AWS Parameter Store Names
    parametername_s3bucket = '/s3/lakehouse/name/raw'
    parametername_snstopic = '/sns/pipeline/wordpressapi/raw'
    parametername_wordpressapi = '/wordpress/amazonwebshark/api/mysqlendpoints'

Finally, I create a new lambdaname object which I’ll use for SNS notifications in my Python script’s body.

Python
    # Lambda name for messages
    lambdaname = 'wordpress_api_raw.py'

Script Body Updates

These changes integrate SNS failure messages into my script. There are no success messages…because I get enough emails as it is.

SNS Parameter Retrieval & Check

There’s now a third use of get_parameter_from_ssm, using parametername_snstopic to get the SNS topic ARN from AWS Parameter Store:

Python
    # Get SNS topic from Parameter Store
    logging.info("Getting SNS parameter...")
    sns_topic = get_parameter_from_ssm(client_ssm, parametername_snstopic)

I’ve also added an SNS parameter check. It behaves differently to the other checks, as it’ll raise a ValueError if nothing is found:

Python
    # Check an SNS topic has been returned.
    if not sns_topic:
        message = "No SNS topic returned."
        logging.warning(message)
        raise ValueError(message)

I want to cause an invocation failure in this situation, as not having the SNS topic ARN is a critical and unrecoverable problem which the automation process will have no way to alert me about.

However, the AWS Lambda service can warn me about invocation failures. This is something I’ll set up later on.

Failure Getting Other Parameters

The get_parameter_from_ssm response checks have changed. Previously, if a parameter request (the API endpoints in this case) returns a blank string then a warning is logged and the invocation ends:

Python
    # Check the API list isn't empty
    if not any(api_endpoints_list):
        logging.warning("No API endpoints returned.")
        return

Now, new subject and message objects are created with details about the error. The message string is added to the log, and both objects are passed to send_sns_message along with the SNS client and SNS topic ARN:

Python
    # Check the API list isn't empty
    if not any(api_endpoints_list):
        message = "No API endpoints returned."
        subject = f"{lambdaname}: Failed"

        logging.warning(message)
        send_sns_message(client_sns, sns_topic, subject, message)
        return

The S3 check now works similarly:

Python
    # Check an S3 bucket has been returned.
    if not s3_bucket:
        message = "No S3 bucket returned."
        subject = f"{lambdaname}: Failed"

        logging.warning(message)
        send_sns_message(client_sns, sns_topic, subject, message)
        return

If either of these checks fail, no WordPress API calls are made and the invocation stops.

Failure During For Loop

Previously, the script’s final output was a log entry showing the endpoint_count_success and endpoint_count_failure values:

Python
    logging.info("WordPress API Raw process complete: " \
                 f"{endpoint_count_success} Successful | {endpoint_count_failure} Failed.")

This section has now been expanded. If endpoint_count_failure is greater than zero, a message object is created including the number of failures.

message is then written to the log, and is passed to send_sns_message with a subject and the SNS client and SNS topic ARN:

Python
    logging.info("WordPress API Raw process complete: " \
                 f"{endpoint_count_success} Successful | {endpoint_count_failure} Failed.")

    # Send SNS notification if any failures found
    if endpoint_count_failure > 0:
        message = f"{lambdaname} ran with {endpoint_count_failure} errors.  Please check logs."
        subject = f"{lambdaname}: Ran With Failures"

        logging.warning(message)
        send_sns_message(client_sns, sns_topic, subject, message)

If a loop iteration fails, the script ends it and starts the next. One or more loop iterations can fail while the others succeed.

That completes the script changes. Next, I’ll test the failure responses.

SNS Notification Testing

SNS should now send me one of two emails depending on which failure occurs. I can test these locally by inverting the logic of some if conditions.

Firstly, I set the S3 bucket check to fail if a bucket name is returned:

Python
    # Check an S3 bucket has been returned.
    if s3_bucket:
        message = "No S3 bucket returned."
        subject = f"{lambdaname}: Failed"

        logging.warning(message)
        send_sns_message(client_sns, sns_topic, subject, message)
        return

Upon invocation, an email arrives with details of the failure:

2024 02 06 wordpress api raw.py Failed

Secondly, I change the loop’s data check condition to fail if data is returned:

Python
        # If no data returned, record failure & end current iteration
        if api_json:
            logging.warning("Skipping attempt due to API call failure.")
            endpoint_count_failure += 1
            continue

This ends the current loop iteration and increments the endpoint_count_failure value. Then, in a check after the loop, an SNS message is triggered when endpoint_count_failure is greater than 0:

Python
    # Send SNS notification if any failures found
    if endpoint_count_failure > 0:
        message = f"{lambdaname} ran with {endpoint_count_failure} errors.  Please check logs."
        subject = f"{lambdaname}: Ran With Failures"

        logging.warning(message)
        send_sns_message(client_sns, sns_topic, subject, message)

Now, a different email arrives with the number of failures:

2024 02 06 wordpress api raw.py RanWithFailures

Success! Now the Python script is working as intended, it’s time to deploy it to AWS.

Lambda & CloudWatch

In this section, I start creating the automation of my WordPress data extraction process by creating and configuring a new AWS Lambda function. Then I deploy my Python script, set some error handling and test everything works.

I made extensive use of Martyn Kilbryde‘s AWS Lambda Deep Dive A Cloud Guru course while completing this section. It was exactly the kind of course I needed – a bridge between theoretical certification content and hands-on experience in my own account.

This section is the result of my first pass through the course. There are better ways of doing what I’ve done here, but ultimately I have to start somewhere. I have several course sections to revisit, so watch this space!

Let’s begin with creating a new Lambda function.

Function Creation

Lambda function creation steps vary depending on whether the function is being written from scratch, or if it uses a blueprint or container image. I’m writing from scratch, so after choosing a name I must choose the function’s runtime. Runtimes consist of the programming language and the specific version. In my case, this is Python 3.12.

Next are the permissions. By design, AWS services need permissive IAM roles to interact with other services. A Lambda function with no IAM role cannot complete actions like S3 reads or CloudWatch writes.

Thankfully, AWS are one step ahead. By default, Lambda creates a basic execution role for each new function with some essential Amazon CloudWatch actions. With this role, the function can record invocations, resource utilization and billing details in a log stream. Additional IAM actions can be assigned to the role as needed.

Script Deployment

Now I have a function, I need to upload my Python script. There are many ways of doing this! I followed the virtual environment process, as I already had one from developing the script in VSCode. This environment’s contents are in the requirements.txt file listed in the Resources section.

While this was successful, the resulting deployment package is probably far bigger than it needs to be. Additionally, I didn’t make use of any of the toolkits, frameworks or pipelines with Lambda functionality. I expect my future deployments to improve!

Lambda Destination

There’s one more Lambda feature I want to use: a Lambda Destination.

From the AWS Compute blog:

With Destinations, you can route asynchronous function results as an execution record to a destination resource without writing additional code. An execution record contains details about the request and response in JSON format including version, timestamp, request context, request payload, response context, and response payload.

https://aws.amazon.com/blogs/compute/introducing-aws-lambda-destinations/

Here, I want a destination that will email me if my Lambda function fails to run. This helps with visibility, and will be vital if the SNS parameter isn’t returned!

With no Destination, the failure would only appear in the function’s log and I might not know about it for days. With a Destination enabled, I’ll know about the failure as soon as the email comes through.

My destination uses the following config:

  • Invocation Type: Asynchronous
  • Condition: On Failure
  • Destination Type: SNS topic

The SNS topic is a general Failed Lambda one that I already have. The Lambda service can use this SNS topic regardless of any script problems.

Lambda & CloudWatch Testing

With the function created and deployed, it’s testing time! Does my function work and log as intended?

Error: Timeout Exceeded

It doesn’t take long to hit my first problem:

Task timed out after 3.02 seconds

All Lambdas created in the console start with a three-second timeout. This is great at preventing runaway invocations, but I clearly need longer than three seconds.

After some local testing, I increased the timeout to two minutes in the function’s config:

2023 12 19 LambdaTimeout

Error: Access Denied

Next, I start hitting permission errors:

An error occurred (AccessDeniedException) when calling the GetParameter operation: User is not authorized to perform: ssm:GetParameter on resource because no identity-based policy allows the ssm:GetParameter action.

My Lambda’s basic execution role can interact with CloudWatch, but nothing else. This is by design in the interests of security. However, this IAM role is currently too restrictive for my needs.

The role’s policy needs to allow additional actions:

To follow IAM best practise, I should also apply least-privilege permissions. Instead of a wildcard character, I should restrict the policy to the specific ARNs of my AWS resources.

For example, this IAM policy is too permissive as it allows access to all parameters in Parameter Store:

JSON
{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "Statement1",
			"Effect": "Allow",
			"Action": [
				"ssm:GetParameter"
			],
			"Resource": [
				"*"
			]
		}
	]
}

Conversely, this IAM policy allows access to specific parameter ARNs only.

(Well, it did before the ARNs were redacted – Ed.)

JSON
"Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ssm:GetParameter"
            ],
            "Resource": [
                "arn:aws:ssm:REDACTED",
                "arn:aws:ssm:REDACTED",
                "arn:aws:ssm:REDACTED"
            ]
        }

My S3 policy does have a wildcard value, but it’s at the prefix level:

JSON
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::REDACTED/wordpress-api/*"
            ]
        }

My Lambda function can now write to my bucket, but only to the wordpress-api prefix. A good way to understand the distinction is to look at an AWS example:

arn:aws:s3:::my_corporate_bucket/*
arn:aws:s3:::my_corporate_bucket/Development/*

In this example, line 1 covers the entire my_corporate_bucket S3 bucket. Line 2 is more focused, only covering all objects in the Development prefix of the my_corporate_bucket bucket.

Error: Memory Exceeded

With the new policy, my function runs smoothly. Until:

Runtime exited with error: signal: killed Runtime.ExitError

This one was weird because the function kept suddenly stopping at different points! I then checked further down the test summary:

2023 12 19 LambdaMaxMemoryHighlight

It’s running out of memory! Lambda assigns a default 128MB RAM to each function, and here my function was hitting 129MB. RAM can be changed in the function’s general configuration. But changed to what?

When a Lambda function runs successfully, it logs memory metrics:

Memory Size: 500 MB	Max Memory Used: 197 MB

After some trial and error, I set the function’s RAM to 250MB and have had no problems since.

Incomplete CloudWatch Logs

The last issue wasn’t an error so much as a bug. CloudWatch was showing my Lambda invocation start and end, but none of the function’s logs:

2023 12 22 LambdaNoLogs

The solution was found in Python’s basicConfig‘s docstring:

This function does nothing if the root logger already has handlers configured, unless the keyword argument force is set to True.

basicConfig docstring

Well, AWS Lambda does have built-in logging. And my basicConfig isn’t forcing anything! One swift update and redeployment later:

Python
    logging.basicConfig(
        level = logging.INFO,
        format = "%(asctime)s [%(levelname)s]: %(message)s",
        datefmt = "%Y-%m-%d %H:%M:%S",
        force = True
        )

And my CloudWatch Log Stream is now far more descriptive!

2023 12 22 LambdaLogs

In the long run I plan to investigate Lamba’s logging abilities, but for now this does what I need.

SNS Destination Email

Finally, I want to make sure my Lambda Destination is working as expected. My function works now, so I need to force a failure. There are many ways of doing this. In this case, I used three steps:

  • Temporarily alter the function’s timeout to 3 seconds.
  • Reconfigure the function’s Asynchronous Invocation retry attempts to zero.
  • Invoke the function with a one-time EventBridge Schedule.

The low timeout guarantees a function invocation failure. Setting zero retries prevents unnecessary retries (because I want the failure to happen!) Finally, the one-time schedule will asynchronously invoke my function, which is what the Destination is looking for.

And…(redacted) success!

2024 02 09 DestinationEmail

I could clean this email up with an EventBridge Input Path (which I’ve done before), but that’s mostly cosmetic in this case.

EventBridge

In this section I configure EventBridge – the AWS service that schedules the automation of my WordPress data extraction process. While I’ve used EventBridge Rules before, this is my first time using EventBridge Scheduler. So what’s the difference?

EventBridge Scheduler 101

From the AWS EventBridge product page:

Amazon EventBridge Scheduler is a serverless scheduler that enables you to schedule tasks and events at scale. With EventBridge Scheduler you have the flexibility to configure scheduling patterns, set a delivery window, and define retry policies to ensure your critical tasks and events are reliably triggered right when you need them.

https://aws.amazon.com/eventbridge/scheduler/

EventBridge Scheduler is a fully managed service that integrates with over 200 AWS services. It supports one-time schedules and start and end dates, and can account for daylight saving time.

Cost-wise, EventBridge Schedules are changed per invocation. EventBridge’s free tier covers the first 14 million(!) invocations each month, after which each further million currently costs $1.00. These invocations can be staggered using Flexible Time Windows to avoid throttling.

AWS has published a table showing the main differences between EventBridge Scheduler and Eventbridge Rules. Essentially, Eventbridge Rules are best suited for event-based activity, while EventBridge Scheduler is best suited for time-based activity.

Schedule Setup

Let’s create a new EventBridge Schedule. After choosing a name, I need a schedule pattern. Here, I want a recurring CRON-based schedule that runs at a specific time.

EventBridge Cron expressions have six required fields which are separated by white space. My cron expression is 0 7 * * ? * which translates to:

  • The 0th minute
  • Of the seventh hour
  • On every day of the month
  • Every month,
  • Day of the week,
  • And year

In response, EventBridge shows some of the future trigger dates so I can check my expression is correct:

Sat, 02 Feb 2024 07:00:00 (UTC+00:00)
Sun, 03 Feb 2024 07:00:00 (UTC+00:00)
Mon, 04 Feb 2024 07:00:00 (UTC+00:00)
Tue, 05 Feb 2024 07:00:00 (UTC+00:00)
Wed, 06 Feb 2024 07:00:00 (UTC+00:00)

I then need to choose a flexible time window setting. This setting distributes AWS service API calls to help prevent throttling, but that’s not a problem here so I select Off.

Next, I choose the target. I have two choices: templated targets or universal targets. Templated targets are a set of popular AWS service operations, needing only the relevant ARN during setup. Universal targets can target any AWS service but require more configuration details. Lambda’s Invoke operation is a targeted template, so I use that.

Next are some optional encryption, retry and state settings. EventBridge Scheduler IAM roles are handled here too, allowing EventBridge to send events to the targeted AWS services. Finally, a summary screen shows the full schedule before creation.

The schedule then appears on the EventBridge console:

2024 02 09 AmazonEventBridgeScheduler

EventBridge Testing

Testing time! Does CloudWatch show Lambda function invocations at 07:00?

It does!

2024 02 08 CloudWatchLogs

While I’m in CloudWatch, I’ll change the log group’s retention setting. It defaults to Never Expire, but I don’t need an indefinite history for this function! Three months is fine – long enough to troubleshoot any errors, but not so long that I’m storing and potentially paying for logs I’ll never need.

Costs

In this section, I examine the current AWS costs for my WordPress data extraction and automation processes using the Billing & Cost Management console.

I began creating pipeline resources in December 2023 using various workshops and tutorials. This table shows my AWS service costs (excluding tax) accrued over December 2023 and January 2024 (the months I currently have full billing periods for):

2024 02 09 Cost Explorer

I’ll examine these costs in two parts:

  • S3 Costs: my AWS costs are all storage-based. I’ll examine my S3 API calls and how each S3 API contributes to my bill.
  • Free Tier Usage: everything else has zero cost. I’ll examine what I used and how it compares to the free tier allowances.

I’ll also take a quick look at February’s costs to date. I’ve not tagged any of the pipeline resources, so these figures are for all activity in this AWS account.

S3 Costs

S3 is the only AWS service in my WordPress data extraction and automation processes that is generating a cost. This Cost Explorer chart shows my S3 API usage over the last two full months:

2024 02 10 Cost ExplorerS3APICalls

PutObject is clearly the most used S3 API, which isn’t surprising given S3’s storage nature. Cost Explorer can also show API request totals, as shown below:

2024 02 10 Cost ExplorerS3APICallsDec23
2024 02 10 Cost ExplorerS3APICallsJan24

Remember that this includes S3 API calls from other services like S3 Inventory, CloudTrail Log Steams and Athena queries.

AWS bills summarise these figures for easier reading. This is my December 2023 S3 bill, where S3 PUT, COPY, POST and LIST requests are grouped:

2024 02 09 Billing202312

January 2024’s bill:

2024 02 09 Billing202401

Going into this depth for $0.08 might not seem worth it. But if the bill suddenly becomes $8 or $80 then having this knowledge is very useful!

The AWS Storage blog has a great post on analyzing S3 API operations that really helped here.

Free Tier Usage

The following services had no cost because my usage fell within their free tier allowances. For each zero cost on the bill, I’ll show the service and, where appropriate, the respective free tier allowance.

CloudTrail:

  • 2023-12: 7970 Events recorded.
  • 2024-01: 6605 Events recorded.

CloudWatch was the same for both months:

  • Sub 1GB-Mo log storage used of 5GB-mo log storage free tier
  • Sub 1GB log data ingested of 5GB log data ingestion free tier

Lambda 2023-12:

  • 36.976 GB-Seconds used of 400,000 GB-seconds Compute free tier
  • 47 Requests used of 1,000,000 Request free tier

Lambda 2024-01:

  • 9.572 GB-Seconds used of 400,000 GB-seconds Compute free tier
  • 8 Requests used of 1,000,000 Request free tier

Parameter Store (billed as Secrets Manager):

  • 2023-12: 31 API Requests used of 10,000 API Request free tier
  • 2024-01: 41 API Requests used of 10,000 API Request free tier

February 2024 Costs

At this time I don’t have full billing data for February, but I wanted to show the EventBridge and SNS usage to date:

EventBridge (billed as CloudWatch Events):

  • 16 Invocations used of 14 million free tier

SNS:

  • 3 Notifications used of 1,000 Email-JSON Notification free tier
  • 227 API Requests used of 1,000,000 API Request free tier

As of Feb 15, Lambda is on 71.742 GB-Seconds and 34 Requests while S3 is on 8,821 PCPL requests, 3,764 GET+ requests and 0.0052 GB-Mo storage.

Resources

The full Python script has been checked into the amazonwebshark GitHub repo, available via the button below. Included is a requirements.txt file for the Python libraries used to extract the WordPress API data. This file is unchanged from last time but is included for completeness.

GitHub-BannerSmall

Summary

In this post, I set up the automation of my WordPress API data extraction script with AWS managed serverless services.

On the one hand, there’s plenty more to do here. I have lots to learn about Lambda, like deployment improvement and resource optimisation. This will improve with time and experience.

However, my function’s logging and alerting are in place, my IAM policies meet AWS standards and I’m using the optimal services for my compute and scheduling. And, most importantly, my automation pipeline works!

My attention now turns to the data itself. My next WordPress Data Pipeline post will look at transforming and loading the data so I can put it to use! If this post has been useful, the button below has links for contact, socials, projects and sessions:

SharkLinkButton 1

Thanks for reading ~~^~~

Categories
Data & Analytics

Using Python & AWS To Extract WordPress API Data

In this post, I use popular Python modules & AWS managed serverless services to extract WordPress API data.

Table of Contents

Introduction

Last year, I tested my Python skills by analysing amazonwebshark’s MySQL database with Python. I could have done the same in 2024, but I wouldn’t have learned anything new and it felt a bit pointless. One of my YearCompass 2023-2024 goals is to build more, so I instead decided to create a data pipeline using popular Python modules & AWS services to extract my WordPress data using their API.

A data pipeline involves many aspects, which future posts will explore. This post focuses on extracting data from my WordPress database and storing it as flat files in AWS.

Firstly, I’ll discuss my architectural decisions for this part of the pipeline. Then I’ll examine the functions in my Python script that interact with AWS and perform data extraction. Finally, I’ll bring everything together and explain how it all works.

Architectural Decisions

In this section, I examine my architectural decisions and outline the pipeline’s processes.

Programming Language

My first decision concerned which programming language to use. I’m using Python here for several reasons:

  • I use Python at work and am always looking to refine my skills.
  • Several AWS services natively support Python.
  • Python SDKs like Boto3 and awswrangler support my use case.

Data Extraction

Next, I chose what data to extract from my WordPress MySQL database. I’m interested in the following tables, which are explained in greater detail in 2023’s Deep Dive post:

In November I migrated amazonwebshark to Hostinger, whose MySQL remote access policy requires an IP address. While this isn’t a problem locally, AWS is a different story. I’d either need an EC2 instance with a static IP, or a Lambda function with several networking components. These are time and money costs I’d prefer to avoid, so no calling the database.

Fortunately, WordPress has an API!

WordPress API

The WordPress REST API lets applications interact with WordPress sites by sending and receiving data as JSON objects. Public content like posts and comments are publicly accessible via the API, while private and password-protected content requires authentication.

While researching options, I stumbled across MiniOrange‘s Custom API for WordPress plugin. It has a simple interface and a good feature list:

Custom API for WordPress plugin allows you to create WordPress APIs / custom endpoints / REST APIs. You can Fetch / Modify / Create / Delete data with an easy-to-use graphical interface. You can also build custom APIs by writing custom SQL queries for your WP APIs.

https://plugins.miniorange.com/custom-api-for-wordpress

This meant I could start using it straight away!

The free plan lets users create as many endpoints as needed. But it also has a pretty vital limitation – API key authentication is only possible on their Premium plan. In the free plan, all endpoints are public!

Now let me be clear – this isn’t necessarily a problem. After all, the WordPress API is public! And my WordPress data doesn’t contain any PII or sensitive data. No – the risk I’m trying to address here isn’t a security one.

Public endpoints can be called by anyone or anything at any time. With WordPress, they have dedicated, optimised resources that auto-scale on demand. Whereas I have one Hostinger server that is doing every site process. Could it be DDoSed into oblivion by tons of API calls from bad actors? Do I want to find out?

As I’m using the plugin’s free tier here, I’ll mitigate my risks by:

  • Adding random strings to the endpoints to make them less guessable.
  • Not showing the endpoints in my script or this post.

So ok – how will I get the API endpoints then?

Parameters

Next, I need to decide how my script will get the endpoints to query and the S3 bucket name to store the results.

With previous scripts, I’ve used features like gitignore and dot sourcing to hide parameters I don’t want to expose. While this works, it isn’t ideal. Dot sourcing breaks if the file paths change, and even with gitignore any credentials are still hardcoded into the script locally.

A better approach is to use a process similar to a password manager, where an authenticated user or role can request and receive credentials using secure channels. AWS has two services for this requirement: AWS Secrets Manager and AWS Systems Manager Parameter Store.

Secrets Manager Vs Parameter Store

Secrets Manager is designed for managing and rotating sensitive information like database credentials, API keys, and other types of secrets. Conversely, Parameter Store is designed for storing configuration data, including plaintext or sensitive information, in a hierarchical structure.

I’m using Parameter Store here for two reasons:

Storage

Next, I need to decide where to store the API data. And I’m already using AWS for parameters, so I was always going to end up using S3. But what makes S3 an obvious fit here?

  • Integration: S3 is one of the oldest and most mature AWS services. It is well supported by both the Python SDK and other AWS services like EventBridge, Glue and Athena for processing and analysis.
  • Scalability: S3 will accept objects from a couple of bytes to terabytes in size (although if I’m generating terabytes of data here something is very wrong!). I can run my script at any time and as often as I want, and S3 will handle all the data it receives.
  • Cost: S3 won’t be entirely free here because I’ll be creating and accessing lots of data during testing. But even so, I expect it to cost me pence. I’m not keeping versions at this stage either, so my costs will only be for the current objects.

Much has been written about S3 over the years, so I’ll leave it at this.

Use Of Flat Files

Finally, let’s examine the decision to store flat files in the first place. The data is already in a database – why duplicate it?

Decoupling: Putting raw data into S3 at an early stage of the pipeline decouples the database at that point. Databases can become inaccessible, corrupted or restricted. The S3 data would be completely unaffected by these database issues, allowing the pipeline to persist with the available data.

Reduced Server Load: Storing data in S3 means the rest of the pipeline reads the S3 objects instead of the database tables. This reduces the Hostinger server’s load, letting it focus on transactional queries and site processes. S3 is almost serving as a read replica here.

Security: It is simpler for AWS services to access data stored in S3 than the same data stored on Hostinger’s server. AWS services accessing server data require MySQL credentials and a whitelisted IP. In contrast, AWS services accessing S3 data require…an IAM policy.

Architectural Diagram

This is an architectural diagram of the expected process:

  1. User triggers the Python function.
  2. Python interacts with AWS Python SDK.
  3. SDK calls Parameter Store for WordPress & S3 parameters. These are returned to Python via the SDK.
  4. Python calls WordPress API. WordPress API returns data.
  5. Python writes API data to S3 bucket via the SDK.

Setup & Config

I completed some local and cloud configurations before I started writing my Python script to extract WordPress API data. This section explores my laptop setup and AWS infrastructure.

Local Machine

I’m using Windows 10 and WSL version 2 to create a Linux environment with the Ubuntu 22.04.3 LTS distribution. I’m using Python 3.12, with a fresh Python virtual environment for installing my dependencies.

AWS Data Storage

I already have an S3 bucket for ingesting raw data, so that’s sorted. I made a wordpress-api prefix in that bucket to partition the uploaded data.

This bucket doesn’t have versioning enabled because it has a high object turnover. Versioning is unneeded and could get very expensive without a good lifecycle policy! While this would be simple to do, it’s a wasted effort at this point in the pipeline.

Another factor against versioning is that I can recreate S3 objects from the MySQL database. As objects are reproducible, there is no need for the delete protection offered by versioning.

AWS Parameters

I’m using Parameter Store to hold two parameters: my S3 bucket name and my WordPress API endpoints. Each of these uses a different parameter type.

The S3 bucket name is a simple string that uses the String Parameter Type. This is intended for blocks of text up to 4096 characters (4kb). The API endpoints are a collection of strings generated by the WordPress plugin. I use the StringList Parameter Type here, which is intended for comma-separated lists of values. This lets me store all the endpoints in a single parameter, optimising my code and reducing my AWS API calls.

Python Script

In this section, I examine the various parts of my Python script that will extract data from the WordPress API. This includes functions, methods and intended functionality.

Advisory

Before continuing I want to make something clear. This advisory is on my amazonwebshark artefacts GitHub repo, but it bears repeating here too:

Artefacts within this post have been created at a certain point in my learning journey. They do not represent best practices, may be poorly optimised or include unexpected bugs, and may become obsolete.

If I find better ways of doing these processes in future then I’ll link to or update posts where appropriate.

Logging

Firstly, I’ll sort out some logging.

The logging module is a core Python library, so I can import it without a pip install command. I then use logging‘s basicConfig function to set my desired parameters:

Python
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s]: %(message)s",
    datefmt="%Y-%m-%d %H:%M:%S"
    )

level sets the logging level to start at. logging.INFO records information about events like authentications, conversions and confirmations.

format sets how the logs will appear in the console. Sections enclosed by % and ( )s are placeholders that will be formatted as strings. Other characters are printed as-is. Here, my logs will return as Date/Time [Log Level]: Log Message.

datefmt sets the date/time format for format‘s asctime using the same directives as time.strftime().

These settings will give me logs in the style of:

2024-01-11 09:44:39 [INFO]: Parameter found.
2024-01-11 09:44:39 [INFO]: API endpoints returned.
2024-01-11 09:44:39 [INFO]: Getting S3 parameter...
2024-01-11 09:44:39 [WARNING]: S3 parameter not found!

This lets me keep track of what stage Python is at when I extract WordPress API data.

boto3 Session

To call the AWS services I want to use, I need to create a boto3 session. This object represents a single connection to AWS, encapsulating options including the configuration settings and credentials. Without this, Python cannot access AWS Parameter Store, and so cannot extract WordPress API data.

To begin, I run pip install boto3 in the terminal. I then script the following:

Python
import logging
import boto3

session = boto3.Session()

This code snippet performs two new actions:

  • Imports the boto3 module
  • Instantiates an instance of the boto3 module’s Session class.

As Session has no arguments, it will use the first AWS credentials it finds. In AWS, these will be from the Lambda function’s IAM role. No problems there. But I have several AWS profiles on my laptop, and my default profile is for a different AWS account!

In response, I can set an AWS profile using VSCode’s launch.json debugging object. By adding "env": {"AWS_PROFILE": "{my_profile_name}"} to the end of the configurations list, I can specify which local AWS profile to use without altering the Python script itself:

JSON
{
	"version": "0.2.0",
    "configurations": [
        {
            "name": "Python: Current File",
            "type": "python",
            "request": "launch",
            "program": "${file}",
            "console": "integratedTerminal",
            "justMyCode": true,
            "env": {"AWS_PROFILE": "profile"}
        }
    ]
}

Functions

This section examines my Python functions that extract WordPress API data. Each function has an embedded GitHub Gist and an explanation of the arguments and processes.

Get Parameters Function

Firstly, I need to get my parameter values from AWS Parameter Store.

Here, I define a get_parameter_from_ssm function that expects two arguments:

  • ssm_client: the boto3 client used to contact AWS.
  • parameter_name: the name of the required parameter.

I use type hints to annotate parameter_name and the returned object type as strings. For a great introduction to type hints, take a look at this short video from AWS Mad Lad Matheus Guimaraes:

I then create a try except block containing a response object which uses the ssm_client.get_parameter function to try getting the requested parameter. If this fails, the AWS error is logged and a blank string is returned. The parameter value is returned if successful.

I am capturing the AWS exceptions using the botocore module because it provides access to the underlying error information returned by AWS services. When an AWS service operation fails, it usually returns an error response that includes details about what went wrong. botocore can access these responses programmatically and log more exception details than the Python default.

I now have two additional changes to my main script:

Python
import logging
import boto3
import botocore

session = boto3.Session()
client_ssm = session.client('ssm')
  • botocore needs to be imported, so I add import botocore to the script. I don’t need to install botocore because it was installed with boto3.
  • I need a Simple Systems Manager (SSM) client to interact with AWS Systems Manager Parameter Store. I create an instance of the SSM client using my existing session and assign it to client_ssm. I can now use client_ssm throughout my script.

Get Filename Function

Next, I want to get each API endpoint’s filename. The filename has some important uses:

  • Logging processes without using the full endpoint.
  • Creating S3 objects.

A typical endpoint has the schema https://site/endpointname_12345/. There are two challenges here:

  • Extracting the name from the string.
  • Removing the name’s random characters.

I define a get_filename_from_endpoint function, which expects an endpoint argument with a string type hint and returns a new string.

Firstly, my name_full variable uses the rsplit method to capture the substring I need, using forward slashes as separators. This converts https://site/endpointname_12345/ to endpointname_12345.

Next, my name_full_last_underscore_index variable uses the rfind method to find the last occurrence of the underscore character in the name_full string.

Finally, my name_partial variable uses slicing to extract a substring from the beginning of the name_full string up to (but not including) the index specified by name_full_last_underscore_index. This converts endpointname_12345 to endpointname.

If the function is unable to return a string, an exception is logged and a blank string is returned instead.

No new imports are needed here. So let’s move on!

Call WordPress API Function

My next function queries a given API endpoint and handles the response.

Here, I define a get_wordpress_api_json function that expects three arguments:

  • requests_session
  • api_url: the WordPress API URL with a string type hint.
  • api_call_timeout: the number of seconds to wait for a response before timing out.

requests.Session is a part of the Requests library, and creates a session object that persists across multiple requests. I can now use the same session throughout the script instead of constantly creating new ones.

I open a try except block and create a response object. requests.Session attempts to call the API URL. If the response status code is 200 OK then the response is returned as a raw JSON dictionary.

This function can fail in three ways:

  • The status code isn’t 200. While this includes 3xx, 4xx and 5xx codes, it also includes the other 2xx codes. This was deliberate, as any 2xx responses other than 200 are still unusual, and something I want to know about.
  • The API call times out.
  • Requests throws an exception.

In all cases, the function raises an exception and doesn’t proceed. This was a conscious choice, as an API call failure represents a critical and unrecoverable problem with the WordPress API that should ring alarm bells.

As I’m using the Requests module now, I need to run pip install requests in the terminal and add import requests to my script. I then create my requests session in the same way as my boto3 session.

I’m also now using json – another pre-installed core Python module ready for import:

Python
import logging
import json
import requests
import boto3
import botocore

session = boto3.Session()
client_ssm = session.client('ssm')
requests_session = requests.Session()

S3 Upload Function

Finally, I need to put my JSON data into S3

I define a put_s3_object function that expects four arguments:

  • s3_client: the boto3 client used to contact AWS.
  • bucket: the S3 bucket to create the new object in
  • name: the name to use for the new object
  • json_data: the data to upload

I give string type hints to the bucket, name and json_data arguments. This is especially important for json_data because of what I plan to do with it.

I open a try except block and try to use put_s3_object to upload the JSON data to S3. In this context:

  • Body is the JSON data I want to store.
  • Bucket is the S3 bucket name from AWS Parameter Store.
  • Key is the S3 object key, using an f-string that includes the name from my get_filename_from_endpoint function.

The JSON data is created by my get_wordpress_api_json function, which returns that data as a dictionary. Passing a dictionary to put_s3_object‘s Body argument will throw a parameter validation error because its type is invalid for the Body parameter. json_data‘s string type hint will help prevent this scenario.

Moving on, the S3 client’s put_object function attempts to upload the data to the S3 bucket’s wordpress-api prefix as a new JSON object. If this operation succeeds, the function returns True. If it fails, a botocore exception is logged and the function returns False.

While no new imports are needed, I do now need an S3 client alongside the SSM one to allow S3 interactions:

Python
session = boto3.Session()
client_ssm = session.client('ssm')
client_s3 = session.client('s3')
requests_session = requests.Session()

Script Body

This section examines the body of my Python script. I look at the script’s flow, the objects passed to the functions and the responses to successful and failed processes.

Variables

In addition to the imports and sessions already listed, I have some additions:

  • The S3 bucket and WordPress API Parameter Store names.
  • An api_call_timeout value for the WordPress API requests in seconds.
  • Three endpoint counts used for monitoring failures, successes and overall progress.
Python
# Parameter Names
parametername_s3bucket = '/s3/lakehouse/name/raw'
parametername_wordpressapi = '/wordpress/amazonwebshark/api/mysqlendpoints'

# Counters
api_call_timeout = 30
endpoint_count_all = 0
endpoint_count_failure = 0
endpoint_count_success = 0

Getting The Parameters

The first part of the script’s body handles getting the AWS parameters.

Firstly, I pass my SSM client and WordPress API parameter name to my get_parameter_from_ssm function.

If successful, the function returns a comma-separated string of API endpoints. I transform this string into a list using .split(",") and assign the list to api_endpoints_list. Otherwise, an empty string is returned.

This empty string is unchanged by .split(",") and is assigned to api_endpoints_list. This is why get_parameter_from_ssm returns a blank string if it hits an exception. split(",") has no issues with a blank string, but throws attribute errors with returns like False and None.

I then check if api_endpoints_list contains anything using if not any(api_endpoints_list). return ends the script execution if the list contains no values, otherwise the number of endpoints is recorded.

A similar process happens with the S3 bucket parameter. My get_parameter_from_ssm function is called with the same SSM client and the S3 parameter name. This time a simple string is returned, so no splitting is needed. This string is assigned to s3_bucket, and if it’s found to be empty then return ends the current execution.

If both api_endpoints_list and s3_bucket pass their tests, the script moves on to the next section.

Getting The Data

The second part of the script’s body handles getting data from the API endpoints.

Firstly, I open a for loop for each endpoint in api_endpoints_list. I pass each endpoint to my get_filename_from_endpoint function to get the name to use for logging and object creation. This name is assigned to object_name.

object_name is then checked. If found to be empty, the loop skips that endpoint to prevent any useless API calls and to preserve the existing S3 data. The failure counter increments by 1, and continue ends the current iteration of the for loop.

Once the name is parsed, my Requests session, timeout values and current API endpoint are passed to the get_wordpress_api_json function. This function returns a JSON dictionary that I assign to api_json. api_json is then checked and, if empty, skipped from the loop using continue.

Next, I need to transform the api_json dictionary object before an S3 upload attempt. If I pass api_json to S3’s put_object as is, the Body parameter throws a ParamValidationError because it can’t accept dictionaries. I use the json.dumps function to transform api_json to a JSON-formatted string and assign it to api_json_string, which put_object‘s Body parameter can accept.

I can now pass my S3 client, S3 bucket name, object_name and api_json_string to my put_s3_object function. This function’s output is assigned to ok, which is then checked and updates the success or failure counter as appropriate.

Once all APIs are processed, the loop ends and the final success and failure totals are logged.

Adding A Handler

Finally, I encapsulate the script’s body into a lambda_handler function. Handlers let AWS Lambda run invoked functions, so I’ll need one when I deploy my script to the cloud.

Resources

The full Python script has been checked into the amazonwebshark GitHub repo, available via the button below. Included is a requirements.txt file for the Python libraries used to extract the WordPress API data.

GitHub-BannerSmall

Summary

In this post, I used popular Python modules & AWS managed serverless services to extract WordPress API data.

I took a lot away from this! The script was a good opportunity to practise my Python skills and try out unfamiliar features like type hints, continue and requests.Session. Additionally, I made several revisions to control flows, logging and error handling that were triggered by writing this post. The script is clearer and faster as a result.

With the script complete, my next step will be deploying it to AWS Lambda and automating its execution. So keep an eye out for that! If this post has been useful, the button below has links for contact, socials, projects and sessions:

SharkLinkButton 1

Thanks for reading ~~^~~