Categories
Developing & Application Integration

Low-Code S3 Key Validation With AWS Step Functions & JSONata

In this post, I use JSONata to add low-code S3 object key validation to an AWS Step Functions state machine.

Table of Contents

Introduction

In 2024, I worked a lot with AWS Step Functions. I built several for different tasks, wrote multiple blog posts about them and talked about them a fair bit. So when AWS introduced JSONata support for Step Functions last year, I was very interested. Although I had no prior JSONata experience, I heard positive feedback and made a mental note to explore its use cases.

Well, there’s no time like the present! And as I was starting to create the first Project Wolfie resources I realised some of my requirements were a perfect fit.

Firstly, I will examine what JSONata is, how it works and why it’s useful. Next, I will outline my architecture and create some low-code S3 key validation JSONata expressions. Finally, I’ll test these expressions and review their outputs.

JSONata & AWS

This section introduces JSONata and examines its syntax and benefits.

Introducing JSONata

JSONata is a lightweight query and transformation language for JSON, developed by Andrew Coleman in 2016. Specifically inspired by XPath and SQL, it enables sophisticated queries using a compact and intuitive notation.

JSONata provides built-in operators and functions for efficiently extracting and transforming data into any JSON structure. It also supports user-defined functions, allowing for advanced expressions that enhance the querying of dynamic JSON data.

For a visual introduction, check out this JSONata overview:

JSONata Syntax Essentials

JSONata has a simple and expressive syntax. Its path-based approach lets developers easily navigate nested structures. It combines functional programming with dot notation for navigation, brackets for filtering and pipeline operators for chaining.

JSONata operations include transformations like:

  • Arithmetic ($price * 1.2)
  • Conditional Logic ($price > 100 ? 'expensive' : 'affordable').
  • Filtering ($orders[status = 'shipped'])
  • String Operations ($firstName & ' ' & $lastName)

The JSONata site includes full documentation and a JSONata Exerciser for experimenting.

JSONata In AWS Step Functions

JSONata was introduced to AWS Step Functions in November 2024. Using JSONata in Step Functions requires setting the QueryLanguage field to JSONata in the state machine definition. This action replaces the traditional JSONPath fields with two JSONata fields:

  • Arguments: Used to customise data sent to state actions.
  • Output: Used to transform results into custom state output.

Additionally, the Assign field sets variables that can be stored and reused across the workflow.

In AWS Step Functions, JSONata expressions are enclosed in {% %} delimiters but otherwise follow standard JSONata syntax. They access data using the $states reserved variable with the following structures:

  • State input is accessed using $states.input
  • Context information is accessed using $states.context
  • Task results (if successful) are accessed using $states.result
  • Error outputs (if existing) are accessed using $states.errorOutput

Step Functions includes standard JSONata functions as well as AWS-specific additions like $partition, $range, $hash, $random, and $uuid. Some functions, such as $eval, are not supported.

Here are some JSONata examples from the AWS Step Functions Developer Guide:

Plaintext
{% $states.input.title %}

{% $current_price <= $states.input.desired_priced %}

{% $parse($states.input.json_string) %}

Talking more about this subject is AWS Principle Developer Advocate Eric Johnson:

JSONata Benefits

So why is JSONata in AWS a big deal?

Low Maintenance: JSONata use removes the need for Lambda runtime updates, dependency management and security patching. JSONata expressions are self-contained and version-free, reducing debugging and testing effort.

Simpler Development Workflow: JSONata’s standardised syntax removes decisions about languages, runtimes and tooling. This improves consistency, simplifies collaboration and speeds up development.

Releases Capacity: JSONata use reduces reliance on AWS Lambda, freeing up Lambda concurrency slots for more complex tasks. This minimises throttling risks and can lower Lambda costs.

Faster Execution: JSONata runs inside AWS services, avoiding cold starts, IAM role checks and network latency. Most JSONata transformations are complete in milliseconds, making it ideal for high-throughput APIs and real-time systems.

Architecture

This section explains the key features and events used in my low-code S3 validation architecture with JSONata.

Object Created Event

My process starts when an S3 object is created. For this post, I’m using Amazon EventBridge‘s sample S3 Object Created event:

JSON
{
  "version": "0",
  "id": "17793124-05d4-b198-2fde-7ededc63b103",
  "detail-type": "Object Created",
  "source": "aws.s3",
  "account": "123456789012",
  "time": "2021-11-12T00:00:00Z",
  "region": "ca-central-1",
  "resources": ["arn:aws:s3:::example-bucket"],
  "detail": {
    "version": "0",
    "bucket": {
      "name": "example-bucket"
    },
    "object": {
      "key": "example-key",
      "size": 5,
      "etag": "b1946ac92492d2347c6235b4d2611184",
      "version-id": "IYV3p45BT0ac8hjHg1houSdS1a.Mro8e",
      "sequencer": "00617F08299329D189"
    },
    "request-id": "N4N7GDK58NMKJ12R",
    "requester": "123456789012",
    "source-ip-address": "1.2.3.4",
    "reason": "PutObject"
  }
}

Here, the highlighted key field is vital as it identifies the uploaded object. This field will be used in the validation processes.

Choice State

In AWS Step Functions, Choice states introduce conditional logic to a state machine. They assess conditions and guide execution accordingly, allowing workflows to branch dynamically based on input data. When used with JSONata, a Choice state must contain the following fields:

  • Condition field – a JSONata expression that evaluates to true/false.
  • Next field – a value that must match a state name in the state machine.

For example, this Choice state checks if a variable foo equals 1:

Plaintext
{"Condition": "{% $foo = 1 %}",  "Next": "NumericMatchState"}

If $foo = 1, the condition is true and the workflow transitions to a NumericMatchState state.

Architecture Diagram

Now let’s put this all together into an architecture diagram:

Here,

  1. A file is uploaded to an Amazon S3 Bucket.
  2. S3 creates an Object Created event.
  3. Amazon EventBridge matches the event record to an event rule.
  4. Eventbridge executes the AWS Step Functions state machine and passes the event to it as JSON input.
  5. The state machine transitions through the various choice states.
  6. The state machine transitions to the fail state if any choice state criteria are not met.
  7. The state machine transitions to the success state if all choice state criteria are met.

Expression Creation

In this section, I create JSONata expressions to perform low-code S3 validation. For clarity, I’ll use this sample S3 event including an object key which closely resembles my actual S3 path:

JSON
{
  "version": "0",
  ...
  "detail": {
    "version": "0",
    "bucket": {
      "name": "data-lakehouse-raw"
    },
    "object": {
      "key": "iTunes/iTunes-AllTunes-2025-02-01.txt",
      "size": 5,
      ...
    },
    "request-id": "N4N7GDK58NMKJ12R",
    "requester": "123456789012",
    "source-ip-address": "1.2.3.4",
    "reason": "PutObject"
  }
}

S3 Key TXT Suffix Check

This JSONata expression checks if the S3 object key ends with txt:

Plaintext
{% $lowercase($split($split($states.input.detail.object.key, '/')[-1], '.')[-1]) = 'txt' %}

For better readability:

Plaintext
{% 
  $lowercase(
    $split(
      $split($states.input.detail.object.key, '/')[-1], 
    '.')[-1]
  ) = 'txt' 
%}

Let’s walk through this step by step:

1. Accessing The S3 Object Key

Extract the key from the event using $states.input:

Plaintext
$states.input.detail.object.key

Output: "iTunes/iTunes-AllTunes-2025-02-01.txt"

2. Splitting By / To Extract The Filename

Break the key into an array with %split using / as the delimiter:

Plaintext
$split($states.input.detail.object.key, '/')

Output: ["iTunes", "iTunes-AllTunes-2025-02-01.txt"]

Now, retrieve the array’s last element (the object name) using [-1]:

Plaintext
$split(...)[-1]

Output: "iTunes-AllTunes-2025-02-01.txt"

3. Splitting By . To Extract The File Suffix

Break the filename with $split again, using . as the delimiter:

Plaintext
$split($split(...)[-1], '.')

Output: ["iTunes-AllTunes-2025-02-01", "txt"]

Now, retrieve the last element (the suffix) using [-1]:

Plaintext
$split($split(...)[-1], '.')[-1]

Output: "txt"

4. Converting To Lowercase For Case-Insensitive Matching

Use $lowercase to convert the suffix to lowercase:

Plaintext
$lowercase($split(...)[-1], '.')[-1])

Output: "txt"

The $lowercase function ensures consistency, as files with TXT, Txt, or tXt extensions will still match correctly. Here, there is no change as txt is already lowercase.

5. Comparing Against txt

Finally, compare the result to 'txt':

Plaintext
$lowercase($split(...)[-1], '.')[-1]) = 'txt'

Output: true

This means that files ending in .txt pass validation, while others fail.

S3 Key iTunes String Check

This JSONata expression checks if the S3 object key contains iTunes.

Plaintext
{% $contains($split($states.input.detail.object.key, '/')[-1], 'iTunes') %}

For better readability:

Plaintext
{% 
  $contains(
    $split(
      $states.input.detail.object.key, '/')[-1],
    'iTunes'
  ) 
%}

I’m not using $lowercase this time, as iTunes is the correct spelling.

1. Extract The Filename

This is unchanged from the last expression:

Plaintext
$split($states.input.detail.object.key, '/')[-1]

Output: "iTunes-AllTunes-2025-02-01.txt"

2. Check If The String Contains 'iTunes

The $contains function checks if the string contains the specified substring. It returns true if the substring exists; otherwise, it returns false.

Plaintext
$contains($split(...)[-1], 'iTunes')

Output: true ✅ if 'iTunes‘ appears anywhere in the filename.

So:

  • "iTunes-AllTunes-2025-02-01.txt"true
  • "itunes-AllTunes-2025-02-01.txt"false (case-sensitive)

S3 Key Date Check

This JSONata expression checks if the S3 object key contains a date with format YYYY-MM-DD.

Plaintext
{% $exists($match($split($states.input.detail.object.key, '/')[-1], /\d{4}-\d{2}-\d{2}/)) %}

For better readability:

Plaintext
$exists(
  $match(
    $split($states.input.detail.object.key, '/')[-1], 
    /\d{4}-\d{2}-\d{2}/
  )
)

1. Extract The Filename

This is unchanged from the first expression:

Plaintext
$split($states.input.detail.object.key, '/')[-1]

Output: "iTunes-AllTunes-2025-02-01.txt"

2. Apply The Regex Match

The $match function applies the substring to the provided regular expression (regex). If found, an array of objects is returned containing the following fields:

  • match – the substring that was matched by the regex.
  • index – the offset (starting at zero) within the substring.
  • groups – if the regex contains capturing groups (parentheses), this contains an array of strings representing each captured group.

In this JSONata expression:

Plaintext
$match(..., /\d{4}-\d{2}-\d{2}/)

The regex looks for:

  • \d{4} → Four digits (year)
  • - → Hyphen separator
  • \d{2} → Two digits (month)
  • - → Another hyphen
  • \d{2} → Two digits (day)

Output:

JSON
{
  "match": "2025-02-01",
  "index": 16,
  "groups": []
}

3. Convert To Boolean With $exists

I can’t use the $match output yet as the Choice state needs a boolean output. Enter $exists. This function returns true for a successful match; otherwise, it returns false.

Plaintext
$exists($match(..., /\d{4}-\d{2}-\d{2}/))

Output: true ✅ if a date is found.

Here, $exists returns true as a date is present. However, ote that JSONata lacks built-in functions to validate dates. For example:

  • "2025-02-01"true (valid date)
  • "2025-02-31"true (invalid date but still matches format)

An AWS Lambda function would be needed for strict date validation.

Combining JSONata Expressions

Although I’ve created separate Choice states for each JSONata expression in this section, I will add that all the expressions can be combined into a single Choice state using and:

Plaintext
{% $lowercase($split($split($states.input.detail.object.key, '/')[-1], '.')[-1]) = 'txt' and $contains($split($states.input.detail.object.key, '/')[-1], 'iTunes') and $exists($match($split($states.input.detail.object.key, '/')[-1], /\\d{4}-\\d{2}-\\d{2}/)) %}

For better readability:

Plaintext
{% 
  $lowercase(
    $split(
      $split(
        $states.input.detail.object.key, '/')[-1], '.')[-1]) = 'txt' 
and 
  $contains(
    $split(
      $states.input.detail.object.key, '/')[-1], 'iTunes') 
and 
  $exists(
    $match(
      $split(
        $states.input.detail.object.key, '/')[-1], /\\d{4}-\\d{2}-\\d{2}/)) 
%}

When deciding whether to do this, consider these benefits:

  • Simplified Structure: Reducing the number of states can make the state machine easier to understand and maintain visually. Instead of multiple branching paths, all logic is in one centralised Choice state.
  • Cost Optimisation: AWS Step Functions Standard Workflows pricing is based on the number of state transitions. Combining multiple Choice states into one reduces transitions, potentially lowering costs for high-volume workflows.
  • Minimises Transition Latency: Each state transition adds a slight delay. By managing all logic within a single Choice state, the workflow runs more efficiently due to the reduced transitions.

Against these tradeoffs:

  • Added Complexity: A complex Choice state with many conditions can be difficult to read, debug, and modify. It may require deeply nested logic, which makes future updates challenging.
  • Limited Observability: If multiple conditions are combined into one state, debugging failures becomes more difficult as it is unclear which condition caused an unexpected transition.
  • Potential Scaling Difficulty: As the workflow evolves, adding more conditions to a single Choice state can become unmanageable. Ultimately, this situation may require breaking it up.

Final Workflows

Finally, let’s see what the workflows look like. Firstly, this workflow has separate Choice states for each JSONata expression:

stepfunctions graph Data Ingestion iTunes

Data-Ingestion-iTunes ASL on GitHub.

Next, this workflow has one Choice state for all JSONata expressions:

stepfunctions graph Data Ingestion iTunes all

Data-Ingestion-iTunes-All ASL on GitHub.

Testing

To ensure my low-code JSONata expressions work as expected, I ran several tests against different S3 object keys. These tests validate:

  • File Suffix (.txt)
  • Key Content (iTunes)
  • Date Format (YYYY-MM-DD)

Suffix Validation Tests

Test CaseS3 KeyExpectedActual
Valid Suffix (.txt)"iTunes/iTunes-2025-02-01.txt"Proceed to iTunes Check✅ Success → Next: iTunes String Check
Invalid Suffix (.csv)"iTunes/iTunes-2025-02-01.csv"Fail (No further checks)❌ Failure → No further checks
Missing Suffix"iTunes/iTunes-2025-02-01"Fail (No further checks)❌ Failure → No further checks

Key Content Validation Tests

Test CaseS3 KeyExpectedActual
Valid “iTunes” Key"iTunes/iTunes-2025-02-01.txt"Proceed to Date Check✅ Success → Next: Date Check
Incorrect Case (itunes instead of iTunes)"iTunes/itunes-2025-02-01.txt"Fail (No further checks)❌ Failure → No further checks
Missing Key String""Fail (No further checks)❌ Failure → No further checks

Date Format Validation Tests

Test CaseS3 KeyExpectedActual
Correct Date Format (YYYY-MM-DD)"iTunes/iTunes-2025-02-01.txt"Success (Validation complete)✅ Success → Validation complete!
Incorrect Date Format (Missing Day)"iTunes/iTunes-2025-02.txt"Fail (No further checks)❌ Failure → No further checks
Missing Date"iTunes/iTunes.txt"Fail (No further checks)❌ Failure → No further checks

Edge Case: Impossible Date

Test CaseS3 KeyExpectedActual
⚠️ Impossible Date (2025-02-31)"iTunes/iTunes-2025-02-31.txt"Fail (Ideally)Unexpected Success (JSONata does not validate real-world dates)

These tests confirm that JSONata expressions can effectively validate S3 object keys based on file suffixes, key contents and date formats. However, while JSONata can check formatting (YYYY-MM-DD) it does not validate real-world dates. If strict date validation is needed then an AWS Lambda function would be required.

Summary

In this post, I used JSONata to add low-code S3 object key validation to an AWS Step Functions state machine. This approach simplifies the validation process and reduces the reliance on more complex Lambda functions.

My first impressions of JSONata are very good! It’s already reduced both the number and size of Project Wolfie’s Lambda functions, and there’s still lots of JSONata to explore. In the meantime, these further videos by Eric Johnson explore more advanced JSONata Step Function applications:

If this post has been useful then the button below has links for contact, socials, projects and sessions:

SharkLinkButton 1

Thanks for reading ~~^~~

Categories
Me

Project Wolfie: Honouring An Absent Friend With Music

In this post, I discuss our late German Shepherd Wolfie and outline a project that utilises music data in his memory.

Table of Contents

Introduction

On 24 January 2025, our German Shepherd Wolfie sadly passed away.

PXL 20230501 121501481 min

Wolfie was a long, fluffy boi with a big presence and a big mane. He enjoyed sniffing things, tilting his head and barking at foxes. He was loud, proud and bushy-browed.

Wolfie was also psychic. No one could leave the house without his knowledge, and no one could enter it without having a snout-first search.

IMG 20200923 121128 min

Wolfie struggled with various health issues throughout his life, including eating difficulties, muscle problems and genetic defects. Despite this, Wolfie was a cherished family member for four years before he sadly lost his battle with kidney failure and suspected cancer.

Our walks often involved music, and I regularly exposed Wolfie to my music library. He grew so accustomed to iPods during walks that he would bark whenever I picked one up!

PXL 20220904 125315538.MP min

After he was gone, I found myself revisiting the songs we shared. Music became a way to cherish those memories, so I wanted to create something meaningful in his memory…

Project Wolfie

This section explores what Project Wolfie is, the music data it utilises and its goals.

Definition

This project has been on my mind for some time now, and I suppose this was the push it needed to take shape. Project Wolfie is a data-driven initiative that explores the patterns hidden in my music collection. It analyses track metadata, listening habits and technical attributes to find insights, trends and recommendations.

Here, Wolfie is short for:

Waveform Observations Library For Intelligence Engineering

Let’s break this down:

Waveform: A visual illustrating a track’s traits like timbre, pitch and dynamics. Time is represented on the horizontal axis, while the vertical axis reflects amplitude.

Here is a sample waveform:

Observations Library: A consolidated data repository containing information about my music’s properties and my listening habits. The data consists of various types, structures and formats, and will be stored, cleaned and enriched for further use.

For Intelligence Engineering: The AI and BI use cases for the observations library. Here, interactive data visualisation and machine learning services will use the data to uncover patterns, predict trends and generate personalised recommendations.

Data

Music files contain more than just sound – they hold layers of metadata that are crucial to Project Wolfie.

This section explores the different types of metadata related to my music collection, highlighting their functions and purposes. I have assigned these categories using my understanding and intended use of the data.

Technical Metadata

Technical Metadata refers to the measurable and technical attributes of a music file. It tends to include numerical values and audio properties, and is commonly found by analysing the track using applications like Audacity, foobar2000 and MixedInKey, as well as Python libraries like Librosa.

Examples include:

  • What is the track’s initial tempo and key?
  • What is the track’s duration, and how loud is it?
  • What are the track’s spectrographic and harmonic properties?

Descriptive Metadata

Descriptive Metadata refers to the contextual and identifying information about a music track. It tends to include text-based details and is commonly found both within the track’s properties and on websites like Beatport and Discogs.

Examples include:

  • Who produced the track, and what is it called?
  • What is the track’s genre?
  • Which label published the track, and when?

Interaction MetaData

Interaction Metadata refers to engagement and listening behaviours. It typically includes dates, integers and timestamps, and is commonly generated by digital music services like iTunes and Spotify.

Examples include:

  • When was the last time a track was played or skipped?
  • How many times has a track been played?
  • What rating has a track been assigned?

Deliverables

Here are the objectives I’m pursuing in Project Wolfie. Given their complexity, they will be divided into multiple epics and spread out over an extended period.

Data Lakehouse

So far, I have discussed the importance, types, and applications of data. To this end, I need to fulfil a few requirements:

  • Ingesting and storing data from multiple sources.
  • Transforming and cleaning data at scale.
  • Enriching and aggregating data for analytics and consumption.

In short, I need a Data Lakehouse. I’ve written about them before and have followed the Medallion Architecture through bronze, silver and gold layers. For Project Wolfie and moving forward, I’ll be using the well-documented and supported AWS reference architecture:

I find this clearer and more regimented than the Medallion Architecture. It also aligns with the points made in Simon Whiteley‘s Advancing Analytics video, which I agree with.

Of course, a good Data Lakehouse isn’t possible without good data…

Quality & Observability

A Data Lakehouse’s effectiveness depends on data quality and observability. Project Wolfie must address factors like:

Veracity & Validation Checks: Verify data accuracy. Checks such as schema validation, null checks and data quality rules can identify issues early, stopping incorrect data from propagating downstream.

Anomaly Detection: Identify patterns often missed by validation like volume spikes and missing periods. Timely anomaly detection shields downstream resources from requiring remedial measures and lowers unforeseen cloud and developer expenses.

Lineage Tracking: Track the data’s journey from ingestion to consumption, documenting all transformations and processes. Vital for debugging, auditing and validation.

Governance & Security

A Data Lakehouse must balance accessibility and control. Governance and security protocols protect data while encouraging responsible usage.

I own all Project Wolfie data, so I have permission to process it. Additionally, there is no sensitive information or PII. However, there are other factors to consider:

Access Controls: Establish guidelines for who and what can access Project Wolfie resources. This safeguards data and services from unauthorised access, misuse and malicious activities.

Data Controls: Establish criteria for availability, backups, and structure. This aids in managing costs, ensuring disaster recovery, and maintaining schema consistency.

Monitoring & Logging: Track access patterns and record changes to data and infrastructure. This improves visibility into both potential threats and cost-related opportunities and vulnerabilities.

AI & BI Use Cases

Finally, I want to extract value and insights from Project Wolfie using Artificial Intelligence (AI) and Business Intelligence (BI). I have data from 2021 onwards from a music collection I started in the early 2000s, so I have lots to work with!

BI Use Cases (Dashboards, Analytics, Insights)

Listening Trends: Identify traits of my collection’s most frequently played and best-represented music. Analyse listening patterns over time to find trends.

Library Optimisation: Find rarely played tracks to add to playlists. Recognise songs that are often played and recommend alternatives for variety.

Distribution Analysis: Analyse my collection’s main genres, publishers and record labels, and investigate the connections between different elements (e.g., “The most popular tracks are typically in the 120-130 BPM range”). Create reports that show diversity and spread (e.g., “90% of house tracks are in five minor keys”).

AI Use Cases (Machine Learning, Automation, Predictions)

AI-Powered Personalised Playlists: Create playlists using the existing library based on properties like BPMs, keys and previous listening patterns, similar to Spotify Wrapped.

Smart Music Recommendations: Use collaborative filtering to suggest search criteria for new music based on my existing collection and listening habits (e.g., “Try G minor tracks at 128 BPM from the early 2010s”).

Predictive Analysis: Use Technical and Descriptive Metadata from new tracks to predict how they will be rated based on my existing library’s metadata (e.g., “This track has harmonic similarities to 70% of your highly rated tracks.“).

Summary

In this post, I discussed our late German Shepherd Wolfie and outlined a project that utilises music data in his memory.

Wolfie enjoyed scent games and retrieving toys, making Project Wolfie’s mission to find and return data and insights a fitting tribute. As the project evolves, I will strengthen its capabilities using new architectures and technologies, honouring Wolfie’s spirit – one track at a time.

Wolfie was more than just a pet; he was a companion and a guardian each day. I miss you big man. Take care out there.

PXL 20211218 180730545.MP min bw

If this post has been useful then the button below has links for contact, socials, projects and sessions:

SharkLinkButton 1

Thanks for reading ~~^~~

Categories
Me

YearCompass 2024-2025

In this post, I use the free YearCompass booklet to reflect on 2024 and to plan some professional goals for 2025.

Table of Contents

Introduction

It’s time for the customary retrospective post! Previously, I’ve written these in December. From now on, I’ll post them on amazonwebshark’s birthday in January because:

  • It gives me more time to consider my goals.
  • If I’m going to write about a year, it makes more sense to write after the year is over.
  • In previous years, I’ve felt personal pressure to produce both a December YearCompass post and a January birthday post. My YearCompass post can cover both of these instead.

Firstly, I’ll examine YearCompass itself. Next, I’ll discuss how 2024 went. Finally, I’ll examine my 2025 YearCompass goals.

About YearCompass

From the YearCompass site:

YearCompass is a free booklet that helps you reflect on the year and plan the next one. With a set of carefully selected questions and exercises, YearCompass helps you uncover your own patterns and design the ideal year for yourself.

YearCompass offers a free alternative to New Year’s Resolutions. Available as A4 and A5 PDFs in several languages, the booklet can be completed digitally or by hand.

YearCompass PDFs contain two sections: reflecting on the past year and planning for the next. Each section features prompts to help users assess priorities and prepare for the future.

There are no rigid guidelines for completing YearCompass. Although the book recommends taking breaks between sections, many people choose to complete it in one go. I often focus on specific sections rather than the entire document, as I usually have what I need by a certain point.

2024 Retrospective

In this section, I reflect on my 2024 goals and evaluate my progress with them.

Build Technology Projects

As a cloud enthusiast, I want to complete valuable project builds so that I can develop and validate my knowledge and skills, and have subject matter for future session abstracts.

Exiting 2023, I felt I had done much theory but little practice. I had completed online learning, attended events and recertified my AWS SysOps Administrator certification, yet I still felt lacking in hands-on experience.

Well no more! Throughout 2024, I worked on my WordPress AWS Data Pipeline series, which greatly enhanced my understanding of several AWS services and cloud architectures. Also, I explored the AWS CloudFormation IaC Generator and DuckDB-WASM in February and June respectively.

Separately, in July, I joined Steamhaus as an AWS Consultant, where I build, scale, and optimise cloud solutions for clients. This role involves unique projects spanning diverse organisations, requiring creative problem-solving and offering many opportunities to learn and grow. And 2025 is already shaping up to be an interesting year!

Additionally, I earned the AWS Certified Data Engineer – Associate certification in August, validating my skills in areas including ETL, orchestration, model design and quality assurance.

Learnings from this certification have already found their way into my blog posts, sessions and client projects.

And speaking of sessions…

Build My Personal Brand

As an IT professional I want to build my personal brand so that I improve my soft skills and define my public image.

And to think that, at the start of 2024, I had no idea how this was going to go…

Having done my first session at 2023’s New Stars Of Data online event, 2024 was the first year I stepped onto an actual stage! After presenting my Building And Automating Serverless Auto-Scaling Data Pipelines In AWS session at AWS Community Summit London in April, I went on to present it at several user groupsa paid event and even internationally!

PXL 20240424 1606465872
comsum-talk

This year, I also launched a YouTube channel, primarily intended to practise and enhance my speaking skills. Although the channel has been somewhat dormant, I’ve been kinda busy!

Recently, I began a series of shorts to boost my spontaneous speaking ability. It appears to be working for me, and there’s still plenty to film!

Finally, in September I met Cat Mawdsley and Dan Knowles at Northern Reach for the first time. Northern Reach focuses on providing technology and innovation-driven business engagement initiatives for partners in both the public and private sectors across Northern England.

I was born and raised in Lancashire, and quickly discovered their ambitious plans for the region. We got chatting, one thing led to another and, well, I’ll have some exciting news to share soon about something I’m part of!

Build A Second Brain

As just a normal man I want to build a second brain so that I can organise my resources and work more efficiently.

So this is the goal with the least progress, but only because the first two goals blew up beyond anything I could have envisaged! Having read Building A Second Brain in January, I started putting some of the ideas into practice.

Firstly, I’m a big fan of the CODE information consumption method:

  • Capture
  • Organise
  • Distill
  • Express

In some ways I was already doing this, but lacked a framework or set of steps to follow. I now use CODE in several areas of my life, and while it’s not yet fully embedded everywhere it is starting to make a difference.

However, I’m no fan of the PARA organisational system:

  • Project
  • Area
  • Resource
  • Archive

Tiago Forte and I define Project very differently, and I dislike using Archive to describe anything. Archive always feels like a nondescript collection of stuff, which is what a Second Brain should not be. Even Tiago defines Archive as:

Anything from the previous three categories that is no longer active, but you might want to save for future reference.

https://fortelabs.com/blog/para/

Nah. Not for me.

Ultimately, my Second Brain isn’t where I want it to be right now. But to be fair, a second brain is never really finished as it constantly grows and evolves like a human brain. In 2025, I’ll be examining some Second Brain-related AI and SaaS tools and might make some related content if I think it’ll be helpful.

2025 Goals

In this section, I use YearCompass to decide on my 2025 professional goals. For each goal, I’ll explain my reasoning and then write a user story.

Community Investment

I am deeply grateful for the tech communities that supported me throughout 2024. They offered me opportunities to learn, grow and connect with like-minded peers who share my passion for technology. I was invited to speak at several local and international events, and I strongly believe that my journey to Steamhaus began at AWS community events.

IMG 3907

In 2025, I plan to build on these experiences by continuing to contribute through speaking engagements, writing, and social support, as well as giving operational user group support. In this way, I hope to strengthen and grow the communities that played such an integral role in my 2024 successes.

On a personal note, I look forward to continuing the growth journey that the communities have nurtured. This includes enhancing my confidence, refining interpersonal skills, strengthening relationships and expanding my experiences. Just like any solid investment – everyone benefits.

As an active tech community member, I want to support and grow these communities through content, involvement and participation so that both the communities and I thrive.

That said…

Gestalt Cycles

I did a lot in 2024, and looking back I didn’t allow much downtime. Feelings of ‘I should be doing something’ constantly ran into evenings, weekends, and annual leave, and I increasingly noticed disrupted sleep, diminished health and fitness and heightened anxiety as a result.

I’ve burned myself out before so I recognise the signs. And as keen as I am to continue on this unexpected, wild and incredible journey I’m currently on, I also don’t want to end up utterly cooked.

Turning this into a goal was hard as nothing really fit. The closest match I’ve found so far is Gestalt Cycles. This describes the natural rhythm of completing an experience, from identifying a need to achieving closure.

Each cycle involves several stages:

The idea focuses on completing each stage fully to maintain balance and well-being, avoiding lingering stress while creating space for rest and renewal. This aligns closely with what I need. My challenge hasn’t been starting or finishing tasks – it’s been allowing enough recovery time between them.

This aligns with Animas Coaching‘s Withdrawal stage definition:

Finally, after the satisfaction of the need or desire, individuals withdraw, returning to a state of relative equilibrium. This stage offers an opportunity for rest and reflection before the cycle recommences with a new sensation.

https://www.animascoaching.com/blog/gestalt-cycle-of-experience/

And with Shea Stevens‘ comments based on the works of Rosemarie Wulf et al:

…the goal is to find relief and meet needs in a way that they are assimilated, such that the organism is truly integrating what it takes in, and what is taken in is a good fit for the organism.

https://www.gestalttherapyblog.com/blog/gestalt-cycle

I have 2025 ambitions around fitness, home improvements and landscaping. To enable these, I’ll need to ease off the accelerator occasionally. So with all this in mind:

As an individual with multiple workstreams, I want to apply Gestalt cycles to recognize when a task is complete and take intentional breaks, so that I can improve my wellbeing and maintain consistent energy and focus.

Be The Change

Definitely the most LinkedIn-sounding of the three. But hear me out. This goal stems from Mahatma Gandhi’s quote:

Be the change you wish to see in the world.

Mahatma Gandhi…erm…

Expect that’s not what he said. But I digress.

“Be the change” is about taking personal responsibility to embody the values and actions you want to inspire in others. It emphasises proactivity and self-empowerment over simply waiting for change to happen.

Throughout my career, I’ve had ideas I wanted to share and a strong desire to contribute. However, I have constantly struggled with self-confidence, self-doubt and articulating my thoughts.

This changed in 2024. Through my professional role, speaking engagements and involvement in the AWS community, I gained the confidence and ability to express and implement my insights and ideas. This let me enhance my skills and uncover new opportunities, leading to beneficial outcomes for myself, my colleagues and clients and the wider tech community.

AWSUGLeeds

In 2025, I’m committed to embracing this newfound confidence to present and support ideas and changes that matter to me. By doing so, I hope to build stronger relationships, deliver impactful projects and advance initiatives that reflect my values and passions.

As a hard working dog dad, I want to confidently embrace and act on my ideas and opinions so that I can make meaningful contributions and changes to myself, my peers and clients and the wider tech community.

Summary

In this post, I used the free YearCompass booklet to reflect on 2024 and to plan some professional goals for 2025.

Reflecting on 2024 has highlighted key lessons and opportunities for growth, helping me shape an exciting vision for 2025. I’m eager to see where the new year leads and to share progress along the way! I’ll post updates here and via my social, project and session links, which are available via the button below:

SharkLinkButton 1

Thanks for reading ~~^~~