Categories
Me

YearCompass 2025-2026

In this post, I use the free YearCompass booklet to reflect on 2025 and to plan some personal goals for 2026.

Introduction

It’s time for the customary retrospective post!

Each year, I use YearCompass to reflect on the past year’s events and consider what lies ahead. This post will follow the same structure: I’ll provide a brief overview of YearCompass, reflect on how 2025 went and then look forward to 2026.

Well, mostly the same structure. My 2026 goals don’t use User Stories anymore, as I’ve replaced them with Identity Statements. Let me explain why…

About Identity Statements

In previous years, I’ve used User Stories for YearCompass goals. These are structured, task-oriented and outcome-focused.

They are effective in the right context, but I no longer think they’re the right fit. They’re rigid, assume a clear definition of success and usually frame things in terms of completion. That doesn’t align well with personal growth, which often exists in shades of grey.

So I’ll now be using Identity Statements instead. As discussed in James Clear‘s 2018 book Atomic Habits, Identity Statements are internal character affirmations that focus on the type of person one wishes to become, rather than on the specific tasks one wants to complete. I’m using them because:

  1. They prioritise character development over achieving specific outcomes.
  2. They offer resilience and adaptability. Unlike rigid goals that feel “failed” when criteria aren’t met, an Identity Statement represents a continuous path that can be strengthened through every small action.
  3. They create lasting change by first transforming self-image. This transformation lets the necessary habits and actions naturally emerge from that changed identity.

An example of an Identity Statement is:

I am a disciplined and curious learner who explores new ideas daily, maintains a consistent reading habit, and shares insights to help others grow.

This interview with James Clear further explains the concept of Identity Statements:

About YearCompass

From the YearCompass site:

YearCompass is a free booklet that helps you reflect on the year and plan the next one. With a set of carefully selected questions and exercises, YearCompass helps you uncover your own patterns and design the ideal year for yourself.

YearCompass provides a free alternative to New Year’s resolutions. The booklet is available as A4 and A5 PDFs in multiple languages and can be completed either digitally or by hand.

YearCompass PDFs include two sections: reflecting on the past year and planning for the upcoming year. Each section features prompts designed to help users assess their priorities and prepare for the future.

There are no strict guidelines for completing YearCompass. Some people take breaks between sections, while others prefer to finish it in one sitting. Personally, I prefer to focus on specific sections rather than go through the entire document, since I usually have what I need by a certain point.

2025 Goals

In this section, I reflect on my YearCompass 2025 goals and evaluate my progress with them.

Community Investment

As an active tech community member, I want to support and grow these communities through content, involvement and participation so that both the communities and I thrive.

How It Went

Let’s start this section by talking about Step On It! – a session I wrote after being invited to speak at DTX Manchester‘s event at Manchester Central in April. This was the beginning of a wild ride, yielding six speaking opportunities both domestically in Manchester, Leeds and London, and internationally in Dublin and Utrecht. It still blows my mind that this little shark gets to do any of this!

1758725006790
AWS Community Day NL

Also, I was accepted into the AWS Community Builder program in March. People apply to the program for various reasons, and my personal reasons were:

  • Early access to sessions on new products and services before official launch.
  • Opportunities to share ideas and challenges with other builders.
  • The promotional credits, which have been very handy!

Community Builders must reapply annually, and I’m hopeful for a second year!

CommunityBuilder

Elsewhere, I said yes to two opportunities that pushed me out of my comfort zone: appearing on Karl & Jon‘s LogiCast podcast and returning to Ada for an IaC lecture. I took both on deliberately because I want to practise being more spontaneous. There is still work to do here, but I am very much up for more of the same in 2026.

1753198852771

There’s also something else to discuss that relates to both this and another of my 2025 goals, but we’ll get to that…

Reflections

Overall, this goal went very well. I’ve decided Community Investment isn’t a goal I’m carrying forward into 2026 for two reasons.

First, I feel that setting the same goal again would be a cop-out, since YearCompass is about progress and moving forward. Additionally, I’ve come to realise that I will continue my community involvement regardless of whether it’s on a goal list. It has become a natural part of how I operate, rather than something I need to commit to explicitly.

So instead of formalising it again, I want to focus on something new that sits alongside it.

Gestalt Cycles

As an individual with multiple workstreams, I want to apply Gestalt cycles to recognize when a task is complete and take intentional breaks, so that I can improve my wellbeing and maintain consistent energy and focus.

How It Went

I set this goal after feeling burned out at the end of 2024. My sleep was off, my energy was low, and while I had plenty of ideas I was doing too much without taking breaks.

Gestalt appealed because it focuses on completing tasks and taking breaks, rather than always being in motion. I wanted to move away from constantly being always-on; instead, I wanted to create time for non-work activities without feeling like I have to earn this time through exhaustion.

In early 2025, I went through a light version of To-Do List Bankruptcy. I reviewed my long-running tasks to determine whether they still held value or if they were just taking up space. Several items were removed or deprioritised. More importantly, I resisted the urge to replace them right away. And while leaving my task list shorter initially felt uncomfortable, that open space was exactly what I had been missing.

This change brought some clear benefits. I found more time for reading – not only by reading more books but also by letting some go. Completing a book or deciding not to finish one became a signal that a cycle was complete, rather than something to optimise continuously. This approach created space for the books I actually wanted to read, including two that I reviewed.

I also used the time to take better care of myself. I joined the Steamhaus Vitality scheme in March and regularly used the provided fitness and wellbeing apps, reaching Vitality’s Platinum status in November.

VitalityCombo

Additionally, I started running again and ended the year with 350 miles, my furthest since 2021. Progress has been slow, but consistent. Running, in particular, gave me a clear sense of a complete cycle: start, effort, finish, recover – then rest before the next run.

Fetch2025

Reflections

In summary, this was a partial success. My physical and mental health have both improved, and I am now much more aware of my energy levels and overall wellbeing than I was a year ago.

However, one part of the Gestalt cycle still needs work – withdrawal. In theory, withdrawal should lead to rest and renewal. In practice, I can recognise when something is finished, but I don’t yet have a clear sense of what proper rest looks like for me.

It’s not a failure of the idea – it’s a signal that further insight is needed. There’s still work to do in understanding how I recover, not just how I stop. This directly connects to one of my goals for 2026.

Be The Change

As a hard working dog dad, I want to confidently embrace and act on my ideas and opinions so that I can make meaningful contributions and changes to myself, my peers and clients and the wider tech community.

Ah – the LinkedIn goal. It was well-intentioned (and loosely based on something already in motion at the time), but the wording isn’t my style and isn’t something I’ll carry forward.

Right. In my YearCompass 2024-2025 post I said:

…I’ll have some exciting news to share soon about something I’m part of!

Well, I launched a user group! Let’s talk about AWS User Group Lancashire.

RoadToAWSLancashire800600

How It Went

Beginnings

In September 2024 I first met Cat Mawdsley and Dan Knowles at Northern Reach. Northern Reach delivers technology and innovation-driven business initiatives for public and private-sector partners in Northern England.

I hadn’t even planned to discuss the prospect of a user group. I mentioned it in passing and Cat snatched my hand off! We exchanged details and agreed to stay in touch. Looking back, I don’t think either of us knew what we’d just gotten ourselves into.

Cat and I bring complementary strengths: she excels at community coordination and enablement, while I bring technical knowledge and connections with community speakers and AWS contacts. We also both have a personal motivation to elevate the region’s profile!

Why Lancashire?

There are several reasons:

  • Personal Connection: I was born and raised in Lancashire, and I’ve always felt a strong connection to the region. It’s home – family, history and a lot of my story starts here.
  • Closing the Distance: Although there are various tech community events in Manchester, Merseyside and Yorkshire – and some in Lancashire – there wasn’t much representation from the major cloud platforms like AWS locally. Often, ‘local’ still meant having to travel. I wanted to create a truly local event to support Lancashire’s growing tech scene.
  • Building a Hub: I wanted to create a space where professionals, students and enthusiasts could come together – somewhere to share ideas, learn and meet others.
  • Growth: I wanted to help put Lancashire more firmly on the UK map for talent, investment and the wider tech community.
Outcomes

The first AWS UG Lancashire event took place in February 2025 as part of the AWS re:Invent recap UKIE tour.

2025ReinventRecapTourPoster

Burnley College hosted us at their BCUC (Burnley College’s University Courses) centre. As a former BC student, this was a proud personal moment. It also meant I got to introduce Matheus Guimaraes to my hometown, during which he met with local authorities and got a tour of the town and college.

That evening, we hosted around forty people – developers, local professionals and a handful of familiar faces from the wider AWS community, all in the same room for the first time.

highres 526373422

We had further events at Chorley in May with Chris Dobson and Matt Houghton, and at Leyland in November with Kate Gowron and Paul Ross.

UGCombo

Reflections

Overall, this was both a big personal win and a wake-up call. I acted on an idea, committed to it publicly and helped create something valuable for Lancashire. Seeing it all come together made the effort incredibly worthwhile.

It also highlighted a few things that are easy to underestimate from the outside.

Leadership Is Scary

Planting a flag and summoning people to it is a vulnerable move. Before the first event, there was a real possibility that nobody would show up – or that they would and regret it. Or that something major would go wrong. Those feelings don’t go away with experience. They just become more familiar and manageable. And I am very much still learning the UG Leader ropes!

The Leyland event illustrated this well. The venue’s large TV, kindly provided by the local council, wasn’t actually attached to the wall! So I ended up balancing a 27-inch monitor on a box of AWS t-shirts and a stack of Costco waters:

PXL 20251126

Nobody in the room seemed to mind, and the event carried on regardless. That experience highlighted something very true about leading community events: things go wrong in unexpected ways, and the job is to stay calm and keep moving.

Leading is still scary. It continues to be scary. And it continues to be worth doing.

Scheduling Is Hard

Scheduling is hard, and scheduling user groups is no exception! Finding dates that worked for speakers, venues, organisers and attendees – all with competing priorities – is a constant challenge. A lot must come together for a community event, and fortunately I have some great people supporting me!

In 2025 there were three UG Lancashire events, and my goal for 2026 is to hold four. As my network of supporters, speakers, and attendees grows, I want to implement processes to manage this expansion and streamline logistics so my ambitions can flourish.

The Demand Is There

One thing became very clear throughout the year: there is real demand for local events in Lancashire. People consistently said how valuable it was to have something nearby instead of having to travel.

This interest extends beyond just AWS. Topics such as serverless computing, modernisation and AI are relevant across multiple vendors and job roles, and there is a clear desire for a local space to discuss them.

That’s not a small thing. It signifies that the work is meaningful and should be done properly. This also connects to the reasons I initially started the group. The focus for 2026 isn’t growth for growth’s sake – it’s about making the group reliable enough to justify the journey, even if it’s just a short one.

Onwards

There’s more to do. Maintaining momentum, ensuring the group stays valuable and balancing it with everything else in my life.

For 2026, the focus is consistency: better scheduling, stronger speaker and promotion pipelines and fewer last-minute scrambles. Not more for the sake of it, but for something reliable, sustainable and worth showing up for.

1740156445555

2026 Goals

In this section, I will use YearCompass to determine my goals for 2026. For each goal, I’ll write an Identity Statement and explain my reasoning.

Cultivate a Maker Mindset

I am an intentional maker – someone who explores the physical and mental world through repair, construction and play. I treat the act of doing as essential for achieving clarity and focus.

Background

2025 was a busy year. Alongside the various work, community and personal projects I’ve already discussed, there were also some smaller activities. While these didn’t receive much attention, they ultimately proved to be very valuable for my mental health.

Two in particular stand out:

Path Maintenance: The back garden paths have gaps where grass grows through, creating uneven and potentially hazardous areas. Weeding is only a temporary fix as the grass quickly regrows, and using weed killer is unsafe for pets and can contaminate the garden.

So, over the summer, I learned how to mix cement, bought the necessary materials and spent some time on part of the path, a section of which is shown below. It’s been grass-free and much safer ever since!

PathsBeforeAfter

Electronics Repair: Our hedge trimmer stopped working due to a damaged cable. A new one would cost £100+, and without it the garden was getting unruly.

So I learned to use wire strippers, safely splice cables and properly apply heat-shrink tubing. After that, I set to work fixing the cable. While the result isn’t the best-looking, it’s electrically safe and I can use the trimmer now!

PXL 20251104 145300142600

So yes – I’ve turned into a tinkerer! And I want to explore this.

Plan

In 2026, I want to intentionally set aside more time for tinkering, building and fixing things. It turns out I enjoy it, both for satisfying my curiosity and as a respite from my technology-focused main workload.

I’ve already felt the benefits. Fixing things around the house, repairing various breakages and even just experimenting – it all created natural breaks between larger pieces of work. It supported exactly what I was aiming for with my previous Gestalt and Cult Of Done goals: a sense of definite completion, followed by a full pause.

So in 2026 I’m going to carve out time and focus for more of it! Some of this time will be spent on larger, varied projects like training roses and fixing iPods. Others will be smaller acts of ‘adult play’, such as soldering electronic kits or building Lego models.

I’m not planning to document all of it. Keeping it informal and unstructured is part of the point. But I’ll share the occasional snapshot on the shark socials, and maybe even dust off that YouTube channel…

Meaningful Milestones

I am someone who pursues goals with intention. Before committing to something significant, I ask what it will mean when it’s done – not just whether it can be done. The work that stays with me is never the work I finished fastest.

Background

As discussed, 2025 was a big year.

Eighteen months into Steamhaus, the role continues to grow both technically and interpersonally. Elsewhere, speaking at conferences and on podcasts became more regular, as did leading the user group. Alongside that, multiple blog posts, three exams, six talks across three countries and finally making a dent in the reading pile.

BooksReadSmall
Some of 2025’s varied reading list

That’s a lot. But looking back with YearCompass, not all of it felt meaningful.

I mentioned in the Gestalt section that I’d already trimmed my to-do list going into 2025. Even so, some surviving goals still felt like going through the motions. The certs didn’t feel like they deserved celebration. Some of the books got speed-read. Tasks were getting done, but…so what?

What those goals had in common was a simple end state: done. There was no answer to the question of what actually changes when they’re finished. Compare that to repairing the hedge trimmer or cementing the garden path. Those stuck out. Not because they were hard, but because the result mattered afterwards. The garden looked better. Equipment returned to use.

The certs and books weren’t wasted time. Certifications validate current knowledge and expose gaps worth filling, and the books were definitely worthwhile. But there’s only one of me and only so many hours in the day. My overloaded task list quietly crowded out personal hobbies, family time and recreation. The ratio was off.

Plan

The lesson going into 2026 is simple: where possible, pursue tasks and milestones that mean something. This won’t apply to everything – quick tasks are fast by nature, and some things just need doing. But for anything that needs sustained effort, having a clear answer to why this matters will be vital.

The AWS DevOps Professional certification is a good example. The practical reason is simple – my SysOps Associate expires in October and the Pro cert will also renew that one. But it’s also the first Professional-tier AWS certification I have attempted. Earning this will do what the Associate certifications haven’t: eliminate the background noise of imposter syndrome with something concrete and tangible. Therein lies the meaningful aspect for me – it’ll improve my self-image.

So keep an eye on my Credly badges this year!

And speaking of improved self-image…

Metacognition

I am an architect of my own thinking. I pay attention to how I think, not just what I think, so that I can respond with clarity rather than react on instinct.

Background

Last year’s Gestalt goal was about recognising the rhythm between effort and rest. And while it changed how I think about energy, it also raised another question: If the pattern of thinking matters, what about the quality of the thinking itself?

That question led me back to The Chimp Paradox by Steve Peters – a book I’d owned for years but never properly finished. And when I finally did, the themes around self-regulation, emotional response, and the gap between instinct and intention stuck with me. This started me on a new journey that eventually led to metacognition.

Metacognition is usually defined as an awareness of one’s own thinking processes – often described as “thinking about thinking.” Psychologist John H. Flavell introduced the term in the 1970s, and it is commonly broken down into three stages: planning, monitoring, and evaluation. Establish a goal and select suitable tools, monitor their effectiveness and evaluate the outcomes.

This approach treats the mind like a system, focusing on visibility, feedback loops and continuous improvement. Something I can easily relate to as a technologist!

Plan

There are two areas of my life where I feel metacognition could help:

Learning Metacognition

Metacognition is a valuable tool in education that helps learners understand their thinking processes and take control of their learning. By reflecting on their performance and learning from successes and failures, students can adjust their strategies and problem-solving methods rather than just trying harder.

This aligns well with my responsibilities at Steamhaus, where I must regularly assess my knowledge and address any gaps in relation to client projects. The rapidly changing technology landscape can feel overwhelming, and by enhancing my metacognitive skills, I can better identify what to learn and recognise when I have a sufficient understanding of a given topic.

This mindset influences not only my work but also my daily life. Running a household, tending a garden and exploring new interests often need me to learn new things, adjust as needed and evaluate what actually works. By using a more thoughtful metacognitive approach in these areas, I can better prepare for unexpected challenges in the future.

Psychological Metacognition

Metacognition in psychology lets individuals recognise their thought patterns, monitor their emotional responses and understand the root causes of their reactions. This understanding fosters the creation of healthier coping strategies and encourages more adaptive behaviours.

I’ve spoken previously about my struggles with anxiety, insecurity and patterns of overthinking. My hope is that metacognition will help me identify emotional triggers, challenge automatic assumptions and respond more intentionally in the moment.

Additionally, I want to investigate whether this approach can enhance my emotional regulation, decision-making under stress, and communication and boundary-setting with others.

Summary

In this post, I used the free YearCompass booklet to reflect on 2025 and to plan some personal goals for 2026.

The process has highlighted several important lessons—especially regarding what feels meaningful to me, how I manage my energy, and how I approach personal growth. As a result, I’m trying something different this year: instead of focusing on what I want to achieve, I am concentrating more on the kind of person I want to become.

I’m curious to see where this leads. I will share updates here and on my social media, as well as in projects and sessions. You can find these by clicking the button below.

SharkLinkButton 1

Thanks for reading ~~^~~

Categories
Data & Analytics

Building A Basic Amazon Bedrock Structured Data Knowledge Base

In this post, I create a structured data knowledge base using Amazon Bedrock and test it by allowing it to query a database directly. Here’s what worked, what didn’t, and what it cost.

Introduction

In my last post, I built an unstructured Bedrock knowledge base using an S3-backed vector store. I now want to see how Bedrock behaves when it can use structured, relational data instead.

2025 12 25 KBCreate

A Bedrock structured knowledge base doesn’t use embeddings or vector search. Instead, it relies on the database schema. Bedrock generates SQL queries from natural language prompts, executes them directly against the data source and then formats the results into a user response.

While researching November’s post, I checked the query engine options for a Bedrock structured data knowledge base out of curiosity. I was greeted by a choice of…Redshift:

2025 12 25 KBQueryEngine

So fair enough – Redshift it is!

In this post, I will first load curated Parquet data into Redshift Serverless via S3. Next, I’ll create a Bedrock structured data knowledge base and synchronise it with my data.

Then I’ll pose increasingly challenging questions to the knowledge base, ranging from simple counts to complex aggregations. I’ll assess Bedrock’s strengths, analyse any misconceptions, and examine how prompt phrasing can affect results. Finally, I’ll review the costs associated with both Bedrock and Redshift.

To manage expectations, this post is essentially a POC. As such, the following are out of scope:

  • Benchmarking SQL or model performance.
  • Evaluating general model accuracy.
  • Production hardening.

Now, let’s get some data into AWS.

Data Preparation

In this section, I’ll create a minimal data pipeline using Parquet, S3 and Redshift Serverless.

Dataset Context

Let’s begin by talking about the data.

The data is sourced from my personal iTunes library and represents a snapshot of track-level metadata at a specific time. Each row corresponds to a single track, with columns capturing both standard attributes and user events.

Some columns mentioned later should be explicitly defined:

  • mtg_artist – The artist associated with the track.
  • mtg_publisher – The record label or publisher associated with the track.
  • my_rating – A personal track rating assigned in iTunes, on a fixed numeric scale.
  • plays – The total number of times each track was played on iTunes at the time of the snapshot.

File Creation

This process starts by creating a Parquet file. Parquet has been an industry standard for a while now, has many benefits and is ideal for this use case.

Now – full disclosure: this process is currently a bit hacky and uses an ad-hoc local Python script. I will write a proper ETL process as part of Project Wolfie. For now, it’s a combination of pandas .join and .to_parquet functions.

As for the data itself, I did mention it briefly in my February 2025 and May 2025 posts, but admittedly not in much depth. Project Wolfie’s Roadmap does include data dictionaries for all data sources, so they will exist – they just don’t currently. That said, the absence of data dictionaries won’t significantly impact this post, as my main focus is on how the Bedrock knowledge base interacts with the structured data.

S3 Setup

Next, let’s create an S3 bucket. I’ve created a temporary new bedrock-test general-purpose bucket for this post with default config. I then created an iTunesTest folder and uploaded my curated_data.parquet file. The names of the folders and buckets aren’t mega important here; what matters most is knowing the S3 path, as I’ll need it soon.

With the S3 bucket taken care of, it’s time to sort out Redshift. But which Redshift?

Redshift Provisioned Vs Redshift Serverless

There are currently two Redshift offerings, each with its own implementation.

Redshift Provisioned is the original deployment model, in which users create and manage a fixed-size cluster with specific node types. Capacity is chosen upfront, and the cluster is always available.

Redshift Provisioned is ideal for steady workloads that need consistent performance and precise control. However, it requires capacity planning and incurs charges even when not in use, making it less suitable for small or irregular workloads.

Redshift Serverless removes the need for cluster management by automatically adjusting compute capacity in response to demand. It mostly behaves like a standard Redshift endpoint, while AWS oversees the underlying infrastructure.

Redshift Serverless is ideal for low-volume or infrequent workloads, as you only pay for the compute power you consume and don’t need to manage capacity. However, it offers less control than provisioned clusters, and costs may exceed those of provisioned clusters if usage goes beyond a certain threshold.

Here, I’m using Redshift Serverless because:

  • Intermittent Workload: The data will be queried infrequently, so an always-on cluster is unnecessary.
  • Operational Simplicity: I want to focus on the AI POC, not on managing and sizing infrastructure.
  • Free Trial: Redshift Serverless offers a free trial for new users. It’d be rude not to…

Decision made. Next, let’s configure Redshift Serverless.

Redshift Serverless Setup

One of Redshift’s Serverless perks is its simple setup. Instead of explaining this myself, I’ll refer to two AWS videos. The first focuses mainly on the settings:

And the second explores Redshift’s Query Editor:

Here, I use the defaults again. I end up with a workgroup for controlling resources, scaling and execution prioritisation, and a namespace for holding tables, workgroups, and other database resources:

2025 12 25 RedshiftCreationDec25

Next, I need to put my Parquet data into a Redshift table. I’ll be using Redshift’s COPY command for this, but as the target table must already exist in the database, I need to write a Data Definition Language (DDL) command first.

To make the process as painless as possible, I write a quick pandas Python script to get the curated_data.parquet file schema:

Python
import pandas as pd

# Use .read_parquet to capture the data in a DataFrame
df = pd.read_parquet("curated_data.parquet")

# Use .dtypes to get the schema
print(df.dtypes)
Plaintext
Name object
Artist object
Album object
Work object
Genre object
Size int64
Time int64
...

And then pass the output to Amazon Q Developer to write a DDL statement for Redshift:

Convert this Parquet schema into a Redshift DDL statement:

Name object
Artist object
Album object
Work object
Genre object
Size int64
Time int64

…

Plaintext
Here's the Redshift DDL statement for your Parquet schema:

CREATE TABLE itunes_data(
    Name VARCHAR(500),
    Artist VARCHAR(500),
    Album VARCHAR(500),
    Work VARCHAR(500),
    Genre VARCHAR(100),
    Size BIGINT,
    Time BIGINT
    ...

Key mappings:

int32 → INTEGER
int64 → BIGINT
float64 → DOUBLE PRECISION
datetime64 → TIMESTAMP
object → VARCHAR(500) (adjust size as needed for your data)

I then run this script in Redshift to create an empty itunes_data table. COPY now has everything it needs to put the data into Redshift:

SQL
COPY dev.public.itunes_data
FROM 's3://bedrock-test/iTunesTest/curated_data.parquet'
IAM_ROLE 'arn:aws:iam::XXXXXXXXXX:role/service-role/AmazonRedshift-CommandsAccessRole-20251225T125455'
PARQUET

In this command:

  • COPY is the name of the target Redshift table.
  • FROM is the path to the S3 objects containing my data.
  • IAM_ROLE is the method the cluster uses to authenticate with, in this case, S3. This role was created during the Redshift Serverless default setup process.
  • PARQUET tells Redshift what format the data is in.

Full documentation is at the COPY from Amazon S3 page. The end result is a Redshift table containing the Parquet data:

2026 01 16 22 19 03 ReshiftResults

I have now prepared my data. Now I can start on the knowledge base!

Knowledge Base Build

In this section, I’ll build a Bedrock structured data knowledge base. Firstly, I’ll set up my query engine and permissions. Next, I’ll create a Bedrock knowledge base and connect it to my Redshift structured data store. Finally, I’ll synchronise the Rdshift structured data store with the Bedrock knowledge base to enable querying.

Knowledge Base Setup

Configuring a Bedrock structured data knowledge base starts out very similarly to my recent unstructured one, albeit with one noticeable difference at the start:

Selecting a structured knowledge base.

This leads to the Query Engine selection screen shown earlier. After selecting Redshift and either creating or selecting an IAM service role for Bedrock, I must then configure the connection options for my chosen Redshift offering:

2025 12 25 KBQueryEngineConnection

The next step identifies where the structured data is stored. In this case, my itunes_data table is located in the dev database’s pubic schema, so I select dev from the database list:

2025 12 25 KBQueryEngineStorageRS

I must now give authentication information to connect to my Redshift database. I can choose between the IAM role created in the previous step and AWS Secrets Manager credentials for authentication.

With this completed, the knowledge base has been created. The next step is the synchronisation process.

Data Store Sync

Currently, Bedrock has visibility into the schema but not into the data itself. As with an unstructured data knowledge base, I must synchronise the Redshift structured data with my Bedrock knowledge base to make the data queryable:

2025 12 25 KBSync

If the credentials are incorrect or lack proper database permissions, the sync will fail:

2025 12 25 KBSyncFail

Bedrock logs issues for review. Typically, IAM updates and/or SQL grants are needed to resolve them.. Here, my Bedrock Execution Role needed access to the public Redshift schema:

SQL
GRANT USAGE ON SCHEMA public TO "IAMR:AmazonBedrockExecutionRoleForKnowledgeBase";

GRANT SELECT ON ALL TABLES IN SCHEMA public TO "IAMR:AmazonBedrockExecutionRoleForKnowledgeBase";

With this done, my sync succeeds and I can start testing my knowledge base.

Knowledge Base Testing

In this section, I will evaluate the Bedrock knowledge base by asking increasingly complex queries. I will start with simple retrievals and then introduce complexity and analytical challenges through ranking, filtering, and aggregation. My goal is to assess comprehension, schema understanding and SQL correctness.

Bedrock isn’t just suggesting SQL here. Instead, it generates and executes SQL directly in Redshift using the knowledge base’s execution role. The queries presented are part of an active execution path rather than just illustrative pseudocode. If the SQL generated is invalid or poorly structured, the query will fail just as it would if I ran it myself.

For each query, I will provide the question I submitted to Bedrock, the SQL that Bedrock generated, and a screenshot of the results. The SQL here is formatted by the excellent CodeBeautify SQL Formatter.

Initial Config

Before I can give queries to the knowledge base, I must configure two settings. Firstly, I must identify the type of response I want:

2025 12 25 KBTestOptions

I’ll use Retrieval and Generation since it returns both the response and the generated SQL. I also need to select a model for Bedrock to use for response generation. In this case, Nova Lite is a suitable option:

2025 12 25 KBTestOptionsModels

The knowledge base can now respond to queries.

Simple Queries

Let’s start by asking about the data as a whole:

How many tracks are in the collection?

Bedrock handles this easily. It generates a straightforward COUNT(*) query and gets the correct number:

SQL
SELECT 
  COUNT(*) AS "total_tracks" 
FROM 
  public.itunes_data;
2025 12 26 22 32 22 QuestionTracks

Next, let’s dig a little deeper. This time I ask about a specific record label:

How many tracks are produced by Anjunabeats?

This should be in the hundreds, but Bedrock returns a far different number:

SQL
SELECT 
  COUNT(*) AS "track_count" 
FROM 
  public.itunes_data 
WHERE 
  artist = 'Anjunabeats';
2025 12 26 22 33 34 QuestionAnjunaWrong

This isn’t a problem with Bedrock. Rather, this is ambiguity within my data. The first Anjunabeats release was Anjunabeats – Volume One, meaning this track and its remixes have both an artist and a publisher of Anjunabeats.

ANJ003
Yes ok this is ANJ003 but the cover is nicer – Ed

Here, Bedrock interprets ‘produced’ in the artist sense and queries on the artist column. Instead, it should use the mtg-publisher column, but it has no way to know that. In response, I change the prompt to use ‘published’ instead of ‘produced’ and try again.

How many tracks are published by Anjunabeats?

This time, Bedrock uses my signposting and gets the correct answer by querying mtg-publisher:

SQL
SELECT 
  COUNT(*) AS "track_count" 
FROM 
  public.itunes_data 
WHERE 
  "mtg-publisher" = 'Anjunabeats';
2025 12 26 22 35 45 QuestionAnjunaRight

So the first issue I encountered wasn’t related to SQL at all; it was the ambiguity in how Bedrock interpreted my prompt. There are more effective ways to address this ambiguity that I will explore later in this section.

Multi-Constraint Queries

Now let’s try some more demanding questions. I’m still asking about Anjunabeats releases, but I’m also now asking about track ratings:

What are the highest-rated tracks published by Anjunabeats?

Bedrock correctly selects the name and my_rating columns, and also decides to order the results despite not being asked to – nice touch!

Unfortunately, Bedrock then slips back into old habits and, despite my prompt continuing to use published, assumes I’m asking about Anjunabeats – Volume One again and queries one of the artist columns:

SQL
SELECT 
  "name", 
  "my_rating" 
FROM 
  public.itunes_data 
WHERE 
  "mtg-albumartist" = 'Anjunabeats' 
ORDER BY 
  "my_rating" DESC 
LIMIT 10;

So Bedrock tells me about the various Volume One remixes I own:

2025 12 26 22 41 01 QUestionRateWrong

Ok – let’s try changing the prompt. Now, I’m asking firstly about the record label, and then the ratings:

What are the tracks published by Anjunabeats with the highest rating?

This time Bedrock queries on mtg-publisher and the results are correct:

SQL
SELECT 
  "name", 
  "my_rating" 
FROM 
  public.itunes_data 
WHERE 
  "mtg-publisher" = 'Anjunabeats' 
ORDER BY 
  "my_rating" DESC 
LIMIT 10;
2025 12 29 TracksAnjHighRating

My collection has many top-rated Anjunabeats tracks, so let’s narrow it down by asking about play counts:

Tell me about the tracks published by Anjunabeats with the highest rating. I want to know which tracks have the most plays. Give me the artist, title and year.

Bedrock modifies its query. The artist, title and year columns are all correctly selected, and it makes sense to remove my_rating since the tracks I’m asking about all have the same rating value. The query also now has i aliases – not much use for this query but helpful for joins and window functions.

Unfortunately, although I asked for tracks in the plural, Bedrock returns only 1 result:

SQL
SELECT 
  i.artist, 
  i.name, 
  i.year 
FROM 
  public.itunes_data i 
WHERE 
  i."mtg-publisher" = 'Anjunabeats' 
ORDER BY 
  i.my_rating DESC, 
  i.plays DESC 
LIMIT 1;
2025 12 29 HighRatingMostPlaysLimit1

Why? No idea. To prevent this from happening again, I request five tracks along with some additional fields.

Tell me about the tracks published by Anjunabeats with the highest rating. I want to know the 5 tracks with the most plays. Give me the artist, title, year, initial key and BPM for each track.

Bedrock responds by…talking about Volume One again:

SQL
SELECT 
  "artist", 
  "name", 
  "mtg-year", 
  "mtg-initialkey", 
  "mtg-bpm" 
FROM 
  public.itunes_data 
WHERE 
  "mtg-albumartist" = 'Anjunabeats' 
ORDER BY 
  "plays" DESC 
LIMIT 5;
2025 12 29 1602 FiveTracksMostPlayed

This is definitely something to consider for the future. Anjunabeats isn’t the only Artist/Publisher combo in this dataset! And while I love me some Volume One, it’s not this query’s focus.

So I spell it out in the prompt’s first line:

Tell me about the tracks where the publisher is Anjunabeats. I want to know the 5 tracks with the highest rating and the most plays. Give me the artist, title, year, initial key, BPM and play count for each track.

Success!

SQL
SELECT 
  "artist", 
  "name", 
  "mtg-year", 
  "mtg-initialkey", 
  "mtg-bpm", 
  "plays" 
FROM 
  public.itunes_data 
WHERE 
  "mtg-publisher" = 'Anjunabeats' 
ORDER BY 
  "my_rating" DESC, 
  "plays" DESC 
LIMIT 5;
2025 12 29 1607 FIveTracksMostPlayedRight

(If these play counts seem low for two-decade-old tracks, all I can say is that record decks don’t collect metadata – Ed)

Complex Query

Let’s see how far we can go! For the last test, I started with a fairly hefty prompt:

Tell me about the tracks where the publisher is Anjunabeats. I want to know about the tracks with the highest rating. I want the following about the tracks meeting these criteria:

– An overall total of tracks

– Totals for each initial key, ordered by initial key

– Totals for each decade (eg 2000 to 2009, 2010 to 2019 etc)

And, well, I broke it:

2025 12 29 1615 QueryFail

Bedrock gave it a good go, but the SQL it sent to Redshift was invalid:

SQL
SELECT 
  COUNT(*) AS "Total Tracks", 
  COUNT(DISTINCT "mtg-initialkey") AS "Total Tracks with Initial Key", 
  COUNT(
    CASE WHEN "mtg-publisher" = 'Anjunabeats' THEN 1 END
  ) AS "Total Anjunabeats Tracks", 
  COUNT(
    CASE WHEN "mtg-publisher" = 'Anjunabeats' 
    AND my_rating IS NOT NULL THEN 1 END
  ) AS "Total Anjunabeats Tracks with Rating", 
  (
    SELECT 
      MAX(my_rating) 
    FROM 
      public.itunes_data 
    WHERE 
      "mtg-publisher" = 'Anjunabeats' 
      AND my_rating IS NOT NULL
  ) AS "Highest Rated Anjunabeats Track", 
  SUM(
    CASE WHEN "mtg-publisher" = 'Anjunabeats' 
    AND my_rating IS NOT NULL THEN 1 ELSE 0 END
  ) OVER (
    PARTITION BY FLOOR("mtg-year" / 10)
  ) AS "Totals by Decade" 
FROM 
  public.itunes_data 
WHERE 
  "mtg-publisher" = 'Anjunabeats' 
  AND my_rating IS NOT NULL 
ORDER BY 
  "mtg-initialkey";
Plaintext
Redshift query execution failed with error: 
ERROR: column "itunes_data."mtg-publisher"" must appear 
in the GROUP BY clause or be used in an aggregate function.

Fair enough. I would probably break this into multiple queries as well. Let’s ease the pressure a bit by splitting the prompt.

Complex Query: Totals

Firstly, let’s request the overall total of tracks published by Anjunabeats with the highest rating:

Tell me about the tracks where the publisher is Anjunabeats. I want to know about the tracks with the highest rating. I want the following about the tracks meeting these criteria:

– An overall total of tracks

This is a COUNT(*) like the initial query. Bedrock also adds a subquery to get the Anjunabeats MAX(my_rating) value instead of the dataset MAX(my_rating) value:

SQL
SELECT 
  COUNT(*) AS "Total Tracks" 
FROM 
  public.itunes_data 
WHERE 
  "mtg-publisher" = 'Anjunabeats' 
  AND my_rating = (
    SELECT 
      MAX(my_rating) 
    FROM 
      public.itunes_data 
    WHERE 
      "mtg-publisher" = 'Anjunabeats'
  );

This is sensible. While the MAX(my_rating) value is 100 for both the Anjunabeats record label and the whole dataset, this isn’t true for other labels in the dataset. This version of the query is easier to reuse.

The answer is correct, and I have my baseline value for the other queries – 44 tracks:

2025 12 29 1619 HighRatingTotal

You might think 44/539 is a small number. But I don’t hand out top ratings lightly!

Complex Query: Initial Key Ordering

Now let’s try the second part of the prompt that introduces key aggregation:

Tell me about the tracks where the publisher is Anjunabeats. I want to know about the tracks with the highest rating. I want the following about the tracks meeting these criteria:

– Totals for each initial key, ordered by initial key

And, uh…

2025 12 29 1619 InitialKey

This response feels like a placeholder. I repeated the same query multiple times, and each time I got the same result. I’ll revisit this later.

Complex Query: Decade Grouping

Finally, let’s try the last part of the initial prompt that introduces decade aggregation:

Tell me about the tracks where the publisher is Anjunabeats. I want to know about the tracks with the highest rating. I want the following about the tracks meeting these criteria:

– Totals for each decade (eg 2000 to 2009, 2010 to 2019 etc)

Bedrock creates a new calculated column for this, using an interesting method to calculate decades that I hadn’t considered! For example, take 2005:

  • 2005 / 10 = 200.5
  • FLOOR(200.5) = 200
  • 200 * 10 = 2000

Much better than the CASE statement I was expecting:

SQL
SELECT 
  FLOOR("mtg-year" / 10) * 10 AS decade, 
  COUNT(*) AS total_tracks 
FROM 
  public.itunes_data 
WHERE 
  "mtg-publisher" = 'Anjunabeats' 
  AND my_rating = (
    SELECT 
      MAX(my_rating) 
    FROM 
      public.itunes_data 
    WHERE 
      "mtg-publisher" = 'Anjunabeats'
  ) 
GROUP BY 
  FLOOR("mtg-year" / 10) * 10 
ORDER BY 
  decade;

Bedrock confirms what I already knew – that the 2000s were a great time and I miss them terribly:

2025 12 29 1620 BPMRange

Also, as in the first query, these results add up to 44.

Complex Query: Initial Key Ordering Retry

Let’s go back to that failing query from earlier. How about I just ask for the totals and drop the ordering requirement:

Tell me about the tracks where the publisher is Anjunabeats. I want to know about the tracks with the highest rating. I want the following about the tracks meeting these criteria:

– Totals for each key

Bedrock apparently has no issues with this:

SQL
SELECT 
  "mtg-initialkey", 
  COUNT(*) AS "Totals" 
FROM 
  public.itunes_data 
WHERE 
  "mtg-publisher" = 'Anjunabeats' 
  AND "my_rating" = (
    SELECT 
      MAX("my_rating") 
    FROM 
      public.itunes_data 
    WHERE 
      "mtg-publisher" = 'Anjunabeats'
  ) 
GROUP BY 
  "mtg-initialkey";
2025 12 29 1621 Keys

But this doesn’t add up to 44! This time I was curious to see what data Bedrock was getting, so I added an ORDER BY clause to the query and ran it in Redshift:

2025 12 29 1625 Redshift

So the SQL is valid, and the data is correct and equals 44. The issue here is that Bedrock is summarising the results. In an attempt to stop this, I asked Bedrock to order the results by totals like I had just done:

Tell me about the tracks where the publisher is Anjunabeats. I want to know about the tracks with the highest rating. I want the following about the tracks meeting these criteria:

– Totals for each key, ordered by totals

Bedrock’s SQL is fine again:

SQL
SELECT 
  "mtg-initialkey", 
  COUNT(*) AS "Totals" 
FROM 
  public.itunes_data 
WHERE 
  "mtg-publisher" = 'Anjunabeats' 
  AND "my_rating" = (
    SELECT 
      MAX("my_rating") 
    FROM 
      public.itunes_data 
    WHERE 
      "mtg-publisher" = 'Anjunabeats'
  ) 
GROUP BY 
  "mtg-initialkey" 
ORDER BY 
  "Totals" DESC;

But this response loses precision after the Gm and Bm keys. Bedrock correctly says 19 keys are in the full list, but is vague beyond this:

2025 12 29 1629 InitialKey

In one final attempt to get the kind of response I’m after, I tell Bedrock not to summarise the data:

Tell me about the tracks where the publisher is Anjunabeats. I want to know about the tracks with the highest rating. I want the following about the tracks meeting these criteria:

– Totals for each key, ordered by totals. do not summarise the data

Bedrock runs the same query as last time…

SQL
SELECT 
  "mtg-initialkey", 
  COUNT(*) AS "Totals" 
FROM 
  public.itunes_data 
WHERE 
  "mtg-publisher" = 'Anjunabeats' 
  AND "my_rating" = (
    SELECT 
      MAX("my_rating") 
    FROM 
      public.itunes_data 
    WHERE 
      "mtg-publisher" = 'Anjunabeats'
  ) 
GROUP BY 
  "mtg-initialkey" 
ORDER BY 
  "Totals" DESC;

…and kind of gives me what I want. Bedrock doesn’t give me all of the keys with 2 tracks and gives me none of the keys with 1 track:

2025 12 29 1630 KeyFInal

I decided that was far enough. This is now crossing into prompt engineering, and the whole point of all this is to test the knowledge base’s ability to use structured data. In that respect, these tests have been successful.

But could I have done things differently?

Future Improvements

I want to end this section by holding my hands up. While Bedrock struggled with some of the tasks I gave it, there were several ways I could have made the process easier for it. As this experience was mainly a proof-of-concept and a learning opportunity, I didn’t implement those measures here.

So instead of concluding with what Bedrock couldn’t achieve, I will outline how I’ll improve future Bedrock implementations as Project Wolfie develops. The following list is not exhaustive – there are definitely unknown unknowns here!

Improved Data Schema: The Parquet file containing my data was created ad hoc, resulting in an unrefined schema with duplicate data and unclear column names. Validation and normalisation processes were also minimal. The Wolfie Data Pipeline will generate a much-improved dataset, which will aid Bedrock’s understanding.

Supportive Descriptions: When creating Bedrock knowledge bases, table and column descriptions, usage notes and additional attributes can be included. These enhance SQL query generation by providing extra context and information about table and column structures. For example, I could have given mtg-publisher a description of ‘record label’ and that might have solved the Volume One issue.

Curated Queries: I can provide predefined examples of questions and answers for the knowledge base, where the questions are natural language queries and the answers are corresponding SQL queries. This gives Bedrock more context for generating SQL. For example, a possible solution to the Anjunabeats confusion could have been:

Plaintext
Q: Tell me about the tracks where the publisher is Anjunabeats.
A: SELECT * FROM public.itunes_data WHERE "mtg-publisher" = 'Anjunabeats'

All things to try out in future versions! Next, let’s talk about costs.

Costs Analysis

This section covers the costs incurred during one week of experimentation in eu-west-1. All figures shown in screenshots are actual charges as reported by AWS. Firstly I’ll examine Bedrock’s costs, followed by Redshift’s.

This account is within my AWS Organisation, and so has access to my Community Builder credits. Some costs will therefore appear as zero until I filter out these credits.

Bedrock Costs

Looking first at Bedrock, my December 2025 bill shows 26 GenerateSQL-StructuredRetrieve requests, totalling USD 0.05:

2025 12 30 20 51 11 CostUsageBill

At the billing level, Bedrock displays a total cost of zero. This indicates how AWS credits are shown, appearing as ‘No region’ credit lines. This information will be relevant shortly!

Now, let’s go to Cost Explorer and adjust the cost granularity. Here are the daily Bedrock expenses for my AWS data sandbox account:

2025 12 30 20 55 36 COstUsageServiceGraph

And this is a table showing the costs filtered by API:

2025 12 30 20 57 53 COstUsageAPITable

Minor rounding artefacts appear at this scale – I’m assuming the 25 December costs are USD 0.00999 and one of the other days is USD 2.00111 or something similar.

Overall, Bedrock costs were low, visible, and easy to reason about. But something is missing…

Redshift Costs

Despite Redshift being central to the architecture, no Redshift charges appear in the billing console during this period. Not even with the debit/credit method shown above. This raises an important question: how exactly does the free trial work?

Well, it turns out that this is by design:

Amazon Redshift Serverless offers a free trial. If you participate in the free trial, you can view the free trial credit balance in the Redshift console, and check free trial usage in the SYS_SERVERLESS_USAGE system view. Note that billing details for free trial usage does not appear in the billing console. You can only view usage in the billing console after the free trial ends.

Billing for Amazon Redshift Serverless

I am NOT a fan of this! I love Cost Explorer for its visibility; everyone with access can view, slice and aggregate the data, quickly identifying trends and hotspots. Not being able to do this with the Redshift Serverless trial feels like a misstep. Surely part of the trial involves understanding Redshift costs like and alongside other AWS services, in order to avoid a big financial shock when the trial ends! Having them separate isn’t great.

Anyway.

There are ways to monitor the costs of a Redshift Serverless trial, but they’re not as intuitive as Cost Explorer. The most obvious way to see how many credits have been used is through the Free Trial widget on the Redshift Serverless dashboard.

2025 12 30 21 00 11 RedshiftTrial

Redshift Serverless is billed using Redshift Processing Units (RPUs). An RPU represents a bundle of compute, memory, and networking capacity, and charges are primarily driven by:

  • The workgroup’s configured Base Capacity.
  • The duration this capacity remains active.

The widget shows USD 56.98 used – higher than I had expected! The billing page mentions SYS_SERVERLESS_USAGE so let’s see what that has to offer:

SQL
SELECT
  * 
FROM
  SYS_SERVERLESS_USAGE
2025 12 30 21 18 56 REdshiftAll

Of particular interest here is the charged_seconds column: the accumulated RPU seconds charged between the start_time and end_time. Handily, the Redshift docs include a query returning aggregated daily costs:

SQL
SELECT
  TRUNC(start_time) "Day",
  (
    sum(charged_seconds)/ 3600 :: DOUBLE PRECISION
  ) * 0.387 as cost_incurred 
FROM
  SYS_SERVERLESS_USAGE
GROUP BY
  1 
ORDER BY
  1
2025 12 30 21 17 26 REdshiftGrouping

That’s… actually more than the widget ($58.62), but I can understand the widget having some lag. So what’s driving these costs?

Redshift Cost Review

Firstly, my workgroup is massively overprovisioned, as the Base Capacity was using the default 128 RPU. Great for performance, but uses a ton of resources and is total overkill for my use case!

Fortunately, Base Capacity can be changed. In Considerations and limitations for Amazon Redshift Serverless capacity, AWS states that configurations with 4 base RPU support managed storage capacity of up to 32 TB. And I have… 2.7GB of total storage used.

I clearly don’t need 128 RPU, so I reduce my Base Capacity to 4 RPUs:

2025 12 30 21 47 37 Performance

While this change led to longer query execution times, expenses quickly stabilised and stopped rising.

Additionally, the Redshift Serverless usage shown is not limited to the Bedrock processes mentioned in this post. During the same time, I was also trying out different features in the Redshift environment, such as:

  • Executing queries manually using my IAM user to explore and test various features of Redshift Serverless.
  • Creating and testing two separate Bedrock knowledge bases (both visible in the Redshift console).
  • Performing ad-hoc exploratory analysis via the Redshift query editor.

Redshift monitors and reports on this activity in the console:

2025 12 30 21 30 03 Users2

My exploratory use of Redshift prolonged workgroup activity, consumed additional RPU and increased costs. As I reduced my exploratory use, the Redshift costs began to align more closely with my expectations.

Summary

In this post, I created a structured data knowledge base using Amazon Bedrock and tested it by allowing it to query a database directly.

Setup Thoughts

Firstly, I was impressed with the respective Redshift and Bedrock setup processes. Although there were some challenges, they primarily involved ensuring services communicated effectively and were resolved quickly. And while some default settings were more helpful than others, they at least provided a starting point for making informed configuration decisions.

Bedrock’s costing is transparent, so I could also make informed cost decisions. And while I’m a big fan of the Redshift Serverless free trial, I wish it was trackable within Cost Explorer. In the meantime, I’ll keep a close eye on SYS_SERVERLESS_USAGE!

Testing Thoughts

Testing the knowledge base was straightforward, and I quickly understood how Bedrock interacted with Redshift. Seeing the actual queries being executed not only helps optimise the knowledge base but also informs potential architectural changes to the database.

The results emphasise the importance of data structure and quality. I’ve worked with data for almost a decade now, and have seen many sales pitches claiming to remove the need for SQL, data modelling or normalisation. The reality is that, especially in the current AI era, well-structured and properly modelled data and an understanding of how to query it effectively remain crucial.

As I’ve shown here, without these data controls, an AI attempting to use such data is likely to produce incorrect, inconsistent or inaccurate outputs. While simple queries and explicit prompts can help mitigate these issues, it is more beneficial to start with clean, well-structured data.

A knowledge base that can access clean, normalised data stores lets users focus on interpreting answers rather than rephrasing questions. This will be a key consideration for the architecture of the Project Wolfie data pipelines.

Finally, I’ll leave you with an AWS demo video in which Lester Sim creates his own Bedrock structured data knowledge base:

Like this post? Click the button below for links to contact, socials, projects and sessions:

SharkLinkButton 1

Thanks for reading ~~^~~

Categories
Data & Analytics

Unstructured Data Embeddings With Amazon S3 Vectors

In this post, I explore how Amazon S3 Vectors works in practice, create an unstructured data Bedrock Knowledge Base using it and review its storage layout, performance and cost.

Introduction

Retrieval-Augmented Generation (RAG) systems rely on three things: accurate embeddings, a reliable vector store and a simple way to link those vectors to an AI model. In July 2025, AWS released Amazon S3 Vectors – a new S3 bucket type that stores vectors in a fast and consistent format. Bedrock Knowledge Bases can also now use S3 Vectors as a managed vector store.

Firstly, I’ll outline the AWS services involved in this post. Next, I’ll build a Bedrock Knowledge Base backed by S3 Vectors and show what a vector bucket actually contains. Finally, I’ll cover S3 Vectors’ main trade-offs in performance, cost and complexity. By the end, you should have a clear view of where S3 Vectors fits in a RAG pipeline and when it’s a good choice for your unstructured data workloads.

For clarity: this is a data-focused post, not an AI-focused post. While Bedrock is involved, my focus here is on using S3 Vectors to expose unstructured data to Bedrock in a simple, scalable way.

AWS Services

This section outlines the AWS data and AI services I’m using in this post.

Amazon S3 Vectors

Amazon S3 Vectors is a new type of S3 bucket built for storing and searching vector embeddings. S3 Vectors manages the indexing and retrieval layer for you, so there’s no need to run your own vector database. The service is intended for large-scale, low-cost vector storage and usually delivers results within the 100-800 millisecond range.

S3 Vectors consists of these key components:

  • Vector Buckets: S3 buckets configured to store and query vector data.
  • Vector Indexes: Structures inside the bucket that organise vectors for similarity search.
  • Vectors: Numerical representations of data used to match and retrieve related content.

This AWS video further explores S3 Vectors:

As of November 2025, Amazon S3 Vectors is in preview release and is available in the following regions:

  • us-east-1 (N. Virginia)
  • us-east-2 (Ohio)
  • us-west-2 (Oregon)
  • eu-central-1 (Frankfurt)
  • ap-southeast-2 (Sydney)

Resources in this post were created in us-east-1 in a sandbox account.

UPDATE: As of 02 December 2025, S3 Vectors is now generally available.

Amazon Bedrock

Amazon Bedrock is a managed AWS service for accessing and managing foundational models (FMs). Bedrock provides tools and integrations to test and customise these models using existing systems and data. It simplifies the creation of generative AI agents and applications while considering security, privacy and responsible AI principles.

Features of Bedrock include:

  • Access to multiple foundation models from vendors including Anthropic, DeepSeek and AWS itself.
  • Serverless configuration, removing the need to provision GPUs, configure model pipelines, or manage any other infrastructure.
  • Integration with other AWS services such as Lambda, S3 and CloudWatch.

While there’s lots more to Bedrock, as it isn’t this post’s focus I’ll link to the AWS Bedrock documentation and include this video:

I will expand on one specific Bedrock feature though…

Bedrock Knowledge Bases

Amazon Bedrock Knowledge Bases is a fully managed RAG service that connects your data to Bedrock models. It supports both unstructured and structured data sources. For unstructured data, it parses documents, chunks them, generates embeddings and stores those embeddings as vectors. Structured data is queried directly without embedding.

Bedrock Knowledge Bases uses vector search to retrieve relevant content and passes that context to a foundation model. It features encryption, IAM integration, data residency controls and automatic scaling under a pay-per-use pricing model. When used with S3 Vectors, AWS manages all vector storage and indexing.

Using S3 Vectors With Bedrock

This section uses S3 Vectors to create an unstructured data Bedrock Knowledge Base.

Firstly, I’ll set up a data source using an S3 general-purpose bucket. Next, I’ll create a vector store using an S3 vector bucket. Finally, I’ll create and test a knowledge base and show what the inside of a vector bucket actually looks like. Since this post focuses on S3 Vectors and unstructured data, I won’t spend too much time on the knowledge base itself.

Data Source Config

To start this process, I need some data for the knowledge base. They support several source file formats, and here I’m using three PDFs from my Calibre library:

These have been chosen deliberately for reasons that’ll become clear. I create a basic, general-purpose S3 bucket and upload the files:

2025 11 12 S3Objects

Then, in the Bedrock Knowledge Base console, I specify this bucket as my data source.

Now there are some decisions to make, and as this post’s primary focus isn’t the knowledge base I use the defaults. As I’m only processing text from these documents, Bedrock’s default parser is ideal for the Parsing Strategy (plus a bonus: it’s free!). The default chunking strategy is also fine. Chunks of 300 tokens are acceptable, and the process respects sentence boundaries which is ideal for the questions I will be asking.

Vector Store Config

With the data source configured, the next step is to configure a vector store. This process is how Bedrock converts my PDFs into vector embeddings.

Firstly, I need to choose a model that supports embedding generation. Currently, the options are Amazon Titan and Cohere Embed – Titan is fine.

Next, I must choose a new or existing vector store type. This section is where we first see S3 Vectors. Regions not offering the S3 Vectors preview currently offer three options for new vector stores:

2025 11 12 VectorStoreNonTrial

In regions that are part of the preview, this section is replaced by a dropdown list with service icons and the S3 Vectors preview option:

2025 11 12 VectorStoreS3V

During the knowledge base creation process, Bedrock creates a new S3 vector bucket. It also establishes the necessary IAM service role to access both the data source bucket and the S3 vector bucket.

When Bedrock finishes, I have a new vector bucket and a knowledge base. At the moment, both are empty though. My vector store needs some vectors…

Data Source Syncing

Once a knowledge base is established, the data needs to be synchronised to make it queryable. This process applies to any vector store, not just S3 Vectors.

Synchronisation transforms the raw data stored in the data store (in this case my PDFs) into vector embeddings within the vector store. The time this process takes depends on factors like data volume and complexity, since these increase the workload:

2025 11 12 KBSync

When it’s done, Bedrock logs the sync job:

2025 11 12 KBSyncHist

And lists the documents indexed within the knowledge base:

2025 11 12 17 59 37 2025 11 12 KBDocumentsEdit

So what does all this look like in S3? Actually, there’s not that much to see. This is what’s in the S3 vector bucket that Bedrock created:

2025 11 12 VectorBucketContents

One vector index is shown – the one created by Bedrock. Additional vector indexes can also be created. Each index can be assigned a dimension (the number of values in a vector), a distance metric (the distance between a query vector and a stored vector), and optional metadata keys for vector enrichment (see the next section for an example).

Everything necessary for the Knowledge Base is now ready. But has it worked?

Knowledge Base Testing

As with any data transformation process, it’s essential to test the new knowledge base and vector store. There are several ways to do this. Here, I’ll query my knowledge base and generate responses based on the retrieved data. Then I’ll personally check them for accuracy.

So let’s give it some prompts! My first one asks for a summary of a specific aspect of data warehouse design:

Tell me about type 2 slowly changing dimensions.

Here’s the AI response:

2025 11 12 TestType2SCD

Sounds reasonable! Each citation links to the document used to create the response – in this case Data Warehouse Toolkit (easily the best choice of the three for this query):

2025 11 12 TestType2SCDExplain
OK I didn’t notice at the time stop laughing at the back – Ed

I can explore further by looking at the Source Details section. Each of the three citations links to the chunk used in the response:

2025 11 12 TestType2SCDDetails

These expand to show the raw text used from the source chunk and, if the data is stored in S3, the metadata associated with the source chunk which is helpful for customisation:

2025 11 12 TestType2SCDDetailsMeta

Let’s try another prompt. This one is more DevOps-oriented, asking about AWS services for a given use case and the initial IaC setup:

I want to deploy a data warehouse with Terraform. Tell me which AWS service I should use, and tell me about how to create a Terraform repo for this.

2025 11 12 TFKBTest

This time Data Warehouse Toolkit is not useful due to its age, so the knowledge base uses Deciphering Data Architectures to recommend Redshift and Terraform Up & Running for a basic HCL template.

Looks good!

S3 Vector Code Examples

Finally, to end this section let’s look at some examples of creating S3 Vectors resources using the AWS CLI. This is how to create a vector bucket called my-vector-bucket:

Bash
# Create a vector bucket
aws s3vectors create-vector-bucket --bucket my-vector-bucket

This command creates a cosine vector index:

Bash
# Create a vector index
aws s3vectors create-vector-index \
  --bucket my-vector-bucket \
  --index-name kb-index \
  --dimension 1024 \
  --distance-metric cosine

And this inserts vectors via embeddings.json:

Bash
# Insert vectors
aws s3vectors put-vectors \
  --bucket my-vector-bucket \
  --index-name kb-index \
  --vectors file://embeddings.json

At this point, we’ve seen S3 Vectors during ingestion, search, and query processing within a Bedrock Knowledge Base. With the basics covered, let’s look at the related costs.

S3 Vectors Costs

This section examines the costs I incurred when using S3 Vectors to create an unstructured data Bedrock Knowledge Base in my sandbox AWS account. The graphs are from Cost Explorer (which I’m somewhat fond of).

These are my data sandbox November costs, filtered to show Bedrock and S3 over the week I was trying them out:

2025 11 18 GraphService

This shows $0.19 for Bedrock and $0.04 for S3. Very reasonable! Here’s the same graph filtered by API operation, showing how those costs are made up:

2025 11 18 GraphAPI

Let’s look deeper into those APIs. Of the two Bedrock APIs used, InvokeModelInference API is the main cost driver:

2025 11 18 BedrockAPINumbers

There are far more S3 API calls but, of the 32 APIs used, only the S3 PutVectors API triggers any cost:

2025 11 18 S3APINumbers

Ultimately, these numbers are from my personal sandbox during a single week of experimentation, so treat them as examples rather than benchmarks. Aspects such as workload, model selection and region will all affect the final price.

So overall I’m pleased with these prices. Manageable costs and no surprises!

S3 Vectors Alternatives

This section examines alternatives to S3 Vectors for handling unstructured data.

Given that S3 Vectors is currently still in preview and subject to change, a full comparison is neither fair nor viable. So this section isn’t intended as a full critique – rather as an examination of options.

For those needing guidance on vector stores, here’s a 2024 AWS vector stores deep dive and some relevant AWS Prescriptive Guidance.

OpenSearch Service

Amazon OpenSearch Service is a distributed search and analytics engine with native vector search and a fully managed serverless mode that auto-scales based on workload. It supports HNSW-based k-NN search, vector-aware filters, and similarity metrics such as cosine and dot-product.

However, OpenSearch Serverless can be expensive. Users on re:Post and Reddit report high bills when using it as a vector store. It can also experience cold-start latency when compute units scale to zero or reload index data from storage.

Choose OpenSearch if…

  • You want a standard high-performance RAG architecture.
  • You have tens or hundreds of millions of embeddings.
  • You need fast ANN search and metadata filtering.
  • You want scalable, high-throughput retrieval.

Aurora PostgreSQL

Amazon Aurora PostgreSQL is a relational DB with optional vector support via the pgvector extension. It allows structured relational data and vector embeddings to be stored together, providing a SQL-first design, ACID guarantees and support for transactional workloads.

Aurora PostgreSQL is well-suited for small-to-medium-sized RAG workloads, particularly those with fewer than 10 million embeddings. It is a good choice for projects that prioritise simplicity over advanced vector performance. Additionally, it is ideal for situations where structured data and embeddings must be stored in the same database, or when maintaining separate vector storage is undesirable.

However, Aurora PostgreSQL struggles with large-scale vector search and high query-per-second (QPS) workloads. It also lacks OpenSearch’s ANN indexing and support for unstructured data, and can be more expensive per GB than S3 and OpenSearch.

Choose Aurora pgvector if…

  • You already use Aurora and want minimal extra infrastructure.
  • Your dataset is small or medium (up to ~5M vectors).
  • You need SQL joins, transactions, and relational modelling.
  • Your application needs both traditional SQL and vector capabilities.

Neptune

Amazon Neptune is a graph database supporting Resource Description Framework (RDF) graphs and Labelled Property graphs (LPGs). This makes it ideal for connected data use cases that are optimised for relationships rather than documents.

Neptune provides graph-based vector search, integrating relationship data with vectors and advanced query capabilities. However, it requires graph modelling, which needs more upfront design effort than the other choices. And if the necessary expertise is lacking, users might find Neptune’s learning curve quite steep.

Choose Neptune if…

  • You run complex relationship-based queries against your data.
  • You want a knowledge graph, not just semantic search.
  • You need graph reasoning and vector similarity.
  • Your RAG workflow depends on graph structure, not document similarity.

Summary

In this post, I explored how Amazon S3 Vectors works in practice, created an unstructured data Bedrock Knowledge Base using it, and reviewed its storage layout, performance and cost.

S3 Vectors provides a substantial cost benefit for the long-term storage of large embedding sets. It is especially effective for handling large volumes of vectors and can achieve retrieval times of 100 to 800 milliseconds.

However, if your workloads require faster lookups or higher throughput, a purpose-built vector database or OpenSearch would be a better choice. Ultimately, the ideal option depends on whether your priority is cost efficiency or query performance.

Finally, I will share a recent AWS video that explores many of the concepts mentioned here. It uses unstructured shareholder letters with S3 Vectors and Bedrock Knowledge Base:

Like this post? Click the button below for links to contact, socials, projects and sessions:

SharkLinkButton 1

Thanks for reading ~~^~~