Month: November 2025

In this post, I explore how Amazon S3 Vectors works in practice, create an unstructured data Bedrock Knowledge Base using it and review its storage layout, performance and cost.

Introduction

Retrieval-Augmented Generation (RAG) systems rely on three things: accurate embeddings, a reliable vector store and a simple way to link those vectors to an AI model. In July 2025, AWS released Amazon S3 Vectors – a new S3 bucket type that stores vectors in a fast and consistent format. Bedrock Knowledge Bases can also now use S3 Vectors as a managed vector store.

Firstly, I’ll outline the AWS services involved in this post. Next, I’ll build a Bedrock Knowledge Base backed by S3 Vectors and show what a vector bucket actually contains. Finally, I’ll cover S3 Vectors’ main trade-offs in performance, cost and complexity. By the end, you should have a clear view of where S3 Vectors fits in a RAG pipeline and when it’s a good choice for your unstructured data workloads.

For clarity: this is a data-focused post, not an AI-focused post. While Bedrock is involved, my focus here is on using S3 Vectors to expose unstructured data to Bedrock in a simple, scalable way.

AWS Services

This section outlines the AWS data and AI services I’m using in this post.

Amazon S3 Vectors

Amazon S3 Vectors is a new type of S3 bucket built for storing and searching vector embeddings. S3 Vectors manages the indexing and retrieval layer for you, so there’s no need to run your own vector database. The service is intended for large-scale, low-cost vector storage and usually delivers results within the 100-800 millisecond range.

S3 Vectors consists of these key components:

• Vector Buckets: S3 buckets configured to store and query vector data.
• Vector Indexes: Structures inside the bucket that organise vectors for similarity search.
• Vectors: Numerical representations of data used to match and retrieve related content.

This AWS video further explores S3 Vectors:

As of November 2025, Amazon S3 Vectors is in preview release and is available in the following regions:

• us-east-1 (N. Virginia)
• us-east-2 (Ohio)
• us-west-2 (Oregon)
• eu-central-1 (Frankfurt)
• ap-southeast-2 (Sydney)

Resources in this post were created in us-east-1 in a sandbox account.

UPDATE: As of 02 December 2025, S3 Vectors is now generally available.

Amazon Bedrock

Amazon Bedrock is a managed AWS service for accessing and managing foundational models (FMs). Bedrock provides tools and integrations to test and customise these models using existing systems and data. It simplifies the creation of generative AI agents and applications while considering security, privacy and responsible AI principles.

Features of Bedrock include:

• Access to multiple foundation models from vendors including Anthropic, DeepSeek and AWS itself.
• Serverless configuration, removing the need to provision GPUs, configure model pipelines, or manage any other infrastructure.
• Integration with other AWS services such as Lambda, S3 and CloudWatch.

While there’s lots more to Bedrock, as it isn’t this post’s focus I’ll link to the AWS Bedrock documentation and include this video:

I will expand on one specific Bedrock feature though…

Bedrock Knowledge Bases

Amazon Bedrock Knowledge Bases is a fully managed RAG service that connects your data to Bedrock models. It supports both unstructured and structured data sources. For unstructured data, it parses documents, chunks them, generates embeddings and stores those embeddings as vectors. Structured data is queried directly without embedding.

Bedrock Knowledge Bases uses vector search to retrieve relevant content and passes that context to a foundation model. It features encryption, IAM integration, data residency controls and automatic scaling under a pay-per-use pricing model. When used with S3 Vectors, AWS manages all vector storage and indexing.

Using S3 Vectors With Bedrock

This section uses S3 Vectors to create an unstructured data Bedrock Knowledge Base.

Firstly, I’ll set up a data source using an S3 general-purpose bucket. Next, I’ll create a vector store using an S3 vector bucket. Finally, I’ll create and test a knowledge base and show what the inside of a vector bucket actually looks like. Since this post focuses on S3 Vectors and unstructured data, I won’t spend too much time on the knowledge base itself.

Data Source Config

To start this process, I need some data for the knowledge base. They support several source file formats, and here I’m using three PDFs from my Calibre library:

• Deciphering Data Architectures – James Serra
• Terraform: Up and Running – Yevgeniy Brikman
• The Data Warehouse Toolkit – Ralph Kimball

These have been chosen deliberately for reasons that’ll become clear. I create a basic, general-purpose S3 bucket and upload the files:

Then, in the Bedrock Knowledge Base console, I specify this bucket as my data source.

Now there are some decisions to make, and as this post’s primary focus isn’t the knowledge base I use the defaults. As I’m only processing text from these documents, Bedrock’s default parser is ideal for the Parsing Strategy (plus a bonus: it’s free!). The default chunking strategy is also fine. Chunks of 300 tokens are acceptable, and the process respects sentence boundaries which is ideal for the questions I will be asking.

Vector Store Config

With the data source configured, the next step is to configure a vector store. This process is how Bedrock converts my PDFs into vector embeddings.

Firstly, I need to choose a model that supports embedding generation. Currently, the options are Amazon Titan and Cohere Embed – Titan is fine.

Next, I must choose a new or existing vector store type. This section is where we first see S3 Vectors. Regions not offering the S3 Vectors preview currently offer three options for new vector stores:

In regions that are part of the preview, this section is replaced by a dropdown list with service icons and the S3 Vectors preview option:

During the knowledge base creation process, Bedrock creates a new S3 vector bucket. It also establishes the necessary IAM service role to access both the data source bucket and the S3 vector bucket.

When Bedrock finishes, I have a new vector bucket and a knowledge base. At the moment, both are empty though. My vector store needs some vectors…

Data Source Syncing

Once a knowledge base is established, the data needs to be synchronised to make it queryable. This process applies to any vector store, not just S3 Vectors.

Synchronisation transforms the raw data stored in the data store (in this case my PDFs) into vector embeddings within the vector store. The time this process takes depends on factors like data volume and complexity, since these increase the workload:

When it’s done, Bedrock logs the sync job:

And lists the documents indexed within the knowledge base:

So what does all this look like in S3? Actually, there’s not that much to see. This is what’s in the S3 vector bucket that Bedrock created:

One vector index is shown – the one created by Bedrock. Additional vector indexes can also be created. Each index can be assigned a dimension (the number of values in a vector), a distance metric (the distance between a query vector and a stored vector), and optional metadata keys for vector enrichment (see the next section for an example).

Everything necessary for the Knowledge Base is now ready. But has it worked?

Knowledge Base Testing

As with any data transformation process, it’s essential to test the new knowledge base and vector store. There are several ways to do this. Here, I’ll query my knowledge base and generate responses based on the retrieved data. Then I’ll personally check them for accuracy.

So let’s give it some prompts! My first one asks for a summary of a specific aspect of data warehouse design:

Tell me about type 2 slowly changing dimensions.

Here’s the AI response:

Sounds reasonable! Each citation links to the document used to create the response – in this case Data Warehouse Toolkit (easily the best choice of the three for this query):

I can explore further by looking at the Source Details section. Each of the three citations links to the chunk used in the response:

These expand to show the raw text used from the source chunk and, if the data is stored in S3, the metadata associated with the source chunk which is helpful for customisation:

Let’s try another prompt. This one is more DevOps-oriented, asking about AWS services for a given use case and the initial IaC setup:

I want to deploy a data warehouse with Terraform. Tell me which AWS service I should use, and tell me about how to create a Terraform repo for this.

This time Data Warehouse Toolkit is not useful due to its age, so the knowledge base uses Deciphering Data Architectures to recommend Redshift and Terraform Up & Running for a basic HCL template.

Looks good!

S3 Vector Code Examples

Finally, to end this section let’s look at some examples of creating S3 Vectors resources using the AWS CLI. This is how to create a vector bucket called my-vector-bucket:

# Create a vector bucket
aws s3vectors create-vector-bucket --bucket my-vector-bucket

# Create a vector bucket
aws s3vectors create-vector-bucket --bucket my-vector-bucket

This command creates a cosine vector index:

# Create a vector index
aws s3vectors create-vector-index \
  --bucket my-vector-bucket \
  --index-name kb-index \
  --dimension 1024 \
  --distance-metric cosine

# Create a vector index
aws s3vectors create-vector-index \
  --bucket my-vector-bucket \
  --index-name kb-index \
  --dimension 1024 \
  --distance-metric cosine

And this inserts vectors via embeddings.json:

# Insert vectors
aws s3vectors put-vectors \
  --bucket my-vector-bucket \
  --index-name kb-index \
  --vectors file://embeddings.json

# Insert vectors
aws s3vectors put-vectors \
  --bucket my-vector-bucket \
  --index-name kb-index \
  --vectors file://embeddings.json

At this point, we’ve seen S3 Vectors during ingestion, search, and query processing within a Bedrock Knowledge Base. With the basics covered, let’s look at the related costs.

S3 Vectors Costs

This section examines the costs I incurred when using S3 Vectors to create an unstructured data Bedrock Knowledge Base in my sandbox AWS account. The graphs are from Cost Explorer (which I’m somewhat fond of).

These are my data sandbox November costs, filtered to show Bedrock and S3 over the week I was trying them out:

This shows $0.19 for Bedrock and $0.04 for S3. Very reasonable! Here’s the same graph filtered by API operation, showing how those costs are made up:

Let’s look deeper into those APIs. Of the two Bedrock APIs used, InvokeModelInference API is the main cost driver:

There are far more S3 API calls but, of the 32 APIs used, only the S3 PutVectors API triggers any cost:

Ultimately, these numbers are from my personal sandbox during a single week of experimentation, so treat them as examples rather than benchmarks. Aspects such as workload, model selection and region will all affect the final price.

So overall I’m pleased with these prices. Manageable costs and no surprises!

S3 Vectors Alternatives

This section examines alternatives to S3 Vectors for handling unstructured data.

Given that S3 Vectors is currently still in preview and subject to change, a full comparison is neither fair nor viable. So this section isn’t intended as a full critique – rather as an examination of options.

For those needing guidance on vector stores, here’s a 2024 AWS vector stores deep dive and some relevant AWS Prescriptive Guidance.

OpenSearch Service

Amazon OpenSearch Service is a distributed search and analytics engine with native vector search and a fully managed serverless mode that auto-scales based on workload. It supports HNSW-based k-NN search, vector-aware filters, and similarity metrics such as cosine and dot-product.

However, OpenSearch Serverless can be expensive. Users on re:Post and Reddit report high bills when using it as a vector store. It can also experience cold-start latency when compute units scale to zero or reload index data from storage.

Choose OpenSearch if…

• You want a standard high-performance RAG architecture.
• You have tens or hundreds of millions of embeddings.
• You need fast ANN search and metadata filtering.
• You want scalable, high-throughput retrieval.

Aurora PostgreSQL

Amazon Aurora PostgreSQL is a relational DB with optional vector support via the pgvector extension. It allows structured relational data and vector embeddings to be stored together, providing a SQL-first design, ACID guarantees and support for transactional workloads.

Aurora PostgreSQL is well-suited for small-to-medium-sized RAG workloads, particularly those with fewer than 10 million embeddings. It is a good choice for projects that prioritise simplicity over advanced vector performance. Additionally, it is ideal for situations where structured data and embeddings must be stored in the same database, or when maintaining separate vector storage is undesirable.

However, Aurora PostgreSQL struggles with large-scale vector search and high query-per-second (QPS) workloads. It also lacks OpenSearch’s ANN indexing and support for unstructured data, and can be more expensive per GB than S3 and OpenSearch.

Choose Aurora pgvector if…

• You already use Aurora and want minimal extra infrastructure.
• Your dataset is small or medium (up to ~5M vectors).
• You need SQL joins, transactions, and relational modelling.
• Your application needs both traditional SQL and vector capabilities.

Neptune

Amazon Neptune is a graph database supporting Resource Description Framework (RDF) graphs and Labelled Property graphs (LPGs). This makes it ideal for connected data use cases that are optimised for relationships rather than documents.

Neptune provides graph-based vector search, integrating relationship data with vectors and advanced query capabilities. However, it requires graph modelling, which needs more upfront design effort than the other choices. And if the necessary expertise is lacking, users might find Neptune’s learning curve quite steep.

Choose Neptune if…

• You run complex relationship-based queries against your data.
• You want a knowledge graph, not just semantic search.
• You need graph reasoning and vector similarity.
• Your RAG workflow depends on graph structure, not document similarity.

Summary

In this post, I explored how Amazon S3 Vectors works in practice, created an unstructured data Bedrock Knowledge Base using it, and reviewed its storage layout, performance and cost.

S3 Vectors provides a substantial cost benefit for the long-term storage of large embedding sets. It is especially effective for handling large volumes of vectors and can achieve retrieval times of 100 to 800 milliseconds.

However, if your workloads require faster lookups or higher throughput, a purpose-built vector database or OpenSearch would be a better choice. Ultimately, the ideal option depends on whether your priority is cost efficiency or query performance.

Finally, I will share a recent AWS video that explores many of the concepts mentioned here. It uses unstructured shareholder letters with S3 Vectors and Bedrock Knowledge Base:

Like this post? Click the button below for links to contact, socials, projects and sessions:

Thanks for reading ~~^~~