Configuring external storage v7

AIDB pipelines can read from two types of data source:

• Postgres tables — reference a table directly by name using the source parameter in aidb.create_pipeline(). See Creating pipelines.
• External storage volumes — connect S3-compatible object stores, Google Cloud Storage, Azure, or local file systems via PGFS. The rest of this page covers how to set this up.

External storage is accessed through the Postgres File System (PGFS) extension, which maps external storage into Postgres as storage locations. AIDB then wraps each storage location in a volume that pipelines reference by name. A volume is a foreign table, so you can list, read, write, and delete objects with standard SQL.

How it works

Connecting external storage to an AIDB pipeline involves two objects:

• PGFS storage location — Defines the external storage provider: its URI, credentials, and connection options. Created with pgfs.create_storage_location().
• AIDB volume — A foreign table that exposes the contents of a PGFS storage location as rows. Stores the data format (Text, Image, Pdf) and an optional sub-path. Created with aidb.create_volume().

Once a volume exists, reference it as the source in aidb.create_pipeline() exactly as you would a Postgres table, or query it directly with SQL.

Step 1: Creating a storage location

Use pgfs.create_storage_location() to define the connection to external storage. The uri parameter identifies the storage backend and path; the options JSONB object carries provider-specific settings such as region and credentials.

S3-compatible object store

-- Private S3 bucket with credentials
SELECT pgfs.create_storage_location(
    name    => 'my_s3_location',
    uri     => 's3://my-bucket/my-folder',
    options => '{"region": "us-east-1", "access_key_id": "<key>", "secret_access_key": "<secret>"}'
);

-- Public S3 bucket (no credentials required)
SELECT pgfs.create_storage_location(
    name    => 'my_public_bucket',
    uri     => 's3://aidb-rag-app',
    options => '{"region": "eu-central-1", "skip_signature": "true"}'
);

Local file system

For local file system access, declare the allowed base paths in postgresql.conf before creating the storage location. PGFS restricts access to these paths for security.

# postgresql.conf
pgfs.allowed_local_fs_paths = '/tmp/pgfs'

After restarting Postgres, create the storage location using a file:// URI:

SELECT pgfs.create_storage_location(
    name => 'local_tmp_pgfs',
    uri  => 'file:///tmp/pgfs/'
);

For full details on storage location options for S3, GCS, and Azure, see the PGFS documentation.

Step 2: Creating a volume

Use aidb.create_volume() to create a foreign table that exposes a PGFS storage location to AIDB. The volume name is the foreign-table name that you reference in pipelines and SQL queries.

SELECT aidb.create_volume(
    name        => 'my_volume',
    server_name => 'my_s3_location',
    path        => '/',
    data_format => 'Text'
);

Parameters:

Path handling

The path argument is canonicalized at creation time: any leading and trailing slashes are stripped, then exactly one trailing slash is appended. This means the following all store as foo/:

• foo
• /foo
• foo/
• /foo/

Passing NULL or / stores as /, which targets the storage location's root. The canonical value is what you see in aidb.volumes and what's used to compute object keys.

Example: PDFs in S3

SELECT pgfs.create_storage_location(
    'pdf_bucket',
    's3://my-docs-bucket',
    options => '{"region": "us-east-1", "access_key_id": "<key>", "secret_access_key": "<secret>"}'
);

SELECT aidb.create_volume('pdf_volume', 'pdf_bucket', '/', 'Pdf');

Example: Images in a local directory

SELECT pgfs.create_storage_location('local_tmp_pgfs', 'file:///tmp/pgfs/');

SELECT aidb.create_volume('ocr_input_volume', 'local_tmp_pgfs', 'ocr_input/', 'Image');

Step 3: Using the volume as a pipeline source

Set the source parameter in aidb.create_pipeline() to the volume name. Volume sources work the same as table sources from the pipeline's perspective:

SELECT aidb.create_pipeline(
    name       => 'my_pdf_pipeline',
    source     => 'pdf_volume',
    step_1     => 'ParsePdf',
    step_2     => 'ChunkText',
    step_3     => 'KnowledgeBase',
    step_3_options => aidb.knowledge_base_config('bert_local', 'Text')
);

See Create a pipeline for a full walkthrough.

Managing volumes

Listing volumes

List all volumes via the aidb.volumes view:

SELECT * FROM aidb.volumes;

 schema |    volume    | storage_location |    path
--------+--------------+------------------+---------------
 public | pdf_volume   | pdf_bucket       | /
 public | text_bucket  | text_bucket      | aidb-rag-app/
(2 rows)

The view returns four columns: schema, volume, storage_location, and path (canonicalized).

Deleting a volume

SELECT aidb.delete_volume('my_volume');

Accessing volume contents with SQL

A volume is a foreign table backed by PGFS, so the storage location's objects are exposed as rows. The columns are:

Listing objects in a volume

SELECT key, size, last_modified
FROM my_volume
ORDER BY key;

       key              | size |       last_modified
------------------------+------+------------------------
 Test 1.html            | 1093 | 2026-05-14 15:49:32+00
 Test 2.html            | 1563 | 2026-05-14 15:50:02+00
 Test 3.html.           | 1395 | 2026-05-14 15:49:32+00
(3 rows)

LIMIT is pushed down to the storage backend, so it's safe to peek at large buckets:

SELECT key, size FROM my_volume LIMIT 5;

Reading a single object

Filter on the exact key to fetch the object body:

-- Read a binary file as BYTEA
SELECT body FROM my_volume WHERE key = 'report.pdf';

-- Read a text file as text
SELECT convert_from(body, 'utf8')
FROM my_volume
WHERE key = 'notes.txt';

Writing, overwriting, and deleting objects

Volumes support standard INSERT, UPDATE, and DELETE. body is BYTEA; cast text to bytea as needed.

-- Upload a new object
INSERT INTO my_volume (key, body)
VALUES ('hello.txt', 'Hello, world'::bytea);

-- Overwrite an existing object
UPDATE my_volume
SET body = 'Hello again'::bytea
WHERE key = 'hello.txt';

-- Delete an object
DELETE FROM my_volume WHERE key = 'hello.txt';

Renaming a key (UPDATE my_volume SET key = ...) is not supported.