Observability v7

See detailed status and progress output for all auto-processing modes.

Two views cover most observability needs:

• aidb.pipeline_metrics (alias aidb.pipem) — one row per pipeline with backlog, source/destination counts, status, and error counts. This is the primary view for tracking processing progress.
• aidb.knowledge_base_metrics (alias aidb.kbm) — one row per knowledge base with pipeline count, embedding count, and aggregated status across all attached pipelines.

Per-pipeline progress

SELECT * FROM aidb.pipem;

      pipeline       | auto processing | table: unprocessed rows | volume: scans completed | count(source records) | count(destination records) |  Status  | count(record errors) | count(blocking errors)
---------------------+-----------------+-------------------------+-------------------------+-----------------------+----------------------------+----------+----------------------+------------------------
 feedback_na_pipe    | Live            |                       0 |                         |                   412 |                        412 | UpToDate |                    0 |                      0
 product_docs_pipe   | Background      |                         |                       6 |                   1842 |                       1842 | UpToDate |                    0 |                      0
(2 rows)

The change detection mechanism is different for volume and table sources, so the view exposes these metrics:

• table: unprocessed rows: How many unique rows are listed in the backlog of change events. Auto-processing captures no new change events when Disabled.
• volume: scans completed: How many full listings of the source have been completed.
• count(source records): How many records exist in the source. Accurate for table sources at all times; updated after each full scan for volume sources.
• count(destination records): How many rows the pipeline has written to its destination (embeddings, chunks, and so on).

Knowledge base overview

SELECT * FROM aidb.kbm;

             name             | pipelines | embeddings |  status
------------------------------+-----------+------------+----------
 public.product_catalog_kb    |         1 |       1842 | UpToDate
 public.customer_feedback_kb  |         2 |        915 | UpToDate
(2 rows)

The status column rolls up the worst pipeline status across all pipelines attached to that KB — drill into aidb.pipeline_metrics to find which pipeline is responsible.