Data Store Service data-store

Containerized file server and REST API for document set discovery, static AST JSON serving, and search index generation.

Service Overview

The data-store service operates as the core content repository backend. It scans the shared data volume (DATA_DIR) for published documentation sets, serves JSON AST files, and builds full-text search indices.

Key Capabilities

  • Documentation Discovery — Recursively scans DATA_DIR for subdirectories containing a toc.json manifest and exposes them via GET /api/docs.
  • Static AST Asset Serving — Serves compiled topic JSON files, image assets, and Table of Contents manifests under /data/....
  • Search Index Generation — Generates an in-memory/JSON MiniSearch index (search-index.json) for each document set upon server startup.

Container Deployment & Configuration

Deploy the service using Docker Compose or a standalone Docker container without requiring local package managers:

# Launch data-store using Docker Compose:
docker compose up -d data-store

# Or run as a standalone Docker container:
docker run -d -p 4000:4000 \
  -v /path/to/data:/app/data \
  ghcr.io/jason-fox/ast-data-store:latest

Configuration Environment Variables:

PORT
HTTP port for the Express REST server (default: 4000).
DATA_DIR
Directory path containing published documentation sets (default: /app/data).
DEFAULT_LANGUAGE
Fallback IETF BCP 47 language code for document set discovery (default: en).
MAX_UPLOAD_SIZE
Max body size accepted by the POST/PUT /api/docs/<id> zip uploads (default: 100mb).
AUTH_TOKEN
Optional bearer token guarding the /api/docs/<id> and /api/chrome write routes. Unset by default, which disables auth entirely.
RAG_SERVICE_URL
Optional rag-service URL. When set, doc set uploads and deletes trigger a background reindex there; failures are logged and never returned to the client.
CHROME_CACHE_TTL_MS
How long GET /api/chrome serves chrome.json from an in-memory cache before re-reading from disk (default: 5000). PUT /api/chrome invalidates the cache immediately on write.
FEATURED_DOCS
Comma-separated doc-set ids to pin first (before priority/alphabetical) in /api/docs ordering. Unset by default.
WEB_CONCURRENCY / WORKERS
Explicit worker process count. Takes precedence over CLUSTER_MODE. Unset by default.
CLUSTER_MODE
Set to true to run one worker per CPU core when WEB_CONCURRENCY/WORKERS aren't set (default: false).

REST Endpoints

  • GET /api/docs — Returns a JSON array of all discovered documentation sets with OASIS metadata.
  • GET /api/docs/<id> — Returns the summary for one doc set. 404 if <id> doesn't exist.
  • HEAD /api/docs/<id> — 200 if <id> exists, 404 otherwise. No body.
  • POST /api/docs/<id> (body: application/zip) — Extracts the zip into a new <id> folder. 201 on success, 409 if <id> already exists.
  • PUT /api/docs/<id> (body: application/zip) — Replaces the existing <id> folder's contents with the zip. 200 on success, 404 if <id> doesn't exist.
  • DELETE /api/docs/<id> — Removes the <id> folder. 204 on success, 404 if it doesn't exist.
  • GET /api/chrome — Returns the shared layout/nav data (chrome.json).
  • PUT /api/chrome (body: application/json) — Overwrites chrome.json with the request body. 200 on success.
  • GET /health — Health check endpoint returning server status and active data directory path.
  • GET /data/... — Static file mount for toc.json, topic files, and search indices.

<id> may contain / for nested doc-set groups (e.g. group/subdoc). Error responses from the write endpoints use RFC 9457 application/problem+json bodies (type, title, status, detail).

POST/PUT /api/docs/<id> bodies must be Content-Type: application/zip and a real zip, checked by its local-file-header magic bytes rather than trusting the header — 415 otherwise. PUT /api/chrome bodies must be Content-Type: application/json and parse as JSON — 415 otherwise. Bodies over MAX_UPLOAD_SIZE get 413 on either route.

Authorization

POST/PUT/DELETE /api/docs/<id> and PUT /api/chrome are guarded by an optional bearer token, set via AUTH_TOKEN. Auth is off by default so the application remains zero-config for local dev — if AUTH_TOKEN is unset, every request to these routes is allowed. If it's set, requests must send Authorization: Bearer <AUTH_TOKEN> or get 401 (with a WWW-Authenticate: Bearer header). GET/HEAD//data stay open either way — this only guards writes.

curl -X POST --data-binary @docset.zip \
  -H "Content-Type: application/zip" \
  -H "Authorization: Bearer $AUTH_TOKEN" \
  http://<data-store>/api/docs/my-doc-set

Where <data-store> is the host and port the service is running on, e.g. localhost:4000.

Publishing chrome.json

The ast-chrome transtype (see Generating Chrome) writes chrome.json to a local output directory. Publish it with PUT /api/chrome rather than copying the file directly — the request is validated the same way regardless of whether data-store is running on the same host or behind a network or container boundary, and nothing on disk changes unless the body passes validation.

dita --input=path/to/your.ditamap \
     --format=ast-chrome \
     --output=path/to/output

curl -X PUT --data-binary @path/to/output/chrome.json \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer $AUTH_TOKEN" \
     http://<data-store>/api/chrome

Omit the Authorization header entirely if AUTH_TOKEN is unset on the server. A successful request returns 200 with the new chrome.json content; see REST Endpoints above for the 415 (bad content type or invalid JSON), 413 (over MAX_UPLOAD_SIZE), and 401 failure cases.