Files Service

Database-backed file storage with typed metadata, upload validation policies, identity-linked provenance, and stable /files serving for AI workloads.

Last updated:

On this page

TL;DR -- systemprompt.io ships a file storage service that works out of the box on local disk. Every file gets a Postgres row with typed metadata, checksums, and identity links (user, session, trace, context). Uploads pass through a configurable validation policy, and public files are served at stable /files/ paths suitable for caching and CDN distribution.

What It Does and Why It Matters

The files service handles upload, storage, retrieval, and lifecycle for every file in a systemprompt.io deployment. Static assets, user uploads, AI-generated images, and internal platform files all flow through the same interface.

This matters because AI workloads produce and consume files constantly. Agents generate reports, store artifacts, and retrieve documents across sessions. Without a unified file layer, each of these operations would require separate plumbing for storage, metadata, and serving. The files service consolidates all of that into a single API surface backed by the database, so every file is queryable, auditable, and linked to the identity that created it.

Key capabilities:

Database-backed metadata -- every file gets a Postgres row with path, MIME type, size, checksums, and typed metadata.
Identity-linked provenance -- files carry optional user_id, session_id, trace_id, and context_id, tying artifacts back to the governance spine.
Upload validation policy -- per-category allow lists (images, documents, audio, video) and a size limit enforced before bytes land on disk.
AI-content tracking -- AI-generated files are flagged and queryable separately (core files ai).
Stable serving -- public files are served under a configurable URL prefix (default /files), cacheable and CDN-ready.

Architecture

The file storage system separates concerns across three parts of the systemprompt-files crate.

API Request / CLI Command / MCP Tool
         │
         ▼
┌─────────────────────────────┐
│   Services                  │  Upload validation, file categories,
│   (FileUploadService,       │  AI-persistence glue
│    FileValidator)           │
└────────┬────────────────────┘
         │
         ▼
┌─────────────────────────────┐
│   Repository                │  sqlx-backed persistence: file rows,
│   (FileRepository)          │  content↔file associations, stats
└────────┬────────────────────┘
         │
         ▼
┌─────────────────────────────┐
│   Storage root              │  Local filesystem under the profile's
│   (paths.storage)           │  paths.storage directory
└─────────────────────────────┘

Service layer

FileUploadService and FileValidator handle validation, content-type detection, and checksum calculation, then persist metadata through the repository. FilesAiPersistenceProvider is the glue that stores AI-generated files (such as generated images) through the same path.

Repository layer

FileRepository owns all database access. Every file gets a row with a unique ID, path, public URL, MIME type, size, integrity checksums, and typed metadata. Deletes are soft (deleted_at), so file history remains auditable.

Storage root

Bytes live on the local filesystem under the profile's paths.storage directory. The service ensures a standard structure (a files/ directory for uploads and an images/ directory for image assets) at startup, and a background FileIngestionJob scans the storage root to reconcile on-disk image files with database rows, so files placed on disk outside the API still get metadata.

Storage Layout

The storage root comes from the profile:

# .systemprompt/profiles/local/profile.yaml
paths:
  storage: /absolute/path/to/storage

The path must be absolute. Under it, the service maintains:

storage/
├── files/           # Uploaded files
│   └── images/      # Image assets (scanned by FileIngestionJob)
└── ...              # Other platform storage (css, js, templates)

In the template repository this is storage/files/, which also holds the CSS and static assets published to web/dist/ by just publish.

Upload Policy Configuration

Upload behavior is configured in an optional services/config/files.yaml. If the file is absent, defaults apply.

# services/config/files.yaml
files:
  urlPrefix: "/files"          # URL prefix for public file serving
  upload:
    enabled: true              # Master switch for uploads
    maxFileSizeBytes: 52428800 # Per-file size limit (default 50 MB)
    persistenceMode: context_scoped
    allowedTypes:
      images: true
      documents: true
      audio: true
      video: false             # Disabled by default

Persistence modes control how uploaded files are scoped:

Mode	Effect
`context_scoped`	Files belong to the conversation context that uploaded them (default)
`user_library`	Files persist in the uploading user's library across contexts
`disabled`	Uploads are rejected

Validation runs before storage: files in a disallowed category or over the size limit are rejected with a typed FileValidationError.

File Metadata

Every file row records:

Property	Description
`id`	Unique file identifier (UUID)
`path`	File path within the storage root
`public_url`	Serving URL under the configured prefix
`mime_type`	Detected or specified MIME type
`size_bytes`	File size in bytes
`ai_content`	Whether the file was AI-generated
`metadata`	Typed metadata: checksums plus image, audio, video, or document details
`user_id`	User associated with the file (optional)
`session_id`	Session that produced the file (optional)
`trace_id`	Trace linking the file to a governed tool call (optional)
`context_id`	Conversation context the file belongs to (optional)
`created_at` / `updated_at`	Timestamps
`deleted_at`	Soft-delete marker

The typed metadata payload includes FileChecksums for integrity verification and type-specific structures (ImageMetadata, AudioMetadata, VideoMetadata, DocumentMetadata). AI-generated images additionally carry ImageGenerationInfo recording the provider that produced them.

Uploading Files

Files are uploaded through the CLI, the API, or MCP tools. Each upload validates against the policy, stores bytes under the storage root, and creates a database record.

# Upload a file into a conversation context (context is required)
systemprompt core files upload /path/to/file.pdf --context <context_id>

# Attribute the upload to a user and session
systemprompt core files upload /path/to/image.png \
  --context <context_id> \
  --user <user_id> \
  --session <session_id>

# Mark a file as AI-generated content
systemprompt core files upload /path/to/generated.png \
  --context <context_id> --ai

# Check a file against the upload policy without uploading
systemprompt core files validate /path/to/file.pdf

Serving Files

Public files are served at stable, predictable URLs under the configured urlPrefix (default /files). The URL maps directly to the storage layout, which makes it straightforward to place a CDN or reverse proxy in front of the endpoint and cache responses with long TTLs. Static assets served this way carry ETags and cache headers appropriate to their type.

File Operations for AI Agents

Agents interact with files through MCP tools and the AI persistence provider. Generated images and artifacts are stored through FilesAiPersistenceProvider, flagged as ai_content, and linked to the originating user, session, trace, and context. That means every AI-generated file is traceable through the same audit spine as the tool call that produced it:

# List AI-generated images
systemprompt core files ai list

# Inspect one, including generation provider info
systemprompt core files ai show <id>

# Count AI-generated images
systemprompt core files ai count

Storage Statistics

# Show file storage statistics
systemprompt core files stats

# Show the active upload configuration
systemprompt core files config

Configuration Reference

Item	Location	Description
Storage root	Profile `paths.storage`	Absolute directory for file storage
URL prefix	`services/config/files.yaml` `files.urlPrefix`	Public serving prefix (default `/files`)
Uploads enabled	`files.upload.enabled`	Master switch for uploads
Max file size	`files.upload.maxFileSizeBytes`	Per-file upload size limit (default 50 MB)
Persistence mode	`files.upload.persistenceMode`	`context_scoped`, `user_library`, or `disabled`
Allowed types	`files.upload.allowedTypes`	Per-category booleans: images, documents, audio, video

CLI Reference

Command	Description
`systemprompt core files list`	List files with pagination and filtering (`--user`, `--mime`)
`systemprompt core files show <identifier>`	Show detailed file information by ID or path
`systemprompt core files upload <path> --context <id>`	Upload a file from the local filesystem
`systemprompt core files delete <id>`	Soft-delete a file by ID (`--dry-run` to preview)
`systemprompt core files validate <path>`	Validate a file against the upload policy
`systemprompt core files config`	Show file upload configuration
`systemprompt core files search <query>`	Search files by path pattern
`systemprompt core files stats`	Show file storage statistics
`systemprompt core files ai list\|show\|count`	AI-generated image operations

Run systemprompt core files <command> --help for detailed options on any command.

Paths Configuration -- configure the storage path where files are kept.
Content Service -- content management that stores rendered output as files.
Security Configuration -- authentication and authorization settings.
MCP Service -- MCP tools that agents use to interact with files.