S3 Connections (Beta)

Connect S3 buckets to Doubleword and automatically sync files for batch processing. Instead of uploading files one at a time through the API or dashboard, point Doubleword at your storage and let it pull files in for you.

Once connected, you can:

• Browse the files in your bucket directly from the dashboard
• Sync files to ingest them as batches for processing
• Track sync progress and inspect any errors

Connections are scoped to the user or organization that creates them. Credentials are encrypted at rest and are never returned via the API.

Prerequisites

Your S3 bucket should contain .jsonl files in the standard batch request format — one JSON object per line with custom_id, method, url, and body fields.

Creating a connection

1. Click Connections in the sidebar
2. Click New Connection
3. Enter a name and your S3 bucket details:
   • Bucket — the S3 bucket name
   • Region — the AWS region (e.g. eu-west-2)
   • Access Key ID and Secret Access Key — IAM credentials with read access to the bucket
   • Prefix (optional) — scope the connection to a specific folder (e.g. inputs/batch/)
4. Click Test Connection to verify your credentials
5. Click Create

Syncing files

There are two ways to sync files from a connection:

Snapshot sync — Click the Sync button on a connection to discover and ingest all new files. Files that have already been synced are automatically skipped. If a file has been modified in the source since it was last synced, it will be re-ingested.

Selective sync — Expand the Source Files section to browse your bucket. Select individual files and click Sync to ingest only those. This is useful when you want to process specific files rather than everything.

You can also force a re-sync of previously synced files by selecting them and choosing Re-sync. This bypasses deduplication and creates new batches regardless of whether the files have changed.

Each synced file becomes a batch. Batches are accepted based on available capacity and are processed at the standard batch priority tier.

Large syncs

The overall time to fully process a sync depends on the number and size of files. A sync with thousands of files may run for longer than 24 hours as batches are accepted progressively based on capacity. Ingestion continues in the background for as long as needed — you don't need to keep the page open.

Once an individual batch is accepted, it follows the standard batch processing SLA.

File validation

Doubleword validates each line of your JSONL files during ingestion:

• Non-JSON lines are skipped entirely and don't create requests. The count of skipped lines is shown on the sync entry.
• Parseable but invalid lines (e.g. missing model field, invalid HTTP method) are flagged as validation errors. You can see exactly which lines failed and why from the sync entry detail view.
• Valid lines are ingested normally and processed as pending requests.

The sync entry detail view shows a breakdown of skipped lines, validation errors with line numbers, and successful requests.

Monitoring syncs

Expand Sync History on a connection to see all sync operations. Each sync shows:

• Status — running, completed, or failed
• Progress — files found, ingested, skipped, and failed
• Entries — expand to see per-file status, including template counts, skipped lines, validation errors, and a link to the resulting batch

Syncs in progress auto-refresh until complete.