Search an S3 bucket from your agent

How it works

Each object under the prefix becomes a document. Its id is its key under the prefix (policies/travel.md), and its metadata has key, etag and size. Every few minutes we list the prefix, read what's new or changed (by ETag), and delete documents whose objects are gone. You don't run anything: no connector, no agent on your side.

• Any text format is read as it is: Markdown, HTML, JSON, CSV, logs, code.
• .gz objects are unzipped.
• Binary objects and objects over 4 MB (unzipped) are left out.

1. Connect the prefix

Make a collection, then tell it where your files are with PUT /api/collections/{name}/source:

curl -X PUT https://stackgrep.com/api/collections/handbook/source \
  -H "Authorization: Bearer $STACKGREP_KEY" \
  -d '{"bucket": "acme-docs", "prefix": "handbook/", "region": "us-east-1"}'

{"bucket": "acme-docs", "prefix": "handbook/", "region": "us-east-1",
 "status": "new", "objects": 0, "last_sync": "", "last_error": "",
 "verify_key": "handbook/.stackgrep-verify", "verify_token": "sg_verify_3f9c...",
 "role": "arn:aws:iam::...:role/stackgrep-app",
 "policy": "{ \"Version\": \"2012-10-17\", \"Statement\": [ ... ] }"}

2. Let us read it

Add the statement in policy to your bucket's policy. It lets our role list the prefix and read objects under it, and nothing else: we never write to your bucket.

3. Show it's yours

Write verify_token into the object named by verify_key. We check it on every sync, so nobody can point a collection at a bucket that isn't theirs.

echo -n "sg_verify_3f9c..." | aws s3 cp - s3://acme-docs/handbook/.stackgrep-verify

4. Sync

Syncs run every few minutes on their own. To start one now, POST /api/collections/{name}/source/sync; it answers 202 and works in the background. GET /api/collections/{name}/source shows how it went.

curl -X POST https://stackgrep.com/api/collections/handbook/source/sync \
  -H "Authorization: Bearer $STACKGREP_KEY"
# 202, {"status": "syncing", ...}

curl https://stackgrep.com/api/collections/handbook/source -H "Authorization: Bearer $STACKGREP_KEY"
# {"status": "ok", "objects": 1843, "last_sync": "2026-10-04T18:22:05Z", ...}

Disconnect with DELETE /api/collections/{name}/source: syncing stops, and the documents already in the collection stay. You can still send documents to a synced collection by API, and search it like any other.