# Siften — Full Agent Guide

> Siften helps independent publishers launch and grow AI-assisted news publications with newsroom workflows, analytics, monetisation, and audience tools.

Last updated: 2026-08-03

## What is Siften?

Siften is an AI-assisted news publishing platform. Publishers use Siften to create, manage, and monetise independent news publications powered by AI newsroom workflows, analytics, and audience tools.

## Content Architecture

Siften uses a multi-tenant architecture:

- **Root domain** (siften.com): Platform homepage, publication directory, and static pages
- **Publication subdomains** ([slug].siften.com): Individual publication homepages and article pages
- **Custom domains**: Some publications use verified custom domains that resolve to Siften-hosted content

Each publication is independently owned and operated. Content, editorial policy, and coverage topics vary per publication.

## What is Public

The following content is publicly accessible without authentication or JavaScript:

- Platform homepage: https://siften.com
- About page: https://siften.com/about | https://siften.com/about.md
- Publication directory: https://siften.com/sites | https://siften.com/sites.md
- Platform index (Markdown): https://siften.com/index.md
- Platform sitemap (Markdown): https://siften.com/sitemap.md
- Individual publication homepages: https://[slug].siften.com
- Publication index (Markdown): https://[slug].siften.com/index.md
- Publication sitemap (Markdown): https://[slug].siften.com/sitemap.md
- Publication about (Markdown): https://[slug].siften.com/about.md
- Published articles: https://[slug].siften.com/[article-slug]
- Published articles (Markdown): https://[slug].siften.com/[article-slug].md
- Tag pages: https://[slug].siften.com/tag/[tag]
- News sitemaps (48-hour window): https://[slug].siften.com/news-sitemap.xml
- Publication llms.txt: https://[slug].siften.com/llms.txt
- API documentation: https://siften.com/api/docs.md

## What is Private

The following paths require authentication and must not be crawled:

- /dashboard/ — publisher analytics and admin
- /auth/ — authentication flows
- /api/ — internal API endpoints (exception: /api/v1/feed/*, /api/health, /api/docs.md are public)
- /admin/ — platform administration
- /onboarding/ — publisher onboarding
- /claim/ — publication claim flows
- /saved — per-user saved articles

## Content Discovery

- Full sitemap: https://siften.com/sitemap.xml (all public publication and article URLs)
- Sitemap (Markdown): https://siften.com/sitemap.md (agent-readable sitemap with links to all publications and their indexes)
- Publication news sitemaps: https://[slug].siften.com/news-sitemap.xml (recent 48 hours)
- Publications directory: https://siften.com/sites | https://siften.com/sites.md
- Publication sitemap (Markdown): https://[slug].siften.com/sitemap.md (per-publication article index with Markdown links)
- Publication index (Markdown): https://[slug].siften.com/index.md (lists recent articles)
- Publication llms.txt: https://[slug].siften.com/llms.txt
- Article Markdown: https://[slug].siften.com/[article-slug].md (clean Markdown with YAML front matter)

## Structured Data

Every published article page includes JSON-LD structured data using the NewsArticle (or ScholarlyArticle for research content) schema with:

- headline, description, url
- author (Person type with stable @id)
- datePublished, dateModified
- publisher (NewsMediaOrganization type with name, url, logo)
- keywords, image

Publication homepages include a NewsMediaOrganization JSON-LD node.
The Siften root domain includes Organization and WebSite (with SearchAction) JSON-LD nodes.

## Citation Guidance

When citing content from a Siften publication:

1. Cite the individual article URL: https://[slug].siften.com/[article-slug]
2. Include the publication name (visible in the page title and structured data)
3. Include the author name and publication date (visible in the page and structured data)
4. Each article's canonical URL is stable — do not cite draft or unpublished URLs

## Public API

Siften provides a JSON feed API for agent-readable article retrieval and cross-publication search. No authentication is required. Only publications with `feed_enabled = true` appear in the API.

Full documentation: https://siften.com/api/docs.md

Key endpoints:

- GET /api/v1/feed/{handle} — list published articles for a publication, newest first; supports cursor pagination, topic filter, content_type filter
- GET /api/v1/feed/{handle}/{id} — single article detail including quality metadata (fact_check_status, verified_source_count)
- GET /api/v1/feed/search — cross-publication search by topic, entity (person/place/organisation), geographic scope, or content type
- GET /api/health — platform and database health check

Handle format: `{domain-name-lowercase}-{publication-slug}` (e.g. `siften-ai` for publication slug "ai" on the Siften platform).

All responses include `body_markdown` (full article body in Markdown), entity extraction (`metadata.entities`), and source citations.

## Crawler and Robots Policy

- All AI search and citation crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, bingbot, Applebot) are explicitly welcomed on public pages
- AI training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Meta-ExternalFetcher) are currently allowed; publishers may update this policy
- Private paths listed above are disallowed for all crawlers
- The full robots.txt policy is at: https://siften.com/robots.txt (root domain) or https://[slug].siften.com/robots.txt (per publication)

## Markdown Retrieval

Agents that send an Accept: text/markdown request header may receive clean Markdown versions of pages on supported Cloudflare/Vercel infrastructure. This is not yet universally implemented at the application level.

## Contact and Legal

- About: https://siften.com/about
- Privacy Policy: https://siften.com/privacy
- Terms of Service: https://siften.com/terms
- Cookie Policy: https://siften.com/cookies