ArcXiv for agents

The problem we kept hitting

When an AI agent needs a research paper, it usually does one of two things: it scrapes an HTML page built for human eyes, or it calls a search endpoint that returns titles without abstracts, let alone anything worth citing. Both paths waste the agent's context window on navigation chrome and leave it guessing about which results matter.

We run a corpus of about three million arXiv records, rank them with editorial quality assessments, and already serve them to a browser. Everything an agent needs was sitting behind an interface designed for scrolling. This release opens it up.

Search that returns evidence

The search endpoint takes free text, an arXiv ID, or an arXiv URL, and returns full abstracts with structured metadata:

curl "https://arcxiv.org/api/agent/search?q=sparse%20autoencoders&pageSize=3"

Every hit carries a qualitySignals block when ArcXiv has assessed the paper: a summary, sixteen metric scores covering rigor, novelty, reproducibility and related dimensions, and the paper's percentile within its category over 30 days. Hits also carry citation counts from OpenAlex and a stable set of links, so one response gives an agent everything it needs to decide what to read next.

Papers as records, then as markdown

The paper endpoint returns the complete record: version history, license, DOI, assessments, and six related papers. The markdown endpoint returns the same metadata rendered as a document an LLM can read directly:

curl "https://arcxiv.org/api/agent/papers/2401.12345"

curl "https://arcxiv.org/abs/2401.12345/markdown"

The markdown document today contains the title, abstract, authors, metadata, and canonical links: exactly what an agent needs to decide whether a paper is worth retrieving, without paying for a full read up front. The full-text converter is built and tested — it walks arXiv's HTML rendering once per paper version, keeps math as LaTeX, and caches the result permanently — and we will switch it on for a paid agent tier once the economics are settled. The API shape will not change when that happens.

How ranking actually works

Search combines four signals. Lexical full-text matching over Postgres with weighted fields, trigram title similarity for fuzzy and prefix matches, editorial assessment scores from our tournament system, and citation counts scaled logarithmically and capped so a heavily cited survey cannot bury fresh relevant work. Concept expansion helps ordinary vocabulary: searching "long context" also surfaces papers that only say "KV cache" or "RoPE".

The honest limitation: there are no embedding vectors yet, so a query that shares no vocabulary with the paper can miss. Lexical search with concept expansion covers most real research questions, and semantic retrieval is the next piece of this platform.

An MCP server with three tools

For agents that speak Model Context Protocol, we ship a small stdio server exposing search_papers, get_paper, and get_paper_markdown. Download the single-file Node bundle, save it somewhere stable, and point your MCP client at its absolute path:

{
  "mcpServers": {
    "arcxiv": {
      "command": "node",
      "args": ["/absolute/path/to/arcxiv-mcp.cjs"]
    }
  }
}

Download the ArcXiv MCP server. The server uses https://arcxiv.org by default; set ARCXIV_API_BASE to point it at another deployment.

Limits, stated plainly

• Read endpoints are public, cached at CloudFront, and limited to 300 requests per five minutes per IP. Contact c@coder.company before sustained bulk use.
• Full-text markdown is off by default and reserved for a paid tier; today every markdown document is metadata plus abstract. When enabled, it depends on arXiv's HTML renderings, which do not exist for every paper, and conversion of complex figures and tables is imperfect.
• A grounded question-answering endpoint exists at POST /api/agent/ask but returns 503 until we settle a quota policy for unauthenticated spend.

Start here

Fetch arcxiv.org/llms.txt for a machine-readable description of every endpoint. If you build something with this, tell us what the API got wrong; agent traffic is the sharpest tool we have for finding where the corpus and the interfaces around it still creak.

For parameters, response fields, research-signal definitions, caching, limits, and MCP setup, read the agent API documentation.