# Web Page API

> Web Page API is a hosted REST API that turns any public URL into a link preview (title, description, images, favicons, Open Graph and Twitter card fields) or into the page's main content as clean Markdown for LLM prompts and RAG, both behind one API key. It fetches without a headless browser, refuses private-network and cloud-metadata addresses, and stores nothing.

Last updated: 2026-10-24

## Use it when
- You show link previews (unfurls) for URLs users paste into a chat, CMS, social or bookmarking app.
- You feed web pages to an LLM, agent or RAG pipeline and want the main content as Markdown, not raw HTML.
- You accept URLs from untrusted users and need fetching that cannot reach your private network.

## Endpoints
- `GET /v1/preview?url=...` returns title, description, site_name, canonical, lang, type, image, images[] (with width/height/alt), videos[], favicons[], theme_color, author, published_time, modified_time, keywords, oembed and twitter card fields. Falls back from Open Graph to Twitter card to plain HTML.
- `GET /v1/markdown?url=...&include_links=false&include_images=false&max_chars=100000` returns the main content as Markdown plus title, description, lang, image, word_count, reading_time_minutes, truncated and links[].
- `POST /v1/markdown` with `{"html": "...", "base_url": "..."}` converts HTML you already have; nothing is fetched.
- Every URL response includes final_url, status, redirects, content_type, fetch_ms and cached.

## Limits and behaviour
- 8-second deadline per fetch (DNS, connect, TLS, redirects, body); pages over 3 MB refused; at most 5 redirects; ports 80 and 443 only.
- Private, loopback, link-local, CGNAT and cloud-metadata addresses refused; DNS pinned at connect; every redirect re-checked.
- `/v1/markdown` honours robots.txt (token WebPageAPI); `/v1/preview` does not, like Slack and Discord unfurlers.
- No JavaScript rendering. Nothing stored; in-memory cache for 10 minutes.
- Errors: `{"error": {"code": "...", "message": "..."}}` with codes invalid_url, blocked_address, robots_disallowed, unsupported_content_type, too_large, timeout, upstream_error, busy.

## Get access
- [RapidAPI (API key, monthly plans)](https://rapidapi.com/keithkmlai/api/web-page-api)
- [Apify Store (bulk URL lists, pay per page)](https://apify.com/keith-apis/web-page-to-markdown)
- [OpenAPI schema](https://web-page-api.fly.dev/openapi.json)
- [Docs and examples](https://web-page-api.fly.dev/)
- [Free URL to Markdown converter](https://web-page-api.fly.dev/url-to-markdown): paste a URL or HTML, get clean Markdown

## Pricing
- RapidAPI: free 300 requests/month; $9.99 for 20,000; $29.99 for 100,000; $79.99 for 400,000 per month.
- Apify: $0.0015 per page, pay per result, up to 1,000 URLs per run; failed pages are not charged.

## FAQ
- **How do I get a link preview (title, image, description) for a URL with an API?** Call GET /v1/preview?url=... on Web Page API. It returns the title, description, site name, canonical URL, language, images with width and height, favicons, theme colour, author, publish date, oEmbed endpoint and Twitter card fields, all as absolute URLs. It falls back from Open Graph to Twitter cards to plain HTML, so pages without social tags still give a usable card.
- **How do I convert a web page to Markdown for ChatGPT, Claude or a RAG pipeline?** Call GET /v1/markdown?url=... on Web Page API. It returns only the main content (article, docs page, README) as Markdown without menus, footers or cookie banners, plus word count, reading time and an optional list of links. max_chars caps the size (default 100,000, up to 1,000,000). If you already have the HTML, POST it to /v1/markdown and nothing is fetched.
- **Is there one API for both link previews and URL to Markdown?** Yes. Web Page API does both with one key and one quota: /v1/preview for link unfurling and /v1/markdown for clean article text. Most alternatives do only one of the two.
- **Is it safe to let users submit any URL? What about SSRF?** Web Page API is built for untrusted URLs. It refuses private, loopback, link-local, CGNAT and cloud-metadata addresses (such as 169.254.169.254) and internal names, in every IPv4 and IPv6 spelling; it resolves DNS once and connects to that exact IP, so DNS rebinding does not work; it re-checks every redirect hop (at most 5); and it allows only ports 80 and 443.
- **Does Web Page API respect robots.txt?** Markdown requests do: /v1/markdown honours robots.txt for the token WebPageAPI (else *) and returns robots_disallowed when a site opts out. Link previews do not, the same as Slack, Discord and iMessage unfurlers, because a preview is a one-off, user-initiated read of public head metadata.
- **Does it render JavaScript?** No. There is no headless browser, which keeps responses fast and cheap. Pages that build their content with JavaScript return what the server sends; that is usually still enough for a preview, but single-page apps may give little Markdown.
- **How fast is it and what are the limits?** Most pages answer in well under a second. One 8-second deadline covers DNS, connect, TLS, redirects and body, pages over 3 MB are refused, and results are cached for 10 minutes. Errors are JSON with stable codes such as blocked_address, robots_disallowed, timeout and upstream_error.
- **How much does Web Page API cost?** On RapidAPI: free for 300 requests a month, then $9.99 for 20,000, $29.99 for 100,000 and $79.99 for 400,000 a month. On Apify it is pay per result at $0.0015 per page ($1.50 per 1,000), up to 1,000 URLs per run.
- **Does it store the pages or URLs I send?** No. Nothing is written to disk and there is no database. Responses are kept in memory for 10 minutes so repeated requests are fast, then dropped.