Search Engines
Lamb helps search engines and other crawlers discover your content out of the box. It generates a sitemap.xml and a robots.txt automatically — there is nothing to configure and no files to drop in by hand.
Available endpoints
| Path | Purpose |
|---|---|
/sitemap.xml | Lists every public URL for crawlers |
/robots.txt | Allows crawling, points at the sitemap, hides admin paths |
Sitemap
/sitemap.xml is a standard sitemaps.org document. It lists the home page followed by every publicly visible post and page, newest first, each with a <lastmod> timestamp taken from when the post was last updated.
It contains exactly what an anonymous visitor can see, so it omits:
Unlike the timeline, the sitemap does include menu-item pages (such as an “About” page) — they are real public URLs worth indexing.
The sitemap is cached and supports conditional requests, so crawlers that revisit it only re-download it when your content has actually changed.
Large sites
sitemaps.org caps a single sitemap document at 50,000 URLs. Once your site passes that, /sitemap.xml automatically becomes a sitemap index instead of a plain sitemap: it lists child sitemaps at /sitemap.xml/page/1, /sitemap.xml/page/2, and so on, each holding up to 50,000 URLs. Crawlers that follow sitemap indexes (all major ones do) pick this up with no configuration change on your part.
robots.txt
/robots.txt allows crawling, advertises the sitemap, and asks crawlers not to waste time on the private routes — the login-gated admin pages and actions (/settings, /edit, /drafts, /trash, /scheduled, /delete, /restore, /upload, /checkbox) plus the internal /login, /logout, and /_cron endpoints. The list is derived automatically from the routes themselves, so it stays complete as the app grows. Those routes already require a login (or are internal), so this is a hint to crawlers rather than a security control.
It also disallows /*?preview=, the preview links that open an unpublished post without a login.
Pages that ask not to be indexed
robots.txt is only read by crawlers that look for it first, and it cannot cover a link someone else has published. So the pages that are not meant to be found say so on the page itself as well — every private page and every preview link is served with both a X-Robots-Tag: noindex, nofollow header and a matching <meta name="robots"> tag:
- the admin pages and actions listed above;
- any URL carrying a
?preview=token.
Ordinary posts and pages carry neither, so nothing public changes.
This matters most for preview links: they are meant to be shared with someone who is not logged in, which is exactly the way an unpublished post ends up in a search index. A preview link also expires after 24 hours.
Overriding robots.txt
If you want full control, drop your own robots.txt into the web root (the src/ directory, next to index.php). When that file exists Lamb serves it verbatim and skips the generated one, so your version always wins.
Submitting your site
Point a search engine’s webmaster tools (for example Google Search Console or Bing Webmaster Tools) at https://your-site/sitemap.xml.
Many of those tools can also ingest a feed directly, so as an alternative — or in addition — you can give them your Atom or JSON feed. The sitemap is the broadest signal (it lists every public page, not just recent posts), so it is the recommended starting point.
Related
- Feeds — Atom / JSON Feed, also accepted by many webmaster tools
- Menu Items — pages that the sitemap includes but the timeline hides
- Scheduling and Drafts — content the sitemap deliberately leaves out