A crawler starts from known URLs and sitemaps, requests each page, reads the HTML, and follows the links it finds to discover more pages. What it can reach and render determines what search engines know about your site. Server responses matter here: a healthy page returns an HTTP 200, while errors or blocks stop the crawler from seeing content.
You influence crawling with robots.txt (which paths bots may request), internal linking (how easily pages are discovered), and site speed. On large sites this ties directly into crawl budget — the finite attention Googlebot gives your domain.
Crawling is not the same as indexing. A page can be crawled but not indexed if it's thin, duplicated or blocked by a noindex tag. Getting crawling right is the foundation that everything else in technical SEO builds on.
