Crawling and Indexing
As mentioned, search engines use robots to crawl your website. Which is why it is integral that your website is built in a way that allows these robots to navigate. A robust internal linking system is a great start.
When the robot arrives at your website, it will look for a robots.txt file. This file works hand in hand with your XML sitemap. When implemented incorrectly, this file can prevent search engines from crawling your pages, which can be disastrous for your rankings, and therefore views.
However, when implemented correctly, the robots.txt file can give search engines a helping hand to access your site to its full potential and prevent it from viewing any pages should you require this.
Once the search engine bots have crawled the site and analysed the content, they store the data about your site, known as indexing. Meaning, your content will be shown in the search results under relevant queries.
For your page to show on the first page of results, it needs excellent, on-page, off-page, and technical SEO.