Who should attend this course?
This course is for anyone wanting to know how search engines crawl and index pages on a site. It's designed for anyone who wants to know more technical aspects of site architecture and crawling and how that impacts overall performance. Beginner to intermediate SEOs wanting to expand their technical SEO skillset should find this course enlightening.
Course content
We'll cover:
- Introduction on crawling, what it is and how it works
- Overview of indexation, proper techniques and common issues
- How search engines crawl sites and index pages
- Setting up crawling with local and web based crawlers
- Benefits of segmentation
- Crawling speed and server resources
- Setting up comparison crawls to track changes
- Differences between text-based crawling and JS Crawling, and when to use them
- How to analyze crawls and present findings to stakeholders (internal or external)
- Common site errors and issues that can lead to indexation issues
- Integrating APIs and external data to your crawl reports to find new solutions
- Understanding depth and internal linking between pages to improve crawling
- How to find uncrawled pages through GSC indexation reports and internal databases
- Case studies on internal linking and meta robots/robots.txt implementations
- Understanding log files and how to review those reports
- Indexation techniques
- For large sites: how to choose what you index and link to
- Links to non indexable pages
- Robots.txt vs meta robots
- Canonicals and hreflang and how they impact indexation
What to expect
A detailed course covering how sites are crawled and indexed by search engines, along with finding and fixing common indexation problems.
The course will cover crawling techniques using a variety of tools and software from local crawlers to GSC and web based crawling solutions. We will go over how to set up crawls and analyze results for medium and large websites, including JS crawling, segmentation, API connections, and more.
We will also go over log file analysis to discover how search engines continuously crawl your site and spotting errors or potential issues with indexation as a result of site structure and internal linking.
At the end of this course, you should have a strong understanding of how your site is indexed, how to report on issues to stakeholders, what and when to index, and have creative solutions to creating a site that is crawled and indexed as intended.
