Managing crawling and indexing

Managing crawling & indexing

Get to know the more technical aspects of site architecture and crawling and how that impacts overall performance.

Michael Louderback

Michael Louderback

Director of SEO at PACIFIC

Courses: 01

Michael is a Director of SEO at a performance marketing agency located in San Diego, CA. He has over 10 years’ experience in SEO, working on small local business sites all the way up to large multinational sites for Fortune 500 companies. Michael has worked on large technical SEO projects spanning reviews of over 100 million pages, finding and solving large indexation and internal linking issues. Outside of work, you can find Michael working on random side projects, jumping on a quick match on the latest FPS, hitting up the gym, or out finding a new place to eat or drink.

Who should attend this course?

This course is for anyone wanting to know how search engines crawl and index pages on a site. It's designed for anyone who wants to know more technical aspects of site architecture and crawling and how that impacts overall performance. Beginner to intermediate SEOs wanting to expand their technical SEO skillset should find this course enlightening.


brightonSEO training workshop attendees

Course content

We'll cover:

  • Introduction on crawling, what it is and how it works
  • Overview of indexation, proper techniques and common issues
  • How search engines crawl sites and index pages
  • Setting up crawling with local and web based crawlers
    • Benefits of segmentation
    • Crawling speed and server resources
    • Setting up comparison crawls to track changes
  • Differences between text-based crawling and JS Crawling, and when to use them
  • How to analyze crawls and present findings to stakeholders (internal or external)
  • Common site errors and issues that can lead to indexation issues
  • Integrating APIs and external data to your crawl reports to find new solutions
  • Understanding depth and internal linking between pages to improve crawling
  • How to find uncrawled pages through GSC indexation reports and internal databases
  • Case studies on internal linking and meta robots/robots.txt implementations
  • Understanding log files and how to review those reports
  • Indexation techniques
    • For large sites: how to choose what you index and link to
    • Links to non indexable pages
    • Robots.txt vs meta robots
    • Canonicals and hreflang and how they impact indexation

brightonSEO trainer leading a workshop

What to expect

A detailed course covering how sites are crawled and indexed by search engines, along with finding and fixing common indexation problems.

The course will cover crawling techniques using a variety of tools and software from local crawlers to GSC and web based crawling solutions. We will go over how to set up crawls and analyze results for medium and large websites, including JS crawling, segmentation, API connections, and more.

We will also go over log file analysis to discover how search engines continuously crawl your site and spotting errors or potential issues with indexation as a result of site structure and internal linking.

At the end of this course, you should have a strong understanding of how your site is indexed, how to report on issues to stakeholders, what and when to index, and have creative solutions to creating a site that is crawled and indexed as intended.

Share this course:

Available Tickets

San Diego

Mon 18 Nov, 2024 | 08:30 AM - 04:15 PM

$1390$1725Prices rise November 6
Book now
100% Satisfaction Guaranteed

100% satisfaction guaranteed

We do all we possibly can to make our courses the best you’ve attended but if you ever feel we didn’t live up to your expectations, we’ll give you a total refund, no questions asked.