2ndHowever, it's fair to assume that these publications recommend running a website scan at least as frequently as they run mini-audits on a monthly basis. What I Learned About SEO Crawl Cadence in 'Real Practice' Best practices are one thing. But how often do SEO professionals Banner Design perform website crawls for client sites in real-world practice? To get an idea, I took to Twitter. (Yes, Twitter polls have their obvious limitations — but it's one of the easiest ways to get hard data.) Three days, and nearly 2,000 votes later, the results are as follows: How often should you Banner Design perform technical website crawls for SEO? About 57% of
SEO professionals who took part in my Banner Design survey fell into the “monthly or longer” bucket, while 43% fell into the “weekly” or shorter bucket. Advertising Continue reading below In other words, we were all over the map. You may not be too surprised by the results of these polls. After all, both ends of the spectrum could make complete sense - depending on the type and size of websites you run. That said, I had an experience at FOX two months ago that made Banner Design me grateful that we were doing weekly website crawls in our core areas . I'd like to share it with all of you here - in case it inspires you to up the tempo of your website's technical explorations. How the Weekly Website Crawl Helped FOX Act Quickly on an allRoutes.json Bug In late July 2020, the FOX SEO team performed a routine weekly crawl of the FOX website and discovered that 100% of our TV episode pages Banner Design were unexpectedly showing error status codes (in due to a bug with the allRoutes.json file). Although the pages displayed well to users, they were throwing 404s at the Google bot, making them ineligible to appear in
Google search results. Not only is “watch time” a major KPI for the site, but these pages also generate significant ad revenue for the business. Needless to say, this Banner Design crawl led us into a serious problem. How often should you perform technical website crawls for SEO? Users were able to navigate and interact with /watch/pages just fine. However, the pages threw 404 codes at Google bots. Thanks to the crawl, we diagnosed the problem quickly. But the solution was Banner Design complex. Over the course of three weeks (July 23 to August 13), we focused on this specific issue and confirmed a sharp drop in clicks and SEO impressions from this set of pages.