A dynamically inserted content spider trap is when you visit a URL that should 404 but returns a 200 OK status code and is similar to the keyword search spider trap

In General,

1. Never-Ending URL: infinite different URLs that point to the same page with duplicated content.

2. Mix and Match Trap: the same information presented in endless different ways (e.g. millions of different ways to sort and filter a list of 1000’s of products).

3. Calendar Trap: pages that are technically unique, but provide no useful information (e.g. an event calendar that goes thousands of years into the future).

4. Session ID Trap: near-duplicates with pages that differ by some infinite detail.


References:-
https://www.advancedwebranking.com/blog/avoid-the-seo-spider-trap-how-to-get-out-of-a-sticky-situation/

There are a number of common techniques that are frequently used to make the spambots run in an infinite loop. To name a few of them :

1.      Sometimes, a cyclic directory structure is used. For example : /path/to/directory/again/path/to/directory. As a result, if a spambot starts crawling the website, it will start running in an infinite loop.

2.      Some websites use unbounded number of dynamic pages. For example, algorithmically generated poetry or including a calendar.

3.      Webpages filled with a large number of characters so that when a lexical analyzer will try to parse it, it will end up crashing.

2. What to look for in queued URLs

http://www.example.org/-4RKgHfoN-6E11VCLhJOgPe79POuU2ZXhnTVbnUg_Zs.eyJpbnN0YW5jZUlkIjoiMTNkZDNmOTYtZTE1OC03NTU5LTEyMWMtZmM1YzE3MmQxMzk4Iiwic2lnbkRhdGUiOivZGUiOmZhbHNlLCJiaVRva2VIy… 

http://www.example.org/media/media/page/sites/css/html-reset.css

http://www.example.org/media/feed/pae/sites/all/themes/dev/custom/sites/all/themes/enviro-c4/css/html-reset.css

In most cases, large numbers of repeating directories or extra directories are a strong indication of a crawler trap. 

http://www.example.org/calendar/events?&page=1&mini=2015-09&mode=week&date=2021-12-04

Calendar pages are not always a problem, but have the potential to generate URLs infinitely into the future and past. 

 
3. How to prevent traps from being crawled

References: https://support.archive-it.org/hc/en-us/articles/208332943-Identify-and-avoid-crawler-traps-

 

So, in order to ensure you have a solid understanding of the potential impact of a spider trap, it’s important that we give an overview of what a spider trap is, how to identify one and how to diagnose one, but before we weave that particular web (pardon the spidery pun!), let’s take a step back and understand the fundamental reason why any of this is important to us as website optimisers, business owners and marketers; it’s all down to the concept of crawl budget and how you can influence website performance in the context of maintaining crawl efficiency through effective URL management.

Google and the other search engines have invested significant capital in creating these wonderful search engines that have become ingrained in our day-to-day lives; gone are the days when we dusted off an old Encyclopaedia Britannica for authoritative answers to our daily queries or the innocent early-internet days of simply ‘Asking Jeeves’ and hoping he served us up a treat! Search engines are big business, and with SEO spend set to skyrocket to $80bn in the US alone by 2020, chances are, they’re here to stay, in one form or another.

With the ever-evolving Google algorithm at the forefront of evrything we in the SEO industry try to build our best practices around, it’s fair to assume that there’s a significant cost associated with Big G maintaining this vast, Rankbrain-driven beast. Needless to say, the bots/crawlers/spiders employed by Lord Googlebot to crawl our sites, index our content and ultimately, display them to our target audiences, cost money to run. Hosted within vast server networks dotted throughout the globe, there is a financial cost associated with physically providing the bandwidth expended by these spiders as they crawl through the web on a continual basis.

Fundamentally, if a website isn’t optimised to allow spider access to its infrastructure in an efficient manner, a spider will inevitably reach a point in time whereby it reaches its allocated bandwidth allowance for a given site and moves on the next website on its crawl schedule, rather than hanging around indefinitely on the same site, hope that it finds a way through a myriad of issues before coming out the other side with a coherent understanding of what it’s just scanned.

Naturally, if there’s been an issue whereby a spider couldn’t complete the task at hand and ceases crawling a site as crawl budget has been reached, then the affected site could in a sense be deemed ‘less than optimal’ in the eyes of the crawler and could in essence be downgraded (when it comes to ranking a site versus, say, a competitor site on which the crawler has no issues accessing) as a result.

Furthermore, if a website has serious crawl issues, then some very important product, category or informational pages may never see the light of day in the SERPs if a spider was unable to reach them to begin with.

To put a slightly weird personification around this concept, consider the following scenario:

In this scenario (in case I've lost you already!), you are the crawler, the shops are the pages of a website and your friends are the SERPs; can you guess who played the role of the spider trap yet?! Those pesky shutters/security guards of course!

Crawl budget, in its purest definition, can be defined as the number of times a search engine crawler visits your website during a given timeframe, which is heavily influenced by its ease of navigation on a website. For example, if Googlebot typically visits your website approximately X times per month, we can assume with a good degree of confidence that that figure is your monthly crawl budget for Googlebot, although this is by no means set in stone.

It’s important to recognise that estimated crawl budget can evolve over time. Many other factors such as PageRank (the kind that Google still most definitely uses internally despite it being discontinued as a publicly-available toolbar metric) and server host load play a role in crawl budget too, as stated by former friend-to-SEOs, Matt Cutts in a revealing interview on the topic with Eric Enge a couple of years back.

A quick scan of your web logs or indeed via Search Console under ‘Crawl Stats’ will help you gain insight into what your average crawl budget for Google may be at a given point in time.

Here’s an example of what calculating estimated crawl budget looks like for our very own new spanky Wolfgang Digital website:

Looking at the above stats, if we take the daily average of 275*30 then we can deduce the average monthly crawl budget for this website from Google is 8250, meaning we now have an estimate of the number of pages we can expect Google to crawl, index and rank within a given timeframe.

If the number of crawlable assets (URLs) we want to rank for is significantly higher (or lower) than this amount, it typically means that we need to look further at crawl budget optimisation as a priority issues from a technical SEO perspective. Thankfully, we’re alright for now, but we regularly encounter scenarios in which websites are struggling with their web IA and oftentimes the root cause can be determined with a simple Screaming Frog crawl, configured to emulate Googlebot.

Bottom line is, if your site can't get indexed, it can't get ranked. Don’t let crawl budget play a part in how a spider treats your site, ensure you’re optimising to allow spiders enter your site, visits what’s important and exit the site left in no uncertain terms that you’ve pulled out all the stops to make it a smooth, logical experience for them.

A spider trap, as bluntly portrayed as an unreasonable security guard or inanimate shutter above, is basically what stands in the way of a spider accessing your website, having a lovely time in there and coming away with nothing but happy memories and some nice indexable pages to rank on SERPs versus the harsh reality of being trapped in a grim, never-ending loop of a section of your site which causes all sorts of trouble and eventually forces them to give up and move on due to pre-programmed bandwidth allowance limits.

Once you have access to the basic tools required in order to perfroma a site crawl, you're ready to commence your crawl analysis to determine whther a spider trap exists on your site.

They’re particularly prevalent on ecommerce sites due to the nature of large inventory management issues and fancy UX-led filtering configurations, but they can be found on pretty much any site with dynamically-served content. Big sites will naturally have more bandwidth allocated to it by search engine spiders, but that doesn’t mean they are less susceptible to being caught in a trap, on the contrary, if fact. Similarly, smaller sites can be equally effected by these issues if they’re not nipped in the bud at an early stage.

Sometimes, a trap is evident at a very early stage, with a loop kicking in with less than half the site crawled.

Here’s an example of a very early stage spider trap on a very small site, where the crawl hits the 8th asset and then loops back through URLs 1-7 before stumbling back over 8 at just 6.4% of crawl completion; this means a staggering 93.6% of this small website is not being crawled, indexed or ranked:

The following crawl completion index illustrates what a mid-stage spider trap looks like, the spiders have crawled a significant chunk of this large site bet can’t get any further, effectively leaving 2/5ths of this large ecommerce site ‘on the shelf’ in terms of potential web visibility:

Other times, it can kick in really late in the day, with almost the entire site crawled, yet the crawl will simply never end due to the crawler continually falling through the infinite loop inflicted by an intricate faceted navigation configuration or a simple coding error.

This can be particularly disheartening as just when you think the crawl is almost complete with only 161 URLs remaining from over 40k, it starts skipping back up around the 200 mark and never falls quite below the 100 mark towards completion.

This kind of trap will likely have less of an impact in the sense that the majority of the site has been crawled, but it’ll undoubtedly cause crawlers to look unfavourably on the site in question and perhaps reduce crawl budget to avoid running through similar issues in future crawls. Best not leave such things to chance in the hope that crawlers deem things ‘OK’ and will index most of what it’s found, we say let’s nip these traps in the bud and make crawl efficiency a non-issue once and for all!

There are four main causes of spider traps which we’ve encountered in recent times, each with varying degrees of complexity to both identify and diagnose; let’s begin with the simpler ones, moving through to the proper headache scenarios for us SEOs (not to mention the poor spiders, who I’m sure by now many of you are visualising as a hairy little Google-critter funnelling its way down the interweb towards your beloved domain!

If you’re more versed in using crawler tools then you may still have some burning questions such as ‘why will this crawl never end?’ or ‘why is smoke emanating from my brand new Lenovo?’; this guide should hopefully put your mind to ease and put you on the path towards an optimal crawler experience for all.

3.Check speed of your website
4.Use various technical checks done under SEO