I have problem where Meta crawler is trying to index URLs that don't exist on a certain page of my website.
I'll explaing with an example what is happening so you can understand easier.
pageA has several galleries that contain 6 images each.
Let's say that images from first gallery are named image-1, image-2, etc. and images from second gallery are named photo-1, photo-2, etc.
so URL like this www.domain.com/pageA?image-1 would open first image of first gallery and that is fine but what I see in cPanel it that Meta crawler is trying to index URLs like this:
www.domain.com/pageA?image-1?photo-1
www.domain.com/pageA?image-2?photo-3
and so on...
Here is one real URL example that Meta crawler is trying to index:
/en/things-to-do-in-split/recreation-and-beaches?split-beach-obojena-svjetlost-2-2-2?split-beach-bacvice-2?split-beach-ovcice-2?split-beach-bacvice-2-2?split-beach-ovcice-2-2-2-2-2
Right now this URL opens the page and no image is shown and there are hundreds of request like this happening which uses all server resources and website cannot load or it takes forever. So I had to ban certain IP range to stop the crawling.
Can you fix this so that BAGallery throws back a 404 error if URL doesn't exists?
Replies are visible only to logged in members with an active subscription.