Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Robots.TXT Disallow Syntax

Tags:

robots.txt

I have dynamically created pages, which can have any name.

www.example.com/pages/ 
www.example.com/pages/dynamicpage1/ 
www.example.com/pages/dynamicpage2/ 

I want the first url to pages to be crawled, but not any of the dynamic pages

User-agent: *
Disallow: /pages/*

The above will block all three pages listed. How do I just block all the dynamic pages, but not the first url.

like image 201
Henry Avatar asked Feb 11 '26 06:02

Henry


1 Answers

I figured it out..

User-agent: *
Disallow: /pages/*/

Happens all the time, solve the problem seconds after you post it.

like image 64
Henry Avatar answered Feb 17 '26 02:02

Henry



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!