robots txt disallow wild card

Question

I am having trouble stopping google crawling a few urls which cause errors.

I want to stop

• /project/123984932842/download/pdf
• /project/123984932842/download/zip

but allow

• /project/123984932842
• /project/123984932842/flat

I tried project/*/download/pdf but it doesn't seem to work. Does anyone know what would?

Answer 1

Do you have a / at the beginning of the Disallow: line?

User-agent: googlebot
Disallow: /project/*/download/pdf