Questions
Linux
Laravel
Mysql
Ubuntu
Git
Menu
HTML
CSS
JAVASCRIPT
SQL
PYTHON
PHP
BOOTSTRAP
JAVA
JQUERY
R
React
Kotlin
×
Linux
Laravel
Mysql
Ubuntu
Git
New posts in robots.txt
How to prevent abusive crawlers from crawling a rails app deployed on Heroku?
Jan 22, 2023
ruby-on-rails
heroku
robots.txt
Why use robot.txt on javascript files?
Jan 18, 2023
robots.txt
What does "User-agent: * Disallow: /" mean? [closed]
Jan 14, 2023
robots.txt
Googlebot cannot access CSS and JS files?
Jan 09, 2023
javascript
css
seo
robots.txt
How Disallow a mirror site (on sub-domain) using robots.txt? [closed]
Dec 28, 2022
seo
cdn
robots.txt
Parsing individual lines in a robots.txt file with C#
Dec 25, 2022
c#
webclient
robots.txt
How to set up robots.txt file for WordPress
Dec 23, 2022
wordpress
robots.txt
robots.txt in Laravel
Dec 19, 2022
php
laravel-5
robots.txt
How and where to add a robots.txt file to an ASP.net web application?
Dec 11, 2022
asp.net
robots.txt
Which is the best programming language to write a web bot? [closed]
Dec 12, 2022
artificial-intelligence
robots.txt
bots
.htaccess not letting robot.txt through
Dec 09, 2022
.htaccess
robots.txt
Can I use non latin characters in my robots.txt and sitemap.xml?
Nov 12, 2022
sitemap
robots.txt
domain-name
punycode
What is the sense of using robots.txt in GitHub pages?
Oct 19, 2022
github-pages
robots.txt
Stop Google indexing subdomains
Oct 07, 2022
search-engine
robots.txt
Twitter meta image is not rendering on Twitter because it "may be restricted by the site's robots.txt file"
Jun 23, 2022
image
meta-tags
robots.txt
twitter-card
How can I get robots.txt to block access to URLs on site after "?" character but index page itself?
Nov 26, 2019
html
seo
robots.txt
Can I use robots.txt to block certain URL parameters?
Aug 29, 2022
robots.txt
Robots.txt Allow sub folder but not the parent
Jul 25, 2022
robots.txt
Serving sitemap.xml and robots.txt with Spring MVC
Oct 19, 2021
java
spring
spring-mvc
sitemap
robots.txt
Older Entries »