robots.txt

Jonathan Davies 1786 0

robots.txt

This is the standard configuration for robots.txt to allow crawler all subpages of the whole site:

User-agent: *
Disallow:

Tags:

Previous Article

Reasonable User Configuration in .bashrc

Next Article

robots-make-pages

What's Your Reaction?

0

0

0

0

0

0

0

Jonathan Davies

After having accumulated thousands of valuable notes in private over the years, I decided to share them on this webpage and make them public for the benefit and improved work efficiency of others. Good luck.

Related Posts

Varient CMS

Jonathan Davies 0 2049

Example pages made with Boostrap

Example pages made with Boostrap

Jonathan Davies 0 1911

Overview over the most efficient Bootstrap 4 Classes

Overview over the most efficient Bootstrap 4 Classes

Jonathan Davies 0 1953

Overview and comparison over the different Javascript libraries for 2021

Overview and comparison over the different Javascript libraries...

Jonathan Davies 0 2742

Schrift & Schriften für ein Webprojekt

Schrift & Schriften für ein Webprojekt

Jonathan Davies 0 2262

jQuery - Introduction & possibilities

jQuery - Introduction & possibilities

Jonathan Davies 0 2068

Comments