4 Steps to create a robots.txt file in Django

robots.txt is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform web robots about which areas of the website should not be processed or scanned. The pages or the URL patterns included in the robots.txt file will not be indexed by the search engines.

When a site owner wishes to give instructions to web robots they place a text file ( called robots.txt) in the root of the website hierarchy (e.g. https://www.example.com/robots.txt). This text file contains the instructions in a specific format (see examples below). Robots that choose to follow the instructions try to fetch this file and read the instructions before fetching or scanning any other file from the website. If this file doesn't exist, web robots assume that the website owner is not wishing to place any limitations on crawling the entire site.

User-agent: *
allow: /blog/
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/

Creating a robots.txt file in Django is a simple task. Just follow the following steps to make a robots.txt file in Django.

Step 1

We will use the Django-robots module to make a robots.txt file for our website.

Install the package by using the following command :
pip install django-robots


Step 2

Include the "robots" app in the INSTALLED_APPS list :

#in settings.py

Now run the migrate command to include robots app tables in the Django admin app -
python manage.py migrate



Step 3

Add a URL pattern for the robots.txt file in the urls.py file.

#in urls.py
url(r'^robots.txt$' , include('robots.urls')),

Use the example.com/robots.txt url for a robots.txt file.

Step 4

Now, go to django admin app and add the URL patterns which you want to allow or disallow in Url .
Set rules for these URLs in the Rules table. You can select the URLs manually to apply rules for them.

Recent Articles

Changing Database from SQLite to MySQL - Django

One of the main benefit to use django for web development is that it comes with an inbuilt database/filesystem i.e SQLITE. Managing database and files is super easy in django. But there are some drawbacks of using SQLITE...

  - 2023-07-12
How to download a file in django

Sometimes we have to serve a file which may be a large file from our project directory. Suppose you want to transfer a file from your project directory to your friend or to some other person/server/platform....

  - 2023-07-12
How to change base url or domain in sitemap - Django

A sitemap is the first important SEO step which elaborates a website's content to search engines. It provides information to search engines about the available content on our site, which helps them to crawl webpages for indexing....

  - 2023-07-12
How to Make a Website

After reading this article you will never ever search for web development guide neither on Google nor on YouTube....

  - 2023-07-12
What is Javascript - Features and usages

I'm kind of a person who always likes to get started by installing the things & writing some code. I love that but this time JavaScript requires a little bit of the theoretical basic so that we can understand what things we are learning, how hard and powerful it is and where they can be applied....

  - 2023-07-12
Top 5 Interview Questions : Tips for HR round

In this article we are talking about the most famous HR questions and the mind-set to answer those HR questions....

  - 2023-07-12
How to get job in IT - Perfect resume guide

In today's article I want to talk about how to prepare the best resume to get a job in IT. Yes, everybody wants to have a job and in order to get a job you have to apply for it and when you apply for that, the resume is one good thing that you always submit....

  - 2023-07-12