{"id":954,"date":"2018-05-03T13:06:13","date_gmt":"2018-05-03T13:06:13","guid":{"rendered":"https:\/\/chemicloud.com\/kb\/?post_type=article&p=954"},"modified":"2023-12-14T11:51:39","modified_gmt":"2023-12-14T11:51:39","slug":"create-robots-txt-file-cpanel","status":"publish","type":"ht_kb","link":"https:\/\/chemicloud.com\/kb\/article\/create-robots-txt-file-cpanel\/","title":{"rendered":"How to Create a robots.txt File in cPanel"},"content":{"rendered":"
If you’ve ever built your website, you may have heard of a robots.txt<\/a> file and wondered, what is this file for? Well, you’re in the right place! Below, we will review this file and why it’s crucial.<\/article>\n
\n

What is a robots.txt file?<\/h3>\n

First of all, the robots.txt<\/strong><\/em> is nothing more than a plain text file (ASCII or UTF-8) located in your domain root\u00a0directory<\/strong>, which blocks (or allows) search engines to access certain areas of your site. The robots.txt<\/em> contains a simple set of commands (or directives), and it\u2019s typically applied to restrict crawler traffic onto your server, thus preventing unwanted resource usage.<\/p>\n

Search engines use so-called crawlers<\/strong> (or bots) to index parts of a website and return those as search results. You might want specific sensitive data stored on your server to be inaccessible for web searches. The robots.txt file helps you do just that.<\/p>\n

Note:<\/strong> Files or pages on your website are not entirely removed from crawlers if these files are indexed\/referenced from other websites. To protect your URL from appearing in Google search engines, you can password-protect the files directly from your server.<\/p>\n<\/article>\n

\n

How to create the robots.txt file
\n<\/strong><\/h3>\n

To create your robots.txt<\/em> file (if not already existent), follow the following steps:<\/p>\n

1.<\/strong> Log into your cPanel account<\/a><\/p>\n

2.<\/strong> Navigate to FILES<\/em> section and click on File Manager<\/strong><\/p>\n

\"\"
cPanel > Files > File Manager<\/figcaption><\/figure>\n

3.<\/strong>\u00a0 Browse File Manager to the website directory ( e.g public_html ) then Click on \u201cFile<\/strong>\u201d\u00a0 >> Type in \u201crobots.txt\u201d\u00a0 >> Click on \u201cCreate New File<\/strong>\u201d.<\/p>\n

\"\"<\/p>\n

4.<\/strong> Now, you are free to edit the content of this file by double-clicking on it.<\/p>\n

Note: <\/strong>you can create only one <\/strong>r<\/em>obots.txt<\/em> file for each domain. Duplicates are not allowed on the same root path. Each domain or sub-domain must contain its own robots.txt<\/em> file. \u00a0<\/strong>\u00a0<\/strong><\/p>\n<\/article>\n

\n

Examples of usage and syntax rules<\/h3>\n

Usually, a robots.txt<\/em> file contains one or more rules, each on its own separate line. Each rule blocks or allows access to a given crawler to a specified file path or the entire website.<\/p>\n