The robots.txt File Protocol

Loading...

Del curso dictado por University of California, Davis

Search Engine Optimization Fundamentals

1074 calificaciones

Explore Related Videos

At Coursera, you will find the best lectures in the world. Here are some of our personalized recommendations for you

University of California, Davis

Search Engine Optimization Fundamentals

1074 calificaciones

Curso 2 de 6 en SpecializationSearch Engine Optimization (SEO)

Gain an understanding of search engine algorithms and how they affect organic search results and websites. Building on this knowledge, you’ll learn the key elements for creating an effective SEO strategy, including how to select keywords and perform keyword research; consumer psychology and search behavior; and how to conduct on-page SEO analysis to identify opportunities to improve a website’s search optimization.

De la lección

Introduction to Technical SEO

Welcome to week 3! In the lessons that follow, you will take a close look at the third leg of a solid SEO Strategy: Technical SEO. You’ll see how important structural components like sitemaps, redirects and other components lay a foundation for your content that will help your site get noticed. By the end of this module, you should be able to define Technical SEO and explain some of the basic aspects of this strategy. As you begin creating sitemaps and robot.txt files as well as planning redirects and managing site errors, you’ll learn how to employ best practices in your Technical SEO strategy.

Introduction to Technical SEO1:05

Laying the Structural Foundation With Technical SEO2:24

HTML Sitemap vs. XML Sitemap8:37

The robots.txt File Protocol6:30

Overcoming Pesky Error Codes6:43

Redirects 1018:37

404 Pages Best Practices5:14

Conoce a los instructores

Rebekah May
SEO Manager & Lead SEO Instructor, Lead Qual
UC Davis Division of Continuing and Professional Education

Hello again.

Now that we've introduced you to site maps, let's discuss another method for

controlling how search engines crawl your website and discover new content.

The robots.txt file, is a protocol that allows you to specify which pages should

or should not be indexed and specify the rate at which files are accessed by search

engine robots in this lesson we'll learn about the importance of these files,

but also recognize their limitations by looking at an example robots.txt file,

we'll learn how to create and place these files in your website.

When we just discussed sitemaps,

we learned that sitemaps are an inclusionary file, meaning

they tell search engines which information you want included in search results.

You will also have the ability to exclude specific pages from being crawled.

You can do this through a robots.txt file.

The robtos.txt file is a protocol that was created in the early days of the Internet

to prevent robots from crawling areas of the web they were not supposed to access.

That protocol today is often to refered to simply as robots.txt.

This is a simple text file that can be uploaded to your server.

If a robot wants to visit a page on a website,

let's say example.com/bluewidgets.

The robot will first go to the website's robot.txt file.

This file will be located at example.com/robots.txt.

Search robots will then check to see

if there are any instructions prohibiting it from crawling that page.

It's important to note that while robots.txt files

provide search engine bots with your preferences for what should or

should not be crawled, robots can choose to ignore information in this file.

Search engines, like Google or Bing, tend to respect this file.

But people create robots for many reasons.

And these reasons can sometimes be malicious.

Some robots are created to scan the web for vulnerabilities so

people can hack into a person's website.

Other robots will crawl websites and

harvest email addresses to sell to companies or spammers.

Also remember that robots.txt is a publicly available file.

Meaning anyone can see what sections of your server

you do not want robots to access.

For example, we can view the robots.txt file of UC David Extension's website.

If you go to extension.ucdavis.edu/robots.txt or

robots.txt, you can see what the Robots.txt file looks like.

If you take a look at their file, you will see a couple different areas that provide

key information to search engine crawlers.

First, is the introductory text that explains what the robots.txt file is.

This information isn't necessary, but

many website content management systems include this information by default.

You will also see here, that the file points out that these instructions

will be ignored unless it is placed at the root of your host.

This means that you always access the sites robots.txt

file at the example.com/robots.txt.

If it is placed anywhere else on the site,

such as example.com/site/robots.txt.

It will be ignored.

Under that area, you will see specific instructions.

For example, User-agent specifies what user agents should crawl website content.

For example, you can set up specific instructions for Google, Bing, or Yahoo.

Most of the time, however, there is no real reason to do so.

They asterisk works as a wildcard & means that all robots should follow

the below instructions.

The crawl delay sets how many seconds a robot should take

before accessing a new page own your site.

This can help prevent overload of server resources which may crash your website.

The pound symbol, or hashtag, is used to make a note.

This is for humans to read and is not read by robots.

In this example, it's a note by the webmaster to remind himself, or

other webmasters, but the below items are specific directories of the site.

The disallow directive instructs robots

not to crawl the content it is referencing.

For example, do not crawl the directory/includes.

So now that we know that the asterisk and disallow directives mean.

If we wanted to exclude robots from accessing our entire site,

we would simply enter the instructions User-agent followed by an asterisk.

And then on a new line, we would enter Disallow followed by a forward slash.

If we wanted to allow robots complete access to the site,

we would simply keep the disallow field blank.

For example, we would simply enter, User-agent,

followed by an asterisk or wild card and

then on a new line type Disallow, and not include any information after that.

You could also simply not use a robots.txt file,

which will automatically make all information public.

You should now understand what a robots.txt file is used for,

how to create and read a robots.txt file,

as well as where this information should be placed in your server.

That completes the video portion of this lesson.

Explora nuestro catálogo

Inscríbete de manera gratuita y obtén recomendaciones personalizadas, actualizaciones y ofertas.

Coursera brinda acceso universal a la mejor educación del mundo, al asociarse con las mejores universidades y organizaciones, para ofrecer cursos en línea.

© 2019 Coursera Inc. Todos los derechos reservados.

Descargar en la App Store

Consíguelo en Google Play