Explore
For Enterprise
Join for Free
Log In

artes y humanidades
Negocios
Ciencias de la Computación
Ciencia de Datos
Tecnología de información
salud
Matemáticas y Lógica
Desarrollo Personal
Ciencias Físicas e Ingeniería
Ciencias sociales
Aprendizaje de un idioma
Grados
Certificados

Explorar todo lo que Coursera tiene para ofrecer

Explorar
Buscar
For Enterprise
Inicia Sesión
Únete de forma gratuita

File content exploration 2

Loading...

Big Data Essentials: HDFS, MapReduce and Spark RDD

4 (345 calificaciones) | 28K estudiantes inscritos

Curso 1 de 5 en Big Data for Data Engineers Programa Especializado

Have you ever heard about such technologies as HDFS, MapReduce, Spark? Always wanted to learn these new tools but missed concise starting material? Don’t miss this course either! In this 6-week course you will: - learn some basic technologies of the modern Big Data landscape, namely: HDFS, MapReduce and Spark; - be guided both through systems internals and their applications; - learn about distributed file systems, why they exist and what function they serve; - grasp the MapReduce framework, a workhorse for many modern Big Data applications; - apply the framework to process texts and solve sample business cases; - learn about Spark, the next-generation computational framework; - build a strong understanding of Spark basic concepts; - develop skills to apply these tools to creating solutions in finance, social networks, telecommunications and many other fields. Your learning experience will be as close to real life as possible with the chance to evaluate your practical assignments on a real cluster. No mocking, a friendly considerate atmosphere to make the process of your learning smooth and enjoyable. Get ready to work with real datasets alongside with real masters! Special thanks to: - Prof. Mikhail Roytberg, APT dept., MIPT, who was the initial reviewer of the project, the supervisor and mentor of half of the BigData team. He was the one, who helped to get this show on the road. - Oleg Sukhoroslov (PhD, Senior Researcher at IITP RAS), who has been teaching MapReduce, Hadoop and friends since 2008. Now he is leading the infrastructure team. - Oleg Ivchenko (PhD student APT dept., MIPT), Pavel Akhtyamov (MSc. student at APT dept., MIPT) and Vladimir Kuznetsov (Assistant at P.G. Demidov Yaroslavl State University), superbrains who have developed and now maintain the infrastructure used for practical assignments in this course. - Asya Roitberg, Eugene Baulin, Marina Sudarikova. These people never sleep to babysit this course day and night, to make your learning experience productive, smooth and exciting.

Destrezas que aprenderás

Python Programming, Apache Hadoop, Mapreduce, Apache Spark

Revisiones

4 (345 calificaciones)

5 stars
168 ratings
4 stars
96 ratings
3 stars
35 ratings
2 stars
16 ratings
1 star
30 ratings

SD

Jun 28, 2018

Absolutely essential for everyone who wants a proper introduction to HDFS, MapReduce and Spark. Brought to you by a great team of geniuses of their time ;)

NP

Apr 27, 2018

The course gave me more techical skill with Hadoop and Spark that help me can confidence in my career. Thank coursera and yandex so much.

De la lección

What are BigData and distributed file systems (e.g. HDFS)?

File system exploration7:38

File system managing6:08

File content exploration 15:54

File content exploration 213:25

Impartido por:

Ivan Puzyrevskiy
Technical Team Lead
Emeli Dral
Evgeniy Riabenko
Alexey A. Dral
Founder and Chief Executive Officer

Transcripción

Hi everybody.

In this video we will continue working with Unix Command Line Interface.

Today you'll learn some commands for file editing.

Also you'll take a look at the vim text editor and

awk programing language as one of the most popular tools for text file processing.

Probably everybody's used to working with text files with the help of text editors.

There are lot of examples of graphical text editors.

Not that everyone can easily remember their favorite one,

for instance, Notepad text edits sub one text and many more.

Vim is very powerful text editor which allow us to work with text files.

Vim is especially suitable for working with programs and

scripts directly from the terminal.

Vim stands for vi improved.

This is an improved version of vi editor which is still backward compatible

with vi.

Vim contains a lot of enhancements compared to vi.

Vim may not be installed from your system by default, so

if you do not have it feel free to install.

You can find some instructions regarding vim installations in the lesson materials.

Most often vim has started to edit the single file using the filename command,

vim filename.

After writing this command, vim editor opens and

you're able to edit your file with help of vim.

For editing the file set option I, which stands for insert.

In the insert mode you can edit your file.

For instance, let's edit the line, Hello, world!, into our empty file.

After finishing editing the file, you may want Acer to save your changes and

create the file or just create the file without saving anything.

In first case, use wq option, it will save your changes before creating vim.

Let's check if our changes have been saved.

Hope you remember at least three different ways from previous videos for

printing files.

Let's for example run the cat command and

see that now our file contains hello world string.

By the way if you have made changes which you do not need to save,

just type you!, when you greet vim.

This will not save any changes, file opening.

Vim allows us to do a lot of things with the file in a very efficient way.

For instance, search for special substrings and characters,

substitute some patterns to others, cut and paste pieces of file and many more.

For doing it, it's better to learn more about vim editor.

But this is a bit out of our scope today.

If you're interested in learning more about vim,

feel free to get more information and links from the lesson optional materials.

I personally encourage you to do it,

you will be surprised with the diversity of vim functionality.

The last thing I would like to mention about vim editor is configuration.

You can specify any options for vim for example the color scheme, the rules to

process text for instance substitution for Tab to several spaces, anything.

For doing this you should set the corresponding option in

.vimrc file stored in your home directory.

Note at the beginning of the file name, make the file invisible.

It will not be printed by default if you run ls command, but

you can print it out by stating option for ls.

More information about vim configuration is also available in the lesson materials.

Now, let us return back to bash language.

I would like to share some handy commands for file processing.

The first command is called cut.

It removes or cuts out section of each line of files.

The syntax is quite simple.

In most case, you will just set the delimiter by -d and

the field to cut out by -f option for choosing fields for the file.

There are some more possibilities to this file.

Feel free to explore them yourself with the help of the man utility.

Let's take a look at the following example.

You set the space symbol as a delimiter and

printout the first field of the file by running

the command cut -d ' ' -f1 multilined_file.txt.

As a result, you have printed the first comment on the file.

This you can easily print out the second column in the file just run

the common cut -d -f2 multilined_file.txt command.

The second command is called tr which stands for translate.

The tr command translates or substitutes one set of characters with another.

The tr utility copies the standard input

into the standard output with substitution or deletion of the selected characters.

That is why the syntax of using this command

is a bit different from everything you used before.

Let's take a look at the example.

Assume that you want to bring in the content of the multilined.txt file with

dots instead of spaces between the work line and number of lines.

For doing this you will use the following command,

cat multilined.txt tr.

As a result, you will see lines with dots instead of spaces.

This command works according to the following scenario.

First you print out the file content by running get command,

then you use tr or the output of the cat command.

To run command over the output of the previous one and we use the symbol.

We will discuss it later in this video.

The second popular scenario of using tr is dropping of the selected characters from

a file, print out the file with dropped spaces.

For doing this, you run cat multilined_file.txt

| tr -d Notice that in both scenarios you did not change the entire file.

You did change only the output.

Sort is an extremely popular tool for sorting files this way, or another.

By default, sort will rearrange lines, so file using the following rules.

The line starting with a number will appear before the line starting with

a letter.

The line starting with the letter that appears earlier in the alphabet

will appear before a line starting with a letter that appears later in the alphabet.

The line starting with lowercase letter will appear before the line starting with

same letter in uppercase.

You can manage the sorting rules easily by setting the corresponding options to

sort command.

Let's play around with it.

For a sorting lines in the reverse order, you use the -r option.

The result is in front of you.

Going deeper, if you would let to reorder the lines to file by a specific column or

columns, you can set it by the -k option.

For example like this, sort -k2, 2 -r multilined_file.txt.

This command will sort file by the second column, and

reverse the results of sorting.

Even more, if you would like to sort the lines by a specific column or

columns using these digits not as strings.

You can simply do it by setting -n option, which stands for numeric sorting.

Let's run the following command, sort- k 2,2 -r -n

multilined_file.txt and see how results are updated.

Now lines are ordered by digit from the second column in the reversed order.

Feel free to play more with the sorting by yourself.

The next tool I would like to discuss in this video is the awk

programming language.

The awk stands for Aho, Weinberger and Kernighan which is

an interpreted programming language which was developed at Bell Labs in 1970s.

And design for text processing especially on the execution of complex pattern

matching operation on streams of text or data.

Awk syntax is a subject to learn, again it is a bit beyond our scope,

so I shared some related materials in the optional section.

Despite this, you can make a simple processing right now,

because the awk syntax is very intuitive.

For example, let's bring the first comment of our file.

You already know how to do it with the help of the cut command.

Here's another way of doing it with the help of the awk.

You can see the comment on this slide.

Here is a few delimiter with a -f option, and

ask awk to print out the first field by typing print $1.

To demonstrate power of awk, let's solve the following task.

Assume you would like to substitute the word line with the word lines in each

line of our file simply by adding an s character to the line word.

Moreover, you would like to multiply each digit from the second field by ten.

And to do all of these in a single command.

With the help of awk, the decision will be quite simple.

You can see it in front of you.

You see that you just concatenated the first field with an s character

by $1"s" command and multiply the second field by 10 by $2*10 command.

Looks very intuitive, doesn't it?

Hopefully now you'll be interested in pressing these variants of awk

syntax yourself.

The last thing for us to be mentioned is bash operators.

As you have already know a lot of handy tools for

the text file processing you may want to write the result of such processing to

files not just to print them to terminal.

The easiest way of doing this is using prompts.

Prompt operator overwrites the file if it exist or creates if it doesn't exist.

And double prompt operator appends the file or

creates the file if it doesn't exist.

The way of using such operators is illustrated in the following example.

Assume you would like to create a new file which contains a list of files and

directories that you got as a result of the ryan.ls commmand.

For doing his just regulate the output of the ls command to the new

file with help of that prompt.

If you run the same command again and

again you will easily see that our file be rewritten each time.

Vise versa if you substitute prompt operator by the double prompt then

the result for ls commands will be appended to the file.

So each run of ls double prompt list_of_my_files.txt command

will lead to editing data at the end of the file.

The pipe operator is used to make outputters.

The first comments acts like as an input for the second comment.

You have already used it with tr comment, but

I'm eager to share with you one of the standard examples of pipe using.

Assume you need to calculate how many times each string can,

is repeated in our file.

For doing this, first you need to print out the file, then sort it and

finally run the unique -c command that will calculate a number of

occurrences of each unique lan assuming that lans are preliminary sorted.

For doing it you just need to write only this comment,

separated by the pipe because you need the output of each comment to serve as

an input for each of the following comments.

Here you go is a comment, cat multilined.txt, pipe, sort, pipe, uniq -c.

You can play around with the pipe and try to write longer list of commands,

it's fun.

Finally I can't help but sharing with you some more widely used

operators which you're not going to use soon in our course,

but you still might be interested in using it for the future.

Ampersand makes a command run in the background.

Double pipe executes the second command only if the execution of the first command

failed.

Double ampersand executes the second command

only if the execution of the first command succeeded.

That's it for today.

You have done a really good job with this video and

have learned a lot of handy commands and tools to work with files.

Now, you can use vim, text editor for editing files, cut command for drop

section of files line by line, tr command for translate one substring to another.

Sort command for sort files by lines, awk programming language for

text editings, and bash operators like pipe, ampersand, and prompt.

Here we finish the book about file editing.

In our next video you will learn some handy commands to control processes

running on your server with help of bash.

See you in the next video.

Explora nuestro catálogo

Inscríbete de manera gratuita y obtén recomendaciones personalizadas, actualizaciones y ofertas.

Coursera brinda acceso universal a la mejor educación del mundo, al asociarse con las mejores universidades y organizaciones, para ofrecer cursos en línea.

© 2019 Coursera Inc. Todos los derechos reservados.

Descargar en la App Store

Consíguelo en Google Play

Coursera
Acerca de
Liderazgo
Empleo
Catálogo
Certificados
Certificados MasterTrack™
Grados
For Enterprise
For Government

Comunidad
Learners
Partners
Developers
Beta Testers
Translators

Conectar
Blog
Facebook
LinkedIn
Twitter
Instagram
Youtube
Blog de Tecnología

Más
Términos
Privacidad
Ayuda
Accesibilidad
Prensa
Contacto
Directorio
Afiliados