Some Linux Commands for Analytics

SED

sed is a stream editor.
A stream editor is used to perform basic text transformations on an input stream (a file or input from a pipeline).
While in some ways similar to an editor which permits scripted edits (such as ed), sed works by making only one pass over the input(s), and is consequently more efficient.
But it is sed’s ability to filter text in a pipeline which particularly distinguishes it from other types of editors.

Examples

The s stands for substitute and the g stands for global.

aaronjohns@pc:~$ cat doc.txt

John is an IT expert. John is the head of IT operations.

John has special access to the datacenter.

He joined the department in 2005. He retires in 2027.

#Non global: Replace John with Nick

aaronjohns@pc:~$ cat doc.txt |sed 's/John/Nick/'

Nick is an IT expert. John is the head of IT operations.

Nick has special access to the datacenter.

He joined the department in 2005. He retires in 2027.

#Global: Replace John with Nick

aaronjohns@pc:~$ cat doc.txt |sed 's/John/Nick/g'

Nick is an IT expert. Nick is the head of IT operations.

Nick has special access to the datacenter.

He joined the department in 2005. He retires in 2027.

#write the changes permanently

aaronjohns@pc:~$ cat doc.txt |sed 's/John/Nick/g' | tee doc.txt

Nick is an IT expert. Nick is the head of IT operations.

Nick has special access to the datacenter.

He joined the department in 2005. He retires in 2027.

aaronjohns@pc:~$ cat doc.txt

Nick is an IT expert. Nick is the head of IT operations.

Nick has special access to the datacenter.

He joined the department in 2005. He retires in 2027.

#Replacing the nth Occurrence of a Pattern in a Line: sed 's/word1/word2/n'

aaronjohns@pc:~$ cat doc.txt |sed 's/Nick/Joshua/2'

Nick is an IT expert. Joshua is the head of IT operations.

Nick has special access to the datacenter.

He joined the department in 2005. He retires in 2027.

#Replace Nick with Joshua if the word IT exists in the sentence

aaronjohns@pc:~$ cat doc.txt |sed '/IT/s/Nick/Joshua/g'

Joshua is an IT expert. Joshua is the head of IT operations.

Nick has special access to the datacenter.

He joined the department in 2005. He retires in 2027.

aaronjohns@pc:~$ ls / > root.txt

aaronjohns@pc:~$ cat root.txt

bin

bin.usr-is-merged

boot

dev

etc

home

lib

lib64

lib.usr-is-merged

lost+found

media

mnt

opt

proc

root

run

sbin

sbin.usr-is-merged

snap

srv

sys

tmp

usr

var

# Print lines 5 to 7 from the document

aaronjohns@pc:~$ sed -n 5,7p root.txt

etc

home

lib

wc

wc (short for word count) is a command in Unix, Plan 9, Inferno, and Unix-like operating systems.
The program reads either standard input or a list of computer files and generates one or more of the following statistics: newline count, word count, and byte count.
If a list of files is provided, both individual file and total statistics follow.

Examples:

aaronjohns@pc:~$ cat doc.txt

Nick is an IT expert. Nick is the head of IT operations.

Nick has special access to the datacenter.

He joined the department in 2005. He retires in 2027.

# Number of characters

aaronjohns@pc:~$ wc -c doc.txt

154 doc.txt

# Number of words and figures

aaronjohns@pc:~$ wc -w doc.txt

29 doc.txt

aaronjohns@pc:~$ grep -Eo "\\b[0-9]+\\b" doc.txt | wc -w

aaronjohns@pc:~$ grep -Eo "\\b[A-Za-z]+\\b" doc.txt | wc -w

awk

AWK (awk)is a domain-specific language designed for text processing and typically used as a data extraction and reporting tool. Like sed and grep, it's a filter, and is a standard feature of most Unix-like operating systems.
The AWK language is a data-driven scripting language consisting of a set of actions to be taken against streams of textual data – either run directly on files or used as part of a pipeline – for purposes of extracting or transforming text, such as producing formatted reports.
The language extensively uses the string datatype, associative arrays (that is, arrays indexed by key strings), and regular expressions. While AWK has a limited intended application domain and was especially designed to support one-liner programs, the language is Turing-complete, and even the early Bell Labs users of AWK often wrote well-structured large AWK programs.

Using Variables with awk

With awk, you can process text files. Awk assigns some variables for each data field found:
- $0 for the whole line.
- $1 for the first field.
- $2 for the second field.
- $n for the nth field.
- The whitespace character like space or tab is the default separator between fields in awk.

Examples:

aaronjohns@pc:~$ cat doc.txt

Nick is an IT expert. Nick is the head of IT operations.

Nick has special access to the datacenter.

He joined the department in 2005. He retires in 2027.

#Print the first and fourth words from each line in the document

aaronjohns@pc:~$ awk '{print $1 " " $4}' doc.txt

Nick IT

Nick access

He department

We will do analysis on a file named employees.txt

Since the default separator is a space, we need to change it to a comma by using the -F flag.

aaronjohns@pc:~$ cat employees.txt

123234877,Michael,Rogers,14

152934485,Anand,Manikutty,14

222364883,Саrol,Smith,37

326587417,Joe,Stevens,37

332154719,Мary-Anne,Foster, 14

332569843,George,ODonnell,77

546523478,John,Doe,59

631231482,David,Smith,77

654873219,Zacary,Efron,59

745685214,Eric,Goldsmith,59

845657245,Elizabeth,Doe,14

845657246,Kumar,Swamy,14

#First and Last Name of employees

aaronjohns@pc:~$ awk -F ',' '{print $2 " " $3}' employees.txt

Michael Rogers

Anand Manikutty

Саrol Smith

Joe Stevens

Мary-Anne Foster

George ODonnell

John Doe

David Smith

Zacary Efron

Eric Goldsmith

Elizabeth Doe

Kumar Swamy

#First name of employees who are 30 years and above in age

aaronjohns@pc:~$ awk -F',' '{if($4>=30) print $2}' employees.txt

Саrol

Joe

George

John

David

Zacary

Eric

#First name of employees who have their first name as George

aaronjohns@pc:~$ awk -F',' '{if($2=="George") print }' employees.txt

332569843,George,ODonnell,77

Sort

SORT command is used to sort a file, arranging the records in a particular order. By default, the sort command sorts file assuming the contents are ASCII. Using options in sort command, it can also be used to sort numerically.
SORT command sorts the contents of a text file, line by line.
sort is a standard command line program that prints the lines of its input or concatenation of all files listed in its argument list in sorted order.
The sort command is a command line utility for sorting lines of text files. It supports sorting alphabetically, in reverse order, by number, by month and can also remove duplicates.
The sort command can also sort by items not at the beginning of the line, ignore case sensitivity and return whether a file is sorted or not. Sorting is done based on one or more sort keys extracted from each line of input.
By default, the entire input is taken as sort key. Blank space is the default field separator.

#Ascending Order

aaronjohns@pc:~$ sort root.txt

bin

bin.usr-is-merged

boot

dev

etc

home

lib

lib64

lib.usr-is-merged

lost+found

media

mnt

opt

proc

root

run

sbin

sbin.usr-is-merged

snap

srv

sys

tmp

usr

var

#Descending Order

aaronjohns@pc:~$ sort -r root.txt

var

usr

tmp

sys

srv

snap

sbin.usr-is-merged

sbin

run

root

proc

opt

mnt

media

lost+found

lib.usr-is-merged

lib64

lib

home

etc

dev

boot

bin.usr-is-merged

bin

Read this intresting article on plotting graphs in the terminal:

https://www.baeldung.com/linux/cli-charting-and-plotting-tools

Page updated

Google Sites

Report abuse