Automating Tasks Using bash [EN].pdf

Overview

• Introduction to command shells & bash

Automating Tasks Using

bash

• bash fundamentals

– I/O redirection, pipelining, wildcard expansion, shell variables

David McCaughan, HPC Analyst

SHARCNET, University of Guelph

dbm@sharcnet.ca

• Shell scripting

– writing bash scripts

– control structures, string operations, pattern matching, command

substitution

– system tools

• Examples

– demo

HPC Resources

What is a Shell?

Brief History of the Major

UNIX Shells

• User interfaces

– GUI, character based, etc.

• 1979: Bourne shell ( sh )

– first UNIX shell

– still widely used as the LCD of shells

• A shell is a character-based user interface

– interprets the text typed in by the user translating them into

instructions to the operating system (and vice versa)

– anyone using SHARCNET systems is already familiar with the

command line your shell provides (typically bash)

• 1981: C shell ( csh )

– part of BSD UNIX

– commands and syntax which resembled C

– introduced aliases, job control

• We tend to see a shell purely as a user interface

– possible to use it as a programming environment also

– shell scripts

• 1988: Bourne again shell ( bash )

– developed as part of GNU project (default shell in Linux)

– incorporated much from csh, ksh and others

– introduced command-line editing, functions, integer arithmetic, etc.

HPC Resources

bash Basics

Reminder: System Tools

• Review of concepts

– bash has a great deal of syntax that you may already be using in

your command lines

• I/O redirection, pipelines, wildcard expansion

– anything we do on the CLI applies equally to scripts (remember,

our command-line is provided by a bash shell!)

• Anything that is usable on the system, can be used in a script---

consider some commonly used utilities:

– echo (output text to stdout)

• e.g. echo “Hello, world!”

• e.g. echo -n “Hello, world!”

– cat (copy input to output)

• e.g. cat somefile.txt

– cut (select columns from text)

• e.g. cut -f 2 -d ‘ ‘ file_notable_field2.txt

• Live review

– use “help” command to obtain a list of commands, and specific

information on any built-in command

– sed (stream editor)

• e.g. sed -e 's/\ */\ /g’ file_excess_ws.txt

– mv , cp , mkdir , ls , file , etc.

HPC Resources

I/O Redirection

Pipelineing

• When we run a program we always have the notion of

“standard input” and “standard output”

– typically the keyboard and terminal respectively

• System calls exist to allow the programmer to connect

stdout of one process to stdin of another

– bash provides a means of doing this on the command-line; we

refer to this as “piping” the output of the first to the input of the

second

– e.g. grabbing just the time and load average for past 15min from

output of uptime command, using cut :

• Redirecting the input/output streams

./myprog arg1 arg2 > output.txt

./myprog arg1 arg2 < input.txt

./myprog arg1 arg2 < input.txt > output.txt

– see also:

• > vs >> (overwrite vs append)

• 1> 2> (stdout [default], stderr)

HPC Resources

Wildcard Expansion

A Note About Meta-

characters

• bash will expand certain meta-characters when used in file names

– ? - matches any single character

– * - matches any sequence of characters, including none

– [] - matches any character in the set (first char ! negates)

• Note that this expansion is performed by the shell

• bash recognizes many characters with “special meaning”

– already we’ve seen: > | * ? [ ]

– there are many more:

• ~ - home directory

• # - comment

• $ - variable

• & - background job

• ; - command separator

• ’ - strong quotation (no interpretation)

• ” - weak quotation (limited interpretation)

• - whitespace

• etc.

HPC Resources

A Note About Meta-

characters (cont.)

Shell Variables

• Quotes

– enclosing a string in single-quotes will prevent the shell from

interpreting them

• A shell variable is a name with an associated string

value

– you have likely already seen these in your shell as environment

variables (PATH, LD_LIBRARY_PATH, etc.)

– by convention we use all upper-case for shell variables, however

it is common to see lower case “temporary” variables in scripts

mkdir ‘Name With Spaces’

cat ‘filenamewitha*.txt’

• Escaping characters

– a backslash “escapes” meta-character that follows

• consider: line continuation, literal quotes in strings, etc.

• Shell variables are created by assignment

VARNAME=string

– note: no whitespace around = (most common error)

– a variable can be “deleted”, if necessary, using unset

• unknown variables are assumed to be the empty string

cat filenamewitha\*.txt

HPC Resources

Shell Variables (cont.)

Shell Programming

• The value of a shell variable can be used in commands

by enclosing the name in ${}

– this is very easy to play with on the command-line (and excellent

way to distinguish single and double quotes)

• In theory we could write increasingly complex command-

lines to produce sophisticated behaviours

– this would quickly become impractical

– we will write shell scripts to facilitate more complex situations

• Script

– a file containing shell commands

– created using a text editor

– can contain any legal bash commands

• i.e. everything you are already used to being able to do on

the command-line, together with the bash shell features you

are learning today (and much more)

HPC Resources

Running a Script

Example: Running a Script

• Instruct your shell to execute the contents of a text file as

bash commands

source scriptname

– executes lines of file as commands in your current shell (as if

you’d typed them in at the command-line)

• More convenient to run them like a program

#!/bin/bash

– should be first line of script (portability)

– set execute permission on the file ( chmod u+x scriptname )

– run it as if it were any other program

– note: this executes commands in a new shell

HPC Resources

Control Structures

Branching: IF + conditions

• We need a means of performing branching and

managing flow of control to be truly useful

if condition; then

commands

[elif condition; then

commands

… ]

[else

commands]

• condition

– any list of commands

– can link conditions using &&, ||

– if tests the exit status of the

last command;

– i.e. “ if program execution

succeeds then do the

following ”

• Branching:

– IF..ELSE

– Also: CASE

• Iteration:

– FOR

– Also: WHILE, UNTIL, SELECT

• Note:

if condition

then

– is equivalent

• syntax: [ condition ]

– [] is a statement; returns an

exit status corresponding to

truth of condition

– necessary as if can only test

exit status

HPC Resources

Condition Tests

IF Examples

• String (i.e. variable) testing

– e.g. [ str1 = str2 ]

str1 = str2 - equal

str1 != str2 - not equal

str1 < str2 - less than

str1 > str2 - greater than

• File testing

– e.g. [ -e ${filename} ]

-e - file exists

-d - file exists + is directory

-f - file exists + is regular

-r - have read perm.

-w - have write perm.

-x - have execute perm.

# detect failure in attempt to copy ${infile} to ${outfile}

if ( ! cp ${infile} ${outfile} >& /dev/null ); then

echo "error copying input file to output location"

exit 2

– unary tests for null strings

-n str - not null

-z str - is null

# test if ${dir} a directory; shows a compound condition

if [ ${dir} = ${targetdir} && -d ${dir} ]; then

mv ${dir}/${file} ${archivedir}

elif [ ${dir} = ${targetdir} && -f ${dir} ]; then

echo “${dir} is a file”

else

echo “${dir} does not exist”

– binary tests for modification

time:

[ file1 -nt file2 ]

[ file1 -ot file 2 ]

HPC Resources

Plik z chomika:

Inne pliki z tego folderu:

Inne foldery tego chomika: