🔙 Quay lại trang tải sách pdf ebook Bash Notes for Professionals Ebooks Nhóm Zalo Notes for Professionals Bash Bash Notes for Professionals 100+ pages of professional hints and tricks GoalKicker.com Free Programming Books Disclaimer This is an unocial free book created for educational purposes and is not aliated with ocial Bash group(s) or company(s). All trademarks and registered trademarks are the property of their respective owners Contents About ................................................................................................................................................................................... 1 Chapter 1: Getting started with Bash ................................................................................................................. 2 Section 1.1: Hello World ................................................................................................................................................. 2 Section 1.2: Hello World Using Variables ..................................................................................................................... 4 Section 1.3: Hello World with User Input ...................................................................................................................... 4 Section 1.4: Importance of Quoting in Strings ............................................................................................................ 5 Section 1.5: Viewing information for Bash built-ins .................................................................................................... 6 Section 1.6: Hello World in "Debug" mode .................................................................................................................. 6 Section 1.7: Handling Named Arguments .................................................................................................................... 7 Chapter 2: Script shebang ........................................................................................................................................ 8 Section 2.1: Env shebang .............................................................................................................................................. 8 Section 2.2: Direct shebang .......................................................................................................................................... 8 Section 2.3: Other shebangs ........................................................................................................................................ 8 Chapter 3: Navigating directories ...................................................................................................................... 10 Section 3.1: Absolute vs relative directories .............................................................................................................. 10 Section 3.2: Change to the last directory .................................................................................................................. 10 Section 3.3: Change to the home directory .............................................................................................................. 10 Section 3.4: Change to the Directory of the Script .................................................................................................. 10 Chapter 4: Listing Files ............................................................................................................................................ 12 Section 4.1: List Files in a Long Listing Format ......................................................................................................... 12 Section 4.2: List the Ten Most Recently Modified Files ............................................................................................ 13 Section 4.3: List All Files Including Dotfiles ................................................................................................................ 13 Section 4.4: List Files Without Using `ls` ..................................................................................................................... 13 Section 4.5: List Files ................................................................................................................................................... 14 Section 4.6: List Files in a Tree-Like Format ............................................................................................................. 14 Section 4.7: List Files Sorted by Size .......................................................................................................................... 14 Chapter 5: Using cat .................................................................................................................................................. 16 Section 5.1: Concatenate files ..................................................................................................................................... 16 Section 5.2: Printing the Contents of a File ............................................................................................................... 16 Section 5.3: Write to a file ........................................................................................................................................... 17 Section 5.4: Show non printable characters ............................................................................................................. 17 Section 5.5: Read from standard input ..................................................................................................................... 18 Section 5.6: Display line numbers with output ......................................................................................................... 18 Section 5.7: Concatenate gzipped files ..................................................................................................................... 18 Chapter 6: Grep ........................................................................................................................................................... 20 Section 6.1: How to search a file for a pattern ......................................................................................................... 20 Chapter 7: Aliasing ..................................................................................................................................................... 21 Section 7.1: Bypass an alias ........................................................................................................................................ 21 Section 7.2: Create an Alias ........................................................................................................................................ 21 Section 7.3: Remove an alias ..................................................................................................................................... 21 Section 7.4: The BASH_ALIASES is an internal bash assoc array .......................................................................... 22 Section 7.5: Expand alias ............................................................................................................................................ 22 Section 7.6: List all Aliases .......................................................................................................................................... 22 Chapter 8: Jobs and Processes ........................................................................................................................... 23 Section 8.1: Job handling ............................................................................................................................................ 23 Section 8.2: Check which process running on specific port .................................................................................... 25 Section 8.3: Disowning background job .................................................................................................................... 25 Section 8.4: List Current Jobs ..................................................................................................................................... 25 Section 8.5: Finding information about a running process ..................................................................................... 25 Section 8.6: List all processes ..................................................................................................................................... 26 Chapter 9: Redirection ............................................................................................................................................. 27 Section 9.1: Redirecting standard output .................................................................................................................. 27 Section 9.2: Append vs Truncate ............................................................................................................................... 27 Section 9.3: Redirecting both STDOUT and STDERR .............................................................................................. 28 Section 9.4: Using named pipes ................................................................................................................................. 28 Section 9.5: Redirection to network addresses ........................................................................................................ 30 Section 9.6: Print error messages to stderr .............................................................................................................. 30 Section 9.7: Redirecting multiple commands to the same file ............................................................................... 31 Section 9.8: Redirecting STDIN .................................................................................................................................. 31 Section 9.9: Redirecting STDERR ............................................................................................................................... 32 Section 9.10: STDIN, STDOUT and STDERR explained ............................................................................................ 32 Chapter 10: Control Structures ............................................................................................................................ 34 Section 10.1: Conditional execution of command lists ............................................................................................. 34 Section 10.2: If statement ............................................................................................................................................ 35 Section 10.3: Looping over an array .......................................................................................................................... 36 Section 10.4: Using For Loop to List Iterate Over Numbers .................................................................................... 37 Section 10.5: continue and break ............................................................................................................................... 37 Section 10.6: Loop break ............................................................................................................................................. 37 Section 10.7: While Loop ............................................................................................................................................. 38 Section 10.8: For Loop with C-style syntax ............................................................................................................... 39 Section 10.9: Until Loop ............................................................................................................................................... 39 Section 10.10: Switch statement with case ................................................................................................................ 39 Section 10.11: For Loop without a list-of-words parameter ..................................................................................... 40 Chapter 11: true, false and : commands .......................................................................................................... 41 Section 11.1: Infinite Loop ............................................................................................................................................. 41 Section 11.2: Function Return ...................................................................................................................................... 41 Section 11.3: Code that will always/never be executed ........................................................................................... 41 Chapter 12: Arrays ...................................................................................................................................................... 42 Section 12.1: Array Assignments ................................................................................................................................. 42 Section 12.2: Accessing Array Elements .................................................................................................................... 43 Section 12.3: Array Modification ................................................................................................................................. 43 Section 12.4: Array Iteration ....................................................................................................................................... 44 Section 12.5: Array Length .......................................................................................................................................... 45 Section 12.6: Associative Arrays ................................................................................................................................. 45 Section 12.7: Looping through an array .................................................................................................................... 46 Section 12.8: Destroy, Delete, or Unset an Array ..................................................................................................... 47 Section 12.9: Array from string ................................................................................................................................... 47 Section 12.10: List of initialized indexes ...................................................................................................................... 47 Section 12.11: Reading an entire file into an array .................................................................................................... 48 Section 12.12: Array insert function ............................................................................................................................ 48 Chapter 13: Associative arrays ............................................................................................................................ 50 Section 13.1: Examining assoc arrays ........................................................................................................................ 50 Chapter 14: Functions ............................................................................................................................................... 52 Section 14.1: Functions with arguments ..................................................................................................................... 52 Section 14.2: Simple Function ..................................................................................................................................... 53 Section 14.3: Handling flags and optional parameters ........................................................................................... 53 Section 14.4: Print the function definition .................................................................................................................. 54 Section 14.5: A function that accepts named parameters ...................................................................................... 54 Section 14.6: Return value from a function ............................................................................................................... 55 Section 14.7: The exit code of a function is the exit code of its last command ..................................................... 55 Chapter 15: Bash Parameter Expansion .......................................................................................................... 57 Section 15.1: Modifying the case of alphabetic characters ..................................................................................... 57 Section 15.2: Length of parameter ............................................................................................................................ 57 Section 15.3: Replace pattern in string ...................................................................................................................... 58 Section 15.4: Substrings and subarrays .................................................................................................................... 59 Section 15.5: Delete a pattern from the beginning of a string ................................................................................ 60 Section 15.6: Parameter indirection ........................................................................................................................... 61 Section 15.7: Parameter expansion and filenames .................................................................................................. 61 Section 15.8: Default value substitution ..................................................................................................................... 62 Section 15.9: Delete a pattern from the end of a string .......................................................................................... 62 Section 15.10: Munging during expansion ................................................................................................................. 63 Section 15.11: Error if variable is empty or unset ...................................................................................................... 64 Chapter 16: Copying (cp) ......................................................................................................................................... 65 Section 16.1: Copy a single file .................................................................................................................................... 65 Section 16.2: Copy folders ........................................................................................................................................... 65 Chapter 17: Find ........................................................................................................................................................... 66 Section 17.1: Searching for a file by name or extension .......................................................................................... 66 Section 17.2: Executing commands against a found file ......................................................................................... 66 Section 17.3: Finding file by access / modification time .......................................................................................... 67 Section 17.4: Finding files according to size .............................................................................................................. 68 Section 17.5: Filter the path ......................................................................................................................................... 69 Section 17.6: Finding files by type .............................................................................................................................. 70 Section 17.7: Finding files by specific extension ........................................................................................................ 70 Chapter 18: Using sort .............................................................................................................................................. 71 Section 18.1: Sort command output ........................................................................................................................... 71 Section 18.2: Make output unique .............................................................................................................................. 71 Section 18.3: Numeric sort .......................................................................................................................................... 71 Section 18.4: Sort by keys ........................................................................................................................................... 72 Chapter 19: Sourcing ................................................................................................................................................. 74 Section 19.1: Sourcing a file ......................................................................................................................................... 74 Section 19.2: Sourcing a virtual environment ........................................................................................................... 74 Chapter 20: Here documents and here strings ........................................................................................... 76 Section 20.1: Execute command with here document ............................................................................................. 76 Section 20.2: Indenting here documents .................................................................................................................. 76 Section 20.3: Create a file ........................................................................................................................................... 77 Section 20.4: Here strings ........................................................................................................................................... 77 Section 20.5: Run several commands with sudo ..................................................................................................... 78 Section 20.6: Limit Strings .......................................................................................................................................... 78 Chapter 21: Quoting ................................................................................................................................................... 80 Section 21.1: Double quotes for variable and command substitution .................................................................... 80 Section 21.2: Dierence between double quote and single quote ......................................................................... 80 Section 21.3: Newlines and control characters ......................................................................................................... 81 Section 21.4: Quoting literal text ................................................................................................................................. 81 Chapter 22: Conditional Expressions ................................................................................................................ 83 Section 22.1: File type tests ......................................................................................................................................... 83 Section 22.2: String comparison and matching ....................................................................................................... 83 Section 22.3: Test on exit status of a command ...................................................................................................... 85 Section 22.4: One liner test ......................................................................................................................................... 85 Section 22.5: File comparison .................................................................................................................................... 85 Section 22.6: File access tests .................................................................................................................................... 86 Section 22.7: Numerical comparisons ....................................................................................................................... 86 Chapter 23: Scripting with Parameters ........................................................................................................... 88 Section 23.1: Multiple Parameter Parsing .................................................................................................................. 88 Section 23.2: Argument parsing using a for loop .................................................................................................... 89 Section 23.3: Wrapper script ...................................................................................................................................... 89 Section 23.4: Accessing Parameters ......................................................................................................................... 90 Section 23.5: Split string into an array in Bash ......................................................................................................... 91 Chapter 24: Bash history substitutions ........................................................................................................... 92 Section 24.1: Quick Reference .................................................................................................................................... 92 Section 24.2: Repeat previous command with sudo ............................................................................................... 93 Section 24.3: Search in the command history by pattern ....................................................................................... 93 Section 24.4: Switch to newly created directory with !#:N ...................................................................................... 93 Section 24.5: Using !$ .................................................................................................................................................. 94 Section 24.6: Repeat the previous command with a substitution .......................................................................... 94 Chapter 25: Math ........................................................................................................................................................ 95 Section 25.1: Math using dc ........................................................................................................................................ 95 Section 25.2: Math using bash capabilities ............................................................................................................... 96 Section 25.3: Math using bc ........................................................................................................................................ 96 Section 25.4: Math using expr .................................................................................................................................... 97 Chapter 26: Bash Arithmetic ................................................................................................................................. 98 Section 26.1: Simple arithmetic with (( )) ................................................................................................................... 98 Section 26.2: Arithmetic command ........................................................................................................................... 98 Section 26.3: Simple arithmetic with expr ................................................................................................................. 99 Chapter 27: Scoping ................................................................................................................................................ 100 Section 27.1: Dynamic scoping in action ................................................................................................................. 100 Chapter 28: Process substitution ...................................................................................................................... 101 Section 28.1: Compare two files from the web ....................................................................................................... 101 Section 28.2: Feed a while loop with the output of a command .......................................................................... 101 Section 28.3: Concatenating files ............................................................................................................................ 101 Section 28.4: Stream a file through multiple programs at once .......................................................................... 101 Section 28.5: With paste command ........................................................................................................................ 102 Section 28.6: To avoid usage of a sub-shell ........................................................................................................... 102 Chapter 29: Programmable completion ....................................................................................................... 103 Section 29.1: Simple completion using function ..................................................................................................... 103 Section 29.2: Simple completion for options and filenames ................................................................................ 103 Chapter 30: Customizing PS1 .............................................................................................................................. 104 Section 30.1: Colorize and customize terminal prompt ......................................................................................... 104 Section 30.2: Show git branch name in terminal prompt ..................................................................................... 105 Section 30.3: Show time in terminal prompt ........................................................................................................... 105 Section 30.4: Show a git branch using PROMPT_COMMAND .............................................................................. 106 Section 30.5: Change PS1 prompt ............................................................................................................................ 106 Section 30.6: Show previous command return status and time .......................................................................... 107 Chapter 31: Brace Expansion ............................................................................................................................... 109 Section 31.1: Modifying filename extension ............................................................................................................. 109 Section 31.2: Create directories to group files by month and year ...................................................................... 109 Section 31.3: Create a backup of dotfiles ................................................................................................................ 109 Section 31.4: Use increments .................................................................................................................................... 109 Section 31.5: Using brace expansion to create lists ............................................................................................... 109 Section 31.6: Make Multiple Directories with Sub-Directories ............................................................................... 110 Chapter 32: getopts : smart positional-parameter parsing .............................................................. 111 Section 32.1: pingnmap ............................................................................................................................................. 111 Chapter 33: Debugging .......................................................................................................................................... 113 Section 33.1: Checking the syntax of a script with "-n" .......................................................................................... 113 Section 33.2: Debugging using bashdb .................................................................................................................. 113 Section 33.3: Debugging a bash script with "-x" .................................................................................................... 113 Chapter 34: Pattern matching and regular expressions ..................................................................... 115 Section 34.1: Get captured groups from a regex match against a string ........................................................... 115 Section 34.2: Behaviour when a glob does not match anything ......................................................................... 115 Section 34.3: Check if a string matches a regular expression .............................................................................. 116 Section 34.4: Regex matching .................................................................................................................................. 116 Section 34.5: The * glob ............................................................................................................................................ 116 Section 34.6: The ** glob ........................................................................................................................................... 117 Section 34.7: The ? glob ............................................................................................................................................ 117 Section 34.8: The [ ] glob ......................................................................................................................................... 118 Section 34.9: Matching hidden files ......................................................................................................................... 119 Section 34.10: Case insensitive matching ................................................................................................................ 119 Section 34.11: Extended globbing ............................................................................................................................. 119 Chapter 35: Change shell ...................................................................................................................................... 121 Section 35.1: Find the current shell .......................................................................................................................... 121 Section 35.2: List available shells ............................................................................................................................. 121 Section 35.3: Change the shell ................................................................................................................................. 121 Chapter 36: Internal variables ........................................................................................................................... 122 Section 36.1: Bash internal variables at a glance ................................................................................................... 122 Section 36.2: $@ ........................................................................................................................................................ 123 Section 36.3: $# ......................................................................................................................................................... 124 Section 36.4: $HISTSIZE ............................................................................................................................................ 124 Section 36.5: $FUNCNAME ....................................................................................................................................... 124 Section 36.6: $HOME ................................................................................................................................................. 124 Section 36.7: $IFS ....................................................................................................................................................... 124 Section 36.8: $OLDPWD ............................................................................................................................................ 125 Section 36.9: $PWD ................................................................................................................................................... 125 Section 36.10: $1 $2 $3 etc.. ....................................................................................................................................... 125 Section 36.11: $* .......................................................................................................................................................... 126 Section 36.12: $! .......................................................................................................................................................... 126 Section 36.13: $? ......................................................................................................................................................... 126 Section 36.14: $$ ........................................................................................................................................................ 126 Section 36.15: $RANDOM .......................................................................................................................................... 126 Section 36.16: $BASHPID ........................................................................................................................................... 127 Section 36.17: $BASH_ENV ....................................................................................................................................... 127 Section 36.18: $BASH_VERSINFO ............................................................................................................................ 127 Section 36.19: $BASH_VERSION .............................................................................................................................. 127 Section 36.20: $EDITOR ............................................................................................................................................ 127 Section 36.21: $HOSTNAME ...................................................................................................................................... 127 Section 36.22: $HOSTTYPE ....................................................................................................................................... 128 Section 36.23: $MACHTYPE ...................................................................................................................................... 128 Section 36.24: $OSTYPE ............................................................................................................................................ 128 Section 36.25: $PATH ................................................................................................................................................ 128 Section 36.26: $PPID .................................................................................................................................................. 128 Section 36.27: $SECONDS ......................................................................................................................................... 128 Section 36.28: $SHELLOPTS ..................................................................................................................................... 129 Section 36.29: $_ ....................................................................................................................................................... 129 Section 36.30: $GROUPS ........................................................................................................................................... 129 Section 36.31: $LINENO ............................................................................................................................................. 129 Section 36.32: $SHLVL .............................................................................................................................................. 129 Section 36.33: $UID .................................................................................................................................................... 131 Chapter 37: Job Control ........................................................................................................................................ 132 Section 37.1: List background processes ................................................................................................................ 132 Section 37.2: Bring a background process to the foreground ............................................................................. 132 Section 37.3: Restart stopped background process .............................................................................................. 132 Section 37.4: Run command in background .......................................................................................................... 132 Section 37.5: Stop a foreground process ................................................................................................................ 132 Chapter 38: Case statement ............................................................................................................................... 133 Section 38.1: Simple case statement ....................................................................................................................... 133 Section 38.2: Case statement with fall through ..................................................................................................... 133 Section 38.3: Fall through only if subsequent pattern(s) match .......................................................................... 133 Chapter 39: Read a file (data stream, variable) line-by-line (and/or field-by-field)? ......... 135 Section 39.1: Looping through a file line by line ..................................................................................................... 135 Section 39.2: Looping through the output of a command field by field ............................................................. 135 Section 39.3: Read lines of a file into an array ....................................................................................................... 135 Section 39.4: Read lines of a string into an array .................................................................................................. 136 Section 39.5: Looping through a string line by line ................................................................................................ 136 Section 39.6: Looping through the output of a command line by line ................................................................ 136 Section 39.7: Read a file field by field ..................................................................................................................... 136 Section 39.8: Read a string field by field ................................................................................................................ 137 Section 39.9: Read fields of a file into an array ..................................................................................................... 137 Section 39.10: Read fields of a string into an array ............................................................................................... 137 Section 39.11: Reads file (/etc/passwd) line by line and field by field ................................................................. 138 Chapter 40: File execution sequence ............................................................................................................. 140 Section 40.1: .profile vs .bash_profile (and .bash_login) ....................................................................................... 140 Chapter 41: Splitting Files .................................................................................................................................... 141 Section 41.1: Split a file ............................................................................................................................................... 141 Chapter 42: File Transfer using scp ................................................................................................................ 142 Section 42.1: scp transferring file ............................................................................................................................. 142 Section 42.2: scp transferring multiple files ........................................................................................................... 142 Section 42.3: Downloading file using scp ................................................................................................................ 142 Chapter 43: Pipelines .............................................................................................................................................. 143 Section 43.1: Using |& ................................................................................................................................................. 143 Section 43.2: Show all processes paginated .......................................................................................................... 144 Section 43.3: Modify continuous output of a command ....................................................................................... 144 Chapter 44: Managing PATH environment variable ............................................................................. 145 Section 44.1: Add a path to the PATH environment variable ................................................................................ 145 Section 44.2: Remove a path from the PATH environment variable .................................................................. 145 Chapter 45: Word splitting .................................................................................................................................. 147 Section 45.1: What, when and Why? ........................................................................................................................ 147 Section 45.2: Bad eects of word splitting ............................................................................................................. 147 Section 45.3: Usefulness of word splitting .............................................................................................................. 148 Section 45.4: Splitting by separator changes ......................................................................................................... 149 Section 45.5: Splitting with IFS .................................................................................................................................. 149 Section 45.6: IFS & word splitting ............................................................................................................................. 149 Chapter 46: Avoiding date using printf ........................................................................................................ 151 Section 46.1: Get the current date ............................................................................................................................ 151 Section 46.2: Set variable to current time .............................................................................................................. 151 Chapter 47: Using "trap" to react to signals and system events ................................................... 152 Section 47.1: Introduction: clean up temporary files .............................................................................................. 152 Section 47.2: Catching SIGINT or Ctl+C ................................................................................................................... 152 Section 47.3: Accumulate a list of trap work to run at exit ................................................................................... 153 Section 47.4: Killing Child Processes on Exit ........................................................................................................... 153 Section 47.5: react on change of terminals window size ...................................................................................... 153 Chapter 48: Chain of commands and operations ................................................................................... 155 Section 48.1: Counting a text pattern ocurrence .................................................................................................... 155 Section 48.2: transfer root cmd output to user file ............................................................................................... 155 Section 48.3: logical chaining of commands with && and || ................................................................................. 155 Section 48.4: serial chaining of commands with semicolon ................................................................................. 155 Section 48.5: chaining commands with | ................................................................................................................. 156 Chapter 49: Type of Shells .................................................................................................................................. 157 Section 49.1: Start an interactive shell ..................................................................................................................... 157 Section 49.2: Detect type of shell ............................................................................................................................ 157 Section 49.3: Introduction to dot files ...................................................................................................................... 157 Chapter 50: Color script output (cross-platform) ................................................................................... 159 Section 50.1: color-output.sh .................................................................................................................................... 159 Chapter 51: co-processes ..................................................................................................................................... 160 Section 51.1: Hello World ........................................................................................................................................... 160 Chapter 52: Typing variables ............................................................................................................................. 161 Section 52.1: declare weakly typed variables ........................................................................................................ 161 Chapter 53: Jobs at specific times .................................................................................................................. 162 Section 53.1: Execute job once at specific time ...................................................................................................... 162 Section 53.2: Doing jobs at specified times repeatedly using systemd.timer .................................................... 162 Chapter 54: Handling the system prompt .................................................................................................. 164 Section 54.1: Using the PROMPT_COMMAND envrionment variable .................................................................. 164 Section 54.2: Using PS2 ............................................................................................................................................ 165 Section 54.3: Using PS3 ............................................................................................................................................. 165 Section 54.4: Using PS4 ............................................................................................................................................ 165 Section 54.5: Using PS1 ............................................................................................................................................. 166 Chapter 55: The cut command .......................................................................................................................... 167 Section 55.1: Only one delimiter character ............................................................................................................. 167 Section 55.2: Repeated delimiters are interpreted as empty fields .................................................................... 167 Section 55.3: No quoting ........................................................................................................................................... 167 Section 55.4: Extracting, not manipulating ............................................................................................................. 167 Chapter 56: Bash on Windows 10 ...................................................................................................................... 169 Section 56.1: Readme ................................................................................................................................................ 169 Chapter 57: Cut Command .................................................................................................................................. 170 Section 57.1: Show the first column of a file ........................................................................................................... 170 Section 57.2: Show columns x to y of a file ............................................................................................................ 170 Chapter 58: global and local variables ......................................................................................................... 171 Section 58.1: Global variables ................................................................................................................................... 171 Section 58.2: Local variables .................................................................................................................................... 171 Section 58.3: Mixing the two together ..................................................................................................................... 171 Chapter 59: CGI Scripts .......................................................................................................................................... 173 Section 59.1: Request Method: GET ......................................................................................................................... 173 Section 59.2: Request Method: POST /w JSON ..................................................................................................... 175 Chapter 60: Select keyword ................................................................................................................................ 177 Section 60.1: Select keyword can be used for getting input argument in a menu format ................................ 177 Chapter 61: When to use eval ............................................................................................................................. 178 Section 61.1: Using Eval ............................................................................................................................................. 178 Section 61.2: Using Eval with Getopt ........................................................................................................................ 179 Chapter 62: Networking With Bash ................................................................................................................. 180 Section 62.1: Networking commands ...................................................................................................................... 180 Chapter 63: Parallel ................................................................................................................................................. 182 Section 63.1: Parallelize repetitive tasks on list of files .......................................................................................... 182 Section 63.2: Parallelize STDIN ................................................................................................................................ 183 Chapter 64: Decoding URL .................................................................................................................................. 184 Section 64.1: Simple example ................................................................................................................................... 184 Section 64.2: Using printf to decode a string ......................................................................................................... 184 Chapter 65: Design Patterns ............................................................................................................................... 185 Section 65.1: The Publish/Subscribe (Pub/Sub) Pattern ....................................................................................... 185 Chapter 66: Pitfalls .................................................................................................................................................. 187 Section 66.1: Whitespace When Assigning Variables ............................................................................................ 187 Section 66.2: Failed commands do not stop script execution .............................................................................. 187 Section 66.3: Missing The Last Line in a File ........................................................................................................... 187 Appendix A: Keyboard shortcuts ...................................................................................................................... 189 Section A.1: Editing Shortcuts ................................................................................................................................... 189 Section A.2: Recall Shortcuts ................................................................................................................................... 189 Section A.3: Macros ................................................................................................................................................... 189 Section A.4: Custome Key Bindings ........................................................................................................................ 189 Section A.5: Job Control ........................................................................................................................................... 190 Credits ............................................................................................................................................................................ 191 You may also like ...................................................................................................................................................... 195 About Please feel free to share this PDF with anyone for free, latest version of this book can be downloaded from: https://goalkicker.com/BashBook This Bash Notes for Professionals book is compiled from Stack Overflow Documentation, the content is written by the beautiful people at Stack Overflow. Text content is released under Creative Commons BY-SA, see credits at the end of this book whom contributed to the various chapters. Images may be copyright of their respective owners unless otherwise specified This is an unofficial free book created for educational purposes and is not affiliated with official Bash group(s) or company(s) nor Stack Overflow. All trademarks and registered trademarks are the property of their respective company owners The information presented in this book is not guaranteed to be correct nor accurate, use at your own risk Please send feedback and corrections to [email protected] GoalKicker.com – Bash Notes for Professionals 1 Chapter 1: Getting started with Bash Version Release Date 0.99 1989-06-08 1.01 1989-06-23 2.0 1996-12-31 2.02 1998-04-20 2.03 1999-02-19 2.04 2001-03-21 2.05b 2002-07-17 3.0 2004-08-03 3.1 2005-12-08 3.2 2006-10-11 4.0 2009-02-20 4.1 2009-12-31 4.2 2011-02-13 4.3 2014-02-26 4.4 2016-09-15 Section 1.1: Hello World Interactive Shell The Bash shell is commonly used interactively: It lets you enter and edit commands, then executes them when Return you press the key. Many Unix-based and Unix-like operating systems use Bash as their default shell (notably Linux and macOS). The terminal automatically enters an interactive Bash shell process on startup. Output Hello World by typing the following: echo "Hello World" #> Hello World # Output Example Notes You can change the shell by just typing the name of the shell in terminal. For example: sh, bash, etc. echo is a Bash builtin command that writes the arguments it receives to the standard output. It appends a newline to the output, by default. Non-Interactive Shell The Bash shell can also be run non-interactively from a script, making the shell require no human interaction. Interactive behavior and scripted behavior should be identical – an important design consideration of Unix V7 Bourne shell and transitively Bash. Therefore anything that can be done at the command line can be put in a script file for reuse. Follow these steps to create a Hello World script: 1. Create a new file called hello-world.sh GoalKicker.com – Bash Notes for Professionals 2 touch hello-world.sh 2. Make the script executable by running chmod +x hello-world.sh 3. Add this code: #!/bin/bash echo "Hello World" Line 1: The first line of the script must start with the character sequence #!, referred to as shebang1. The shebang instructs the operating system to run /bin/bash, the Bash shell, passing it the script's path as an argument. E.g. /bin/bash hello-world.sh Line 2: Uses the echo command to write Hello World to the standard output. 4. Execute the hello-world.sh script from the command line using one of the following: ./hello-world.sh – most commonly used, and recommended /bin/bash hello-world.sh bash hello-world.sh – assuming /bin is in your $PATH sh hello-world.sh For real production use, you would omit the .sh extension (which is misleading anyway, since this is a Bash script, not a sh script) and perhaps move the file to a directory within your PATH so that it is available to you regardless of your current working directory, just like a system command such as cat or ls. Common mistakes include: 1. Forgetting to apply execute permission on the file, i.e., chmod +x hello-world.sh, resulting in the output of ./hello-world.sh: Permission denied. 2. Editing the script on Windows, which produces incorrect line ending characters that Bash cannot handle. A common symptom is : command not found where the carriage return has forced the cursor to the beginning of line, overwriting the text before the colon in the error message. The script can be fixed using the dos2unix program. An example use: dos2unix hello-world.sh dos2unix edits the file inline. 3. Using sh ./hello-world.sh, not realizing that bash and sh are distinct shells with distinct features (though since Bash is backwards-compatible, the opposite mistake is harmless). Anyway, simply relying on the script's shebang line is vastly preferable to explicitly writing bash or sh (or python or perl or awk or ruby or...) before each script's file name. A common shebang line to use in order to make your script more portable is to use #!/usr/bin/env bash instead of hard-coding a path to Bash. That way, /usr/bin/env has to exist, but beyond that point, bash just GoalKicker.com – Bash Notes for Professionals 3 needs to be on your PATH. On many systems, /bin/bash doesn't exist, and you should use /usr/local/bin/bash or some other absolute path; this change avoids having to figure out the details of that. 1 Also referred to as sha-bang, hashbang, pound-bang, hash-pling. Section 1.2: Hello World Using Variables Create a new file called hello.sh with the following content and give it executable permissions with chmod +x hello.sh. Execute/Run via: ./hello.sh #!/usr/bin/env bash # Note that spaces cannot be used around the `=` assignment operator whom_variable="World" # Use printf to safely output the data printf "Hello, %s\n" "$whom_variable" #> Hello, World This will print Hello, World to standard output when executed. To tell bash where the script is you need to be very specific, by pointing it to the containing directory, normally with ./ if it is your working directory, where . is an alias to the current directory. If you do not specify the directory, bash tries to locate the script in one of the directories contained in the $PATH environment variable. The following code accepts an argument $1, which is the first command line argument, and outputs it in a formatted string, following Hello,. Execute/Run via: ./hello.sh World #!/usr/bin/env bash printf "Hello, %s\n" "$1" #> Hello, World It is important to note that $1 has to be quoted in double quote, not single quote. "$1" expands to the first command line argument, as desired, while '$1' evaluates to literal string $1. Security Note: Read Security implications of forgetting to quote a variable in bash shells to understand the importance of placing the variable text within double quotes. Section 1.3: Hello World with User Input The following will prompt a user for input, and then store that input as a string (text) in a variable. The variable is then used to give a message to the user. GoalKicker.com – Bash Notes for Professionals 4 #!/usr/bin/env bash echo "Who are you?" read name echo "Hello, $name." The command read here reads one line of data from standard input into the variable name. This is then referenced using $name and printed to standard out using echo. Example output: $ ./hello_world.sh Who are you? Matt Hello, Matt. Here the user entered the name "Matt", and this code was used to say Hello, Matt.. And if you want to append something to the variable value while printing it, use curly brackets around the variable name as shown in the following example: #!/usr/bin/env bash echo "What are you doing?" read action echo "You are ${action}ing." Example output: $ ./hello_world.sh What are you doing? Sleep You are Sleeping. Here when user enters an action, "ing" is appended to that action while printing. Section 1.4: Importance of Quoting in Strings Quoting is important for string expansion in bash. With these, you can control how the bash parses and expands your strings. There are two types of quoting: Weak: uses double quotes: " Strong: uses single quotes: ' If you want to bash to expand your argument, you can use Weak Quoting: #!/usr/bin/env bash world="World" echo "Hello $world" #> Hello World If you don't want to bash to expand your argument, you can use Strong Quoting: #!/usr/bin/env bash world="World" echo 'Hello $world' GoalKicker.com – Bash Notes for Professionals 5 #> Hello $world You can also use escape to prevent expansion: #!/usr/bin/env bash world="World" echo "Hello \$world" #> Hello $world For more detailed information other than beginner details, you can continue to read it here. Section 1.5: Viewing information for Bash built-ins help This will display the Bash help (manual) page for the specified built-in. For example, help unset will show: unset: unset [-f] [-v] [-n] [name ...] Unset values and attributes of shell variables and functions. For each NAME, remove the corresponding variable or function. Options: -f treat each NAME as a shell function -v treat each NAME as a shell variable -n treat each NAME as a name reference and unset the variable itself rather than the variable it references Without options, unset first tries to unset a variable, and if that fails, tries to unset a function. Some variables cannot be unset; also see `readonly'. Exit Status: Returns success unless an invalid option is given or a NAME is read-only. To see a list of all built-ins with a short description, use help -d Section 1.6: Hello World in "Debug" mode $ cat hello.sh #!/bin/bash echo "Hello World" $ bash -x hello.sh + echo Hello World Hello World The -x argument enables you to walk through each line in the script. One good example is here: $ cat hello.sh #!/bin/bash echo "Hello World\n" GoalKicker.com – Bash Notes for Professionals 6 adding_string_to_number="s" v=$(expr 5 + $adding_string_to_number) $ ./hello.sh Hello World expr: non-integer argument The above prompted error is not enough to trace the script; however, using the following way gives you a better sense where to look for the error in the script. $ bash -x hello.sh + echo Hello World\n Hello World + adding_string_to_number=s + expr 5 + s expr: non-integer argument + v= Section 1.7: Handling Named Arguments #!/bin/bash deploy=false uglify=false while (( $# > 1 )); do case $1 in --deploy) deploy="$2";; --uglify) uglify="$2";; *) break; esac; shift 2 done $deploy && echo "will deploy... deploy = $deploy" $uglify && echo "will uglify... uglify = $uglify" # how to run # chmod +x script.sh # ./script.sh --deploy true --uglify false GoalKicker.com – Bash Notes for Professionals 7 Chapter 2: Script shebang Section 2.1: Env shebang To execute a script file with the bash executable found in the PATH environment variable by using the executable env, the first line of a script file must indicate the absolute path to the env executable with the argument bash: #!/usr/bin/env bash The env path in the shebang is resolved and used only if a script is directly launch like this: script.sh The script must have execution permission. The shebang is ignored when a bash interpreter is explicitly indicated to execute a script: bash script.sh Section 2.2: Direct shebang To execute a script file with the bash interpreter, the first line of a script file must indicate the absolute path to the bash executable to use: #!/bin/bash The bash path in the shebang is resolved and used only if a script is directly launch like this: ./script.sh The script must have execution permission. The shebang is ignored when a bash interpreter is explicitly indicated to execute a script: bash script.sh Section 2.3: Other shebangs There are two kinds of programs the kernel knows of. A binary program is identified by it's ELF (ExtenableLoadableFormat) header, which is usually produced by a compiler. The second one are scripts of any kind. If a file starts in the very first line with the sequence #! then the next string has to be a pathname of an interpreter. If the kernel reads this line, it calls the interpreter named by this pathname and gives all of the following words in this line as arguments to the interpreter. If there is no file named "something" or "wrong": #!/bin/bash something wrong echo "This line never gets printed" bash tries to execute its argument "something wrong" which doesn't exist. The name of the script file is added too. To see this clearly use an echo shebang: GoalKicker.com – Bash Notes for Professionals 8 #"/bin/echo something wrong # and now call this script named "thisscript" like so: # thisscript one two # the output will be: something wrong ./thisscript one two Some programs like awk use this technique to run longer scripts residing in a disk file. GoalKicker.com – Bash Notes for Professionals 9 Chapter 3: Navigating directories Section 3.1: Absolute vs relative directories To change to an absolutely specified directory, use the entire name, starting with a slash /, thus: cd /home/username/project/abc If you want to change to a directory near your current on, you can specify a relative location. For example, if you are already in /home/username/project, you can enter the subdirectory abc thus: cd abc If you want to go to the directory above the current directory, you can use the alias ... For example, if you were in /home/username/project/abc and wanted to go to /home/username/project, then you would do the following: cd .. This may also be called going "up" a directory. Section 3.2: Change to the last directory For the current shell, this takes you to the previous directory that you were in, no matter where it was. cd - Doing it multiple times effectively "toggles" you being in the current directory or the previous one. Section 3.3: Change to the home directory The default directory is the home directory ($HOME, typically /home/username), so cd without any directory takes you there cd Or you could be more explicit: cd $HOME A shortcut for the home directory is ~, so that could be used as well. cd ~ Section 3.4: Change to the Directory of the Script In general, there are two types of Bash scripts: 1. System tools which operate from the current working directory 2. Project tools which modify files relative to their own place in the files system For the second type of scripts, it is useful to change to the directory where the script is stored. This can be done with the following command: GoalKicker.com – Bash Notes for Professionals 10 cd "$(dirname "$(readlink -f "$0")")" This command runs 3 commands: 1. readlink -f "$0" determines the path to the current script ($0) 2. dirname converts the path to script to the path to its directory 3. cd changes the current work directory to the directory it receives from dirname GoalKicker.com – Bash Notes for Professionals 11 Chapter 4: Listing Files Option Description -a, --all List all entries including ones that start with a dot -A, --almost-all List all entries excluding . and .. -c Sort files by change time -d, --directory List directory entries -h, --human-readable Show sizes in human readable format (i.e. K, M) -H Same as above only with powers of 1000 instead of 1024 -l Show contents in long-listing format -o Long -listing format without group info -r, --reverse Show contents in reverse order -s, --size Print size of each file in blocks -S Sort by file size --sort=WORD Sort contents by a word. (i.e size, version, status) -t Sort by modification time -u Sort by last access time -v Sort by version -1 List one file per line Section 4.1: List Files in a Long Listing Format The ls command's -l option prints a specified directory's contents in a long listing format. If no directory is specified then, by default, the contents of the current directory are listed. ls -l /etc Example Output: total 1204 drwxr-xr-x 3 root root 4096 Apr 21 03:44 acpi -rw-r--r-- 1 root root 3028 Apr 21 03:38 adduser.conf drwxr-xr-x 2 root root 4096 Jun 11 20:42 alternatives ... The output first displays total, which indicates the total size in blocks of all the files in the listed directory. It then displays eight columns of information for each file in the listed directory. Below are the details for each column in the output: Column No. Example Description 1.1 d File type (see table below) 1.2 rwxr-xr-x Permission string 2 3 Number of hard links 3 root Owner name 4 root Owner group 5 4096 File size in bytes 6 Apr 21 03:44 Modification time 7 acpi File name GoalKicker.com – Bash Notes for Professionals 12 File Type The file type can be one of any of the following characters. Character File Type - Regular file b Block special file c Character special file C High performance ("contiguous data") file d Directory D Door (special IPC file in Solaris 2.5+ only) l Symbolic link M Off-line ("migrated") file (Cray DMF) n Network special file (HP-UX) p FIFO (named pipe) P Port (special system file in Solaris 10+ only) s Socket ? Some other file type Section 4.2: List the Ten Most Recently Modified Files The following will list up to ten of the most recently modified files in the current directory, using a long listing format (-l) and sorted by time (-t). ls -lt | head Section 4.3: List All Files Including Dotfiles A dotfile is a file whose names begin with a .. These are normally hidden by ls and not listed unless requested. For example the following output of ls: $ ls bin pki The -a or --all option will list all files, including dotfiles. $ ls -a . .ansible .bash_logout .bashrc .lesshst .puppetlabs .viminfo .. .bash_history .bash_profile bin pki .ssh The -A or --almost-all option will list all files, including dotfiles, but does not list implied . and ... Note that . is the current directory and .. is the parent directory. $ ls -A .ansible .bash_logout .bashrc .lesshst .puppetlabs .viminfo .bash_history .bash_profile bin pki .ssh Section 4.4: List Files Without Using `ls` Use the Bash shell's filename expansion and brace expansion capabilities to obtain the filenames: GoalKicker.com – Bash Notes for Professionals 13 # display the files and directories that are in the current directory printf "%s\n" * # display only the directories in the current directory printf "%s\n" */ # display only (some) image files printf "%s\n" *.{gif,jpg,png} To capture a list of files into a variable for processing, it is typically good practice to use a bash array: files=( * ) # iterate over them for file in "${files[@]}"; do echo "$file" done Section 4.5: List Files The ls command lists the contents of a specified directory, excluding dotfiles. If no directory is specified then, by default, the contents of the current directory are listed. Listed files are sorted alphabetically, by default, and aligned in columns if they don’t fit on one line. $ ls apt configs Documents Fonts Music Programming Templates workspace bin Desktop eclipse git Pictures Public Videos Section 4.6: List Files in a Tree-Like Format The tree command lists the contents of a specified directory in a tree-like format. If no directory is specified then, by default, the contents of the current directory are listed. Example Output: $ tree /tmp /tmp ├── 5037 ├── adb.log └── evince-20965 └── image.FPWTJY.png Use the tree command's -L option to limit the display depth and the -d option to only list directories. Example Output: $ tree -L 1 -d /tmp /tmp └── evince-20965 Section 4.7: List Files Sorted by Size The ls command's -S option sorts the files in descending order of file size. GoalKicker.com – Bash Notes for Professionals 14 $ ls -l -S ./Fruits total 444 -rw-rw-rw- 1 root root 295303 Jul 28 19:19 apples.jpg -rw-rw-rw- 1 root root 102283 Jul 28 19:19 kiwis.jpg -rw-rw-rw- 1 root root 50197 Jul 28 19:19 bananas.jpg When used with the -r option the sort order is reversed. $ ls -l -S -r /Fruits total 444 -rw-rw-rw- 1 root root 50197 Jul 28 19:19 bananas.jpg -rw-rw-rw- 1 root root 102283 Jul 28 19:19 kiwis.jpg -rw-rw-rw- 1 root root 295303 Jul 28 19:19 apples.jpg GoalKicker.com – Bash Notes for Professionals 15 Chapter 5: Using cat Option Details -n Print line numbers -v Show non-printing characters using ^ and M- notation except LFD and TAB -T Show TAB characters as ^I -E Show linefeed(LF) characters as $ -e Same as -vE -b Number nonempty output lines, overrides -n -A equivalent to -vET -s suppress repeated empty output lines, s refers to squeeze Section 5.1: Concatenate files This is the primary purpose of cat. cat file1 file2 file3 > file_all cat can also be used similarly to concatenate files as part of a pipeline, e.g. cat file1 file2 file3 | grep foo Section 5.2: Printing the Contents of a File cat file.txt will print the contents of a file. If the file contains non-ASCII characters, you can display those characters symbolically with cat -v. This can be quite useful for situations where control characters would otherwise be invisible. cat -v unicode.txt Very often, for interactive use, you are better off using an interactive pager like less or more, though. (less is far more powerful than more and it is advised to use less more often than more.) less file.txt To pass the contents of a file as input to a command. An approach usually seen as better (UUOC) is to use redirection. tr A-Z a-z file It will let you write the text on terminal which will be saved in a file named file. cat >>file will do the same, except it will append the text to the end of the file. Ctrl+D N.B: to end writing text on terminal (Linux) A here document can be used to inline the contents of a file into a command line or a script: cat <file Hello, World. END The token after the << redirection symbol is an arbitrary string which needs to occur alone on a line (with no leading or trailing whitespace) to indicate the end of the here document. You can add quoting to prevent the shell from performing command substitution and variable interpolation: cat <<'fnord' Nothing in `here` will be $changed fnord (Without the quotes, here would be executed as a command, and $changed would be substituted with the value of the variable changed -- or nothing, if it was undefined.) Section 5.4: Show non printable characters This is useful to see if there are any non-printable characters, or non-ASCII characters. e.g. If you have copy-pasted the code from web, you may have quotes like ” instead of standard ". $ cat -v file.txt $ cat -vE file.txt # Useful in detecting trailing spaces. e.g. $ echo '” ' | cat -vE # echo | will be replaced by actual file. M-bM-^@M-^] $ You may also want to use cat -A (A for All) that is equivalent to cat -vET. It will display TAB characters (displayed GoalKicker.com – Bash Notes for Professionals 17 as ^I), non printable characters and end of each line: $ echo '” `' | cat -A M-bM-^@M-^]^I`$ Section 5.5: Read from standard input cat < file.txt Output is same as cat file.txt, but it reads the contents of the file from standard input instead of directly from the file. printf "first line\nSecond line\n" | cat -n The echo command before | outputs two lines. The cat command acts on the output to add line numbers. Section 5.6: Display line numbers with output Use the --number flag to print line numbers before each line. Alternatively, -n does the same thing. $ cat --number file 1 line 1 2 line 2 3 4 line 4 5 line 5 To skip empty lines when counting lines, use the --number-nonblank, or simply -b. $ cat -b file 1 line 1 2 line 2 3 line 4 4 line 5 Section 5.7: Concatenate gzipped files Files compressed by gzip can be directly concatenated into larger gzipped files. cat file1.gz file2.gz file3.gz > combined.gz This is a property of gzip that is less efficient than concatenating the input files and gzipping the result: cat file1 file2 file3 | gzip > combined.gz A complete demonstration: echo 'Hello world!' > hello.txt echo 'Howdy world!' > howdy.txt gzip hello.txt gzip howdy.txt GoalKicker.com – Bash Notes for Professionals 18 cat hello.txt.gz howdy.txt.gz > greetings.txt.gz gunzip greetings.txt.gz cat greetings.txt Which results in Hello world! Howdy world! Notice that greetings.txt.gz is a single file and is decompressed as the single file greeting.txt. Contrast this with tar -czf hello.txt howdy.txt > greetings.tar.gz, which keeps the files separate inside the tarball. GoalKicker.com – Bash Notes for Professionals 19 Chapter 6: Grep Section 6.1: How to search a file for a pattern To find the word foo in the file bar : grep foo ~/Desktop/bar To find all lines that do not contain foo in the file bar : grep –v foo ~/Desktop/bar To use find all words containing foo in the end (WIldcard Expansion): grep "*foo" ~/Desktop/bar GoalKicker.com – Bash Notes for Professionals 20 Chapter 7: Aliasing Shell aliases are a simple way to create new commands or to wrap existing commands with code of your own. They somewhat overlap with shell functions, which are however more versatile and should therefore often be preferred. Section 7.1: Bypass an alias Sometimes you may want to bypass an alias temporarily, without disabling it. To work with a concrete example, consider this alias: alias ls='ls --color=auto' And let's say you want to use the ls command without disabling the alias. You have several options: Use the command builtin: command ls Use the full path of the command: /bin/ls Add a \ anywhere in the command name, for example: \ls, or l\s Quote the command: "ls" or 'ls' Section 7.2: Create an Alias alias word='command' Invoking word will run command. Any arguments supplied to the alias are simply appended to the target of the alias: alias myAlias='some command --with --options' myAlias foo bar baz The shell will then execute: some command --with --options foo bar baz To include multiple commands in the same alias, you can string them together with &&. For example: alias print_things='echo "foo" && echo "bar" && echo "baz"' Section 7.3: Remove an alias To remove an existing alias, use: unalias {alias_name} Example: # create an alias $ alias now='date' # preview the alias $ now Thu Jul 21 17:11:25 CEST 2016 # remove the alias $ unalias now GoalKicker.com – Bash Notes for Professionals 21 # test if removed $ now -bash: now: command not found Section 7.4: The BASH_ALIASES is an internal bash assoc array Aliases are named shortcuts of commands, one can define and use in interactive bash instances. They are held in an associative array named BASH_ALIASES. To use this var in a script, it must be run within an interactive shell #!/bin/bash -li # note the -li above! -l makes this behave like a login shell # -i makes it behave like an interactive shell # # shopt -s expand_aliases will not work in most cases echo There are ${#BASH_ALIASES[*]} aliases defined. for ali in "${!BASH_ALIASES[@]}"; do printf "alias: %-10s triggers: %s\n" "$ali" "${BASH_ALIASES[$ali]}" done Section 7.5: Expand alias Assuming that bar is an alias for someCommand -flag1. Ctrl + alt + e Type bar on the command line and then press you'll get someCommand -flag1 where bar was standing. Section 7.6: List all Aliases alias -p will list all the current aliases. GoalKicker.com – Bash Notes for Professionals 22 Chapter 8: Jobs and Processes Section 8.1: Job handling Creating jobs To create an job, just append a single & after the command: $ sleep 10 & [1] 20024 Ctrl z You can also make a running process a job by pressing + : $ sleep 10 ^Z [1]+ Stopped sleep 10 Background and foreground a process To bring the Process to the foreground, the command fg is used together with % $ sleep 10 & [1] 20024 $ fg %1 sleep 10 Now you can interact with the process. To bring it back to the background you can use the bg command. Due to the Ctrl z occupied terminal session, you need to stop the process first by pressing + . $ sleep 10 ^Z [1]+ Stopped sleep 10 $ bg %1 [1]+ sleep 10 & Due to the laziness of some Programmers, all these commands also work with a single % if there is only one process, or for the first process in the list. For Example: $ sleep 10 & [1] 20024 $ fg % # to bring a process to foreground 'fg %' is also working. sleep 10 or just $ % # laziness knows no boundaries, '%' is also working. sleep 10 Additionally, just typing fg or bg without any argument handles the last job: $ sleep 20 & $ sleep 10 & $ fg GoalKicker.com – Bash Notes for Professionals 23 sleep 10 ^C $ fg sleep 20 Killing running jobs $ sleep 10 & [1] 20024 $ kill %1 [1]+ Terminated sleep 10 The sleep process runs in the background with process id (pid) 20024 and job number 1. In order to reference the process, you can use either the pid or the job number. If you use the job number, you must prefix it with %. The default kill signal sent by kill is SIGTERM, which allows the target process to exit gracefully. Some common kill signals are shown below. To see a full list, run kill -l. Signal name Signal value Effect SIGHUP 1 Hangup SIGINT 2 Interrupt from keyboard SIGKILL 9 Kill signal SIGTERM 15 Termination signal Start and kill specific processes Probably the easiest way of killing a running process is by selecting it through the process name as in the following example using pkill command as pkill -f test.py (or) a more fool-proof way using pgrep to search for the actual process-id kill $(pgrep -f 'python test.py') The same result can be obtained using grep over ps -ef | grep name_of_process then killing the process associated with the resulting pid (process id). Selecting a process using its name is convinient in a testing environment but can be really dangerous when the script is used in production: it is virtually impossible to be sure that the name will match the process you actually want to kill. In those cases, the following approach is actually much safe. Start the script that will eventually killed with the following approach. Let's assume that the command you want to execute and eventually kill is python test.py. #!/bin/bash if [[ ! -e /tmp/test.py.pid ]]; then # Check if the file already exists python test.py & #+and if so do not run another process. echo $! > /tmp/test.py.pid else echo -n "ERROR: The process is already running with pid " cat /tmp/test.py.pid echo fi This will create a file in the /tmp directory containing the pid of the python test.py process. If the file already exists, we assume that the command is already running and the script return an error. GoalKicker.com – Bash Notes for Professionals 24 Then, when you want to kill it use the following script: #!/bin/bash if [[ -e /tmp/test.py.pid ]]; then # If the file do not exists, then the kill `cat /tmp/test.py.pid` #+the process is not running. Useless rm /tmp/test.py.pid #+trying to kill it. else echo "test.py is not running" fi that will kill exactly the process associated with your command, without relying on any volatile information (like the string used to run the command). Even in this case if the file does not exist, the script assume that you want to kill a non-running process. This last example can be easily improved for running the same command multiple times (appending to the pid file instead of overwriting it, for example) and to manage cases where the process dies before being killed. Section 8.2: Check which process running on specific port To check which process running on port 8080 lsof -i :8080 Section 8.3: Disowning background job $ gzip extremelylargefile.txt & $ bg $ disown %1 This allows a long running process to continue once your shell (terminal, ssh, etc) is closed. Section 8.4: List Current Jobs $ tail -f /var/log/syslog > log.txt [1]+ Stopped tail -f /var/log/syslog > log.txt $ sleep 10 & $ jobs [1]+ Stopped tail -f /var/log/syslog > log.txt [2]- Running sleep 10 & Section 8.5: Finding information about a running process ps aux | grep shows processes matching search-term Example: root@server7:~# ps aux | grep nginx root 315 0.0 0.3 144392 1020 ? Ss May28 0:00 nginx: master process /usr/sbin/nginx www-data 5647 0.0 1.1 145124 3048 ? S Jul18 2:53 nginx: worker process www-data 5648 0.0 0.1 144392 376 ? S Jul18 0:00 nginx: cache manager process root 13134 0.0 0.3 4960 920 pts/0 S+ 14:33 0:00 grep --color=auto nginx root@server7:~# GoalKicker.com – Bash Notes for Professionals 25 Here, second column is the process id. For example, if you want to kill the nginx process, you can use the command kill 5647. It is always adviced to use the kill command with SIGTERM rather than SIGKILL. Section 8.6: List all processes There are two common ways to list all processes on a system. Both list all processes running by all users, though they differ in the format they output (the reason for the differences are historical). ps -ef # lists all processes ps aux # lists all processes in alternative format (BSD) This can be used to check if a given application is running. For example, to check if the SSH server (sshd) is running: ps -ef | grep sshd GoalKicker.com – Bash Notes for Professionals 26 Chapter 9: Redirection Parameter Details internal file descriptor An integer. direction One of >, < or <> external file descriptor or path & followed by an integer for file descriptor or a path. Section 9.1: Redirecting standard output > redirect the standard output (aka STDOUT) of the current command into a file or another descriptor. These examples write the output of the ls command into the file file.txt ls >file.txt > file.txt ls The target file is created if it doesn't exists, otherwise this file is truncated. The default redirection descriptor is the standard output or 1 when none is specified. This command is equivalent to the previous examples with the standard output explicitly indicated: ls 1>file.txt Note: the redirection is initialized by the executed shell and not by the executed command, therefore it is done before the command execution. Section 9.2: Append vs Truncate Truncate > 1. Create specified file if it does not exist. 2. Truncate (remove file's content) 3. Write to file $ echo "first line" > /tmp/lines $ echo "second line" > /tmp/lines $ cat /tmp/lines second line Append >> 1. Create specified file if it does not exist. 2. Append file (writing at end of file). # Overwrite existing file $ echo "first line" > /tmp/lines # Append a second line $ echo "second line" >> /tmp/lines $ cat /tmp/lines first line second line GoalKicker.com – Bash Notes for Professionals 27 Section 9.3: Redirecting both STDOUT and STDERR File descriptors like 0 and 1 are pointers. We change what file descriptors point to with redirection. >/dev/null means 1 points to /dev/null. First we point 1 (STDOUT) to /dev/null then point 2 (STDERR) to whatever 1 points to. # STDERR is redirect to STDOUT: redirected to /dev/null, # effectually redirecting both STDERR and STDOUT to /dev/null echo 'hello' > /dev/null 2>&1 Version ≥ 4.0 This can be further shortened to the following: echo 'hello' &> /dev/null However, this form may be undesirable in production if shell compatibility is a concern as it conflicts with POSIX, introduces parsing ambiguity, and shells without this feature will misinterpret it: # Actual code echo 'hello' &> /dev/null echo 'hello' &> /dev/null 'goodbye' # Desired behavior echo 'hello' > /dev/null 2>&1 echo 'hello' 'goodbye' > /dev/null 2>&1 # Actual behavior echo 'hello' & echo 'hello' & goodbye > /dev/null NOTE: &> is known to work as desired in both Bash and Zsh. Section 9.4: Using named pipes Sometimes you may want to output something by one program and input it into another program, but can't use a standard pipe. ls -l | grep ".log" You could simply write to a temporary file: touch tempFile.txt ls -l > tempFile.txt grep ".log" < tempFile.txt This works fine for most applications, however, nobody will know what tempFile does and someone might remove it if it contains the output of ls -l in that directory. This is where a named pipe comes into play: mkfifo myPipe ls -l > myPipe grep ".log" < myPipe myPipe is technically a file (everything is in Linux), so let's do ls -l in an empty directory that we just created a pipe in: GoalKicker.com – Bash Notes for Professionals 28 mkdir pipeFolder cd pipeFolder mkfifo myPipe ls -l The output is: prw-r--r-- 1 root root 0 Jul 25 11:20 myPipe Notice the first character in the permissions, it's listed as a pipe, not a file. Now let's do something cool. Open one terminal, and make note of the directory (or create one so that cleanup is easy), and make a pipe. mkfifo myPipe Now let's put something in the pipe. echo "Hello from the other side" > myPipe You'll notice this hangs, the other side of the pipe is still closed. Let's open up the other side of the pipe and let that stuff through. Open another terminal and go to the directory that the pipe is in (or if you know it, prepend it to the pipe): cat < myPipe You'll notice that after hello from the other side is output, the program in the first terminal finishes, as does that in the second terminal. Now run the commands in reverse. Start with cat < myPipe and then echo something into it. It still works, because a program will wait until something is put into the pipe before terminating, because it knows it has to get something. Named pipes can be useful for moving information between terminals or between programs. Pipes are small. Once full, the writer blocks until some reader reads the contents, so you need to either run the reader and writer in different terminals or run one or the other in the background: ls -l /tmp > myPipe & cat < myPipe More examples using named pipes: Example 1 - all commands on the same terminal / same shell $ { ls -l && cat file3; } >mypipe & $ cat mypipe & $ cat file3 >mypipe & GoalKicker.com – Bash Notes for Professionals 29 $ cat mypipe # Output: Prints the output of ls directly on screen Mind that the variable $pipedata is not available for usage in the main terminal / main shell since the use of & invokes a subshell and $pipedata was only available in this subshell. Example 4 - all commands on the same terminal / same shell $ export pipedata $ pipedata=$(mypipe $ echo "$pipedata" #Output : Prints correctly the contents of mypipe This prints correctly the value of $pipedata variable in the main shell due to the export declaration of the variable. The main terminal/main shell is not hanging due to the invocation of a background shell (&). Section 9.5: Redirection to network addresses Version ≥ 2.04 Bash treats some paths as special and can do some network communication by writing to /dev/{udp|tcp}/host/port. Bash cannot setup a listening server, but can initiate a connection, and for TCP can read the results at least. For example, to send a simple web request one could do: exec 3&3 cat <&3 and the results of www.google.com's default web page will be printed to stdout. Similarly printf 'HI\n' >/dev/udp/192.168.1.1/6666 would send a UDP message containing HI\n to a listener on 192.168.1.1:6666 Section 9.6: Print error messages to stderr Error messages are generally included in a script for debugging purposes or for providing rich user experience. Simply writing error message like this: GoalKicker.com – Bash Notes for Professionals 30 cmd || echo 'cmd failed' may work for simple cases but it's not the usual way. In this example, the error message will pollute the actual output of the script by mixing both errors and successful output in stdout. In short, error message should go to stderr not stdout. It's pretty simple: cmd || echo 'cmd failed' >/dev/stderr Another example: if cmd; then echo 'success' else echo 'cmd failed' >/dev/stderr fi In the above example, the success message will be printed on stdout while the error message will be printed on stderr. A better way to print error message is to define a function: err(){ echo "E: $*" >>/dev/stderr } Now, when you have to print an error: err "My error message" Section 9.7: Redirecting multiple commands to the same file { echo "contents of home directory" ls ~ } > output.txt Section 9.8: Redirecting STDIN < reads from its right argument and writes to its left argument. To write a file into STDIN we should read /tmp/a_file and write into STDIN i.e 0 /tmp/list.txt $ echo "a" >> /tmp/list.txt $ echo "c" >> /tmp/list.txt $ sort < /tmp/list.txt a b c GoalKicker.com – Bash Notes for Professionals 31 Section 9.9: Redirecting STDERR 2 is STDERR. $ echo_to_stderr 2>/dev/null # echos nothing Definitions: echo_to_stderr is a command that writes "stderr" to STDERR echo_to_stderr () { echo stderr >&2 } $ echo_to_stderr stderr Section 9.10: STDIN, STDOUT and STDERR explained Commands have one input (STDIN) and two kinds of outputs, standard output (STDOUT) and standard error (STDERR). For example: STDIN root@server~# read Type some text here Standard input is used to provide input to a program. (Here we're using the read builtin to read a line from STDIN.) STDOUT root@server~# ls file file Standard output is generally used for "normal" output from a command. For example, ls lists files, so the files are sent to STDOUT. STDERR root@server~# ls anotherfile ls: cannot access 'anotherfile': No such file or directory Standard error is (as the name implies) used for error messages. Because this message is not a list of files, it is sent to STDERR. STDIN, STDOUT and STDERR are the three standard streams. They are identified to the shell by a number rather than a name: 0 = Standard in 1 = Standard out 2 = Standard error By default, STDIN is attached to the keyboard, and both STDOUT and STDERR appear in the terminal. However, we GoalKicker.com – Bash Notes for Professionals 32 can redirect either STDOUT or STDERR to whatever we need. For example, let's say that you only need the standard out and all error messages printed on standard error should be suppressed. That's when we use the descriptors 1 and 2. Redirecting STDERR to /dev/null Taking the previous example, root@server~# ls anotherfile 2>/dev/null root@server~# In this case, if there is any STDERR, it will be redirected to /dev/null (a special file which ignores anything put into it), so you won't get any error output on the shell. GoalKicker.com – Bash Notes for Professionals 33 Chapter 10: Control Structures Parameter to [ or test Details File Operators Details -e "$file" Returns true if the file exists. -d "$file" Returns true if the file exists and is a directory -f "$file" Returns true if the file exists and is a regular file -h "$file" Returns true if the file exists and is a symbolic link String Comparators Details -z "$str" True if length of string is zero -n "$str True if length of string is non-zero "$str" = "$str2"True if string $str is equal to string $str2. Not best for integers. It may work but will be inconsitent "$str" != "$str2" True if the strings are not equal Integer Comparators Details "$int1" -eq "$int2" True if the integers are equal "$int1" -ne "$int2" True if the integers are not equals "$int1" -gt "$int2" True if int1 is greater than int 2 "$int1" -ge "$int2" True if int1 is greater than or equal to int2 "$int1" -lt "$int2" True if int1 is less than int 2 "$int1" -le "$int2" True if int1 is less than or equal to int2 Section 10.1: Conditional execution of command lists How to use conditional execution of command lists Any builtin command, expression, or function, as well as any external command or script can be executed conditionally using the &&(and) and ||(or) operators. For example, this will only print the current directory if the cd command was successful. cd my_directory && pwd Likewise, this will exit if the cd command fails, preventing catastrophe: cd my_directory || exit rm -rf * When combining multiple statements in this manner, it's important to remember that (unlike many C-style languages) these operators have no precedence and are left-associative. Thus, this statement will work as expected... cd my_directory && pwd || echo "No such directory" If the cd succeeds, the && pwd executes and the current working directory name is printed. Unless pwd fails (a rarity) the || echo ... will not be executed. If the cd fails, the && pwd will be skipped and the || echo ... will run. But this will not (if you're thinking if...then...else)... GoalKicker.com – Bash Notes for Professionals 34 cd my_directory && ls || echo "No such directory" If the cd fails, the && ls is skipped and the || echo ... is executed. If the cd succeeds, the && ls is executed. If the ls succeeds, the || echo ... is ignored. (so far so good) BUT... if the ls fails, the || echo ... will also be executed. It is the ls, not the cd, that is the previous command. Why use conditional execution of command lists Conditional execution is a hair faster than if...then but its main advantage is allowing functions and scripts to exit early, or "short circuit". Unlike many languages like C where memory is explicitly allocated for structs and variables and such (and thus must be deallocated), bash handles this under the covers. In most cases, we don't have to clean up anything before leaving the function. A return statement will deallocate everything local to the function and pickup execution at the return address on the stack. Returning from functions or exiting scripts as soon as possible can thus significantly improve performance and reduce system load by avoiding the unnecessary execution of code. For example... my_function () { ### ALWAYS CHECK THE RETURN CODE # one argument required. "" evaluates to false(1) [[ "$1" ]] || return 1 # work with the argument. exit on failure do_something_with "$1" || return 1 do_something_else || return 1 # Success! no failures detected, or we wouldn't be here return 0 } Section 10.2: If statement if [[ $1 -eq 1 ]]; then echo "1 was passed in the first parameter" elif [[ $1 -gt 2 ]]; then echo "2 was not passed in the first parameter" else echo "The first parameter was not 1 and is not more than 2." fi The closing fi is necessary, but the elif and/or the else clauses can be omitted. The semicolons before then are standard syntax for combining two commands on a single line; they can be omitted only if then is moved to the next line. It's important to understand that the brackets [[ are not part of the syntax, but are treated as a command; it is the exit code from this command that is being tested. Therefore, you must always include spaces around the brackets. GoalKicker.com – Bash Notes for Professionals 35 This also means that the result of any command can be tested. If the exit code from the command is a zero, the statement is considered true. if grep "foo" bar.txt; then echo "foo was found" else echo "foo was not found" fi Mathematical expressions, when placed inside double parentheses, also return 0 or 1 in the same way, and can also be tested: if (( $1 + 5 > 91 )); then echo "$1 is greater than 86" fi You may also come across if statements with single brackets. These are defined in the POSIX standard and are guaranteed to work in all POSIX-compliant shells including Bash. The syntax is very similar to that in Bash: if [ "$1" -eq 1 ]; then echo "1 was passed in the first parameter" elif [ "$1" -gt 2 ]; then echo "2 was not passed in the first parameter" else echo "The first parameter was not 1 and is not more than 2." fi Section 10.3: Looping over an array for loop: arr=(a b c d e f) for i in "${arr[@]}";do echo "$i" done Or for ((i=0;i<${#arr[@]};i++));do echo "${arr[$i]}" done while loop: i=0 while [ $i -lt ${#arr[@]} ];do echo "${arr[$i]}" i=$(expr $i + 1) done Or i=0 while (( $i < ${#arr[@]} ));do echo "${arr[$i]}" ((i++)) GoalKicker.com – Bash Notes for Professionals 36 done Section 10.4: Using For Loop to List Iterate Over Numbers #! /bin/bash for i in {1..10}; do # {1..10} expands to "1 2 3 4 5 6 7 8 9 10" echo $i done This outputs the following: 1 2 3 4 5 6 7 8 8 10 Section 10.5: continue and break Example for continue for i in [series] do command 1 command 2 if (condition) # Condition to jump over command 3 continue # skip to the next value in "series" fi command 3 done Example for break for i in [series] do command 4 if (condition) # Condition to break the loop then command 5 # Command if the loop needs to be broken break fi command 6 # Command to run if the "condition" is never true done Section 10.6: Loop break Break multiple loop: arr=(a b c d e f) for i in "${arr[@]}";do echo "$i" GoalKicker.com – Bash Notes for Professionals 37 for j in "${arr[@]}";do echo "$j" break 2 done done Output: a a Break single loop: arr=(a b c d e f) for i in "${arr[@]}";do echo "$i" for j in "${arr[@]}";do echo "$j" break done done Output: a a b a c a d a e a f a Section 10.7: While Loop #! /bin/bash i=0 while [ $i -lt 5 ] #While i is less than 5 do echo "i is currently $i" i=$[$i+1] #Not the lack of spaces around the brackets. This makes it a not a test expression done #ends the loop Watch that there are spaces around the brackets during the test (after the while statement). These spaces are necessary. This loop outputs: i is currently 0 i is currently 1 i is currently 2 i is currently 3 GoalKicker.com – Bash Notes for Professionals 38 i is currently 4 Section 10.8: For Loop with C-style syntax The basic format of C-style for loop is: for (( variable assignment; condition; iteration process )) Notes: The assignment of the variable inside C-style for loop can contain spaces unlike the usual assignment Variables inside C-style for loop aren't preceded with $. Example: for (( i = 0; i < 10; i++ )) do echo "The iteration number is $i" done Also we can process multiple variables inside C-style for loop: for (( i = 0, j = 0; i < 10; i++, j = i * i )) do echo "The square of $i is equal to $j" done Section 10.9: Until Loop Until loop executes until condition is true i=5 until [[ i -eq 10 ]]; do #Checks if i=10 echo "i=$i" #Print the value of i i=$((i+1)) #Increment i by 1 done Output: i=5 i=6 i=7 i=8 i=9 When i reaches 10 the condition in until loop becomes true and the loop ends. Section 10.10: Switch statement with case With the case statement you can match values against one variable. The argument passed to case is expanded and try to match against each patterns. If a match is found, the commands upto ;; are executed. GoalKicker.com – Bash Notes for Professionals 39 case "$BASH_VERSION" in [34]*) echo {1..4} ;; *) seq -s" " 1 4 esac Pattern are not regular expressions but shell pattern matching (aka globs). Section 10.11: For Loop without a list-of-words parameter for arg; do echo arg=$arg done A for loop without a list of words parameter will iterate over the positional parameters instead. In other words, the above example is equivalent to this code: for arg in "$@"; do echo arg=$arg done In other words, if you catch yourself writing for i in "$@"; do ...; done, just drop the in part, and write simply for i; do ...; done. GoalKicker.com – Bash Notes for Professionals 40 Chapter 11: true, false and : commands Section 11.1: Infinite Loop while true; do echo ok done or while :; do echo ok done or until false; do echo ok done Section 11.2: Function Return function positive() { return 0 } function negative() { return 1 } Section 11.3: Code that will always/never be executed if true; then echo Always executed fi if false; then echo Never executed fi GoalKicker.com – Bash Notes for Professionals 41 Chapter 12: Arrays Section 12.1: Array Assignments List Assignment If you are familiar with Perl, C, or Java, you might think that Bash would use commas to separate array elements, however this is not the case; instead, Bash uses spaces: # Array in Perl my @array = (1, 2, 3, 4); # Array in Bash array=(1 2 3 4) Create an array with new elements: array=('first element' 'second element' 'third element') Subscript Assignment Create an array with explicit element indices: array=([3]='fourth element' [4]='fifth element') Assignment by index array[0]='first element' array[1]='second element' Assignment by name (associative array) Version ≥ 4.0 declare -A array array[first]='First element' array[second]='Second element' Dynamic Assignment Create an array from the output of other command, for example use seq to get a range from 1 to 10: array=(`seq 1 10`) Assignment from script's input arguments: array=("$@") Assignment within loops: while read -r; do #array+=("$REPLY") # Array append array[$i]="$REPLY" # Assignment by index let i++ # Increment index done < <(seq 1 10) # command substitution GoalKicker.com – Bash Notes for Professionals 42 echo ${array[@]} # output: 1 2 3 4 5 6 7 8 9 10 where $REPLY is always the current input Section 12.2: Accessing Array Elements Print element at index 0 echo "${array[0]}" Version < 4.3 Print last element using substring expansion syntax echo "${arr[@]: -1 }" Version ≥ 4.3 Print last element using subscript syntax echo "${array[-1]}" Print all elements, each quoted separately echo "${array[@]}" Print all elements as a single quoted string echo "${array[*]}" Print all elements from index 1, each quoted separately echo "${array[@]:1}" Print 3 elements from index 1, each quoted separately echo "${array[@]:1:3}" String Operations If referring to a single element, string operations are permitted: array=(zero one two) echo "${array[0]:0:3}" # gives out zer (chars at position 0, 1 and 2 in the string zero) echo "${array[0]:1:3}" # gives out ero (chars at position 1, 2 and 3 in the string zero) so ${array[$i]:N:M} gives out a string from the Nth position (starting from 0) in the string ${array[$i]} with M following chars. Section 12.3: Array Modification Change Index Initialize or update a particular element in the array array[10]="elevenths element" # because it's starting with 0 GoalKicker.com – Bash Notes for Professionals 43 Version ≥ 3.1 Append Modify array, adding elements to the end if no subscript is specified. array+=('fourth element' 'fifth element') Replace the entire array with a new parameter list. array=("${array[@]}" "fourth element" "fifth element") Add an element at the beginning: array=("new element" "${array[@]}") Insert Insert an element at a given index: arr=(a b c d) # insert an element at index 2 i=2 arr=("${arr[@]:0:$i}" 'new' "${arr[@]:$i}") echo "${arr[2]}" #output: new Delete Delete array indexes using the unset builtin: arr=(a b c) echo "${arr[@]}" # outputs: a b c echo "${!arr[@]}" # outputs: 0 1 2 unset -v 'arr[1]' echo "${arr[@]}" # outputs: a c echo "${!arr[@]}" # outputs: 0 2 Merge array3=("${array1[@]}" "${array2[@]}") This works for sparse arrays as well. Re-indexing an array This can be useful if elements have been removed from an array, or if you're unsure whether there are gaps in the array. To recreate the indices without gaps: array=("${array[@]}") Section 12.4: Array Iteration Array iteration comes in two flavors, foreach and the classic for-loop: a=(1 2 3 4) # foreach loop GoalKicker.com – Bash Notes for Professionals 44 for y in "${a[@]}"; do # act on $y echo "$y" done # classic for-loop for ((idx=0; idx < ${#a[@]}; ++idx)); do # act on ${a[$idx]} echo "${a[$idx]}" done You can also iterate over the output of a command: a=($(tr ',' ' ' <<<"a,b,c,d")) # tr can transform one character to another for y in "${a[@]}"; do echo "$y" done Section 12.5: Array Length ${#array[@]} gives the length of the array ${array[@]}: array=('first element' 'second element' 'third element') echo "${#array[@]}" # gives out a length of 3 This works also with Strings in single elements: echo "${#array[0]}" # gives out the lenght of the string at element 0: 13 Section 12.6: Associative Arrays Version ≥ 4.0 Declare an associative array declare -A aa Declaring an associative array before initialization or use is mandatory. Initialize elements You can initialize elements one at a time as follows: aa[hello]=world aa[ab]=cd aa["key with space"]="hello world" You can also initialize an entire associative array in a single statement: aa=([hello]=world [ab]=cd ["key with space"]="hello world") Access an associative array element echo ${aa[hello]} # Out: world Listing associative array keys GoalKicker.com – Bash Notes for Professionals 45 echo "${!aa[@]}" #Out: hello ab key with space Listing associative array values echo "${aa[@]}" #Out: world cd hello world Iterate over associative array keys and values for key in "${!aa[@]}"; do echo "Key: ${key}" echo "Value: ${array[$key]}" done # Out: # Key: hello # Value: world # Key: ab # Value: cd # Key: key with space # Value: hello world Count associative array elements echo "${#aa[@]}" # Out: 3 Section 12.7: Looping through an array Our example array: arr=(a b c d e f) Using a for..in loop: for i in "${arr[@]}"; do echo "$i" done Version ≥ 2.04 Using C-style for loop: for ((i=0;i<${#arr[@]};i++)); do echo "${arr[$i]}" done Using while loop: i=0 while [ $i -lt ${#arr[@]} ]; do echo "${arr[$i]}" i=$((i + 1)) done Version ≥ 2.04 GoalKicker.com – Bash Notes for Professionals 46 Using while loop with numerical conditional: i=0 while (( $i < ${#arr[@]} )); do echo "${arr[$i]}" ((i++)) done Using an until loop: i=0 until [ $i -ge ${#arr[@]} ]; do echo "${arr[$i]}" i=$((i + 1)) done Version ≥ 2.04 Using an until loop with numerical conditional: i=0 until (( $i >= ${#arr[@]} )); do echo "${arr[$i]}" ((i++)) done Section 12.8: Destroy, Delete, or Unset an Array To destroy, delete, or unset an array: unset array To destroy, delete, or unset a single array element: unset array[10] Section 12.9: Array from string stringVar="Apple Orange Banana Mango" arrayVar=(${stringVar// / }) Each space in the string denotes a new item in the resulting array. echo ${arrayVar[0]} # will print Apple echo ${arrayVar[3]} # will print Mango Similarly, other characters can be used for the delimiter. stringVar="Apple+Orange+Banana+Mango" arrayVar=(${stringVar//+/ }) echo ${arrayVar[0]} # will print Apple echo ${arrayVar[2]} # will print Banana Section 12.10: List of initialized indexes Get the list of inialized indexes in an array GoalKicker.com – Bash Notes for Professionals 47 $ arr[2]='second' $ arr[10]='tenth' $ arr[25]='twenty five' $ echo ${!arr[@]} 2 10 25 Section 12.11: Reading an entire file into an array Reading in a single step: IFS=$'\n' read -r -a arr < file Reading in a loop: arr=() while IFS= read -r line; do arr+=("$line") done Version ≥ 4.0 Using mapfile or readarray (which are synonymous): mapfile -t arr < file readarray -t arr < file Section 12.12: Array insert function This function will insert an element into an array at a given index: insert(){ h=' ################## insert ######################## # Usage: # insert arr_name index element # # Parameters: # arr_name : Name of the array variable # index : Index to insert at # element : Element to insert ################################################## ' [[ $1 = -h ]] && { echo "$h" >/dev/stderr; return 1; } declare -n __arr__=$1 # reference to the array variable i=$2 # index to insert at el="$3" # element to insert # handle errors [[ ! "$i" =~ ^[0-9]+$ ]] && { echo "E: insert: index must be a valid integer" >/dev/stderr; return 1; } (( $1 < 0 )) && { echo "E: insert: index can not be negative" >/dev/stderr; return 1; } # Now insert $el at $i __arr__=("${__arr__[@]:0:$i}" "$el" "${__arr__[@]:$i}") } Usage: insert array_variable_name index element GoalKicker.com – Bash Notes for Professionals 48 Example: arr=(a b c d) echo "${arr[2]}" # output: c # Now call the insert function and pass the array variable name, # index to insert at # and the element to insert insert arr 2 'New Element' # 'New Element' was inserted at index 2 in arr, now print them echo "${arr[2]}" # output: New Element echo "${arr[3]}" # output: c GoalKicker.com – Bash Notes for Professionals 49 Chapter 13: Associative arrays Section 13.1: Examining assoc arrays All needed usage shown with this snippet: #!/usr/bin/env bash declare -A assoc_array=([key_string]=value \ [one]="something" \ [two]="another thing" \ [ three ]='mind the blanks!' \ [ " four" ]='count the blanks of this key later!' \ [IMPORTANT]='SPACES DO ADD UP!!!' \ [1]='there are no integers!' \ [info]="to avoid history expansion " \ [info2]="quote exclamation mark with single quotes" \ ) echo # just a blank line echo now here are the values of assoc_array: echo ${assoc_array[@]} echo not that useful, echo # just a blank line echo this is better: declare -p assoc_array # -p == print echo have a close look at the spaces above\!\!\! echo # just a blank line echo accessing the keys echo the keys in assoc_array are ${!assoc_array[*]} echo mind the use of indirection operator \! echo # just a blank line echo now we loop over the assoc_array line by line echo note the \! indirection operator which works differently, echo if used with assoc_array. echo # just a blank line for key in "${!assoc_array[@]}"; do # accessing keys using ! indirection!!!! printf "key: \"%s\"\nvalue: \"%s\"\n\n" "$key" "${assoc_array[$key]}" done echo have a close look at the spaces in entries with keys two, three and four above\!\!\! echo # just a blank line echo # just another blank line echo there is a difference using integers as keys\!\!\! i=1 echo declaring an integer var i=1 echo # just a blank line echo Within an integer_array bash recognizes artithmetic context. echo Within an assoc_array bash DOES NOT recognize artithmetic context. echo # just a blank line echo this works: \${assoc_array[\$i]}: ${assoc_array[$i]} echo this NOT!!: \${assoc_array[i]}: ${assoc_array[i]} GoalKicker.com – Bash Notes for Professionals 50 echo # just a blank line echo # just a blank line echo an \${assoc_array[i]} has a string context within braces in contrast to an integer_array declare -i integer_array=( one two three ) echo "doing a: declare -i integer_array=( one two three )" echo # just a blank line echo both forms do work: \${integer_array[i]} : ${integer_array[i]} echo and this too: \${integer_array[\$i]} : ${integer_array[$i]} GoalKicker.com – Bash Notes for Professionals 51 Chapter 14: Functions Section 14.1: Functions with arguments In helloJohn.sh: #!/bin/bash greet() { local name="$1" echo "Hello, $name" } greet "John Doe" # running above script $ bash helloJohn.sh Hello, John Doe 1. If you don't modify the argument in any way, there is no need to copy it to a local variable - simply echo "Hello, $1". 2. You can use $1, $2, $3 and so on to access the arguments inside the function. Note: for arguments more than 9 $10 won't work (bash will read it as $10), you need to do ${10}, ${11} and so on. 3. $@ refers to all arguments of a function: #!/bin/bash foo() { echo "$@" } foo 1 2 3 # output => 1 2 3 Note: You should practically always use double quotes around "$@", like here. Omitting the quotes will cause the shell to expand wildcards (even when the user specifically quoted them in order to avoid that) and generally introduce unwelcome behavior and potentially even security problems. foo "string with spaces;" '$HOME' "*" # output => string with spaces; $HOME * 4. for default arguments use ${1:-default_val}. Eg: #!/bin/bash foo() { local val=${1:-25} echo "$val" } GoalKicker.com – Bash Notes for Professionals 52 foo # output => 25 foo 30 # output => 30 5. to require an argument use ${var:?error message} foo() { local val=${1:?Must provide an argument} echo "$val" } Section 14.2: Simple Function In helloWorld.sh #!/bin/bash # Define a function greet greet () { echo "Hello World!" } # Call the function greet greet In running the script, we see our message $ bash helloWorld.sh Hello World! Note that sourcing a file with functions makes them available in your current bash session. $ source helloWorld.sh # or, more portably, ". helloWorld.sh" $ greet Hello World! You can export a function in some shells, so that it is exposed to child processes. bash -c 'greet' # fails export -f greet # export function; note -f bash -c 'greet' # success Section 14.3: Handling flags and optional parameters The getopts builtin can be used inside functions to write functions that accommodate flags and optional parameters. This presents no special difficulty but one has to handle appropriately the values touched by getopts. As an example, we define a failwith function that writes a message on stderr and exits with code 1 or an arbitrary code supplied as parameter to the -x option: # failwith [-x STATUS] PRINTF-LIKE-ARGV # Fail with the given diagnostic message # # The -x flag can be used to convey a custom exit status, instead of # the value 1. A newline is automatically added to the output. failwith() GoalKicker.com – Bash Notes for Professionals 53 { local OPTIND OPTION OPTARG status status=1 OPTIND=1 while getopts 'x:' OPTION; do case ${OPTION} in x) status="${OPTARG}";; *) 1>&2 printf 'failwith: %s: Unsupported option.\n' "${OPTION}";; esac done shift $(( OPTIND - 1 )) { printf 'Failure: ' printf "$@" printf '\n' } 1>&2 exit "${status}" } This function can be used as follows: failwith '%s: File not found.' "${filename}" failwith -x 70 'General internal error.' and so on. Note that as for printf, variables should not be used as first argument. If the message to print consists of the content of a variable, one should use the %s specifier to print it, like in failwith '%s' "${message}" Section 14.4: Print the function definition getfunc() { declare -f "$@" } function func(){ echo "I am a sample function" } funcd="$(getfunc func)" getfunc func # or echo "$funcd" Output: func () { echo "I am a sample function" } Section 14.5: A function that accepts named parameters foo() { while [[ "$#" -gt 0 ]] GoalKicker.com – Bash Notes for Professionals 54 do case $1 in -f|--follow) local FOLLOW="following" ;; -t|--tail) local TAIL="tail=$2" ;; esac shift done echo "FOLLOW: $FOLLOW" echo "TAIL: $TAIL" } Example usage: foo -f foo -t 10 foo -f --tail 10 foo --follow --tail 10 Section 14.6: Return value from a function The return statement in Bash doesn't return a value like C-functions, instead it exits the function with a return status. You can think of it as the exit status of that function. If you want to return a value from the function then send the value to stdout like this: fun() { local var="Sample value to be returned" echo "$var" #printf "%s\n" "$var" } Now, if you do: var="$(fun)" the output of fun will be stored in $var. Section 14.7: The exit code of a function is the exit code of its last command Consider this example function to check if a host is up: is_alive() { ping -c1 "$1" &> /dev/null } This function sends a single ping to the host specified by the first function parameter. The output and error output of ping are both redirected to /dev/null, so the function will never output anything. But the ping command will have exit code 0 on success, and non-zero on failure. As this is the last (and in this example, the only) command of the function, the exit code of ping will be reused for the exit code of the function itself. GoalKicker.com – Bash Notes for Professionals 55 This fact is very useful in conditional statements. For example, if host graucho is up, then connect to it with ssh: if is_alive graucho; then ssh graucho fi Another example: repeatedly check until host graucho is up, and then connect to it with ssh: while ! is_alive graucho; do sleep 5 done ssh graucho GoalKicker.com – Bash Notes for Professionals 56 Chapter 15: Bash Parameter Expansion The $ character introduces parameter expansion, command substitution, or arithmetic expansion. The parameter name or symbol to be expanded may be enclosed in braces, which are optional but serve to protect the variable to be expanded from characters immediately following it which could be interpreted as part of the name. Read more in the Bash User Manual. Section 15.1: Modifying the case of alphabetic characters Version ≥ 4.0 To uppercase $ v="hello" # Just the first character $ printf '%s\n' "${v^}" Hello # All characters $ printf '%s\n' "${v^^}" HELLO # Alternative $ v="hello world" $ declare -u string="$v" $ echo "$string" HELLO WORLD To lowercase $ v="BYE" # Just the first character $ printf '%s\n' "${v,}" bYE # All characters $ printf '%s\n' "${v,,}" bye # Alternative $ v="HELLO WORLD" $ declare -l string="$v" $ echo "$string" hello world Toggle Case $ v="Hello World" # All chars $ echo "${v~~}" hELLO wORLD $ echo "${v~}" # Just the first char hello World Section 15.2: Length of parameter # Length of a string $ var='12345' $ echo "${#var}" GoalKicker.com – Bash Notes for Professionals 57 5 Note that it's the length in number of characters which is not necessarily the same as the number of bytes (like in UTF-8 where most characters are encoded in more than one byte), nor the number of glyphs/graphemes (some of which are combinations of characters), nor is it necessarily the same as the display width. # Number of array elements $ myarr=(1 2 3) $ echo "${#myarr[@]}" 3 # Works for positional parameters as well $ set -- 1 2 3 4 $ echo "${#@}" 4 # But more commonly (and portably to other shells), one would use $ echo "$#" 4 Section 15.3: Replace pattern in string First match: $ a='I am a string' $ echo "${a/a/A}" I Am a string All matches: $ echo "${a//a/A}" I Am A string Match at the beginning: $ echo "${a/#I/y}" y am a string Match at the end: $ echo "${a/%g/N}" I am a strinN Replace a pattern with nothing: $ echo "${a/g/}" I am a strin Add prefix to array items: $ A=(hello world) $ echo "${A[@]/#/R}" Rhello Rworld GoalKicker.com – Bash Notes for Professionals 58 Section 15.4: Substrings and subarrays var='0123456789abcdef' # Define a zero-based offset $ printf '%s\n' "${var:3}" 3456789abcdef # Offset and length of substring $ printf '%s\n' "${var:3:4}" 3456 Version ≥ 4.2 # Negative length counts from the end of the string $ printf '%s\n' "${var:3:-5}" 3456789a # Negative offset counts from the end # Needs a space to avoid confusion with ${var:-6} $ printf '%s\n' "${var: -6}" abcdef # Alternative: parentheses $ printf '%s\n' "${var:(-6)}" abcdef # Negative offset and negative length $ printf '%s\n' "${var: -6:-5}" a The same expansions apply if the parameter is a positional parameter or the element of a subscripted array: # Set positional parameter $1 set -- 0123456789abcdef # Define offset $ printf '%s\n' "${1:5}" 56789abcdef # Assign to array element myarr[0]='0123456789abcdef' # Define offset and length $ printf '%s\n' "${myarr[0]:7:3}" 789 Analogous expansions apply to positional parameters, where offsets are one-based: # Set positional parameters $1, $2, ... $ set -- 1 2 3 4 5 6 7 8 9 0 a b c d e f # Define an offset (beware $0 (not a positional parameter) # is being considered here as well) $ printf '%s\n' "${@:10}" 0 a b c d e f GoalKicker.com – Bash Notes for Professionals 59 # Define an offset and a length $ printf '%s\n' "${@:10:3}" 0 a b # No negative lengths allowed for positional parameters $ printf '%s\n' "${@:10:-2}" bash: -2: substring expression < 0 # Negative offset counts from the end # Needs a space to avoid confusion with ${@:-10:2} $ printf '%s\n' "${@: -10:2}" 7 8 # ${@:0} is $0 which is not otherwise a positional parameters or part # of $@ $ printf '%s\n' "${@:0:2}" /usr/bin/bash 1 Substring expansion can be used with indexed arrays: # Create array (zero-based indices) $ myarr=(0 1 2 3 4 5 6 7 8 9 a b c d e f) # Elements with index 5 and higher $ printf '%s\n' "${myarr[@]:12}" c d e f # 3 elements, starting with index 5 $ printf '%s\n' "${myarr[@]:5:3}" 5 6 7 # The last element of the array $ printf '%s\n' "${myarr[@]: -1}" f Section 15.5: Delete a pattern from the beginning of a string Shortest match: $ a='I am a string' $ echo "${a#*a}" m a string Longest match: $ echo "${a##*a}" string GoalKicker.com – Bash Notes for Professionals 60 Section 15.6: Parameter indirection Bash indirection permits to get the value of a variable whose name is contained in another variable. Variables example: $ red="the color red" $ green="the color green" $ color=red $ echo "${!color}" the color red $ color=green $ echo "${!color}" the color green Some more examples that demonstrate the indirect expansion usage: $ foo=10 $ x=foo $ echo ${x} #Classic variable print foo $ foo=10 $ x=foo $ echo ${!x} #Indirect expansion 10 One more example: $ argtester () { for (( i=1; i<="$#"; i++ )); do echo "${i}";done; }; argtester -ab -cd -ef 1 #i expanded to 1 2 #i expanded to 2 3 #i expanded to 3 $ argtester () { for (( i=1; i<="$#"; i++ )); do echo "${!i}";done; }; argtester -ab -cd -ef -ab # i=1 --> expanded to $1 ---> expanded to first argument sent to function -cd # i=2 --> expanded to $2 ---> expanded to second argument sent to function -ef # i=3 --> expanded to $3 ---> expanded to third argument sent to function Section 15.7: Parameter expansion and filenames You can use Bash Parameter Expansion to emulate common filename-processing operations like basename and dirname. We will use this as our example path: FILENAME="/tmp/example/myfile.txt" To emulate dirname and return the directory name of a file path: echo "${FILENAME%/*}" #Out: /tmp/example To emulate basename $FILENAME and return the filename of a file path: echo "${FILENAME##*/}" GoalKicker.com – Bash Notes for Professionals 61 #Out: myfile.txt To emulate basename $FILENAME .txt and return the filename without the .txt. extension: BASENAME="${FILENAME##*/}" echo "${BASENAME%%.txt}" #Out: myfile Section 15.8: Default value substitution ${parameter:-word} If parameter is unset or null, the expansion of word is substituted. Otherwise, the value of parameter is substituted. $ unset var $ echo "${var:-XX}" # Parameter is unset -> expansion XX occurs XX $ var="" # Parameter is null -> expansion XX occurs $ echo "${var:-XX}" XX $ var=23 # Parameter is not null -> original expansion occurs $ echo "${var:-XX}" 23 ${parameter:=word} If parameter is unset or null, the expansion of word is assigned to parameter. The value of parameter is then substituted. Positional parameters and special parameters may not be assigned to in this way. $ unset var $ echo "${var:=XX}" # Parameter is unset -> word is assigned to XX XX $ echo "$var" XX $ var="" # Parameter is null -> word is assigned to XX $ echo "${var:=XX}" XX $ echo "$var" XX $ var=23 # Parameter is not null -> no assignment occurs $ echo "${var:=XX}" 23 $ echo "$var" 23 Section 15.9: Delete a pattern from the end of a string Shortest match: $ a='I am a string' $ echo "${a%a*}" I am Longest match: GoalKicker.com – Bash Notes for Professionals 62 $ echo "${a%%a*}" I Section 15.10: Munging during expansion Variables don't necessarily have to expand to their values - substrings can be extracted during expansion, which can be useful for extracting file extensions or parts of paths. Globbing characters keep their usual meanings, so .* refers to a literal dot, followed by any sequence of characters; it's not a regular expression. $ v=foo-bar-baz $ echo ${v%%-*} foo $ echo ${v%-*} foo-bar $ echo ${v##*-} baz $ echo ${v#*-} bar-baz It's also possible to expand a variable using a default value - say I want to invoke the user's editor, but if they've not set one I'd like to give them vim. $ EDITOR=nano $ ${EDITOR:-vim} /tmp/some_file # opens nano $ unset EDITOR $ $ ${EDITOR:-vim} /tmp/some_file # opens vim There are two different ways of performing this expansion, which differ in whether the relevant variable is empty or unset. Using :- will use the default if the variable is either unset or empty, whilst - only uses the default if the variable is unset, but will use the variable if it is set to the empty string: $ a="set" $ b="" $ unset c $ echo ${a:-default_a} ${b:-default_b} ${c:-default_c} set default_b default_c $ echo ${a-default_a} ${b-default_b} ${c-default_c} set default_c Similar to defaults, alternatives can be given; where a default is used if a particular variable isn't available, an alternative is used if the variable is available. $ a="set" $ b="" $ echo ${a:+alternative_a} ${b:+alternative_b} alternative_a Noting that these expansions can be nested, using alternatives becomes particularly useful when supplying arguments to command line flags; $ output_file=/tmp/foo $ wget ${output_file:+"-o ${output_file}"} www.stackexchange.com # expands to wget -o /tmp/foo www.stackexchange.com $ unset output_file GoalKicker.com – Bash Notes for Professionals 63 $ wget ${output_file:+"-o ${output_file}"} www.stackexchange.com # expands to wget www.stackexchange.com Section 15.11: Error if variable is empty or unset The semantics for this are similar to that of default value substitution, but instead of substituting a default value, it errors out with the provided error message. The forms are ${VARNAME?ERRMSG} and ${VARNAME:?ERRMSG}. The form with : will error our if the variable is unset or empty, whereas the form without will only error out if the variable is unset. If an error is thrown, the ERRMSG is output and the exit code is set to 1. #!/bin/bash FOO= # ./script.sh: line 4: FOO: EMPTY echo "FOO is ${FOO:?EMPTY}" # FOO is echo "FOO is ${FOO?UNSET}" # ./script.sh: line 8: BAR: EMPTY echo "BAR is ${BAR:?EMPTY}" # ./script.sh: line 10: BAR: UNSET echo "BAR is ${BAR?UNSET}" The run the full example above each of the erroring echo statements needs to be commented out to proceed. GoalKicker.com – Bash Notes for Professionals 64 Chapter 16: Copying (cp) Option Description -a,-archive Combines the d, p and r options -b, -backup Before removal, makes a backup -d, --no-deference Preserves links -f, --force Remove existing destinations without prompting user -i, --interactive Show prompt before overwriting -l, --link Instead of copying, link files instead -p, --preserve Preserve file attributes when possible -R, --recursive Recursively copy directories Section 16.1: Copy a single file Copy foo.txt from /path/to/source/ to /path/to/target/folder/ cp /path/to/source/foo.txt /path/to/target/folder/ Copy foo.txt from /path/to/source/ to /path/to/target/folder/ into a file called bar.txt cp /path/to/source/foo.txt /path/to/target/folder/bar.txt Section 16.2: Copy folders copy folder foo into folder bar cp -r /path/to/foo /path/to/bar if folder bar exists before issuing the command, then foo and its content will be copied into the folder bar. However, if bar does not exist before issuing the command, then the folder bar will be created and the content of foo will be placed into bar GoalKicker.com – Bash Notes for Professionals 65 Chapter 17: Find find is a command to recursively search a directory for files(or directories) that match a criteria, and then perform some action on the selected files. find search_path selection_criteria action Section 17.1: Searching for a file by name or extension To find files/directories with a specific name, relative to pwd: $ find . -name "myFile.txt" ./myFile.txt To find files/directories with a specific extension, use a wildcard: $ find . -name "*.txt" ./myFile.txt ./myFile2.txt To find files/directories matching one of many extensions, use the or flag: $ find . -name "*.txt" -o -name "*.sh" To find files/directories which name begin with abc and end with one alpha character following a one digit: $ find . -name "abc[a-z][0-9]" To find all files/directories located in a specific directory $ find /opt To search for files only (not directories), use -type f: find /opt -type f To search for directories only (not regular files), use -type d: find /opt -type d Section 17.2: Executing commands against a found file Sometimes we will need to run commands against a lot of files. This can be done using xargs. find . -type d -print | xargs -r chmod 770 The above command will recursively find all directories (-type d) relative to . (which is your current working directory), and execute chmod 770 on them. The -r option specifies to xargs to not run chmod if find did not find any files. If your files names or directories have a space character in them, this command may choke; a solution is to use the following GoalKicker.com – Bash Notes for Professionals 66 find . -type d -print0 | xargs -r -0 chmod 770 In the above example, the -print0 and -0 flags specify that the file names will be separated using a null byte, and allows the use of special characters, like spaces, in the file names. This is a GNU extension, and may not work in other versions of find and xargs. The preferred way to do this is to skip the xargs command and let find call the subprocess itself: find . -type d -exec chmod 770 {} \; Here, the {} is a placeholder indicating that you want to use the file name at that point. find will execute chmod on each file individually. You can alternatively pass all file names to a single call of chmod, by using find . -type d -exec chmod 770 {} + This is also the behaviour of the above xargs snippets. (To call on each file individually, you can use xargs -n1). A third option is to let bash loop over the list of filenames find outputs: find . -type d | while read -r d; do chmod 770 "$d"; done This is syntactically the most clunky, but convenient when you want to run multiple commands on each found file. However, this is unsafe in the face of file names with odd names. find . -type f | while read -r d; do mv "$d" "${d// /_}"; done which will replace all spaces in file names with underscores.(This example also won't work if there are spaces in leading directory names.) The problem with the above is that while read -r expects one entry per line, but file names can contain newlines (and also, read -r will lose any trailing whitespace). You can fix this by turning things around: find . -type d -exec bash -c 'for f; do mv "$f" "${f// /_}"; done' _ {} + This way, the -exec receives the file names in a form which is completely correct and portable; the bash -c receives them as a number of arguments, which will be found in $@, correctly quoted etc. (The script will need to handle these names correctly, of course; every variable which contains a file name needs to be in double quotes.) The mysterious _ is necessary because the first argument to bash -c 'script' is used to populate $0. Section 17.3: Finding file by access / modification time On an ext filesystem, each file has a stored Access, Modification, and (Status) Change time associated with it - to view this information you can use stat myFile.txt; using flags within find, we can search for files that were modified within a certain time range. To find files that have been modified within the last 2 hours: $ find . -mmin -120 GoalKicker.com – Bash Notes for Professionals 67 To find files that have not been modified within the last 2 hours: $ find . -mmin +120 The above example are searching only on the modified time - to search on access times, or changed times, use a, or c accordingly. $ find . -amin -120 $ find . -cmin +120 General format: -mmin n : File was modified n minutes ago -mmin -n : File was modified less than n minutes ago -mmin +n : File was modified more than n minutes ago Find files that have been modified within the last 2 days: find . -mtime -2 Find files that have not been modified within the last 2 days find . -mtime +2 Use -atime and -ctime for access time and status change time respectively. General format: -mtime n : File was modified nx24 hours ago -mtime -n : File was modified less than nx24 hours ago -mtime +n : File was modified more than nx24 hours ago Find files modified in a range of dates, from 2007-06-07 to 2007-06-08: find . -type f -newermt 2007-06-07 ! -newermt 2007-06-08 Find files accessed in a range of timestamps (using files as timestamp), from 1 hour ago to 10 minutes ago: touch -t $(date -d '1 HOUR AGO' +%Y%m%d%H%M.%S) start_date touch -t $(date -d '10 MINUTE AGO' +%Y%m%d%H%M.%S) end_date timeout 10 find "$LOCAL_FOLDER" -newerat "start_date" ! -newerat "end_date" -print General format: -newerXY reference : Compares the timestamp of the current file with reference. XY could have one of the following values: at (access time), mt (modification time), ct (change time) and more. reference is the name of a file whe want to compare the timestamp specified (access, modification, change) or a string describing an absolute time. Section 17.4: Finding files according to size Find files larger than 15MB: GoalKicker.com – Bash Notes for Professionals 68 find -type f -size +15M Find files less than 12KB: find -type f -size -12k Find files exactly of 12KB size: find -type f -size 12k Or find -type f -size 12288c Or find -type f -size 24b Or find -type f -size 24 General format: find [options] -size n[cwbkMG] Find files of n-block size, where +n means more than n-block, -n means less than n-block and n (without any sign) means exactly n-block Block size: 1. c: bytes 2. w: 2 bytes 3. b: 512 bytes (default) 4. k: 1 KB 5. M: 1 MB 6. G: 1 GB Section 17.5: Filter the path The -path parameter allows to specify a pattern to match the path of the result. The pattern can match also the name itself. To find only files containing log anywhere in their path (folder or name): find . -type f -path '*log*' To find only files within a folder called log (on any level): find . -type f -path '*/log/*' To find only files within a folder called log or data: GoalKicker.com – Bash Notes for Professionals 69 find . -type f -path '*/log/*' -o -path '*/data/*' To find all files except the ones contained in a folder called bin: find . -type f -not -path '*/bin/*' To find all file all files except the ones contained in a folder called bin or log files: find . -type f -not -path '*log' -not -path '*/bin/*' Section 17.6: Finding files by type To find files, use the -type f flag $ find . -type f To find directories, use the -type d flag $ find . -type d To find block devices, use the -type b flag $ find /dev -type b To find symlinks, use the -type l flag $ find . -type l Section 17.7: Finding files by specific extension To find all the files of a certain extension within the current path you can use the following find syntax. It works by making use of bash's built-in glob construct to match all the names having the .extension. find /directory/to/search -maxdepth 1 -type f -name "*.extension" To find all files of type .txt from the current directory alone, do find . -maxdepth 1 -type f -name "*.txt" GoalKicker.com – Bash Notes for Professionals 70 Chapter 18: Using sort Option Meaning -u Make each lines of output unique sort is a Unix command to order data in file(s) in a sequence. Section 18.1: Sort command output sort command is used to sort a list of lines. Input from a file sort file.txt Input from a command You can sort any output command. In the example a list of file following a pattern. find * -name pattern | sort Section 18.2: Make output unique If each lines of the output need to be unique, add -u option. To display owner of files in folder ls -l | awk '{print $3}' | sort -u Section 18.3: Numeric sort Suppose we have this file: test>>cat file 10.Gryffindor 4.Hogwarts 2.Harry 3.Dumbledore 1.The sorting hat To sort this file numerically, use sort with -n option: test>>sort -n file This should sort the file as below: 1.The sorting hat 2.Harry 3.Dumbledore 4.Hogwarts 10.Gryffindor Reversing sort order: To reverse the order of the sort use the -r option GoalKicker.com – Bash Notes for Professionals 71