• sed – regular expression replacing
• awk – tokenizing strings
• seq – returns a list of integers
• cat – read a file
• find – get a list of files matching a certain criteria

This gives you a quick overview where the largest directories are

$> du --max-depth=3 -x / |sort -n

$> find . -type d -empty -delete

$> cat script.sh |ssh user@host "bash -s"

To read filenames from a file and do something (eg. echo them) you might wanna do this:

#!/bin/bash
 
for x in `cat files.txt`
do
    echo "$x"
done

Unfortunately this breaks when some of these filenames contain spaces. read is the rescue:

#!/bin/bash
 
cat files.txt |while read x
do
    echo "$x"
done

If you need to scp them you need another trick to escape those spaces:

#!/bin/bash
 
cat files.txt |while read x
do
    scp user@remote:"\"$x\"" .
done

You can use Perl to replace a certain string with another in multiple files. But first you should make sure your regexp matches the right thing. The following is looking for a path in quotes starting with the dir static in all php files below the current directory:

$> find -name '*.php' | xargs perl -n -e 'm/"(\/?static\/.+?)"/g && print "$ARGV: $1\n"'

If you're happy use the following to do the replacement. Here we surround the string with the PHP function PI:

$> find -name '*.php' | xargs perl -p -i -e 's/"(\/?static\/.+?)"/"<?PI("$1")?>"/g'

This handy function returns the absolute filename of a given file. Example:

$> absname ~/.bashrc
/home/andi/.bashrc

function absname {
  if [ -d "$1" ] ; then   # Only a directory name.
    dir="$1"
    unset file
  elif [ -f "$1" ] ; then # Strip off and save the filename.
    dir=$(dirname "$1")
    file="/"$(basename "$1")
  else
    # The file did not exist.
    # Return null string as error.
    echo
    return 1
  fi
 
  # Change to the directory and display the absolute pathname.
  cd "$dir"  > /dev/null
  echo ${PWD}${file}
}

Here is a simpler one which doesn't print nice paths but just prepends $PWD - this comes handy in scripts sometimes:

#!/bin/bash
 
p=`echo "$0" | sed -re "s§^([^/])§$PWD/\\1§"`
echo "The scripts absolute path is $p"

Or you could just use readlink, which also resolves symlinks:

  $> readlink -f ~/.bashrc
  /home/andi/.bashrc

This small script prints all available ANSI-colors on the terminal.

#!/bin/bash
 
for i in 30 31 32 33 34 35 36 37 39
do
 for j in 40 41 42 43 44 45 46 47 49
 do
   # skip if same fore- and backgroundcolor
   if [ $j -eq $[ i + 10 ] ]; then
     continue
   fi
   echo -e $i $j "\033[${i};${j}mCOLOR\033[0m"
 done
done

This is a short snippet to copy whole directory trees with all permissions, symlinks and special files (devices,fifos…).

$> cd /home/user
$> ssh user@remote.host.com "cd /home/user && find . \
   |grep -v bigfile | cpio -H crc -o" | cpio -idum --verbose

The above statement will copy everything but *bigfile* from the remote directory /home/user to the local directory /home/user

Just found at http://slashdot.org/comments.pl?sid=113431&cid=9607753

$> tar czf /backupdir/`date +%Y%m%d`.tar.gz /home/directory
$> find /backupdir -name \*.tar.gz -mtime +30 -print0 | xargs -0r rm -f

$> for x in `find -type d -maxdepth 1|grep -v '^.$'`; do tar -czvf $x.tgz $x && rm -rf $x; done

Sometimes you want to set the system date of Unix system to some sane value. You could do it manually or use some NTP client. The first idea is too much work and for the second one you need a time server and some client installed. But why not use the HTTP Response Header Date? Well you could use again some client tool like htp or just use this snippet.

#> date -s "`curl -I http://www.kernel.org |grep 'Date:'|awk -F': ' '{print $2}'`"

The following command uses afio to create a gzipped backup stored on a remote machine (using ssh).

#> find / -path '/proc' -prune -o -print0 | afio -o -v -M 100m -Z -0 user@backup.host%ssh:/backups/backup.afio

Creating multiple maximum 1GB sized chunks:

#> find / -path '/proc' -prune -o -print0 | \
   afio -o -v -M 100m -Z -0 -s 1000m \
   -H 'bash -c "mv \$2 \$2.\$1" -s' backup.afio

This is to prefer over other solutions using split as this makes sure each chunk is a valid afio archive itself.

Unfortunately this does not work with the user@backup.host%ssh:/backups/backup.afio syntax

This shows lines 713-720 from longlogfile.log – you can use $ to specify the last line. Omitting the comma and the second number just prints a single line.

$> sed -n '713,720p' longlogfile.log

These are handy in the .bashrc

alias dos2unix="perl -pi -e 's/\r\n/\n/;'"
alias unix2dos="perl -pi -e 's/\n/\r\n/;'"

I've got trouble with unix2dos while calling it on DOS files. So I replaced it with a new version. I'm not very familar with “regular expressions”, so maybe there is a shorter way to find LF without a leading CR. But it works fine for me.

alias unix2dos="perl -pi -e 's/([^\r])\n/$1\r\n/;'"

Command Line (Windows to Unix):

col -bx < win_file.txt > unix_file.txt

The following command removes the UTF-8 Byte Order Mark from given file(s). See bomstrip for solutions in other languages.

$> sed -i -s -e '1s/^\xef\xbb\xbf//' <files>

You can print out man pages – in this case, 2 man pages on one paper sheet:

$> function lprman () { man -t $1 $2 | psnup -2 | lpr -P $3 }

Parameter 1: man page category
Parameter 2: man page name
Parameter 3: name of the queue to print with

Sample call:

$> lprman 5 fstab mylaser

Notes:

• you need psutils to have that working – psnup belongs into this package
• you can put this function in a file that is sourced when you log in, for example /etc/profile.local
• of course you can omit -P $3 when you always print with your default printer
• on my machines, the duplex setting is ignored, it's always printing on one side of the paper