====== Shell ======

===== Useful standard utilities =====

  * [[man>sed]] -- regular expression replacing
  * [[man>awk]] -- tokenizing strings
  * [[man>seq]] -- returns a list of integers
  * [[man>cat]] -- read a file
  * [[man>find]] -- get a list of files matching a certain criteria

===== Find space wasters =====

This gives you a quick overview where the largest directories are

  $> du --max-depth=3 -x / |sort -n

===== Delete empty directories =====

  $> find . -type d -empty -delete

===== Execute a local shellscript on a remote host =====


  $> cat script.sh |ssh user@host "bash -s"


===== Looping over lines with spaces =====

To read filenames from a file and do something (eg. echo them) you might wanna do this:

<code bash>
#!/bin/bash

for x in `cat files.txt`
do
    echo "$x"
done
</code>

Unfortunately this breaks when some of these filenames contain spaces. [[man>read]] is the rescue:

<code bash>
#!/bin/bash

cat files.txt |while read x
do
    echo "$x"
done
</code>

If you need to [[man>scp]] them you need another trick to escape those spaces:

<code bash>
#!/bin/bash

cat files.txt |while read x
do
    scp user@remote:"\"$x\"" .
done
</code>



===== Replace by regular expression in multiple files =====

You can use Perl to replace a certain string with another in multiple files. But first you should make sure your regexp matches the right thing. The following is looking for a path in quotes starting with the dir ''static'' in all ''php'' files below the current directory:

  $> find -name '*.php' | xargs perl -n -e 'm/"(\/?static\/.+?)"/g && print "$ARGV: $1\n"'

If you're happy use the following to do the replacement. Here we surround the string with the PHP function PI:

  $> find -name '*.php' | xargs perl -p -i -e 's/"(\/?static\/.+?)"/"<?PI("$1")?>"/g'

===== Return absolute filename =====

This handy function returns the absolute filename of a given file. Example:

  $> absname ~/.bashrc
  /home/andi/.bashrc

<code bash>
function absname {
  if [ -d "$1" ] ; then   # Only a directory name.
    dir="$1"
    unset file
  elif [ -f "$1" ] ; then # Strip off and save the filename.
    dir=$(dirname "$1")
    file="/"$(basename "$1")
  else
    # The file did not exist.
    # Return null string as error.
    echo
    return 1
  fi

  # Change to the directory and display the absolute pathname.
  cd "$dir"  > /dev/null
  echo ${PWD}${file}
}
</code>

Here is a simpler one which doesn't print nice paths but just prepends ''$PWD'' - this comes handy in scripts sometimes:

<code bash>
#!/bin/bash

p=`echo "$0" | sed -re "s§^([^/])§$PWD/\\1§"`
echo "The scripts absolute path is $p"
</code>

Or you could just use readlink, which also resolves symlinks:

    $> readlink -f ~/.bashrc
    /home/andi/.bashrc

===== Print ANSI colors =====

This small script prints all available ANSI-colors on the terminal.

<code bash>
#!/bin/bash

for i in 30 31 32 33 34 35 36 37 39
do
 for j in 40 41 42 43 44 45 46 47 49
 do
   # skip if same fore- and backgroundcolor
   if [ $j -eq $[ i + 10 ] ]; then
     continue
   fi
   echo -e $i $j "\033[${i};${j}mCOLOR\033[0m"
 done
done
</code>

===== Copy remote directories =====

This is a short snippet to copy whole directory trees with all permissions, symlinks and special files (devices,fifos...).

  $> cd /home/user
  $> ssh user@remote.host.com "cd /home/user && find . \
     |grep -v bigfile | cpio -H crc -o" | cpio -idum --verbose

The above statement will copy everything but //*bigfile*// from the remote directory ''/home/user'' to the local directory ''/home/user''

===== 30 days backup =====

Just found at http://slashdot.org/comments.pl?sid=113431&cid=9607753

  $> tar czf /backupdir/`date +%Y%m%d`.tar.gz /home/directory
  $> find /backupdir -name \*.tar.gz -mtime +30 -print0 | xargs -0r rm -f

===== TAR all directories and delete them afterwards =====

  $> for x in `find -type d -maxdepth 1|grep -v '^.$'`; do tar -czvf $x.tgz $x && rm -rf $x; done

===== Set system date by time from webserver =====

Sometimes you want to set the system date of Unix system to some sane value. You could do it manually or use some NTP client. The first idea is too much work ;-) and for the second one you need a time server and some client installed. But why not use the HTTP Response Header ''Date''? Well you could use again some client tool like [[http://www.clevervest.com/htp/intro.html|htp]] or just use this snippet.

  #> date -s "`curl -I http://www.kernel.org |grep 'Date:'|awk -F': ' '{print $2}'`"

===== Backups with afio =====

The following command uses [[man>afio]] to create a gzipped backup stored on a remote machine (using [[man>ssh]]).

  #> find / -path '/proc' -prune -o -print0 | afio -o -v -M 100m -Z -0 user@backup.host%ssh:/backups/backup.afio

Creating multiple maximum 1GB sized chunks:

  #> find / -path '/proc' -prune -o -print0 | \
     afio -o -v -M 100m -Z -0 -s 1000m \
     -H 'bash -c "mv \$2 \$2.\$1" -s' backup.afio

This is to prefer over other solutions using [[man>split]] as this makes sure each chunk is a valid afio archive itself.

Unfortunately this does not work with the ''%%user@backup.host%ssh:/backups/backup.afio%%'' syntax :-/

===== Show specified lines from a file =====

This shows lines 713-720 from ''longlogfile.log'' -- you can use ''$'' to specify the last line. Omitting the comma and the second number just prints a single line.

  $> sed -n '713,720p' longlogfile.log

===== dos2unix unix2dos =====

These are handy in the .bashrc

<code bash>
alias dos2unix="perl -pi -e 's/\r\n/\n/;'"
alias unix2dos="perl -pi -e 's/\n/\r\n/;'"
</code>

I've got trouble with unix2dos while calling it on DOS files. So I replaced it with a new version. I'm not very familar with "regular expressions", so maybe there is a shorter way to find LF without a leading CR. But it works fine for me.

<code bash>
alias unix2dos="perl -pi -e 's/([^\r])\n/$1\r\n/;'"
</code>

Command Line (Windows to Unix):
  col -bx < win_file.txt > unix_file.txt

===== Strip UTF-8 Byte Order Mark (BOM) =====

The following command removes the UTF-8 Byte Order Mark from given file(s). See [[http://www.xs4all.nl/~mechiel/projects/bomstrip/|bomstrip]] for solutions in other languages.

  $> sed -i -s -e '1s/^\xef\xbb\xbf//' <files>


===== Print formatted man pages =====

You can print out man pages -- in this case, 2 man pages on one paper sheet:

  $> function lprman () { man -t $1 $2 | psnup -2 | lpr -P $3 }

Parameter 1: man page category \\ Parameter 2: man page name \\ Parameter 3: name of the queue to print with

Sample call:
  $> lprman 5 fstab mylaser

Notes:
  * you need psutils to have that working -- psnup belongs into this package
  * you can put this function in a file that is sourced when you log in, for example /etc/profile.local
  * of course you can omit ''-P $3'' when you always print with your default printer
  * on my machines, the duplex setting is ignored, it's always printing on one side of the paper