Sunday, May 30, 2010

Simple Backup for Spare-Time-Projects

In my spare time -- the remainder of the time my child, wife and job don't demand ;) -- I do some programming for private projects.

I was searching a very simple backup solution considering that I add new stuff just once in a while. The requirements:

  • Check once a day if I added new stuff
  • Just in case new or changed files are detected do the backup
  • Store the last 12 files locally on the development server
  • All backups should be stored on a remote FTP server

Finally, I wrote the following shell-script that does the job:
(If you like to use it just change the variable at the beginning and the FTP commands below):

#!/bin/bash

# Adjust those parameters to your needs
 
REPO_PATH="/var/myproject/"
BACKUP_PATH="/mybackup/myproject/"
BACKUP_FILE_NAME="myproject_backup"
TIME_FILE="${BACKUP_PATH}last_backup_time"
NR_OF_BACKUPS=12
BACKUP_EXCLUDES="/log/"


BACKUP_FILE="${BACKUP_PATH}${BACKUP_FILE_NAME}"

if [ ! -f "$TIME_FILE" ]
then
  echo "No $TIME_FILE found. Create one!"
  touch -d 731211 $TIME_FILE
fi

CHANGED_FILES=`find $REPO_PATH -newer $TIME_FILE | grep -v "$BACKUP_EXCLUDES" | wc -l`

echo "Found $CHANGED_FILES files in backup folder!"

if [ "$CHANGED_FILES" -gt "0" ]
then
  # do the backup
  tar -cjf "${BACKUP_FILE}_$(date +%F_%H%M%S).tar.bz2" -C $REPO_PATH . && touch $TIME_FILE && echo "Backup succeeded!" || echo "Backup FAILED !!!" 
  # put tar on FTP-Backup
  echo Putting backup-tar on FTP-Server
  cd $BACKUP_PATH
  TAR_FILE=`ls -tr1 *.tar* | tail -1`
  ftp -n -i my-ftp-server.hosteurope.de <<EOF
user ftpuser ftppasswd
cd remoteDir
put $TAR_FILE
quit
EOF
  # should old ones be removed ?
  COUNT_BACKUPS=`ls -l ${BACKUP_FILE}* | wc -l`
  if [ "$COUNT_BACKUPS" -gt "$NR_OF_BACKUPS" ]
  then
    echo Found more then $NR_OF_BACKUPS backups. Delete the oldest one.
    ls -t1 ${BACKUP_FILE}* | tail -1 | xargs rm
  fi
fi

Then add the script to your cronjobs. For example:

15 4 * * * /usr/local/bin/mybackup >/mybackup/myproject/backup.log 2>>/mybackup/myproject/error_backup.log

One final note:
If you do backups of "transactional" databases or version-control-systems, don't use the simple tar-backup-command inside the script. Instead use a "transaction-safe" application specific tool. For example in case of doing a Subversion backup use something like:

hot-backup.py --archive-type=bz2 --num-backups=12 $REPO_PATH $BACKUP_PATH && touch $TIME_FILE && echo "Backup succeeded!" || echo "Backup FAILED !!!"

Saturday, May 15, 2010

Increase your Bash History Size

Working with the linux bash shell, the command history could be one of your best friends.
Using the history command (for example joined with a grep) lets you relax your brain and fingers. Sure, even more relaxed is the usage of the reverse-search-history function (Ctrl-R).

To make sure the history's brain isn't to small, you should increase its size. The following lines need to be added to your ~/.bashrc file. It increases the history to the last 10.000 commands, erases previous duplicates and makes sure that the commands are appended (and do no override) on shell exit:

export HISTSIZE=10000
export HISTCONTROL=erasedups
shopt -s histappend

Friday, May 7, 2010

Online Web-Developer Tools

This is a small collection of tools that could be very useful in the daily job of a web-developer. I'll try to keep this one up-to-date:

Where? What?
www.generateit.net A lot of useful generators. For example colors, Lorem Impsum, Cron Jobs and CSS stuff.
tools.web-max.ca Misc tools for encodings (MD5, Sha-1, URL, Base64...), IP stuff and others
hogehoge.tk/tool-i De- and encodings for character-sets, URL, MD5, SHA-1, Hex
kuler.adobe.com Let you generate a nice and harmonious color-scheme
www.typetester.org Test and review fonts and screen types

Sunday, May 2, 2010

Install latest Version of MySQL on Ubuntu

Some days ago I installed the latest Subversion Version to Ubuntu 8.04 (Hardy). It wasn't quite easy, so I was afraid that installing the up-to-date version of MySQL would be a similar trouble.

Luckily, it turns out that it's easy to do. Here we go:

First check the version that is available currently:
root@dev:~# apt-cache madison mysql-server
mysql-server | 5.0.51a-3ubuntu5.5 | http://archive.ubuntu.com hardy-updates/main Packages

Ok, this one too old. We need at least version 5.1.45. Fortunately we just need to add the Lenny debian repository:
root@dev:~#vi /etc/apt/sources.list
add --> deb http://packages.dotdeb.org lenny all
root@dev:~# apt-get update
.....
Fetched 2380kB in 1s (2140kB/s)
Reading package lists... Done
root@dev:~# apt-cache madison mysql-server
mysql-server | 5.1.45-0.dotdeb.0 | http://packages.dotdeb.org lenny/all Packages
mysql-server | 5.0.51a-3ubuntu5.5 | http://archive.ubuntu.com hardy-updates/main Packages

Install the new version of MySQL server (it removes an old MySQL version automatically):
root@dev:~# apt-get install mysql-server-5.1
Important: Select always 'y' on questions during the installation process. This includes the question to override the /etc/mysql/my.cnf. Otherwise you need to adjust it manually, because the "skip-bdb" option doesn't work with version 5.1.

Finally check:
root@dev:~# mysql --version
mysql  Ver 14.14 Distrib 5.1.45, for debian-linux-gnu (x86_64) using readline 5.2

Finishing basic MySQL setup:

Now you need to configure your DB-system as described in the MySQL manual: http://dev.mysql.com/doc/refman/5.1/en/unix-post-installation.html

Problem:
root@dev:~# mysql_install_db --user=mysql
Installing MySQL system tables...
ERROR: 1136  Column count doesn't match value count at row 1
100502 13:26:07 [ERROR] Aborting
100502 13:26:07 [Note] /usr/sbin/mysqld: Shutdown complete
Installation of system tables failed!  Examine the logs in
/var/lib/mysql for more information.

Ok, there are databases in the old 5.0.x format, let's upgrade them:
root@dev:~# /etc/init.d/mysql stop
root@dev:~# /usr/sbin/mysqld --skip-grant &
root@dev:~# mysql_upgrade
root@dev:~# mysql_install_db --user=mysql

Alternativly, delete the stuff in your mysql-data folder (in case you setup a fresh DB):
root@dev:~# /etc/init.d/mysql stop
root@dev:~# rm -R /var/lib/mysql/*
root@dev:~# mysql_install_db --user=mysql

Now follow the MySQL documentation:
root@dev:~# mysqld_safe --user=mysql &
root@dev:~# mysqladmin version
mysqladmin  Ver 8.42 Distrib 5.1.45, for debian-linux-gnu on x86_64
root@dev:~# mysql -e "SELECT * FROM user" mysql
root@dev:~# mysql -e "select version();" mysql

And secure and setup some accounts:
root@dev:~# mysql -u root
mysql> SET PASSWORD FOR 'root'@'localhost' = PASSWORD('mypasswd');
mysql> SET PASSWORD FOR 'root'@'127.0.0.1' = PASSWORD('mypasswd');
mysql> CREATE USER 'debian-sys-maint'@'localhost' IDENTIFIED BY 'wze0EguNSaU52jCH';
mysql> GRANT ALL PRIVILEGES ON *.* TO 'debian-sys-maint'@'localhost' WITH GRANT OPTION;

If you need external access to your MySQL (instead of just allowing it from localhost) do something like this:
mysql> CREATE USER 'root'@'%' IDENTIFIED BY 'mypasswd';
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' WITH GRANT OPTION;
root@dev:~# vi /etc/mysql/my.cnf
           --> Change:  "bind-address" to "0.0.0.0"
root@dev:~# /etc/init.d/mysql restart