btrfs for a simple and powerful backup system

I’ve been testing btrfs for some months now. One of the most interesting features of this file-system is its snapshoting capabilities. Before that I was using rsnapshot. The issue with rsnapshot is that its lowest atomic level for snapshotting is the files themselves using hard-links. So any database table where one row is changed is copied completely. Btrfs as you might guess will only copy the modified chunks (I don’t know the atomicity of them [but who cares?]).

Here is a simple I’ve been using during these last months to backup my laptop and (remotely hosted) servers.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
#!/bin/sh
 
# === ARGUMENTS PARSING ===
 
# We don't want to define a default period
PERIOD=
 
while echo $1 | grep ^- > /dev/null; do
 
    if [ "$1" = "--daily" ]; then
        PERIOD=daily
    fi
 
    if [ "$1" = "--monthly" ]; then
        PERIOD=monthly
    fi
 
    if [ "$1" = "--period" ]; then
        PERIOD=$2
        shift
    fi
 
    shift
done
 
if [ "${PERIOD}" = "" ]; then
        echo "You have to define a period  with the --period arg !" >&2
        exit 1
fi
 
# === END OF ARGUMENTS PARSING ===
 
# === PARAMETERS ===
 
# * Device we will use
DISK=/mnt/externe3
 
# * Subvolume used for the backup
SUBVOLUME=${DISK}/servers-backup
 
# * Current date (you could limit the date to +%Y-%m-%d)
DATE=`/bin/date +%Y-%m-%d_%H-%M-%S`
 
# * snapshot directory that will be used
SNAPDIR=${DISK}/snap/servers-backup
 
# * snapshot volume that will be used
SNAPVOL=${SNAPDIR}/${PERIOD}-${DATE}
 
# * max days to keep daily backups
MAX_DAYLY=60
 
# * max days to keep monthly backups
MAX_MONTHLY=365
 
# * Alert limit
LIMIT_ALERT=95
 
# * High limit
LIMIT_HIGH=90
 
# * Low limit
LIMIT_LOW=85
 
# === END OF PARAMETERS ===
 
# We get the space used over the total allocated space and the total percentage use.
# This is NOT the device total size but it's a lot more reliable than "df -h"
DISK_USED=`/sbin/btrfs filesystem df ${DISK}|grep Data|grep -Po "used=([0-9]*)"|cut -d= -f2`
DISK_TOTAL=`/sbin/btrfs filesystem df ${DISK}|grep Data|grep -Po "total=([0-9]*)"|cut -d= -f2`
DISK_PERC=`echo 100*${DISK_USED}/${DISK_TOTAL}|bc`
 
# We create the snapshot dir if it doesn't exist
if [ ! -d ${SNAPDIR} ]; then
        mkdir -p ${SNAPDIR}
fi
 
cd ${SNAPDIR}
 
# If we are over the low free space limit,
# we delete two days of daily backup.
if [ $DISK_PERC -gt $LIMIT_LOW ]; then
        echo "LOW LIMIT reached: $DISK_PERC > $LIMIT_LOW : Deleting 2 days" >&2
 
        OLDEST_FILES=`ls --sort=time -r|grep "daily-.*"|head -2`
        for file in $OLDEST_FILES; do
                /sbin/btrfs subvolume delete $file;
        done
 
fi
 
# If we are over the high free space limit,
# we delete a month of monthly backup
if [ $DISK_PERC -gt $LIMIT_HIGH ]; then
        echo "HIGH LIMIT reached: $DISK_PERC > $LIMIT_HIGH : Deleting 1 month" >&2
 
        OLDEST_FILES=`ls --sort=time -r|grep "monthly-.*"|head -1`
        for file in $OLDEST_FILES; do
                /sbin/btrfs subvolume delete $file;
        done
 
fi
 
# If we are over the alert free space limit,
# we delete the first two oldest files we can find
if [ $DISK_PERC -gt $LIMIT_ALERT ]; then
        echo "ALERT LIMIT reached: $DISK_PERC > $LIMIT_ALERT : Deleting the 2 oldest" >&2
 
        OLDEST_FILES=`ls --sort=time -r|head -2`
        for file in $OLDEST_FILES; do
                /sbin/btrfs subvolume delete $file;
        done
fi
 
 
# We touch the subvolume to change the modification date
touch ${SUBVOLUME}
 
# We do a snapshot of the subvolume
if [ ! -d "${SNAPVOL}" ]; then
        /sbin/btrfs subvolume snapshot ${SUBVOLUME} ${SNAPVOL}
fi
 
# We delete the backups older than MAX_DAYLY
find ${SNAPDIR} -mindepth 1 -maxdepth 1 -mtime +${MAX_DAYLY} -name "daily-*" -exec /sbin/btrfs subvolume delete {} \;
 
# We delete the backups older than MAX_MONTHLY
find ${SNAPDIR} -mindepth 1 -maxdepth 1 -mtime +${MAX_MONTHLY} -name "monthly-*" -exec /sbin/btrfs subvolume delete {} \;
 
 
# This is the actual backup code
# You need to save your data into the ${SUBVOLUME} directory
 
# We will only do the actual backup for the daily task
if [ "${PERIOD}" = "daily" ]; then
 
rsync -auv --inplace /usr/local/bin ${SUBVOLUME}/localhost/usr/local
 
fi

Then this is how you can use it by adding these cron-tasks :

1
2
0 12 * * *  user /usr/local/bin/backup-servers --period daily   >/var/log/backup-servers-daily.log
55 10 1 * * user /usr/local/bin/backup-servers --period monthly >/var/log/backup-servers-monthly.log

2 thoughts on “btrfs for a simple and powerful backup system”

  1. You are right, I fixed it. I had it in the server backup code which I should also include.

    It looks like this:
    RSYNC_ARGS="-avz --inplace --timeout=3600 --exclude /tmp --exclude /proc --exclude /sys --exclude /dev --exclude swapfile --stats"

Leave a Reply

Your email address will not be published. Required fields are marked *