Difference between revisions of "Backup with Affa and FreeDup"

From SME Server
Jump to navigationJump to search
m
(10 intermediate revisions by 4 users not shown)
Line 1: Line 1:
 +
=== Version ===
 +
{{ #smeversion: freedup}}
 
=== Purpose ===
 
=== Purpose ===
There are situations in backup process that Affa (with or without Rsync) can't handle well.
+
There are situations in backup process that [[Affa]] (with or without Rsync) can't handle well.
  
 
Imagine backing up user files in a company where documents are distributed on regular basis. It is very likely that all of your users will have a copy of the same file in their documents.
 
Imagine backing up user files in a company where documents are distributed on regular basis. It is very likely that all of your users will have a copy of the same file in their documents.
Line 6: Line 8:
 
Freedup walks through the file trees (directories) you specify. When it finds two identical files on the same device, it hard links them together. In this case two or more files still exist in their respective directories, but only one copy of the data is stored on disk; both directory entries point to the same data blocks.  
 
Freedup walks through the file trees (directories) you specify. When it finds two identical files on the same device, it hard links them together. In this case two or more files still exist in their respective directories, but only one copy of the data is stored on disk; both directory entries point to the same data blocks.  
  
Rsync partially does this, but it won't search for identical files within your backup. This software does just that. '''In my case I saved ONE THIRD of the backup diskspace on one single backup alone.''' And what is more, FreeDup found more identical files within backups' trees themselves, hence correcting Rsync and saving me terabytes of space.
+
Rsync partially does this, but it won't search for identical files within your backup. This software does just that.  
  
'''Without FreeDup's hardlinks my current off-site backup would be 3.4TB, but instead it is only 270MB. That is 12 backups in total - 3x monthly, 4x weekly and 5x daily backups + the current backup running.'''
+
This document describes the installation and configuration options of the freedup in Affa's post job for the purpose to reclaim free space on your disk occupied by duplicate files.
  
This document describes the installation and configuration options of the freedup in Affa's post job for the purpose to reclaim free space on your disk occupied by duplicate files.
+
=== Installation ===
  
 +
====SME8====
 +
The FreeDup package is in the Dag repository, if you have not installed it already you need to do that first.
 +
You need to activate the [[Dag]] reposity before installing this contrib.<br />
  
=== Installation ===
+
see [[dag|dag repository]] <br />
  
To install FreeDup, visit project's homepage and download the latest rpm and install:
 
  
  http://software.neuper.de/freedup/download.html
+
To install the package issue the following command:
 +
  yum install freedup --enablerepo=dag
  
CentOS or RedHat binary works fine.
+
====SME9====
  
 +
To install the package issue the following command:
 +
yum install freedup --enablerepo=smecontribs,smedev
  
 
=== Usage ===
 
=== Usage ===
 
+
To use FreeDup with Affa, first install and setup latest version of Affa. Installation instructions for Affa can be found [[:Affa | here]].
To use FreeDup with Affa, first install and setup latest version of Affa.
 
  
 
Then add this
 
Then add this
Line 35: Line 41:
  
 
Command line options are beyond the scope of this document, in my case I only use -T to preserve timestamps and -v for some verbosity.
 
Command line options are beyond the scope of this document, in my case I only use -T to preserve timestamps and -v for some verbosity.
 
  
 
=== Various Tests ===
 
=== Various Tests ===
 
 
To see how much your backup occupies, you may find these commands below useful.
 
To see how much your backup occupies, you may find these commands below useful.
  
Line 52: Line 56:
  
 
  find <insert your directory with backups> -xdev -type f -links +1 -printf '%i %p\n' | sort -n > links.txt
 
  find <insert your directory with backups> -xdev -type f -links +1 -printf '%i %p\n' | sort -n > links.txt
 +
{{Note box|msg=Please be aware that once you launch FreeDup, it can take several hours to complete, depending on the size of backups as it its algorithm is making sure that files that are to be hardlinked are the same. }}
  
 +
{{Note box|Also, make sure you have the latest version of Affa installed. At the time of writing this document, it was Affa 1.0.0-2.}}
  
=== NOTE! ===
+
===User experience===
 
+
In my case I saved ONE THIRD of the backup diskspace on one single backup alone. And what is more, FreeDup found more identical files within backups' trees themselves, hence correcting Rsync and saving me terabytes of space.
Please be aware that once you launch FreeDup, it can take several hours to complete, depending on the size of backups as it its algorithm is making sure that files that are to be hardlinked are the same. My backups
 
 
 
Also, make sure you have the latest version of Affa, at the time of writing this document, it is this one:
 
 
 
http://neddix.com/sme7/smeserver-affa-1.0.0-2.el4.sme.noarch.rpm
 
 
 
 
 
=== Document Maintainer ===
 
  
Robert Gruber
+
Without FreeDup's hardlinks my current off-site backup would be 3.4TB, but instead it is only 270MB. That is 12 backups in total - 3x monthly, 4x weekly and 5x daily backups + the current backup running. - [[User:Giskaard | Giskaard]] ([[User_talk:Giskaard|talk]] | [[ Special:Contributions/Giskaard| contribs]])
  
 +
=== References ===
 +
* Wikipedia article http://en.wikipedia.org/wiki/Freedup
 +
* FreeDup homepage http://freedup.org
 +
* Wiki article [[Affa]]
 +
----
 
[[Category:Backup]]
 
[[Category:Backup]]

Revision as of 07:44, 6 March 2016

Version

Contrib 10:
Contrib 9:
freedup
The latest version of freedup is available in the SME repository, click on the version number(s) for more information.


Purpose

There are situations in backup process that Affa (with or without Rsync) can't handle well.

Imagine backing up user files in a company where documents are distributed on regular basis. It is very likely that all of your users will have a copy of the same file in their documents.

Freedup walks through the file trees (directories) you specify. When it finds two identical files on the same device, it hard links them together. In this case two or more files still exist in their respective directories, but only one copy of the data is stored on disk; both directory entries point to the same data blocks.

Rsync partially does this, but it won't search for identical files within your backup. This software does just that.

This document describes the installation and configuration options of the freedup in Affa's post job for the purpose to reclaim free space on your disk occupied by duplicate files.

Installation

SME8

The FreeDup package is in the Dag repository, if you have not installed it already you need to do that first. You need to activate the Dag reposity before installing this contrib.

see dag repository


To install the package issue the following command:

yum install freedup --enablerepo=dag

SME9

To install the package issue the following command:

yum install freedup --enablerepo=smecontribs,smedev

Usage

To use FreeDup with Affa, first install and setup latest version of Affa. Installation instructions for Affa can be found here.

Then add this

freedup -Tv <insert your directory with backups>

into

/usr/lib/affa/postJob.sh

Command line options are beyond the scope of this document, in my case I only use -T to preserve timestamps and -v for some verbosity.

Various Tests

To see how much your backup occupies, you may find these commands below useful.

Find out real, physical space occupied by your backup:

du -sh <insert your directory with backups>

Find out the space occupied by your backup if you weren't using hardlinks (counts sizes many times if hard linked):

du -shl <insert your directory with backups>

Find out what files are hardlinked:

find <insert your directory with backups> -xdev -type f -links +1 -printf '%i %p\n' | sort -n > links.txt
Important.png Note:
Please be aware that once you launch FreeDup, it can take several hours to complete, depending on the size of backups as it its algorithm is making sure that files that are to be hardlinked are the same.



Important.png Note:
Also, make sure you have the latest version of Affa installed. At the time of writing this document, it was Affa 1.0.0-2.


User experience

In my case I saved ONE THIRD of the backup diskspace on one single backup alone. And what is more, FreeDup found more identical files within backups' trees themselves, hence correcting Rsync and saving me terabytes of space.

Without FreeDup's hardlinks my current off-site backup would be 3.4TB, but instead it is only 270MB. That is 12 backups in total - 3x monthly, 4x weekly and 5x daily backups + the current backup running. - Giskaard (talk | contribs)

References