Difference between revisions of "BackupPC"

From SME Server
Jump to navigationJump to search
(10 intermediate revisions by 3 users not shown)
Line 6: Line 6:
  
 
{{#smeversion: smeserver-BackupPC }}
 
{{#smeversion: smeserver-BackupPC }}
 +
{{#smeversion: smeserver-BackupPC4 }}
 
{{#smeversion: BackupPC }}
 
{{#smeversion: BackupPC }}
 +
{{#smeversion: BackupPC4 }}
  
 
=== Description ===
 
=== Description ===
Line 46: Line 48:
  
 
=== Requirements ===
 
=== Requirements ===
*SME Server 7.X, 8.X or 9.X
+
*SME Server 7.x, 8.x, 9.x or 10.x.
*a dedicated volume is recommended (can be a simple disk or a RAID array)
+
*A dedicated volume is recommended (can be a simple disk or a RAID array)
  
 
=== Installation ===
 
=== Installation ===
Line 107: Line 109:
 
*updates configuration files following the installation
 
*updates configuration files following the installation
 
  signal-event backuppc-update
 
  signal-event backuppc-update
 +
 +
====install the rpms (SME 10.X)====
 +
yum --enablerepo=smecontribs,epel,fws install smeserver-BackupPC
 +
 +
*updates configuration files following the installation
 +
signal-event backuppc-update
 +
  
 
You should now be able to access the interface in the server-manager or directly on https://my.domain.tld/BackupPC
 
You should now be able to access the interface in the server-manager or directly on https://my.domain.tld/BackupPC
Line 123: Line 132:
  
 
When you submit, the configuration from smeserver-template will be copied on my_server. You then can edit my_server's config
 
When you submit, the configuration from smeserver-template will be copied on my_server. You then can edit my_server's config
 +
 +
====install the BackupPC4 rpms (SME 9.X)====
 +
Important you can not install both BackupPC (3) and BackupPC4 on the same server.
 +
 +
first the needed repo:
 +
yum install smeserver-extrarepositories-fws smeserver-extrarepositories-epel -y
 +
signal-event yum-modify
 +
then install it:
 +
yum --enablerepo=smecontribs,fws,epel install smeserver-BackupPC4
 +
 +
*updates configuration files following the installation
 +
signal-event backuppc-update
 +
 +
You should now be able to access the interface in the server-manager or directly on https://my.domain.tld/BackupPC
  
 
=== Upgrade from smeserver-backuppc.fws-3.0-1 ===
 
=== Upgrade from smeserver-backuppc.fws-3.0-1 ===
Line 147: Line 170:
  
 
You should be done ;)
 
You should be done ;)
 +
 +
 +
 +
 +
===upgrade from BackupPC (3) to BackupPC4 rpms (SME 9.X)===
 +
TODO.
 +
You will mostly have some debugging to do regarding the bug {{Bugzilla:10514}}
 +
Also according to BackupPC documentaton this should be doable, but might need some manual tweaking in order to use both v3 and v4 pools during the transition delay:
 +
# http://backuppc.sourceforge.net/BackupPC-4.0.0.html#BackupPC-4.0
 +
# https://github.com/backuppc/backuppc/blob/master/bin/BackupPC_migrateV3toV4
 +
# https://www.srbu.se/index.php?option=com_content&view=article&id=210:migrating-backuppc-v3-pools-to-v4&catid=12&Itemid=121
 +
# https://github.com/backuppc/backuppc/issues/249
  
 
=== DB properties ===
 
=== DB properties ===
Line 220: Line 255:
 
  (confirm the host key)
 
  (confirm the host key)
  
 +
*ssh port different from 22
 +
 +
See this post :
 +
https://forums.contribs.org/index.php/topic,53586.0.html
  
 
====Restore or migrate a smeserver====
 
====Restore or migrate a smeserver====
Line 335: Line 374:
  
 
  nano /var/lib/BackupPC/pc/<client>/backups
 
  nano /var/lib/BackupPC/pc/<client>/backups
With ctrl+k you can remove the line with <backup_number>.
+
With ctrl+k you can remove the line with <backup_number>.
  
 
Relog to your web interface and voila! No more <backup_number>.
 
Relog to your web interface and voila! No more <backup_number>.
Line 368: Line 407:
 
One other way could be to set in the setting of backupPC (RsyncClientCmd)the default value for all hosts
 
One other way could be to set in the setting of backupPC (RsyncClientCmd)the default value for all hosts
 
  $Conf{RsyncClientCmd} = '$sshPath -o ServerAliveInterval=120 -q -x -l root $host $rsyncPath $argList+';
 
  $Conf{RsyncClientCmd} = '$sshPath -o ServerAliveInterval=120 -q -x -l root $host $rsyncPath $argList+';
 +
 +
* trying backing up a SME Server you might encounter error or timeout on ssh, this is because of the AutoBlock function on sshd
 +
config setprop sshd AutoBlock disabled
 +
signal-event remoteaccess-update
 +
{{Tip box|if you disabled Autoblock, install denyhosts or fail2ban not to fall security.}}
  
 
=== Uninstall ===
 
=== Uninstall ===

Revision as of 21:18, 27 October 2020


Maintainer

Daniel B. from Firewall Services

Version

Contrib 10:
Contrib 9:
smeserver-BackupPC
The latest version of smeserver-BackupPC is available in the SME repository, click on the version number(s) for more information.


Contrib 9:
smeserver-BackupPC4
The latest version of smeserver-BackupPC4 is available in the SME repository, click on the version number(s) for more information.


Contrib 9:
BackupPC
The latest version of BackupPC is available in the SME repository, click on the version number(s) for more information.


Contrib 10:
BackupPC4
The latest version of BackupPC4 is available in the SME repository, click on the version number(s) for more information.


Description

BackupPC is a powerful, high-performance, enterprise-grade system for backing up a large number of Linux and Windows PCs and laptops to a server's disk. BackupPC is highly configurable and easy to install and maintain.

With the decreasing cost of disks and raid systems, it is practical and cost effective to backup a large number of machines onto a server's local disk or network storage. This is what BackupPC does. For some sites, this might be the complete backup solution. For other sites, additional permanent archives could be created by periodically backing up the server to tape. A variety of Open Source systems are available for doing backup to tape.

BackupPC is written in Perl and extracts backup data via SMB using Samba, tar over ssh/rsh/nfs, or rsync. It is robust, reliable, well documented and freely available as Open Source on SourceForge.

More information & support is available at http://backuppc.sourceforge.net


Features

A clever pooling scheme minimizes disk storage and disk I/O. Identical files across multiple backups of the same or different PCs are stored only once resulting in substantial savings in disk storage and disk I/O.

One example of disk use: 95 laptops with each full backup averaging 3.6GB each, and each incremental averaging about 0.3GB. Storing three weekly full backups and six incremental backups per laptop is around 1200GB of raw data, but because of pooling and compression only 150GB is needed.

Optional compression support further reducing disk storage. Since only new files (not already pooled) need to be compressed, there is only a modest impact on CPU time.

No client-side software is needed. The standard smb protocol is used to extract backup data on Windows clients. On linux clients, tar over ssh/rsh/nfs is used to backup the data. With version 2.0.0, rsync is also supported on any client that has rsync or rysncd.

A powerful web (http/cgi) user interface allows administrators to view log files, configuration, current status and allows users to initiate and cancel backups and browse and restore files from backups.

A full set of restore options is supported, including direct restore (via smbclient, tar, or rsync/rsyncd) or downloading a zip or tar file.

Supports mobile environments where laptops are only intermittently connected to the network and have dynamic IP addresses (DHCP).

Flexible configuration parameters allow multiple backups to be performed in parallel, specification of which shares to backup, which directories to backup or not backup, various schedules for full and incremental backups, schedules for email reminders to users and so on. Configuration parameters can be set system-wide or also on a per-PC basis.

Users are sent periodic email reminders if their PC has not recently been backed up. Email content, timing and policies are configurable.

Tested on Linux, Freenix and Solaris hosts, and Linux, Windows 95, Windows 98, Windows 2000, Windows XP, and Windows Vista clients.

Detailed documentation.

Open Source hosted by SourceForge and freely available under GPL.

Requirements

  • SME Server 7.x, 8.x, 9.x or 10.x.
  • A dedicated volume is recommended (can be a simple disk or a RAID array)

Installation

  • Mount a volume on /var/lib/BackupPC

This part is optional but highly recommended. Backups should always be on a separate volume. First, prepare your hard drive for ext3 fs (you can use any fs which support hardlinks, I always use ext3 as it's very reliable) In this example, I use a simple partition /dev/sdb1, it could also be a RAID array (/dev/md3) or a LVM logical volume (or a combination of RAID and lvm). By the way, if you know how to manage LVM, its usage is also recommended as it will be easier to manage backups data later.

mkfs.ext3 -L BACKUPPC -m 1 /dev/sdb1

Now, we need to create the directory

mkdir -p /var/lib/BackupPC

Add the mount point in fstab. Just add something like:

/dev/sdb1    /var/lib/BackupPC       ext3    noatime 0 0

And check everything is ok

mount -a
mount | grep BackupPC
/dev/sdb1 on /var/lib/BackupPC type ext3 (rw,noatime)

  • create some symlinks

This part is optional but recommended. With these commands, all BackupPC related files (config and logs) will be stored on the BackupPC volume. You'll then be able to swap the BackupPC disk between two servers.

mkdir -p /var/lib/BackupPC/{etc,log}
ln -s /var/lib/BackupPC/etc /etc/BackupPC
ln -s /var/lib/BackupPC/log /var/log/BackupPC
  • configure needed repositories (needed on SME 8.X)


install the rpms (SME 7.X)

You need to configure the EPEL directory as explained on this page: Epel and the sme7contribs as explained here due to SME7 now at EOL: SME8.0_Contribs_QA#Setup

yum --enablerepo=sme7contribs install smeserver-BackupPC
signal-event backuppc-update

install the rpms (SME 8.X)

yum --enablerepo=smecontribs install smeserver-BackupPC
  • remove -N option for Samba >=3.5 from /etc/BackupPC/config.pl ( 3 lines) for SME8
cp /etc/BackupPC/config.pl /etc/BackupPC/config.pl.orig
cat /etc/BackupPC/config.pl.orig|sed 's/ -E -N -d 1/ -E -d 1 /g'>/etc/BackupPC/config.pl

regarding upstream :http://sourceforge.net/apps/mediawiki/backuppc/index.php?title=BackupPC_on_Fedora_Core_9 http://sourceforge.net/apps/mediawiki/backuppc/index.php?title=Troubleshooting and also bugs : bugzilla:7016 and bugzilla:7281

  • updates configuration files following the installation
signal-event backuppc-update

install the rpms (SME 9.X)

yum --enablerepo=smecontribs install smeserver-BackupPC
  • updates configuration files following the installation
signal-event backuppc-update

install the rpms (SME 10.X)

yum --enablerepo=smecontribs,epel,fws install smeserver-BackupPC
  • updates configuration files following the installation
signal-event backuppc-update


You should now be able to access the interface in the server-manager or directly on https://my.domain.tld/BackupPC

You'll find three sample configuration:

  • smeserver-template is a sample config file to backup a SME Server.
  • localhost-template is a sample config file to backup the server itself.
  • windows-template is a sample to backup a windows host using cygwin+rsyncd

To create a host using a template, create a new host, add =<template_name>

For example

myserver=smeserver-template

When you submit, the configuration from smeserver-template will be copied on my_server. You then can edit my_server's config

install the BackupPC4 rpms (SME 9.X)

Important you can not install both BackupPC (3) and BackupPC4 on the same server.

first the needed repo:

yum install smeserver-extrarepositories-fws smeserver-extrarepositories-epel -y
signal-event yum-modify

then install it:

yum --enablerepo=smecontribs,fws,epel install smeserver-BackupPC4
  • updates configuration files following the installation
signal-event backuppc-update

You should now be able to access the interface in the server-manager or directly on https://my.domain.tld/BackupPC

Upgrade from smeserver-backuppc.fws-3.0-1

If you already use the previous version of the contrib, some other steps are needed as the default location of the backups has changed from /opt/backuppc/files to /var/lib/BackupPC

The easiest way is just to symlink the old location to the new one:

ln -s /opt/backuppc/files /var/lib/BackupPC

Then, you can install the new release:

yum --enablerepo=smecontribs install smeserver-BackupPC

Then, just enter the following commands:

mv /etc/BackupPC/config.pl.back /etc/BackupPC/config.pl
mv /etc/BackupPC/hosts.back /etc/BackupPC/hosts
mv /etc/BackupPC/pc/localhost.pl.back /etc/BackupPC/pc/localhost.pl
db configuration delprop backuppc version
usermod -d /var/lib/BackupPC/ -m backuppc
mv /home/e-smith/files/users/backuppc/.ssh/ /var/lib/BackupPC/
signal-event backuppc-update

You should be done ;)



upgrade from BackupPC (3) to BackupPC4 rpms (SME 9.X)

TODO. You will mostly have some debugging to do regarding the bug {{Bugzilla:10514}} Also according to BackupPC documentaton this should be doable, but might need some manual tweaking in order to use both v3 and v4 pools during the transition delay:

  1. http://backuppc.sourceforge.net/BackupPC-4.0.0.html#BackupPC-4.0
  2. https://github.com/backuppc/backuppc/blob/master/bin/BackupPC_migrateV3toV4
  3. https://www.srbu.se/index.php?option=com_content&view=article&id=210:migrating-backuppc-v3-pools-to-v4&catid=12&Itemid=121
  4. https://github.com/backuppc/backuppc/issues/249

DB properties

there's only two properties you can change:

  • status: (enabled|disabled) Do you want the service to start automatically when the server boot ?
  • examples: (enabled|disabled) Do you want the sample configuration files to be added again if they are removed ?
  • sudo: a list of commands backuppc will be able to run as root using sudo. The default is "/usr/bin/rsync,/bin/gtar"

When you have configured this, you should run

signal-event backuppc-update


Important.png Note:
If you make changes to the "sudo" properties, you should then check the sudoers syntax to be sure everything is OK. For this, you should run
visudo

And close the file. If visudo detects an error, it'll tell you.


as an example if you want to be able to wake on lan :

 config setprop backuppc sudo `config getprop backuppc sudo`,/sbin/ether-wake
signal-event backuppc-update

then simply add a script to your prebackup command or your ping command like :


#!/bin/sh
# Set the mac address for each host
host1_mac=<mac address>
host1_name=<host name>
host2_mac=<mac address>
host2_name=<host name>
# How long does the host take to boot
boottime=120
# Which interface to send the WoL packets out of
interface=eth0
# Function to check if the host pings
pingcheck()
{
ping -w 5 -c 2 $1
export pingstatus=$?
}
case $1 in
       host1)
               sudo etherwake -i $interface $host1_mac
               sleep $boottime
               pingcheck $host1_name
               exit $pingstatus
       ;;
       host2)
               sudo etherwake -i $interface $host2_mac
               sleep $boottime
               pingcheck $host2_name
               exit $pingstatus
       ;;
       *)
               echo "Unknown machine $1"
               exit 1
       ;;
esac


NB. If using the default backuppc admin user, you will need to use the full path for the above commands

   Scripts should not be placed in /etc/BackupPC as permissions will be overwritten

Operation

Backup a smeserver

To configure a SME Server client to be backed up using a configuration based on the provided smeserver-template, configure SSH from the backup server as follows:

[root@backupserver ~]# su -s /bin/bash backuppc
bash-3.2$ ssh-keygen -t rsa -b 2048
(used a blank passphrase)
bash-3.2$ cat /var/lib/BackupPC/.ssh/id_rsa.pub | ssh root@myserver.mydomain.com \
"cat >> /root/.ssh/authorized_keys2"
(confirm the host key)
  • ssh port different from 22

See this post : https://forums.contribs.org/index.php/topic,53586.0.html

Restore or migrate a smeserver

If you simply want to restore a few files or a whole directory you just have to use the web interface, and within a few clicks you are done, with your files restored directly in place, or aside if you want.


This will cover the opportunity to use this full or almost full backup to use the restoration option while installing a new manager, which will help you to migrate between versions.

  • change SERVERNAME with the actual name of your backup
  • change -n -1 with the number of the most uptodate fullbackup you want to use. (-1 means relative to the end: last available in this case)
su - backuppc -s /bin/sh
cd ~
mkdir myrestore
cd myrestore/

to include and restore everything that is outside of the usual backuped folder in SME:

/usr/share/BackupPC/bin/BackupPC_tarCreate  -h SERVERNAME -n -1 -s /  / > smeserver.tar

the base backup in the SME Server console would be an equivalent to this :

 /usr/share/BackupPC/bin/BackupPC_tarCreate  -h SERVERNAME -n -1 -s  /    home/e-smith etc/e-smith/templates-custom etc/e-smith/templates-user-custom \
etc/ssh root etc/sudoers  etc/passwd  etc/shadow etc/group etc/gshadow etc/samba/secrets.tdb  etc/samba/smbpasswd > smeserver.tar


then you have just to copy this on a USB disk, or alternatively copy it in a virtual disk if your SME will be virtualized and can not plug any USB device

mkdir /mnt/MyDisk
mount /dev/disk/by-label/MyDisk /mnt/MyDisk
mv /var/lib/BackupPC/myrestore/smeserver.tar /mnt/MyDisk/ 
umount /mnt/MyDisk/ 
  1. install SME server
  2. reboot
  3. plug your drive when you see the screen asking for restoration

and you are done.

Backup a Linux host through rsync

To configure a Linux host client to be backed up, configure SSH from the backup server as follows in order to exchange certificates and to perform a login without passwords.

At the BackupPC server command line as root:

su -s /bin/bash backuppc
ssh-keygen -t rsa -b 2048  # use a blank passphrase
exit

Assuming SSH access to the remote host to be backed up is already configured, as root:

su -s /bin/bash backuppc 
ssh-copy-id -i ~backuppc/.ssh/id_rsa.pub root@your.host.name
exit

You can test the key exchange:

su -s /bin/bash backuppc  #become backuppc user
ssh -l root your.host.name                                                     # add "-p 2222" for port 2222
exit                      #from the remote_host
exit                      #from the backuppc user back to root

You should be able to log in without a password.

Note: There are times when the backuppc user needs a real shell specified in /etc/passwd, for example, when the backuppc user is used to run sudo commands or pre/post scripts.

Now you can go to the admin backuppc panel to set your host

Backup a Windows Client with rsync

To configure a Windows client to be backed up via rsync, install rsync on the Windows client. rysnc needs to be running as a service on the Windows client to ensure communication. There are several options/versions of rsync – the first I was able to get working was from http://sourceforge.net/projects/backuppc/files/cygwin-rsyncd/2.6.8_0/ The zip file contains 7 files – the site goes into more detail about each of the files. Depending on how you wish to use BackupPC will determine which files to edit. I use rsync without ssh keys – to do so create a local user with sufficient rights to the directory or directories on the Windows client, then edit the file rsync.secrets to include the UserID & Password. You then use that same UserID & Password within the configuration from the BackupPC webpage – where you chose to back-up the client with Samba, rsync etc. The contents of rsync.secrets file is simply:

 UserID:Password

You will have to create a “Cygwin share” on the Windows box – that is done using the rsync.conf file. The “Cygwin share” is the name you choose within the square brackets. My rsync.conf file with a share name of “cdocs” (after removal of all personal information of course) looks like this:

[cdocs]
   path = c:/User
   comment = c: Drive documents
   auth users = <valid-back-up-user>
   secrets file = c:/rsyncd/rsyncd.secrets
   hosts allow = 192.168.1.1
   strict modes = false
   read only = true
   list = false

We have share name (cdocs), path, the valid user created on the Windows client, the path to the rysnc.secrets file, an allowable host (this being the local backup server IP), modes, read only and list. The options are mostly explained in the file – what it doesn’t show is that if you have a different version of rsync, the rsync.conf file looks a bit different. This one is for rsync 3.0.9

[cdocs]
  Path = /cygdrive/c/Users/<USERNAME>
  comment = c: User Data
  strict modes = false
  auth users = <valid-back-up-user>
  secrets file = c:/rsyncd/rsyncd.secrets
  hosts allow = 192.168.1.1
  read only = false
  list = false

The final thing that must happen to make rsync work properly is to load rysnc as a service on the Windows client. The file service.bat shows the following command to accomplish this:

c:/rsyncd/rsync.exe -a "--config=c:/rsyncd/rsyncd.conf --daemon --no-detach"

There is also a “gotcha” – sometimes port 873 is blocked by Windows Firewall, so you may need to fix that.

Remove backups

  • Remove all backups of a client
rm -rf /var/lib/BackupPC/pc/<client>/
sudo -u backuppc /usr/share/BackupPC/bin/BackupPC_serverMesg BackupPC_nightly run
  • Remove One backup of a client
cd /var/lib/BackupPC/pc/<client>/
ls
rm -fR <backup_number>
sudo -u backuppc /usr/share/BackupPC/bin/BackupPC_serverMesg BackupPC_nightly run


Warning.png Warning:
Removing a single backup must be done with extreme care. You can safly remove the last backup for example, but if you remove a backup on which another one is based on, you can screw up your backups


After this, you will see space reclaimed and you can check it with df, but you still need to remove statistic line, otherwise you will see it in your web interface. To do that:

nano /var/lib/BackupPC/pc/<client>/backups

With ctrl+k you can remove the line with <backup_number>.

Relog to your web interface and voila! No more <backup_number>.

Known Issues

  • When restarting the BackupPC service you may receive the error as indicated below :

[root@backuppc ~]# service backuppc restart
Shutting down BackupPC:                                    [  OK  ] 
Starting BackupPC:                                         [  OK  ] 
BackupPC: WARNING:  Your BackupPC $Conf{TopDir} is not listed in the locate 
BackupPC: database configuration's PRUNEPATHS.  This may cause all of your 
BackupPC: backed up files to be indexed! 

To remove the warning

you should create a custom template /etc/e-smith/templates-custom/etc/updatedb.conf/10conf (copy the default /etc/e-smith/templates/etc/updatedb.conf/10conf)

mkdir -p /etc/e-smith/templates-custom/etc/updatedb.conf/
cp /etc/e-smith/templates/etc/updatedb.conf/10conf /etc/e-smith/templates-custom/etc/updatedb.conf/

add the directory /var/lib/BackupPC to the PRUNEPATHS list, then

expand-template /etc/updatedb.conf
  • “aborted by signal=PIPE”

Sometime the backup fails with this error, this error is related to the remote ssh service. You must set in the /etc/ssh/ssh_config of your remote host

 ServerAliveInterval 120

then restart your ssh service

One other way could be to set in the setting of backupPC (RsyncClientCmd)the default value for all hosts

$Conf{RsyncClientCmd} = '$sshPath -o ServerAliveInterval=120 -q -x -l root $host $rsyncPath $argList+';
  • trying backing up a SME Server you might encounter error or timeout on ssh, this is because of the AutoBlock function on sshd
config setprop sshd AutoBlock disabled
signal-event remoteaccess-update
Information.png Tip:
if you disabled Autoblock, install denyhosts or fail2ban not to fall security.


Uninstall

yum remove smeserver-BackupPC BackupPC

Bugs

Please raise bugs under the SME-Contribs section in bugzilla and select the smeserver-BackupPC component or use this link


Below is an overview of the current issues for this contrib:

IDProductVersionStatusSummary (4 tasks)
12197SME Contribs10.0CONFIRMEDcleanup /var/log/httpd-bkpc/
12196SME Contribs10.0CONFIRMEDremove old way logrotate
12123SME Contribs10.0CONFIRMEDimport deps from fws
11517SME Contribs10.0rcCONFIRMEDcleanup remove use of sv run script for service

Changelog

Only released version in smecontrib are listed here.

smeserver-BackupPC Changelog: SME 10 (smecontribs)
2023/02/11 Zsolt Vasarhelyi 0.2-15.sme
- expand-updatedb-conf patch [SME: 10513]
2023/02/10 Zsolt Vasarhelyi 0.2-14.sme
- $Conf{TopDir} not in locate configuration's PRUNEPATHS [SME: 10513]

2022/07/31 Jean-Philippe Pialasse 0.2-13.sme
- update to httpd 2.4 syntax [SME: 12036]

including Requires, and bkpc-conf
2022/07/30 Brian Read 0.2-12.sme
- Re-build and link to latest devtools [SME: 11997]
2022/07/25 Zsolt Vasarhelyi 0.2-11.sme
- update to httpd 2.4 syntax [SME: 12036]