Affa

From SME Server
Jump to navigationJump to search
Geographylogo.png Languages: 

English  • français


Maintainer

Michael Weinberger

Description

The main purpose of this affa package is to make a SME 7 Server a dedicated backup box in a few minutes. Affa backs up as many SME servers as you like or any other servers which have sshd running and rsync installed. Once it is configured, Affa runs reliably unattended and sends warning messages in case of any errors.

All backup archives are full backups, as Affa make use of the hardlink technique. Therefore a new full backup only needs disk space for the differences plus the filesystem overhead for the hardlinks and directories (which is typically 2-3%).

Affa is based on the rsync program and supports the rsync --compress option. This allows you to run backups over the internet or VPN. A typical setup is one or more Affa backup servers placed in different locations, which backup the production server(s) over the VPN.

A special feature is the rise option, which allows you to rise the backup server to your production server from a backup archive in case of a dead loss of your production server. The rise is executed within a extremely short time, even with huge amount of data. The rise feature uses hardlinks and therefore does not use up additional disk space.

Affa is a command line tool for system administrators and is intentionally designed without a GUI. Therefore it can be efficiently managed on the console and over slow internet connections.

Affa features at a glance

  • Affa is secure: All transfers and connections are made by using the ssh protocol with public/private key authentication
  • Periodically runs unattended full backups. Only differences to the previous full backup are transferred over the network
  • Using rsync with optional bandwidth limit and compression allows backups over the internet
  • Uses hardlink technique, i.e. physical disk space only needed for the differences between two full backups
  • Keeps a configurable number of scheduled, daily, weekly, monthly and yearly full backup archives
  • The archives are browseable. No need to unpack them first.
  • Backup jobs are started by the cron daemon
  • Interrupted jobs continuing with already transfered data when restarted
  • Backups the default e-smith directories and files, when property SMEServer is set to yes
  • Additional directories and files can be included
  • Directories and files can be excluded from the backup
  • Non-SME server linux systems can be backed up by setting the SMEServer property to no and using an include list
  • Configurable nice level for rsync processes on the backup and source server
  • Optional run of custom programs before and after a job run (e.g. running tape backup)
  • Checks the disk space left after a job run with warning levels strict, normal or risky
  • Extensive checking of failure conditions
  • Sends failure messages to a configurable list of email addresses
  • Sends a warning message, if the backup server runs out of disk space
  • Installs an optional watchdog on the source server in case the backupserver fails
  • Watchdog sends warning, if an expected backup did not run
  • Watchdog sends a daily reminder message, if the error continues unchecked
  • Option to display current status of all jobs showing times of last and next run, size and disk usage
  • Status can be mailed on a daily, weekly or monthly schedule
  • Option to display all existing archives of a job shown date, size, nbr of files and disk usage
  • Option to send (and revoke) the public key to the source server (SME Server only)
  • Option to rise the backup server to a production server from a backup. For SME 7 only
  • The rise feature does not physically move data and therefore is extremly fast and needs (almost) no extra disk space
  • Rise option can be run remotely as the NIC driver configuration of the backup server are preserved
  • Compares installed RPMs on source with backup server. Sends warning message, if not in sync
  • Undo rise option to restore the backup server
  • Configurable via a e-smith style db, with one record for each job and a default record for all jobs
  • Logs to /var/log/affa/JOB.log and /var/log/affa/affa.log with optional debug switch for higher verbosity
  • Log files are rotated weekly, with 5 logs kept

Installation or Update

/usr/bin/yum install --enablerepo=smecontribs smeserver-affa

When you have installed Affa for the first time run the following command to initialize the Affa database 

affa --make-cronjobs

and logout and re-login to the console to take the bash auto-completion (TAB key) in effect.


Important.png Note:
After upgrade from Affa versions <=0.7.0 you need to run the command 
affa --send-key

(w/o any arguments) to generate the ssh known host entries.


Quick start example

You have a SME 7 production server with hostname 'prodbox‘ and IP 10.200.48.1.
Set up a second SME 7 box as your backupserver with hostname 'affabox‘ and IP 10.200.48.2.

  1. log into the 'affabox' and install the packages as described above.
  2. copy the config helper script sample cp /usr/lib/affa/jobconfig-sample.pl /root/prodbox-job.pl
  3. edit /root/prodbox-job.pl and set my $jobname='prodbox'; and 'remoteHostName‘=>'10.200.48.1', optionally, if the remote server port is configured to e.g. 2222 then set 'sshPort'=>2222,
  4. write the configuration (this makes the database entries and sets up the cronjobs) /root/prodbox-job.pl
  5. generate the DSA keys and send the public key to the 'prodbox'
    Important.png Note:
    When initially doing this step, you will need to temporarily enable "Allow secure shell access using standard passwords" on the production server 

    affa --send-key prodbox
  6. run the job manually affa --run prodbox

Configuration

The configuration is stored in an e-smith style database. Use the db command to configure Affa. The jobname is the record key with the type 'job'.
To setup a new job with the name 'prodbox' enter: 

db affa set prodbox job

then set the properties 

db affa setprop prodbox remoteHostName 192.168.1.1
db affa setprop prodbox TimeSchedule '0030,0730,1130,1330,1730,2030'
db affa setprop prodbox Description 'My Production Server'
db affa setprop prodbox status enabled

and so on...

Alternatively you can you use a script as described above in the 'Quick start' chapter.

To verify your work, type: 

db affa show prodbox

Finally set up the cronjobs: 

affa --make-cronjobs

Job configuration properties

Property Value Default Description
remoteHostName FQHN or IP FQHN or IP of the source host (mandatory)
TimeSchedule HHMM,HHMM,... doesn't need to be ordered. At least one time is mandatory
Description text string
scheduledKeep
dailyKeep
weeklyKeep
monthlyKeep
yearlyKeep 		integer >= 1 2
7
4
12
2
 how many of the scheduled, daily, weekly, monthly or yearly backups should be kept
SMEServer yes or no yes when set to yes the default e-smith directories are automatically included and the property RPMCheck=yes can be used
Include[0]
Include[1]
... 		full path additional files or directories to include
Exclude[0]
Exclude[1]
... 		full path additional files or directories exclude from backup
RPMCheck yes or no no Only applicable to jobs that backups a SME 7 server.
Compares the packages installation of the source host with this affa backup host. Sends a message with diff list if not in sync. This check is usefull, if you want have the option to rise the backup server to a production server from a backup.
DiskSpaceWarn strict or normal or risky or none strict run a disk space check after a job has been completed. With level 'strict' a warning message will be sent, if the available space is less then the size of the just completed backup. With level 'normal'/'risky' the message is sent, if less than 50%/10% of the backup size is still available.
localNice -19...+19 0 run rsync local process niced.
remoteNice -19...+19 0 run rsync process on source niced.
Watchdog yes or no yes Only applicable to jobs that backups a SME 7 server.
When a job is started, affa installs a watchdog script on the source in /etc/cron.d/, which sends a warning message, if the next scheduled job (taken from the TimeSchedule property + 10 minutes) did not run. This guarantees, that you will be notfied even in case of a affa server outage. The watchdog script send a daily reminder message, if the error continues. The next run job replaces the watchdog script with a new trigger time.
sshPort service port 22 When sshd on the source host or your firewall listen on a non-standard port set the port here.
ConnectionCheckTimeout seconds 120 before the rsync process is started on the remote source host, affa checks the ssh connection and exits with an error after the configured time, if the host does not respond.
rsyncTimeout seconds 900 Rsync exits, if no data is transferred for the configured time. This avoids infinitely hanging in case of a network error.
BandwidthLimit integer>=0 kilobytes per second 0 Limits the data transfer rate. A value of zero specifies no limit.
rsyncCompress yes or no no compress the transferred data. May be useful with slow internet connections. Increases CPU load on source and backup host.
EmailAddresses name@domain.com,name@domain.com,... admin comma separated list of mail addresses, where the messages should be sent to
Note: By default Affa only sends messages on errors, never on success (see property chattyOnSuccess).
chattyOnSuccess integer >= 0 0 when set to a value>0, Affa sends a message on a successfully completed job run and decrements the chattyOnSuccess value. When the value has reached zero, Affa falls back to the default and only sends messages on errors.
AutomountDevice
AutomountPoint 		full path Device and mountpoint of backup device (e.g. USB disk). Device is automounted before a job starts and unmounted after job completion. With both properties empty no automount is done.
AutomountOptions string An option string passed to the mount command
AutoUnmount yes or no yes When set to 'no' the automounted device stay mounted after the Affa run.
preJobCommand
postJobCommand 		full path programs (local on the affa server) to be executed before/after a job run. The job name and type (scheduled, daily etc.) are passed as arguments to the program. The exit code is additionally passed to the post job command program. See /usr/lib/affa/ for sample perl scripts.
RootDir full path /var/affa where to store the backup archives, Do not use /home/e-smith or /root as these are included in the backup and therefore the rise option will not work! Recommended: /var/affa
Debug yes or no no set to yes to increase log verbosity
status enabled or disabled enabled with set to disabled, no cron entries will made. You can still run a job manually.
rsync--inplace yes or no yes set to no, if the rsync version on the source does not support this option (like rsync on SME6)
rsync--modify-window integer >= 0 0 When comparing two timestamps, rsync treats the timestamps as being equal if they differ by no more than the modify-window value. This is normally 0 for an exact match. A value >= 0 is useful if you can't get the clocks of the source and the Affa server in sync.
rsyncdMode yes or no no set to yes to connect to the rsync daemon on the remote host (instead of running rsync over ssh)
rsyncdModule string AFFA the rsyncd module name (only applicable with rsyncdMode=yes)
rsyncdUser string affa the username for authentication to the rsync daemon (only applicable with rsyncdMode=yes)
rsyncdPassword string the password for authentication to the rsync daemon (only applicable with rsyncdMode=yes)
remoteOS cygwin with remoteOS=cygwin the options --send-key and --revoke-key uses the account 'Administrator' and the correct path for the public key on a Windows/Cygwin remote host.

Default configuration properties

All properties can be set as defaults in the DefaultAffaConfig record. This is useful, when you set up many similar jobs. Example: You want to set the property 'localNice' to 19 for all jobs. Then run 

db affa setprop DefaultAffaConfig localNice 19

and don't set this property for the jobs. Properties set in the job record overrides the defaults.

The special property 'sendStatus' is only applicable to the DefaultAffaConfig record. It controls the status report sent by email and can be set to the values 'none', 'daily', 'weekly' or 'monthly'. To setup a weekly status report run: 

db affa setprop DefaultAffaConfig sendStatus weekly

then setup the cronjob: 

affa --make-cronjobs

Global disable

All jobs can be disabled with setting the AffaGlobalDisable record type to 'yes'. 

db affa set AffaGlobalDisable yes
affa --make-cronjobs

to re-enable run: 

db affa set AffaGlobalDisable no
affa --make-cronjobs

Usage and command line options

Important.png Note:
Options can be abbreviated to uniqueness, e.g. --mak is equal to --make-cronjobs 


affa --run JOB

Starts a job run. Usually done by the cronjob. 

affa --make-cronjobs

Configures the cronjobs as scheduled in the jobs records. 

affa --send-key JOB

affa --send-key --host=TARGETHOST [--port=PORT] [--remoteOS=cygwin]

This first generates the DSA key for the Affa Server, if not already done. Then it sends the public key to the host 'remoteHostName' as configured in the record of job JOB and generates the job specific ssh known host entry.

Important.png Note:
When initially doing this step, you will need to temporarily enable "Allow secure shell access using standard passwords" on the production server.


Important.png Note:
By default, the --send-key option works for a SME Server as a remote server and for systems where the keys are stored in /root/.ssh/authorized_keys2 and the commands /bin/cat, /bin/touch, /bin/grep and /bin/mv are available. With remoteOS=cygwin it works for a Cygwin/Windows remote server. 


affa --full-restore JOB [ARCHIVE]

Does a full restore of the standard files and directories from the backup ARCHIVE on the remote source server as defined in the JOB record. If ARCHIVE is not given, the archive 'scheduled.0' is used as the default. The full restore reconstructs the server as it was at the time of the backup. After the restore the source host reboots. 


affa --rise [--all] JOB [ARCHIVE]

Runs a full restore on the Affa server (!) of all standard files and directories from the backup ARCHIVE of job JOB. In other words: After completion, the Affa box reboots as a clone of the source server. Ensure, that the source server has been powered off before you reboot the Affa box, otherwise the network interface will not come up. This is important, when you run --rise remotely. The --rise feature only works with SME 7 servers und should only be used on dedicated backup servers.

With option --all, all files and directories of the archive as defined by the include[] properties are restored. Files or directories with the same name on the Affa server will be overwritten and cannot be restored by a undorise. This should not be an issue on a dedicated Affa server which does not hold any other data. After a possible undorise those additional restored dada must be removed manually.

Please note, that the rise process backs up the the Affa server itself before doing the restore from the archive. This backup is used by a possible undorise run to restore the Affa server. Only the standard files and directories are backed up. Data in non-standard loctions (like /opt) are untouched and will still exist after the rise run. See also: Backup_server_config#Standard_backup_.26_restore_inclusions 


affa --create-backup-file JOB [ARCHIVE] [--outfile=FILE]

Creates a gzipped tar archive FILE from the ARCHIVE of job JOB. During creation the MD5 checksum is calculated against which the written tar is checked. The default ARCHIVE is scheduled.0 and default the FILE is ./smeserver.tgz. The checksum is written to FILE.md5sum.

A smeserver.tgz file stored on an USB disk can be used for restore during a SME Server fresh install. 

affa --list-archives [--csv] JOB

Displays a table of all present archives of job JOB with date, number of files, size and disk usage. See chapter 'Restore' for an output example. With --csv, the output is in machine readable colon separared format. 

affa --status [--csv]

Displays a table of all configured jobs with enable status, time of last and next run, size, disk usage and the number of scheduled (s), daily (d), weekly (w), monthly (m) and yearly (y) archives. Last time shows 'failed', if a job did not run in the last 24h. For disabled jobs 'Last' always shows 'failed' after 24 h. To see the date and time of the last run of those jobs use the --list-archives option. Column 'Next' shows the time when the next run will be started, if 'Enabled' is 'yes'. Column 'Disk usage' shows the usage of the partion, where the RootDir of the job is located on. If all jobs are located in the same RootDir, identical disk usage is shown for all jobs. If RootDir is /var/affa, the usage of the SME server system partition is shown. 

Affa version 1.0.0-1 on backup.mydomain.de (10.204.48.2)
+---------------+------+--------+-------+--------+------------+----------------+
| Job           | ENBL | Last   | Next  |   Size | Disk usage | N of s,d,w,m,y |
+---------------+------+--------+-------+--------+------------+----------------+
| bookkeep      | yes  | 23:48  | 23:30 |  4.5GB |  616GB/47% | 2,7,4,2,0      |
| crm           | yes  | 11:20  | 13:20 |   47MB |  616GB/47% | 7,7,4,2,0      |
| fespdc        | yes  | 09:55  | 12:45 |   40GB |  616GB/47% | 6,7,4,2,0      |
| helpdesk      | yes  | failed | 13:40 |   70MB |  616GB/47% | 7,7,4,2,0      |
| imageserv     | yes  | 23:01  | 23:00 |   19GB |  616GB/47% | 2,7,4,1,0      |
| intraweb      | yes  | 10:32  | 13:30 |  1.4GB |  616GB/47% | 7,7,4,2,0      |
| pdcaus2       | yes  | 23:01  | 12:15 |  5.4GB |  616GB/47% | 2,7,4,2,0      |
| persoff       | yes  | running (pid 17521)     |  616GB/47% | 2,7,4,2,0      |
| primmail      | yes  | 10:09  | 13:00 |   45GB |  616GB/47% | 7,7,4,2,0      |
| rayofhope     | yes  | 22:32  | 22:30 |   20GB |  616GB/47% | 2,7,4,1,0      |
| sozserv       | yes  | 22:30  | 22:30 |  8.0GB |  616GB/47% | 2,7,4,2,0      |
+---------------+------+--------+-------+--------+------------+----------------+
| az32share     | no   | -      | 02:20 |  234MB |  616GB/47% | 7,7,2,0,0      |
| azpdc         | no   | -      | 21:00 |   42GB |  616GB/47% | 2,7,1,0,0      |
+---------------+------+--------+-------+--------+------------+----------------+

With --csv, the output is printed in a machine readable colon separated format. 


affa --show-schedule

Prints a 'graphical' timetable for all enabled jobs. The resolution is 30 minutes. Disabled jobs are not shown. 

Affa version 1.0.0-1 on backup.mydomain.de (10.204.48.2)
     TIME 0:00     4:00     8:00     12:00    16:00    20:00   
      crm -X------ -------- -------- -------- -------- --------
  sozserv --X----- -------- -------- -------- -------- --------
   fespdc ----X--- -------- -------- -------- -------- --------
imageserv -----X-- -------- -------- -------- -------- --------
 primmail ------X- -------- -------- -------- -------- --------
 intraweb -------- X------- -------- -------- -------- --------
 bookkeep -------- ------X- X-X-X-X- X-X-X-X- X-X-X-X- X-X-X---
  pdcaus2 -------- -------- -------- X------- -------- ------X-
  persoff -------- -------- -------- -------- -------- ---X----
 helpdesk -------- -------- -------- -------- -------- -----X--
rayofhope -------- -------- -------- -------- -------- -----X--
2 disabled jobs not listed



affa --send-status

Sends the status table to the email adresses configured in the 'DefaultAffaConfig' record. Used by the cronjob 'affa-status'. 

affa --mailtest JOB

Sends a test email to the email adresses configured in the JOB record. With property Watchdog=yes, a test email is sent from the remote host, too. Use this to verify, that your mail processing is functional.
Note: By default Affa only sends messages on errors, never on success (see property chattyOnSuccess). 

affa --cleanup JOB

After you have lowered a keep value, e.g. scheduledKeep, then archives with a higher indices will no longer be shifted and will exist for ever. This option finds these archives and deletes them. 

affa --rename-job JOB NEWNAME

Renames the job JOB to NEWNAME including all database records and archive directories. 

affa --move-archive JOB NEWROOTDIR

Moves the archive directory of job JOB to the rootdir NEWROOTDIR and adjusts the property RootDir. NEWROOTDIR must be a full path starting with a slash. As moving across filesystems (e.g. from an external USB drive to the local disk) is not possible, Affa uses a copy command in this case and deletes the source directory after that. Depending on the archive size, copying across filesystems can take a long time. 

affa --delete-job [--revoke-key] JOB

Irreversibly deletes a job including all archives, configuration and report databases. With given --revoke-key option, the public key on the remote server will be deleted. 

affa --revoke-key JOB

affa --revoke-key --host=TARGETHOST [--port=PORT] [--remoteOS=cygwin]

Deletes the public dsa key on the remote server.

Important.png Note:
By default, the --send-key option works for a SME Server as a remote server and for systems where the keys are stored in /root/.ssh/authorized_keys2. With remoteOS=cygwin it works for a Cygwin/Windows remote server. 


affa --check-connections

Checks the ssh login for all configured jobs. For jobs where the public key was not yet sent, you are prompted for the password and the key will be sent then. 

affa --kill JOB

Terminates the running job JOB and all its child processes (rsync processes).

Example setups

[Todo] Standard

Dedicated Affa server to backup all production servers
...


[Todo] Local Affa server plus a Affa server in remote location

meanwhile see forum post

Backup single ibays

Suppose you want to backup the ibays 'staff1' and 'staff2' on your production server with WAN IP 82.123.1.1 to an Affa server in a different location over the internet every night at 2:30am.

  1. log into the Affa server and install the packages as described above.
  2. copy the config helper script sample cp /usr/lib/affa/jobconfig-sample.pl /root/ibay-staff-job.pl
  3. edit /root/ibay-staff-job.pl and set my $jobname='ibay-staff'; and 'remoteHostName‘=>'82.123.1.1', 'TimeSchedule'=>'0230', 'SMEServer'=>'no', 'Include[0]'=>'/home/e-smith/files/ibays/staff1', 'Include[1]'=>'/home/e-smith/files/ibays/staff2',
  4. write the configuration /root/ibay-staff-job.pl
  5. send the public key to the production server affa --send-key ibay-staff
  6. check next morning affa --list-archives ibay-staff affa --status ls /var/affa/ibay-staff

[Todo] Two production servers backup each other

...


Backing up a Windows computer

Backing up data from a Windows system requires the Cygwin Rsyncd daemon installed and configured on Windows. The standard procedure rsync over ssh does not work, as the Cygwin rsync process always hangs after some files were transferred.

Rsyncd setup on the Windows computer

Install the Cygwin base, the rsync package and configure the Rsyncd service as described in this document: Rsyncd setup on a windows computer for use with Affa backup'. The installation of the sshd service is optional and not needed for the backup itself, but having a ssh login can be very helpful for administration or executing scripts on the Windows system. Affa supports sending the public key to a Windows Cygwin for password-less login.

Note: Affa does not backup the Windows Access Control List (ACL) information. You may need to correct the ACLs manually after a restore.

Affa Rsyncd mode setup (Quick start example)

You want to backup the My Documents folders of the users ssorglos and bsimpson from the Windows computer 'ws001' with IP 192.168.1.65 to your Affa server 'affabox' with IP 192.168.1.3.

1. log into the 'affabox' and copy the Cygwin config helper script sample 

cp /usr/lib/affa/jobconfig-cygwin-sample.pl /root/ws001-mydocs-job.pl

2. edit /root/ws001-mydocs-job.pl and set 

my $jobname='ws001-mydocs';

and 

'remoteHostName‘=>'192.168.1.65',
'rsyncdPassword'=>'secretword',
'Include[0]'=>'/c/Documents and Settings/ssorglos/My Documents/', # don't use backslashes in pathnames!
'Include[1]'=>'/c/Documents and Settings/bsimpson/My Documents/',

where secretword must be replaced by the password you have chosen in the rsyncd.secretsfile on the Windows box.

3. write the configuration (this makes the database entries and sets up the cronjobs) 

/root/ws001-mydocs-job.pl

4. run the job manually. After completion check the archive /var/affa/ws001-mydocs/scheduled.0 and the logfile /var/log/affa/ws001-mydocs.log 

affa --run ws001-mydocs
Affa Rsyncd mode manual setup

In case you want to do the setup manually using the db command, these are the mandatory settings for Cygwin Rsyncd mode 

db affa setprop JOB rsyncdMode yes
db affa setprop JOB rsyncdModule AFFA
db affa setprop JOB rsyncdUser affa
db affa setprop JOB rsyncdPassword secretword
db affa setprop JOB SMEServer no
db affa setprop JOB RPMCheck no
db affa setprop JOB Watchdog no

Optional for ssh login 

db affa setprop JOB remoteOS cygwin


Use Affa to backup to a NFS-mounted NAS or a local attached USB drive

You want to backup your SME 7 production server with hostname 'prodbox‘ and IP 10.200.48.1 on a mounted filesystem instead of setting up a dedicated Affa box.

Setup NAS

You have a FreeNAS box with IP 10.200.48.2 up and running with NFS service enabled. The disk is mounted to /mnt/affashare. You have authorized the IP address of your prodbox server to access share /mnt/affashare.

  1. log into the 'prodbox' and install the NFS packages /usr/bin/yum install --enablerepo=smecontribs smeserver-nfs Now enable and start the portmapper service config setprop portmap status enabled service portmap start
  2. mount the NFS share mkdir -p /mnt/affadevice mount 10.200.48.2:/mnt/affashare /mnt/affadevice
Alternatively setup a USB drive
  1. log into the 'prodbox'
  2. Connect a USB hard disk to the USB Bus. Now you must determine what device the kernel has assigned to the drive. View the /var/log/message and search for Initializing USB Mass Storage driver. A few lines below you'll find the name of the device. In this example it is sdh. Replace /dev/sdh by your device in following instructions.
  3. Use the fdisk program to create a linux partition. Verify that this is really the attached USB drive before you continue! fdisk /dev/sdh You'll most likely find an existing vfat dos partition, which you have to delete first. In the following we assume, that you have created a single partition /dev/sdh1.
  4. Now format the drive with an ext3 filesystem mkfs.ext3 /dev/sdh1
  5. Make the mount point mkdir -p /mnt/affadevice
  6. Add the following line to the /etc/fstab /dev/sdh1 /mnt/affadevice ext3 defaults
  7. Mount the drive mount /mnt/affadevice
  8. Crosscheck your work using the df command df
Setup Affa

You want to run backups on 11:30 h, 15:30 h and 19:30 h and you want to keep the last 3 scheduled backups, 7 daily, 5 weekly, 12 monthly and 1 yearly backups.

  1. log into the 'prodbox' and install the Affa packages as described above.
  2. copy the config helper script sample cp /usr/lib/affa/jobconfig-sample.pl /root/prodbox-job.pl
  3. edit /root/prodbox-job.pl and set my $jobname='prodbox'; and 'remoteHostName‘=>'localhost', 'TimeSchedule'=>'1130,1530,1930', 'scheduledKeep'=>3, 'dailyKeep'=>7, 'weeklyKeep'=>5, 'monthlyKeep'=>12, 'yearlyKeep'=>1, 'RootDir=>'/mnt/affadevice', Review the other properties and change them to your needs.
  4. write the configuration /root/prodbox-job.pl
  5. run the job manually affa --run prodbox
Limitations

With this kind of setup you cannot use the affa rise feature, as it requires the backup archive to be located on the same fileystem as the server installation. The rise option uses hardlinks, which are not working across filesystems.

Automount

Having the backup archives in the same filesystem is always a risk, which can be minimized by using the automount feature. Then the external filesystem is only mounted during a job run.

In the NAS example set 

'AutomountDevice=>'10.200.48.2:/mnt/affashare',
'AutomountPoint =>'mnt/affadevice',

and skip the step 2.

In the USB drive example set 

'AutomountDevice=>'/dev/sdc1',
'AutomountPoint =>'mnt/affadevice',

and skip the steps 5 to 8.

The mount point will be automatically created, if it does not exist.
For access to the archive directory, you need to mount it manually.

Restore

Restore single files or directories

Example 1: It's Tuesday April 8th 2008, when user 'briedlin' asks you to restore the messages of his mailbox 'orders' he has inadvertently deleted on Monday morning.

  1. You first must check what backup archives are available. The jobname of this server backup is 'prodserv'. On the Affa backup server do: affa --list-archives prodserv 
    Affa version 1.0.0-1 on affa1.mydomain.de (10.204.48.2)
+------------------------------------------------------------------------------+
| Job: prodserv                                                                |
| Description: File- and Mailserver Alfdorf                                    |
| Directory: /var/affa/prodserv/                                               |
| Hostname: 10.204.48.1                                                        |
| Email: sysadmin@mydomain.de                                                  |
+-----------------------+----------------+--------------+--------+-------------+
| Date                  | Archive        |        Files |   Size | Disk usage  |
+-----------------------+----------------+--------------+--------+-------------+
| Sat 2007 Apr 28 00:45 | monthly.10     |        39625 |  7.7GB |    118GB/9% |
| Sat 2007 May 26 00:41 | monthly.9      |        39928 |  7.8GB |     88GB/6% |
| Sat 2007 Jun 30 00:41 | monthly.8      |        40343 |  7.8GB |   155GB/11% |
| Fri 2007 Jul 27 00:41 | monthly.7      |        39530 |  7.9GB |   172GB/13% |
| Sat 2007 Aug 25 00:41 | monthly.6      |        39613 |  7.9GB |   179GB/13% |
| Sat 2007 Sep 29 00:42 | monthly.5      |        40157 |  8.2GB |   198GB/15% |
| Sat 2007 Oct 27 00:46 | monthly.4      |        40453 |  7.6GB |   213GB/16% |
| Sat 2007 Dec 01 00:43 | monthly.3      |        40545 |  7.6GB |   228GB/17% |
| Sun 2007 Dec 30 00:42 | monthly.2      |        40378 |  7.6GB |   284GB/21% |
| Sun 2008 Jan 27 00:44 | monthly.1      |        40545 |  7.5GB |   422GB/32% |
| Sun 2008 Mar 02 00:42 | monthly.0      |        40924 |  7.7GB |   479GB/36% |
+-----------------------+----------------+--------------+--------+-------------+
| Sun 2008 Mar 09 00:41 | weekly.3       |        40976 |  7.7GB |   501GB/38% |
| Sun 2008 Mar 16 00:42 | weekly.2       |        41027 |  7.7GB |   506GB/38% |
| Sun 2008 Mar 23 00:41 | weekly.1       |        41051 |  7.7GB |   500GB/38% |
| Sun 2008 Mar 30 00:41 | weekly.0       |        41062 |  7.7GB |   503GB/38% |
+-----------------------+----------------+--------------+--------+-------------+
| Tue 2008 Apr 01 00:58 | daily.6        |        41090 |  7.7GB |   500GB/38% |
| Wed 2008 Apr 02 00:43 | daily.5        |        47680 |  7.8GB |   501GB/38% |
| Thu 2008 Apr 03 00:46 | daily.4        |        47680 |  7.8GB |   507GB/38% |
| Fri 2008 Apr 04 00:42 | daily.3        |        47686 |  7.8GB |   513GB/39% |
| Sat 2008 Apr 05 00:45 | daily.2        |        47687 |  7.8GB |   512GB/39% |
| Sun 2008 Apr 06 00:42 | daily.1        |        47687 |  7.8GB |   511GB/39% |
| Mon 2008 Apr 07 00:42 | daily.0        |        47687 |  7.8GB |   511GB/39% |
+-----------------------+----------------+--------------+--------+-------------+
| Tue 2008 Apr 08 00:43 | scheduled.0    |        47688 |  7.8GB |   525GB/40% |
+-----------------------+----------------+--------------+--------+-------------+
  2. Choose the daily.0 archive, which was created Monday night. Now restore the mailbox 'orders' using the rsync command.
  3. Now run the rsync command (note the trailing slash!) on the Affa backup server: export RDIR=/home/e-smith/files/users/briedlin/Maildir/.orders/ rsync -av /var/affa/prodserv/daily.0/$RDIR 10.204.48.1:$RDIR
    4. If the servers are configured to use a different ssh port eg 2222, then instead do: export RDIR=/home/e-smith/files/users/briedlin/Maildir/.orders/ rsync -av -e 'ssh -p 2222' /var/affa/prodserv/daily.0/$RDIR 10.204.48.1:$RDIR


Example 2: A user has deleted the file orderform.pdf from ibay 'docs' and asks you to restore it.

  1. You have searched and found the latest version of this file in archive weekly.
  2. Copy it back to the server - On the Affa backup server do: export RFILE=/home/e-smith/files/ibays/docs/files/orderform.pdf rsync -av /var/affa/prodserv/weekly.1/$RFILE 10.204.48.1:$RFILE
    3. If the servers are configured to use a different ssh port eg 2222, then instead do: export RFILE=/home/e-smith/files/ibays/docs/files/orderform.pdf rsync -av -e 'ssh -p 2222' /var/affa/prodserv/weekly.1/$RFILE 10.204.48.1:$RFILE


Example 3: Restoring a file with special characters in the path

Do not use shell variables to shorten the command. It would complicate things more than it would help. Quote the source and destination path. In the destination path escape blank characters with a double backslash and brackets with a single backslash. On the Affa backup server do: 

rsync -av "/var/affa/fileshare/daily.4/home/e-smith/files/ibays/mechfiles/files/Valve Control (Design Gr)/VALVE LIST FOR ISSUED.xls" "192.168.1.7:/home/e-smith/files/ibays/mechfiles/files/Valve\\ Control\\ \(Design\\ Gr\)/VALVE\\ LIST\\ FOR\\ ISSUED.xls"

If the servers are configured to use a different ssh port eg 2222, then instead do: 

rsync -av -e 'ssh -p 2222' "/var/affa/fileshare/daily.4/home/e-smith/files/ibays/mechfiles/files/Valve Control (Design Gr)/VALVE LIST FOR ISSUED.xls" "192.168.1.7:/home/e-smith/files/ibays/mechfiles/files/Valve\\ Control\\ \(Design\\ Gr\)/VALVE\\ LIST\\ FOR\\ ISSUED.xls"

Full restore

To run a full restore of user and configuration data run on the Affa server 

affa --full-restore <JOB> [<ARCHIVE>]

This rsyncs the data from the backup ARCHIVE back to the 'remoteHostname' defined in the configuration of the job JOB.

Example: You have backuped your production server 'prodsrv' as job 'prodbox'. To restore from the latest backup run 

affa --full-restore prodbox

To restore from the older archive daily.3 run 

affa --full-restore prodbox daily.3


Warning.png Warning:
A full restore reconstructs the server as it was at the time of the backup. That means, that all files created or server configuration changes made after the backup will be lost. After the restore is done, the restored server reboots automatically.



Moving a SME 7 server installation to new hardware using the Affa rise feature

Please see this Howto: Moving SME to new Hardware

FAQ

Can I use Affa to backup a SME 6 server?
To backup a SME 6 server set the property 'rsync--inplace' to 'no' and install the perl-TimeDate package on the SME 6 box. The perl-TimeDate package is needed by the watchdog script running on the SME 6. Use the RPM from DAG: perl-TimeDate-1.16-0.rh73.dag.noarch.rpm
You cannot use the RPMCheck property.


How can I move a backup archive from my production server in a remote branch office to the Affa server using an USB disk? I want to avoid to download the first full backup over the internet. Install Affa on your production server, connect and mount the USB disk. Setup a job to write the archive to the USB disk. After the job run, connect the USB disk to your Affa server and setup the job in the same way. Use the setup script from the archive directory. You only need to change the remoteHostName property. Now use the --move-archive option to move the archive to the local disk. If not already done, send the public key to your production server. You're done.


How do I backup two SME servers behind a firewall? First you need to configure port forwardings for the ssh service on your firewall. Use a non-standard port ,e.g 2200->22, for your second server. Setup a job on your Affa server for each of your production servers. Set up the job property sshPort=2200 for second server.

Uninstall

This removes the installed Affa package, all configuration data and all backup archives.

Run the following commands for each job: 

affa --revoke-key JOBNAME
affa --delete-job JOBNAME

Verify that all jobs have been deleted 

affa --status

Remvove the Affa package 

rpm -e smeserver-affa

and, if perl-Filesys-DiskFree is not needed by any other packages: 

rpm -e perl-Filesys-DiskFree

Cleaning up 

rm -f /etc/cron.d/affa /etc/cron.d/affa-status
rm -rf /home/e-smith/db/affa /home/e-smith/db/affa-report
rm -rf /var/affa /var/log/affa

Additional information

Performance

It is hard to predict how much time a backup job needs to complete. It depends on the number of files, the total file size, the file changes since last run, the network speed and not least on the CPU power, disk speed and RAM of the source and backup server. The following table of measured values will give you an idea of what you can expect.

Backup server Source server Data on source server Transferred Data Connection Compression Affa run time
2x3.2GHz Xeon
2 GB RAM, 1.5 TB RAID6 		2x3.2GHz Dual Core Xeon
4 GB RAM, RAID5, SME 7.1
Intranet Web Server + MySQL		 1.4 GB, 12,000 files 300 MB, 16 files Internet 2 Mbit yes 2 minutes
2x3.2GHz Xeon
2 GB RAM, 1.5 TB RAID6 		2x3.2GHz Dual Core Xeon
4 GB RAM, RAID5
SME 7.1 Mailserver		 43 GB, 410,000 files 140 MB, 2,700 files Internet 2 Mbit yes 10 minutes
2x2GHz Dual Core Xeon 5130
6 GB RAM, 1 TB RAID 5 		2x2GHz Dual Core Xeon 5130
6 GB RAM, 1 TB RAID 5
SME 7.1 File- and Mailserver		 125 GB, 98,000 files 3,2 GB, 3,000 files Gbit LAN no 25 minutes
2x2GHz Dual Core Xeon 5130
6 GB RAM, 874 GB RAID 5 		2x2GHz Dual Core Xeon 5130
6 GB RAM, 1 TB RAID 5
SME 7.1 File- and Mailserver		 125 GB, 98,000 files 3.5 GB, 2000 files Internet 2 Mbit yes 17 minutes
2x800MHz Pentium 3
1 GB RAM, 300 GB RAID1 		2x2.8GHz Xeon,
1GB RAM, 140 GB RAID5
SME 7.1 File- and Mailserver		 39 GB, 370,000 files 12 GB, 4,000 files 100Mbit LAN no 52 minutes
1xP4 2.4GHz
256 MB RAM
SME 7.1 		2xP4 1GHz
1 GB RAM
SME 6		 7.4 GB, 134,790 files 7.4 GB, 134,790 files 100Mbit LAN no 35 minutes

Note: The last action of a job run is to remove the oldest backup, e.g. if archive scheduled.11 exists and you have set the scheduledKeep property to 12, then it must be deleted. This can take a significantly long time, which increases the total job execution time.

A affa --rise run on a SME 7.2, 40 GB data, 411.500 files, 2x2.8 Xeon CPU, 1GB RAM, SW-RAID 5 with 4 U320 SCSI drives took 7 minutes including the final reboot.

Bug report

Affa is listed in the bugtracker contribs section. Please report all bugs, new feature requests and documentation issues there.

Changelog

* Wed Apr 23 2008 Michael Weinberger 1.0.0-2
  Bug fix #4210, command "affa --show-schedule" executes postJobCommand
* Fri Apr 04 2008 Michael Weinberger 1.0.0-1
  Final release 
  man page updated 
* Fri Apr 04 2008 Michael Weinberger 0.10.0-2
  added --all option to --rise
* Mon Mar 10 2008 Michael Weinberger 0.10.0-1
  Increased release number
  Added 10 sec timeout in Net::DNS::Resolver->search
* Mon Mar 10 2008 Michael Weinberger 0.9.0-11
  Specfile fix
* Mon Mar 10 2008 Michael Weinberger 0.9.0-10
  Delete locally created watchdog file after remotecopy
  Bugfix 4035
  Added default AutoUnmount property to sample script
  Fix man page
  Do not delete md5sum in --create-backup-file when verify is skipped by user
  Set lock in --create-backup-file
* Sun Mar 09 2008 Michael Weinberger 0.9.0-9
  trap signals in --create-backup-file
* Sat Mar 08 2008 Michael Weinberger 0.9.0-8
  Permissions in /usr/lib/affa
* Sat Mar 08 2008 Michael Weinberger 0.9.0-7
  md5sum in --create-backup-file
* Fri Mar 07 2008 Michael Weinberger 0.9.0-6
  Added option --create-backup-file
  Using line number as error code in affaErrorExit()
  Fixed /etc/profile.d/affa.sh
* Mon Mar 03 2008 Michael Weinberger 0.9.0-5
  delete new style Dovecot's index files dovecot.index* 
  delete dovecot index files on remote server in --full-restore
* Fri Feb 29 2008 Michael Weinberger 0.9.0-4
  set localhost as default remoteHostName in /etc/e-smith/templates//etc/cron.d/affa/00jobs
* Wed Feb 27 2008 Michael Weinberger 0.9.0-3
  removed %post from spec file
* Sat Feb 02 2008 Michael Weinberger 0.9.0-2
  Bugfix: Report db was not deleted in deleteJob()
  don't use -q on ssh commands with Debug=yes
* Fri Feb 1 2008 Shad L. Lords <slords@mail.com> 0.9.0-1
  Fix specfile for building in new buildsystem
* Fri Jan 18 2008 Michael Weinberger
  Version 0.9.0
  Bugfix: run cronSetup() after job delete
  Throw error if nothing was backed up (empty archive)
  Added rsyncd support for backing up Windows/Cygwin
  Added option and property remoteOS. Which remoteOS=cgywin the 
  send key and revoke key functions use the right path to the 
  authorized_keys2 file and login as user Administrator
* Sat Dec 29 2007 Michael Weinberger
  Version 0.8.1
  Automount Bugfix [Bug 3165]: Only exit with error on mount failure when in backup run. Otherwise log the error and  continue.
* Thu Dec 20 2007 Michael Weinberger
  Version 0.8.0
  added bash completion script
  added option --kill JOB
  improved INT and TERM signal handling
  replaced all scp commands by rsync
  added poperty BandwidthLimit (unit KBytes/sec)
  each job write its own known hosts entry using ssh option HostKeyAliasOption
  affa --send-key w/o arguments sends the keys for all jobs
  removed Property StrictHostKeyChecking
  UPDATE FROM VERSIONS<8: run the command 'affa --send-key' to generate the known hosts entries for all jobs
* Wed Dec 12 2007 Michael Weinberger
  Version 0.7.0
  Modified logging: Job run logs to /var/log/affa/JOBNAME.log, all other to /var/log/affa/affa.log
  Bugfix: Check error status of --send-key  and --revoke-key
  New: Options --show-schedule
  Modified: Option --list-archives now accepts multiple args or none
  Fix: install watchdog on source server before every run not only scheduled, to avoid watchdog triggered if a non-scheduled run take to long time.
* Wed Nov 28 2007 Michael Weinberger
  Version 0.6.8
  Bugfix: Automount was done too late. Daily backup never ran.
* Wed Nov 21 2007 Michael Weinberger
  Version 0.6.7
  Bugfix: Size and No. of files were not shown in --status
  Improved deleting of directories
  added --check-connections option
  Increased width of job column in --stats table
* Sun Nov 11 2007 Michael Weinberger
  Version 0.6.6
  Property rsync--modify-window added
  Modified recursivly deleting directories
* Wed Nov 07 2007 Michael Weinberger
  Version 0.6.5
  ssh options added to scp commands
  added post-backup event
* Wed Aug 08 2007 Michael Weinberger
  Version 0.6.4
  delete dovcot index files after a --rise
* Thu Jul 05 2007 Michael Weinberger
  Version 0.6.3
  Bugfix: with RPMCheck=yes the affa-rpmlist was not found in the 
  archive. Wrong path to schedule.0 instead of scheduled.running. 
  Side effect of changes in version 0.5.0. [Bug 3128]
* Tue Jul 03 2007 Michael Weinberger
  Version 0.6.2
  Bugfix: Filesys::DiskFree requires $ENV{LANG}="en_US" [Bug 3119]
  Bug reported to bug-Filesys-DiskFree [at] rt.cpan.org
* Mon Jun 25 2007 Michael Weinberger 
  Version 0.6.1
  Fix 3080 was not applied.
* Mon Jun 18 2007 Michael Weinberger
  Version 0.6.0
  Replaced Filesys::DiskSpace by Filesys::DiskFree as the latter supports CIFS file system
  using Filesys::DiskFree->mount and ->device instead of scanning /proc/mounts
  mkdir RootDir only on job run to avoid making it in an unmounted mountpoint
* Mon Jun 18 2007 Michael Weinberger
  Version 0.5.2
  Sending ssh key failed. When grep returned an error
  the mv command was not executed. Using ';' instead of '&&' 
  for chaining commands. [Bug 3080]
  /root/.ssh/authorized_keys2: No such file or directory warning.
  Run a touch to guarantee, that the file exists. [Bug 3080]
  same issue with --revoke-key
* Thu Jun 14 2007 Michael Weinberger
  Version 0.5.1
  Bugfix automount: getStatus() unmounts the device while a job is running
  Status table: Last=never, if there are no archives at all
* Tue Jun 12 2007 Michael Weinberger
  Version 0.5.0
  New property AutomountOptions
  status table: grouped disabled jobs. Show Last='-' for disabled jobs instead of 'failed'
  unlink lockfile in getLock()
  Changes to improve robustness in case of network outage:
  - writing to a temporary archive scheduled.running and linkdest
    against the latest existing archive (usually scheduled.0). Then
    shift archives and rename scheduled.running to scheduled.0
  - scheduledKeep=1 is now allowed
  - also check connection in daily,weekly etc., not only in scheduled run
  - don't run daily, weekly, monthly or yearly, when scheduled.0 doesn't exist.
  new option --revoke-key
  added optional option --revoke-key to --delete-job
  avoid multiple adding public key to authorized_keys2 on source server
* Wed Jun 06 2007 Michael Weinberger
  Version 0.4.7
  use StrictHostKeyChecking=no in sendKeys()
* Tue Jun 05 2007 Michael Weinberger
  Version 0.4.6
  mailtest: Send also a testmail from the remotehost, when property Watchdog=yes
  removed useless code in mailTestWatchdogRemote()
* Mon Jun 04 2007 Michael Weinberger
  Version 0.4.5
  write job config file to the archive dir
* Thu May 31 2007 Michael Weinberger
  Version 0.4.4
  added property StrictHostKeyChecking (ssh option)
  string comparison instead of numric for $job{'doneYearly'} ne $thisYear etc.
* Wed May 30 2007 Michael Weinberger
  Version 0.4.3
  Minor bugfixes:
  calculation of RootDirFilesystemUsage in .AFFA-REPORT
  improved error handling with rsync status and df in DiskspaceWarn()
* Tue May 29 2007 Michael Weinberger
  Version 0.4.2
  Infinite loop in execPostJobCommand() if command could not be executed:
  Don't call execPostJobCommand() in affaErrorExit() if err==115
  RPMCheck property was ignored
  added Property chattyOnSuccess
  modified jobconfig-sample.pl to preserve 'doneDaily','doneWeekly','doneMonthly','doneYearly' and 'chattyOnSuccess'
  write error codes of affaErrorExit() to log
* Thu May 24 2007 Michael Weinberger
  Version 0.4.1
  fixed bug in disk usage calculation
* Mon May 21 2007 Michael Weinberger
  Version 0.4.0
  added option --rename-job
  changed syntax of --send-keys (!)
  added option --move-archive
* Sun May 20 2007 Michael Weinberger
  Version 0.3.3
  rebuild cronjobs after rise/undorise
  ignore job of own backup, when creating cronjobs (job appears after a rise run)
* Sun May 20 2007 Michael Weinberger
  Version 0.3.2
  man: added sshPort propperty
  chdir /tmp to avoid cwd warnings when the cwd disappears while running rise or undo rise
  bugfix: undorise() did not found own backup archive. Was searching for a wrong name
* Wed May 16 2007 Michael Weinberger
  Version 0.3.1 minor bugfixes
  checkCrossFS() did not work (used in --rise) 
  jobconfig-sample.pl: deleting record before setting props
  Perl errors with --status before a job run
* Thu May 10 2007 Michael Weinberger
  Version 0.3.0
  man page completed
  mark archives with indices > keep setting with '*' in --list-acrchive output
  Option --delete-job
  Option --cleanup 
  added --job=JOB alternative to --send-keys
* Wed May 09 2007 Michael Weinberger
  Version 0.2.0
  added --mailtest option
* Tue May 08 2007 Michael Weinberger
  Version 0.1.5
  improved --status output
  removed options --report and --send-report
  added option --show-archives
  added --csv for status and show-archives output in CSV format
  added property 'sshPort'
* Mon May 07 2007 Michael Weinberger
  Version 0.1.4
  don't install the remote watchdog, when remotehost is eq localhost
  improved check for remoteHostName eq localhost using DNS
  ssh -o PasswordAuthentication=no in checkConnection()
  added --full-restore
  missing check for HOSTNAME argument in --send-keys added
  improved error check
  prevent run of --rise of localhost from own backup
* Mon Apr 30 2007 Michael Weinberger
  Version 0.1.3
  modified 'use constant* syntax in watchdog script for compatibility with perl 5.6 on SME6
* Fri Apr 27 2007 Michael Weinberger
  Version 0.1.2
  Bugfix: Preserve of ethernet driver setting with --rise did not work.
  Also preserve NIC bonding.
* Mon Apr 23 2007 Michael Weinberger
  Version 0.1.1
  scheduledKeep must be>=2 for --link-dest
  set scheduledKeep to 2 if <2
  get lastrun date from affa-report rather than from report file
  added auto mount function
  added AutomountDevice and AutomountPoint to jobconfig-sample.pl
* Wed Apr 18 2007 Michael Weinberger
  Version 0.0.8
  dont die if report db does not exist
* Wed Apr 18 2007 Michael Weinberger
  Version 0.0.7
  run checkConnection() only for scheduled backups
  added Size and Disk usage information to --status
* Thu Apr 12 2007 Michael Weinberger
  Version 0.0.5
  fixed calculation of lastrun-now
* Thu Apr 12 2007 Michael Weinberger
  Version 0.0.4
  added --send-status plus templates 
  fixed format error of times in affa --status
  show 'failed', if lastrun is older 1 day in affa --status 
  fixed typo. default status=disabled (was disable)
* Fri Apr 06 2007 Michael Weinberger
  Version 0.0.3
  watchdog reminder was not deleted on soure
  wrong version mismatch list in  rpm compare
* Thu Apr 05 2007 Michael Weinberger
  added 'rsync--inplace' property
* Mon Apr 02 2007 Michael Weinberger
  initial release

Files

/etc/e-smith/events/actions/affa-make-cronjobs
/etc/e-smith/events/post-upgrade/S90affa-make-cronjobs
/etc/e-smith/templates/etc/cron.d/affa-status/00run
/etc/e-smith/templates/etc/cron.d/affa/00jobs
/etc/logrotate.d/affa
/etc/profile.d/affa.sh
/sbin/e-smith/affa
/sbin/e-smith/affa-rpmlist.sh
/usr/lib/affa/COPYING
/usr/lib/affa/jobconfig-cygwin-sample.pl
/usr/lib/affa/jobconfig-sample.pl
/usr/lib/affa/postJobCommand-sample.pl
/usr/lib/affa/preJobCommand-sample.pl
/usr/lib/affa/watchdog-mailtest.template
/usr/lib/affa/watchdog.template
/usr/man/man1/affa.1.gz


Acronym

Affa stands for Automatische Festplatten Fernarchivierung

Retrieved from "https://wiki.koozali.org/index.php?title=Affa&oldid=10114"