Tuesday, January 29, 2013

Duplicity fails on Too many open files

If you have got any error message like “Too many open files” in Ubuntu machine below are the steps to overcome the error message.
I have faced this error message while restoring Duplicity backup volumes. To restore backup volume I have used Java code.

There is simple solution that I can maximize the Open file Descriptor in Ubuntu OS, to complete that follow the below instructions.

System-wide File Descriptors (FD) Limits

The number of concurrently open file descriptors throughout the system can be changed via /etc/sysctl.conf file under Linux operating systems.

The Number Of Maximum Files Was Reached, How Do I Fix This Problem?

Many application such as Oracle database or Apache web server needs this range quite higher. So you can increase the maximum number of open files by setting a new value in kernel variable /proc/sys/fs/file-max as follows (login as the root):# sysctl -w fs.file-max=100000
Above command forces the limit to 100000 files immediately. You need to edit /etc/sysctl.conf file and put following line so that after reboot the setting will remain as it is:
# vi /etc/sysctl.conf
Append a config directive as follows:
fs.file-max = 100000
Save and close the file. Users need to log out and log back in again to changes take effect or just type the following command:
# sysctl -p
Verify your settings with command:
# cat /proc/sys/fs/file-max
OR
# sysctl fs.file-max



Source : http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/

Duplicity Backup Restore process

I have used duplicity backup script for backing up user data and also I have used Private and Public keys encryption method for encrypting the backup.

To restore backup used below commands and initially copied Duplicity volume gpg files from Remote location to local Computer.
Then we must issue below commands to decrypt the backup volumes. After the decryption we can view there are two folders created in the backup restore location. Folders are named as “multivol_snapshot” and “snapshot”.

If you encrypted your backup, first you must decrypt the volume by using your private key. Say you have duplicity-full.20110127T131352Z.vol1.difftar.gpg:
gpg --output duplicity-full.20110127T131352Z.vol1.difftar --decrypt duplicity-full.20110127T131352Z.vol1.difftar.gpg
Or to do all at once (This is the easiest way to do ...):
gpg --multifile --decrypt duplicity-*.*.*.difftar.gpg
Now you have either a .difftar or a .difftar.gz volume (depending on whether you had to decrypt it or not). Use tar on whichever one you have to extract the individual patch files:
tar xvf duplicity-full.20110127T131352Z.vol1.difftar
Or again, to do all at once:
for t in duplicity-*.*.*.difftar; do tar xf $t; done

If your file is in snapshot/ then you're done. Otherwise find the directory in multivol_snapshot/ at the path where your file used to be: you need to join together all the files in this directory to recreate the original file. The files are numbered, and can be joined together using the cat command. Depending on how large the original was, there may be many parts.

cat * > rescued-file


Problem with original instructions
The directions linked above suggest using cat * > rescued-file. Unfortunately this simple approach fails if you have more than 9 parts. Since * expands in dictionary order, not numeric order, 10 would be listed before 2, and the file would be reconstructed in the wrong order.
Workaround
One simple approach is to remember that dictionary order does work when numbers are the same length, and that ? matches a single character. So if your largest file has three digits, you can manually enter:
cat ? ?? ??? > rescued-file
Add or remove ? patterns as necessary, depending on the largest file number.

If there are more than 9 parts then you should go for scripting to recover the data. There you can use either shell scripting or Java code which include below.
But there is major limitation that it won't work with Incremental backups.

If you have a lot of files to recover and don't fancy typing that for all of them, you might prefer to use a script such as this. It lists the containing directory for every file, removes duplicates from the list, then goes to each directory and creates a content file from the fragments there. (spacer is just to make$1 work.)


find multivol_snapshot/ -type f -printf '%h\0' | \
  sort -uz | \
  xargs -0 -n 1 sh -c 'cd "$1" ; cat $(ls | sort -n) > content' spacer

Now you just have to add /content to the end of any filename you were looking for, and you should find it.


Or you can use below Java code to restore backup without issue. (No need to change single letter of the code it works fantastically for me.)



import java.io.File;
import java.io.FileFilter;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.Arrays;
import java.util.Comparator;
import java.util.ListIterator;
import java.util.Vector;

import org.apache.commons.io.filefilter.DirectoryFileFilter;

public class DeFrankensteiner {

static String untaredRoot = "/media/big5wf/untared2test";
static Vector<File> resultDirs = new Vector<File>();

public static void main(String[] args) {
if (args.length>0) {
if (args[0] != null) {
untaredRoot = args[0];
if(!new File(untaredRoot).exists()){
System.err.println("Directory does not exist");
}
}

} else {
System.out.println("Program takes two arguments: root folder of the backup and an optional target folder");
System.out
.println("Please rerun and specifiy the root of your untared duplicity backup");
System.out
.println("The directory contains two folders, 'snapshot' and 'multivol_snapshot'");
System.exit(0);
}
getLeafDirectories(new File(untaredRoot
+ System.getProperty("file.separator") + "multivol_snapshot"));

ListIterator iter = resultDirs.listIterator();
while (iter.hasNext()) {
File sourceDir = (File) iter.next();
File[] the64KbBlocks = sourceDir.listFiles();
// We need a non alphabetic, simple higher is better sort
Arrays.sort(the64KbBlocks, new IntValueComparator());
String targetFileName = sourceDir.getAbsolutePath().replace(
"multivol_snapshot", "snapshot");
System.out.println("Will save file to " + targetFileName
+ " after merging " + the64KbBlocks.length
+ " blocks.");
// instead of /bin/bash try to use java onboard methods
try {

File targe = new File(targetFileName);

if(targe.exists())targe.delete();

FileOutputStream fos = new FileOutputStream(targetFileName,
true);
int i = 0;
for (File file : the64KbBlocks) {
i++;
FileInputStream fis = new FileInputStream(file);
byte[] bytesOfA64KbBlock = bytesOfA64KbBlock = getBytesFromFile(file);
fos.write(bytesOfA64KbBlock);
fis.close();
}

System.err.println("Written file file://" + targetFileName);
fos.close();
} catch (IOException e) {
e.printStackTrace();
}
}

}

public static byte[] getBytesFromFile(File file) throws IOException {
InputStream is = new FileInputStream(file);

// Get the size of the file
long length = file.length();

// Create the byte array to hold the data
byte[] bytes = new byte[(int) length];

// Read in the bytes
int offset = 0;
int numRead = 0;
while (offset < bytes.length
&& (numRead = is.read(bytes, offset, bytes.length - offset)) >= 0) {
offset += numRead;
}

// Ensure all the bytes have been read in
if (offset < bytes.length) {
throw new IOException("Could not completely read file "
+ file.getName());
}

// Close the input stream and return bytes
is.close();
return bytes;
}

public static class IntValueComparator implements Comparator<File> {

public int compare(final File file1, final File file2) {

int res = 0;
try {
res = (Integer.valueOf(file1.getName()) - Integer.valueOf(file2
.getName()));
} catch (NumberFormatException e) {
System.err.println("Something is here that shouldnt be here: "
+ e.getMessage());
System.err.println(file1.getAbsolutePath());
System.err.println(file2.getAbsolutePath());
// e.printStackTrace();
System.exit(-1);
}
return res;
}

}

public static void getLeafDirectories(File dir) {
File listFile[] = dir.listFiles();
if (listFile != null) {
for (int i = 0; i < listFile.length; i++) {
if (listFile[i].isDirectory()) {

File[] subdirs = listFile[i]
.listFiles((FileFilter) DirectoryFileFilter.DIRECTORY);
if (subdirs.length == 0)
resultDirs.add(listFile[i]);
getLeafDirectories(listFile[i]);
}
}
}

}

}

You must have installed Java run time and IDE for compile Java code. I have installed NetBeans IDE and then create Public class called “DeFrankensteiner” and added above code to that class and compile it.
You should take “<name>.jar” file from the comlied location which is under “dist” folder. Then issue the below commands to Recover the “multivol_snapshot” folder.

  • Go into the dist folder
    • Ex : - cd /user/Desktop/java/backuprestore/dist
  • Run the jar file with the source directory and Destination directory arguments.
    • Ex:- java -jar Backuprestore.jar /root/Documents/Test/ /root/restoreddata


Limitations
This doesn't restore any of the original file permissions or ownership. It also doesn't deal with incremental backups, but then the inked instructions also hit a bit of a dead end on this point — they just suggest using rdiff to stitch the files together' and refer the reader to man rdiff.


https://answers.launchpad.net/ubuntu/+source/duplicity/+question/186098

Wednesday, January 16, 2013

My SQL error jos_session' is marked as crashed and should be repaired SQL=INSERT INTO `jos_session`

This is one of the most common error in Joomla and it only means that the table jos_session has crashed or is damage. This table stores your session and you can easily fix this via cPanel or phpmyadmin.

Before you do anything you should backup the database which is located in /var/lib/mysql and then tar the related database. After that you are totally fine for table repair.

Using phpMyAdmin;

First take phpMyAdmin and then place that phpMyAdmin into apache document root. (ex : /var/www/html)
Then using your web browser access the phpMyAdmin interface and login using related Database credentials. Then issue the command REPAIR TABLE `tbl_name`"
That's it. It will fix the issue.

Or you can use command line and use below commands to repair the table.
  • mysql -u root -p
Enter database password.
  • use “database_name”
  • show tables ; This will show you all the tables name under particular database.
  • Then run either truncate table jos_session or repair table jos_session
This will fix the table crashed issue.



Thursday, January 10, 2013

Allowing Calling user to Transfer Call in FREEPBX

Even though you have allowed Call Transfer, Calling user unable to Transfer the Call with Free PBX and Grandstream phones.
There is way to enable that facility as well.
If you want to transfer calls you have made then in the outbound dial rules on FreePBX general settings check that there is a "T".
Follow the Screen shot for configure Calling user transfer option.







Then Save the changes and Submit it, then enjoyed VOIP with FreePBX.










Thursday, November 29, 2012

Cannot retrieve repository metadata (repomd.xml) for repository: updates. Please verify its path and try again


Error: Cannot retrieve repository metadata (repomd.xml) for repository: updates. Please verify its path and try again

If you see this kind of error intially you don't do anything just reboot the server. This might work for you.
After the reboot try the “yum update” command.
But if you getting same error after the reboot then try the below.
  1. vi /etc/yum.repos.d/fedora.repo
  2. Comment out the “baseurl”.
  3. Save it and exit.
  4. Then try to reboot and run the yum update command.

Then you should be able to update your servers.

If it still having the same then if you having proxy on your network you can try out with that with the below configuration changes.

  1. vi /etc/yum.conf
  2. Add to this line “proxy=http://192.168.1.8:3128
Then save and reboot with those configuration you should be able update your repository.

Monday, November 19, 2012

Edit Virtual Machine Startup and Shutdown Settings

You can configure virtual machines running on an ESXi host to start up and shut down with the host. You can also set the default timing and startup order for selected virtual machines. This ability allows the operating system to save data when the host enters maintenance mode or is being powered off for another reason. This setting is disabled when DRS cluster is enabled.
Procedure
1. In the vSphere Client inventory, select the host where the virtual machine is located and click the Configuration tab.
2. Under Software, click Virtual Machine Startup/Shutdown and click Properties.
The Virtual Machine Startup and Shutdown dialog box opens.
image



3. Select Allow virtual machines to start and stop automatically with the system.

image


image



Friday, November 9, 2012

Configure fail2ban in Fedora server

If you need to monitor or mail the unauthorized login to linux server fail2ban is perfect tool for that purpose. It will trace the unauthorized access by looking at secure log file and will mail it to mention email address ( by default root ).

To install fail2ban use the yum command
  • yum -y install fail2ban
Change the configuration settings

Need to configure below files in roder to work fail2ban properly.

  1. /etc/fail2ban/jail.conf
  2. /etc/fail2ban/action.d/sendmail-whois.conf
  3. /etc/fail2ban/filter.d/sshd.conf

First go to jail.conf file and change the below configuration

# "ignoreip" can be an IP address, a CIDR mask or a DNS host. Fail2ban will not
# ban a host which matches an address in this list. Several addresses can be
# defined using space separator.
ignoreip = 127.0.0.1

Add your internal IP range to the ignore IP address

# "bantime" is the number of seconds that a host is banned.
bantime = 600

The defualt ban time is 10 minutes and if you want to increase banned time then change above setting and time calculate in Seconds.

# "maxretry" is the number of failures before a host get banned.
maxretry = 3
Either you can change maxretry level here or in particular service directory which will meet u below.

# This jail corresponds to the standard configuration in Fail2ban 0.6.
# The mail-whois action send a notification e-mail with a whois request
# in the body.

[ssh-iptables]

enabled = true
filter = sshd
action = iptables[name=SSH, port=ssh, protocol=tcp]
sendmail-whois[name=SSH, dest=chamara@nic.lk, sender=fail2ban@ukusu.nic.lk]
logpath = /var/log/secure
maxretry = 3

In the above settings enabled should be true and in dest and sender you should edit as necessary to enable your mail fascility. Even if you didn't mention mail IDs then it will forwarded to root mail folder.
Note : If you use diffenrnt port and different protocol for SSH then you should port=ssh and protocol=tcp accordingly.

In sendmail-whois.conf comment the bewlo lines if you don not need the fail2ban service start and stop mails.

#actionstart = printf %%b "Subject: [Fail2Ban] <name>: started
# From: Fail2Ban <<sender>>
# To: <dest>\n
# Hi,\n
# The jail <name> has been started successfully.\n
# Regards,\n
# Fail2Ban" | /usr/sbin/sendmail -f <sender> <dest>

# Option: actionstop
# Notes.: command executed once at the end of Fail2Ban
# Values: CMD
#
#actionstop = printf %%b "Subject: [Fail2Ban] <name>: stopped
# From: Fail2Ban <<sender>>
# To: <dest>\n
# Hi,\n
# The jail <name> has been stopped.\n
# Regards,\n
# Fail2Ban" | /usr/sbin/sendmail -f <sender> <dest>


In sshd.conf just go through that file and check if there is unwanted filtering for ssh ban.

Enjoy with the fail2ban service for SSH.