Friday, November 16, 2007

Bash Script for FTP

#!/bin/bash
USER=myusername
PASS=mypasswd
FTPSERVER=192.168.0.X
ftp -i -n $FTPSERVER << EOF
user $USER $PASS
mkdir test
cd test
put myfile
bye
>>

But FTP will allow transfer of files only,not the directory tree.
If you want to transfer the Directory structure through FTP you can use LFTP or similar FTP clients. A variety of GUI Based clients are available

LFTP

lftp has builtin mirror which can download or update a whole directory tree. There is also reverse mirror (mirror -R) which uploads or updates a directory tree on server. Mirror can also synchronize directories between two remote servers, using FXP if available.
It can be downloaded from http://lftp.yar.ru/get.html or http://rpm.pbone.net

Here is a sample BASH Script to automate the FTP Transfer

#!/bin/bash
USER=ftpuser
PASS=ftppasswd
FTPSERVER=192.168.0.X
LOCALDIR=/home/USER/LOCAL
REMDIR=REMOTE
lftp -u $USER,$PASS $FTPSERVER << EOF
mirror -R $LOCALDIR $REMDIR
quit
>>



Now see how to play with the data we have to upload.That is you can decide whatever folders or files have to be uploaded to the FTP Server.I use two scripts fro this.But we can consolidate it into a single one.

Script 1 - ftp_initiate.sh
#!/bin/sh
LIST=/root/scripts/datalist.txt
#echo Where is the Data List
#read LIST

count=`wc -l $LIST | cut -f1 -d" "`
n=1

while [ $n -le $count ]
do
{
data=`head -$n $LIST | tail -1`
sh /root/scripts/ftp_upload.sh $data
n=$[$n +1]
}
done

Script 2 - ftp_upload.sh
#!/bin/sh
USERNAME='username'
PASSWORD='password'
SERVER='192.168.0.X'

# local directory to pickup
SOURCE=/some/where/in/your/home

# remote server directory to upload backup
BACKUPDIR=/backup/folder/in/FTP/Server

data=$SOURCE/$1
lftp -u $USERNAME,$PASSWORD $SERVER << EOF
mirror -R $data $BACKUPDIR/
quit
>>
Here the if a deletion takes place at the SOURCE it won't affect the DESTINATION .Means the deleted contents never get deleted from the DESTINATION.
You can optionally delete those files in the DESTINATION also by specifying the --delete switch of the MIRROR command as below
mirror -R -e --delete $LOCALDIR $REMDIR
In some environment with Firewalls,Mix of OSs and FTP Services a few problems may arise in connectivity like SSL Communication,Proxy,etc
Here I have faced an issue with the SSL .By default SSL is enabled in LFTP
After connecting to the FTP Server I just tried to List the contents which turned into errors as below
lftp ftpuser@192.168.0.1:~> ls
'ls' at 0 [FEAT TLS negotiaition..]
'ls' at 0 [ Delaying before Reconnect 29..]
'ls' at 0 [Not Connected..]
lftp ftpuser@192.168.0.1:~>
It repeats
What I did was just disabled SSL
set -a will list all the variables and values for the FTP session
lftp ftpuser@192.168.0.1:~> set -a
SSL was enabled . I turned it to disabled state
lftp ftpuser@192.168.0.1:~> set ftp:ssl-allow no

Thereafter it worked
lftp ftpuser@192.168.0.1:~>ls
12-10-07 11:04PM DIR dir1
12-10-07 11:10PM DIR tesfile.txt
12-07-07 09:48AM DIR TestDir
12-09-07 11:05PM DIR mydata


The same can be applied to the BASH Script also
#!/bin/bash
USER=ftpuser
PASS=ftppasswd
FTPSERVER=192.168.0.X
LOCALDIR=/home/USER/LOCAL
REMDIR=REMOTE
lftp -u $USER,$PASS $FTPSERVER << EOF
set ftp:ssl-allow no
mirror -R $LOCALDIR $REMDIR
quit
>>
To know more about FTP-SSL See RFC2228
FXP Mirroring

server A -> server B
When mirroring is done between two remote servers the File eXchange Protocol is used. Obviously, both servers must support this protocol for this operation to succeed.
Technically, FXP is not a protocol but an extension of FTP. It is used to transfer data from one remote server to another without routing this data through the client. The client sends and receives control data to make everything work.
In an FXP session, the client maintains a standard FTP connection to both servers, and can direct either server to connect to the other to initiate a data transfer. The advantage of using FXP (server A -> server B) instead of (twice using) FTP (server A -> client -> server B) is evident when both servers are high-bandwidth but the client is low-bandwidth.
Enabling FXP support, however, can make a server vulnerable to a denial-of-service attack, known as the FTP bounce attack
In such a scenario, the "client" is a compromised machine that bombards server B.FXP is also frequently used for warez
trafficking.

Due to these considerations, FXP is often disabled by default on FTP servers.

Reference
lftp Man Page
PapaMike

Follow ups
Cert.org
LinuxForums

2 comments:

Matt (the_cha) said...

Hello thanks for this great post it is exactly what I'm looking for... I have a folder on my server that I would like to upload to my FTP server daily...

What I would like to happen is that All of the folders inside /Public/WebSites on my server are put into the folder new/Classes on the FTP server.

However what this script does is it puts all the folders of /Public/WebSites into new/Classes/new/Classes on the FTP server. Do you know what I might be able to do to the script to fix it so the at it works as I intend?

#!/bin/bash
USER=******
PASS=******
FTPSERVER=*****
LOCALDIR=/Public/WebSites
REMDIR=new/Classes
lftp -u $USER,$PASS $FTPSERVER << EOF
cd $REMDIR
mirror -R $LOCALDIR $REMDIR
quit

hareeshvv@skype said...

I guess your local directory /Public/WebSites is having new/Classes inside it. If so can you try resetting LOCALDIR to /Public/WebSites/new/Classes