LinuxQuestions.org
Register a domain and help support LQ
Go Back   LinuxQuestions.org > Forums > Enterprise Linux Forums > Linux - Enterprise
User Name
Password
Linux - Enterprise This forum is for all items relating to using Linux in the Enterprise.

Notices



Reply
 
Search this Thread
Old 07-19-2014, 08:15 AM   #1
callis07
LQ Newbie
 
Registered: Jul 2014
Posts: 5

Rep: Reputation: Disabled
script to move file from local filesystem to hadoop filesystem


Hi All

Can you provide me a shell script which will move file from local filseystem to hadoop filesytem , by the checking the count of files in hadoop filesystem if it is less than 20 fils then move 10 files from local filesystem to hadoop filesystem.
 
Old 07-19-2014, 08:24 AM   #2
T3RM1NVT0R
Senior Member
 
Registered: Dec 2010
Location: Internet
Distribution: Linux Mint, Ubuntu, SLES, CentOS
Posts: 1,790

Rep: Reputation: 324Reputation: 324Reputation: 324Reputation: 324
From the requirement it appears that you only want to move files from local filesystem to hadoop filesystem if there are 20 files on hadoop file system. Taking that into consideration, if you have got the following mount points /local and /hadoop on the system you can do it as follows:

Code:
#!/bin/bash
NUM=`ls /hadoop | wc -l`
if [ $NUM -lt 20 ]
then
    {
      for i in `ls /local | head -n 10`
      do
      {
         mv /local/$i /hadoop
         echo "10 files moved from local to hadoop fs"
      }
      done
     }
else
     {
       echo "Hadoop fs has got more than 20 files"
     }
fi
EDIT - I forgot to put wc -l for file NUM count. Updated that in code section.

Last edited by T3RM1NVT0R; 07-19-2014 at 08:30 AM.
 
Old 07-19-2014, 09:14 AM   #3
callis07
LQ Newbie
 
Registered: Jul 2014
Posts: 5

Original Poster
Rep: Reputation: Disabled
From the requirement it appears that you only want to move files from local filesystem to hadoop fil

yes. some script like this.
But hadoop filesystem will be read something like
hadoop fs -ls /hadoop filesystem path/ -- to read a files in a specific directory
And word count cannot be done like using wc -l

and for moving the files we need to use something like below

hadoop fs -moveFromLocal /local path/ /hadoop filesytem path/

before performing this operation we need to login as hbase user

su - hbase

I will try to modify it from your scripts and see if it works. And Thanks for your script.

Thanks//
 
Old 07-19-2014, 09:31 AM   #4
callis07
LQ Newbie
 
Registered: Jul 2014
Posts: 5

Original Poster
Rep: Reputation: Disabled
I tried to modify something like this but amgetting error, Please can you correct me.

[scripts]# ./process.sh
./process.sh: line 2: hadoop: command not found
./process.sh: line 3: [: -lt: unary operator expected
Hadoop fs has got more than 20 files
[root@TMSKCRDM01 scripts]#

#!/bin/bash
NUM=`hadoop fs -count /var/CRS/processing/SDP`
if [ $NUM -lt 20 ]
then
{
for i in `ls /var/CRS/cdr_bkp/june01/SDP1_0601/var/DWS/incoming/CS40/Mediation/CDRmedSend | head -n 10`
do
{
hadoop fs -moveFromLocal /var/CRS/cdr_bkp/june01/SDP1_0601/var/DWS/incoming/CS40/Mediation/CDRmedSend/$i /var/CRS/incoming/SDP
echo "10 files moved from local to hadoop fs"
}
done
}
else
{
echo "Hadoop fs has got more than 20 files"
}
fi
 
Old 07-19-2014, 09:37 AM   #5
T3RM1NVT0R
Senior Member
 
Registered: Dec 2010
Location: Internet
Distribution: Linux Mint, Ubuntu, SLES, CentOS
Posts: 1,790

Rep: Reputation: 324Reputation: 324Reputation: 324Reputation: 324
It says hadoop command not found, it appears that the search path is not set to find hadoop related commands. So instead of:

Code:
NUM=`hadoop fs -count /var/CRS/processing/SDP`
use:

Code:
NUM=`/full_path_to_hadoop_binaries/hadoop fs -count /var/CRS/processing/SDP`
Wherein, full path to hadoop binaries is the actual path where hadoop binaries are located.

Instead of:

Code:
if [ $NUM -lt 20 ]
you can try:

Code:
if [ $NUM < 20 ]
Also run process.sh with sh -x process.sh which will give step by step information on what script is doing.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: SSHFS(Secure SHell FileSystem): Securely Mount remote filesystem over ssh LXer Syndicated Linux News 0 11-13-2013 08:30 PM
Move root filesystem HOWTO processcontrol Linux - General 1 03-01-2012 06:10 AM
new filesystem: convert, or move data? slackhack Linux - General 5 04-13-2008 04:35 PM
Need to move filesystem journal to another array Thaidog Linux - General 1 06-18-2005 03:11 PM
DISCUSSION: Virtual Filesystem: Building a Linux Filesystem from an Ordinary File mchirico LinuxAnswers Discussion 0 10-28-2004 11:35 PM


All times are GMT -5. The time now is 03:16 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration