LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Enterprise (https://www.linuxquestions.org/questions/linux-enterprise-47/)
-   -   script to move file from local filesystem to hadoop filesystem (https://www.linuxquestions.org/questions/linux-enterprise-47/script-to-move-file-from-local-filesystem-to-hadoop-filesystem-4175511587/)

callis07 07-19-2014 07:15 AM

script to move file from local filesystem to hadoop filesystem
 
Hi All

Can you provide me a shell script which will move file from local filseystem to hadoop filesytem , by the checking the count of files in hadoop filesystem if it is less than 20 fils then move 10 files from local filesystem to hadoop filesystem.

T3RM1NVT0R 07-19-2014 07:24 AM

From the requirement it appears that you only want to move files from local filesystem to hadoop filesystem if there are 20 files on hadoop file system. Taking that into consideration, if you have got the following mount points /local and /hadoop on the system you can do it as follows:

Code:

#!/bin/bash
NUM=`ls /hadoop | wc -l`
if [ $NUM -lt 20 ]
then
    {
      for i in `ls /local | head -n 10`
      do
      {
        mv /local/$i /hadoop
        echo "10 files moved from local to hadoop fs"
      }
      done
    }
else
    {
      echo "Hadoop fs has got more than 20 files"
    }
fi

EDIT - I forgot to put wc -l for file NUM count. Updated that in code section.

callis07 07-19-2014 08:14 AM

From the requirement it appears that you only want to move files from local filesystem to hadoop fil
 
yes. some script like this.
But hadoop filesystem will be read something like
hadoop fs -ls /hadoop filesystem path/ -- to read a files in a specific directory
And word count cannot be done like using wc -l

and for moving the files we need to use something like below

hadoop fs -moveFromLocal /local path/ /hadoop filesytem path/

before performing this operation we need to login as hbase user

su - hbase

I will try to modify it from your scripts and see if it works. And Thanks for your script.

Thanks//

callis07 07-19-2014 08:31 AM

I tried to modify something like this but amgetting error, Please can you correct me.

[scripts]# ./process.sh
./process.sh: line 2: hadoop: command not found
./process.sh: line 3: [: -lt: unary operator expected
Hadoop fs has got more than 20 files
[root@TMSKCRDM01 scripts]#

#!/bin/bash
NUM=`hadoop fs -count /var/CRS/processing/SDP`
if [ $NUM -lt 20 ]
then
{
for i in `ls /var/CRS/cdr_bkp/june01/SDP1_0601/var/DWS/incoming/CS40/Mediation/CDRmedSend | head -n 10`
do
{
hadoop fs -moveFromLocal /var/CRS/cdr_bkp/june01/SDP1_0601/var/DWS/incoming/CS40/Mediation/CDRmedSend/$i /var/CRS/incoming/SDP
echo "10 files moved from local to hadoop fs"
}
done
}
else
{
echo "Hadoop fs has got more than 20 files"
}
fi

T3RM1NVT0R 07-19-2014 08:37 AM

It says hadoop command not found, it appears that the search path is not set to find hadoop related commands. So instead of:

Code:

NUM=`hadoop fs -count /var/CRS/processing/SDP`
use:

Code:

NUM=`/full_path_to_hadoop_binaries/hadoop fs -count /var/CRS/processing/SDP`
Wherein, full path to hadoop binaries is the actual path where hadoop binaries are located.

Instead of:

Code:

if [ $NUM -lt 20 ]
you can try:

Code:

if [ $NUM < 20 ]
Also run process.sh with sh -x process.sh which will give step by step information on what script is doing.


All times are GMT -5. The time now is 08:05 PM.