script to move file from local filesystem to hadoop filesystem
Hi All
Can you provide me a shell script which will move file from local filseystem to hadoop filesytem , by the checking the count of files in hadoop filesystem if it is less than 20 fils then move 10 files from local filesystem to hadoop filesystem. |
From the requirement it appears that you only want to move files from local filesystem to hadoop filesystem if there are 20 files on hadoop file system. Taking that into consideration, if you have got the following mount points /local and /hadoop on the system you can do it as follows:
Code:
#!/bin/bash |
From the requirement it appears that you only want to move files from local filesystem to hadoop fil
yes. some script like this.
But hadoop filesystem will be read something like hadoop fs -ls /hadoop filesystem path/ -- to read a files in a specific directory And word count cannot be done like using wc -l and for moving the files we need to use something like below hadoop fs -moveFromLocal /local path/ /hadoop filesytem path/ before performing this operation we need to login as hbase user su - hbase I will try to modify it from your scripts and see if it works. And Thanks for your script. Thanks// |
I tried to modify something like this but amgetting error, Please can you correct me.
[scripts]# ./process.sh ./process.sh: line 2: hadoop: command not found ./process.sh: line 3: [: -lt: unary operator expected Hadoop fs has got more than 20 files [root@TMSKCRDM01 scripts]# #!/bin/bash NUM=`hadoop fs -count /var/CRS/processing/SDP` if [ $NUM -lt 20 ] then { for i in `ls /var/CRS/cdr_bkp/june01/SDP1_0601/var/DWS/incoming/CS40/Mediation/CDRmedSend | head -n 10` do { hadoop fs -moveFromLocal /var/CRS/cdr_bkp/june01/SDP1_0601/var/DWS/incoming/CS40/Mediation/CDRmedSend/$i /var/CRS/incoming/SDP echo "10 files moved from local to hadoop fs" } done } else { echo "Hadoop fs has got more than 20 files" } fi |
It says hadoop command not found, it appears that the search path is not set to find hadoop related commands. So instead of:
Code:
NUM=`hadoop fs -count /var/CRS/processing/SDP` Code:
NUM=`/full_path_to_hadoop_binaries/hadoop fs -count /var/CRS/processing/SDP` Instead of: Code:
if [ $NUM -lt 20 ] Code:
if [ $NUM < 20 ] |
All times are GMT -5. The time now is 08:52 PM. |