LinuxQuestions.org
Register a domain and help support LQ
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 01-29-2010, 03:29 PM   #1
Supreme1012
LQ Newbie
 
Registered: Jun 2007
Location: Florida, USA
Distribution: Ubuntu 7.04, Kubuntu 7.04, Kubuntu 7.10
Posts: 22

Rep: Reputation: 15
Bash Script for Moving X number files from /direct1 to /direct2?


Hi guys. I have no Bash skills but Im badly in need of a script to move the first 20,000 or whatever number of files from a directory containing over 200,000 files to a new directory. The problem is that I cant access the directory because its so large so I want to break it into chunks, but keep the files in order if possible. If you know of a script to meet my needs, please post it. Otherwise Ill post my fumblings with Bash until I find the right way to fix my problem.

Should I use

Code:
For
 seq 0 20000
Then
 mv /direct2
Else
 echo "Messed up"
fi
or

Code:
i=0
Do
 mv file
 i+1
Until
 i=20000
Else
 echo "Messed Up"
Fi
 
Old 01-29-2010, 03:55 PM   #2
EricTRA
Guru
 
Registered: May 2009
Location: Gibraltar, Gibraltar
Distribution: Fedora 20 with Awesome WM
Posts: 6,805
Blog Entries: 1

Rep: Reputation: 1291Reputation: 1291Reputation: 1291Reputation: 1291Reputation: 1291Reputation: 1291Reputation: 1291Reputation: 1291Reputation: 1291
Hello,

Might I suggest that you first start by reading a tutorial on Bash like for example this one? What you are putting as 'example' doesn't make any sense at all and is missing all the basics.

For example you put a 'fi' to end an 'if' statement that you haven't even started. The 'mv' command needs two parameters at least like
Code:
mv * /direct2
to move everything from the directory where you execute the command to /direct2.

I believe you need to learn the basics first before even attempting bash.

Kind regards,

Eric
 
Old 01-29-2010, 04:15 PM   #3
Supreme1012
LQ Newbie
 
Registered: Jun 2007
Location: Florida, USA
Distribution: Ubuntu 7.04, Kubuntu 7.04, Kubuntu 7.10
Posts: 22

Original Poster
Rep: Reputation: 15
Thanks for the reply. Im just feeling my way at this point. I used to know autoit really well but its been years since Ive done any scripting. I put "fi" at the end b/c the quide I was looking at had it, and I just assumed it was short for "finish". But anyways, Ill take your suggestions and make some changes. Its a work in progress until someone either helps me finish it or writes it for me.

Code:
#!/bin/bash
for FILE in /dir1/;
n=0
do
 mv * /dir2;
 n++
Until
 n=20000
done
basically what I want to do is start off with n=0 and then move one of the files and add n+1 each time a file is moved until n=20000

Im sure this is coded different in bash scripting but I havent found anything that shows exactly what I want to do, so Im going to make some really obvious mistakes until I figure it out

edit: Im not sure about the "for FILE in /dir1/;" if its even needed. and Im unsure if I should use n+1, n+, or n++ to make the loop add 1 to n

Last edited by Supreme1012; 01-29-2010 at 04:17 PM.
 
Old 01-29-2010, 04:36 PM   #4
EricTRA
Guru
 
Registered: May 2009
Location: Gibraltar, Gibraltar
Distribution: Fedora 20 with Awesome WM
Posts: 6,805
Blog Entries: 1

Rep: Reputation: 1291Reputation: 1291Reputation: 1291Reputation: 1291Reputation: 1291Reputation: 1291Reputation: 1291Reputation: 1291Reputation: 1291
Hi,

LinuxQuestions is all about helping others, but not in doing all the work for them, so if you'll put in some time, so will I.

Ok, let's get some basics set up:
Code:
#!/bin/bash 
COUNTER=0
while [  $COUNTER -lt 10 ]; do
    echo The counter is $COUNTER
    let COUNTER=COUNTER+1 
done
The above is an example for a counter used in a while loop. Examples like this are found all over the internet.

You could use the above example like this:
Code:
#!/bin/bash 
COUNTER=0
while [  $COUNTER -lt 20000 ]; 
do
   your 'move' command
   let COUNTER=COUNTER+1 
done
That gets the counter part figured out. Now you'll have to think up a move command taking into account that you cannot use wildcards and take one file at the time to keep everything under control. If for example you use 'move * /destination'; that one command will move everything in the origin directory to the destination directory before adding one to the counter and that's not what you want.

Set a variable to hold your origin directory and one for your destination directory, so you can easily keep track of them in your commands.

Some tips on figuring out how to move your files one by one:
- Use the ls command to make a list of the files in the directory
- Sort the files by size, biggest first, so that the 'weight' of the directory decreases in the fastest way possible
- Get the filename of the first line and put it in a variable (use awk for that)
- then at last use that 'file' variable to mv the file to the destination directory
- use 'piping' to have your commands work together, that is, use the output of one as input for the next one

Using those tips you should be able to come up with something by yourself. Once you do, post it here and someone will have a look at it and help you out (if I'm online I will too).

As you said, it's a work in progress but you'll have to put in some 'Google' time and most of the effort. That way you'll learn the most. Give a man a fish and he'll have food for a day, teach him how to catch a fish and he'll never be hungry again (don't remember who said that, so if someone does, please let me know in order to give him/her credit )

Kind regards,

Eric
 
Old 01-29-2010, 05:15 PM   #5
Supreme1012
LQ Newbie
 
Registered: Jun 2007
Location: Florida, USA
Distribution: Ubuntu 7.04, Kubuntu 7.04, Kubuntu 7.10
Posts: 22

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by EricTRA View Post
Hi,

LinuxQuestions is all about helping others, but not in doing all the work for them, so if you'll put in some time, so will I.

Ok, let's get some basics set up:
Code:
#!/bin/bash 
COUNTER=0
while [  $COUNTER -lt 10 ]; do
    echo The counter is $COUNTER
    let COUNTER=COUNTER+1 
done
The above is an example for a counter used in a while loop. Examples like this are found all over the internet.

You could use the above example like this:
Code:
#!/bin/bash 
COUNTER=0
while [  $COUNTER -lt 20000 ]; 
do
   your 'move' command
   let COUNTER=COUNTER+1 
done
That gets the counter part figured out. Now you'll have to think up a move command taking into account that you cannot use wildcards and take one file at the time to keep everything under control. If for example you use 'move * /destination'; that one command will move everything in the origin directory to the destination directory before adding one to the counter and that's not what you want.

Set a variable to hold your origin directory and one for your destination directory, so you can easily keep track of them in your commands.

Some tips on figuring out how to move your files one by one:
- Use the ls command to make a list of the files in the directory
- Sort the files by size, biggest first, so that the 'weight' of the directory decreases in the fastest way possible
- Get the filename of the first line and put it in a variable (use awk for that)
- then at last use that 'file' variable to mv the file to the destination directory
- use 'piping' to have your commands work together, that is, use the output of one as input for the next one

Using those tips you should be able to come up with something by yourself. Once you do, post it here and someone will have a look at it and help you out (if I'm online I will too).

As you said, it's a work in progress but you'll have to put in some 'Google' time and most of the effort. That way you'll learn the most. Give a man a fish and he'll have food for a day, teach him how to catch a fish and he'll never be hungry again (don't remember who said that, so if someone does, please let me know in order to give him/her credit )

Kind regards,

Eric
wow. ok, this is very helpful.

so "$COUNTER -lt 20000" is the count for the less than 20,000 files. The problem that I have is how to move only one file at a time? I could create a list of the files with ls -1 and save it to a text file, but that seems like more of a hassle than it should be. how do you move an individual file without knowing its specific file name? Is there anyway to take just the first file it finds? I mean, the computer knows how many items are in the folder, why cant I tell it to take x number of items from folder to new folder? Ill dig more when I get some free time but Ive been putting off too many other things trying to figure this out so Ill post again if I learn anything else. Please post if you have any other suggestions.

-Supreme1012
 
Old 01-29-2010, 05:20 PM   #6
EricTRA
Guru
 
Registered: May 2009
Location: Gibraltar, Gibraltar
Distribution: Fedora 20 with Awesome WM
Posts: 6,805
Blog Entries: 1

Rep: Reputation: 1291Reputation: 1291Reputation: 1291Reputation: 1291Reputation: 1291Reputation: 1291Reputation: 1291Reputation: 1291Reputation: 1291
Hi,

Have a look at 'head'. For example if you use:
Code:
ls | head -1
in any directory you'll only get the first filename as output. Assign that output to a variable as I said in my previous post and use it in the move command.

Kind regards,

Eric

Last edited by EricTRA; 01-29-2010 at 05:22 PM.
 
Old 01-29-2010, 06:04 PM   #7
MBybee
Member
 
Registered: Jan 2009
Location: wherever I can make a living
Distribution: PC-BSD / FreeBSD / Debian / Ubuntu / Win7 / OpenVMS
Posts: 438

Rep: Reputation: 57
Actually, this might also be a good use of find:

Code:
find . -atime +1 -type f -exec mv {} TMP \; # mv files older then 1 day to dir TMP
For example.

http://johnmeister.com/CS/UNIX/FIND/find-usage.html

I use stuff like this all the time when a directory gets out of control and is too large to do * type operations. No scripts needed.
 
Old 01-29-2010, 07:09 PM   #8
Supreme1012
LQ Newbie
 
Registered: Jun 2007
Location: Florida, USA
Distribution: Ubuntu 7.04, Kubuntu 7.04, Kubuntu 7.10
Posts: 22

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by EricTRA View Post
Hi,

Have a look at 'head'. For example if you use:
Code:
ls | head -1
in any directory you'll only get the first filename as output. Assign that output to a variable as I said in my previous post and use it in the move command.

Kind regards,

Eric
Idiot that I am, I spent an hour trying to find out how to get the name of the first file in a directory before I saw your post. I believe I have it figured out. lmk if you see anything that I could do to improve it.

I set up a small test with 2 directories and 10 randomly named files. Heres the script
Code:
#!/bin/bash

COUNTER=0
while [  $COUNTER -lt 5 ];
do
  FILE=$(ls -X ~/test/dir1/ | head -1)
  mv ~/test/dir1/$FILE ~/test/dir2/
  let COUNTER=COUNTER+1 
done
The next step is to make the directories variable so that I can change them easily. Plus I would like to have some sort of safeguards in place in case it messes up. Like a popup message telling me it finished or failed.


Thanks!
-Supreme1012
 
Old 01-29-2010, 07:29 PM   #9
PTrenholme
Senior Member
 
Registered: Dec 2004
Location: Olympia, WA, USA
Distribution: Fedora, (K)Ubuntu
Posts: 4,153

Rep: Reputation: 331Reputation: 331Reputation: 331Reputation: 331
Quote:
Originally Posted by Supreme1012 View Post
. . . The problem is that I can't access the directory because its so large . . .
What did you mean by "I can't access the directory?"
And how do you define the concept of "first" for a directory? Alphabetically? Chronologically? Size? Something else?

That being said, and assuming you meant "first" to be chronologically, this code might work:
Code:
#!/bin/bash
#
# Verify arguments, display help if none supplied.
[ $# -eq 0 ] && echo Usage: $0 {location of directory to split} [files per subdirectory] >/dev/stderr && exit 1
SOURCE=$1
[ -d "$SOURCE" ] || (echo $0: \"$SOURCE\" is not the name of a directory. > /dev/stderr; exit 2)
if [ $# -gt 1 ]; then
  SIZE=$((0$2))
else
  SIZE=20000
fi
[ $SIZE -le 0 ] && SIZE=20000
SET=0
# Define the function to create the name of the next target directory
function nextTarget() {
  local flag
  SET=$(($SET+1))
  TARGET=$SOURCE_$SET
  flag=0
  while [ -e $TARGET -a $flag -lt 20 ];do
    echo -n $0: The target directory \"$TARGET\" already exists.
    SET=$(($SET+1))
    TARGET=$SOURCE_$SET
    echo -n " Trying $TARGET"
    flag=$(($flag+1))
  done
  [ $flag -gt 0 ] && echo
  [ $flag -eq 20 ] && echo $0 Too many target directories already exist. Aborting. > /dev/stderr && exit 3
}
#Loop through the directory, splitting off the "top" $SIZE entries
while true;do
  list=$(ls -ABc1 $SOURCE | head -$SIZE)
  count=$(echo $list | wc -w)
  [ $count -lt $SIZE ] && break
  nextTarget
  mkdir $TARGET
  for fn in $list;do
    mv $SOURCE/$fn $TARGET/
    [ $? -ne 0 ] && echo $0: Error moving \"$fn\" from $SOURCE to $TARGET/. Aborting. > /dev/stderr $$ exit 4
  done
done
if [ "$SET" -eq 0 ]
then
  if [ $count -eq 0 ]
  then
    echo $0: \"$SOURCE\" appears to be an empty directory.
  else
    echo $0: \"$SOURCE\" contains fewer then $SIZE entries.
  fi
else
  nextTarget
  mv $SOURCE $TARGET
  [ $? -ne 0 ] && echo $0: Error renaming $SOURCE to $TARGET. Aborting. > /dev/stderr $$ exit 5
fi
exit 0
WARNING: The above is untested code.

Last edited by PTrenholme; 01-29-2010 at 07:31 PM. Reason: Oops! Left debugging stuff in the code
 
Old 01-29-2010, 09:57 PM   #10
Supreme1012
LQ Newbie
 
Registered: Jun 2007
Location: Florida, USA
Distribution: Ubuntu 7.04, Kubuntu 7.04, Kubuntu 7.10
Posts: 22

Original Poster
Rep: Reputation: 15
ok, thanks. I meant that Dolphin freezes when loading the directory and I have to kill the process or else it just hangs forever or is too slow to use. and yes, I wanted the files listed chronologically, which "ls -X" does I believe, so I used that in the small script I wrote. I have to admit that I dont understand most of what your script does. I need to run the script remotely so the directories have to be specified in the script, I cant put the script in the folder and click on it since the folder hangs.

I believe my script does everything I want it to do, but is there anyway to have it show a popup message when starting and after completing? or especially if it messes up
Code:
#!/bin/bash

COUNTER=0
while [  $COUNTER -lt 5 ];
do
  FILE=$(ls -X ~/test/dir1/ | head -1)
  mv -n ~/test/dir1/$FILE ~/test/dir2/
  let COUNTER=COUNTER+1 
done
How do I use "mv -n" instead of just "mv" without it stopping the script once it finds a duplicate?
 
Old 01-30-2010, 01:55 AM   #11
EricTRA
Guru
 
Registered: May 2009
Location: Gibraltar, Gibraltar
Distribution: Fedora 20 with Awesome WM
Posts: 6,805
Blog Entries: 1

Rep: Reputation: 1291Reputation: 1291Reputation: 1291Reputation: 1291Reputation: 1291Reputation: 1291Reputation: 1291Reputation: 1291Reputation: 1291
Hello,

Have a look at the man page of ls to find different ways to sort:
Code:
man ls
The script posted by PTrenholme has everything covered that you need in my opinion (haven't tested it, only read across) but in my opinion is way to difficult for you to understand as you stated yourself. If you start out the way you are doing right now you'll soon know the basics and will move on to more complicated scripts by yourself.

For the safeguards you're looking for, have a look at the script posted by PTrenholme . Read it carefully line by line, try to figure out what it does. For example the [ -d $SOURCE ] checks if the directory you provide as parameter when running the script exists.

Kind regards,

Eric
 
Old 01-30-2010, 02:57 AM   #12
Supreme1012
LQ Newbie
 
Registered: Jun 2007
Location: Florida, USA
Distribution: Ubuntu 7.04, Kubuntu 7.04, Kubuntu 7.10
Posts: 22

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by EricTRA View Post
Hello,

Have a look at the man page of ls to find different ways to sort:
Code:
man ls
The script posted by PTrenholme has everything covered that you need in my opinion (haven't tested it, only read across) but in my opinion is way to difficult for you to understand as you stated yourself. If you start out the way you are doing right now you'll soon know the basics and will move on to more complicated scripts by yourself.

For the safeguards you're looking for, have a look at the script posted by PTrenholme . Read it carefully line by line, try to figure out what it does. For example the [ -d $SOURCE ] checks if the directory you provide as parameter when running the script exists.

Kind regards,

Eric
I already checked out man ls and decided to use ls -X to get the files in chronological order. Ive been using the script some, and I have to say the one I wrote is really slow. Is there some preset default delays that are slowing it down? b/c it moves like 5 files per second, which with 20,000 files will take a very long time lol.

I feel that I have most of what PTrenholme posted figured out, but I would like to specify the SOURCE directory. would that be
Code:
SOURCE=$(~/test/dir1)
 
Old 01-30-2010, 04:03 AM   #13
ghostdog74
Senior Member
 
Registered: Aug 2006
Posts: 2,697
Blog Entries: 5

Rep: Reputation: 241Reputation: 241Reputation: 241
Code:
ls| sort -n | head -20000| xargs -i mv "{}" /destination
 
Old 01-30-2010, 11:48 AM   #14
Supreme1012
LQ Newbie
 
Registered: Jun 2007
Location: Florida, USA
Distribution: Ubuntu 7.04, Kubuntu 7.04, Kubuntu 7.10
Posts: 22

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by ghostdog74 View Post
Code:
ls| sort -n | head -20000| xargs -i mv "{}" /destination
This is perfect! I knew there had to be an easier way lol. Thanks!
 
Old 01-30-2010, 06:08 PM   #15
MBybee
Member
 
Registered: Jan 2009
Location: wherever I can make a living
Distribution: PC-BSD / FreeBSD / Debian / Ubuntu / Win7 / OpenVMS
Posts: 438

Rep: Reputation: 57
Quote:
Originally Posted by Supreme1012 View Post
This is perfect! I knew there had to be an easier way lol. Thanks!
Yup - you've got two oneliners now
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Question about moving a range of files with a Bash script surfrock66 Linux - Newbie 6 11-22-2009 04:37 PM
commands for bash script that handles files of varying number of lines BBFeltham Linux - Newbie 1 07-26-2008 11:18 AM
Issues moving large number of files gauge73 Linux - Software 7 05-08-2008 02:23 PM
bash script to download a number of things linksocc Linux - Software 3 12-10-2003 01:18 PM


All times are GMT -5. The time now is 11:51 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration