Welcome to the most active Linux Forum on the web.
Go Back > Forums > Linux Forums > Linux - Newbie
User Name
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!


  Search this Thread
Old 10-02-2006, 07:42 PM   #1
LQ Newbie
Registered: Oct 2006
Posts: 8

Rep: Reputation: 0
Question Need help with a short Linux command line script - working with split files

Hi All,

First off, thank s in advance to anyone that can help me on this one..

The following is what I want to script. I know bits and pieces of it already , but I am missing the more complicated sections that work with variables, etc.. here goes:

# /bin/sh
## First, I want to split the main file into smaller files of 850 lines each

cd /splitfiles

split -l 850 /sourcefolder/mainfile.txt part

## Result: partxaa.txt , partxab.text etc.

## Next, I want to count how many pieces were made

ls *.txt | wc -w

## Result: x number of text files
## ie. two text files would be output: 2

** Now I need help!!!

## Next, I want to create x number of folders in a directory in the amount 
## of the number given from amount of text files (following our example from above
## this would be 2). I have a list of folder names in file dirnames.txt, each on its 
## own line. From our example I want to create 2 folders with 2 random name selections from 
## the folder name list, for example names: foo foo2

cat dirnames.txt ...some command - pull 2 random names from the list

... some command sequence as described above then:

cd /home/var/www/mysite
mkdir foo *randomfoldername1* foo2 *randomfoldername2*

## once the folders have been created, I want to copy files to each folder
## The files for each folder are the same, EXCEPT for our split text files from the first step

[root@ mysite]# 
cp /path/to/filesandfolderstocopy/*.* mysite/foo *randomfoldername1* then
cp /path/to/filesandfolderstocopy/*.* mysite/foo2 *randomfoldername2*

## If, possible, could the two lines above be condensed to something with a variable? $1?

## Now grab first split file from our first step, and put it into the first folder created:

... some command to identify the first file in the split files folder

mv /splitfiles/partxaa.txt /home/var/www/mysite/foo/partxaa.txt

## Now continue in the same fashion for the next file or files in the "splitfiles" folder:

mv /splitfilespartx..etc /home/var/www/mysite/foo2/partxab.txt ..etc

## continue until all split files are moved to their folders. There will be enough folders present
## because we counted the number of splits in the second step.

## Finally, chmod all folders 755 - using a helpful code snippet I found:

find /home/var/www/mysite -maxdepth 1 -type d ...collect the directory names ie. foo , foo2 
chmod 755 foo , foo2 , others

## DONE!
Old 10-02-2006, 10:05 PM   #2
Senior Member
Registered: Nov 2002
Location: Edmonton AB, Canada
Distribution: Gentoo x86_64; Gentoo PPC; FreeBSD; OS X 10.9.4
Posts: 3,760
Blog Entries: 4

Rep: Reputation: 78
This seems a rather arbitrary thing to do. Can you explain what you are trying to accomplish?

Anyway, I don't think there is a way to pick a random number in shell directly, so try something like:
RAND_UPPER=$(cat dirnames.txt | wc -l)
RAND_NUMBER=$(ruby -e "puts rand(1-${RAND_UPPER}+1)")
RAND_filename=$(sed -n ${RAND_NUMBER}p dirnames.txt)
Here, we set a lower bounds of 1, and an upper bounds of the number of filenames in your file.
We get a random number from Ruby, and grab a line from your file using said random number.

You will of course need to loop over this to grab as many names from your file as you have your 'chunks'.
Also, this makes no provisions to make sure it doesn't grab the same name twice. You will have to keep track of your first and subsequent 'random' line numbers and try for another if you pull a duplicate.

As for your 'find' line, have a look at the '-exec' command in the find man page. It will run a command for each file it finds.

This would really be _so_ much easier if you were using a 'real' language like Ruby, Python, or Perl...

Anyway, hopefully I have given you a few ideas. Work some more on it, and post back if you get stuck...

Last edited by bulliver; 10-02-2006 at 10:09 PM.
Old 10-03-2006, 10:54 PM   #3
LQ Newbie
Registered: Oct 2006
Posts: 8

Original Poster
Rep: Reputation: 0
Lightbulb Thanks Bulliver, I came across this script in Perl which may help

Hi Bulliver, others,

Thanks for your tip about my script. I think Perl might be the way to go on this one, since I believe it can be executed from the command line or as a CRON job...Admittedly, I really have no clue how to code Perl, just a general understanding of it through countless trial and error efforts.

I've found the a script for my first step, named, which is invoked by the following:

Step 1:
[root@]# perl -l 500 wordlist.txt file

Where -l = lines for each file, then source file, then output file prefix

I'd list the URL for the source, but I don't have sufficient privileges yet.

In my next step, I originally wanted to call a command line function that counts the number of files created, then uses that count to pull random new folder names from a text list of possible names.

Bulliver pointed out the error with this step: there is a chance that the same new folder name could be pulled from the list, thus screwing up the whole process.

So, now I will simplify it, but I don't know the perl for doing the rest:

Step 2: Collect all file names created by the split process, for example: fooaaa, fooaab, fooaac but *without* the .txt extension for now - basename I think...

Step 3: Go to a selected web directory and create new folders named according to the filenames collected in step 2; for example /var/www/user1/web/fooaaa , then make /var/www/user1/web/fooaab , /var/www/user1/web/fooaac etc. Make sure to create a folder for each name...

Step 4: After all folders have been created, go to the first folder, fooaaa, and copy files from a predefined source on the server: example copy /home/sourcefilesandfolders/*.* to /var/www/user1/web/foooaaa/ .

Step 5: Among the folders and files copied to these new directories will be a sub-folder named: abc , such that the complete directory path to folder abc would be: /var/www/user1/web/fooaaa/abc . Now return to the directory containing the split files and move file fooaaa to /var/www/user1/web/fooaaa/abc/mylist.txt - making sure to rename it to mylist.txt .

Repeat in the same fashion for each remaining split file, moving it to sub-directory abc with name mylist.txt of the folder that was created with its original "split name"...

In the end, we would have:

Step 6: Chmod all /var/www/user1/web/foo* folders to 755

Step 7: Open crontab for user1 and begin with the following commands:

0 0 * * * /usr/bin/php /var/www/user1/web/fooaaa/abc/myphpfunction.php
Step 8: Starting at Cron time 0 0 * * * . Write command for myphpfunction.php for each foo* directory created, but for each successive command, increment the Cron Exec time by 5 minutes. When 12 sets have been written to the crontab, increment hour value by 1 (since 12 x 5 minutes = one hour has past) and continue until commands for *every* folder created have been written...

It is not likely that 24 hours worth of new folder commands will be needed, since this would be 12/hour x 24 hours equaling 288 unique commands, which is essentially 288 new foo* created folders.

Thanks again for the help!
Old 10-04-2006, 02:08 AM   #4
LQ Newbie
Registered: Oct 2006
Posts: 8

Original Poster
Rep: Reputation: 0
Lightbulb OK...I switched to Perl, and this is what I have so far, but I get an error


Ok I switched over to Perl and was able to scrounge up some snippets for the following process, which effectively takes care of steps 1-3 in my amended plan, * SEE POST IMMEDIATELY ABOVE*.

Now, however, I want to copy the contents of the following variable, assigned by my file path: $sourcefiles = "/home/soure"; . Inside folder "source" is all the files I wish to copy over to my new folders...

I tried to use "foreach" twice, being naive and not aware of any other way of copying (not moving) files..

I tried copy::file, but there is no wildcard for copying all files as in copying files from the Command Line...


use File::Find;
use File::Copy;

$userdir = "/var/www/web12/web/";
$sourcefiles = "/home/source";
$sourcedir = "play";
$ftd1 = "bm";
opendir $sourcedir, ".";
@contents = grep /$ftd1/, readdir $sourcedir;
closedir $sourcedir;
foreach $listitem ( @contents )
  chdir $userdir;
  mkdir $listitem, 0755;

opendir $sourcefiles, ".";
	@contents2 = grep !/^\.\.?$/, readdir $sourcefiles;
	closedir $sourcefiles;
	foreach $listitem2 ( @contents2 )
		copy($listitem2, $userdir.$listitem."/")
Thanks again to all for any help!


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
bash script 'for each command line argument' true_atlantis Linux - Newbie 3 01-28-2009 01:51 PM
working with hidden files from the command line indygreen Linux - General 4 01-04-2006 08:50 AM
edit wav files via command line in a script legolin Linux - Software 4 12-21-2005 10:09 AM
Bash Script, no new line for echo command jorisb Linux - General 5 11-05-2005 12:08 AM
print files in PDF or html format from the linux command line IBKnobel Linux - Software 3 07-12-2004 09:29 PM

All times are GMT -5. The time now is 08:10 PM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration