[SOLVED] About 5000000 file's i need trim

Vodkaholic1983 · 02-24-2011, 06:20 AM

Ok mate am lost now is this a sh script? if so i don't have sh on my server

also will this delete all numbers? the reason i ask is this bit

Code:

mv "${FILE}" "${FILE/#[0-9]* - /}"

I don't know what it does but the 0-9 bit scares me as some of my image's have numbers in them which i need to keep

last time i did this i had a file i moved in each folder than edited it abit like this you said before

Code:

sed -i 's/^......//' infile

the more dots it had the more it trim'ed from the start of the file's

I am really a newbie at linux so sorry if i read this wrong
Thank's

druuna · 02-24-2011, 06:55 AM

Hi,

Quote:

Originally Posted by Vodkaholic1983

Ok mate am lost now is this a sh script? if so i don't have sh on my server

This is indeed a shell script. The first line of the script itself (#!/bin/bash) points to bash (which just about every linux distribution has). Linux does not look at extensions, the .sh part is there for the user to recognize it as a shell script.

Quote:

also will this delete all numbers? the reason i ask is this bit

Code:

mv "${FILE}" "${FILE/#[0-9]* - /}"

I don't know what it does but the 0-9 bit scares me as some of my image's have numbers in them which i need to keep

This won't delete all the numbers. It (#[0-9]* - ) looks for files that start with numbers, followed by a space, a dash and a space.

123 - abc.abc.png will become: abc.abc.png
321 - abc.123.png will become: abc.123.png
345 - 123.123.png will become: 123.123.png

Quote:

last time i did this i had a file i moved in each folder than edited it abit like this you said before

Code:

sed -i 's/^......//' infile

the more dots it had the more it trim'ed from the start of the file's

You shouldn't use the sed examples given before in this post. It was assumed that you wanted to change the content of the file, not the file name itself.

How to use the script I provided in post #15:

- cd into the directory that holds Directory 1, Directory 2 etc,
- using an editor, safe the code that I provided (I used changer.sh as the script name, but it can be almost anything you like),
- change the permissions to make the script executable: chmod 750 changer.sh
- execute the script: ./changer.sh

About the red part: I would strongly suggest to create a temporary directory and copy some of the files and directories into that temporary directory and do the given steps there. If something goes wrong or isn't to your liking you still have the original files (ie: no harm done). Once you are satisfied with the results you can run the script from your original directory (and remove the temporary directory).

If unsure: Please ask.

Hope this clears things up a bit.

Vodkaholic1983 · 02-24-2011, 02:12 PM

Quote:

Originally Posted by druuna

Hi,
This is indeed a shell script. The first line of the script itself (#!/bin/bash) points to bash (which just about every linux distribution has). Linux does not look at extensions, the .sh part is there for the user to recognize it as a shell script.

This won't delete all the numbers. It (#[0-9]* - ) looks for files that start with numbers, followed by a space, a dash and a space.

123 - abc.abc.png will become: abc.abc.png
321 - abc.123.png will become: abc.123.png
345 - 123.123.png will become: 123.123.png

You shouldn't use the sed examples given before in this post. It was assumed that you wanted to change the content of the file, not the file name itself.

How to use the script I provided in post #15:

- cd into the directory that holds Directory 1, Directory 2 etc,
- using an editor, safe the code that I provided (I used changer.sh as the script name, but it can be almost anything you like),
- change the permissions to make the script executable: chmod 750 changer.sh
- execute the script: ./changer.sh

About the red part: I would strongly suggest to create a temporary directory and copy some of the files and directories into that temporary directory and do the given steps there. If something goes wrong or isn't to your liking you still have the original files (ie: no harm done). Once you are satisfied with the results you can run the script from your original directory (and remove the temporary directory).

If unsure: Please ask.

Hope this clears things up a bit.

Thank you this sound's like a dream to me
Test folder worked!

Here goes one big folder.

Wow lolol Thank you so much 100000 folder done in about 20 second's lol

If i could see you i would Kiss you

druuna · 02-24-2011, 02:53 PM

Hi,

This will work on a single directory:

Code:

#!/bin/bash

find . -maxdepth 1 -type f | sed 's%^\./%%' | grep "^[0-9]* - " | \
while read FILE
do
  mv "${FILE}" "${FILE/#[0-9]* - /}"
done

The above should work on single directories (again: test this first. It does work on my side, but yours might be different).

BTW: You do not have to move the script itself into the directories. I'm not familiar with your set-up so I keep it as simple as possible:

- Go to your home directory: cd ~
- Using an editor, safe the new script (I called it changer2.sh, but that's up to you)
- Make it executable: chmod 750 changer2.sh
- Go to the desired (test) directory: cd /some/dir/with/files
- To run the changer2.sh script in the directory you are now standing in: ~/changer2.sh

Quote:

Test folder worked!

Here goes one big folder.

Wow lolol Thank you so much 100000 folder done in about 20 second's lol

If i could see you i would Kiss you

It's always nice to see a happy penguin

Hope this helps.

Vodkaholic1983 · 02-24-2011, 03:01 PM

Quote:

Originally Posted by druuna

Hi,

This will work on a single directory:

Code:

#!/bin/bash

find . -maxdepth 1 -type f | sed 's%^\./%%' | grep "^[0-9]* - " | \
while read FILE
do
  mv "${FILE}" "${FILE/#[0-9]* - /}"
done

The above should work on single directories (again: test this first. It does work on my side, but yours might be different).

BTW: You do not have to move the script itself into the directories. I'm not familiar with your set-up so I keep it as simple as possible:

- Go to your home directory: cd ~
- Using an editor, safe the new script (I called it changer2.sh, but that's up to you)
- Make it executable: chmod 750 changer2.sh
- Go to the desired (test) directory: cd /some/dir/with/files
- To run the changer2.sh script in the directory you are now standing in: ~/changer2.sh

It's always nice to see a happy penguin

Hope this helps.

Thank you again!

I also have a question about htaccess could I post about it on this site do you think?

druuna · 02-24-2011, 03:07 PM

Hi,

Quote:

Originally Posted by Vodkaholic1983

Thank you again!

You're welcome

Quote:

I also have a question about htaccess could I post about it on this site do you think?

Yes, open a new thread in Linux Newbie or one of the other sub-forums, if that one seems more appropriate.

BTW: If this is solved, can you put up the [SOLVED] flag (first post -> Thread Tools menu).

Vodkaholic1983 · 04-13-2011, 12:48 PM

Hey again question does this "delete" files if there names are the same?
I run this with 25k files and after it was done there was only 8k

Ive reupload the missing files

H_TeXMeX_H · 04-13-2011, 02:09 PM

You should use the 'mv -n' option, or it will overwrite files (deleting them).

Vodkaholic1983 · 04-13-2011, 02:50 PM

Quote:

Originally Posted by H_TeXMeX_H

You should use the 'mv -n' option, or it will overwrite files (deleting them).

Hey thanks for that so is that

Code:

mv -n ./changer.sh

in the putty?
Thanks again

druuna · 04-14-2011, 12:04 AM

Hi

You don't use the mv -n on the script, you need to add -n to the mv command:

Code:

 mv -n "${FILE}" "${FILE/#[0-9]* - /}"

Linux is case sensitive which means that if you have these 3 files in one directory: Foo, FoO and FOO and move (mv) them to foo you end up with one file called foo, the -n option (Do not overwrite an existing file.) prevents that from happening.

Vodkaholic1983 · 04-14-2011, 06:52 AM

Quote:

Originally Posted by druuna

Hi

You don't use the mv -n on the script, you need to add -n to the mv command:

Code:

 mv -n "${FILE}" "${FILE/#[0-9]* - /}"

Linux is case sensitive which means that if you have these 3 files in one directory: Foo, FoO and FOO and move (mv) them to foo you end up with one file called foo, the -n option (Do not overwrite an existing file.) prevents that from happening.

Yea I worked that out in the end hehe thanks again

Vodkaholic1983 · 04-23-2011, 09:56 AM

Hi again (last lot of files) am on now
and the file names have change some

example

all now start with this

animelon.com animelon.com - 1 - akanbe itou_noiji nagato_yuki suzumiya_haruhi_no_yuutsu white 120.jpg
animelon.com animelon.com - 2 - akanbe itou_noiji nagato_yuki suzumiya_haruhi_no_yuutsu white 120.jpg
animelon.com animelon.com - 3 - akanbe itou_noiji nagato_yuki suzumiya_haruhi_no_yuutsu white 120.jpg

can this be changed to remove the "animelon.com animelon.com - 3 - "

Thanks again you guys have saved me soon very much work

MTK358 · 04-23-2011, 10:07 AM

Code:

sed -r 's/[a-zA-Z0-9]+\.[a-z]+[ \t]+[a-zA-Z0-9]+\.[a-z]+[ \t]+-[ \t]+[0-9]+[ \t]+-[ \t]+(.*)/\1/'

Vodkaholic1983 · 04-23-2011, 10:15 AM

Quote:

Originally Posted by MTK358

Code:

sed -r 's/[a-zA-Z0-9]+\.[a-z]+[ \t]+[a-zA-Z0-9]+\.[a-z]+[ \t]+-[ \t]+[0-9]+[ \t]+-[ \t]+(.*)/\1/'

Thanks could you show me where I add that in the change.sh file?
As I don't see any sed in there

Code:

#!/bin/bash

find $PWD -type d | grep -v "^$PWD$" | \
while read THISDIR
do
  cd "$THISDIR"
  ls | while read FILE
  do
    mv -n "${FILE}" "${FILE/#[0-9]* - /}"
  done
done

MTK358 · 04-23-2011, 10:18 AM

If you want the result in a different file:

Code:

sed -r 'expression' source_file > dest_file

If you want the original file to be overwritten with the new data:

Code:

sed -i -r 'expression' file

Hopefully you can figure it out now.

EDIT: Also, you shold NEVER use ls output in a script. It's meant for human reading only.

Use this instead to loop over files in a directory:

Code:

for FILE in *
do
    # stuff
done