[SOLVED] Bash script: Search folder names in file and cp text after "," into new file

Well, my first real question, sorry for asking such a big thing. Maybe, with some help I can figure out the missing pieces by myself. I have spend some evenings on figuring this out so my description can be a bit uberthought - please ask if it's too chaotic :P


I want to write a bash script (or python script) which helps me sorting a download folder.

This should be done by copying similar named files in subfolders named after the first 15 chars of the filename or create a subfolder if there is none present.

Then the script should make a new file in the (new) subfolder where a part of a logfile gets pasted including other text.

At last, it should copy the matching files to the subfolder.


To be more concrete I thought of the following steps:

check if there is a subfolder which name is similar to the first (15*) chars of a file in the script folder

if not create a subfolder and name it after the first 15* chars of the lonleyfile

find in  ./.scripts/text_file.log a line beginning which is similar to (the first 15 chars of) the subfolder_names from the script folder

get the part_of_the_line_after the comma ","

create a new file in the subfolder

paste a piece of text and text_including$part_of_the_line_after into a new_file_in_subfolder.desktop named after the subfolders first 15 letters

check if there are any files which are names similar with the subfolder_names and copy them into the created subfolder.

For example:

$ls
Abelmosch 123
Acyandstuff
Blahontea
Anther_All_pics
Greenandfunny
NotInTHEGARDEN01
Abelmosch 123.jpg
Blahontea.wmv
Green_introductino to gardening.pdf
...

cat specific_text_file.log
Downloaded Flower Pics.log
Abelmosch 123,Abelmoschus
Acyandstuff,Acalypha
Blahontea,Anisodontea
Anther_All_pics,Anthericum
Greenandfunny,Anaphalis
NotInTHEGARDEN01,Clitoria
...

*runningmagicscript*

$ls -R
.:
Abelmosch 123
Acyandstuff
Blahontea
Anther_All_pics
Greenandfunny
NotInTHEGARDEN01

./Abelmosch 123
Abelmosc.desktop
Abelmosch 123.jpg

./Acyandstuff
Acyandst.desktop

./Blahontea
Blahonte.desktop
Blahontea.wmv

./Anther_All_pics
Anther_A.desktop

./Greenandfunny
Greenan.desktop
Green_introductino to gardening.pdf

./NotInTHEGARDEN01
NotInTHE.desktop

cat ./Greenandfunny/Greenan.desktop

[Desktop Entry]
Version=1.0
Type=Link
URL=https://en.wikipedia.org/wiki/Anaphalis

I have a snippet which does some part of the job.


script2folder.sh

#!/bin/bash

find * -prune -type f ! -name "*.sh" | while read file;
do

	f=$(basename "$file")
	f1=${f%.*}
	mkdir "$f1"
#	mv "$f" "$f1"
done

Two points I assume important:
a) the string containing the file/subfolder name should be similar, not exactly matching.
b) the script should check if the things it creates already exist, if so, goto next step

Maybe it is better to check if the whole filename matches and exclude spaces, underscores and file endings (like all $foo after the last dot).

Thank you for reading this far

In bash you can use parameter expansion in the form of ${parameterffset:length} to get a part of the string in the variable like

echo "${VAR:0:15}"

which would take the first 15 characters.
As for your example a process substition would be better:

while IFS='' read -r FILE; do
    BASENAME=${FILE##*/}
    DIRPATH=${FILE%/*}
    NOEXT=${BASENAME%.*}
    FIRST15=${NOEXT:0:15}
    SUBDIR=$DIRPATH/$FIRST15
    [[ -d $SUBDIR ]] || mkdir -p "$SUBDIR"
    cp -v "$FILE" "$SUBDIR" ## mv?
    # ... do other things
done < <(exec find -prune -type f ! -name "*.sh")

## Is it only for the files in the current directory? If so some steps above could be skipped.

Thank you konsolebox, now I also have variables I can use further on!

If the .desktop file doesn't exist in the (created)subfolder, then the script should make it. Going one step deeper in the path. But that I can solve with a if ! present then. I also made the part which reads out my logfile.

#!/bin/bash

while IFS="," read f1 f2
do

echo -e "[Desktop Entry]\nVersion=1.0\nType=Link\nURL=URL=https://en.wikipedia.org/wiki/$f2" > "$f1.desktop"

done < specific_text_file.log

If I use it within your loop, does the second IFS= change the behaviour of your loop?

Edit: I tested your script, it doesn't do anything. And there is no error message. I tried echoing the variables in the loop and nothing happens too.

Edit2: When I change -prune to -maxdepth 1 it works pretty well!! I also changed cp to mv as you suggested

Btw: Sorry for posting in the non-*nix-section...

How many entries does your specific_text_file.log contain? If it's many wouldn't that produce more than one .desktop files within the directory? If one .desktop file does exist, how about the others in specific_text_file.log which could have been not included yet? Also guessing that f1 refers to the form which has the first 15 characters, echo -e "[Desktop Entry]\nVersion=1.0\nType=Link\nURL=URL=https://en.wikipedia.org/wiki/$f2" > "$f1.desktop" would actually just overwrite the contents of "$f1.desktop". Is that intended or should it be appended? If data is updated that could also possibly cause redundancy with old data, some old data would remain even if they (the files referred) are no longer there. But this wouldn't apply if you wouldn't update it anyway, however if you expect new files to have similar first 15 character names with the old ones then you have to pick a solution whether you'll have to reset and refresh the .desktop files or not.

Is it also necessary to have specific_text_file.log? Are we basing on the contents of it or the files we found? Is it a temporary form for <first15>,<original_filename>? It might no longer be necessary in the process. Also, filenames even the first 15 characters of those could have commas in them as well so if you do have to use the logfile, then better use other separators which are never part of filenames at least practically like | or tabs.

Btw, if you mean adding the whole loop as a subloop of the previous loop would not affect the read command if the read command would have its context-specific assignment of IFS done everytime it's called i.e. compounded in single statement like IFS='' read something as everytime the line is called IFS gets updated back.

while IFS='' read ...; do
    while IFS=, read ...; do
        ...
    done < some_file.txt
done < <(exed find ...)

Many many thanks konsolebox, your last comment got me working better. I wanted as less code as possible (possible for me^^), so I went through the whole thing again. It works now *enough* for me and I comment it now and post my solution. Ah, I'm proud, my first homemade script YAY!!

Let's start by assuming that we get 2 parameters passed on by a download manager as soon as the download is finished <id> and <name>. The application is also capable of running a script after it finishes the download. The script now creates a folder called done/$name in the users download folder. Then it creates the .desktop file, using the <id> to paste a path to the download site.

I use this link to get more information on the topic from the website for my Zim desktop wiki (manually, until I learn to use Scrapy Scrapy would paste an excerp of the scraped information in my Wiki, already in Zim style formatted).

I assume that I download files and folders at this point we have to take care of this. After creating the .desktop file in done/$name the script checks if the folder .notready/$name exists. If so, it moves its contents to done/$name, waits until mv is finished and then deletes the folder .notready/$name. If the folder doesn't exists in the first place we have downloaded a file. We move the file .notready/$name to done/ wait for mv (do we need to?) and viola, we are done.

#!/bin/bash

# This script will be started as the download is finished.
# It copies the dl-files to the done/$fname15 folder, makes .desktop links and cleans up afterwards.

# Let us get the passed on parameters / format the names of the strings

fileid=$1
filename=$2


# Be flexible - shorten long boring names and provide them as variables

fname="${filename%.*}"
fname15="${fname:0:15}"


# Set working dir to shorten the script - has not worked, but has created funny folders all over the place xD
# cd "~/downloads/files/"


# create done/$name directory from file name

mkdir -p "/home/derpina/downloads/files/done/$fname15"


# make .desktop file with a link to the source website

echo -e "[Desktop Entry]\nVersion=1.0\nType=Link\nURL=https://my-beloved-linux-iso-site.com/search/?id=$fileid" > "/home/derpina/downloads/files/done/$fname15/$fname15.desktop"


# If folder .notready/$filename exists mv .notready/$filename/* to done/$fname15; rmdir .notready/$filename else move .notready/$filename to done/$fname15; wait for it... fi

if [ -d "/home/derpina/downloads/files/.notready/$filename" ]; then

    mv "/home/derpina/downloads/files/.notready/$filename/*" "/home/derpina/downloads/files/done/$fname15"
    wait mv
    rmdir "/home/derpina/downloads/files/.notready/$filename"

else

    mv "/home/derpina/downloads/files/.notready/$filename" "/home/derpina/downloads/files/done/$fname15"
    wait mv

fi

Sorry for beeing too specific at the beginning. I thought I can solve this by solving small parts of the puzzle and then stacking them together like Lego bricks.

A last question remains: How does someone who is familiar with programming start things like this or a bit bigger projects? Do you draw a scheme?

I'm not really a veteran programmer compared to others but I do write schemes or create plans for projects sometimes but it depends on the complexity or value of the target. Sometimes it's only when I encounter a complex part where I try to plan well on how it should be done. Sometimes a simple draft is enough. And sometime I just imagine it and let the code in skeletal form be the scheme itself. Commercial projects are different from hobby projects. Commercial projects most of the time needs serious thinking at first. Hobby projects like mine didn't really start at serious planning. Everytime I had an idea I just try to implement it right away like playing a game but still try to make it perfect everytime and it just grew. Although I did take serious planning to some of it.

Btw I wonder but I think you no longer have to call wait mv.