creating symlinks except

kzo81 · 03-18-2017, 03:43 PM

Hi:-)

Now I have a working code:

Code:

#!/bin/sh

echo "[+] Creating symbolic links ..."

SD="/var/spool/storage/SD_DISK/"
ARRANGED_FOLDER="$SD/videos_by_day"

while read i
do

DAY=$(echo "$i" | cut -d'/' -f10 | cut -d'_' -f1 | sort | uniq)
	
	if [ ! -d "/$ARRANGED_FOLDER/$DAY" ]; then
		mkdir -p /"$ARRANGED_FOLDER"/"$DAY"
	fi
	
	cd "$ARRANGED_FOLDER"/"$DAY" || exit 1
	
	if [ `find "$ARRANGED_FOLDER" -samefile $i | wc -l` eq 0 ]; then 
		ln -sfn "$i"
	fi

done < $(find "$SD" -type f -name "*.mkv")

There is still some bug in it, since it complained for only one file:

/var/spool/storage/SD_DISK/20170215/09/20170215_093234_3E4A_ACCC8E565EC5/20170215_09/20170215_093234_29AF_ACCC8E565EC5.mkv: File name too long

It seems that now the script only creates symlinks of the newly arrived files.

Quote:

is that "suppose" to give you a file name? because it comes up empty when I run it. just run this simple version with a path you give it to search. any path will work it is not changing any data.

Yes, it gives me the day, which will be the folder to create later.
Thank you so much for the efforts you all make for me.

kzo81 · 03-19-2017, 06:19 AM

Guys, I did some tests on the video server. And still the for cycle version gives me the least error message.
The one that I created:

Code:

#!/bin/sh

SD="/var/spool/storage/SD_DISK/"
ARRANGED_FOLDER="/var/spool/storage/SD_DISK/videos_by_day"

for i in $(find "$SD" -type f -name "*.mkv"); do

DAY=$(echo "$i" | cut -d'/' -f10 | cut -d'_' -f1 | sort | uniq)
	
	if [ ! -d "/$ARRANGED_FOLDER/$DAY" ]; then
		mkdir -p /"$ARRANGED_FOLDER"/"$DAY"
	fi
	
	cd "$ARRANGED_FOLDER"/"$DAY"
	
	if ! ln -sfn "$i"; then
		echo "[-] Error: Could not create symbolic links"
	fi
done

The While version, this one:

Code:

#!/bin/sh


SD="/var/spool/storage/SD_DISK/"
ARRANGED_FOLDER="/var/spool/storage/SD_DISK/videos_by_day"


while read i; do

DAY=$(echo "$i" | cut -d'/' -f10 | cut -d'_' -f1 | sort | uniq)
	
	if [ ! -d "/$ARRANGED_FOLDER/$DAY" ]; then
		mkdir -p /"$ARRANGED_FOLDER"/"$DAY"
	fi
	
	cd "$ARRANGED_FOLDER"/"$DAY"
	
	if ! ln -sfn "$i"; then
		echo "[-] Error: Could not create symbolic links"
	fi
done < `find "$SD" -type f -name "*.mkv"`

Complains:

Quote:

+ DAY=�{ckא�A6�S��6�?i�x:�(�d�.QF��1<�A�ޙ��'>1`z��IG��Z��*r�+!Juu��m)T��,�O�l�U6{9P
+ [ ! -d //var/spool/storage/SD_DISK/videos_by_day/�{ckא�A6�S��6�?i�x:�(�d�.QF��1<�A�ޙ��'>1`z��IG��Z��*r�+!Juu��m)T��,�O�l�U6{9P ]
+ cd /var/spool/storage/SD_DISK/videos_by_day/�{ckא�A6�S��6�?i�x:�(�d�.QF��1<�A�ޙ��'>1`z��IG��Z��*r�+!Juu��m)T��,�O�l�U6{9P

I know that shellcheck website, and You suggest me not using a for loop for, but as you can see I don't get any error message when I use the for loop version.

The only think that still bothers me, that I still couldn't implement a check that checks if an original file has a symlink already.
That command substitution is way complicated for me, I checked your suggested code, with the corrected if clause and it is doing terrible things, I mean this code:

Code:

SD="/var/spool/storage/SD_DISK/"
ARRANGED_FOLDER="/var/spool/storage/SD_DISK/videos_by_day"

while read FILENAME
do

 c=$FILENAME
 xpath=${c%/*} 
 xbase=${c##*/}
 
 xfext=${xbase##*.}
 xpref=${xbase%.*}
 
 path=${xpath}
 fname=${xpref}
 ext=${xfext}

ckfile="$fname"."$ext"

DAY=$(echo "$FILENAME" | cut -d'/' -f10 | cut -d'_' -f1 | sort | uniq)
	
	DailyFiles="$ARRANGED_FOLDER"/"$DAY"
        mkdir -p "$DailyFiles" 

        # check to see if in directory already / which it should not be?
[[ -h "$DailyFiles/$ckfile" ]] || ln -sfn $FILENAME "$DailyFiles/$ckfile"

done < $(find "$SD" -type f -name "*.mkv")

output:

Quote:

ln: /var/spool/storage/SD_DISK/videos_by_day/�h�c��t�e�c�W�뙁�@ø:�yw�� ig&�f�ȯ��J��.E��i�I�S4�`{�;�A
!kr��/�h�c��t�e�c�W�뙁�@ø:�yw�� ig&�f�ȯ��J��.E��i�I�S4�`{�;�A
�QI��O�2�e#f��L9�Hk��̀,�s��ȷ��[ !kr��_gg6�o-!��^��
-`��C�W��=H��I�wo�9[�?(zT�b�� zK��Kr��?n��p��00Ѹ��
��6�S%A�: File name too long

and if I put set -x in the script:

+ xbase=_B�X��Ѕ��I*�H�3ar�$��0�q��^�Gߓ=GE��׹��![��(��?��Dx�E��=�7;��x�p��q�^Wa�ܟ��v�7n
2��x��+��.��sp;��516�sR��m:��
z��?��T)�eB�:�t��C��F�V#.�5 ��}�Һ��׾��ӵ|�=�(�u'��
g��{�
+ xfext=�5 ��}�Һ��׾��ӵ|�=�(�u'��

I really appreciate your help, but the problem of - to much SD_DISK write, when rerun the script - is still there.

I noticed that if I omit the -fn arguments, it gives me an error, but at least the script didn't recreate the link:

Code:

#!/bin/sh

SD="/var/spool/storage/SD_DISK/"
ARRANGED_FOLDER="/var/spool/storage/SD_DISK/videos_by_day"

for i in $(find "$SD" -type f -name "*.mkv"); do

DAY=$(echo "$i" | cut -d'/' -f10 | cut -d'_' -f1 | sort | uniq)
	
	if [ ! -d "/$ARRANGED_FOLDER/$DAY" ]; then
		mkdir -p /"$ARRANGED_FOLDER"/"$DAY"
	fi
	
	cd "$ARRANGED_FOLDER"/"$DAY"

	ln -s "$i" >/dev/null 2>&1
done

exit 0

So at least it's a workaround for now.

pan64 · 03-19-2017, 10:08 AM

I do not really understand where those strange chars are coming from. Are they filenames or something else?
if the for cycle and while loop work differently that means the while is ok and the for loop makes something wrong.
if

Quote:

+ DAY=�{ckא�A6�S��6�?i�x:�(�d�.QF��1<�A�ޙ��'>1`z��IG��Z��*r�+!Juu��m)T��,�O�l�U6{9P

that means calculating the DAY is not ok.

BW-userx · 03-19-2017, 02:22 PM

I got a ask you do you know what a soft link basically is? It is just a fake copy of the same file pointing back to the original file. it looks feels and smells just like the original one. Q2: so how do you check to see if a file is present within a directory? apply the same methodology.

-L FILE
FILE exists and is a symbolic link (same as -h)

Code:

if [[ -L "$strFile" ]] && [[ -a "$strFile" ]];then 
  echo "'$strFile' link present"; 
fi

that bit of code I gave you that you didn't understand I. That slices up the complete path to a file and breaks it down to path to file - file name - extension.

BW-userx · 03-19-2017, 02:41 PM

you do not need all of this code

Code:

	if [ ! -d "/$ARRANGED_FOLDER/$DAY" ]; then
		mkdir -p /"$ARRANGED_FOLDER"/"$DAY"
	fi

Just this will work because mkdir does the check to see if a directory by that name is already there for you, all you need to do is add just this line.

Code:

mkdir -p /"$ARRANGED_FOLDER"/"$DAY"

what that leading / slash is for I do not know. I am surprised it works. Because if it is for a root dir on your SD Card then that should be within your variable name instead. just coding styles

Code:

ARRANGED_FOLDER="/root-new-whatever-dir"

example

Code:

userx@voider⚡️~⚡️$ newdir=/tmp
userx@voider⚡️~⚡️$ ls $newdir
cron.1FMpX9
userx@voider⚡️~⚡️$ ls /tmp
cron.1FMpX9

kzo81 · 03-19-2017, 02:47 PM

Hi, I know what is a softlink. Though I had rarely used it until this problem emerged with this axis video server.
I also understand the test tool. As I mentioned, I achieved the effect what I wished by omiting arguments (ls -sfn).
I think I'll rework the code tomorrow to a WHILE cycle, I don't really understand why it doesn't work the same as the FOR version of the code.

This is wrong I, know, but it's working:

Code:

#!/bin/sh

SD="/var/spool/storage/SD_DISK/"
ARRANGED_FOLDER="/var/spool/storage/SD_DISK/videos_by_day"

for i in $(find "$SD" -type f -name "*.mkv"); do

DAY=$(echo "$i" | cut -d'/' -f10 | cut -d'_' -f1 | sort | uniq)
	
	if [ ! -d "/$ARRANGED_FOLDER/$DAY" ]; then
		mkdir -p /"$ARRANGED_FOLDER"/"$DAY"
	fi
	
	cd "$ARRANGED_FOLDER"/"$DAY"

	ln -s "$i" >/dev/null 2>&1
done

exit 0

But this doesn't work:

Code:

#!/bin/sh

set -x 

SD="/var/spool/storage/SD_DISK/"
ARRANGED_FOLDER="/var/spool/storage/SD_DISK/videos_by_day"

while read i; do

DAY=$(echo "$i" | cut -d'/' -f10 | cut -d'_' -f1 | sort | uniq)
	
	if [ ! -d "/$ARRANGED_FOLDER/$DAY" ]; then
		mkdir -p /"$ARRANGED_FOLDER"/"$DAY"
	fi
	
	cd "$ARRANGED_FOLDER"/"$DAY"
	
	ln -s "$i" >/dev/null 2>&1

done < `find "$SD" -type f -name "*.mkv"`

This gives an error:

Quote:

/var/spool/storage/SD_DISK/20170319/10/20170319_102601_27E2_ACCC8E565EC5/20170319_10/20170319_102601_2F6F_ACCC8E565EC5.mkv: no such file

The FOR version runs seamless. I don't know why. That file doesn't exist on the original place, only the While version of the code finds it...

BW-userx · 03-19-2017, 02:56 PM

this

Code:

[[ -h "$DailyFiles/$ckfile" ]] || ln -sfn $FILENAME "$DailyFiles/$ckfile"

can be written like this too
if file link not there then create one.

Code:

[[ ! -h "$DailyFiles/$ckfile" ]] && ln -sfn $FILENAME "$DailyFiles/$ckfile"

-s, --symbolic
make symbolic links instead of hard links

is this really needed? you're logic is this - if link not there then make one else skip - right?
-f, --force
remove existing destination files

-n, --no-dereference
treat destination that is a symlink to a directory as if it were a normal file

BW-userx · 03-19-2017, 03:00 PM

why are you going into the directory?

Code:

cd "$ARRANGED_FOLDER"/"$DAY"

all of your paths are already set to work outside of directory regardless if your inside it or not. so what is the point of being inside of it? just trying to understand your logic.

kzo81 · 03-19-2017, 04:02 PM

Quote:

Originally Posted by BW-userx

why are you going into the directory?

Because if I dont, it'll create the symlinks in the directory where the script is. If you look at the man page of ln, see the 2nd form. It says that : "create a link to TARGET in the current directory".

If I only use ln -s, it wont recreate the symlink which is already created before although it throughs an error, which I redirect to the black-hole:-)

BW-userx · 03-19-2017, 04:45 PM

Quote:

Originally Posted by kzo81

Because if I dont, it'll create the symlinks in the directory where the script is. If you look at the man page of ln, see the 2nd form. It says that : "create a link to TARGET in the current directory".

I do not specify remember if I've ever ran a ln -s like this outside of both directories before on cli - but I do think I have. I got a give that a try and see the results on that one.

Quote:

Originally Posted by kzo81

If I only use ln -s, it wont recreate the symlink which is already created before although it throughs an error, which I redirect to the black-hole:-)

I understand the -s it is the other two I was having issues with, specifically -f forcing it to link when there is no link to begin with. Just being cautious ??

kzo81 · 03-19-2017, 04:55 PM

You guys told me about string manipulation.
Do you know an elegant way of extracting the red part of this?

i="/var/spool/storage/SD_DISK/20170319/10/20170319_102601_27E2_ACCC8E565EC5/20170319_10/20170319_102601_2F6F_ACCC8E565EC5.mkv"

something like this: echo "${i%%_*.mkv}"

kzo81 · 03-19-2017, 05:00 PM

Quote:

Originally Posted by BW-userx

it is the other two I was having issues with, specifically -f forcing it to link when there is no link to begin with. Just being cautious ??

first I thought that ln -sf will help me, but it didnt. Same goes with ln -sfn.
And interestingly many tools are not available on this videoserver. For instance there is no awk, but sed with no possibility of using arguments -oE... So I have to cook from that is available

BW-userx · 03-19-2017, 05:11 PM

test it in your terminal to see what it does first like this

Code:

userx@slackwhere⚡️~⚡️$ newname=20170319_102601_2F6F_ACCC8E565EC5.mkv
userx@slackwhere⚡️~aname=${newname%%_*}
userx@slackwhere⚡️~⚡️$ echo $aname
20170319

WTH? go the other way ? using this one --> #

http://tldp.org/LDP/abs/html/string-manipulation.html

look in that and you will find it

Code:

userx@slackwhere⚡️~⚡️$ aname=${newname#*_}                                      
userx@slackwhere⚡️~⚡️$ echo $aname                                              
102601_2F6F_ACCC8E565EC5.mkv

kzo81 · 03-19-2017, 05:28 PM

I tested it:

Code:

a="20111108_XYZA_121212.mkv"
echo $a "${a%%_*}"
20111108_XYZA_121212.mkv 20111108

But the string is much longer. I could eliminate the full path in front of the mkv file

Code:

i="/var/spool/storage/SD_DISK/20170319/10/20170319_102601_27E2_ACCC8E565EC5/20170319_10/20170319_102601_2F6F_ACCC8E565EC5.mkv"
echo "${i##/*/}"

20170319_102601_2F6F_ACCC8E565EC5.mkv

how could I go further and extract only the 8 characters, there is no similar example in that tutorial.

BW-userx · 03-19-2017, 06:33 PM

Quote:

Originally Posted by kzo81

I tested it:

Code:

a="20111108_XYZA_121212.mkv"
echo $a "${a%%_*}"
20111108_XYZA_121212.mkv 20111108

But the string is much longer. I could eliminate the full path in front of the mkv file

Code:

i="/var/spool/storage/SD_DISK/20170319/10/20170319_102601_27E2_ACCC8E565EC5/20170319_10/20170319_102601_2F6F_ACCC8E565EC5.mkv"
echo "${i##/*/}"

20170319_102601_2F6F_ACCC8E565EC5.mkv

how could I go further and extract only the 8 characters, there is no similar example in that tutorial.

yes there is I think

${string

osition:length}

Extracts $length characters of substring from $string at $position.

Code:

 again=${newfile#[0-9]*}

using the [0-9] every number 0 thru 9 - first number