[SOLVED] sequential : how to find the missing numbers within a sequence of files that have sequential numbers attached to them?

BW-userx · 07-07-2017, 08:30 AM

Counting how many files one has is a simple matter, but when I got a few hundred and some are missing within the total sequence what is a quick way to find out which ones are missing within the sequence of the total and put them in a file listing only the numbers that are not presence within the sequence?

say we have here these numbers, whereas it is to be all numbers within that sequence of 1-19.
we search though the files with the attached numbers and only see these numbers.

1, 2 , 3, 5, 7, 8, 19.

the means to search this and report back would result in this means saying that, 4, 6, 9-18 are missing and spelling out the last sequence in numbered order as well. The whole sequence of missing numbers being printed out or into a file.

the actual pattern being.

Code:

FileName-xxx-xxxxxxx.ext

where the leading zeros are place holders. the sequence pattern for the numbers then is: 001, 002, 003, -- 010, 011, -- 120 etc

the middle three x's being the ones to look at to find the missing numbers within the entire sequence of numbers.

TenTenths · 07-07-2017, 08:50 AM

Not tested but you could try something like:

Code:

testfor.sh

#!/bin/bash

PREFIX=$1
SUFFIX=$2

for I in $(seq -f "%03g" 0 19) ; do

  CHECKFOR=${PREFIX}${I}${SUFFIX}

  if [[ ! -f ${CHECKFOR} ]] ; then

    echo -ne "${I}, "

  fi

done

Then run it in the folder with:

Code:

./testfor.sh /path/to/folder/FileName- -*.ext

BW-userx · 07-07-2017, 08:57 AM

Ok I got a script to get the numbers to use in a input file - removed the leading zeros, but the output is out of sequence.

Code:

#!/bin/bash

working_dir=/run/media/userx/250GB/NumberedFiles

while read file
do

f=$file
path=${f%/*}
xfile=${f##*/}
title=${xfile%.*}
ext=${xfile##*.}

numbers=${title#*-}
numbers=${numbers%%-*}
#remove all leading zero's
numbers=$(echo $numbers | sed 's/^0*//')

echo "$numbers" >> Numbers

done< <(find "$working_dir" -type f )

just a sniplet of the results.

Quote:

57
58
59
60
61
62
63
64
65
66
67
68
69
70
18
36
53
71
89
108
125
143
177
201
219
237
255
72
73
74
75
76

BW-userx · 07-07-2017, 10:21 AM

Ok this is defiantly not working yets.

the sequence is
1 - 270

Code:

#!/bin/bash

working_dir=/run/media/userx/250GB/numberedFiles

a=0

while read file
do

f=$file
path=${f%/*}
xfile=${f##*/}
title=${xfile%.*}
ext=${xfile##*.}
t
#get numbers off of files
numbers=${title#*-}
numbers=${numbers%%-*}

#remove all leading zero's
numbers=$(echo $numbers | sed 's/^0*//')

# make sure they are actual digits
if [[ $numbers =~ ^-?[0-9]+$ ]] ; then  NumberedArray[$a]=$numbers ; ((a++)) ; fi
done< <(find "$working_dir" -type f )

# put them in order of 1 - 270

sortedNums=( $( printf "%s\n" "${NumberedArray[@]}" | sort -n ) )

# print them into a file

for (( b= 0 ; $b < "${#sortedNums[@]}" ; b++ ))
{
 echo "${sortedNums[$b]}" >> Numbers
} 

#not working like I need
#print out missing numbers
#awk '$1!=p+1{print p+1"-"$1-1}{p=$1}'  Numbers

awk '$1!=p+1{print p+1}{p=$1}'  Numbers

giving me the results of

Code:

userx%slackwhere ⚡ production ⚡> ./getNumberOffFiles
162
164
172
181
186
195
259

using this in the script

Code:

awk '$1!=p+1{print p+1"-"$1-1}{p=$1}'  Numbers

gives me this

Code:

I know there are more missing than that. 164 - 170 for starters.

maybe I got that awk backwards or something I snagged it off the net.

it is not incrementing to the next valid number and printing out the missing ones between the first missing one and the next valid number.

BW-userx · 07-07-2017, 10:22 AM

Quote:

Originally Posted by TenTenths

Not tested but you could try something like:

Code:

testfor.sh

#!/bin/bash

PREFIX=$1
SUFFIX=$2

for I in $(seq -f "%03g" 0 19) ; do

  CHECKFOR=${PREFIX}${I}${SUFFIX}

  if [[ ! -f ${CHECKFOR} ]] ; then

    echo -ne "${I}, "

  fi

done

Then run it in the folder with:

Code:

./testfor.sh /path/to/folder/FileName- -*.ext

that is interesting, a lot shorter than what I've got.
I might have to play with it to get just the numbers off the file?

I'll give it a go just to see what that does.

BW-userx · 07-07-2017, 11:43 AM

well I got it to a point - and maybe this is too much for all of

Code:


c=1
  
for (( b = 1 ; $b < "${#sortedNums[@]}" ; b++ ))
{
 if [[ "${sortedNums[$b]}" != "$((c+1))" ]] ; then
     echo "Missing:  $((b+1))"
 fi
c=${sortedNums[$b]}
} 

#awk '$1!=p+1{print p+1"-"$1-1}{p=$1}'  Numbers

#awk '$1!=p+1{print p+1}{p=$1}'  Numbers

adding that to the entire script,

Code:

Missing:  162
Missing:  166 < present
Missing:  167
----
168
169
170
---
Missing:  172
----
173
174
175
---
Missing:  176 < present
Missing:  184 < present
Missing:  230 < present

awk

awk '$1!=p+1{print p+1"-"$1-1}{p=$1}' Numbers

Code:

I am just wanting the exact numbers missing printed in sequence

Laserbeak · 07-07-2017, 11:55 AM

Do you want a Perl version?

BW-userx · 07-07-2017, 12:47 PM

Quote:

Originally Posted by Laserbeak

Do you want a Perl version?

can it print out the missing numbers in sequence?
if range is between 1 - 100 and missing is 4 - 10, 44, 85 then it prints
4 5 6 7 8 9 10 44 85 NOT 4-10 and whatever else gets printed.

scasey · 07-07-2017, 02:39 PM

Quote:

Originally Posted by BW-userx

Ok I got a script to get the numbers to use in a input file - removed the leading zeros, but the output is out of sequence.

Code:

sort -n Numbers

as the last step should put the numbers in sequence.

Edit: Oh...you already got that...

I can almost see the perl in my minds eye. Be interested to see what Laserbeak comes up with.

BW-userx · 07-07-2017, 02:45 PM

Quote:

Originally Posted by scasey

Code:

sort -n Numbers

as the last step should put the numbers in sequence.

Edit: Oh...you already got that...

I can almost see the perl in my minds eye. Be interested to see what Laserbeak comes up with.

yeah I got the sequencing in order done so,

yeah lets just see if he is up for the challenge MUHahahahaha

scasey · 07-07-2017, 04:17 PM

I couldn't resist. Am interested in Laserbeak's comments and/or version.
Hopefully comments are self-explanatory.

Code:

#!/usr/bin/perl  
## ^^ set to location of your perl

$working_dir="/run/media/userx/250GB/NumberedFiles";
##~ $working_dir=".";   # testing
$max = $ARGV[0];   ## get max number from the command line
$max++;

## get list of file name in array
@files=`find "$working_dir" -type f`;

## remove leading and trailing parts
foreach $file (@files) {
	$file =~ s/^.*?-//;    #remove from beginning to first hyphen
	$file =~ s/-.*$//;	#remove from second hyphen to end
	$existnums[$file]=$file;  #save in array
}

## populate array with all the numbers: 1-input value 
for ($i = 1; $i < $max; $i++) {
    $allnums[$i] = $i;
}

## remove existing numbers from full list
foreach $nbr (@existnums) {
    $allnums[$nbr] = 0
}

## print out the remaining (i.e. missing) numbers
## note, no sorting required because the allnums array is populated in sequence
foreach $nbr  (@allnums) {
	if ($allnums[$nbr] ne 0) {      ## only print the entries that are not -0-
		print "$allnums[$nbr] ";   ## or 
##~ 		print "$allnums[$nbr]\n";  ## to do one per line
	}
}

print "\n";   ## when printing all on one line.

Run with /path/to/thisscript.pl maxvalue > savefile
or /path/to/thisscript.pl maxvalue to see on STDOUT

schneidz · 07-07-2017, 04:25 PM

what i would do is create a file containing the integers in the order you want.
create another file with the filenames.
and then diff the two files.

BW-userx · 07-07-2017, 04:49 PM

Quote:

Originally Posted by schneidz

what i would do is create a file containing the integers in the order you want.
create another file with the filenames.
and then diff the two files.

hummm funny

Code:

userx%slackwhere ⚡ production ⚡> diff Numbers FILES 
1,258c1,259
< 1
< 2
< 3
< 4

just prints out numbers then the file names

BW-userx · 07-07-2017, 04:51 PM

Quote:

Originally Posted by scasey

I couldn't resist. Am interested in Laserbeak's comments and/or version.
Hopefully comments are self-explanatory.

Run with /path/to/thisscript.pl maxvalue > savefile
or /path/to/thisscript.pl maxvalue to see on STDOUT

Nothing I get nothing -

Code:

userx%slackwhere ⚡ scripts ⚡> ./perl-find-missing-numbers maxvalue

userx%slackwhere ⚡ scripts ⚡> 


userx%slackwhere ⚡ scripts ⚡> ./perl-find-missing-numbers 

userx%slackwhere ⚡ scripts ⚡>

does it actually need a .pl tag/ext on the end of it- or is that just a naming convention to indicate a perl script?

chnged it - same output

then checked to be sure I got perl ..

Code:

userx%slackwhere ⚡ scripts ⚡> perl -version
This is perl 5, version 22, subversion 2 (v5.22.2) built for x86_64-linux-thread-multi

scasey · 07-07-2017, 05:19 PM

you need the output of

Code:

which perl

to match what's on the #! line.

on my server:

Code:

$ which perl
/usr/bin/perl

Hmm. Me too. let me see what I broke...
Oh. You get nothing if you don't supply a max value on the command line -- there should probably be a test for that:

Code:

./perl-find-missing-numbers.pl 270

(or whatever the current name of the script is...)
(your #! must be right, or you'd have got errors)