Linux - KernelThis forum is for all discussion relating to the Linux kernel.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Lately I noticed that thunderbird changed it's directory structure, so now not reading all emails. TBird files containing the actual emails, that you scrub addresses from have no extension, so been trying to write a function that will recursively read/record all these email files for processing as I think my "grep" statements are working correctly.
However having issues. Here is my code. It is not reading any files at all to my knowledge:
Code:
# Function to read directories recursively
function get_file_array {
for f in $1/*; do
if [ -d "$f" ]; then
get_file_array $file;
else
# Skip files with extensions .dat .msf
fnam=$(basename "$f");
fext="${fnam##*.}";
fnam="${fnam%.*}";
if [ -z "$fext" ]; then
fil_ray+=("$fnam");
fi
fi # end if -d $f
done # end do
} # end function
I'm calling it from this loop:
Code:
# Declare the Local Folders dir for Thunderbird
ptdir="/home/$myuser/.thunderbird"; # Parent dir
prfln=$( cat "$ptdir/profiles.ini" | grep "Path=" ); # Read profile
profn="${prfln:5}"; # Substring profile name
tbrot="$ptdir/$profn"; # TBird root dir
flray=("Mail/Local Folders" "ImapMail"); # Default Mail Folders
for i in "${flray[@]}"; do # Loop for each Folder set
tbdir="$tbrot/$i/"; # Set processing Folder
get_file_array "$tbdir"; # Run file array process
echo "FR=> $fil_ray"; # Show results
done # end for i loop
exit;
Could use some help figuring out why my function is not recording the files I'm looking for into the array!
Have tried processing my recursive directory reads 2 ways as follows:
Code:
function get_file_array () {
echo "1=> $1";
for f in $1/*; do
echo "F=> $f";
if [ -d "$f" ]; then
get_file_array $file;
else
fnam=$(basename "$f");
fext="${fnam##*.}";
fnam="${fnam%.*}";
# Skip files with extensions .dat .msf
if [ -z "$fext" ]; then
fil_ray+=("$fnam");
fi
fi # end if -d $f
done # end do
} # end function
This way the parms are not passing, so the routine starts processing from the "/" directory, not where it should.
Code:
function get_file_array () {
echo "1=> $1";
fil_ray=$( IFS=$'\n'; find "$1" -type f -follow | grep --exclude=\*.{dat,msf} );
dir_str=$( IFS=$'\n'; find "$1" -type f -follow );
IFS=' ' read -r -a fil_ray <<< "$dir_str";
for i in "${fil_ray[@]}"; do
echo "I=> $i";
fnam=$(basename "$i");
fext="${fnam##*.}";
fnam="${fnam%.*}";
# Skip files with extensions
if [ -z "$fext" ]; then
new_ray+=("$fnam");
fi # end if -z $fext
done # end do
} # end function
The "find" here does not save/assign as an array, but as a string and not able to parse it to an array for processing correctly!
When I change the 2nd to:
Code:
function get_file_array () {
echo "1=> $1";
#fil_ray=$( IFS=$'\n'; find "$1" -type f -follow | grep --exclude=\*.{dat,msf} );
#dir_str=$( IFS=$'\n'; find "$1" -type f -follow );
fil_ray=$( IFS=$'\n'; find "$1" -type f -follow );
#IFS=' ' read -r -a fil_ray <<< "$dir_str";
for i in "${fil_ray[@]}"; do
echo "I=> $i";
fnam=$(basename "$i");
fext="${fnam##*.}";
fnam="${fnam%.*}";
# Skip files with extensions
if [ -z "$fext" ]; then
new_ray+=("$fnam");
fi # end if -z $fext
done # end do
} # end function
I get all the files but as a string, not an array. My attempts to split this into an array failed, so not able to eliminate the files with extensions. Notice the $1 argument passes correctly here, but not in the 1st scenario.
If I could get these results to parse into an array, to apply the extension filtering all would be good.
find dirname -type f !-name '*.dat' ! -name '*.msf' --follow -exec sh -c '
for f; do # here is the array, ready for some loving
echo "$f"
# do more stuff
done' _ {} +
# Function to read directories recursively
function get_file_array () {
fil_ray=$( IFS=$'\n'; find "$1" -type f -follow '!' -regex '^.*\.\(\dat\|\msf\)$' );
echo "$fil_ray" >> $tbfiles; # Save "find" string to file
} # end function
# Function to read file string and parse to array
function parse_f2_array () {
echo "1=> $1";
#IFS=$( -regex '\r\n' ); read
IFS=$'\r\n' GLOBIGNORE='*' command eval 'fil_ray=($(cat $1))';
#echo "$fil_ray";
echo "FR=> $fil_ray";
} # end function
# Get current directory
DIR="$( cd -P "$( dirname "$0" )" && pwd )"
curdir="$DIR";
myuser='youruserID';
#echo "C=>$curdir";
# Declare the Local Folders dir for Thunderbird
ptdir="/home/$myuser/.thunderbird"; #TBird Parent directory
prfln=$( cat "$ptdir/profiles.ini" | grep "Path=" ); #Cet profile path name
profn="${prfln:5}"; #Get profile name
tbrot="$ptdir/$profn"; #Set the profile root dir
flray=("Mail/Local Folders" "ImapMail"); #Declare the processing dirs
g_opts="grep -o -E '[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,4}'";
s_opts="sort -f | uniq -i";
tbfiles="/home/$myuser/myfiles/TBird_Files.txt"; #Declare the output file
tbarray="/home/$myuser/myfiles/TB_rayout.txt"; #Declare the trap file
tbaddrs="/home/$myuser/myfiles/all-addresses.txt"; #Declare the output file
rm -f $tbfiles && touch $tbfiles; #Clear the output file
rm -f $tbarray && touch $tbarray; #Clear the trap file
for i in "${flray[@]}"; do #Process the dirs
tbdir="$tbrot/$i/"; #Set the dir path
get_file_array "$tbdir"; #Get the files to process
echo "TF=> $tbfiles";
parse_f2_array "$tbfiles"; #Parse the string into an array
for f in "${fil_ray[@]}"; do #Process each file
echo "FL=> $f";
# Run the grep statement to get the addresses
#grep '^\(From\|To\|Cc\|CC\Bcc\|BCC\):' "$f" | "$g_opts" | "$s_opts" >> $tbaddrs;
done # end for f loop
done # end for i loop
#nano $tbarray;
#nano $tbfiles;
exit;
I think I may have the parse_f2_array FUNCTION finally working right. Had some awesome help from 2 dudes and IRC /#bash. They suggested I can do all the process in the "find" command I'm using "-exec grep" with my grep and using xargs somehow, but not versed on that, so open to comments on that here to optimize this code.
Oops! found that I somehow messed up my address grep statements, but see I'm processing file by file, so a little more tweaking and all will be good!
OK fixed my "GREP" statements and now it's working.
In old version, would only get 3261 addresses. Been processing now for 15 min and already up to 10213 addresses, so see it is working correctly.
I get a lot of CraigList emails and other junk, so will have to figure out how to run these results into array and process a "filtering" function to eliminate those email, that I never want to reply to!
Hope you enjoy this code. Now you can gather all email addresses together for processing. I dump mine into PHPList for auto emailing.
First of all that is not what they have been told you, as i have been there.
Second they are the ones who write the stuff or refer to the stuff you consider to "make no sense" (wikiwooledge and shellcheck).
Best might be to post your result over there and ask about it.
In case what i post makes no sense to you. You will get pretty much similar notes, assuming someone is willing to look through that mess.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.