LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software > Linux - Kernel
User Name
Password
Linux - Kernel This forum is for all discussion relating to the Linux kernel.

Notices


Reply
  Search this Thread
Old 01-18-2017, 03:59 PM   #1
TBotNik
Member
 
Registered: May 2016
Location: Greenville, TX
Distribution: Kubuntu 18.04
Posts: 796

Rep: Reputation: Disabled
Reading Thunderbird Email addresses


All,

I wrote a bash script a couple years ago to glean all email addresses from Thunderbird.

I published it at:

http://superuser.com/questions/35318...in-thunderbird

Lately I noticed that thunderbird changed it's directory structure, so now not reading all emails. TBird files containing the actual emails, that you scrub addresses from have no extension, so been trying to write a function that will recursively read/record all these email files for processing as I think my "grep" statements are working correctly.

However having issues. Here is my code. It is not reading any files at all to my knowledge:

Code:
# Function to read directories recursively
function get_file_array {
	for f in $1/*;	do
		if [ -d "$f" ]; then
			get_file_array $file;
		else
			# Skip files with extensions .dat .msf
			fnam=$(basename "$f");
			fext="${fnam##*.}";
			fnam="${fnam%.*}";
			if [ -z "$fext" ]; then
				fil_ray+=("$fnam");
			fi
		fi	# end if -d $f
	done	# end do
}			# end function
I'm calling it from this loop:
Code:
# Declare the Local Folders dir for Thunderbird
ptdir="/home/$myuser/.thunderbird";                    # Parent dir
prfln=$( cat "$ptdir/profiles.ini" | grep "Path=" );   # Read profile
profn="${prfln:5}";                                    # Substring profile name
tbrot="$ptdir/$profn";                                 # TBird root dir
flray=("Mail/Local Folders" "ImapMail");               # Default Mail Folders
for i in "${flray[@]}"; do                             # Loop for each Folder set
	tbdir="$tbrot/$i/";                            # Set processing Folder
	get_file_array "$tbdir";                       # Run file array process
	echo "FR=> $fil_ray";                          # Show results
done	# end for i loop
exit;
Could use some help figuring out why my function is not recording the files I'm looking for into the array!

Cheers!

TBNK

Last edited by TBotNik; 01-19-2017 at 10:58 AM.
 
Old 01-19-2017, 01:25 AM   #2
TBotNik
Member
 
Registered: May 2016
Location: Greenville, TX
Distribution: Kubuntu 18.04
Posts: 796

Original Poster
Rep: Reputation: Disabled
Neither Works

All,

Have tried processing my recursive directory reads 2 ways as follows:

Code:
function get_file_array () {
	echo "1=> $1";
	for f in $1/*;	do
		echo "F=> $f";
		if [ -d "$f" ]; then
			get_file_array $file;
		else
			fnam=$(basename "$f");
			fext="${fnam##*.}";
			fnam="${fnam%.*}";
			# Skip files with extensions .dat .msf
			if [ -z "$fext" ]; then
				fil_ray+=("$fnam");
			fi
		fi	# end if -d $f
	done	# end do
}			# end function
This way the parms are not passing, so the routine starts processing from the "/" directory, not where it should.

Code:
function get_file_array () {
	echo "1=> $1";
	fil_ray=$( IFS=$'\n'; find "$1" -type f -follow | grep --exclude=\*.{dat,msf} );
	dir_str=$( IFS=$'\n'; find "$1" -type f -follow );
	IFS=' ' read -r -a fil_ray <<< "$dir_str";
	for i in "${fil_ray[@]}"; do
		echo "I=> $i";
		fnam=$(basename "$i");
		fext="${fnam##*.}";
		fnam="${fnam%.*}";
		# Skip files with extensions
		if [ -z "$fext" ]; then
			new_ray+=("$fnam");
		fi	# end if -z $fext
	done	# end do
}			# end function
The "find" here does not save/assign as an array, but as a string and not able to parse it to an array for processing correctly!

When I change the 2nd to:

Code:
function get_file_array () {
	echo "1=> $1";
	#fil_ray=$( IFS=$'\n'; find "$1" -type f -follow | grep --exclude=\*.{dat,msf} );
	#dir_str=$( IFS=$'\n'; find "$1" -type f -follow );
	fil_ray=$( IFS=$'\n'; find "$1" -type f -follow );
	#IFS=' ' read -r -a fil_ray <<< "$dir_str";
	for i in "${fil_ray[@]}"; do
		echo "I=> $i";
		fnam=$(basename "$i");
		fext="${fnam##*.}";
		fnam="${fnam%.*}";
		# Skip files with extensions
		if [ -z "$fext" ]; then
			new_ray+=("$fnam");
		fi	# end if -z $fext
	done	# end do
}			# end function
I get all the files but as a string, not an array. My attempts to split this into an array failed, so not able to eliminate the files with extensions. Notice the $1 argument passes correctly here, but not in the 1st scenario.

If I could get these results to parse into an array, to apply the extension filtering all would be good.

All help appreciated!

Cheers!

TBNK

Last edited by TBotNik; 01-19-2017 at 11:01 AM.
 
Old 01-23-2017, 10:07 AM   #3
nodir
Member
 
Registered: May 2016
Posts: 222

Rep: Reputation: Disabled
from the top of my head
Code:
find dirname -type f !-name '*.dat' ! -name '*.msf' --follow -exec sh -c ' 
  for f; do # here is the array, ready for some loving
    echo "$f"
    # do more stuff
  done' _ {} +
http://mywiki.wooledge.org/UsingFind
 
Old 01-23-2017, 08:03 PM   #4
TBotNik
Member
 
Registered: May 2016
Location: Greenville, TX
Distribution: Kubuntu 18.04
Posts: 796

Original Poster
Rep: Reputation: Disabled
Parse Array ?Working?

All,

Progress in that code is now:

Code:
	# Function to read directories recursively
	function get_file_array () {
		fil_ray=$( IFS=$'\n'; find "$1" -type f -follow '!' -regex '^.*\.\(\dat\|\msf\)$' );
		echo "$fil_ray" >> $tbfiles;  # Save "find" string to file
	}			# end function

	# Function to read file string and parse to array
	function parse_f2_array () {
		echo "1=> $1";
		#IFS=$( -regex '\r\n' ); read 
		IFS=$'\r\n' GLOBIGNORE='*' command eval  'fil_ray=($(cat $1))';
		#echo "$fil_ray";
		echo "FR=> $fil_ray";
	}			# end function

	# Get current directory
	DIR="$( cd -P "$( dirname "$0" )" && pwd )"
	curdir="$DIR";
	myuser='youruserID';
	#echo "C=>$curdir";

	# Declare the Local Folders dir for Thunderbird
	ptdir="/home/$myuser/.thunderbird";                  #TBird Parent directory
	prfln=$( cat "$ptdir/profiles.ini" | grep "Path=" ); #Cet profile path name
	profn="${prfln:5}";											  #Get profile name
	tbrot="$ptdir/$profn";                               #Set the profile root dir
	flray=("Mail/Local Folders" "ImapMail");				  #Declare the processing dirs
	g_opts="grep -o -E '[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,4}'";
	s_opts="sort -f | uniq -i";
	tbfiles="/home/$myuser/myfiles/TBird_Files.txt";     #Declare the output file
	tbarray="/home/$myuser/myfiles/TB_rayout.txt";       #Declare the trap file
	tbaddrs="/home/$myuser/myfiles/all-addresses.txt";   #Declare the output file
	rm -f $tbfiles && touch $tbfiles;                    #Clear the output file
	rm -f $tbarray && touch $tbarray;                    #Clear the trap file
	for i in "${flray[@]}"; do                           #Process the dirs
		tbdir="$tbrot/$i/";                               #Set the dir path
		get_file_array "$tbdir";                          #Get the files to process
		echo "TF=> $tbfiles";
		parse_f2_array "$tbfiles";                        #Parse the string into an array
		for f in "${fil_ray[@]}"; do							  #Process each file
			echo "FL=> $f";
			# Run the grep statement to get the addresses
			#grep '^\(From\|To\|Cc\|CC\Bcc\|BCC\):' "$f" | "$g_opts" | "$s_opts" >> $tbaddrs;
		done	# end for f loop
	done		# end for i loop
	#nano $tbarray;
	#nano $tbfiles;
	exit;
I think I may have the parse_f2_array FUNCTION finally working right. Had some awesome help from 2 dudes and IRC /#bash. They suggested I can do all the process in the "find" command I'm using "-exec grep" with my grep and using xargs somehow, but not versed on that, so open to comments on that here to optimize this code.

Oops! found that I somehow messed up my address grep statements, but see I'm processing file by file, so a little more tweaking and all will be good!

Cheers!

TBNK
 
Old 01-23-2017, 08:54 PM   #5
TBotNik
Member
 
Registered: May 2016
Location: Greenville, TX
Distribution: Kubuntu 18.04
Posts: 796

Original Poster
Rep: Reputation: Disabled
Working

All,

OK fixed my "GREP" statements and now it's working.

In old version, would only get 3261 addresses. Been processing now for 15 min and already up to 10213 addresses, so see it is working correctly.

I get a lot of CraigList emails and other junk, so will have to figure out how to run these results into array and process a "filtering" function to eliminate those email, that I never want to reply to!

Hope you enjoy this code. Now you can gather all email addresses together for processing. I dump mine into PHPList for auto emailing.

Cheers!

TBNK

Last edited by TBotNik; 01-23-2017 at 08:58 PM.
 
Old 01-23-2017, 09:37 PM   #6
nodir
Member
 
Registered: May 2016
Posts: 222

Rep: Reputation: Disabled
http://mywiki.wooledge.org/BashFAQ/050
and
http://www.shellcheck.net/

and, like i tried to say above already:
http://mywiki.wooledge.org/BashPitfa...8ls_.2A.mp3.29
Quote:
files=($(find . -type f)) # Wrong!
(also note the difference used there files=($( ... (assuming you want an array, and it sounds like it).
and
https://superuser.com/questions/3230...ess-use-of-cat

Last edited by nodir; 01-23-2017 at 09:52 PM.
 
Old 01-24-2017, 10:15 AM   #7
TBotNik
Member
 
Registered: May 2016
Location: Greenville, TX
Distribution: Kubuntu 18.04
Posts: 796

Original Poster
Rep: Reputation: Disabled
This maxed out at over 28,000 addresses, but still have to perform a sort -unique to make sure there are no dups.

Cheers!

TBNK
 
Old 01-24-2017, 10:17 AM   #8
TBotNik
Member
 
Registered: May 2016
Location: Greenville, TX
Distribution: Kubuntu 18.04
Posts: 796

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by nodir View Post
http://mywiki.wooledge.org/BashFAQ/050
and
http://www.shellcheck.net/

and, like i tried to say above already:
http://mywiki.wooledge.org/BashPitfa...8ls_.2A.mp3.29

(also note the difference used there files=($( ... (assuming you want an array, and it sounds like it).
and
https://superuser.com/questions/3230...ess-use-of-cat
nodir,

The stuff in your links make no sense and are contrary to what the gurus on IRC /#bash told me.

Cheers!

TBNK
 
Old 01-24-2017, 09:58 PM   #9
nodir
Member
 
Registered: May 2016
Posts: 222

Rep: Reputation: Disabled
First of all that is not what they have been told you, as i have been there.
Second they are the ones who write the stuff or refer to the stuff you consider to "make no sense" (wikiwooledge and shellcheck).

Best might be to post your result over there and ask about it.
In case what i post makes no sense to you. You will get pretty much similar notes, assuming someone is willing to look through that mess.

Last edited by nodir; 01-24-2017 at 10:00 PM.
 
Old 01-25-2017, 10:25 AM   #10
TBotNik
Member
 
Registered: May 2016
Location: Greenville, TX
Distribution: Kubuntu 18.04
Posts: 796

Original Poster
Rep: Reputation: Disabled
Posted in Wrong Thread

All;

Sorry all that script got posted in the wrong thread!

Cheers!

TBNK

Last edited by TBotNik; 01-26-2017 at 05:32 PM. Reason: Posted in Wrong Thread
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Got this email from Google after trying to log into Thunderbird. Is Thunderbird in-secure? Ajwad Linux - Newbie 2 02-19-2016 11:59 AM
Ubuntu 12.04/Thunderbird - Migrate Thunderbird Email & Settings Ineed2know Linux - Newbie 3 05-04-2012 10:59 PM
[SOLVED] Best email client for moderate to large email database (Evolution, thunderbird, kmail, Claws mail) Carpincho Linux - Software 1 08-24-2011 05:19 AM
Thunderbird email & addresses remotely or CLI jantman Linux - Software 5 10-16-2006 10:53 PM
Reading Ethernet IP addresses linux_lover2005 Linux - Networking 1 02-10-2005 04:41 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software > Linux - Kernel

All times are GMT -5. The time now is 09:21 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration