Hi Everyone,

I'm a newbie and I'm excited to join this forum to learn and share where I can.

My platform is Windows10 Enterprise.
I have 24 GB RAM.
System type: 64-bit Operating System, x64-based processor.
Processor is i7-9800X CPU @ 3.8GHz

I just started to use Cygwin32 at work. I am merging text files from various sources into a pipe delimited format file. I chose gawk since I've used it more than other languages and I need to get this project completed in a short time-frame. Just started the job 1 months ago and I need to prove that I was the right candidate for the position!!

I'm reading a 800+k line file and 2 smaller files using gawk.

gawk -v p1="file1.txt" -v p2="file2.txt" -v p3="file800lines.txt" -f awkscript.txt afile.txt > gawkout.txt


In the awkscript, I read file1 and file2 into arrays.
I then read the file800lines file. I use the getline function.
I'm able to process all the 800k lines and populate array values based on the associative values in files file1 and file2.
When I attempt to print out the 800lines using a for loop the script just hangs. I then put print statements inside the for loop and found out it hangs around the 250k line. There are no error messages. I just Ctrl C to get back to the prompt.

Thankfully, I had Cygwin64 installed along with Cygwin32.
I started the Cygwin64 window and reran the script and it worked!!
It may appear that Cygwin64 by default allocates a large portion of memory, hence, the reason why my script was able to complete (I think).

I checked the web and found this link:
https://cygwin.com/cygwin-ug-net/setup-maxmem.html

After reading it, this may be a solution to the problem when using Cygwin32, but I defer this to people smarter than me.

If this is the right path for a solution, can someone provide some direction using the peflags utility since I'm lost as to how to implement the command. There's also a link: 4-Gigabyte Tuning that goes to: https://docs.microsoft.com/en-us/win...ectedfrom=MSDN

If Cygwin64 is superior to Cywgin32 and handles memory allocation dynamically and I won't have any future issues if I use Cygwin64 reading 1M+ lines, then, I guess I don't have a problem!!

However, just to be safe, are the above links a way to increase memory when using Cygwin64 to eliminate any future memory issues?

Sorry for being so verbose, but I felt its better to provide enough information the first time without having to go back and forth with Q&As.

Thanks in advance guys!

Any reason why you can't just use

Yes Jeremy. the 800line file require manipulation of the fields to perform references to the data in file1 and file2 which have been loaded into associative arrays. I also perform certain logic within the gawk script to generate values based on certain business rules. Examples are dates that are in different formats and I have to change them to specific formats plus I have to concatenate fields to create new fields using business rules, for the output file.