Here's a quick summary of your options:
applies a regex pattern to each line, and if it matches, it prints out the whole line by default. However it can optionally print out just the part that matches, or the lines that don't match, or that line plus the ones just before or after it. It's main limitation is that it can't match a string, then print out only part of the matching section (i.e. you can't search for "abc=xyz", and print only "xyz").
is a more flexible editing tool. It applies regex patterns to each line of input, and can print, delete, insert, alter, or extract text from them. It can do the substring extraction that grep cannot. While it does have some multi-line editing capability as well, it's not well designed for it.
is not just a program, but a full scripting language, and comes in several variations based on the interpreter used. Linux generally uses gawk
. awk divides the input up into records, and then subdivides those records into fields. These are user defined, but by default are one line per record, and one word per field. You can then manipulate these fields with an impressive collection of functions.
The text processing capabilities of bash
and other shells is string-based and fairly powerful. If you can break the text down into clearly defined, reasonably-sized chunks (a single line from a file, for example), then you can store them in variables and operate on them with various internal and external tools. Bash is one of the more advanced shells, and has full regex ability built into its [[..]]
extended test pattern, in addition to other string manipulation
Now, since you used quote tags above, which don't preserve formatting, rather than [code][/code]
tags, which do, I'm not yet sure about the exact layout of the text you posted. Does each line contain a single, unbroken url, or can they contain more than one, or can the urls can span multiple lines? If the latter, then it's going to require some additional work to extract them.
The basic procedure will probably be very simple. Just extract the urls using one of the above tools, store them in variables, and run your curl commands on them.
If the file contains unbroken urls, then a simple bash loop iterating through the lines would do the job. But if there can be multi-line urls, or if the file is very large, then it would probably be better to use sed or awk to extract them first, then run the loop on the results.