Bash Problem

disno · 11-18-2013, 10:55 AM

i use this command:
curl http://xxx.com | lynx -dump -listonly -nonumbers -stdin

the output i got is urls like this:
http://qq.com
http://aa.com
http://zz.com

now i got two problems some of the URLS are duplicates i want to remove the duplicates and choose a random URL and store it in paramater "QA"
can someone help me to handle this problem?

Thanks For helpers

danielbmartin · 11-18-2013, 11:29 AM

With this InFile ...

Code:

http://qq.com
http://aa.com
http://zz.com
http://qq.com
http://aa.com
http://zz.com
http://qq.com
http://aa.com
http://zz.com

... this code ...

Code:

QA=$(sort $InFile |uniq |shuf |head -1)
echo "QA=" $QA

... produces a result of this form ...

Code:

QA= http://zz.com

Daniel B. Martin

PTrenholme · 11-18-2013, 01:10 PM

That's a workable solution, but it could be somewhat simplified: QA=$(sort -u ${InFile} | shuf -n 1) works as well.

Note that, if you wanted the likelihood of a URL being selected to be proportional the frequency of it's occurrence in the page, then the unique sort could be skipped and a QA=$(shuf -n 1 ${InFile}) used.

Also, if you wanted to do it all in one line, this

QA=$(curl http://xxx.com | lynx -dump -listonly -nonumbers -stdin | sort -u | shuf -n 1)

should do the trick.

disno · 11-18-2013, 02:24 PM

thanks guyz it works