[SOLVED] Find URL in Debian package index via awk/sed (=find a line, then search from there)
Linux - SoftwareThis forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Introduction to Linux - A Hands on Guide
This guide was created as an overview of the Linux Operating System, geared toward new users as an exploration tour and getting started guide, with exercises at the end of each chapter.
For more advanced trainees it can be a desktop reference, and a collection of the base knowledge needed to proceed with system and network administration. This book contains many real life examples derived from the author's experience as a Linux system and network administrator, trainer and consultant. They hope these examples will help you to get a better understanding of the Linux system and that you feel encouraged to try out things on your own.
Click Here to receive this Complete Guide absolutely free.
Find URL in Debian package index via awk/sed (=find a line, then search from there)
Hello everybody,
I would like to create a small script that searches a huge file in a special way.
I assume that sed and awk may be able to do this, but have not the slightest clue how to start.
The task in general:
1. Search the text file for a line with the content "aaa".
2. From there find the next line beginning with "bbb:"
3. Get the value behind "bbb:"
4. Search the text file for a line with the content "ccc:" plus value from last step
5. From there find the next line beginning with "ddd:"
6. Get the value behing "ddd:"
Steps 1-3 and 4-6 are the same just with different values.
The task in my case:
I want to find the download URL of a virtual debian package.
As this package is for a different architecture I can not use apt, dpkg, etc. (can't I?)
First I download and extract the Packages index to get the text file I want to search in:
nano Packages
# CTRL+W, "Package: linux-image-orion5x"
# Copy value from "Depends:" line, e.g. "linux-image-2.6.26-2-orion5x"
# CTRL+W, "Package: " + value copied, e.g. "Package: linux-image-2.6.26-2-orion5x"
# Copy value from "Filename:" line, e.g. "pool/main/l/linux-2.6/linux-image-2.6.26-2-orion5x_2.6.26-21_armel.deb"
Can sed/awk do this? In an "understandable" form?
If so, could you also give some clues why you choosed your way and give some small explanations.
This would help me with this reoccurring task, and also help me and others to get into sed/awk.
it's difficult to give you advise on that without knowing what your file actually looks like. Why not search for bbb in the first place? And is the line with bbb following immidiately after the line with aaa or are there some lines in between? Please clarify.
Thanks for your reply.
I gave instructions in "The task in my case" section about how to download the file and extract it.
And also what I currently do by hand to find the value I need.
There are several hundreds of lines that start with "bbb:" (in my case "Depends:"), but I want to find the one following "aaa" (in my case "Package: linux-image-orion5x").
And there can be some other unimportant lines after "aaa" and before "bbb".
Relevant excerpt from the file:
Code:
Package: linux-image-ixp4xx
Priority: optional
Section: admin
Installed-Size: 32
Maintainer: Debian Kernel Team <debian-kernel@lists.debian.org>
Architecture: armel
Source: linux-latest-2.6 (17+lenny1)
Version: 2.6.26+17+lenny1
Provides: linux-latest-modules-2.6.26-2-ixp4xx
Depends: linux-image-2.6.26-2-ixp4xx
Filename: pool/main/l/linux-latest-2.6/linux-image-ixp4xx_2.6.26+17+lenny1_armel.deb
Size: 2514
MD5sum: 3bdd134c4704a3d18e3a31c5471a9436
SHA1: aa422f4ad5f992a8fc4777d0796a57c1484e70f6
SHA256: 5c5a90e03e1f35bb803f383d1cadef04ba8aacc72229b2b9af5f8e1756918745
Description: Linux image on IXP4xx
This package depends on the latest binary image for Linux kernel on IXP4xx
based (Linksys NSLU2, etc) machines.
Tag: admin::kernel, role::dummy
Package: linux-image-orion5x
Priority: optional
Section: admin
Installed-Size: 32
Maintainer: Debian Kernel Team <debian-kernel@lists.debian.org>
Architecture: armel
Source: linux-latest-2.6 (17+lenny1)
Version: 2.6.26+17+lenny1
Provides: linux-latest-modules-2.6.26-2-orion5x
Depends: linux-image-2.6.26-2-orion5x
Filename: pool/main/l/linux-latest-2.6/linux-image-orion5x_2.6.26+17+lenny1_armel.deb
Size: 2526
MD5sum: a6c9721c85fa20012d8f8416a15c8236
SHA1: 8bf5552d2c2a4ee6e6eced35c1b5a2cc9815e8bf
SHA256: cab463cf0950a17a5962dc0fdcb6b21a85953395e51b22cefb7fcc8116773788
Description: Linux image on Orion
This package depends on the latest binary image for Linux kernel on Orion
5181, 5182 and 5281 based (QNAP TS-109/TS-209, etc) machines.
Tag: admin::kernel, qa::low-popcon, role::dummy
Package: linux-image-versatile
Priority: optional
Section: admin
Installed-Size: 32
Maintainer: Debian Kernel Team <debian-kernel@lists.debian.org>
Architecture: armel
Source: linux-latest-2.6 (17+lenny1)
Version: 2.6.26+17+lenny1
Provides: linux-latest-modules-2.6.26-2-versatile
Depends: linux-image-2.6.26-2-versatile
Filename: pool/main/l/linux-latest-2.6/linux-image-versatile_2.6.26+17+lenny1_armel.deb
Size: 2504
MD5sum: 9e50b372cd67bedb0999491fa19ad64c
SHA1: 8868af24e081d77ea15c0e32ceb6170e94863199
SHA256: d376774482acfd8f765790da030b0087b35bdb760cb1c19298f1d6ca5d84f989
Description: Linux image on Versatile
This package depends on the latest binary image for Linux kernel on
Versatile (PB, AB, Qemu) machines.
Tag: admin::kernel, role::dummy
So the result of the first search (steps 1-3) should be "linux-image-2.6.26-2-orion5x".
#!/bin/sed -nf
/Package: linux-image-orion5x/ {
x
d
}
/Package: linux-image-orion5x/ !{
H
}
/Depends:/ {
H
x
/Package: linux-image-orion5x/ {
s/\(.*\)\(Depends: \)\(.*\)/\3/
p
d
}
}
You will have to type (or better yet paste) this script into a textfile and safe it. Afterwards make the file executable by
Code:
chmod 774 sed-script
Let me know if it works for you.
BTW, I use sed version 4.1.5 and bash version 3.2.39.
I tried it with the following sample data. I copied your data and pasted it twice into a textfile with slight modifications to also test against some unexpected cases. However, since your file suggests that it is in a standardized format I do not think that this was necessary; but one never knows what might be still to come in the future.
So here is the modified data:
Code:
Package: linux-image-ixp4xx
Priority: optional
Section: admin
Installed-Size: 32
Maintainer: Debian Kernel Team <debian-kernel@lists.debian.org>
Architecture: armel
Source: linux-latest-2.6 (17+lenny1)
Version: 2.6.26+17+lenny1
Provides: linux-latest-modules-2.6.26-2-ixp4xx
Depends: linux-image-2.6.26-2-ixp4xx
Filename: pool/main/l/linux-latest-2.6/linux-image-ixp4xx_2.6.26+17+lenny1_armel.deb
Size: 2514
MD5sum: 3bdd134c4704a3d18e3a31c5471a9436
SHA1: aa422f4ad5f992a8fc4777d0796a57c1484e70f6
SHA256: 5c5a90e03e1f35bb803f383d1cadef04ba8aacc72229b2b9af5f8e1756918745
Description: Linux image on IXP4xx
This package depends on the latest binary image for Linux kernel on IXP4xx
based (Linksys NSLU2, etc) machines.
Tag: admin::kernel, role::dummy
Package: linux-image-orion5x
Priority: optional
Section: admin
Installed-Size: 32
Maintainer: Debian Kernel Team <debian-kernel@lists.debian.org>
Architecture: armel
Source: linux-latest-2.6 (17+lenny1)
Version: 2.6.26+17+lenny1
Provides: linux-latest-modules-2.6.26-2-orion5x
Depends: linux-image-2.6.26-2-orion5x
Depends: THIS WILL BE IGNORED !!!
Filename: pool/main/l/linux-latest-2.6/linux-image-orion5x_2.6.26+17+lenny1_armel.deb
Size: 2526
MD5sum: a6c9721c85fa20012d8f8416a15c8236
SHA1: 8bf5552d2c2a4ee6e6eced35c1b5a2cc9815e8bf
SHA256: cab463cf0950a17a5962dc0fdcb6b21a85953395e51b22cefb7fcc8116773788
Description: Linux image on Orion
This package depends on the latest binary image for Linux kernel on Orion
5181, 5182 and 5281 based (QNAP TS-109/TS-209, etc) machines.
Tag: admin::kernel, qa::low-popcon, role::dummy
Package: linux-image-versatile
Priority: optional
Section: admin
Installed-Size: 32
Maintainer: Debian Kernel Team <debian-kernel@lists.debian.org>
Architecture: armel
Source: linux-latest-2.6 (17+lenny1)
Version: 2.6.26+17+lenny1
Provides: linux-latest-modules-2.6.26-2-versatile
Depends: linux-image-2.6.26-2-versatile
Filename: pool/main/l/linux-latest-2.6/linux-image-versatile_2.6.26+17+lenny1_armel.deb
Size: 2504
MD5sum: 9e50b372cd67bedb0999491fa19ad64c
SHA1: 8868af24e081d77ea15c0e32ceb6170e94863199
SHA256: d376774482acfd8f765790da030b0087b35bdb760cb1c19298f1d6ca5d84f989
Description: Linux image on Versatile
This package depends on the latest binary image for Linux kernel on
Versatile (PB, AB, Qemu) machines.
Tag: admin::kernel, role::dummy
Package: linux-image-ixp4xx
Priority: optional
Section: admin
Installed-Size: 32
Maintainer: Debian Kernel Team <debian-kernel@lists.debian.org>
Architecture: armel
Source: linux-latest-2.6 (17+lenny1)
Version: 2.6.26+17+lenny1
Provides: linux-latest-modules-2.6.26-2-ixp4xx
Depends: linux-image-2.6.26-2-ixp4xx
Filename: pool/main/l/linux-latest-2.6/linux-image-ixp4xx_2.6.26+17+lenny1_armel.deb
Size: 2514
MD5sum: 3bdd134c4704a3d18e3a31c5471a9436
SHA1: aa422f4ad5f992a8fc4777d0796a57c1484e70f6
SHA256: 5c5a90e03e1f35bb803f383d1cadef04ba8aacc72229b2b9af5f8e1756918745
Description: Linux image on IXP4xx
This package depends on the latest binary image for Linux kernel on IXP4xx
based (Linksys NSLU2, etc) machines.
Tag: admin::kernel, role::dummy
Package: linux-image-orion5x
Priority: optional
Section: admin
Installed-Size: 32
Maintainer: Debian Kernel Team <debian-kernel@lists.debian.org>
Architecture: armel
Source: linux-latest-2.6 (17+lenny1)
Version: 2.6.26+17+lenny1
Provides: linux-latest-modules-2.6.26-2-orion5x
Depends: linux-image-2.6.26-2-orion5x AND SOMETHING ELSE !!!
Filename: pool/main/l/linux-latest-2.6/linux-image-orion5x_2.6.26+17+lenny1_armel.deb
Size: 2526
MD5sum: a6c9721c85fa20012d8f8416a15c8236
Depends: THIS WILL BE IGNORED !!!
SHA1: 8bf5552d2c2a4ee6e6eced35c1b5a2cc9815e8bf
SHA256: cab463cf0950a17a5962dc0fdcb6b21a85953395e51b22cefb7fcc8116773788
Description: Linux image on Orion
This package depends on the latest binary image for Linux kernel on Orion
5181, 5182 and 5281 based (QNAP TS-109/TS-209, etc) machines.
Tag: admin::kernel, qa::low-popcon, role::dummy
Package: linux-image-versatile
Priority: optional
Section: admin
Installed-Size: 32
Maintainer: Debian Kernel Team <debian-kernel@lists.debian.org>
Architecture: armel
Source: linux-latest-2.6 (17+lenny1)
Version: 2.6.26+17+lenny1
Provides: linux-latest-modules-2.6.26-2-versatile
Depends: linux-image-2.6.26-2-versatile
Filename: pool/main/l/linux-latest-2.6/linux-image-versatile_2.6.26+17+lenny1_armel.deb
Size: 2504
MD5sum: 9e50b372cd67bedb0999491fa19ad64c
SHA1: 8868af24e081d77ea15c0e32ceb6170e94863199
SHA256: d376774482acfd8f765790da030b0087b35bdb760cb1c19298f1d6ca5d84f989
Description: Linux image on Versatile
This package depends on the latest binary image for Linux kernel on
Versatile (PB, AB, Qemu) machines.
Tag: admin::kernel, role::dummy
That sed script works on the original file too. Thanks crts.
Will try to understand it over the next few days and then report back.
If anybody thinks there is another solution to it, or has comments to the current solution feel free to add your thoughts.
Maddes
After I read your original post I realised that you do not want to print Depends. I had it adjusted. Not sure which version of the script you have but the current one will not print the preceding 'Depends: ' but just the value.
Have the latest script version and the result is fine.
One immediate question:
Can the strings "Package: linux-image-orion5x" and "Depends: " be replaced with variables, something similar as $1 $2 in shell scripts?
e.g. to call it like ./sed_script "Package: linux-image-orion5x" "Depends: " <filename>
Have the latest script version and the result is fine.
One immediate question:
Can the strings "Package: linux-image-orion5x" and "Depends: " be replaced with variables, something similar as $1 $2 in shell scripts?
e.g. to call it like ./sed_script "Package: linux-image-orion5x" "Depends: " <filename>
Maddes
You will have to make it a bash script then:
Code:
#!/bin/bash
sed -n '
'/"$1"/' {
x
d
}
'/"$1"/' !{
H
}
'/"$2"/' {
H
x
'/"$1"/' {
s/\(.*\)\('"$2"'\)\(.*\)/\3/
p
d
}
}' "$3"
If you call it the quote the parameters in 'single-quotes' like
Recently I got much more into sed and awk, so here's the same solution in awk.
Note that awk is more like a programming language than a stream editor like sed, therefore it also takes longer to process the file.
Code:
#!/bin/sh
awk --posix '
BEGIN {
process = 0
}
/Package:[[:space:]]*/ {
# before checking the new block: end the previous block if that was processed
if (process == 1) {
print "----------------------------------------"
process = 0
}
# if wanted block then process that block
if (/Package:[[:space:]]*'"${1}"'/) {
print $0
process = 1
}
}
# when block should be processed, look for wanted pattern and print those lines
process == 1 {
if (/'"${2}"'/) {
print $0
}
}
END {
if (process == 1) {
print "----------------------------------------"
}
}' ${3:+"$3"}
Additionally it turned out that an empty line marks the end of a package inside the list, therefore the block can be found and processed much easier with sed:
What you really appear to want is just to extract specific lines from individual entries so that you can process them with other commands. If you set awk's record and field separators properly, then this should be reasonably easy to do.
Now you can use the resulting variables as desired. Note that the fields variable is written as a regular expression, since that's the way awk treats them in tests. It also assumes there will be only a single match of the packagename.
If we can absolutely guarantee that all field entries are always present and in the same order then it can be even easier. You can just print out the field numbers that correspond to the lines you want.
Last edited by David the H.; 06-28-2013 at 07:38 AM.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.