i want to search among millions of files for ones that have a specific string. if any file has that string in an interesting way, it will be within the first 512 bytes of the file. so what i would like to find is a grep program that can be told (by option, environment variable, configuration file, whatever) to give up after 512 bytes (or some higher number if choices are limited).

i have already considered the idea of copying a limited size of the file if is larger than 512 byes. i've ruled this out for performance issues (the reason i'm looking for this, in the first place).

the grep command does have a limit feature with option -n but this counts number of matches, not number of non-matches.

so, i am looking for a better for of grep with this ability.

If I am understanding all of this correctly, this will work:

As for perfomance? I have no idea. There is probably a more 'one-liner-y' way to do this. If you need that just reach back out.

binwalk might work for your needs. It will search for a raw string with a specified sequence and you can limit the number of bytes to scan.

1 members found this post helpful.

binwalk is a cool program. Nice suggestion.

FWIW: when i installed it on my debian system just now, it had to install 491MB of other packages to make it work!

I've never used the program but it is designed to scan binary files looking for specific signatures so that makes sense there is a bunch of other dependencies.

At this scale? It's time feed them into ElasticSearch.

What's wrong with dd ?. Use it to read a single sector.
No way to stop this all polluting the system tho'.

One off or regular/intermittent requirement ?.

anyway, if there are really millions of files to scan better to forget shell and grep (one by one).
Theoretically you can use dd to read only 512 bytes, but that will be slow again (because you need to fork dd for every file).
Much faster solution would be to use a language which can read files directly and check the content without any external/other tool.
Like python or perl.
But simply grep -m 1 -r <pattern> <dir> might work for you.

Also: solid drive or platter drive? If it’s a platter drive then the seeks are going to be a huge bottleneck.

right. and it is a few spinning platters for now. i'm hoping to get this to be solid next year.

nothing really wrong with it, but it's kind of like head. it involves piping the data between 2 processes, which appears to be the best solution short of hacking grep.

maybe with a larger -m.

Is this your use case?

https://www.linuxquestions.org/quest...9/#post6422884

no.

the comments thing involves generating a string that is to be added by others to source to be managed. when this key added as a comment is used, the file is being handled for managing. actually they can be inserted anywhere. but it would likely be easier to add the key in the form of a comment at the front or back of the file. for some languages like Python, a form which is a large string literal that is discarded or otherwise does not affect the program could be used to hold the key.

the grep thing is to look for some files in my personal archive of all kinds of files of which maybe about 5% is source code. i just happen to know that what i am looking for is only in short files (easy to filter) in some cases or at the beginning of larger files (not at the end) in some other cases. for this i usually need to manually check files to see if they are what i need to find, but i can only remember poorly what strings are involved. i recently needed to find someone's name i could not remember but i could remember their street. but i this case the files could have lots of data appended. i could have spent a few hours on it and worked out a way to find it. but i did know the street this person lived (same as my own back then). the long scan gave me a list of about 50 files, which was a small enough list to check manually.

anyway, you can use that grep command alone, or you can implement a more suitable tool for yourself. Just remember forking a new process (or more) for every and each file will extremely slow down this search, so better to avoid that. You ought to use a language like c, perl, python for that, which can recognize file types, can limit the search for the beginning of files and you can also implement any kind of filters. bash is not really suitable for this. (and probably awk is usable, but I would rather try something else).
(from my side I don't know what's wrong with that grep, it will list you all the files where the pattern was found much faster than any other solution posted here, anyway just tell us if you found something better).