Maybe because the first bytes of those files are reserved for other uses. On the other hand, the magic number specification provides the ability to specify an offset and the offset can be retrieved from the file, as clearly stated in man magic
Offsets do not need to be constant, but can also be read from the file being examined. If the first character following the last > is a ( then the string after the parenthesis is interpreted as an indirect offset.
Just take in mind that magic numbers in many cases are not there to be... magic numbers. They are unique sequences of bytes that help programs in *nix systems (along with the file
command) to identify the file type, but they could have been inserted originally for other purposes.