That is a thing that the individual needs to decide :}
If, for instance, an older kernel has a vulnerability
for local attacks but you're the only one ever to have
physical access to the box you might not think that
the upgrade is necessary ... if you don't have any
hardware that's not supported by the old kernel
stick with it ...