I think that is impossible to do precisely in shellscript. Keys pressed during the sleep go into a buffer. Even if it were possible to discard the buffer it would take some time, however little, and more keystrokes could arrive during that time. Abandon hope all ye who enter the messy world of asynchronous and non-atomic processes!
You may be able to approximate what you want to do by using a timed read with the -n1 option in a loop. This will read a single "character" (in quotes because it also gets non-characters such as arrow keys). If the time is short you could (imprecisely) assume the character was entered during the sleep. The first character that takes a longer time should be the first character entered after the sleep.
Not perfect but it may be "good enough" for what you want to do.