It sounds like you're trying to process input character by character, right? But the problem is, standard input is line buffered, which means that your program doesn't receive any input from the device until the user has entered an entire line. You need to set standard input to be unbuffered, so that you get each character as it is typed. What you need to do is set the terminal device's CBREAK flag, and turning off echoing would be a good idea too.
But you really don't want to do that directly yourself, because you would create a lot of problems for your code to handle. For example, those arrow keys send escape sequences of several characters, and you would have to know how many to read in. Also, different terminals use different sequences for the arrow keys. Fortunately, all these problems have been solved. What you should be using is "ncurses" (do a "man ncurses" to get started). This is a terminal-independent, text-based display package. While it's main focus is on creating output on text terminals, it also handles all the details of input processing for you, which is what you're after. It provides functions such as cbreak() and noecho() to do the things you want. See curs_inopts(3X) for more details.