You can kill your program in many ways.
First, you should be looking into binding "xkill" to a key combo. Look at the fluxbox docs to know how to do that, I don't really remember. You could bind xkill to control+alt+esc if you want to have the same combo that you said above you had in kde.
Second, control+alt+F<n> is used to change between different vt's. Usually, most distros come configured to show 6 of them, from F1 to F6. X usually takes the next free vt, so you just need to keep pressing from F1 to F12 until you find in which one X is. By taking a look into /etc/inittab you can find how many VT's are initialized at boot time, if you are even interested in that.
Third, you could do as someone said above, however if the opengl app is really in full screen, or even if it's windowed but the fluxbos policy puts it over the rest of windows, then you are going to have a hard time to get the focus into the alt+f2 dialog, if it's possible at all. I am not too familiar with the flux focus policy though.
Fourth and last, the first number in the ps output is the "process identification number", PID for short, to kill a process you just need to use either kill with the pid or pkill with its name. If the name doesn't work, use the pid, it's unique and should work most times.