According to the manual, you can press F10 to switch between windowed and fullscreen modes.
or add the --fullscreen parameter in the startup command. (edit the quickstart app menu link in gnome)
e.g. tuxmath --fullscreen
Hope this helps
(note: i currently don't have tuxmath at hand, but looked it up for you nevertheless)