You can insert a newline by typing newline where you want one.
You can make a small gap with a backtick "`" and a larger gap with "~".
Some tips for using the formula editor more effectively:
1) Learn the "language" for typing formulae - don't rely on the buttons. For example, PI is inserted with %PI or %pi for upper or lower case pi.
2) Using oomath, create some templates for commonly used structures, such as a 3x3 matrix, 4x4 matrix, etc. Then you can simply insert one, using "Insert - OLE - From File" in Writer, and then edit it to add the data you want. You can insert the blank boxes in a formula using <?>.
3) There is a list of all the commands in the help section.
I hope this helps