quick thought:
if you use itemheight, it will override fontsize.
remove itemheight, and try to adjust iconsize via fontsize.
also consider padding values.
more detailed:
fluxbox wiki and man pages are good and cover everything.
on this page:
http://fluxbox.org/help/ you should read every man page; fluxbox-style seems most relevant.(*)
if you want to override a value, you need to be very specific, e.g. window.*.border will NOT override window.focused.border.
(*) the layout of the site is broken. if you "view source" you get properly formatted code.
but it should be the same as the installed man pages, anyway.