first: Read the documentation!!
http://lartc.org/howto
yes, i know, it is toomuch... but it pays off
for make you a little short, i use HTB on the internet access, but to prioritize outbound traffic.
see
http://lartc.org/howto/lartc.cookbook.ultimate-tc.html
in my case, the bottleneck is in the upload part of the link, so the right thing to do if i had in your situation is to give max priority to the connections coming from tcp/22 and tcp/5900 on my lan/dmz.
if you read the entire howto, you will get a really more comprehensive point of view of your problem and you will be able to design the better solution to it
best regards