I have solved this problem now. It wasn't a firewall issue, the webmin server and client are on the same local network and I wasn't trying to access it from outside.
Turns out the answer is in the documentation:
Basically the default configuration on debian is set to only allow access from listed IP addresses, and the only listed IP address is 127.0.0.1, ie the webmin server itself.
To edit the configuration, I first installed the lynx web-browser, since I'm not running a desktop on the webmin server, and used that to access webmin:
apt-get install lynx
From there I was able to login and add my local network (192.168.10.0) to the allowed list via Webmin Configuration -> IP Access Control. I did get a bit bored of the "unknown security certificate" dialogue popping up on every page, so next time I would choose "Always Accept" the first time it asks.
FInally, restart webmin with
and I can now login to webmin from my PC!