How to do secure remote management of off-the-shelf network devices (router, wireless AP, PC)?
Currently I can manage the network devices locally (at the office) by a web browser on Windows or linux PC. How to do that remotely through the internet from my home?
Because I cannot modify the off-the-shelf network devices, to provide a secure connection across the internet, I need ssh connection, so I can put a ssh server at the office PC and a ssh client at my home PC.
How to make the web browser (HTTP client) at my home PC to talk to the HTTP server embedded inside off-the-shelf network devices with a secure connection?
What further setup is required?

Presumably you have restricted the web access on the router to the local network. Therefore you either need a VPN connection to that network in which case your remote PC then appears inside the network and you can then simply manage it in your remote browser using the normal router IP address. Alternatively you could use VNC over ssh (I presume it can be done but haven't done it myself) to remote control one of the PC's at the office, in which case you are connecting locally with that PC.

I use a VPN and have also used VNC over the VPN to do all sorts of stuff remotely.