You are thinking of the window manager called Windowmaker. See
When you minimize a window it becomes an icon on the desktop.
You might also be thinking about the Enlightenment window manager. It does something similar, but you have to see it to understand it. Google them both and see if maybe they are installed on your system, or if you can get them.