Keeping track of a maximized window’s previous state is the job of a window manager. As such, you will need to interface with your window manager to do this. The most “portable” way is to use the
EWMH spec. In particular, you will need to send a
_NET_WM_STATE event to the root window, with the window id of the desired window in the event. You will want to specify the
_NET_WM_STATE_REMOVE action on the
_NET_WM_STATE_MAXIMIZED_VERT and
_NET_WM_STATE_MAXIMIZED_HORZ properties.
Alternatively, you might use a window-manager-specific message (e.g., through dbus or DCOP). While this will probably take up less code, it will also be non-portable.