Depends what you mean by non-plug-and-play, or for that matter, plug and play. "True" plug and play was simply a term used by hardware makers before the advent of PCI (which has PnP built in). Even so, plug and play in this sense isn't something you just want to yank out of a system. This is partly because most devices are powered by the system bus, and the resulting surge or loss in power from the device being removed or inserted can cause all sorts of problems. (Aside from the obvious shorting that can happen if you don't have it aligned correctly.)
USB/Firewire, and a couple other things (like server racks) are able to deal with being pulled while working. For many peripherals, yanking the cord won't kill them or the system, as there was some tolerence for this designed into them, but neither should you expect everything to work normally if you start playing with their cords. Some things, depending on the vendor, can be plugged and unplugged repeatedly with no effect, due to the fact that the system doesn't use them that often. An example would be parallel port printers (a dying breed). As long as you're not printing, you can dink around with the cords all day long. If you catch it in the middle of a job, you'll find it isn't so forgiving.
A good rule of thumb is: If it's inside the case, don't remove it while the powers on (this is a safety thing too, atx boards carry power even when the system is off, unless it's unplugged). If it's outside the box, it's safe to unplug, but there's no guarantee it'll work after you disconnect and reconnect it.
One point I should add. In older systems (p2 or slower, generally), not having the keybord plugged in is a bad thing. The keyboard connector on these older AT motherboards uses the keyboard as a way of dealing with power surges. In other words, if you don't have the keyboard plugged in and a power surge hits your system, game over. Not really a problem for most people that use UPS's, but it's still an interesting fact.