Okay, I'm making some progress. As it turns out, Enigmail wasn't using gpg-agent at all. Instead it was using the gnome keyring daemon. Once I killed that process Thunderbird went paranoid and asked me for passphrase for each email.
EDIT: Some relevant info on http://docs.xfce.org/xfce/xfce4-session/advanced
SSH and GPG Agents
By default xfce4-session tries to start the gpg- or ssh-agent. To disable this run the following command:
xfconf-query -c xfce4-session -p /startup/ssh-agent/enabled -n -t bool -s false
To force the ssh-agent instead of the gpg-agent use the following command:
xfconf-query -c xfce4-session -p /startup/ssh-agent/type -n -t string -s ssh-agent
In case you want to use gnome-keyring enable the Launch GNOME services on startup in the Advanced tab of the settings dialog. If you do this xfce4-session will not attempt to start the gpg- or ssh-agent.
EDIT 2: Unticking that "Launch GNOME services on startup" checkbox seems to have solved the problem.