Code:
sudo apt-get --purge remove conky conky-all
sudo apt-get --purge auto-remove
Maybe it created some configuration (dot) files in your users home directory and you want to remove those too.
Code:
find ~/ -iname '*infinity*' -o -iname '*conky*'
Then you can manually delete those files.