You can't do it with the tools you have.
You installed the Quota package - these are just program to manage quotas, but they need backup from the underlying operating system. Webmin correctly detected that there is none.
You need to ask your vps provider to enable it for you.
<- more about 2nd level quotas