Is there any way to tell Apache2 to prompt for username and password
when it sees a .htaccess file in a directory?

At the moment, I need to define the directory in the following
fashion in Apache2 conf file but it becomes a problem as I have thousands of directories that needs .htaccess files.

<Directory "/srv/www/htdocs/dir">
AllowOverride All
Options Indexes
Order allow,deny
Allow from all
AuthUserFile /srv/www/htdocs/dir/.htaccess
</Directory>

Thanks.

I'm a little confused about the way you're going about things here. Is it that you want to protect a whole directory structure with a username and password, or do you want to protect each of thousands of directories with different usernames and passwords? Protecting one directory would protect all directories below it (e.g. protecting private/ will also protect private/quotes/ and private/orders/).
A few issues here:

• It's missing an AuthType [Basic|Digest] directive
• It's missing an AuthName directive
• It's missing a Require directive
• Using .htaccess files (which is enabled with AllowOverride All) for most directories is only a good idea if you do not have access to the main config files (it can sap performance of the server).
• .htaccess files are not normally the AuthUserFile - and the AuthUserFile should not be in a publically accessible directory. The file created using htpasswd - should be stored above the document root.

The section should read something like:
Using AuthType Digest provides further security, by hashing the username and password before transmission (though a few more changes would be needed as well). The Require statement can instead list a number of users from the AuthUserFile, rather than valid-user (= all users). See here for more.

If you do decide you want to go with .htaccess files, you would need to include the essential directives shown above (AuthType, AuthName, AuthUserFile and Require) within <Limit... > ... </Limit> tags, within that file (at the root of each branch you want to protect).

Hope this helps,
Rob

I want to protect each of thousands of directories with a different usernames and passwords. Assuming /srv/www is DirectoryRoot then protect /srv/www/a1, /srv/www/z1 etc.

I am aware of the fact that protecting one directory would protect all directories below it.

The issues that you have raised about a few directives missing are all described in the .htaccess file (and it works fine). Following is an example. .htaccess is not an issue; it works fine.

Wouldn't you then just need to modify the AllowOverride directive for your document root, to something like: