squid performance slowing for html webpages
 

Is there a way to cache everything in squid except for the video files and mp3s?
I had a problem before when I turned caching on and it made the performance very poor because it was trying to cache large video files.
Right now, I have a problem on the server where even if iptraf lists ingoing+outgoing at 5000 kbits/sec viewing video works fine, but it takes a while for normal webpages to load and I can't figure out where the bottleneck is.

Yes there are 2 configuration entries that may help you.

In /etc/squid/squid.conf

you should find the 2 entries listed below. The first is the maximum size of an object (html page, video file, image etc....) that will be stored in the cache's memory. If this is set too large then you will be trying to store lots of large files in memory. Reducing this size may help you avoid trying to cache video in memory. The size listed is the default on Fedora 11.

The second entry below is the same setting but the maximum size of a file that will be cached, this is usually quite big meaning that videos will be cached but only on disk.

You will need to see what these are set to on your system, but tuning them may help.



Lee

The problem is that it is streamed video so I am not sure how squid deals with that especially with caching since the whole file is not downloaded first.

my appologies, I assume you have looked at the squid stats in the cachemgr to try and idenfity what the squid cache is seeinng as the bottle neck if anything. Also I assume there are no issues with disk io etc.... on the server which is hosting the squid proxy.

Cache manager is distributed with squid and is able to give you a large number of stats about what squid is doing. If you have squid installed from an RPM it should include the cachemgr.cgi script. This will need to be placed on a web server in the appropriate location (/var/www/cgi-bin for RH/Fedora systems). By default on RH systems there is an entry in the cache acl list for the manager which allows access from the localhost.


Defines the manager entry and the default local host entry.

Somewhere in your http_access rules you should have:

Which allows the cgi script to connect to the squid server from the local host to access the cache manager objects.

I have attached 2 screen shots of the cache manager as it looks when its configured, maybe that will help you pinpoint the source of your problem, I have used it for performance tuning squid caches before.

There are extra settings in the squid.conf for setting access control rights for the cachemgr, the above entries only use IP based control access.

Lee

I access the server through SSH so can I still use the localhost for this?
I believe I would need to access it via the web interface available externally to do this?
Presumably the cache needs to be turned on for the cachemgr to work?
My problem was last time I had the cache turned on the performance was slow because it must have been trying to cache streamed video files. If I could turn that off immediately then it would be ok but not sure if it is an option and don't want to take the squid server offline for too long...

have followed the tutorial here: http://wiki.squid-cache.org/SquidFaq/CacheManager
and set up acls to access the cache manager cgi on my server. I have to access this externally for the moment as that is the only access to the server that I have (SSH or web).
The cache manager login appears when I access: http://myexternalipaddress/cgi-bin/cachemgr.cgi
I have set the cache manager login and password in the squid.conf
# TAG: cache_mgr
# Email-address of local cache manager who will receive
# mail if the cache dies. The default is "root".
#
#Default:
# cache_mgr root
cache_mgr aaa@aaa.com
cachemgr_passwd aaa all

#Recommended minimum configuration:

acl all src 0.0.0.0/0.0.0.0

acl manager proto cache_object

acl localhost src 127.0.0.1/255.255.255.255

acl cacheadmin src 88.xxx.xxx.xx9/255.255.255.255 #external IP address?

acl to_localhost dst 127.0.0.0/8

# Only allow cachemgr access from localhost
http_access allow ncsa_users
http_access allow manager localhost
http_access allow manager cacheadmin
http_access deny manager

However, whenever I enter the password and select localhost port 8080 from the cgi script I get:

The following error was encountered:


Cache Access Denied.




Sorry, you are not currently allowed to request:
cache_object://localhost/
from this cache until you have authenticated yourself.

You need to change the following

to

This will allow access from cacheadmin and localhost for management queries i.e. cachemgr.cgi and then will fall through to allow ncsa_users if its not a management request.

You have set the username and password options in squid.conf but I would try with the above config first with the squid.conf settings commented and then once you have basic IP authentication working then build more security if you want/require it from there with the username and password options.

Regards

lee

Done that and it works with password - thanks.
Can anything be controlled from the cache manager or is it simply a reporting tool?
Are there any guides to understand what the cache manager info means?

On my distribution there is a large amount of info in the FAQ documentation included with squid.

/usr/share/doc/squid-3.0.STABLE20

The version/dir name may be slightly different on your system. Section 9 of the FAQ discusses the cachemgr including the meanings of the Metadata entries and a large number of other parameters mean. These can then be used to tune the performance of your cache.

You will find the same (possibly more up-to-date version) of the documentation here http://wiki.squid-cache.org/SquidFaq/CacheManager