LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 02-04-2010, 09:52 AM   #1
qwertyjjj
Senior Member
 
Registered: Jul 2009
Location: UK
Distribution: Cent OS5 with Plesk
Posts: 1,012

Rep: Reputation: 30
squid performance slowing for html webpages


Is there a way to cache everything in squid except for the video files and mp3s?
I had a problem before when I turned caching on and it made the performance very poor because it was trying to cache large video files.
Right now, I have a problem on the server where even if iptraf lists ingoing+outgoing at 5000 kbits/sec viewing video works fine, but it takes a while for normal webpages to load and I can't figure out where the bottleneck is.
 
Old 02-05-2010, 03:59 AM   #2
cardy
Member
 
Registered: Jan 2005
Location: Shropshire, England, UK
Distribution: RedHat, Fedora, CentOS..........
Posts: 121

Rep: Reputation: 19
Yes there are 2 configuration entries that may help you.

In /etc/squid/squid.conf

you should find the 2 entries listed below. The first is the maximum size of an object (html page, video file, image etc....) that will be stored in the cache's memory. If this is set too large then you will be trying to store lots of large files in memory. Reducing this size may help you avoid trying to cache video in memory. The size listed is the default on Fedora 11.

The second entry below is the same setting but the maximum size of a file that will be cached, this is usually quite big meaning that videos will be cached but only on disk.

You will need to see what these are set to on your system, but tuning them may help.

Quote:
# TAG: maximum_object_size_in_memory (bytes)
# Objects greater than this size will not be attempted to kept in
# the memory cache. This should be set high enough to keep objects
# accessed frequently in memory to improve performance whilst low
# enough to keep larger objects from hoarding cache_mem.
#
#Default:
maximum_object_size_in_memory 2048 KB

Quote:
# TAG: maximum_object_size (bytes)
# Objects larger than this size will NOT be saved on disk. The
# value is specified in kilobytes, and the default is 4MB. If
# you wish to get a high BYTES hit ratio, you should probably
# increase this (one 32 MB object hit counts for 3200 10KB
# hits). If you wish to increase speed more than your want to
# save bandwidth you should leave this low.
#
# NOTE: if using the LFUDA replacement policy you should increase
# this value to maximize the byte hit rate improvement of LFUDA!
# See replacement_policy below for a discussion of this policy.
#
#Default:
maximum_object_size 40960 KB

Lee
 
Old 02-05-2010, 07:39 AM   #3
qwertyjjj
Senior Member
 
Registered: Jul 2009
Location: UK
Distribution: Cent OS5 with Plesk
Posts: 1,012

Original Poster
Rep: Reputation: 30
Quote:
Originally Posted by cardy View Post
Yes there are 2 configuration entries that may help you.

In /etc/squid/squid.conf

you should find the 2 entries listed below. The first is the maximum size of an object (html page, video file, image etc....) that will be stored in the cache's memory. If this is set too large then you will be trying to store lots of large files in memory. Reducing this size may help you avoid trying to cache video in memory. The size listed is the default on Fedora 11.

The second entry below is the same setting but the maximum size of a file that will be cached, this is usually quite big meaning that videos will be cached but only on disk.

You will need to see what these are set to on your system, but tuning them may help.







Lee
The problem is that it is streamed video so I am not sure how squid deals with that especially with caching since the whole file is not downloaded first.
 
Old 02-05-2010, 09:18 AM   #4
cardy
Member
 
Registered: Jan 2005
Location: Shropshire, England, UK
Distribution: RedHat, Fedora, CentOS..........
Posts: 121

Rep: Reputation: 19
my appologies, I assume you have looked at the squid stats in the cachemgr to try and idenfity what the squid cache is seeinng as the bottle neck if anything. Also I assume there are no issues with disk io etc.... on the server which is hosting the squid proxy.

Cache manager is distributed with squid and is able to give you a large number of stats about what squid is doing. If you have squid installed from an RPM it should include the cachemgr.cgi script. This will need to be placed on a web server in the appropriate location (/var/www/cgi-bin for RH/Fedora systems). By default on RH systems there is an entry in the cache acl list for the manager which allows access from the localhost.


Quote:
acl manager proto cache_object
acl localhost src 127.0.0.1/32
Defines the manager entry and the default local host entry.

Somewhere in your http_access rules you should have:

Quote:
http_access allow manager localhost
Which allows the cgi script to connect to the squid server from the local host to access the cache manager objects.

I have attached 2 screen shots of the cache manager as it looks when its configured, maybe that will help you pinpoint the source of your problem, I have used it for performance tuning squid caches before.

There are extra settings in the squid.conf for setting access control rights for the cachemgr, the above entries only use IP based control access.

Lee
Attached Thumbnails
Click image for larger version

Name:	Screenshot-Cache Manager Interface - Mozilla Firefox.png
Views:	15
Size:	38.0 KB
ID:	2663   Click image for larger version

Name:	Screenshot-CacheMgr@localhost: menu - Mozilla Firefox.png
Views:	16
Size:	98.9 KB
ID:	2664  

Last edited by cardy; 02-05-2010 at 09:20 AM. Reason: Updated regarding availability of access controls for cache mgr
 
Old 02-07-2010, 09:49 AM   #5
qwertyjjj
Senior Member
 
Registered: Jul 2009
Location: UK
Distribution: Cent OS5 with Plesk
Posts: 1,012

Original Poster
Rep: Reputation: 30
I access the server through SSH so can I still use the localhost for this?
I believe I would need to access it via the web interface available externally to do this?
Presumably the cache needs to be turned on for the cachemgr to work?
My problem was last time I had the cache turned on the performance was slow because it must have been trying to cache streamed video files. If I could turn that off immediately then it would be ok but not sure if it is an option and don't want to take the squid server offline for too long...

Last edited by qwertyjjj; 02-07-2010 at 11:07 AM.
 
Old 02-07-2010, 04:34 PM   #6
qwertyjjj
Senior Member
 
Registered: Jul 2009
Location: UK
Distribution: Cent OS5 with Plesk
Posts: 1,012

Original Poster
Rep: Reputation: 30
have followed the tutorial here: http://wiki.squid-cache.org/SquidFaq/CacheManager
and set up acls to access the cache manager cgi on my server. I have to access this externally for the moment as that is the only access to the server that I have (SSH or web).
The cache manager login appears when I access: http://myexternalipaddress/cgi-bin/cachemgr.cgi
I have set the cache manager login and password in the squid.conf
# TAG: cache_mgr
# Email-address of local cache manager who will receive
# mail if the cache dies. The default is "root".
#
#Default:
# cache_mgr root
cache_mgr aaa@aaa.com
cachemgr_passwd aaa all

#Recommended minimum configuration:

acl all src 0.0.0.0/0.0.0.0

acl manager proto cache_object

acl localhost src 127.0.0.1/255.255.255.255

acl cacheadmin src 88.xxx.xxx.xx9/255.255.255.255 #external IP address?

acl to_localhost dst 127.0.0.0/8

# Only allow cachemgr access from localhost
http_access allow ncsa_users
http_access allow manager localhost
http_access allow manager cacheadmin
http_access deny manager

However, whenever I enter the password and select localhost port 8080 from the cgi script I get:

The following error was encountered:


Cache Access Denied.




Sorry, you are not currently allowed to request:
cache_object://localhost/
from this cache until you have authenticated yourself.
 
Old 02-11-2010, 10:16 AM   #7
cardy
Member
 
Registered: Jan 2005
Location: Shropshire, England, UK
Distribution: RedHat, Fedora, CentOS..........
Posts: 121

Rep: Reputation: 19
You need to change the following

Quote:
# Only allow cachemgr access from localhost
http_access allow ncsa_users
http_access allow manager localhost
http_access allow manager cacheadmin
http_access deny manager
to

Quote:
# Only allow cachemgr access from localhost
http_access allow manager localhost cacheadmin
http_access deny manager
http_access allow ncsa_users
This will allow access from cacheadmin and localhost for management queries i.e. cachemgr.cgi and then will fall through to allow ncsa_users if its not a management request.

You have set the username and password options in squid.conf but I would try with the above config first with the squid.conf settings commented and then once you have basic IP authentication working then build more security if you want/require it from there with the username and password options.

Regards

lee

Last edited by cardy; 02-11-2010 at 10:20 AM.
 
Old 02-11-2010, 10:23 AM   #8
qwertyjjj
Senior Member
 
Registered: Jul 2009
Location: UK
Distribution: Cent OS5 with Plesk
Posts: 1,012

Original Poster
Rep: Reputation: 30
Done that and it works with password - thanks.
Can anything be controlled from the cache manager or is it simply a reporting tool?
Are there any guides to understand what the cache manager info means?
 
Old 02-11-2010, 10:43 AM   #9
cardy
Member
 
Registered: Jan 2005
Location: Shropshire, England, UK
Distribution: RedHat, Fedora, CentOS..........
Posts: 121

Rep: Reputation: 19
On my distribution there is a large amount of info in the FAQ documentation included with squid.

/usr/share/doc/squid-3.0.STABLE20

The version/dir name may be slightly different on your system. Section 9 of the FAQ discusses the cachemgr including the meanings of the Metadata entries and a large number of other parameters mean. These can then be used to tune the performance of your cache.

You will find the same (possibly more up-to-date version) of the documentation here http://wiki.squid-cache.org/SquidFaq/CacheManager
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
squid: howto deny cache of all dynamic asp webpages? cccc Linux - Server 2 02-14-2009 07:16 AM
Slow squid performance the_gripmaster Linux - Server 6 08-08-2008 11:02 AM
Converting HTML Webpages to PDF or JPEG with konqueror mrkirkland Linux - Software 2 01-31-2006 10:16 AM
Bandwidth problems slowing down the server performance ravilohot Linux - Networking 2 10-15-2005 11:52 PM
Squid Performance cli_man Linux - General 2 06-27-2003 05:48 PM


All times are GMT -5. The time now is 05:23 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration