I was recently offered an interview with a company managing about 3000 servers running a mix of CentOS, Redhat and Solaris (most probably running on VMWare). There would be a total of 6 sysadmins for managing all this.

I have no idea how difficult it would be to manage so many servers. I know the description of the job is quite vague but would be glad if anyone could share his/her experience of working in such a large environment.

Thanks.

Hi,
My question to the interviewer would be the Admin hierarchy and responsibilities. How management distributes the management work loads will dictate who does what.

What I gathered from the job description is pretty much all 6 persons would be doing the same. One person on-call a week and next on-call week is after 5 more weeks, so one person on call every week. It is basically making sure all services on all servers are running smoothly. I am quite sure they have something like Nagios setup for monitoring. It is actually a white page/yellow page and other associated web based service company.

Have you looked at the numbers, and thought about it??? SIX people for 3000 servers?? Unless all you're doing is VERY simple stuff (NOT admin work), there is NO WAY six people could do that job with any degree of professionalism. You'd never keep up, even with basic things. Simple math says that even if you had one thing that took only a minute per server, that's 50 hours of work right there.

Get job details...and ask why they're looking for a new person. Could be they ALWAYS are, since people quit quickly...

6 good linux admins should be able to manage 3000 machines easily.. with tea time and smoke breaks.

I worked on a team of two supporting 8000 for 3 years.

umm.. you shouldnt need to logon manually to every server. Thats what scripts are for.

Even better, config management tools like puppet; "ssh in a loop is not the solution"

You should simply ask questions about how they go about managing their server farm. What tools and techniques do they use (e.g. Nagios?) for observing the status of the machines? What techniques do they use (e.g. Subversion?) to manage software configuration? In their experience, how many issues arise each week and what sort of issues are they?

It is quite straightforward to manage a farm of "thousands of servers, all alike," as long as the hardware and software infrastructure is properly set up and maintained ... and as long as the team is properly managed. In other words, if the entire setup is such that "success is actually achievable," success is actually likely. But... if the setup is such that people are likely to be thrown under the bus ... that, too, is likely.

Ask probing questions. You need to assess the true nature of the situation before you insert yourself into it. It just might be fine.

Upon further thought, it does depend on what level of support, and the environment, as sundialcvs pointed out. If it's nothing but a managed-hosting environment, where most things are already set up and done, and the customer is responsible for any tweaks, then yes, it's a doable number.

The environments I've worked in are banks and large corps. NOTHING is cookie cuttered out, and things are a bit more involved.