LinuxQuestions.org - Blogs - From A System Admin's Desk by mufy

Oracle on Linux PoC - Start of a Journey ...

mufy — Tue, 12 Jul 2011 08:38:14 GMT

Our environment is mostly filled with Windows servers and AIX boxes. Recently there have been some shifts in thoughts for moving towards Linux. As a guy who has had prior experience working on Linux, I was pulled into setup the environment for the proof of concept.

Actual work on the site began last week with storage allocation and switch side configuration for the 2 IBM eServer x366 rack mountable servers. Two days ago I was located physically in our DR site (where the test boxes are hosted). After a little bit of tug-of-war I installed RHEL 6.1 on one of the boxes.

Getting the GUI part was tricky (I've not yet done it) during the installation. So I opted to take the 'linux text' method. Successfully completed a bare metal installation.

Groping to find an answer as to why the GUI was having hiccups I found that the model I was given to work with was not on the Red Hat HCL for 6.1. So waiting for our infrastructure guys to provide different set of boxes to work on.

Meanwhile I thought of playing around with the installation on my workstation, an XP box. Installed VMware server. And as I eagerly managed to get the VM remote console up to get the installation going, BANG!, I was having a 64-bit image and my box was 32-bit.

So now... downloading the 30-day evaluation of RHEL 6.1 32-bit.

(To be continued ...)

A Journey With 'mksysb'

mufy — Sat, 22 Jan 2011 04:04:36 GMT

When the shit hits the fan, you take cover. Or, you cold stay right there, enjoy the spray and get new insights. Well, that's what I did - stood there and it was a learning experience.

It all began with an 'mksysb' backup failing on one of my development boxes. I'd start the backup, leave it for the day, and find the process stuck at one point when I come in the next day.

Code:

Creating information file (/image.data) for rootvg..

Creating list of files to back up.................
Backing up 238748 files..............................
9251 of 238748 files (3%)..............................
19979 of 238748 files (8%)..............................
49828 of 238748 files (20%)..............................
56914 of 238748 files (23%)..............................
82208 of 238748 files (34%)..............................
82208 of 238748 files (34%)..............................
82208 of 238748 files (34%)..............................

I was kind of blind. I didn't know whether the process was looping, progressing or had entered an infinite world. A couple of good people online were willing to share my trouble to see if it could be resolved. According to one, such a behavior was expected in the case of a large file.

The problem was I did not have anything tangile to look at as I did not know where to look. Then I learned that under /tmp directory the 'mksysb' command created a directory as 'mksys.'. In it there was a file which got populated with the names of files that were getting processed. This was the breakthrough I needed. Identifying the file that was causing the issue and to take the troubleshooting to another level was what I intended to do.

It turns out an application process was hanging onto this particular file preventing 'mksysb' from progressing. Bringing down the application resolved that.

Sounds simple right? Even that's what I felt at the end of it all. The details of the discussion may be found here.

Scripting/Programming As A System Administrator

mufy — Thu, 06 Jan 2011 11:47:24 GMT

The vast experience I gained into programming was during my school days in C++. There was nothing that I hated in it neither did I find any sort of pull towards programming.

The years that followed my post schooling I have found myself drifting away from coding, intentionally many a times. Starting my career as a System Administrator I found the need to carry out tasks with the help of bash scripts in Linux. It has been an on and off thing ever since.

Now I deal more with Korn scripts as my platform is AIX. Today, for some unknown reason, I found myself getting drawn to Perl and have made an effort to polish my skills.

And the first problem I hit, I've fired this post for help :-).

18 Days Of Troubleshooting

mufy — Mon, 03 Jan 2011 10:50:40 GMT

I never thought I'd be blogging on a technical website. But here I am, taking that new step towards a change this new year.

I'd not call what follows a purely technical account, but is definitely related to my work and my work is definitely technical.

The last 18 days I was actively pursuing a small problem that surfaced at work. Normally, it is something that people (including me) would brush over considering it not worth spending time on. After all, who would want to troubleshoot a small script that becomes unresponsive after giving the desired output? Moreover, there was no attached benefit in resolving the same either.

I started posting in forums and searching for pointers on Google. People were quite generous with their feed-backs, I must say, and I began to gain clarity into the problem. There were times when it didn't look all that promising, but I'd will myself to push-on. Definitely, it paid the price.

Coming to the problem (some were posted on LQ and some on LinkedIn), it was as simple as a light-weight in-house script not behaving the way it should. To compound the irritability, it was working perfectly in test environment.

So, the general inclination was to blame the difference in the environment. Easier said.

That being so it was more irritating as I knew we had a problem, but no visibility into it to trace. That's when suggestions in the forums started clearing the way. Using truss, lsof, etc. kind of zeroed in on the problem.

And in the end it was the difference in the sudo version that was the villain.

To summarize, learned a few things on the journey and that was definitely the up-side.