LinuxQuestions.org - Blogs - sag47

sshd chroot jails based on hostname and SELinux

sag47 — Mon, 01 Apr 2013 15:46:42 GMT

Why chroot jails based on host name?
If you don't care about the background info just jump to Setting up sshd chroot jail and SELinux.

So I have started configuring automated deployments through my organization using a continuous integration server called Jenkins. I'll post more on Jenkins regularly so that eventually you can have a series of posts which wrap up into a nice and secure howto for Jenkins. So here's why I want chroot jails based on host names.

I have a Jenkins master node running on RHEL 6 and several child nodes mixed with Windows and RHEL Linux for building a variety of cross platform applications. Different groups in my org use this tool so it's great to be so versatile. Jenkins is secured from the public a number of ways such as using LDAP, SSL, and firewall rules for IP blocks. However, Jenkins implementation of security gives you a false sense of security because anyone who actually has access to creating Jobs in Jenkins has the ability to do whatever they want, unhindered by the web interface. I'll explain.

On the master node, Jenkins is running as the jboss user which does not have root user access. The master node is what primarily runs the web interface.
On the Windows nodes (aka build agent), Jenkins is running as a domain account, call it domain-user-jenkins. In Windows specifically, domain-user-jenkins is required to have local Administrator rights in order to do the service bits it requires (according to the documentation). Eventually I'll investigate that further if it *really* needs admin rights but for now we'll take it at face value.

Why is this information important?
Because in Jenkins jobs you can run any action through a batch script, a shell script, a powershell script, a groovy script, even any executable if you set it up with svn. That means a user creating a Job in Jenkins is capable of doing anything the local build agent privileges has the ability to do. On the Windows nodes, where domain-user-jenkins has local administrator rights, that could be anything a local Administrator can do; which is anything. On the Master Node, where the user is restricted by the rights of the local jboss user it is possible for the user to modify web server configurations or even Jenkins core configurations and open up the security access.

How can we improve this model?

First and foremost, since the security model of Jenkins is very bad to begin with and gives you a false sense of security I recommend: be very careful who you give access to this system. While they may not be able to access many advanced features from the web interface a "normal" user who has "create jobs" permission on the master node can easily create a job that accesses and modifies any advanced feature of Jenkins on the command line using a simple bash script. So you put a lot of trust in users who get the ability to create build jobs. It's one of those security vs automation issues.

Jenkins is used for more than just building and compiling. It's used for deploying applications to both production and test servers. Therefore, it is important to think about how you approach deployments. This is important in case the Jenkins server itself is compromised and unparalleled access is given to an evil-doer across production if incorrectly configured.

Enough about Jenkins; the main focus of this is sshd chroot jails and SELinux.

Setting up sshd chroot jail and SELinux
Here's what I came up with. You should set up all of your deployments with a reversed access security model. You should give access to the Jenkins server as sftp-internal only with SELinux enabled inside of a ssh chroot jail. The user should only be able to read from the chroot jail and not write. This way Jenkins doesn't have access to servers at all and all of the servers have access to Jenkins in the scope of a read-only chroot jail.

Create ssh chroot
Create a chroot environment for ssh. Remember, the user on the system that will be locked in the chroot jail is called jboss.

Code:

#create a template jail which will be used by default if no system is specified
for x in /chroot/jail/home/jboss /chroot/jail/usr/libexec/openssh /chroot/jail/lib64;do
  mkdir -p $x
done
chmod -R 755 /chroot
chown jboss\: /chroot/jail/home/jboss
#use ldd to determine which libraries belong to
ldd /usr/libexec/openssh/sftp-server
#note the following libraries are used by sftp-server via ldd
for x in /usr/lib64/libcrypto.so.10 /lib64/libutil.so.1 /lib64/libz.so.1 /lib64/libnsl.so.1 /lib64/libcrypt.so.1 /lib64/libresolv.so.2 /lib64/libgssapi_krb5.so.2 /lib64/libkrb5.so.3 /lib64/libk5crypto.so.3 /lib64/libcom_err.so.2 /usr/lib64/libnss3.so /lib64/libc.so.6 /lib64/libdl.so.2 /lib64/libfreebl3.so /lib64/libkrb5support.so.0 /lib64/libkeyutils.so.1 /lib64/libpthread.so.0 /lib64/ld-linux-x86-64.so.2 /usr/lib64/libnssutil3.so /lib64/libplc4.so /lib64/libplds4.so /lib64/libnspr4.so /lib64/libselinux.so.1;do
  cp $x /chroot/jail/lib64/
done
cp /usr/libexec/openssh/sftp-server /chroot/jail/usr/libexec/openssh/

Set up some basic selinux rules for the chroot using the proper context.

Code:

#change selinux context
chcon -R -t chroot_user_t /chroot
#Do not allow ssh users to have rw access to chroot directory
setsebool -P ssh_chroot_rw_homedirs off
#disable selinux temporarily to test out the chroot; we'll handle that in a bit
setenforce 0

Modify /etc/ssh/sshd_config at the end so that the jboss user is chrooted by the jail automatically. I have a separate Match rule for the ChrootDirectory directive because once it is set it doesn't change. It must be done this way to allow different hosts to have different chroot directories.

Code:

Match User jboss
	PasswordAuthentication no
	AllowAgentForwarding no
	AllowTcpForwarding no
	GSSAPIAuthentication no
	PubkeyAuthentication yes
	PermitEmptyPasswords no
	RSAAuthentication no
	X11Forwarding no
	X11UseLocalHost no
#catchall
Match User jboss
	ChrootDirectory /chroot/jail
	ForceCommand internal-sftp

Restart sshd and test logging in as the jboss user using a public key from another system. I found the context of chroot_user_t by running the following command with SELinux set to permissive.

Code:

ps auZx | grep internal-sftp

SELinux and ssh chroot
Build a module so that local unconfined users have rw access to chroot directory. By default SELinux does not allow any user, not even root, to write or modify a chroot_user_t context type directory. This is why we have to build a module.

First you must create a policy violation alert with SELinux enforcing. Remember, we're trying to provoke a violation with all the different ways we might use the chroot jail for our deployments. That way we can write a module to allow that violation (so it will not longer be a violation).

Code:

mkdir /tmp/selinux_play
cd /tmp/selinux_play


#Set up your chroot jail for some violation tests
cd /chroot/jail/home/jboss
setenforce 0
touch env2.sh  env3.sh
mkdir -p ./dir/dir3
echo "test5" > env2.sh
echo "test6" > env3.sh
\rm env.sh
chown jboss\: *
setenforce 1

#now run tests with SELinux enforcing
#Log into Jenkins and try to run a job which attempts to write a file to the chroot directory.
#Now do some local commands on it
cd /chroot/jail/home/jboss
\echo "test" > env2.sh 
\touch env5.sh
\mv -f env3.sh env5.sh
\mv -f env3.sh /tmp/
\rm -f env3.sh 
ls
\mv -f dir dir2
\rmdir -f dir
\mkdir dir2
\rm -rf dir
\chmod 600 env2.sh
\chmod -f 600 env2.sh
\touch env3.sh

#do the same thing as the jboss user
su - jboss
cd /chroot/jail/home/jboss
\echo "test" > env2.sh 
\touch env5.sh
\mv -f env3.sh env5.sh
\mv -f env3.sh /tmp/
\rm -f env3.sh 
ls
\mv -f dir dir2
\rmdir -f dir
\mkdir dir2
\rm -rf dir
\chmod 600 env2.sh
\chmod -f 600 env2.sh
\touch env3.sh

Now let's create a module.

Code:

cd /tmp/selinux_play
audit2allow -M unconfined_user_rw_ssh_chroot -l -i /var/log/audit/audit.log

Here's the module that I came up with (along with a few manual edits).

source: unconfined_user_rw_ssh_chroot.te

Code:

#SELinux module by Sam Gleske
module unconfined_user_rw_ssh_chroot 1.1;

require {
	type unconfined_t;
	type fs_t;
	type locate_t;
	type chroot_user_t;
	type inetd_child_t;
	class dir { write rmdir rename getattr setattr remove_name create add_name };
	class file { rename create unlink setattr getattr };
	class filesystem associate;
}
#============= inetd_child_t ==============
allow inetd_child_t chroot_user_t:file setattr;

#============= locate_t ==============
allow locate_t chroot_user_t:dir getattr;

#============= chroot_user_t ==============
allow chroot_user_t fs_t:filesystem associate;

#============= unconfined_t ==============
allow unconfined_t chroot_user_t:dir { write rmdir setattr remove_name create add_name rename getattr };
allow unconfined_t chroot_user_t:file { create unlink setattr rename getattr };

If you need to edit and recompile the module here's how you can do it.

Code:

\rm -f unconfined_user_rw_ssh_chroot.mod unconfined_user_rw_ssh_chroot.pp
checkmodule -M -m -o unconfined_user_rw_ssh_chroot.mod unconfined_user_rw_ssh_chroot.te
semodule_package -o unconfined_user_rw_ssh_chroot.pp -m unconfined_user_rw_ssh_chroot.mod

Now install the module.

Code:

semodule -i unconfined_user_rw_ssh_chroot.pp

If you need to change and recompile the module you should remove it first.

Code:

semodule -l | grep chroot
semodule -r unconfined_user_rw_ssh_chroot
#recompile your changes and install again

Add a host to the chroot jail
Now that we've got the chroot jail and SELinux enabled and properly working. We want to start segregating hosts and chroot jails. That means your Jenkins job which is deploying should write any environment files or deployment files to the jboss home directory of the chroot jail for that host. Primarily because the jboss user coming from that host will only have access to the chroot jail with the sftp utility.

Let's set up the jail for myhost.com using our jail template.

Code:

cd /chroot
cp -r jail myhost.com
chcon -R -t chroot_user_t myhost.com
chown jboss\: myhost.com/home/jboss

Now edit /etc/ssh/sshd_config and add the host to use the new jail.

Code:

Match User jboss
	PasswordAuthentication no
	AllowAgentForwarding no
	AllowTcpForwarding no
	GSSAPIAuthentication no
	PubkeyAuthentication yes
	PermitEmptyPasswords no
	RSAAuthentication no
	X11Forwarding no
	X11UseLocalHost no
Match Host myhost.com
	ChrootDirectory /chroot/myhost.com
	ForceCommand internal-sftp
#catchall
Match User jboss
	ChrootDirectory /chroot/jail
	ForceCommand internal-sftp

The reverse call

Here's the last bit which I almost forgot to mention. I know you were probably wondering, "what is a reversed access security model?" Here's how I would define it since I pulled that phrase out of thin air:

Quote:

From the master node (Jenkins) set up your ssh keys. Copy your master node public key to the authorized_keys file of myhost.com. Modify the authorized keys file so that it executes a command rather than dropping into a shell when connected. That command will be a script which will sftp to the master node to gather all of the files from Jenkins.

So essentially the master node will only be able to execute a single command on the "deploy to" server and nothing else. And the "deploy to" server will only have access to the chroot directory on the master node. Here's a sample authorized_keys file.

Code:

command="/opt/deploy_something.sh" ssh-rsa KEY GIBERISH

Conclusion
Now whenever we ssh to the master node from the server myhost.com we get chrooted into a chroot directory. If we create more chroot directories for other hosts then we'll be able to keep a separate chroot jail for each host. This way hosts can't interfere with deployments to other hosts.

It helps to mitigate the amount of access a continuous integration server has to all of the production machines and visa versa. Hopefully I was concise enough for good understanding. For more information on the Match directive in sshd_config see the sshd_config man page.

Feel free to ask me any questions in the comments.

man page keyboard shortcuts

sag47 — Sun, 24 Feb 2013 14:30:38 GMT

Here's a short list of keyboard shortcuts for reading man pages that I've learned by usage. I've not found a good place shortcuts are documented; man page or otherwise.

Code:

man bash

Use SHIFT+G to scroll to bottom of the page and look at FILES. You'll also learn more about how bash loads the bashrc if you read the INVOCATION section of the bash man page.

Helpful shortcuts for man pages...

k and j go up and down one line.
Ctrl+u and Ctrl+d go up and down half a page
/ search forward for text
? search backward for text
n find next occurrence of searched word
N find previous occurrence of searched word
g go to top of document
G go to bottom of document
q quit man page

To learn more about the manual pages execute "man man".

Headless file conversion using LibreOffice as a service

sag47 — Wed, 13 Feb 2013 04:16:16 GMT

The content outlined here also works for OpenOffice. This post was inspired by a question I was answering on the forums.

System Setup
I installed LibreOffice packages using a PPA in Kubuntu. Here is information about my environment.

Code:

Tue Feb 12 23:00:54 EST 2013
Ubuntu 12.04.2 LTS
Linux 3.2.0-37-generic x86_64 GNU/Linux
GNU bash, version 4.2.24(1)-release (x86_64-pc-linux-gnu)
LibreOffice 3.5 (Version: 1:3.5.4-0ubuntu1.1)
unoconv 0.4
Python 2.7.3
gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3

You must have libreoffice and unoconv installed to proceed. soffice is in my $PATH. Depending one which office version you're using you may need soffice.bin instead.

Converting your first document

Let's start with the goal of converting a single document. Let's say we want to convert the following html document into a PDF (test.html).

Code:

This is a test

This is my title

This is some text in the page

Visit Gleske Homepage

For this step you'll need to make use of either the program screen or simply open two terminals. Open the first terminal and start the LibreOffice service.

Code:

soffice --nologo --headless --nofirststartwizard --accept='socket,host=127.0.0.1,port=2220,tcpNoDelay=1;urp'

Now convert test.html into test.pdf. With unoconv you can connect to the LibreOffice API through the network socket and use the existing service to do the heavy lifting.

Code:

unoconv --connection 'socket,host=127.0.0.1,port=2220,tcpNoDelay=1;urp;StarOffice.ComponentContext' -f pdf test.html

If all goes well you should now see test.pdf in the same directory as test.html. You can use name globbing to specify multiple files like *.html.

To see a list of formats unoconv supports with the -f option run the command.

Code:

unoconv --show
#grep for certain formats
unoconv --show 2>&1 1>/dev/null | grep pdf

Daemonize the LibreOffice service
You could start the LibreOffice daemon (as the current user) with the following command.

Code:

soffice --nologo --headless --nofirststartwizard --accept='socket,host=127.0.0.1,port=2220,tcpNoDelay=1;urp;StarOffice.Service'

It will now run until a shutdown command for the service has been issued. To issue a shutdown command do the following.

Code:

soffice --nologo --headless --nofirststartwizard --unaccept='socket,host=127.0.0.1,port=2220,tcpNoDelay=1;urp;StarOffice.Service'

That's it! You could also alternatively start the service as another user like so (in this case the user is "sam")...

Code:

sudo su - sam -c "soffice --nologo --headless --nofirststartwizard --accept='socket,host=127.0.0.1,port=2220,tcpNoDelay=1;urp;StarOffice.Service'"

Pretty cool LibreOffice tricks I learned today. If you want to know more then look up the LibreOffice/OpenOffice API. That is technically what is being used. Also, you could google terms like "StarOffice.Service" or "StarOffice.ComponentContext".

SAM

Using a script to get your IP address

sag47 — Mon, 07 Jan 2013 17:33:15 GMT

A friend of mine was complaining today how their work place doesn't parse the IP address from ifconfig but asks for manual input from the user. This is the dumbest thing ever. In case you're ever encountered by this here's a horrible one liner to get the job done quick and dirty.

Also, you may need to replace eth2 with your correct interface (such as eth1).

Code:

ifconfig | sed -ne '/^eth2/,/^\s\+Interrupt/p' | awk '$0 ~ /inet addr/ { print $2 }' | cut -d: -f2

It works, move along.

Better filtered mail notifications

sag47 — Wed, 31 Oct 2012 14:18:42 GMT

Note: this post is not about mail filtering but about getting better notified on mail that is already filtered. I will assume you have an understanding of bash scripting and how grep works. This includes modifying paths within scripts for your own environment. This is one of those hack together one job scripts.

Background

I am a Linux Systems Administrator at Drexel University and we're primarily a RedHat shop. My work day is high in volume with communication on supporting systems. I receive anywhere from 0-100 text messages a day on my work phone (yes I keep my work phone completely separate). I receive anywhere between a few hundred to a few thousand emails every day. The text messages and some emails come from automated monitoring systems such as Icinga. The mail comes from a number of sources such as:

cron from each individual system
RT tickets for customer (student/faculty/staff) facing ticketing
An in-house built hourly log summary email of all systems which mails out using perl. The logs are agregated to a central system running syslog-ng and sent by the systems using either syslog or rsyslog (see /etc/syslog.conf or /etc/rsyslog.conf man pages for your own system).

So I need a little something extra to assist me in picking out the information that is most important to me. There's a specific set of systems I support which I am most concerned and I filter my email to account for that. E.G. I have a "tickets important" mail folder which gets funneled using keywords for specific applications or tickets I want immediate attention. All other tickets go into a generic tickets folder for reviewing when I exhaust my "tickets important" folder to help out other admins. I do the same thing with Icinga alerts (with an "Icinga" and "Icinga Critical" folder). I check my email often but sometimes not often enough. Sure the "important" and "Critical" mail folders helped but I needed a better alerting system to warn me when I am deeply buried in a terminal and forget to check my email. I use KDE and with it KMail. KMail has notifications for new mail but it's the same notification for all new mail with no ability to specify the type of new mail. Luckily the KDE notifications can output to a log file in addition to its "New Mail" sound. This is where using the Linux kernel inotify functionality is useful.

Introducing inotify and incrond

inotify is a Linux kernel subsystem that acts to extend filesystems to notice changes to the filesystem (i.e. monitoring inodes). This is useful because we can monitor changes of the log file generated by KDE notifications and then do a little something extra.

inotify is actually an API accessed via system calls using C. So it's not a program. This is where incrond comes in handy because the author, Lukas Jelinek, wrote a cron-like daemon which can execute shell scripts when inotify events occur. Look in your vendors package system to install incrond. In KUbuntu 12.04 the package is incron.

Code:

sudo apt-get install incron

Setting up your notification

System Settings > Common Appearance and Behavior > Application and System Notifications.

Once you're there choose the drop down box next to event source and select KMail. Once you have KMail selected choose the "New Mail Arrived" alert and add the option to "Log to a file". If you notice in the screen shot I'm choosing to log to a file at /home/sam/mail-notify.log.

screenshot

Now that I have my KMail notifications logging to a file I now want to use inotify and watch that file for changes. First let's write a script which will parse the KDE notifications log file and then chime a specific sound different from other sounds.

Code:

#!/bin/bash
#Created by Sam Gleske
#Mon Oct 22 16:17 EST 2012
#Environment:
#  Linux 3.2.0-32-generic #51-Ubuntu SMP x86_64 GNU/Linux
#  GNU bash, version 4.2.24(1)-release (x86_64-pc-linux-gnu)
#  play: SoX v14.3.2
#  incrond 0.5.9

count="$(wc -L /home/sam/mail-notify.log | awk '{print $1}')"

if [ "$count" -eq "0" ];then
  exit 0
fi

sleep 2

incomming="$(grep 'Drexel IMAP' /home/sam/mail-notify.log | grep -i 'important\|Critical')"

#using a sound file which came with KDE
if [ ! -z "$incomming" ];then
  play /usr/share/sounds/freedesktop/stereo/complete.oga > /dev/null 2>&1
fi

#invoked logging
#date >> /home/sam/.local/invoked.log

#clear old logs and refresh incrond
echo '' > /home/sam/mail-notify.log
incrontab -d

Now run the command "incrontab -e" and edit your incrontab to be the following.

Code:

/home/sam/mail-notify.log IN_CLOSE_WRITE /home/sam/.local/mail-important-notify.sh

You should start to hear notifications upon arrival of new important mail. For more information see the incrond man pages.

Code:

man incrond
man 1 incrontab
man 5 incrontab

Extending an LVM volume on a VMWare host

sag47 — Thu, 23 Aug 2012 17:17:11 GMT

Back story
This is going to be a short blog post. Mostly with commands. Basically I run a media server for online courses/lectures called Echo360 ESS. At some point the LVM volume which holds the lecture captures nearly filled up. I was getting Icinga alerts on it and by looking at the rate of data generation in munin showed me I had better hop to it. So that's what this post is about.

The fun stuff

Add disk to VM

I added a disk to the virtual machine and then rescanned disks so that it would be apparent in the VM.

Code:

#Add the disk to VMWare disk in VM settings
#then rescan the disks so it shows up
echo "- - -" > /sys/class/scsi_host/host0/scan
#check out what the new device is called
dmesg
#in my case the new device is /dev/sdf

Extending an LVM volume

Code:

#look at the devices in the volume
lvdisplay
#look at each device
pvdisplay /dev/sdb
pvdisplay /dev/sdd
#deactivate the logical volumes in the volume group
vgchange -an vg_echo360
#initialize the disk for use by LVM
pvcreate /dev/sdf
#get the VG Name and extend it
vgdisplay
vgextend vg_echo360 /dev/sdd
#get the LV name
lvdisplay
#now that we have the LV name let's extend it with the new LV in our LV group
lvextend /dev/vg_echo360/lv_echo360 /dev/sdf
#check that the volume has been successfully extended
vgdisplay -v
lvdisplay
#activate the logical volumes in the volume group vg_echo360
vgchange -ay vg_echo360
#check the filesystem before resizing
e2fsck -f /dev/mapper/vg_echo360-lv_echo360
#proceed to resize the ext2/3 filesystem to the new volume size
resize2fs /dev/mapper/vg_echo360-lv_echo360

Taking LVM snapshots for recovering changes

Take a snapshot of the disk before removing raw files older than 120 days. Here is the find command which will be used.

Code:

find . -mtime +120 -type f -name '*.h264' -o -mtime +120 -type f -name '*.aac' -exec rm {} \;
dd if=/dev/zero of=/app/loop bs=1 count=1 seek=10737418240
losetup -f /app/loop
pvcreate /dev/loop0
vgextend vg_echo360 /dev/loop0
#create the snapshot
lvcreate -s -n echo_snapshot1 /dev/vg_echo360/lv_echo360 -L 9.9G

Recover data from the snapshot

Code:

mkdir /mnt/snapshot_recovery
mount /dev/vg_echo360/echo_snapshot1 /mnt/snapshot_recovery
#Now the path /mnt/snapshot_recovery contains the filesystem at a state from when the snapshot was taken.

Removing the snapshot

Code:

lvremove /dev/vg_echo360/echo_snapshot1
#Do not allow the LVG to allocate data to /dev/loop0 any longer.  This sets the PV Status to NOT allocatable.
pvchange -xn /dev/loop0
#Check out the status of /dev/loop0 compared to the other devices.
vgdisplay -v
#Remove the loopback device from the LVG entirely.
vgreduce vg_echo360 /dev/loop0
#See that the device is missing in the following command?
vgdisplay -v
#Now that the device is removed from the LVG you can remove the loopback device from the system entirely.
losetup -a
#/dev/loop0 is the device I want to remove which is using /app/loop.
losetup -d /dev/loop0
rm /app/loop

#If you accidentally removed the loopback device before removing it from the LVM that's okay.  Just be sure to run the following command on the LVG.
vgreduce --removemissing vg_echo360

That's about it. Use the man pages to exercise a better understanding!

SAM

SELinux and Icinga

sag47 — Sun, 05 Aug 2012 05:39:28 GMT

Today I ran into an SELinux problem when installing icinga. I followed their documentation "Adjusting the SELinux settings". However my cgis still would not run. When I ran,

Code:

cat /var/log/audit/audit.log | audit2allow -v

I received the following output,

Code:

#============= httpd_sys_script_t ==============
# src="httpd_sys_script_t" tgt="initrc_tmp_t" class="file", perms="{ read getattr open }"
# comm="tac.cgi" exe="" path=""
allow httpd_sys_script_t initrc_tmp_t:file { read getattr open };

I had to create a policy module and build it using the following commands...

Code:

cd /usr/local/src
mkdir icinga-selinux
cd icinga-selinux
audit2allow -M icinga-cgi -l -i /var/log/audit/audit.log

I removed icinga-cgi.pp and modified icinga-cgi.te to be the following.

Code:

module icinga-cgi 1.0;

require {
        type initrc_tmp_t;
        type httpd_sys_script_t;
        class file { read write getattr open };
}

#============= httpd_sys_script_t ==============
allow httpd_sys_script_t initrc_tmp_t:file { read getattr open };

I then proceeded to compile the module and install it.

Code:

checkmodule -M -m -o icinga-cgi.mod icinga-cgi.te
semodule_package -o icinga-cgi.pp -m icinga-cgi.mod
semodule -i icinga-cgi.pp

--------------------

I then had to compile a second module to get "reschedule next service check" working.

Code:

audit2allow -M icinga-schedule-cmd -l -i /var/log/audit/audit.log

Module Source.

Code:

module icinga-schedule-cmd 1.0;

require {
        type httpd_sys_script_t;
        type usr_t;
        class fifo_file getattr;
}

#============= httpd_sys_script_t ==============
allow httpd_sys_script_t usr_t:fifo_file getattr;

Compile and install module,

Code:

checkmodule -M -m -o icinga-schedule-cmd.mod icinga-schedule-cmd.te
semodule_package -o icinga-schedule-cmd.pp -m icinga-schedule-cmd.mod
semodule -i icinga-schedule-cmd.pp

Leap second and Java woes

sag47 — Fri, 27 Jul 2012 17:25:18 GMT

This is kind of an old event however I thought I'd talk about it for both myself in the future and for anyone looking for an answer when it happens again. There is a bug in java or the kernel in which leap seconds cause some really nasty behavior. For instance, All app servers which include Jetty, Tomcat, and JBoss were pegged at 100% CPU usage and ridiculously high load averages. I don't know if this affects weblogic because if I recall they ship their own custom build of java which may or may not have this issue fixed. Restarting the app server does not resolve the issue however bouncing the entire server does resolve the issue (and destroys uptime). After the community put their heads together on the issue an easy fix was discovered. Simply restarting ntpd with some additional commands resolves the problem and java will chug on normally without the extra cpu load. In a nutshell here are those commands and the source where I found them.

Code:

/etc/init.d/ntpd stop
date
date $(date +"%m%d%H%M%C%y.%S")
/etc/init.d/ntpd start

SELinux, syslog-ng, and the like

sag47 — Tue, 19 Jun 2012 15:12:25 GMT

So recently in my home setup I installed a scheme for centralized logging. I use rsyslog or syslog on the client servers and syslog-ng on the central server. This is nothing new and has been done before.

rsyslog + syslog-ng = gold.

I have a script which parses the syslog-ng logs across all the servers, filters them, and then sends and email with a summary and unusual logs. Each time I got an email I would get, on average, ~200-500 log messages of SELinux errors and alerts.

The errors looked something like this.

Code:

Jun 19 08:00:01 stealth: setroubleshoot: SELinux is preventing /sbin/syslog-ng from using the fowner capability. For complete SELinux messages. run sealert -l 3e6b6041-071a-483d-a3f8-79e7ec50e2a6
Jun 19 08:00:01 stealth.home: setroubleshoot: dbus avc(node=stealth.home type=AVC msg=audit(1340107201.216:9899): avc:  denied  { fowner } for  pid=462 comm="syslog-ng" capability=3  scontext=system_u:system_r:syslogd_t:s0 tcontext=system_u:system_r:syslogd_t:s0 tclass=capability node=stealth.home type=SYSCALL msg=audit(1340107201.216:9899): arch=c000003e syscall=91 success=no exit=-1 a0=e a1=1a0 a2=3e9 a3=1 items=0 ppid=1 pid=462 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="syslog-ng" exe="/sbin/syslog-ng" subj=system_u:system_r:syslogd_t:s0 key=(null)
Jun 19 08:00:01 stealth.home: setroubleshoot: AuditRecordReceiver.feed() got node=stealth.home type=AVC msg=audit(1340107201.216:9899): avc:  denied  { fowner } for  pid=462 comm="syslog-ng" capability=3  scontext=system_u:system_r:syslogd_t:s0 tcontext=system_u:system_r:syslogd_t:s0 tclass=capability

And those messages repeated on. After googling around I found that certain policy stuff was out of date and that it needed to be modified for the current syslog-ng. I was looking up modifying the policy and started looking in the package repository (yum on Fedora 16) for the policy sources. I talked to a buddy who is better at SELinux than I and he said it's all about policy modules now. You no longer have to modify policy source code and rebuild the sources.

This is neat I thought, so I googled around and found this great article.
http://fedorasolved.org/security-sol...odule-building

I then proceeded to build my policy module and install it. I did all commands as root user.

Code:

cd /root
mkdir selinux
cd selinux
man audit2allow
#you always want to look at the man page an understand the options before running any command
audit2allow -M syslog-ng-modified -l -i /var/log/audit/audit.log
ls
#syslog-ng-modified.pp and syslog-ng-modified.te show up in the directory
vim syslog-ng-modified.te
semodule -i syslog-ng-modified.pp

Here is the source of syslog-ng-modified.te.

Code:

module syslog-ng-modified 1.0;

require {
        type syslogd_t;
        class capabilityfowner;
}

#============= syslogd_t ==============
allow syslogd_t self:capability fowner;

After running semodule -i the policy was installed. I was curious where it was installed so I looked it up.

Code:

updatedb
locate syslog-ng-modified.pp

I found it at the following locattions.

/etc/selinux/targeted/modules/active/modules/syslog-ng-modified.pp
/root/selinux/syslog-ng-modified.pp

I further investigated what other modules there were but no need for me to keep ranting about that. Since then, my logs have been relatively quiet except for security concerns and I got SELinux and syslog-ng to play nice together. Pretty cool experience.

SAM

Removing files with find, and LVM snapshots

sag47 — Tue, 24 Apr 2012 16:21:47 GMT

Take an LVM snapshot using a loopback device
Create the snapshot
Take a look at the devices in your volume group. First there needs to be sufficient space in the volume group for a snapshot to occur. Check this by typing: vgdisplay and checking for Free PE / Size

Code:

vgdisplay -v

Create a loopback device to use for the snapshot. My volume group is named vg_echo360. I want my snapshot to be 9.9GB in size so I'm creating a loopback device which is 10GB in size.

Code:

dd if=/dev/zero of=/app/loop bs=1 count=1 seek=10737418240
losetup -f /app/loop
losetup -a

Now that you see which loopback device your /app/loop file is using you can add it to the volume group. For me it's /dev/loop0.

Code:

pvcreate /dev/loop0
vgextend vg_echo360 /dev/loop0

Create the snapshot.

Code:

lvcreate -s -n echo_snapshot1 /dev/vg_echo360/lv_echo360 -L 9.9G
lvs

Recover files from an existing snapshot
It's relatively easy to recover files from an existing snapshot. Basically just...

Code:

mkdir /mnt/snapshot_recovery
mount /dev/vg_echo360/echo_snapshot1 /mnt/snapshot_recovery

Now the path /mnt/snapshot_recovery contains the filesystem at a state from when the snapshot was taken.

Remove the snapshot when no longer needed
Be aware that LVM will throw out the snap space if it overflows. Keep an eye on it while changing things with the "lvs" command.

Code:

lvremove /dev/vg_echo360/echo_snapshot1

Do not allow the LVG to allocate data to /dev/loop0 any longer. This sets the PV Status to NOT allocatable.

Code:

pvchange -xn /dev/loop0

Check out the status of /dev/loop0 compared to the other devices.

Code:

vgdisplay -v

Remove the loopback device from the LVG entirely.

Code:

vgreduce vg_echo360 /dev/loop0

See that the device is missing in the following command?

Code:

vgdisplay -v

Now that the device is removed from the LVG you can remove the loopback device from the system entirely.

Code:

losetup -a

/dev/loop0 is the device I want to remove which is using /app/loop.

Code:

losetup -d /dev/loop0
rm /app/loop

FINAL NOTE ON LVM SNAPSHOTS: If you accidentally removed the loopback device before removing it from the LVM that's okay. Just be sure to run the following command on the LVG.

Code:

vgreduce --removemissing vg_echo360

Using find to remove files
Today at work I had to find all raw files a media server archived which were older than 120 days and delete them. They are regular files and have the following two possible extensions: .aac and .h264. So I built a single find command. I first created an LVM snapshot in case I needed to revert changes.

Before running the remove command put the found file names in a temporary file.

Code:

find . -mtime +120 -type f -name '*.h264' -o -mtime +120 -type f -name '*.aac' | tr '\n' ' ' > /tmp/rawfiles

You can see how much space you will save by doing that.

Code:

du -shc $(cat /tmp/rawfiles)

Now delete all of the files.

Code:

find . -mtime +120 -type f -name '*.h264' -exec rm {} \; -o -mtime +120 -type f -name '*.aac' -exec rm {} \;

Now that all the files are deleted make sure that there aren't any left over.

Code:

du -shc $(cat /tmp/rawfiles) 2> /dev/null

The disk usage should say 0 total used space.

See man find.
OPERATORS section for -o option.
See TESTS subsection in the EXPRESSIONS section of the find man page for the rest of the options.

Another thing to note, when using the -o option is that you have to treat each expression on either side like a completely new find command. It essentially allows you to put two find commands on a single line.

Making predictions and thinking like a computer

sag47 — Fri, 24 Feb 2012 01:33:00 GMT

Another user from a different forum asked me a question. I gave, what I though was, a good answer so I'm posting it here for all LQ to enjoy.

Quote:

Originally Posted by Capt Kenpachi

Sar & Jim;

If I were to build a database that were to compare historical purchases for a single individual to forecast what they are likely to purchase again in the future...

I know I would use a SQL database to store the Historical Data on the individual.

But what would I use to comb through the Data to determine the forecast?

(for example jean purchases)
id | Purch Date | Dark Jeans | Light Jeans | Zipper crotch | Button Crotch | etc

I would fill in yes or no's into each category (other then name and purchase date) to simplify the input process and create less chance for error (spelling differences/typos/etc). I would want to code to scan the known historical data on each person and come up with a profile on the type of jeans they want to buy (Light Jeans, zipper Crotch, Boot Cut, etc etc etc).

What scripting language would allow for me to compile the historical data together for the forecast prediction?

Thanks,
Jon

Well you could make the yes or no an integer or boolean. 0=no and 1=yes.

Then you can select all entries in the database and run a length command for the entries returned. Let's call this return length variable num_purchases.

You can handle the prediction in one of four ways. There are other methods that exist but I'll mention four.

FIRST
Static prediction as in make the same prediction every time. Predict the same thing every time. Keep track of the number of times the prediction was right in a separate database table. We'll call this variable correct_predictions.

Accuracy:
correct_predictions/num_purchases*100 = percentage of correct predictions.

SECOND
Start out with a static prediction. Iterate through all of the returned values and create a sum for each category. Let's call these values sum_darkj, sum_lightj, sum_zipperc, and sum_buttonc respectively.

Category #1
sum_darkj/num_purchases*100 = percentage_of_time_item_is_purchased
sum_lightj/num_purchases*100 = percentage_of_time_item_is_purchased

Category #2
sum_zipperc/num_purchases*100 = percentage_of_time_item_is_purchased
sum_buttonc/num_purchases*100 = percentage_of_time_item_is_purchased

Choose only one highest percentage from each category and make that your next prediction. Keep track of correct predictions and calculate accuracy like you did in equation #1.

As you accumulate more data, statistical forecasts may be more accurate but as you accumulate more data the predictions will be more static like static prediction. Only predicting the highest percentage of chosen features in an item each time. This method is usually more accurate than method two. But only slightly depending on how the user spends.

THIRD
The simple two bit branch predictor. See the following articles.
http://en.wikipedia.org/wiki/Branch_predictor
http://cs.utsa.edu/~dj/cs3853/bpexample.html

You have two bits to work with. This means that your branch has four states of yes/no prediction.
00=strongly light jeans (T)
01=weakly light jeans (t)
10=weakly dark jeans (n)
11=strongly dark jeans (N)

You start out by predicting a default feature weakly (weakly taken state) in our case weakly light jeans. If you get it right then you subtract one from the state and it moves to strongly light jeans. If you get it wrong then you add one and it moves to weakly dark jeans for the next prediction. Every time you get the answer right you keep predicting the same option as before and switch to a more strong stance on predicting that state. If you get that prediction wrong then you move to a weaker state of prediction on your next prediction.

A simple example of a buyer and predictor with the initial state at 01.
Actually purchased item, prediction, correct prediction?, next state?
light jeans, t, yes, T
dark jeans, T, no, t
dark jeans, t, no, n
dark jeans, n, yes, N

The simple two bit branch predictor can be wrong 3 times in a row before it gets the prediction right. This is not always optimal which is why a more advanced two bit branch prediction scheme was invented.

FOURTH
A more advanced two bit prediction scheme is more simple than it sounds. It's the exact same as the simple two bit predictor. With exception that if it gets two wrong predictions in a row then it will automatically switch to choosing the opposite jean.

Do some tables of prediction and the more you do you'll find at times it's equal or better.

Hopefully that sheds some light on what you want to accomplish.

To sum it all up basically you can calculate accuracy using method #1, split each feature of an item into two categories like in method #2, and then apply 2-bit branch prediction on each category. You'll find with more numbers your predictions will be most accurate over other methods applied.

Monitoring sound over time for long periods to see trends using a microphone

sag47 — Thu, 09 Feb 2012 21:54:39 GMT

Recently a post was made requesting a solution to detecting sound in a studio to see if it is populated with lots of noise (lots of noise=people/bands playing) this way the person monitoring it wouldn't have to go through the trouble of hulking all their stuff to the studio.

I posted an interesting solution so I'm sharing it here and a little more on this blog. See post #9 of this thread.

I'm going to expand upon that munin plugin and make it a little more robust and informative. I'll rewrite the munin plugin a bit to include more information on the same graph and be a little better.

Code:

#!/bin/sh

case $1 in
   config)
           cat < /dev/null
echo -n "mxamp.value " 
sox /tmp/foo.wav -n stat 2>&1 | grep 'Maximum amplitude' | awk '{print $3}'
echo -n "mnamp.value "
sox /tmp/foo.wav -n stat 2>&1 | grep 'Minimum amplitude' | awk '{print $3}'
echo -n "mdamp.value "
sox /tmp/foo.wav -n stat 2>&1 | grep 'Midline amplitude' | awk '{print $3}'
rm -f /tmp/foo.wav

Of course, that plugin can be further improved by writing an awk script which parses out all of the values and displays all three values at once that way the sox command is only run once and it uses less CPU. Though power savings will be almost negligible by doing that. I kept it this way for readability mostly.

The new munin plugin improves upon the last one by providing more information which is a little more useful than just knowing the maximum amplitude of the 1 second sample recorded at the time of running the plugin. At any rate I thought that was a cool little project I got myself into so I documented it here.

SAM

Creating special devices to solve problems

sag47 — Thu, 19 Jan 2012 21:53:49 GMT

With tax season coming up I have been asked to set up a certain centralized tax service for a company. I won't mention the company nor the well known tax program involved as this post isn't about flaming a product but just to show you how cool commands in Linux can be used to solve, what would originally seem like, complicated problems.

Here's the scenario. The software runs on JBoss (a java app server) and it has log output. During client testing with fake social security numbers I noticed that they were getting output into the log. For security purposes this is unacceptable. After thinking for a while the solution I came up with was this... I want to output the log into /dev/null. But how? How indeed, and here's what I did.

I remember that the mknod command is used for special devices so I looked at the man documentation.

The relevant portions of the man page are...

Code:

--- *snip* ---
SYNOPSIS
       mknod [OPTION]... NAME TYPE [MAJOR MINOR]

DESCRIPTION
--- *snip* ---
       c, u   create a character (unbuffered) special file
--- *snip* ---

Now let's look at the file listing for /dev/null.

Code:

$ ls -l /dev/null
crw-rw-rw- 1 root root 1, 3 2012-01-12 10:12 /dev/null

The first character in the permissions is the filetype. Normally you see d for directory, - for file, or l for a symbolic link as an example. Here specifically it is a character special file.

Also notice the "1, 3" in the file listing. Those are the device special major and minor numbers. I'm not sure where there's a list of all of the different types of numbers but if you know please post in the comments. Here though we basically know how we want the file to behave and had a file we could analyze to get the major and minor numbers so that mknod knows what kind of special device we're creating.

So now I just go to where the log file is outputting and create a null special device in the file name where the log name would normally be. Let's say, hypothetically, that our offending log output file is located at /opt/jboss/log/program.log. Here's the commands to create the special device and match it be the same type as /dev/null.

Code:

cd /opt/jboss/log/
rm program.log
mknod program.log c 1 3
chmod 666 program.log

My jboss instance runs under user jboss but my program.log special device is owned by root with permissions 666. That's because I don't want jboss or the app to be able to rename or move the special device.

Now when log output is written to the log file it will simply be going into a null-like file which isn't recorded. Of course if you ever need to troubleshoot you'll have to remove this special file so that it can write to a log file again for output. But for normal usage of the app I'd say this is a pretty good solution to a closed source proprietary program which was simple for the problem at hand.

SAM

Ventrilo and RHEL/Fedora

sag47 — Wed, 04 Jan 2012 01:19:00 GMT

Well I just set ventrilo up on my internal dev server. Just trying it out because I've heard other people talk about it and thought I would integrate it better with my system. I'm running Fedora 14 on my server but all of these scripts should work on Fedora 12/14 and RedHat EL 5/6. Untested on the new Fedora 16. Here's the service script.

Code:

#!/bin/bash
#Created by Sam Gleske
#For RHEL and Fedora-like operating systems

# chkconfig: - 0 0
# description: ventrilo is the ventrilo daemon. \
# It is a team speak-like service which runs on \
# port 3784.

### BEGIN INIT INFO
# Provides: ventrilo
# Short-Description: start and stop ventrilo
# Description: ventrilo is the ventrilo daemon. It is a team speak-like
#              service which runs on port 3784.


# source function library
. /etc/rc.d/init.d/functions

# Startup ventrilo servers.
#working directory of the service
SRVPATH=/home/ventrilo/ventsrv
#Service binary
SRVBIN=$SRVPATH/ventrilo_srv
#user which the subshell will run
SUBUSER=ventrilo
#service name for status messages
SRVNAME=Ventrilo

start(){
  if [ -f "$SRVPATH/ventrilo_srv.pid" ];then
    echo -n "$SRVNAME is already running."
    failure
    echo ""
  else
    su $SUBUSER -c "$SRVBIN -f$SRVBIN -d"
    #renice -5 `cat $SRVPATH/ventrilo_srv.pid`
    success
    echo "$SRVNAME has started."
  fi
}
stop(){
  if [ -f "$SRVPATH/ventrilo_srv.pid" ];then
    kill -s 15 `cat $SRVPATH/ventrilo_srv.pid` &> /dev/null
  else
    failure
    echo "$SRVNAME is not running."
    return 1
  fi
  if [ "$?" -eq "0" ];then
    echo -n "$SRVNAME is stopping."
    wait && success
    echo ""
  else
    failure
    echo "$SRVNAME is not running."
  fi
}
wait(){
  while true;do
    if [ ! -f "$SRVPATH/ventrilo_srv.pid" ];then
      return 0
    else
      sleep 1
      echo -n "."
    fi
  done
}
restart(){
  stop
  start
}

#control center
case "$1" in
start)
  start
  ;;
stop)
  stop
  ;;
restart)
  restart
  ;;
status)
  status ventrilo_srv
  ;;
*)
  echo "usage: $0 (start|stop|restart|status|help)"
esac

HOWTO: Ventrilo install on RedHat/Fedora

1) Create the user ventrilo

Code:

adduser ventrilo

2) Write my vent daemon script to /etc/init.d/ventrilo.

3) Download ventrilo_srv-3.0.3-Linux-i386.tar.gz from ventrilo.com to ~ventrilo and extract it.

4) Make sure ownership of all ventrilo files and folders are to user ventrilo.

5) Control the service using the following commands.

Code:

# service ventrilo help
usage: /etc/init.d/ventrilo (start|stop|restart|status|help)

6) Add the service to the machine startup process if you want ventrilo to start on boot.

Code:

chkconfig --add ventrilo
chkconfig --levels 35 ventrilo on

Don't forget to poke a hole in your firewall (I hope you're using a firewall) for port 3784 and that should be it!

SAM

SELinux and sshd

sag47 — Mon, 02 Jan 2012 22:55:53 GMT

So I've started using the new Fedora 16 on my main media server (previously F14) and decided to try something new this time. I've always set SELinux to permissive which is a bad idea normally for anything public (which this server isn't but I still want it). Anyway I believe I was testing something before I left for the holidays and ran

Code:

setenforce 0

which disables SELinux. I guess at some point it turned back on somehow and I was locked out of ssh. I dug through my logs (/var/log/messages, /var/log/audit/audit.log) and didn't find much but then ran

Code:

tail -f /var/log/secure

and tried to log in via ssh again. Well I found a hint of what was going wrong because I found the following secure log errors.

Code:

Jan  2 17:11:49 stealth sshd[9225]: error: Could not get shadow information for sam
Jan  2 17:11:49 stealth sshd[9225]: Failed password for sam from 127.0.0.1 port 55123 ssh2

I thought maybe it was an SELinux context thing and ran

Code:

ls -Z /etc/ssh/
restorecon -R /etc/ssh/
systemctl restart sshd.service

which had no effect. Finally after I felt I've exhausted my local resources I turned to google. I googled "selinux ssh could not get shadow information for *" without quotes.

Clicked on the first link which was this one. Read the whole thread and at the bottom the person resolved it by removing /etc/ssh/ and then reinstalling openssh-server. This, by the way, is the dumbest solution ever and anytime someone reinstalls or reboots to fix something in Linux just means that the problem will crop up again. So our journey continues.

The next link I click on was a redhat ticket. I rebuilt my policy database by running "semodule -DB" but that had no effect. It wasn't until I got to the bottom of the thread where the user posted back with their solution. Apparently SSHD doesn't use PAM (Pluggable Authentication Modules) by default. It was commented out in their installation so I checked mine. Sure enough the setting "UsePAM" was commented out so I uncommented it.

Code:

UsePAM yes

The odd thing about it is that I had uncommented another setting "PasswordAuthentication yes" and thought I had resolved this. That's what I get for not reading the entire config the first time I configured it.

For those who are curious I have set the following changes away from defaults on my sshd_config.

Code:

PermitRootLogin no
AllowGroups wheel
ChallengeResponseAuthentication no
UsePAM yes

If you use PKAuth (public key authentication) then you don't want the ChallengeResponseAuthentication set to no.

In addition to that, for samba, I had to allow a few extra SELinux rules which was a little pain.

Code:

setsebool -P samba_export_all_ro 1
setsebool -P samba_export_all_rw 1
man samba_selinux
chcon -R -t samba_share_t /my/share

That was my adventure after the holidays.

SELinux and Apache ProxyPass

sag47 — Wed, 07 Dec 2011 18:39:32 GMT

Today I was having trouble getting a httpd ProxyPass and ProxyPassReverse to work. As a background I'm working in RedHat Enterprise Linux 6.1. It took me a while to figure out what was wrong (longer than it should have) and as it turns out it worked after I disabled SELinux. So my configs were correct in /etc/httpd/conf.d/* and it passed all config tests. I'm sure you don't want to read a whole post of talking so I'll just get right into the grit of the commands. For the rest of the explanation you can use the man pages. I'm assuming a higher level audience in this post though with man pages anyone can learn.

Code:

tail -f /var/log/httpd/error_log
#shows an error
setenforce 0
#config works so enable selinux again
setenforce 1
getsebool --help
getsebool -a
getsebool -a | grep ^httpd
cat /var/log/audit/audit.log | audit2allow -v
#audit2allow will tell us what entries can be enabled to allow selinux to work
setsebool httpd_can_network_relay on
#test config so does it work?  Yes
getenforce
#make sure selinux was still on
#now save the changes to be permanent for when the machine reboots.
setsebool -P httpd_can_network_relay on

Yay for speed blogging! I feel better now that it's out of the way.

Stress testing a new hard drive...

sag47 — Mon, 21 Nov 2011 04:38:24 GMT

You should always stress test a new harddrive when you buy it.

Some people say, "Just hook it up for a few months and keep unimportant data on it. If it breaks in that 1-3 month time then it will be good for 3 or 4 years."

While that metric is okay at best I prefer to take a more systematic approach to it. I want to fill the whole hard drive with random 1 and 0 over a constant and long period of time. I do that because you are stressing the hard drive out with max writes for a long period of time (I usually do it for 7 days before I'll pass the hard drive). If the hard drive fails within those 7 days then it was going to break within a month or two likely.

I encountered just that situation over the past day. Here are the scripts I ran to stress it out (run as root).

hdd-stress.sh

Code:

#!/bin/bash

while true;do
  dd if=/dev/zero of=/dev/sdk &> /root/log.txt
  sleep 2
done

hdd-stress2.sh

Code:

#!/bin/bash

while true;do
  dd if=/dev/urandom of=/dev/sdk &> /root/log2.txt
  sleep 2
done

hdd-stress-killall.sh

Code:

#!/bin/bash
#kill the scripts
ps aux | grep 'bash.*hdd' | grep -v 'grep' | awk '{print $2}' | xargs kill -15
killall dd

To run them just sub it so that it doesn't kill when you log out.

Code:

su - -c '/root/hdd-stress.sh &'

After 3 straight days of letting it run with a nice value of 0 here is the result from the tail of dmesg.

Code:

... it continues for a long time up ...

[42931.558804] sd 7:0:0:0: [sdk] Unhandled error code
[42931.558807] sd 7:0:0:0: [sdk] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[42931.558812] sd 7:0:0:0: [sdk] CDB: Read(10): 28 00 e8 e0 88 a8 00 00 08 00
[42931.558821] end_request: I/O error, dev sdk, sector 3907029160
[42931.558825] Buffer I/O error on device sdk, logical block 488378645
[42931.565946] Buffer I/O error on device sdk, logical block 2
[42931.567031] Buffer I/O error on device sdk, logical block 3
[42931.568285] sd 7:0:0:0: [sdk] Unhandled error code
[42931.568289] sd 7:0:0:0: [sdk] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[42931.568295] sd 7:0:0:0: [sdk] CDB: Read(10): 28 00 00 00 00 00 00 00 20 00
[42931.568310] end_request: I/O error, dev sdk, sector 0
[42931.569451] Buffer I/O error on device sdk, logical block 0
[42931.570577] Buffer I/O error on device sdk, logical block 1
[42931.570615] sd 7:0:0:0: [sdk] Unhandled error code
[42931.570619] sd 7:0:0:0: [sdk] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[42931.570624] sd 7:0:0:0: [sdk] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
[42931.570634] end_request: I/O error, dev sdk, sector 0
[42931.570638] Buffer I/O error on device sdk, logical block 0

It's a good thing I did this before giving my brother his present. It would have been really annoying if he had to been the one to have to RMA it.

SAM

Limiting and blocking connections dynamically.

sag47 — Mon, 31 Oct 2011 20:46:13 GMT

Today I feel like talking about limiting connections which get made to a server. There are a few ways to do it; some cooler than others. I'd like to feature some open source software while I'm at it.

Today I happened upon this thread which hilariously got closed for good reason. Once you wade through all the crap you'll see some pretty cool posts which explain how to limit incoming connections within a certain time period and other suggestions. I'll point out the most useful related ones and then recommend an open source solution on top of it.

Post Highlights

My first highlight is post #8. This cool post shows how you can limit incoming connection attempts on any given port (in the example port 22) to just 3 times a minute. I'm not an iptables guru but I thought that little gem was pretty cool.

Second highlight would be post #12 and post #14. Post #14 being a simple breakdown explaining what post #12 does. This is nice, because it shows how you can put the suggested rules in post #8 within a script if you don't already have a file for iptables rules and get it up and running fast. Quick and dirty but it works until the server is bounced (restarted).

This brings me to my last highlight of the thread, post #20. Building upon all the previously mentioned highlighted posts, this post shows you how to make the rules more permanent which could then optionally be enabled by default when the machine is rebooted (or loses power, or whatever).

Followup results
Here's a quick command tutorial (commands only) which explain the process of enabling that in CentOS (Actually tested on RedHat but methodically the same). This also applies to Fedora 14 (Only other version I've tested on but likely for all versions of Fedora as well). If you're running Ubuntu server then you'll want to read this instead of following my commands verbatim.

Also note these commands are run as root. These commands use information from all previously mentioned posts and then elaborated.

Code:

service iptables stop
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -i eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A INPUT -i eth0 -p tcp --dport 5060 -m state --state NEW -m recent --set
iptables -A INPUT -i eth0 -p tcp --dport 5060 -m state --state NEW -m recent --update --seconds 60 --hitcount 4 -j DROP
iptables -A INPUT -i eth0 -p tcp -m tcp -m state --state NEW -j ACCEPT
iptables -A INPUT -i eth0 -p udp -m udp -m state --state NEW -j ACCEPT
iptables -A INPUT -i eth0 -p icmp -m icmp --icmp-type any -j ACCEPT
iptables -A INPUT -i eth0 -j REJECT --reject-with icmp-host-prohibited
iptables-save > /etc/sysconfig/iptables
chkconfig --add iptables
chkconfig --levels 35 iptables on
service iptables start

Remember, the rules I mentioned above are purely within the scope of the thread this blog post is about. Your system performance requirements may be different and you'll likely want more/less/different restricting port rules than the ones mentioned above.

I'd also like to state there is no such thing as an IP Phone system or application which is incompatible with SELinux, iptables, or any other security layer outlined in the PCI Security Standards. It just means you have to configure these services yourself for the given application assuming the application developers were not nice enough to include security as a section in their docs. So unless you're running a blog about cheese sandwiches then I suggest turning on as much security as you have available for your platform. If you can't be within security standards then you're using the wrong OS or you're an incompetent sysadmin.

"incompetent sysadmin" is not an insult but should be taken as an incentive to pick up a book and learn about these skills.

Recommended open source project
Now that we're on the same page with iptables it's time to introduce you to Fail2ban. Fail2ban is a cool piece of software we use here at Drexel University. Fail2ban scans auth logs associated with different services for failed login attempts. After a certain number of failed login attempts (configurable) the violating IP address attempting the logins gets banned.

Basically it reads the source IP of the auth request and then updates the iptables firewall rules to block incomming connections just from that IP address.

This is more elaborate than the previous iptables rules because it dynamically updates the firewall rules based on current and past connections. You can make the bans permanent or temporary.

Fail2ban is not a replacement for iptables but rather a helper application which further improves the security and integrity of any given system. When bullets don't work, bombs and nuclear missiles will.

DISCLAIMER: I am not being paid to promote fail2ban or any open source project on my LQ blog.

Server latency

sag47 — Thu, 27 Oct 2011 13:26:43 GMT

I wanted a quick and dirty way to determine server latency on my Kubuntu machine. This just parses simple output from ping and provides only the time values.

Code:

for x in drexel.edu google.com;do echo -n "[$x]: "; ping -c 1 $x | head -n 2 | tail -n 1 | awk '{split($8,arr,"=")} END{print arr[2],$9}';done

Output

Code:

[drexel.edu]: 0.576 ms
[google.com]: 6.23 ms

You can put as many servers or IPs as you want in the for loop to get a comparison. Just simple little interesting statistic.

SAM

Network usage on a single device and log diving

sag47 — Tue, 27 Sep 2011 19:57:08 GMT

I wanted to know the received network usage on a device in kB so this one liner was produced for the desired result.

Code:

while true; do y=`ifconfig eth0 | grep RX\ bytes| sed -e 's/.*bytes\://' -e 's/\ .*//'`; sleep 1; x=`ifconfig eth0 | grep RX\ bytes| sed -e 's/.*bytes\://' -e 's/\ .*//'`; echo $(((x-y)/1024)); done

Press Ctrl+C to stop analyzing the network traffic on that device.

Here's another one liner of me looking through multiple log locations.

Code:

for x in /fake1/logs/ /fake2/logs/;do find $x -type f -print0 | xargs -0 grep -iH 'search for something' | cut -d: -f1 | sort -u;done

Hell yea.

Simulate -print0 with ls

sag47 — Wed, 07 Sep 2011 08:39:39 GMT

Recently I wanted to manipulate files only in the current directory (not sub-directories). I usually use the following combination...

Code:

find . -type f -print0 | xargs -0 somecommand

The only problem I had with that was it ran through all sub-directories as well. Then I remembered the -m switch for ls and decided to depend on my good old friend sed.

Code:

ls -m | sed 's/, /\x0/g' | xargs -0 somecommand

Though I guess it would be better to do it like so...

Code:

find  . -maxdepth  1 -type f -print0 | xargs -0 somecommand

So many ways to do it and I'm just playing around.

SAM

rsync commands and how I backup

sag47 — Thu, 23 Jun 2011 22:06:33 GMT

I run the following command when I want to just update one directory to another. It only copies files that have changed. If any files have been deleted from A they still exists in directory B. The trailing slash after A/ is necessary to tell rsync to move the contents. Otherwise A will create a folder in B called /media/B/A and move the files over.

Code:

rsync -ruptv /media/A/ /media/B

I run the following command when I want to mirror one directory to another. Any files that are deleted from A are also deleted from B. Only modified files in A are moved to B.

Code:

rsync -ruptv --delete /media/A/ /media/B

This command is similar to the last command but it excludes certain files/types.

Code:

rsync -ruptv --delete --exclude=Thumbs.db /media/A/ /media/raid/B

I utilize those commands on a schedule with crontab -e.

Code:

0 3 * * 3,6 /home/sam/backup.sh

Basically that command runs every Wednesday and Saturday at 3am. This is pretty easy to use.

SAM

Convert and resize images using imagemagick

sag47 — Tue, 31 May 2011 03:07:40 GMT

This past memorial day weekend I took a bunch of pictures which I wanted to post to the web. Web photos are best viewed in 800x600px and I had a lot of them. So I used a simple command to do it.

Code:

for x in `ls -1`;do convert -resize 800x600 $x `echo $x | sed 's/.JPG/.png/'`;done
mkdir web_album
for x in `ls -1 *.png`;do convert $x web_album/`echo $x | sed 's/.png/.jpg/'`;done
rm *.png

Now I don't have to spend all my time in Gimp with that simple batch command.

DVD Ripping

sag47 — Mon, 25 Apr 2011 15:54:18 GMT

I work for a Scientific Institution and a company which recorded talks decided to give us all the talks in DVD format instead of a hard drive full of video files. This is kind of counter intuitive however luckily I use Linux at work. Long story short mencoder with lame installed solved my problems.

Code:

mencoder 
-vf harddup 
-vf-add smartblur=.6:-.5:0,unsharp=l5x5:.8:c5x5:.4 
-xvidencopts fixed_quant=4:profile=dxnhtntsc 
-lameopts cbr:br=128:aq=0:vol=1 
-oac mp3lame 
-ovc xvid 
infile.vob -o outfile.avi

Source of my command.

MAC Mania

sag47 — Thu, 07 Apr 2011 19:01:47 GMT

Today I was having mac address problems. My corp network runs a mac filtered domain so in order to avoid the hassle of contacting IT (which are stupid) I simply copied the macs from other machines and issued them to my servers (I'm a technical head for my department). Well I didn't save the MAC address and the server was shutdown when we moved it which blew away the mac. What did I do? Search the logs for every MAC address ever.

Code:

find . -type f -print0 | xargs -0 grep -iH '\([0-9a-f]\{2\}[:-]\)\{5\}[0-9a-f]\{2\}\|[0-9a-f]\{12\}' | cut -d: -f1 | sort -u

Kernel Panic!

sag47 — Sat, 16 Oct 2010 22:30:45 GMT

Everybody dreads it. When you see it, it is difficult to know what to do. Luckily Unix provides utilities which allow us to traverse the filesystem and parse files quickly and effortlessly.

Here's a search command which will search all your log files for keywords such as shutdown, poweroff, and panic.

Code:

find /var/log/ -type f -print0 | xargs -0 grep -iH 'shutdown\|poweroff\|panic' | cut -d: -f1 | sort -u

This will allow you to narrow which log files you should be looking at. Once you know which log files you need to look into then you need to grep each one of them to find out which lines the error occurrence shows up within each file. It is best to inspect the files individually. Lets call the file shutdown.err for this example.

Code:

grep -iHn 'shutdown\|poweroff\|panic' /var/log/shutdown.err | cut -d: -f2

Lets say that line 498 showed up in the results. You can use vim or the less pager to view the log at line 498 to see what caused the shutdown error.

Code:

less +498 /var/log/shutdown.err
vim +498 /var/log/shutdown.err

Based on what error messages you see you can google around to hopefully find a concise accurate solution to your problem.

openbox and repeating keys

sag47 — Mon, 11 Oct 2010 06:50:16 GMT

Recently I've been using openbox but it has been driving me crazy! Mainly because when you hold down a key it does not repeat like it does on all other WMs and OSs. After googling around forever I found I can enable my auto repeating keys in ~/.config/openbox/autostart.sh with the following code.

Code:

xset r on

Handy.

Batch convert images to PDF

sag47 — Sun, 10 Oct 2010 10:48:27 GMT

Recently (within the past year) I have helped a user with a problem converting hundreds of images into individual PDF files. I was pleasantly surprised to find imagemagick to effortlessly do this.

Command line tool imagemagick does what you need if they're images.
Installation:

Code:

sudo apt-get install imagemagick

Usage: (coverts all images into one pdf)

Code:

convert *.jpg foo.pdf

Usage: (converts each image into an individual pdf)
This is if you want .jpg to be replaced with .pdf. This command will work with any image. Replace .jpg with whatever image extension you want.

Code:

IFS=$'\n';for x in `ls -w 1 *.jpg`;do convert $x `echo $x | sed 's/\.jpg$/.pdf/'`;done

Here is the previous command but a little simpler. This is if you want the file extension to be .jpg.pdf.

Code:

IFS=$'\n';for x in `ls -w 1 *.jpg`;do convert $x $x.pdf;done

Here's your bulk solution for converting all image files to pdf individually...

Code:

IFS=$'\n';for x in `ls -w 1 *.jpg *.jpeg *.png *.gif *.tif *.tiff`;do convert $x $x.pdf;done

As you can see with a little bit of terminal ninja scripting you can get the job done relatively effortlessly. (search shell script howto in google)

3 easy ways to create ridiculous passwords

sag47 — Wed, 18 Aug 2010 01:12:14 GMT

A password can be easy to create if you use the right devices to remember it.

Method number 1: use a sentence
For example your password could be a sentence:

Code:

Ilovemydogfoofie

Dictionary attacks only work if your password is a single word. For that previous password a dictionary attack wouldn't work so a cracker would either have to resort to rainbow tables or brute force.

Method number 2: start and end all passwords with the same sequence.
For instance I always start certain passwords with a capital letter. I also start it with %123 and end it with a #.
So here's 3 insecure passwords:

Code:

fartsauce
happiness
poopy

And here's 3 passwords which are very secure:

Code:

%123Fartsauce#
%123Happiness#
%123Poopy#

You can also put them all together and have a super secure password uncrackable with AES-256

Code:

%123Fartsauce#%123Happiness#%123Poopy#

I mix and match and add to passwords like that all the time. So I can have ridiculously long passwords but they're all using passwords I already know and commonly use.

The only thing to remember is the %123 and the #. Words are easy.

Method number 3: draw shapes on your keyboard.
Here's a password which is pretty wild yet when you look at the keyboard it's so easy to remember. And all you have to do is count in your head 1 2 3 4 1 2 3 4. Just type the characters with your left index finger and middle finger. Kind of like playing the piano.

Code:

1q2w3e4r!Q@W#E$R

All I did was type 1q2w3e4r and then I held the shift key and typed 1q2w3e4r. A very simple combination when you put it graphically but when put into a linear context on the screen it looks overly complicated to remember and even harder to crack.

Mix and match.
You could also apply method #2 with method #3 and make it look even more confusing but easy for you in your head ;).

Code:

%1231q2w3e4r!Q@W#E$R#

Or use method #1 with method #2.

Code:

%123Ilovemydogfoofie#

Those are just 3 easy ways to create ridiculously crazy passwords

Streamlining the process of uploading

sag47 — Tue, 17 Aug 2010 16:21:01 GMT

Today I had to zip up about 50 folders and upload them to a software directory for my work. That's a huge hassle and they didn't need to be in any particular order. The names of the zip files had to be the same name as the folder so that made my life easier. I zipped them all, moved them into a filestoupload folder and then uploaded them.

A process which would have taken 10-20 minutes on a Windows machine took less than a minute of my precious time on my Linux box. For most of it I could walk away and get other things done.

Code:

sudo bash #simulate root login, I hate the sudo command so I use it as little as possible
for x in `ls -w 1`;do zip -r $x.zip $x;done;mkdir filestoupload;mv *.zip filestoupload/;chown -R sam\: filestoupload

Walk away, come back, bulk upload files, Done :)