When we were setting up our locking cabinets at the new data center for my job I was wondering how to actively monitor the security around our cabinets. Being that only myself and the data center staff have keys to the cabinets I wanted to be able to monitor when anyone entered them. I came up with a simple solution to be able to monitor the racks and keep track of changes. First I downloaded ZoneMinder, which is cam software that runs on Linux. It allows you to setup cheap consumer grade cams to monitor security. With such advanced features as motion detection. This made it ideal for my needs since I could go buy two cheap $30 web cams to use for front and back of cabinets. Then all I had to do was install ZoneMinder on our monitoring server and configure it for motion detection captures. Now the only step for me to do was put a sign-in sheet inside our cabinet with fields for techs name, date, time, and what was done in cabinet. Then require the data center staff to fill out the sheet every time they enter the rack. Another good idea is to do this before signing a contract with the data center you have chosen. Then you can stipulate that if they enter the rack without signing the sheet you get X amount off your bill that month for each infraction.
For our new data center I considered using Nagios, Cacti, and Zabbix. At the time Zabbix had it’s 1.1 version in Beta and looked very promising. In the past I had used Nagios and Cacti with much success even though each has their own caveats. But from the test install I had done of Zabbix I was impressed by a lot of its features. Along with setup being not too hard. Configuration as with all Open Source monitoring solutions is somewhat of a pain. But it is far easier than say Nagios. I also like the email and SMS alerting features that Zabbix offers. Along with the ease of creating custom events to monitor from various daemons such as MySQL. I will be sure to document more tip and tricks with Zabbix as I come across them.
I have come to the conclusion that as a whole Sendmail is an evil that must be stopped. It is right up there in my book with BIND as an inherent evil program that has been maintained well past its usefulness. A good example of this is I have been configuring a server to send out emails for various web app we run. First you must install sendmail-cf and m4 in order to get any real functionality. Then you have to go through and edit one of the worst designed configuration files in the history of config files. The only config file that may rival it is the Dovecot one. Which still at least makes sense even though it is 10 pages long. Just do yourselves a favor and uninstall all sendmail and replace with Qmail or Postfix.
At my job I have recently setup an easy means of managing code as well as the whole chain of development. All without the need of giving developers direct login access to servers. Another bonus of using SVN is the ability to roll back changes in real time if needed. For our organization I have set up the following:
-Development environment consisting of a large server running OpenVZ ( I will write another entry soon on server virtualization) with images identical to the production servers. This allows the developers to have their own VPS (Virtual Private Server) to code and test on. All without fear of blowing it up. If this happens I simply type a total of 5 commands to destroy and rebuild the VPS.
-Staging environment consisting of a staging server that also hosts a stripped down data set of our production DBs. Here we have the development code that has made it to the point of being ready to be QA’d. All developers have access to Staging via SVN. They check out a working copy of the repository and work on it via their VPS or desktop. At the same time the /var/www of the Staging server itself is a checkout of the SVN repository. Using SVN hooks (namely post-commit) that once a commit is made it forces a svn update /var/www to insure that when a developer or client visit the staging site they see the changes.
-Finally there are the live web servers which have their own seperate repository where the QA person moves files from the staging working directory to the live working directory. Then does an add/commit.
All of this creates an audit trail of what has been pushed live and who/what changes were made. With our change request system each commit is refernced to its CR. So by viewing the SVN log for both staging and live you can see what is what. If a file pushed live has an issue you simple roll back to the previous version. There is no need to guess which files and if all were pushed to begin with.
I am very pleased with how this is working and the developers like it much more than our previous versioning system. For the Windows minded developers I have found Tortoise to be the easiest for them to use and understand.