Puppet CMS success story

Our company has been offering services in the IT outsourcing market for more than 7 years. Our customers are very different, as well as their servers and configurations. Companies like ours which have large numbers of servers, at a certain point, begin to ask questions like: “How do we organize all of the chaos that we run on servers? How can the work that system administrators have to do be reduced? How can IT processes be optimized?” As a growing company, we are continuously seeking answers to these questions. Almost all of the questions were answered, until we were faced with The Puppet. It is a configuration management system, but we call it an infrastructure management system (IMS) because it is not performing simple configuration management. Here is the story.

For a long time, all of our work was minimized and optimized by bash/perl scripts. Everything was fine, up to the point when we realized that maintaining these scripts sometimes took more time and resources than just doing everything manually. We began to search for a solution. We investigated solutions such as Cfengine, Chef, and Puppet. At last, we decided to try Puppet, as it was more flexible then Cfengine and more developed then Chef.

First things that were done: we automated the use of common settings, configurations, and software on all of the servers that we administer. It was rather hard for us to make the proper tool to administer the infrastructure using IMS. Before we finished with internal IMS implementation we got requests from two customers (with more then 50 servers each) who wanted to implement something like Puppet in their infrastructure. It was another challenge for us, because implementing such tools in their infrastructure meant total review of infrastructure, procedures, processes and sometimes even the approach to the concept of work. After the review, in most cases everything was updated, changed and standardized for IMS implementation.

During implementation of these 3 projects, Internal IMS and IMS for 2 customers, many interesting features were implemented and new concepts were developed. Here is what we have up to this moment:

1. Managed SSH configuration files and SSH keys
- Enabled root login
- Managed predefined keys
- Managed per-server specific keys

2. Managed/configured sysctl (Linux, FreeBSD)
- Maintained whole systcl.conf file, same on all servers
- Defined custom values for sysctl variables
3. Enabled/disabled BSD accounting (Linux, FreeBSD)

4. Managed time zone
- Configured time zone of servers

5. Managed syslog
- Installed/configured rlogging software
- Defined custom per-server configuration
- Managed server’s running remote logging collect part (TODO)

6. Managed /etc/resolv.conf
- Managed resolver domain, searched domains, resolvers (predefined and custom)

7. Enabled/disabled bsdsar (FreeBSD)

8. Managed/configured inetd/xinetd
- Start/stop, configured needed services

9. Managed Puppet - client configuration file and Puppet service

10. Managed NTP client
- Installed, configured and start/stop service
- Defined NTP servers

11. Managed FreeBSD rc.conf
- Managed common rc.conf values and settings

12. Ability to determine what panel is installed (cPanel, DirectAdmin, Plesk)

13. DNS management
- Managing of DNS records in simple files or MySQL databases (using Puppet manifests for nodes) (TODO)

14. Advanced management of Nagios
- Complex integration into Puppet infrastructure (Nagios configuration is managed through Puppet)
- Automatic re-configuration
- Easy to administer (service is added by a 1 line define)

15. Advanced management of NRPE
- Managed, installed and configured NRPE plugins and Nagios records

16. Advanced management/configuration of Bacula server and client
- Triggers for Bacula client configuration and monitoring

To allow all functions to continue to work properly, our developers had to implement these next features in Puppet:
1. Manage lines in files using patterns
2. Manage simple configuration files (for hosts where Augeas not available)
3. More advanced work with arrays (what Puppet by default can not do or it is complicated to do)
4. More advanced work with hashes (what Puppet by default can not do or it is complicated to do)
5. Many custom facts.

All of these gave, us and customers, the possibility to free up many human resources for other projects and things to do. For technical staff, now it is easier to administer servers. They can now add up to 6 services, in 1 line, into monitoring. The concept of standardization and unification showed us that we were performing things correctly. We already feel all benefits of IMS implementation and know that this is not the end and we still have many todo`s as nothing stays the same if we want to be called a modern and growing company. It is understood that you have to follow the market and IT world trends and needs.