Josh’s Blog

Foreman Upgrade to 1.7 Broke My RunDeck Integration!

In the not too distant past I put in the effort to get r10k integrated with our Puppet & Foreman systems. For those not familiar with r10k, when executed it will handle the dynamic creation of puppet environments on your master. Your r10k config is stored in a git repo, and for each branch of that repo, a puppet environment will get generated. Each branch has a PuppetFile, which in turn points to the puppet modules you wish to install in the environment.

To automate this whole process, I glued whole process of executing r10k and it’s associated tasks into our RunDeck instance. When a technician commits to either the r10k git repo, or one of our puppet module repos, a webhook from our git system (we use Stash) to a job on RunDeck via it’s API will fire off. This job wil connect to our puppet server, run the r10k deploy command, followed by instructing Foreman to reload it’s puppet environments via it’s API. The whole process is quite slick, and allows us to quickly iterate on puppet module development.

In December, I began the process of upgrading our Foreman instance from 1.6x to 1.7. For the most part, the upgrade was smooth. What I did not notice is that the foreman plugin that handles rundeck integration was no longer included with the main Foreman package. This Foreman blog entry explains this change, but of course I did not see that until it was rendered fubar.

At the time of working this issue, the Foreman wiki stated that one needs to now install this plugin via rpm package. The only problem was that the rpm package didn’t exist. It may now. Here’s what I did to manually install it:

  1. Install the module
    1
    2
    3
    4
    5
    6
    7
    8
    
    root@puppet log]# scl enable ruby193 bash
    
    [root@puppet log]# gem install –ignore-dependencies foreman_host_rundeck
    Fetching: foreman_host_rundeck-0.0.2.gem (100%)
    Successfully installed foreman_host_rundeck-0.0.2
    1 gem installed
    Installing ri documentation for foreman_host_rundeck-0.0.2…
    Installing RDoc documentation for foreman_host_rundeck-0.0.2…
  2. Restart Foreman
    1
    2
    
    [root@puppet log]# service foreman restart
    Restarting foreman rails app:                              [  OK  ]

After performing the steps above, I was certain that we would be back in business. Of course I was wrong! I immediately noticed that one of my other RunDeck -> Foreman integrations was failing. I have a cronjob on Rundeck that queries the Foreman API, asking for a list of hosts that are members of a specific Foreman Host Group. The data returned is in JSON and in a structure that RunDeck likes. In this paticular usage case I had Foreman producing a dynamic list of our HAProxy hosts, so that RunDeck in turn knew of all the different hosts it needed to update when our HAProxy config was updated in git. At this point Foreman wasn’t returing anything. Attempts at manually running the API call from curl returned nothing.

Back over on the Foreman system, I checked the About->Plugins section, and sure enough I couldn’t find any reference for the foreman_host_rundeck plugin. What?

I took to reading more about Foreman plugins, and found that each plugin should have a listing in /usr/share/foreman/bundler.d. When I looked in that directory, I see:

1
2
3
4
5
6
7
8
9
[josh@puppet ~]$ ls -la /usr/share/foreman/bundler.d/
total 32
drwxr-xr-x  2 root root 4096 Dec 12 14:23 .
drwxr-xr-x 14 root root 4096 Dec 12 11:13 ..
-rw-r--r--  1 root root   82 Dec  2 02:20 fog.rb
-rw-r--r--  1 root root   23 Nov 18 01:13 foreman_bootdisk.rb
-rw-r--r--  1 root root   50 Nov 20 05:00 foreman_setup.rb
-rw-r--r--  1 root root   36 Dec  2 02:20 postgresql.rb
-rw-r--r--  1 root root   49 Dec  2 02:20 vmware.rb

Note that there isn’t anything for the foreman_host_rundeck plugin. Examining the structure of these plugin files, I decided to make my own for foreman_host_rundeck. I created a file foreman_host_rundeck.rb, with the following content:

1
gem 'foreman_host_rundeck'

After a quick restart of the foreman service, Foreman now lists the RunDeck plugin! A manual run of the previous curl command generated the expected results:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
[root@rundeck josh]# curl -k "https://haproxy:*************************@puppet.library.ucsb.edu/hosts?search=+hostgroup+%3D++haproxy&rundeck=true&format=yaml"
---
haproxy-dev-2.library.ucsb.edu:
  description: ''
  hostname: haproxy-dev-2.library.ucsb.edu
  nodename: haproxy-dev-2.library.ucsb.edu
  Environment: josh_dev
  osArch: x86_64
  osFamily: Redhat
  osName: CentOS
  osVersion: '6.6'
  tags:
  - class=accounts
  - class=bash
  - class=haproxy
  - class=iptables
  - class=ntp
  - class=puppet_agent
  - class=rsyslog
  - class=sshd_server
  - class=sudo
  username: rundeck
haproxy-module-testing.library.ucsb.edu:
  description: ''
  hostname: haproxy-module-testing.library.ucsb.edu
  nodename: haproxy-module-testing.library.ucsb.edu
  Environment: josh_dev
  osArch: x86_64
  osFamily: Redhat
  osName: CentOS
  osVersion: '6.6'
  tags:
  - class=accounts
  - class=bash
  - class=concat::setup
  - class=haproxy
  - class=iptables
  - class=ntp
  - class=puppet_agent
  - class=rsyslog
  - class=sshd_server
  - class=sudo
  username: rundeck

Hope this helps out any poor soul retain a couple hours of their life.