Third-party monitoring is good for a number of reasons, mainly being that your services can be checked from a number of locations, alleviating the burden of setting this up yourself, however, with it comes the possibility of false-positives. These either end up with someone getting called out of bed, or lots of emails (or both).
Pingdom is such an example but there are over 100 different IPs that the checks could come from. Multiply this by the number of data centers, and you have a LOT of IP addresses to manage.
If you have the luxury of a firewall that can read an RSS feed or something like that (i.e. MineMeld from Palo Alto) then this is easy enough to do, however, I am working with Cisco ASAs and there is nothing like this.
So, while searching for a solution, I decided to give Ansible a go. It was either that or write a bash script with a load of expect scripts.
To begin with, I used an Ubuntu 16.04 server, and installed python (sudo apt-get install python) and python-pip (sudo apt-get install python), then ran “sudo pip install ansible” to install Ansible.
First things first, let’s see how we can grab the IPs which, handily, are located at https://my.pingdom.com/probes/ipv4.
Script 1:
We need to specify a host here, so I am using the local host, the connection is local. To grab the IP addresses, we use the lookup command, passing it the URL and formatting it as a list. The script is called “fetch-pingdom.yml”.
# Grab Pingdom IPs - hosts: 127.0.0.1 connection: local become: yes tasks: - name: List Pingdom IPv4 addresses debug: msg="{{ lookup('url', 'https://my.pingdom.com/probes/ipv4', wantlist=True) }}"
This gets the result I wanted:
[email protected] ~/playbooks$ ansible-playbook -k fetch-pingdom.yml SSH password: PLAY [127.0.0.1] ************************************************************* TASK [Gathering Facts] ******************************************************* ok: [127.0.0.1] TASK [List Pingdom IPv4 addresses] ******************************************* ok: [127.0.0.1] => { "msg": [ "5.172.196.188", "5.178.78.77", "23.22.2.46", "23.83.129.219", "23.111.152.74", "23.111.159.174", "27.122.14.7", "37.252.231.50", "43.225.198.122", "43.229.84.12", "46.20.45.18", "46.165.195.139", "50.16.153.186", "50.22.90.227", <trimmed> "207.244.80.239", "208.64.28.194", "209.126.117.87", "209.126.120.29", "211.44.63.35" ] } PLAY RECAP ****************************************************************** 127.0.0.1 : ok=2 changed=0 unreachable=0 failed=0
Script 2:
For this next stage, we need to save the list to be used later on. Below, the IP addresses are held in a variable called PingdomProbeNodes:
# Grab Pingdom IPs - hosts: 127.0.0.1 connection: local become: yes vars: PingdomProbeNodes: "{{ lookup('url', 'https://my.pingdom.com/probes/ipv4', wantlist=True) }} " tasks: - name: List Pingdom IPv4 addresses debug: msg="The current probe addresses are {{ PingdomProbeNodes }}"
This gets what we need, slightly differently to the first version:
[email protected] ~/playbooks$ ansible-playbook -k fetch-pingdom.yml SSH password: PLAY [127.0.0.1] *********************************************************************************************************************************************************** TASK [Gathering Facts] ***************************************************************************************************************************************************** ok: [127.0.0.1] TASK [List Pingdom IPv4 addresses] ***************************************************************************************************************************************** ok: [127.0.0.1] => { "msg": "The current probe addresses are [u'5.172.196.188', u'5.178.78.77', u'23.22.2.46', u'23.83.129.219', u'23.111.152.74', u'23.111.159.174', u'27.122.14.7', u'37.252.231.50', u'43.225.198.122', u'43.229.84.12', u'46.20.45.18', u'46.165.195.139', u'50.16.153.186', u'50.22.90.227', u'50.23.28.35', u'52.0.204.16', u'52.24.42.103', u'52.48.244.35', u'52.52.34.158', u'52.52.95.213', u'52.52.118.192', u'52.57.132.90', u'52.59.46.112', u'52.59.147.246', u'52.62.12.49', u'52.63.142.2', u'52.63.164.147', u'52.63.167.55', u'52.67.148.55', u'52.73.209.122', u'52.89.43.70', u'52.197.31.124', u'52.197.224.235', u'52.198.25.184', u'52.201.3.199', u'52.209.34.226', u'52.209.186.226', u'52.210.232.124', u'54.68.48.199', u'54.70.202.58', u'54.94.206.111', u'64.120.6.122', u'64.237.49.203', u'64.237.55.3', u'66.165.229.130', u'66.165.233.234', u'69.59.28.19', u'70.32.40.2', u'72.46.140.106', u'72.46.153.26', u'76.72.167.90', u'76.72.167.154', u'76.72.172.208', u'76.164.194.74', u'76.164.234.106', u'76.164.234.170', u'82.103.136.16', u'82.103.139.165', u'83.170.113.210', u'85.17.156.11', u'85.17.156.76', u'85.93.93.123', u'85.93.93.124', u'85.93.93.133', u'89.163.146.247', u'89.163.242.206', u'94.247.174.83', u'95.141.32.46', u'95.211.198.87', u'95.211.217.68', u'96.47.225.18', u'103.47.211.210', u'104.129.24.154', u'104.129.30.18', u'108.62.115.226', u'109.123.101.103', u'138.219.43.186', u'159.8.146.132', u'162.218.67.34', u'168.1.92.58', u'169.51.2.22', u'169.51.80.85', u'173.248.147.18', u'173.254.206.242', u'174.34.156.130', u'174.34.162.242', u'174.34.224.167', u'175.45.132.20', u'178.255.152.2', u'178.255.153.2', u'178.255.154.2', u'178.255.155.2', u'179.50.12.212', u'184.75.208.210', u'184.75.209.18', u'184.75.210.90', u'184.75.210.226', u'184.75.214.66', u'185.39.146.214', u'185.39.146.215', u'185.70.76.23', u'185.93.3.92', u'185.152.65.167', u'185.180.12.65', u'185.246.208.82', u'188.138.40.20', u'188.172.252.34', u'199.87.228.66', u'201.33.21.5', u'207.244.80.239', u'208.64.28.194', u'209.126.117.87', u'209.126.120.29', u'211.44.63.35'] > /tmp/PingomProbeIPs.txt" } PLAY RECAP ***************************************************************************************************************************************************************** 127.0.0.1 : ok=2 changed=0 unreachable=0 failed=0 [email protected] ~/playbooks$
Script 3:
So far, so good, we actually need the list saved as a file, so that we can strip out the junk, and have it as a proper list.
To achieve this, I added the following to the end of the existing script:
- copy: content: "{{ lookup('url', 'https://my.pingdom.com/probes/ipv4', wantlist=True) }} " dest: "/tmp/PingdomProbeIPs.txt"
We do get the same results, but now we have the output in a text file:
[email protected] ~/playbooks$ cat /tmp/PingdomProbeIPs.txt ["5.172.196.188", "5.178.78.77", "23.22.2.46", "23.83.129.219", "23.111.152.74", "23.111.159.174", "27.122.14.7", "37.252.231.50", "43.225.198.122", "43.229.84.12", "46.20.45.18", "46.165.195.139", "50.16.153.186", "50.22.90.227", "50.23.28.35", "52.0.204.16", "52.24.42.103", "52.48.244.35", "52.52.34.158", "52.52.95.213", "52.52.118.192", "52.57.132.90", "52.59.46.112", "52.59.147.246", "52.62.12.49", "52.63.142.2", "52.63.164.147", "52.63.167.55", "52.67.148.55", "52.73.209.122", "52.89.43.70", "52.197.31.124", "52.197.224.235", "52.198.25.184", "52.201.3.199", "52.209.34.226", "52.209.186.226", "52.210.232.124", "54.68.48.199", "54.70.202.58", "54.94.206.111", "64.120.6.122", "64.237.49.203", "64.237.55.3", "66.165.229.130", "66.165.233.234", "69.59.28.19", "70.32.40.2", "72.46.140.106", "72.46.153.26", "76.72.167.90", "76.72.167.154", "76.72.172.208", "76.164.194.74", "76.164.234.106", "76.164.234.170", "82.103.136.16", "82.103.139.165", "83.170.113.210", "85.17.156.11", "85.17.156.76", "85.93.93.123", "85.93.93.124", "85.93.93.133", "89.163.146.247", "89.163.242.206", "94.247.174.83", "95.141.32.46", "95.211.198.87", "95.211.217.68", "96.47.225.18", "103.47.211.210", "104.129.24.154", "104.129.30.18", "108.62.115.226", "109.123.101.103", "138.219.43.186", "159.8.146.132", "162.218.67.34", "168.1.92.58", "169.51.2.22", "169.51.80.85", "173.248.147.18", "173.254.206.242", "174.34.156.130", "174.34.162.242", "174.34.224.167", "175.45.132.20", "178.255.152.2", "178.255.153.2", "178.255.154.2", "178.255.155.2", "179.50.12.212", "184.75.208.210", "184.75.209.18", "184.75.210.90", "184.75.210.226", "184.75.214.66", "185.39.146.214", "185.39.146.215", "185.70.76.23", "185.93.3.92", "185.152.65.167", "185.180.12.65", "185.246.208.82", "188.138.40.20", "188.172.252.34", "199.87.228.66", "201.33.21.5", "207.244.80.239", "208.64.28.194", "209.126.117.87", "209.126.120.29", "211.44.63.35"]
Now how do we get a nicely formatted list?
Script 4:
Actually just need the copy, and then some replaces:
Now the script becomes:
# Grab Pingdom IPs - hosts: 127.0.0.1 connection: local become: yes tasks: - copy: content: "{{ lookup('url', 'https://my.pingdom.com/probes/ipv4', wantlist=True) }} " dest: "/tmp/PingdomProbeIPs.txt" - replace: path: /tmp/PingdomProbeIPs.txt regexp: '", "' replace: "\n" - replace: path: /tmp/PingdomProbeIPs.txt regexp: '\[\"' replace: '' - replace: path: /tmp/PingdomProbeIPs.txt regexp: '\"\]' replace: "\n"
Much easier and the output is much cleaner. We get the list of IP addresses needed. Now we are ready to start with the ASA stuff!
Basic ASA config:
TestASA# sh run | i username username ansible password Password123 privilege 15 TestASA# TestASA# sh run int gi 0/0 ! interface GigabitEthernet0/0 nameif Inside security-level 100 ip address 192.168.1.81 255.255.255.0 TestASA#
Script 5:
The script is now renamed “UpdatePingdom.yml”. The hosts entry is now changed from the localhost to Cisco_ASAs. This is held in a file called hosts in the same directory as the UpdatePingdom.yml file. It looks like this:
[Cisco_ASAs] 192.168.3.170
We also have a secrets file, which looks like this:
[email protected] ~$ cat secrets.yml --- creds: username: ansible password: Password123 auth_pass: Password123 [email protected] ~$
The script is much longer now, so I will break it down into sections.
- hosts: Cisco_ASAs connection: local tasks: - name: GET CREDS include_vars: secrets.yml - name: DEFINE CONNECTION set_fact: connection: authorize: yes host: "{{ inventory_hostname }}" username: "{{ creds['username'] }}" password: "{{ creds['password'] }}" auth_pass: "{{ creds['auth_pass'] }}" timeout: 30
We are limiting our usage of this, or our application of it, to just the hosts defined under the Cisco_ASAs stanza of the hosts file.
We then define the start of our tasks and tell the script to use the variables defined in the secrets.yml file.
Next, we define the connection parameters, using the set fact command. We pass through the username and password, along with the auth_pass password (which in Cisco terminology is our enable password). I also set a timeout to thirty, but that was to solve a different issue (hint: you need to be on Ansible version 2.6).
The next part of the file is the copy and replace commands we saw earlier:
- copy: content: "{{ lookup('url', 'https://my.pingdom.com/probes/ipv4', wantlist=True) }} " dest: "/tmp/PingdomProbeIPs.txt" - replace: path: /tmp/PingdomProbeIPs.txt regexp: '", "' replace: "\n" - replace: path: /tmp/PingdomProbeIPs.txt regexp: '\[\"' replace: '' - replace: path: /tmp/PingdomProbeIPs.txt regexp: '\"\]' replace: "\n"
Lastly, we have the actual Cisco commands. We use the “asa_config” command, and specify the lines to add. This is where we need to pass all of our IPs from the PingdomProbeIPs.txt file. Thankfully, instead of writing these all out by hand, we can use the {{ item }} option (you can change “item” to whatever you want), and then pass the IPs through a loop using the “with_lines” command. We need to tell the ASA where the commands are going though, and we do this using the parent command – which also creates the object group for us if it is not already present.
Lastly, we use the “asa_command” to save the changes.
- asa_config: lines: - network-object host {{ item }} parents: ['object-group network Pingdom'] provider: "{{ connection }}" with_lines: cat /tmp/PingdomProbeIPs.txt register: result - debug: var=result - asa_command: commands: - wr mem provider: "{{ connection }}"
To run the script, we use the command: ansible-playbook UpdatePingdom.yml –i hosts
It’s probably not the cleanest of scripts, but then I have only been using Ansible since Friday, and I partied all weekend, so it’s not bad (if I do say so myself) considering I started it on Friday and finished it on Monday morning.
Ansible is surprisingly easy to learn, so it’s been fun.
Since then, I have written two more scripts, one of which I’ll post about tomorrow.