Recently I had to quickly test and deploy CentOS on 128 physical nodes, just to test hardware and that all currently "supported" CentOS releases could be installed quickly when needed. The interesting bit is that it was a completely new infra, without any traditional deployment setup in place, so obviously, as sysadmin, we directly think about pxe/kickstart, which is so trivial to setup. That was the first time I had to "play" with SeaMicro devices/chassis though, and so understanding how they work (the SeaMicro 15K fabric chassis, to be precise). One thing to note is that those seamicro chassis don't provide remote VGA/KVM feature (but who cares, as we'll automate the whole thing, right ? ) but they instead provide either cli (ssh) or rest api access to the management interface, so that you can quickly reset/reconfigure a node, changing vlan assignement, and so on.
It's not a secret that I like to use Ansible for ad-hoc tasks, and I thought that it would be (again) a good tool for that quick task. If you have used Ansible already, you know that you have to declare nodes and variables (not needed, but really useful) in the inventory (if you don't gather inventory from an external source). To configure my pxe setup (and so being able to reconfigure it when needed) I obviously needed to get mac addresses from all 64 nodes in each chassis, decide that hostnames will be n${slot-number}., etc .. (and yes in Seamicro slot 1 = 0/0, slot 2 = 1/0, and so on ...)
The following quick-and-dirty bash script let you do that quickly in 2 seconds (ssh into chassis, gather information, and fill some variables in my ansible host_vars/${hostname} file) :
1 2 3 4 5 6 7 8 | #!/bin/bash
ssh admin@hufty.ci.centos.org "enable ; show server summary | include Intel ; quit" | while read line ;
do
seamicrosrvid=\$(echo \$line |awk '{print \$1}')
slot=\$(echo \$seamicrosrvid| cut -f 1 -d '/')
id=\$(( \$slot + 1)); ip=\$id ; mac=\$(echo \$line |awk '{print \$3}')
echo -e "name: n${id}.hufty.ci.centos.org \nseamicro_chassis: hufty \nseamicro_srvid: $seamicrosrvid \nmac_address: $mac \nip: 172.19.3.$ip \ngateway: 172.19.3.254 \nnetmask: 255.255.252.0 \nnameserver: 172.19.0.12 \ncentos_dist: 6" > inventory/n${id}.hufty.ci.centos.org
done
|
Nice so we have all \~/ansible/hosts/host_vars/${inventory_hostname}
files in one go (I let you add ${inventory_hostname} in the
\~/ansible/hosts/hosts.cfg file with the same script, but modify to your
needs
For the next step, we assume that we already have dnsmasq installed on
the "head" node, and that we also have a httpd setup to provide the
kickstart to the nodes during installation.
So our basic ansible playbook looks like this :
---
- hosts: ci-nodes
sudo: True
gather_facts: False
vars:
deploy_node: admin.ci.centos.org
seamicro_user_login: admin
seamicro_user_pass: obviously-hidden-and-changed
seamicro_reset_body:
action: reset
using-pxe: "true"
username: "{{ seamicro_user_login }}"
password: "{{ seamicro_user_pass }}"
tasks:
- name: Generate kickstart file[s] for Seamicro node[s]
template: src=../templates/kickstarts/ci-centos-{{ centos_dist}}-ks.j2 dest=/var/www/html/ks/{{ inventory_hostname }}-ks.cfg mode=0755
delegate_to: "{{ deploy_node }}"
- name: Adding the entry in DNS (dnsmasq)
lineinfile: dest=/etc/hosts regexp="\^{{ ip }} {{ inventory_hostname }}" line="{{ ip }} {{ inventory_hostname }}"
delegate_to: "{{ deploy_node }}"
notify: reload_dnsmasq
- name: Adding the DHCP entry in dnsmasq
template: src=../templates/dnsmasq-dhcp.j2 dest=/etc/dnsmasq.d/{{ inventory_hostname }}.conf
delegate_to: "{{ deploy_node }}"
register: dhcpdnsmasq
- name: Reloading dnsmasq configuration
service: name=dnsmasq state=restarted
run_once: true
when: dhcpdnsmasq|changed
delegate_to: "{{ deploy_node }}"
- name: Generating the tftp configuration boot file
template: src=../templates/pxeboot-ci dest=/var/lib/tftpboot/pxelinux.cfg/01-{{ mac_address | lower | replace(":","-") }} mode=0755
delegate_to: "{{ deploy_node }}"
- name: Resetting the Seamicro node[s]
uri: url=https://{{ seamicro_chassis }}.ci.centos.org/v2.0/server/{{ seamicro_srvid }}
method=POST
HEADER_Content-Type="application/json"
body='{{ seamicro_reset_body | to_json }}'
timeout=60
delegate_to: "{{ deploy_node }}"
- name: Waiting for Seamicro node[s] to be available through ssh ...
action: wait_for port=22 host={{ inventory_hostname }} timeout=1200
delegate_to: "{{ deploy_node }}"
handlers:
- name: reload_dnsmasq
service: name=dnsmasq state=reloaded
The first thing to notice is that you can use Ansible to provision nodes
that aren't already running : people think than ansible is just to
interact with already provisioned and running nodes, but by providing
useful informations in the inventory, and by delegating actions, we can
already start "managing" those yet-to-come nodes.
All the templates used in that playbook are really basic ones, so
nothing "rocket science". For example the only diff for the kickstart.j2
template is that we inject ansible variables (for network and storage) :
network --bootproto=static --device=eth0 --gateway={{ gateway }}
--ip={{ ip }} --nameserver={{ nameserver }} --netmask={{ netmask }}
--ipv6=auto --activate
network --hostname={{ inventory_hostname }}
\<snip\>
part /boot --fstype="ext4" --ondisk=sda --size=500
part pv.14 --fstype="lvmpv" --ondisk=sda --size=10000 --grow
volgroup vg_{{ inventory_hostname_short }} --pesize=4096 pv.14
logvol /home --fstype="xfs" --size=2412 --name=home --vgname=vg_{{
inventory_hostname_short }} --grow --maxsize=100000
logvol / --fstype="xfs" --size=8200 --name=root --vgname=vg_{{
inventory_hostname_short }} --grow --maxsize=1000000
logvol swap --fstype="swap" --size=2136 --name=swap --vgname=vg_{{
inventory_hostname_short }}
\<snip\>
The dhcp step isn't mandatory, but at least in that subnet we only allow dhcp to "already known" mac address, retrieved from the ansible inventory (and previously fetched directly from the seamicro chassis) :
# {{ name }} ip assignement
dhcp-host={{ mac_address }},{{ ip }}
Same thing for the pxelinux tftp config file :
SERIAL 0 9600
DEFAULT text
PROMPT 0
TIMEOUT 50
TOTALTIMEOUT 6000
ONTIMEOUT {{ inventory_hostname }}-deploy
LABEL local
MENU LABEL (local)
MENU DEFAULT
LOCALBOOT 0
LABEL {{ inventory_hostname}}-deploy
kernel CentOS/{{ centos_dist }}/{{ centos_arch}}/vmlinuz
MENU LABEL CentOS {{ centos_dist }} {{ centos_arch }}- CI Kickstart
for {{ inventory_hostname }}
{% if centos_dist == 7 -%}
append initrd=CentOS/7/{{ centos_arch }}/initrd.img net.ifnames=0 biosdevname=0 ip=eth0:dhcp inst.ks=http://admin.ci.centos.org/ks/{{ inventory_hostname }}-ks.cfg console=ttyS0,9600n8
{% else -%}
append initrd=CentOS/{{ centos_dist }}/{{ centos_arch }}/initrd.img ksdevice=eth0 ip=dhcp ks=http://admin.ci.centos.org/ks/{{ inventory_hostname }}-ks.cfg console=ttyS0,9600n8
{% endif %}
The interesting part is the one on which I needed to spend more time : as said, it was the first time I had to play with SeaMicro hardware, so I had to dive into documentation (which I *always* do, RTFM FTW !) and understand how to use their Rest API but once done, it was a breeze. Ansible by default doesn't provide a native resource for Seamicro, but that's why Rest exists, right and thanfully, Ansible has a native URI module, which we use here . The only thing on which I had to spend more time was to understand how to properly construct the body, but declaring in the yaml file as a variable/list and then converting it on the fly to json (with the magical body='{{ seamicro_reset_body | to_json }}' ) was the way to go and is so self-explained when read now.
And here we go, calling that ansible playbook and suddenly 128 physical machines were being installed (and reinstalled with different CentOS versions - 5,6,7 - and arches i386,x86_64)
Hope this helps if you have to interact with Seamicro chassis from within an ansible playbook too