Initial support for Vagrant

During development it's not always possible or convenient to
have access to an OpenStack cloud to host the virtual machines
for the infrastructure.

This change adds a Vagrantfile which utilizes libvirt as the
provisioning backend for deploying the base infrastructure
as managed via Vagrant.

These changes have been tested against both a Vagrant deployment with
libvirt, and against the existing OpenStack deployment system.

.gitignore

@@ -3,5 +3,6 @@ hosts/
 !hosts/localhost
 roles/
 inventory
-files/jobs/
-files/nfv_jobs_config/
+files/
+!files/filebeat.yml
+.vagrant/

README.md

@@ -6,18 +6,36 @@ Deploy a continuous integration reference architecture with Jenkins to test
 OpenStack with [TripleO
 Quickstart](https://github.com/openstack/tripleo-quickstart).
 
-## Requirements
+# Requirements
 
-You'll need to install the `shade` dependency so that you can interact with
-OpenStack (assuming you are deploying to an OpenStack cloud).
-
-    pip install --user shade
+There are two ways to install CIRA. You deploy locally into a development
+environment using Vagrant, or you can deploy to an OpenStack instance. Below
+you will find the list of requirements for each of the deployment scenarios.
 
 For Ansible, several roles are required, and you can install them as follows:
 
     ansible-galaxy install -r requirements.yml
 
-## Setup OpenStack Connection
+## Vagrant
+
+Deployment to Vagrant should be straight forward. The only real dependency is
+Vagrant itself, along with whatever provider backend you wish to utilize. Our
+preferred provider is libvirt (KVM). In order to use Vagrant with the libvirt
+provider, you'll need to install a new provider plugin.
+
+    vagrant plugin install vagrant-libvirt
+
+Additional information about other dependencies required by vagrant-libvirt are
+available at https://github.com/vagrant-libvirt/vagrant-libvirt
+
+## OpenStack
+
+You'll need to install the `shade` dependency so that you can interact with
+OpenStack (assuming you are deploying to an OpenStack cloud).
+
+    pip install --user shade
+
+### Setup OpenStack Connection
 
 If you're going to install to an OpenStack cloud, you'll need to configure a
 cloud to connect to. You can do this by creating the `~/.config/openstack/`
@@ -32,11 +50,17 @@ that directory (adjust to your own cloud connection):
                 password: cloud_pass
                 project_name: "My Cloud Project"
 
-## Overrides / Private Info
+# Overrides / Private Info
 
 There may be some variables you don't want to expose into a Git repo. You can
-store those in the `~/.ansible/vars/cira_vars.yml` file. For example, the
-following variables are being utilized by the author:
+store those in the `~/.ansible/vars/cira_vars.yml` file.
+
+> **NOTE**: You *must* create a `~/.ansible/vars/cira_vars.yml` file, even if
+> it is blank. This file is loaded via `var_files` directives in Ansible and
+> your deployment will fail if the file doesn't exist.
+
+The following list of options are required when deploying to an OpenStack
+cloud:
 
 **Cloud Configuration**
 * cloud_name_prefix
@@ -47,17 +71,20 @@ following variables are being utilized by the author:
 * cloud_flavor
 * cloud_key_name
 
-**Jenkins Job Builder Configuration**
-* jenkins_job_builder_git_jobs_src
-* jenkins_job_config_git_src
-* jenkins_job_builder_config_jenkins_user
-* jenkins_job_builder_config_jenkins_password
+The `jenkins_scp_sites` variable is required when you need to copy
+configuration files off the slave to the master. Note that the hostname is
+relative to the master (in this case, files are copied off the slave *into* the
+master node, since that's where the SCP command is run).
 
 **SCP Site Configuration**
 
     jenkins_scp_sites:
       - hostname: 127.0.0.1
-        path: "{{ jenkins_master_results_directory  }}"
+        path: "{{ jenkins_master_results_directory }}"
+
+When adding slaves, you would do so by creating a new file in the `hosts/`
+directory. For example you would create a `hosts/slaves` file and add your
+Jenkins slaves via the `[jenkins_slave]` and `[jenkins_slave:vars]` headers.
 
 **Jenkins Slave Configuration**
 * slave_name
@@ -68,7 +95,7 @@ following variables are being utilized by the author:
 * slave_credentialsId
 * slave_label
 
-### Example Override Variable File
+## Example Override Variable File
 Many of the values can be found in your OpenStack RC file, which can typically
 be found in the _Access & Security_ section of the Horizon dashboard.
 
@@ -79,20 +106,35 @@ be found in the _Access & Security_ section of the Horizon dashboard.
     cloud_image: c0a97bbd-0cdd-4ed1-b6c1-052123456789    # unique image ID
     cloud_flavor: m1.medium
     cloud_key_name: my_pub_key                 # name of your keypair
-
+ 
     jenkins_job_builder_git_jobs_src: gitserver.tld:leifmadsen/nfv-jenkins-jobs.git   # branched from upstream for customization purposes
     jenkins_job_config_git_src: gitserver.tld:nfvpe/nfv-job-configs.git
     jenkins_job_builder_config_jenkins_user: admin       # default username
     jenkins_job_builder_config_jenkins_password: admin   # default password
-
+ 
     # Can only specify a single site to SCP files to at the end of the run.
     jenkins_scp_sites:
       - hostname: 127.0.0.1
         path: "{{ jenkins_master_results_directory }}"   # defined in vars/main.yml
 
-## Deployment
+# Deployment
 
-### Base Deployment
+Deployment can be done via two methods: OpenStack cloud, or Vagrant development
+environment.
+
+## Base Deployment (Vagrant)
+
+Deploying into a Vagrant development environment should be as simple as
+running:
+
+    vagrant up
+
+This will deploy all the virtual machines and apply the `site.yml` Ansible
+configuration to the virtual machines. The deployment uses the built in default
+networking configuration that Vagrant instantiates. At the end of the run, the
+web interface addresses for Jenkins and Kibana will be displayed.
+
+## Base Deployment (OpenStack)
 
 You may need to modify the `host_vars/localhost` file to adjust the
 `security_group` names, as the playbook does not currently create security
@@ -108,6 +150,10 @@ security groups, and opened the corresponding ports:
 * web_ports
   * `TCP: 80, 443`
 
+> **NOTE**: The security groups are only relevant for OpenStack cloud
+> deployments. There are no firewall rules managed by CIRA within a Vagrant
+> deployment.
+
 The base set of four VMs created for the CI components in OpenStack are listed
 as follows (as defined in `host_vars/localhost`):
 

Vagrantfile

@@ -0,0 +1,52 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+Vagrant.configure(2) do |config|
+  config.vm.define :jenkins_master do |jenkins_master|
+    jenkins_master.vm.box = "centos/7"
+    jenkins_master.vm.provider :libvirt do |domain|
+      domain.machine_arch = 'x86_64'
+      domain.cpu_mode = 'host-passthrough'
+      domain.memory = "2048"
+    end
+  end
+
+  config.vm.define :elasticsearch do |elasticsearch|
+    elasticsearch.vm.box = "centos/7"
+    elasticsearch.vm.provider :libvirt do |domain|
+      domain.machine_arch = 'x86_64'
+      domain.cpu_mode = 'host-passthrough'
+      domain.memory = "1024"
+    end
+  end
+
+  config.vm.define :logstash do |logstash|
+    logstash.vm.box = "centos/7"
+    logstash.vm.provider :libvirt do |domain|
+      domain.machine_arch = 'x86_64'
+      domain.cpu_mode = 'host-passthrough'
+      domain.memory = "1024"
+    end
+  end
+
+  config.vm.define :kibana do |kibana|
+    kibana.vm.box = "centos/7"
+    kibana.vm.provider :libvirt do |domain|
+      domain.machine_arch = 'x86_64'
+      domain.cpu_mode = 'host-passthrough'
+      domain.memory = "2048"
+    end
+  end
+
+  config.vm.provision :ansible do |ansible|
+    ansible.extra_vars = {
+      use_openstack_deploy: false,
+      vars_files_relative: "../../../.."    # this sets the relative path from
+                                            # from the inventory file to the
+                                            # vars/ directory.
+    }
+    ansible.limit = "all"
+    ansible.skip_tags = "jenkins_slave"
+    ansible.playbook = 'site.yml'
+  end
+end

ansible.cfg

@@ -1,11 +1,14 @@
 [defaults]
 roles_path = ./roles
-gathering = smart
-fact_caching = jsonfile
-fact_caching_connection = ~/.ansible/cachedir
-fact_caching_timeout = 86400
-host_key_checking = False
 inventory = ./hosts/
+host_key_checking = False
+
+# NOTE: You can enable this to speed up deployments, but note with a teardown /
+#       spin-up that you may run into outdated facts.
+#gathering = smart
+#fact_caching = jsonfile
+#fact_caching_connection = ~/.ansible/cachedir
+#fact_caching_timeout = 86400
 
 [ssh_connection]
 ssh_args = -o ControlMaster=auto -o ControlPersist=300s -o ForwardAgent=yes

elk.yml

@@ -1,44 +1,52 @@
 # vim: ft=ansible
 ---
 - hosts: elasticsearch
+  become: true
   tags:
     - elasticsearch
     - elk
 
   vars_files:
-    - "{{ inventory_dir  }}/../vars/main.yml"
+    - "{{ inventory_dir  }}/{{ vars_files_relative }}/vars/main.yml"
 
   roles:
     - { role: 'geerlingguy.repo-epel' }
     - { role: 'leifmadsen.elasticsearch' }
 
 - hosts: logstash
+  become: true
   tags:
     - logstash
     - elk
 
   vars_files:
-    - "{{ inventory_dir  }}/../vars/main.yml"
+    - "{{ inventory_dir  }}/{{ vars_files_relative }}/vars/main.yml"
 
   roles:
     - { role: 'geerlingguy.repo-epel' }
     - { role: 'geerlingguy.java' }
     - { role: 'leifmadsen.logstash' }
 
 - hosts: kibana
+  become: true
   tags:
     - kibana
     - elk
 
   vars_files:
-    - "{{ inventory_dir  }}/../vars/main.yml"
+    - "{{ inventory_dir  }}/{{ vars_files_relative }}/vars/main.yml"
 
   roles:
     - { role: 'geerlingguy.repo-epel' }
     - { role: 'geerlingguy.nginx' }
     - { role: 'leifmadsen.kibana-4' }
 
   post_tasks:
+    - name: Ensure libsemanage-python is installed
+      yum:
+        name: libsemanage-python
+        state: present
+
     - name: Validate SELinux is enabled
       selinux:
         policy: targeted

filebeat.yml

@@ -3,12 +3,13 @@
 
 # Deploy Jenkins Master
 - hosts: jenkins_master
+  become: true
   tags:
     - jenkins_master
     - logging
 
   vars_files:
-    - "{{ inventory_dir  }}/../vars/main.yml"
+    - "{{ inventory_dir  }}/{{ vars_files_relative  }}/vars/main.yml"
     - ~/.ansible/vars/cira_vars.yml
 
   roles:

group_vars/all

@@ -0,0 +1,2 @@
+# set the relative path from the inventory file to the vars/ directory.
+vars_files_relative: ".."

jenkins.yml

@@ -4,15 +4,17 @@
 # Deploy Jenkins Master
 ########################
 - hosts: jenkins_master
+  become: true
   tags:
     - jenkins_master
 
   vars_files:
-    - "{{ inventory_dir  }}/../vars/main.yml"
+    - "{{ inventory_dir }}/{{ vars_files_relative }}/vars/main.yml"
     - ~/.ansible/vars/cira_vars.yml
 
   roles:
     - { role: 'franklinkim.sudo' }
+    - { role: 'geerlingguy.repo-epel' }
     - { role: 'geerlingguy.nginx' }
     - { role: 'leifmadsen.jenkins' }
 
@@ -30,6 +32,11 @@
         jenkins_admin_username: "{{ jenkins_admin_username }}"
         jenkins_admin_password: "{{ jenkins_admin_password }}"
 
+    - name: Install python-pip
+      yum:
+        name: python-pip
+        state: present
+
   post_tasks:
     # Operating system configuration and setup
     - name: Add jenkins user to wheel group
@@ -52,6 +59,11 @@
         password: "{{ jenkins_admin_password |password_hash('sha512')}}"
         shell: /bin/bash
 
+    - name: Ensure libsemanage-python is installed
+      yum:
+        name: libsemanage-python
+        state: present
+
     - name: Validate SELinux is enabled
       selinux:
         policy: targeted
@@ -119,7 +131,7 @@
     - { role: 'geerlingguy.java' }
 
   vars_files:
-    - "{{ inventory_dir  }}/../vars/main.yml"
+    - "{{ inventory_dir  }}/{{ vars_files_relative }}/vars/main.yml"
     - ~/.ansible/vars/cira_vars.yml
 
   vars: