Elasticsearch Ansible Setup

[thepsalmist]

This PR introduces Ansible playbooks and roles for deploying ad managing an Elasticsearch cluster.
This builds off from the discussion on #366 and the draft at #367

[DavidTheProgrammer]

Looks good, I've left a few comments on the general structure and I have a couple of questions about the role we're using, is it something we've hand-rolled or it's available on Ansible Galaxy? It seems to be environment dependent with the vars files and that is setting off some flags for me as roles should not be environment dependent. Is there any reason we can't use a 3rd party installation modules for ES 8+ Like this one: https://galaxy.ansible.com/ui/standalone/roles/geerlingguy/elasticsearch/install/ ?

[DavidTheProgrammer]

This directory ideally should only contain variables shared by specific groups and since we don't have a group named "vault", I'd recommend we move this one level up to the inventory level. The rationale is that this is the vault used for this particular inventory.

[DavidTheProgrammer]

I have found the alternative directory layout located in the Ansible best practices that seperates the inventories based on environment to work quite well in the past. Reduces filters needed and allows us to actually use all group for things like agent installation that needs to happen on all hosts while only affecting specific environments.

https://docs.ansible.com/ansible/2.8/user_guide/playbooks_best_practices.html#alternative-directory-layout

[DavidTheProgrammer]

Can we default to staging? I think that's a safer default.

[DavidTheProgrammer]

Use include_role instead of the traditional roles: section

Following this article we should prefer include_roles instead of the roles section. I think it's okay.

https://www.ansiblejunky.com/blog/ansible-101-standards/#playbooks

[DavidTheProgrammer]

Can we move these to a standalone role and include it in this playbook instead of having the tasks written directly? This also comes from the recommendations in the Ansible 101 article.

https://www.ansiblejunky.com/blog/ansible-101-standards/#playbooks

[philbudne]

I started the ansible files. Regarding selection of a role to install ES: I surveyed the available roles, and I'm pretty sure the geerlingguy module was one I glanced at. I didn't keep notes on why I rejected it (the fact that the docs don't show ES 8.x support, and it requires a separate module for Java install probably didn't weigh in it's favor). I ended up choosing between two forks of the Elastic role that had been adapted for ES8, selecting the more lightly modified one, and forked it to make one or two changes necessary to get it to run, keeping it as a separate (mediacloud) repo because (a) it had it's own licence, and (b) I didn't want to get too deep into modifiying it. Some of my thoughts / principles: To keep the work compact, and avoid spreading stuff out into too many files: Keeping the node names ONLY in the YAML inventory file, with per-group settings in the same file, allowing testing by selecting an alternate inventory group. I regard ansible as a necessary evil (hateful in many ways, tho the best/only game in town). Past experience has shown me that data organization is key (being a data driven solution), and that it's easy to make a twisted mess that's hard to untangle (and noone wants to touch because it's so easy to break). Like in many things, I'm not a purist or true believer. My experience is that it's likely that any ansible file written a few years ago won't run as-it today, and I have no reason to believe the same won't be true of today's files in five years: I tried to avoid things that caused overt warnings, but I view "best practices" and "future proofing" as at best unnecessary, or at worst harmful to clarity and simplicity (making something that gets the job done, and future people can understand and adapt quickly) as opposed to maximally reusable by others. Trying to be a "hero" was an ABSOLUTE non-goal. I quickly realized that my install of statsd-agent.py SHOULD have been written as a sequence of tasks, rather than a slab of shell commands, AND that if I made it a role, it could be reused to install other tools in the same repo that could be used to rebuild other servers (ie; tarbell, the web server), *BUT* I didn't do either of those things; it simply isn't worth the investment of time (I find coding ANYTHING for ansible takes 3-5 times longer than it would to write a shell script, debugging to be painful, and documentation to be barely adequate). My concern with Xavier's work so far is that it increased the number of files and duplication of data, and that I might be the only one qualified to review it, and thus would have to wrap my head around it or accept it as-is. We DO NOT have the luxury of time, to have this be a multi-week incremental crawl towards a solution. In my mind, something that works today is good enough, and is (at least) superior to written documentation, because future people will at least know that it worked, and was used at some point in time. The one thing I would view as least desireable is any need for human intervention in the install/configuration process.