added a bunch more stuff

Author: dunkOnIT

README.md

@@ -30,6 +30,10 @@ If changes aren't showing up, please:
 
 ## FAQ
 
+### Can I make draft pages?
+
+You can! Just prefix the page name with a `_`, and Jekyll will ignore it. Remove the `_` when it is ready to publish.
+
 ### How do you edit the left navigation bar? 
 
 If you're familiar with Jekyll, we do not use Collections to organize the content. JustTheDocs provides navigation tools which we make use of. 

TODO.md

@@ -0,0 +1,16 @@
+---
+title: Docs TODO
+layout: default
+nav_order: 5
+---
+
+## TODO
+- [ ] Add a "DevOps" section under WST processes for deployments and related stuff
+- [ ] figure out how to add/link to images, and document that
+- [x] add WARNING, INFO and OPTION callouts
+- [ ] Work through all existing documentation and add it somewhere
+- [ ] work through all pages, and either finalize or convert to drafts
+- [ ] add links etc to index.md
+- [ ] finish off and publish draft pages
+
+## IDEAS / THOUGHTS

_config.yml

@@ -12,3 +12,15 @@ aux_links:
 nav_external_links:
   - title: API 
     url: https://wca-rest-api.robiningelbrecht.be/
+
+
+callouts:
+  warning:
+    title: Warning
+    color: red
+  info:
+    title: Info
+    color: blue
+  help:
+    title: Help Wanted
+    color: yellow

contributing/contributor_guidelines.md renamed to contributing/_contributor_guidelines.md

@@ -6,3 +6,8 @@ layout: default
 
 This is where we'll tell beginners how to contribute
 
+
+# TODO
+- add development tips and tricks
+    - byebug
+    - rails console

contributing/Beginner's Guide.md renamed to contributing/detailed_setup_guide.md

@@ -5,4 +5,37 @@ layout: default
 nav_order: 1
 ---
 
-This is where the information for quickly setting up the dev environment will go.
+{: .info}
+> This assumes you are an experienced developer with the necessary dependencies - refer to [Detailed Setup Guide](./detailed_setup_guide) if you need more information.
+
+There are 3 steps to run the WCA website locally with docker[^1]:
+
+## (1) GET THE SERVER RUNNING
+1. Clone and navigate to the repo
+2. Run using docker[^2] with `docker-compose up` (or `docker compose up`, depending on your docker compose config)
+
+Now the website will run locally. You have some options on how to set up the database and run tests.
+
+## (2) DATABASE SETUP
+There are two options here - a "test" database with generated data, or the production database (with sensitive information removed).
+1. **Test database**: Quick to set up, but may not mirror real-world data well. `bin/rake db:reset`
+2. **Exported database**: Takes 60-90 minutes to set up, but uses actual data from the live website. `bin/rake db:load:development`
+
+## (3) RUNNING TESTS
+Two options - running while your docker server is running, or running tests independently. 
+
+### **RUNNING INDEPENDENTLY**
+```
+docker-compose exec wca_on_rails bash -c "RAILS_ENV=test bin/rake db:reset && RAILS_ENV=test bin/rake assets:precompile && bin/rspec"
+```
+
+### **RUNNING WHILE SERVER IS LIVE**
+With your docker server running:
+1. Connect to the `wca_on_rails` container using `docker exec -it {container_id} bash`
+    1. _Use `docker ps` to list containers and their id's_
+2. Run `bin/rspec` from inside the container's terminal
+
+----
+
+[^1]: The website can also be run with `rails server` - see [Running with Rails](/guides/running_with_rails)
+[^2]: If you're using Visual Studio Code to develop, you can [attach it to the Docker container](https://code.visualstudio.com/docs/remote/containers) so that your extensions can take advantage of the Ruby environment, and so the terminal runs from inside the container

contributing/quickstart.md

@@ -0,0 +1,50 @@
+---
+title: Running Server with Rails
+layout: default
+parent: Guides
+---
+
+{: .help}
+> After Monolith Improvements, we should consider whether to deprecate this guide. If we keep it, this page needs some formatting cleanup. 
+
+### Run directly with Ruby (lightweight, but it involves manual installation which happens at your own risk and we won't be able to assist with detailed troubleshooting)
+
+- Ensure you have the correct [Ruby version](./.ruby-version) installed. We recommend using a Ruby version manager like [rvm](https://rvm.io/rvm/install) or [rbenv](https://github.com/rbenv/rbenv). They should both read the `.ruby-version` file to use the correct version (`rvm current` or `rbenv version` to confirm).
+- Ensure [Bundler 2](https://bundler.io/v2.0/guides/bundler_2_upgrade.html) is installed
+  - To update from bundler 1:
+    ```
+    gem update --system
+    bundle update --bundler
+    ```
+  - Or, if you haven't installed bundler previously:
+    ```
+    gem update --system
+    gem install bundler
+    ```
+- Set up git pre-commit hook. Optional, but very useful.
+  ```shell
+  (cd WcaOnRails; bundle install && bin/yarn && bundle exec overcommit --install)
+  ```
+  If some changes are made to this hook, you will have to update it running this command from the repository's root directory: `BUNDLE_GEMFILE=WcaOnRails/Gemfile bundle exec overcommit --sign`.
+- Install [MySQL 8.0](https://dev.mysql.com/doc/refman/8.0/en/linux-installation.html), and set it up with a user with username "root" with an empty password.
+  If it poses problems, try the following:
+  ```shell
+  # Run MySQL CLI as administrator and set an empty password for the root user:
+  sudo mysql -u root
+  ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY '';
+  ```
+- Install dependencies and load development database.
+  1. `cd WcaOnRails/`
+  2. [Install Node.js](https://nodejs.org/en/) and [yarn](https://yarnpkg.com/en/docs/install), we need them for our javascript assets.
+  Feel free to take a look at our [chef recipe](https://github.com/thewca/worldcubeassociation.org/blob/master/chef/site-cookbooks/wca/recipes/default.rb#L6-L23)
+  for the accurate versions we use and how we install them.
+  Please note that other versions may work, but it is not guaranteed.
+  3. Install `libyaml` using your package manager of choice, eg. [Mac](https://formulae.brew.sh/formula/libyaml) or [Ubuntu](https://launchpad.net/ubuntu/+source/libyaml)
+  4. `bundle install && bin/yarn`
+  5. `bin/rake db:load:development` - Download and import the [developer's database export](https://github.com/thewca/worldcubeassociation.org/wiki/Developer-database-export). Depending on your computer it may take a long time. Alternatively you can run `bin/rake db:reset` which will create the database and seed it with random data (it's way faster, but less representative of our website content).
+  6. `bin/rails server` - Run rails. The server will be accessible at localhost:3000
+- Run tests.
+  1. `RAILS_ENV=test bin/rake db:reset` - Set up test database.
+  2. `RAILS_ENV=test bin/rake assets:precompile` - Compile some assets needed for tests to run.
+  3. `bin/rspec` - Run tests.
+- [Mailcatcher](http://mailcatcher.me/) is a good tool for catching emails in development.

guides/installing_docker.md

@@ -6,8 +6,8 @@ nav_order: 1
 
 ## Quick Links
 
-- Development Quickstart
-- Beginner's Guide to Contributing
+- Quickstart
+- Detailed Guide to Contributing
 - Current Development Vision
 
 ## Contributing to the Docs

guides/running_with_rails.md

@@ -0,0 +1,42 @@
+---
+title: Building Regulations Locally
+layout: default
+parent: Wiki Imports
+---
+
+This page aims to guide you through the process of building the WCA Regulations and Guidelines, which is something that you may want to do when you first clone the website repository.
+
+# Requirements
+The requirements to build the Regulations locally are pretty small, and can be found in the chef recipes for the [regulations](https://github.com/thewca/worldcubeassociation.org/blob/a604f5a600dc52943101a00e7a3d2e24c086eecd/chef/site-cookbooks/wca/recipes/regulations.rb), but here is a commented step-by-step setup:
+
+The following system packages are needed:
+`git`, `python-pip`, `fonts-unfonts-core`, `fonts-wqy-microhei`, `fonts-ipafont`, `lmodern`
+
+I'm fairly confident the first two are likely to be already installed on your system, and the last four are needed to build pdfs for the CJK translations of the Regulations.
+Please note that the name may differ a bit depending on your Linux distribution.
+
+The html/pdf/json generation is done by the [WCA Regulations Compiler](https://github.com/thewca/wca-regulations-compiler), which can be installed through `pip`:
+
+```
+pip install wrc --upgrade
+```
+
+To generate pdf, `wrc` relies on a tool named `wkhtmltopdf`, so you need to get it too. We use a precompiled binary that can be found [here](https://wkhtmltopdf.org/downloads.html).
+The following two lines will get it for you and install it to `/usr/local`:
+```
+wget https://github.com/wkhtmltopdf/wkhtmltopdf/releases/download/0.12.4/wkhtmltox-0.12.4_linux-generic-amd64.tar.xz -O wkhtml.tar.xz
+tar -xf wkhtml.tar.xz --strip-components=1 -C /usr/local
+```
+
+# How to build
+
+From the root of the repository, just run:
+```
+./scripts/deploy.sh rebuild_regs
+```
+
+The script will detect if anything needs to be (re)built and do it.
+On a first run it will build everything that lives under the `/regulations` route on the website, which includes the WCA Regulations and Guidelines, their translations, the regulations history, the scramble programs, and the json used by omnisearch to search in the Regulations.
+
+
+

index.md

@@ -0,0 +1,35 @@
+---
+title: Caching and DB Optimizations
+layout: default
+parent: Wiki Imports
+---
+
+This is a list of things we are currently caching or using the Read replica for instead of using the primary database:
+
+# HTML Request Caches
+These cache full or partial HTML requests or json objects using the built in Rails Cache
+
+## API
+- /v0/records: json, no expiration date set
+
+## /results
+- /results/records: HTML fragment, no expiration date set
+- /results/rankings: HTML fragment, no expiration date set
+
+## /competitions
+- public wcif: json is cached for 5 minutes
+- by_person: HTML fragment, no expiration date set
+
+## /persons
+- _results_by_event: HTML fragment, no expiration date set
+
+## /registrations
+- index_html: HTML fragment, no expiration date set. This works by using "users updated last" as part of the cache key
+
+# Model Caches
+
+## Cachable
+This is a module which caches SQL query results by saving them in a class variable, this is used for Continent, Country, Event, Format, Round_type and Team.
+
+## CachedResult
+This is a Class which caches a SQL Query result for Results used in /results/records and /results/rankings

wiki_imports/Building-Regulations-locally.md

@@ -0,0 +1,30 @@
+---
+title: Infrastructure Costs
+layout: default
+parent: Wiki Imports
+---
+
+Current expenses:
+
+- **AWS** *
+- **Tarsnap**
+- **Google Domains**
+
+The non-AWS expenses are all pretty minor (they all add up to less than $50 per year) expenses.
+
+*We could potentially save some money by getting a reserved instance (instead of on demand), but that does lock us into that particular instance for one or three years. We could also investigate other cloud service providers (but we'd need to find a good alternative to Amazon SES). For example, DigitalOcean might actually be cheaper for our purposes.
+
+## generating cost reports
+
+go to https://console.aws.amazon.com/billing/home?region=us-west-2#/costexplorer
+and click Launch Cost Explorer.
+
+On the advanced options > include costs related to uncheck "Credits".
+
+The rest behaves as you would expect 
+
+some example decent configs are 
+ - under time range put in some forecast
+![](http://i.imgur.com/5q1zDRR.png)
+ - grouping costs by instance type
+![](http://i.imgur.com/yC1GIPA.png)

wiki_imports/Caching-and-other-Database-Optimizations.md

@@ -0,0 +1,27 @@
+---
+title: Dependabot PR's
+layout: default
+parent: Wiki Imports
+---
+
+
+For handling Dependabot PRs, we generally rely on our test suite. The exact procedure depends on the kind of update, but the tests are always the first and most important indicator for merging.
+
+There is usually a changelog contained in the PR description, to help you decide whether to go ahead with the merge. For more concrete rules, refer to the listing below.
+
+## Bugfix version bumps
+For example, `1.1.3` --> `1.1.4` (last digit)
+
+Merge these at will if the test suite checks out.
+
+## Minor version bumps
+For example, `6.0.4` --> `6.1.0` (middle digit)
+
+Merge these at will if the test suite checks out.
+Exception: If it's a crucial part of our infrastructure like `rails` or `typescript` or similar, keep an eye on the changelog.
+
+## Major version bumps
+For example, `3.2.1` --> `4.0.0` (first digit)
+
+You should skim the changelog for any breaking changes that could be relevant.
+If it's a crucial part of our infrastructure (see above), get a second pair of eyes to review.

wiki_imports/Costs.md

@@ -0,0 +1,30 @@
+---
+title: Developer Database Export
+layout: default
+parent: Wiki Imports
+---
+
+{: .help}
+> This can be organized under a more general "Accessing WCA Data" guide
+
+## Are you a third party developer?
+
+We maintain two database exports:
+
+1. The well known public database export: https://www.worldcubeassociation.org/export/results
+2. The not so well known developer database export: https://www.worldcubeassociation.org/wst/wca-developer-database-dump.zip
+
+The "public database export" is used by tons of software, and is great if you want to run statistics against all the results ever achieved in competition. We try not to change it, and consequently it does not have some of the new interesting data that we've started tracking our database over the years.
+
+The "developer database export" confusingly enough, is **also** publicly accessible. It is a production database dump with sensitive data (birthdays, emails, etc) removed. The schema should match [exactly what's in master's `structure.sql`](https://github.com/thewca/worldcubeassociation.org/blob/master/WcaOnRails/db/structure.sql), so naturally, its schema changes with every database migration. This dump is used by WCA Software Team members for website development purposes, and third party developers who want to do interesting data analyses not possible with the public database export (for an example, see @jonatanklosko's awesome [wca_statistics](https://github.com/jonatanklosko/wca_statistics) project).
+
+## Are you investigating bugs in the developer export?
+
+See [lib/database_dumper.rb](https://github.com/thewca/worldcubeassociation.org/blob/master/WcaOnRails/lib/database_dumper.rb) and the [relevant test](https://github.com/thewca/worldcubeassociation.org/blob/master/WcaOnRails/spec/lib/database_dumper_spec.rb) and the [corresponding rake file](https://github.com/thewca/worldcubeassociation.org/blob/master/WcaOnRails/lib/tasks/db.rake).
+
+There are two commands to deal with the developer database export:
+
+1. `bin/rake db:dump:development` - Generates a dump of our database with sensitive information stripped, safe for public viewing.
+2. `bin/rake db:load:development` - Download and import the publicly accessible database dump from the production server.
+
+`db:dump:development` automatically [runs every 3 days](https://github.com/thewca/worldcubeassociation.org/blob/891553f59cc9f2e66f87bba03302c2333f3a285d/WcaOnRails/app/jobs/dump_developer_database.rb#L9) in production.

wiki_imports/Dependabot-PRs.md

@@ -0,0 +1,18 @@
+---
+title: Emails
+layout: default
+parent: Wiki Imports
+---
+
+We use Google GSuite Groups for our mailing lists. This is way better than free Google Groups, because it gives us an API to automate group membership. It also saves us some hassle setting up redirects. Steps involved in creating a group:
+
+1. Create the group on the GSuite admin panel. See how to [Create a group](https://docs.google.com/document/d/1tFuNMyUcOwsYiGxN8OLG2w41AsLGZpKN-_5GcRGCr2g/edit)
+2. Update [SyncMailingListsJob].(https://github.com/thewca/worldcubeassociation.org/blob/master/WcaOnRails/app/jobs/sync_mailing_lists_job.rb) if you want membership of the group to be automated. You can see an example of this in [PR 3803](https://github.com/thewca/worldcubeassociation.org/pull/3803).
+
+### Misc documentation
+
+- Relevant code: [SyncMailingListsJob]
+- [WCA Suite Rollout Plan](https://docs.google.com/document/d/1dx9DsVwD1RUBmT3dmNahARYRKTCiN0Nq9jGRBJsoyHs/edit?ts=5bf0785b#)
+- [Automating GSuite group membership](https://docs.google.com/spreadsheets/d/1I2l_ht2NhcXurG9wezludKNb6ZFjpuWS4jrF_h8DaIw/edit?disco=AAAACUXPzDE&ts=5bef0e17#gid=0)
+
+[SyncMailingListsJob]: https://github.com/thewca/worldcubeassociation.org/blob/master/WcaOnRails/app/jobs/sync_mailing_lists_job.rb