From e321982564c4b87e1f4639b4a705dc6c1d9ac064 Mon Sep 17 00:00:00 2001 From: Chris McCormick Date: Mon, 12 Aug 2019 14:00:27 +0800 Subject: [PATCH] Piku bootstrap extra playbooks (#83) * Remove --pi hack in favour of pi@HOST. * Properly deploy systemd startup in bootstrap. Fixes #71. * piku-bootstrap using separate ansible playbooks. * Prefer built-in playbooks in piku-bootstrap. * Document new piku-bootstrap options. * Doc piku-bootstrap built-in playbooks usage. * Move 'using' doc to top. * Typos + doc fixes in piku-bootstrap. * Improved built-in playbook detection. * Added node playbook. * Documentation tweaks. * Documentation tweaks. --- README.md | 54 ++++++--- piku-bootstrap | 257 ++++++++++-------------------------------- playbooks/nodeenv.yml | 7 ++ playbooks/nodejs.yml | 21 ++++ playbooks/piku.yml | 186 ++++++++++++++++++++++++++++++ 5 files changed, 311 insertions(+), 214 deletions(-) create mode 100644 playbooks/nodeenv.yml create mode 100644 playbooks/nodejs.yml create mode 100644 playbooks/piku.yml diff --git a/README.md b/README.md index 775311e..e602eb2 100644 --- a/README.md +++ b/README.md @@ -6,11 +6,28 @@ The tiniest Heroku/CloudFoundry-like PaaS you've ever seen. [![asciicast](https://asciinema.org/a/Ar31IoTkzsZmWWvlJll6p7haS.svg)](https://asciinema.org/a/Ar31IoTkzsZmWWvlJll6p7haS) +## Using `piku` + +`piku` supports a Heroku-like workflow, like so: + +* Create a `git` SSH remote pointing to your `piku` server with the app name as repo name. + `git remote add piku piku@yourserver:appname`. +* Push your code: `git push piku master`. +* `piku` determines the runtime and installs the dependencies for your app (building whatever's required). + * For Python, it segregates each app's dependencies into a `virtualenv`. + * For Go, it defines a separate `GOPATH` for each app. + * For Node, it installs whatever is in `package.json` into `node_modules`. + * For Java, it builds your app depending on either `pom.xml` or `build.gradle` file. +* It then looks at a `Procfile` and starts the relevant workers using [uWSGI][uwsgi] as a generic process manager. +* You can optionally also specify a `release` worker which is run once when the app is deployed. +* You can then remotely change application settings (`config:set`) or scale up/down worker processes (`ps:scale`). +* You can also bake application settings into a file called [`ENV` which is documented here](./docs/ENV.md). + ## Install To use `piku` you need a VPS, Raspberry Pi, or other server bootstrapped with `piku`'s requirements. You can use a single server to run multiple `piku` apps. -**Warning**: You should use a fresh server or VPS instance without anything important running on it already, as `piku` will make changes to configuration files, running services, etc. +**Warning**: You should use a fresh server or VPS instance without anything important running on it already, as `piku-bootstrap` will make changes to configuration files, running services, etc. Once you've got a fresh server, download the [piku-bootstrap](./piku-bootstrap) shell script onto your local machine and run it: @@ -29,20 +46,7 @@ The script will display a usage message and you can then bootstrap your server: If you put the `piku-bootstrap` script on your `PATH` somewhere, you can use it again to provision other servers in the future. -## Using `piku` - -`piku` supports a Heroku-like workflow, like so: - -* Create a `git` SSH remote pointing to `piku` with the app name as repo name (e.g. `git remote add piku piku@yourserver:appname`). -* `git push piku master` your code. -* `piku` determines the runtime and installs the dependencies for your app (building whatever's required). - * For Python, it segregates each app's dependencies into a `virtualenv`. - * For Go, it defines a separate `GOPATH` for each app. - * For Node, it installs whatever is in `package.json` into `node_modules`. - * For Java, it builds your app depending on either `pom.xml` or `build.gradle` file. -* It then looks at a `Procfile` and starts the relevant workers using [uWSGI][uwsgi] as a generic process manager. -* You can then remotely change application settings (`config:set`) or scale up/down worker processes (`ps:scale`) at will. -* You can also bake application settings into a file called [`ENV` which is documented here](./docs/ENV.md). +See below for instructions on [installing other custom dependencies](#installing-other-dependencies) that your apps might need like a database etc. ### `piku` client @@ -65,7 +69,25 @@ $ piku # <- will show help for the remote app If you put this `piku` script on your `PATH` you can use the `piku` command across multiple apps on your local. -### Examples +### Installing other dependencies + +`piku-bootstrap` uses Ansible internally and it comes with several extra built-in playbooks which you can use to bootstrap common components onto your `piku` server. + +Use `piku-bootstrap list-playbooks` to show a list of built-in playbooks, and then to install one add it as an argument to the bootstrap command. + +For example, to deploy `nodeenv` onto your server: + +```shell +piku-bootstrap root@yourserver.net nodeenv.yml +``` + +You can also use `piku-bootstrap` to run your own Ansible playbooks like this: + +```shell +piku-bootstrap root@yourserver.net ./myplaybook.yml +``` + +## Examples You can find examples for deploying various kinds of apps into a `piku` server in the [Examples folder](./examples). diff --git a/piku-bootstrap b/piku-bootstrap index fce41c2..7e3c9ab 100755 --- a/piku-bootstrap +++ b/piku-bootstrap @@ -7,6 +7,7 @@ PBD=${PIKU_BOOTSTRAP_DIR:-~/.piku-bootstrap} VENV="${PBD}/virtualenv" +REPO="${PBD}/piku" VIRTUALENV_VERSION="16.0.0" LOG="${PBD}/install.log" @@ -23,6 +24,7 @@ main() { # ensure we have a dir mkdir -p "${PBD}" + # ensure we have a virtualenv setup if [ ! -d "$VENV" ]; then echo " #> Virtualenv setup not found. Installing it into ${PBD}." ensure_virtualenv @@ -31,6 +33,12 @@ main() { # get into virtualenv . "$VENV/bin/activate" + # ensure we have the piku repo checked out + if [ ! -d "${REPO}" ]; then + echo " #> Piku repo not found. Installing it into ${REPO}." + git clone https://github.com/rcarmo/piku "${REPO}" + fi + # ensure ansible if [ "`command -v ansible-playbook`" = "" ] then @@ -41,16 +49,25 @@ main() { if [ "$1" = "" ] then echo - echo "Usage: `basename $0` [USER@]HOST [ANSIBLE_ARGS...]" + echo "Usage:" + echo " `basename $0` [USER@]HOST [PLAYBOOK] [ANSIBLE_ARGS...]" + echo " `basename $0` list-playbooks" + echo " `basename $0` update" echo - echo " HOST\tCreates a user 'piku' on the machine 'HOST'," - echo "\tinstalls git, and sets up the piku.py script." + echo " HOST Creates a user 'piku' on the machine 'HOST'," + echo " installs git, and sets up the piku.py script" + echo " by default, unless PLAYBOOK is specified." echo - echo " USER\tOptional non-root user to log in as (e.g. 'pi')." + echo " USER Optional non-root user to log in as (e.g. 'pi')." echo - echo "WARNING: This script installs software and makes changes" - echo " on the target server. Only use a freshly provisioned" - echo " server which you do not mind being modified." + echo " PLAYBOOK Optional playbook to deploy on to the server." + echo " For example 'nodeenv.yml' installs nodeenv." + echo " Can be an absolute path to your own playbook." + echo + echo " list-playbooks" + echo " List available built-in playbooks." + echo + echo " update Pull the piku repo to get the latest playbooks." echo echo "Notes:" echo @@ -62,198 +79,42 @@ main() { echo echo "\t`basename $0` pi@raspberrypi.local" echo + echo " ** WARNING **" + echo " This script installs software and makes changes on the target" + echo " server. Only use a freshly provisioned server which you do not" + echo " mind being modified and reconfigured." + echo else - host="$1"; shift - echo "Bootstrapping piku onto ${host}" - PYTHONWARNINGS="ignore" ansible-playbook -i "${host}", "$@" /dev/stdin << EOF ---- -- hosts: all - become: yes - gather_facts: no - pre_tasks: - - name: Install python2 required by Ansible - raw: "( /usr/bin/python --version 2>&1 | grep -c 'Python' > /dev/null ) || apt-get update && apt-get -y install python" - -- hosts: all - become: yes - tasks: - - name: Add piku user - user: - name: piku - password: ! - comment: PaaS access - group: www-data - - - name: Install Debian Packages - apt: - pkg: ['bc', 'git', 'build-essential', 'libpcre3-dev', 'zlib1g-dev', 'python', 'python3', 'python3-pip', 'python3-dev', 'python-pip', 'python-setuptools', 'python3-setuptools', 'nginx', 'incron', 'acl'] - #, 'python-dev', 'python3', 'python3-virtualenv', 'python3-pip'] - update_cache: true - state: present - - - name: Install Python packages - pip: - executable: pip3 - name: ['setuptools', 'click==7.0', 'virtualenv==15.1.0', 'uwsgi==2.0.15'] - register: packages_installed - - - shell: which uwsgi - register: uwsgi_location - when: packages_installed is changed - - - name: Create uwgsi symlink - file: - src: "{{uwsgi_location.stdout}}" - dest: /usr/local/bin/uwsgi-piku - owner: root - group: root - state: link - when: packages_installed is changed - - - name: Install uwsgi dist script - get_url: - url: https://raw.githubusercontent.com/rcarmo/piku/master/uwsgi-piku.dist - dest: /etc/init.d/uwsgi-piku - mode: 0700 - when: packages_installed is changed - - - name: Install uwsgi-piku dist script - shell: update-rc.d uwsgi-piku defaults - args: - creates: /etc/rc2.d/S01uwsgi-piku - when: packages_installed is changed - - - name: Install uwsgi-piku systemd script - get_url: - url: https://raw.githubusercontent.com/rcarmo/piku/master/uwsgi-piku.service - dest: /etc/systemd/system/uwsgi-piku.service - mode: 0600 - when: packages_installed is changed - - - name: Create piku ansible tmp dir - file: - path: ~piku/.ansible/tmp - mode: 0700 - owner: piku - group: www-data - state: directory - -- hosts: all - become: yes - become_user: piku - tasks: - ### TODO: use pyenv like this instead - - #- name: Download pyenv installer - # get_url: - # url: https://pyenv.run - # dest: ~/pyenv-installer - # mode: 0755 - - #- name: Run pyenv installer - # shell: - # argv: ~/pyenv-installer - # creates: ~/.pyenv - - #- name: Install python3 - # shell: ~/.pyenv/bin/pyenv install 3.6.8 - - #- name: Use python3 - # shell: ~/.pyenv/bin/pyenv local 3.6.8 - - - name: Fetch piku.py script - get_url: - url: https://raw.githubusercontent.com/rcarmo/piku/master/piku.py - dest: ~/piku.py - mode: 0700 - - - name: Run piku setup - shell: python3 ~/piku.py setup - args: - creates: ~/.piku - - - name: Copy up my SSH key for piku - copy: src=~/.ssh/id_rsa.pub dest=/tmp/id_rsa.pub - - - name: Ask piku to use SSH key - shell: python3 ~/piku.py setup:ssh /tmp/id_rsa.pub - args: - creates: ~/.ssh/authorized_keys - - - name: Check if acme.sh is already installed - stat: - path: ~/.acme.sh/acme.sh - register: acme_stat_result - - - name: Download acme.sh - get_url: - url: https://raw.githubusercontent.com/Neilpang/acme.sh/6ff3f5d/acme.sh - dest: ~/acme.sh - mode: 0755 - when: acme_stat_result.stat.exists == False - register: acme_installer - - - name: Execute acme.sh installer - shell: ./acme.sh --install - args: - chdir: ~/ - creates: ~/.acme.sh/acme.sh - executable: /bin/bash - when: acme_installer is defined - - - name: Remove acme.sh installer - file: path=~/acme.sh state=absent - when: acme_installer is defined - - -- hosts: all - become: yes - tasks: - - - name: Test if systemctl is present - shell: command -v systemctl - register: systemctl - - - name: Enable uwsgi-piku service - systemd: - name: uwsgi-piku - enabled: yes - state: started - masked: no - when: '"systemctl" in systemctl.stdout' - - - name: Start uwsgi init script - service: - name: uwsgi-piku - state: started - when: '"systemctl" not in systemctl.stdout' - - - name: Get nginx default config - get_url: - url: https://raw.githubusercontent.com/rcarmo/piku/master/nginx.default.dist - dest: /etc/nginx/sites-available/default - force: yes - register: nginx_config_installed - - - name: Restart nginx service - service: - name: nginx - state: restarted - when: nginx_config_installed is changed - - - name: Get incron config - get_url: - url: https://raw.githubusercontent.com/rcarmo/piku/master/incron.dist - dest: /etc/incron.d/piku - register: incron_config_installed - - - name: Restart incron service - service: - name: incron - state: restarted - when: incron_config_installed is changed - -EOF + case "$1" in + update) + echo "Updating piku repo." + cd "${REPO}" + git pull + ;; + list-playbooks) + ls "${REPO}/playbooks" + ;; + *) + host="$1"; shift + if [ ! "$1" = "" -a -z "${1##*.yml*}" ]; + then + playbook="$1"; shift; + else + playbook="piku.yml" + fi + if [ -z "${playbook##*.yml*}" ]; then + echo "Bootstrapping piku onto ${host}" + builtin="${REPO}/playbooks/${playbook}" + if [ ! -f "${playbook}" -a -f "${builtin}" ]; then + echo "Using built-in playbook: ${playbook}" + playbook="${builtin}" + fi + PYTHONWARNINGS="ignore" ansible-playbook -i "${host}", "${playbook}" "$@" + else + echo "${playbook} is not a valid playbook name." + fi + ;; + esac fi } diff --git a/playbooks/nodeenv.yml b/playbooks/nodeenv.yml new file mode 100644 index 0000000..c539cb6 --- /dev/null +++ b/playbooks/nodeenv.yml @@ -0,0 +1,7 @@ +--- +- hosts: all + tasks: + - name: Install nodeenv + apt: + name: ["nodeenv"] + diff --git a/playbooks/nodejs.yml b/playbooks/nodejs.yml new file mode 100644 index 0000000..e7ddbb3 --- /dev/null +++ b/playbooks/nodejs.yml @@ -0,0 +1,21 @@ +--- +- hosts: all + tasks: + - name: Install node packages + apt: + name: ["nodejs", "npm"] + + - shell: which nodejs + args: + creates: /usr/sbin/node + register: nodejs_location + + - name: Symlink expected node binary name + file: + src: "{{nodejs_location.stdout}}" + dest: /usr/sbin/node + owner: root + group: root + state: link + when: nodejs_location is changed + diff --git a/playbooks/piku.yml b/playbooks/piku.yml new file mode 100644 index 0000000..0db4af0 --- /dev/null +++ b/playbooks/piku.yml @@ -0,0 +1,186 @@ +--- +- hosts: all + become: yes + gather_facts: no + pre_tasks: + - name: Install python2 required by Ansible + raw: "( /usr/bin/python --version 2>&1 | grep -c 'Python' > /dev/null ) || apt-get update && apt-get -y install python" + +- hosts: all + become: yes + tasks: + - name: Add piku user + user: + name: piku + password: ! + comment: PaaS access + group: www-data + + - name: Install Debian Packages + apt: + pkg: ['bc', 'git', 'build-essential', 'libpcre3-dev', 'zlib1g-dev', 'python', 'python3', 'python3-pip', 'python3-dev', 'python-pip', 'python-setuptools', 'python3-setuptools', 'nginx', 'incron', 'acl'] + #, 'python-dev', 'python3', 'python3-virtualenv', 'python3-pip'] + update_cache: true + state: present + + - name: Install Python packages + pip: + executable: pip3 + name: ['setuptools', 'click==7.0', 'virtualenv==15.1.0', 'uwsgi==2.0.15'] + register: packages_installed + + - shell: which uwsgi + register: uwsgi_location + when: packages_installed is changed + + - name: Create uwgsi symlink + file: + src: "{{uwsgi_location.stdout}}" + dest: /usr/local/bin/uwsgi-piku + owner: root + group: root + state: link + when: packages_installed is changed + + - name: Install uwsgi dist script + get_url: + url: https://raw.githubusercontent.com/rcarmo/piku/master/uwsgi-piku.dist + dest: /etc/init.d/uwsgi-piku + mode: 0700 + when: packages_installed is changed + + - name: Install uwsgi-piku dist script + shell: update-rc.d uwsgi-piku defaults + args: + creates: /etc/rc2.d/S01uwsgi-piku + when: packages_installed is changed + + - name: Install uwsgi-piku systemd script + get_url: + url: https://raw.githubusercontent.com/rcarmo/piku/master/uwsgi-piku.service + dest: /etc/systemd/system/uwsgi-piku.service + mode: 0600 + when: packages_installed is changed + + - name: Create piku ansible tmp dir + file: + path: ~piku/.ansible/tmp + mode: 0700 + owner: piku + group: www-data + state: directory + +- hosts: all + become: yes + become_user: piku + tasks: + ### TODO: use pyenv like this instead + + #- name: Download pyenv installer + # get_url: + # url: https://pyenv.run + # dest: ~/pyenv-installer + # mode: 0755 + + #- name: Run pyenv installer + # shell: + # argv: ~/pyenv-installer + # creates: ~/.pyenv + + #- name: Install python3 + # shell: ~/.pyenv/bin/pyenv install 3.6.8 + + #- name: Use python3 + # shell: ~/.pyenv/bin/pyenv local 3.6.8 + + - name: Fetch piku.py script + get_url: + url: https://raw.githubusercontent.com/rcarmo/piku/master/piku.py + dest: ~/piku.py + mode: 0700 + + - name: Run piku setup + shell: python3 ~/piku.py setup + args: + creates: ~/.piku + + - name: Copy up my SSH key for piku + copy: src=~/.ssh/id_rsa.pub dest=/tmp/id_rsa.pub + + - name: Ask piku to use SSH key + shell: python3 ~/piku.py setup:ssh /tmp/id_rsa.pub + args: + creates: ~/.ssh/authorized_keys + + - name: Check if acme.sh is already installed + stat: + path: ~/.acme.sh/acme.sh + register: acme_stat_result + + - name: Download acme.sh + get_url: + url: https://raw.githubusercontent.com/Neilpang/acme.sh/6ff3f5d/acme.sh + dest: ~/acme.sh + mode: 0755 + when: acme_stat_result.stat.exists == False + register: acme_installer + + - name: Execute acme.sh installer + shell: ./acme.sh --install + args: + chdir: ~/ + creates: ~/.acme.sh/acme.sh + executable: /bin/bash + when: acme_installer is defined + + - name: Remove acme.sh installer + file: path=~/acme.sh state=absent + when: acme_installer is defined + + +- hosts: all + become: yes + tasks: + + - name: Test if systemctl is present + shell: command -v systemctl + register: systemctl + + - name: Enable uwsgi-piku service + systemd: + name: uwsgi-piku + enabled: yes + state: started + masked: no + when: '"systemctl" in systemctl.stdout' + + - name: Start uwsgi init script + service: + name: uwsgi-piku + state: started + when: '"systemctl" not in systemctl.stdout' + + - name: Get nginx default config + get_url: + url: https://raw.githubusercontent.com/rcarmo/piku/master/nginx.default.dist + dest: /etc/nginx/sites-available/default + force: yes + register: nginx_config_installed + + - name: Restart nginx service + service: + name: nginx + state: restarted + when: nginx_config_installed is changed + + - name: Get incron config + get_url: + url: https://raw.githubusercontent.com/rcarmo/piku/master/incron.dist + dest: /etc/incron.d/piku + register: incron_config_installed + + - name: Restart incron service + service: + name: incron + state: restarted + when: incron_config_installed is changed