Merge branch 'master' into plugins-master-merge

Conflicts:
	piku.py
pull/129/head
Chris McCormick 2019-11-26 12:20:04 +08:00
commit 5ac6ae2d1d
7 zmienionych plików z 337 dodań i 204 usunięć

36
.github/workflows/pylint.yml vendored 100644
Wyświetl plik

@ -0,0 +1,36 @@
name: Linting
on: [push, pull_request]
jobs:
build:
runs-on: ubuntu-latest
strategy:
max-parallel: 3
matrix:
# This is currently overkill since we are targeting 3.5
# but affords us visibility onto syntax changes in newer Pythons
python-version: [3.5, 3.6, 3.7, 3.8]
steps:
- uses: actions/checkout@v1
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v1
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
- name: Lint with flake8
run: |
pip install flake8
# stop the build if there are Python syntax errors or undefined names
flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
# Notes:
# --exit-zero treats all errors as warnings, but we don't use it anymore
# Allow longer lines for inlining SSH entries and set a complexity threshold that will pass for now
# Ignore W605 (https://lintlyci.github.io/Flake8Rules/rules/W605.html) because
# flake8 does not understand escaping dots inside templates for nginx and SSH
# TODO: pare down complexity and line length as we shrink piku core
flake8 . --ignore=W605 --count --max-complexity=60 --max-line-length=255 --statistics

Wyświetl plik

@ -6,7 +6,13 @@ The tiniest Heroku/CloudFoundry-like PaaS you've ever seen.
[![asciicast](https://asciinema.org/a/Ar31IoTkzsZmWWvlJll6p7haS.svg)](https://asciinema.org/a/Ar31IoTkzsZmWWvlJll6p7haS) [![asciicast](https://asciinema.org/a/Ar31IoTkzsZmWWvlJll6p7haS.svg)](https://asciinema.org/a/Ar31IoTkzsZmWWvlJll6p7haS)
### Documentation: [Procfile](docs/DESIGN.md#procfile-format) | [ENV](./docs/ENV.md) | [Examples](./examples/README.md) ### Documentation: [Procfile](docs/DESIGN.md#procfile-format) | [ENV](./docs/ENV.md) | [Examples](./examples/README.md) | [Roadmap](https://github.com/piku/piku/projects/2)
## Goals and Motivation
(New text being summarized and drafted in #134, soon to find its way here)
I kept finding myself wanting an Heroku/CloudFoundry-like way to deploy stuff on a few remote ARM boards and [my Raspberry Pi cluster][raspi-cluster], but since [dokku][dokku] didn't work on ARM at the time and even `docker` can be overkill sometimes, I decided to roll my own.
## Using `piku` ## Using `piku`
@ -102,74 +108,6 @@ You can also use `piku-bootstrap` to run your own Ansible playbooks like this:
```shell ```shell
piku-bootstrap root@yourserver.net ./myplaybook.yml piku-bootstrap root@yourserver.net ./myplaybook.yml
``` ```
## Examples
You can find examples for deploying various kinds of apps into a `piku` server in the [Examples folder](./examples).
## Motivation
I kept finding myself wanting an Heroku/CloudFoundry-like way to deploy stuff on a few remote ARM boards and [my Raspberry Pi cluster][raspi-cluster], but since [dokku][dokku] didn't work on ARM at the time and even `docker` can be overkill sometimes, I decided to roll my own.
## Project Status/To Do:
This is currently being used for production deployments of [my website](https://taoofmac.com) and a few other projects of mine that run on Azure and other IaaS providers. Regardless, there is still room for improvement:
From the bottom up:
- [ ] Prebuilt Raspbian image with everything baked in
- [ ] `chroot`/namespace isolation (tentative)
- [ ] Relay commands to other nodes
- [ ] Proxy deployments to other nodes (build on one box, deploy to many)
- [ ] Support Clojure/Java deployments through `boot` or `lein`
- [ ] Sample Go app
- [ ] Support Go deployments (in progress)
- [ ] nginx SSL optimization/cypher suites, own certificates
- [ ] Review deployment messages
- [ ] WIP: Review docs/CLI command documentation (short descriptions done, need `help <cmd>` and better descriptions)
- [ ] Lua/WSAPI support
- [x] Support for Java Apps with maven/gradle (in progress through jwsgi, by @matrixjnr)
- [x] Django and Wisp examples (by @chr15m)
- [x] Project logo (by @chr15m)
- [x] Various release/deployment improvements (by @chr15m)
- [x] Support Node deployments (by @chr15m)
- [x] Let's Encrypt support (by @chr15m)
- [x] Allow setting `nginx` IP bindings in `ENV` file (`NGINX_IPV4_ADDRESS` and `NGINX_IPV6_ADDRESS`)
- [x] Cleanups to remove 2.7 syntax internally
- [x] Change to Python 3 runtime as default, with `PYTHON_VERSION = 2` as fallback
- [x] Run in Python 3 only
- [x] (experimental) REPL in `feature/repl`
- [x] Python 3 support through `PYTHON_VERSION = 3`
- [x] static URL mapping to arbitrary paths (hat tip to @carlosefr for `nginx` tuning)
- [x] remote CLI (requires `ssh -t`)
- [x] saner uWSGI logging
- [x] `gevent` activated when `UWSGI_GEVENT = <integer>`
- [x] enable CloudFlare ACL when `NGINX_CLOUDFLARE_ACL = True`
- [x] Autodetect SPDY/HTTPv2 support and activate it
- [x] Basic nginx SSL config with self-signed certificates and UNIX domain socket connection
- [x] nginx support - creates an nginx config file if `NGINX_SERVER_NAME` is defined
- [x] Testing with pre-packaged [uWSGI][uwsgi] versions on Debian Jessie (yes, it was painful)
- [x] Support barebones binary deployments
- [x] Complete installation instructions (see `INSTALL.md`, which also has a draft of Go installation steps)
- [x] Installation helper/SSH key setup
- [x] Worker scaling
- [x] Remote CLI commands for changing/viewing applied/live settings
- [x] Remote tailing of all logfiles for a single application
- [x] HTTP port selection (and per-app environment variables)
- [x] Sample Python app
- [X] `Procfile` support (`wsgi` and `worker` processes for now, `web` processes being tested)
- [x] Basic CLI commands to manage apps
- [x] `virtualenv` isolation
- [x] Support Python deployments
- [x] Repo creation upon first push
- [x] Basic understanding of [how `dokku` works](http://off-the-stack.moorman.nu/2013-11-23-how-dokku-works.html)
## Internals
This is an illustrated example of how `piku` works for a Python deployment:
![](img/piku.png)
## Supported Platforms ## Supported Platforms
`piku` is intended to work in any POSIX-like environment where you have Python, [uWSGI][uwsgi] and SSH, i.e.: `piku` is intended to work in any POSIX-like environment where you have Python, [uWSGI][uwsgi] and SSH, i.e.:
@ -185,26 +123,6 @@ But there are already a few folk using `piku` on vanilla `x64` Linux without any
`piku` currently supports deploying apps (and dependencies) written in Python, with Go, Clojure (Java) and Node (see [above](#project-statustodo)) in the works. But if it can be invoked from a shell, it can be run inside `piku`. `piku` currently supports deploying apps (and dependencies) written in Python, with Go, Clojure (Java) and Node (see [above](#project-statustodo)) in the works. But if it can be invoked from a shell, it can be run inside `piku`.
## FAQ
**Q:** Why `piku`?
**A:** Partly because it's supposed to run on a [Pi][pi], because it's Japanese onomatopeia for 'twitch' or 'jolt', and because I know the name will annoy some of my friends.
**Q:** Why Python/why not Go?
**A:** I actually thought about doing this in Go right off the bat, but [click][click] is so cool and I needed to have [uWSGI][uwsgi] running anyway, so I caved in. But I'm very likely to take something like [suture](https://github.com/thejerf/suture) and port this across, doing away with [uWSGI][uwsgi] altogether.
Go also (at the time) did not have a way to vendor dependencies that I was comfortable with, and that is also why Go support fell behind. Hopefully that will change soon.
**Q:** Does it run under Python 3?
**A:** Right now, it _only_ runs on Python 3, even though it can deploy apps written in both major versions. It began its development using 2.7 and using`click` for abstracting the simpler stuff, and I eventually switched over to 3.5 once it was supported in Debian Stretch and Raspbian since I wanted to make installing it on the Raspberry Pi as simple as possible.
**Q:** Why not just use `dokku`?
**A:** I used `dokku` daily for most of my personal stuff for a good while. But it relied on a number of `x64` containers that needed to be completely rebuilt for ARM, and when I decided I needed something like this (March 2016) that was barely possible - `docker` itself was not fully baked for ARM yet, and people were at the time trying to get `herokuish` and `buildstep` to build on ARM.
[click]: http://click.pocoo.org [click]: http://click.pocoo.org
[pi]: http://www.raspberrypi.org [pi]: http://www.raspberrypi.org
[dokku]: https://github.com/dokku/dokku [dokku]: https://github.com/dokku/dokku

Wyświetl plik

@ -0,0 +1,64 @@
users:
- name: piku
gecos: PaaS access
primary_group: www-data
groups: www-data
apt_update: true
apt_upgrade: true
packages:
- ntp
- tmux
- htop
- vim
- fail2ban
- curl
- build-essential
- certbot
- git
- incron
- libjpeg-dev
- libxml2-dev
- libxslt1-dev
- zlib1g-dev
- nginx
- python-certbot-nginx
- python-dev
- python-pip
- python-virtualenv
- python3-dev
- python3-pip
- python3-click
- python3-virtualenv
- uwsgi
- uwsgi-plugin-asyncio-python3
- uwsgi-plugin-gevent-python
- uwsgi-plugin-python
- uwsgi-plugin-python3
- uwsgi-plugin-tornado-python
- nodejs
- npm
write_files:
- path: /etc/nginx/sites-available/default
content: |
server {
listen 80 default_server;
listen [::]:80 default_server;
root /var/www/html;
index index.html index.htm;
server_name _;
location / {
try_files $uri $uri/ =404;
}
}
include /home/piku/.piku/nginx/*.conf;
- path: /etc/incron.d/paas
content: |
/home/piku/.piku/nginx IN_MODIFY,IN_NO_LOOP /bin/systemctl reload nginx
runcmd:
- timedatectl set-timezone Europe/Lisbon
- ln /home/piku/.piku/uwsgi/uwsgi.ini /etc/uwsgi/apps-enabled/piku.ini
- sudo su - piku -c "wget https://raw.githubusercontent.com/piku/piku/master/piku.py && python3 ~/piku.py setup

Wyświetl plik

@ -86,6 +86,12 @@ As to runtime isolation, `piku` only provides `virtualenv` support until 1.0. Py
This separation makes it easier to cope with long/large deployments and restore apps to a pristine condition, since the app will only go live after the deployment clone is reset (via `git checkout -f`). This separation makes it easier to cope with long/large deployments and restore apps to a pristine condition, since the app will only go live after the deployment clone is reset (via `git checkout -f`).
## Components
This diagram (available as a `dot` file in the `img` folder) outlines how its components interact:
![](../img/piku.png)
[uwsgi]: https://github.com/unbit/uwsgi [uwsgi]: https://github.com/unbit/uwsgi
[emperor]: http://uwsgi-docs.readthedocs.org/en/latest/Emperor.html [emperor]: http://uwsgi-docs.readthedocs.org/en/latest/Emperor.html
[12f]: http://12factor.net [12f]: http://12factor.net

29
docs/FAQ.md 100644
Wyświetl plik

@ -0,0 +1,29 @@
# FAQ
**Q:** Why `piku`?
**A:** Partly because it's supposed to run on a [Pi][pi], because it's Japanese onomatopeia for 'twitch' or 'jolt', and because I know the name will annoy some of my friends.
**Q:** Why Python/why not Go?
**A:** I actually thought about doing this in Go right off the bat, but [click][click] is so cool and I needed to have [uWSGI][uwsgi] running anyway, so I caved in. But I'm very likely to take something like [suture](https://github.com/thejerf/suture) and port this across, doing away with [uWSGI][uwsgi] altogether.
Go also (at the time) did not have a way to vendor dependencies that I was comfortable with, and that is also why Go support fell behind. Hopefully that will change soon.
**Q:** Does it run under Python 3?
**A:** Right now, it _only_ runs on Python 3, even though it can deploy apps written in both major versions. It began its development using 2.7 and using`click` for abstracting the simpler stuff, and I eventually switched over to 3.5 once it was supported in Debian Stretch and Raspbian since I wanted to make installing it on the Raspberry Pi as simple as possible.
**Q:** Why not just use `dokku`?
**A:** I used `dokku` daily for most of my personal stuff for a good while. But it relied on a number of `x64` containers that needed to be completely rebuilt for ARM, and when I decided I needed something like this (March 2016) that was barely possible - `docker` itself was not fully baked for ARM yet, and people were at the time trying to get `herokuish` and `buildstep` to build on ARM.
[click]: http://click.pocoo.org
[pi]: http://www.raspberrypi.org
[dokku]: https://github.com/dokku/dokku
[raspi-cluster]: https://github.com/rcarmo/raspi-cluster
[cygwin]: http://www.cygwin.com
[uwsgi]: https://github.com/unbit/uwsgi
[wsl]: https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux

Wyświetl plik

@ -0,0 +1,58 @@
# Roadmap
This was the original roadmap, filed here in 2019-11-21 for future reference:
- [ ] Prebuilt Raspbian image with everything baked in
- [ ] `chroot`/namespace isolation (tentative)
- [ ] Relay commands to other nodes
- [ ] Proxy deployments to other nodes (build on one box, deploy to many)
- [ ] Support Clojure/Java deployments through `boot` or `lein`
- [ ] Sample Go app
- [ ] Support Go deployments (in progress)
- [ ] nginx SSL optimization/cypher suites, own certificates
- [ ] Review deployment messages
- [ ] WIP: Review docs/CLI command documentation (short descriptions done, need `help <cmd>` and better descriptions)
- [ ] Lua/WSAPI support
- [x] Support for Java Apps with maven/gradle (in progress through jwsgi, by @matrixjnr)
- [x] Django and Wisp examples (by @chr15m)
- [x] Project logo (by @chr15m)
- [x] Various release/deployment improvements (by @chr15m)
- [x] Support Node deployments (by @chr15m)
- [x] Let's Encrypt support (by @chr15m)
- [x] Allow setting `nginx` IP bindings in `ENV` file (`NGINX_IPV4_ADDRESS` and `NGINX_IPV6_ADDRESS`)
- [x] Cleanups to remove 2.7 syntax internally
- [x] Change to Python 3 runtime as default, with `PYTHON_VERSION = 2` as fallback
- [x] Run in Python 3 only
- [x] (experimental) REPL in `feature/repl`
- [x] Python 3 support through `PYTHON_VERSION = 3`
- [x] static URL mapping to arbitrary paths (hat tip to @carlosefr for `nginx` tuning)
- [x] remote CLI (requires `ssh -t`)
- [x] saner uWSGI logging
- [x] `gevent` activated when `UWSGI_GEVENT = <integer>`
- [x] enable CloudFlare ACL when `NGINX_CLOUDFLARE_ACL = True`
- [x] Autodetect SPDY/HTTPv2 support and activate it
- [x] Basic nginx SSL config with self-signed certificates and UNIX domain socket connection
- [x] nginx support - creates an nginx config file if `NGINX_SERVER_NAME` is defined
- [x] Testing with pre-packaged [uWSGI][uwsgi] versions on Debian Jessie (yes, it was painful)
- [x] Support barebones binary deployments
- [x] Complete installation instructions (see `INSTALL.md`, which also has a draft of Go installation steps)
- [x] Installation helper/SSH key setup
- [x] Worker scaling
- [x] Remote CLI commands for changing/viewing applied/live settings
- [x] Remote tailing of all logfiles for a single application
- [x] HTTP port selection (and per-app environment variables)
- [x] Sample Python app
- [X] `Procfile` support (`wsgi` and `worker` processes for now, `web` processes being tested)
- [x] Basic CLI commands to manage apps
- [x] `virtualenv` isolation
- [x] Support Python deployments
- [x] Repo creation upon first push
- [x] Basic understanding of [how `dokku` works](http://off-the-stack.moorman.nu/2013-11-23-how-dokku-works.html)
[click]: http://click.pocoo.org
[pi]: http://www.raspberrypi.org
[dokku]: https://github.com/dokku/dokku
[raspi-cluster]: https://github.com/rcarmo/raspi-cluster
[cygwin]: http://www.cygwin.com
[uwsgi]: https://github.com/unbit/uwsgi
[wsl]: https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux

190
piku.py
Wyświetl plik

@ -4,17 +4,16 @@
try: try:
from sys import version_info from sys import version_info
assert version_info >= (3,5) assert version_info >= (3, 5)
except AssertionError: except AssertionError:
exit("Piku requires Python 3.5 or above") exit("Piku requires Python 3.5 or above")
from importlib import import_module from importlib import import_module
from click import argument, command, group, get_current_context, option, secho as echo, pass_context, CommandCollection from click import argument, command, group, get_current_context, option, secho as echo, pass_context, CommandCollection
from collections import defaultdict, deque from collections import defaultdict, deque
from datetime import datetime
from fcntl import fcntl, F_SETFL, F_GETFL from fcntl import fcntl, F_SETFL, F_GETFL
from glob import glob from glob import glob
from hashlib import md5 from grp import getgrgid
from json import loads from json import loads
from multiprocessing import cpu_count from multiprocessing import cpu_count
from os import chmod, getgid, getuid, symlink, unlink, remove, stat, listdir, environ, makedirs, O_NONBLOCK from os import chmod, getgid, getuid, symlink, unlink, remove, stat, listdir, environ, makedirs, O_NONBLOCK
@ -22,16 +21,18 @@ from os.path import abspath, basename, dirname, exists, getmtime, join, realpath
from re import sub from re import sub
from shutil import copyfile, rmtree, which from shutil import copyfile, rmtree, which
from socket import socket, AF_INET, SOCK_STREAM from socket import socket, AF_INET, SOCK_STREAM
from sys import argv, stdin, stdout, stderr, version_info, exit from sys import argv, stdin, stdout, stderr, version_info, exit, path as sys_path
from sys import path as sys_path
from stat import S_IRUSR, S_IWUSR, S_IXUSR
from subprocess import call, check_output, Popen, STDOUT, PIPE
from tempfile import NamedTemporaryFile
from traceback import format_exc
from time import sleep
from urllib.request import urlopen
from pwd import getpwuid from pwd import getpwuid
from grp import getgrgid from shutil import copyfile, rmtree, which
from socket import socket, AF_INET, SOCK_STREAM
from stat import S_IRUSR, S_IWUSR, S_IXUSR
from subprocess import call, check_output, Popen, STDOUT
from tempfile import NamedTemporaryFile
from time import sleep
from traceback import format_exc
from urllib.request import urlopen
from click import argument, group, secho as echo, pass_context
# === Make sure we can access all system binaries === # === Make sure we can access all system binaries ===
@ -40,8 +41,8 @@ if 'sbin' not in environ['PATH']:
# === Globals - all tweakable settings are here === # === Globals - all tweakable settings are here ===
PIKU_ROOT = environ.get('PIKU_ROOT', join(environ['HOME'],'.piku')) PIKU_ROOT = environ.get('PIKU_ROOT', join(environ['HOME'], '.piku'))
PIKU_BIN = join(environ['HOME'],'bin') PIKU_BIN = join(environ['HOME'], 'bin')
PIKU_SCRIPT = realpath(__file__) PIKU_SCRIPT = realpath(__file__)
PLUGIN_ROOT = abspath(join(PIKU_ROOT, "plugins")) PLUGIN_ROOT = abspath(join(PIKU_ROOT, "plugins"))
APP_ROOT = abspath(join(PIKU_ROOT, "apps")) APP_ROOT = abspath(join(PIKU_ROOT, "apps"))
@ -53,7 +54,7 @@ UWSGI_AVAILABLE = abspath(join(PIKU_ROOT, "uwsgi-available"))
UWSGI_ENABLED = abspath(join(PIKU_ROOT, "uwsgi-enabled")) UWSGI_ENABLED = abspath(join(PIKU_ROOT, "uwsgi-enabled"))
UWSGI_ROOT = abspath(join(PIKU_ROOT, "uwsgi")) UWSGI_ROOT = abspath(join(PIKU_ROOT, "uwsgi"))
UWSGI_LOG_MAXSIZE = '1048576' UWSGI_LOG_MAXSIZE = '1048576'
ACME_ROOT = environ.get('ACME_ROOT', join(environ['HOME'],'.acme.sh')) ACME_ROOT = environ.get('ACME_ROOT', join(environ['HOME'], '.acme.sh'))
ACME_WWW = abspath(join(PIKU_ROOT, "acme")) ACME_WWW = abspath(join(PIKU_ROOT, "acme"))
# === Make sure we can access piku user-installed binaries === # # === Make sure we can access piku user-installed binaries === #
@ -192,12 +193,13 @@ INTERNAL_NGINX_UWSGI_SETTINGS = """
uwsgi_param SERVER_NAME $server_name; uwsgi_param SERVER_NAME $server_name;
""" """
# === Utility functions === # === Utility functions ===
def sanitize_app_name(app): def sanitize_app_name(app):
"""Sanitize the app name and build matching path""" """Sanitize the app name and build matching path"""
app = "".join(c for c in app if c.isalnum() or c in ('.','_')).rstrip().lstrip('/') app = "".join(c for c in app if c.isalnum() or c in ('.', '_')).rstrip().lstrip('/')
return app return app
@ -215,7 +217,7 @@ def get_free_port(address=""):
"""Find a free TCP port (entirely at random)""" """Find a free TCP port (entirely at random)"""
s = socket(AF_INET, SOCK_STREAM) s = socket(AF_INET, SOCK_STREAM)
s.bind((address,0)) s.bind((address, 0))
port = s.getsockname()[1] port = s.getsockname()[1]
s.close() s.close()
return port return port
@ -233,7 +235,7 @@ def write_config(filename, bag, separator='='):
def setup_authorized_keys(ssh_fingerprint, script_path, pubkey): def setup_authorized_keys(ssh_fingerprint, script_path, pubkey):
"""Sets up an authorized_keys file to redirect SSH commands""" """Sets up an authorized_keys file to redirect SSH commands"""
authorized_keys = join(environ['HOME'],'.ssh','authorized_keys') authorized_keys = join(environ['HOME'], '.ssh', 'authorized_keys')
if not exists(dirname(authorized_keys)): if not exists(dirname(authorized_keys)):
makedirs(dirname(authorized_keys)) makedirs(dirname(authorized_keys))
# Restrict features and force all SSH commands to go through our script # Restrict features and force all SSH commands to go through our script
@ -249,20 +251,24 @@ def parse_procfile(filename):
workers = {} workers = {}
if not exists(filename): if not exists(filename):
return None return None
with open(filename, 'r') as procfile: with open(filename, 'r') as procfile:
for line in procfile: for line_number, line in enumerate(procfile):
line = line.strip()
if line.startswith("#") or not line:
continue
try: try:
kind, command = map(lambda x: x.strip(), line.split(":", 1)) kind, command = map(lambda x: x.strip(), line.split(":", 1))
workers[kind] = command workers[kind] = command
except: except Exception:
echo("Warning: unrecognized Procfile entry '{}'".format(line), fg='yellow') echo("Warning: unrecognized Procfile entry '{}' at line {}".format(line, line_number), fg='yellow')
if not len(workers): if len(workers) == 0:
return {} return {}
# WSGI trumps regular web workers # WSGI trumps regular web workers
if 'wsgi' in workers or 'jwsgi' in workers: if 'wsgi' in workers or 'jwsgi' in workers:
if 'web' in workers: if 'web' in workers:
echo("Warning: found both 'wsgi' and 'web' workers, disabling 'web'", fg='yellow') echo("Warning: found both 'wsgi' and 'web' workers, disabling 'web'", fg='yellow')
del(workers['web']) del workers['web']
return workers return workers
@ -281,7 +287,7 @@ def command_output(cmd):
try: try:
env = environ env = environ
return str(check_output(cmd, stderr=STDOUT, env=env, shell=True)) return str(check_output(cmd, stderr=STDOUT, env=env, shell=True))
except: except Exception:
return "" return ""
@ -293,12 +299,12 @@ def parse_settings(filename, env={}):
with open(filename, 'r') as settings: with open(filename, 'r') as settings:
for line in settings: for line in settings:
if '#' == line[0] or len(line.strip()) == 0: # ignore comments and newlines if line[0] == '#' or len(line.strip()) == 0: # ignore comments and newlines
continue continue
try: try:
k, v = map(lambda x: x.strip(), line.split("=", 1)) k, v = map(lambda x: x.strip(), line.split("=", 1))
env[k] = expandvars(v, env) env[k] = expandvars(v, env)
except: except Exception:
echo("Error: malformed setting '{}', ignoring file.".format(line), fg='red') echo("Error: malformed setting '{}', ignoring file.".format(line), fg='red')
return {} return {}
return env return env
@ -315,18 +321,19 @@ def check_requirements(binaries):
return False return False
return True return True
def found_app(kind): def found_app(kind):
"""Helper function to output app detected"""
echo("-----> {} app detected.".format(kind), fg='green') echo("-----> {} app detected.".format(kind), fg='green')
return True return True
def do_deploy(app, deltas={}, newrev=None): def do_deploy(app, deltas={}, newrev=None):
"""Deploy an app by resetting the work directory""" """Deploy an app by resetting the work directory"""
app_path = join(APP_ROOT, app) app_path = join(APP_ROOT, app)
procfile = join(app_path, 'Procfile') procfile = join(app_path, 'Procfile')
log_path = join(LOG_ROOT, app) log_path = join(LOG_ROOT, app)
env_file = join(APP_ROOT, app, 'ENV')
config_file = join(ENV_ROOT, app, 'ENV')
env = {'GIT_WORK_DIR': app_path} env = {'GIT_WORK_DIR': app_path}
if exists(app_path): if exists(app_path):
@ -339,17 +346,18 @@ def do_deploy(app, deltas={}, newrev=None):
if not exists(log_path): if not exists(log_path):
makedirs(log_path) makedirs(log_path)
workers = parse_procfile(procfile) workers = parse_procfile(procfile)
if workers and len(workers): if workers and len(workers) > 0:
settings = {} settings = {}
if exists(join(app_path, 'requirements.txt')) and found_app("Python"): if exists(join(app_path, 'requirements.txt')) and found_app("Python"):
settings.update(deploy_python(app, deltas)) settings.update(deploy_python(app, deltas))
elif exists(join(app_path, 'package.json')) and found_app("Node") and (check_requirements(['nodejs', 'npm']) or check_requirements(['nodeenv'])): elif exists(join(app_path, 'package.json')) and found_app("Node") and (
check_requirements(['nodejs', 'npm']) or check_requirements(['nodeenv'])):
settings.update(deploy_node(app, deltas)) settings.update(deploy_node(app, deltas))
elif exists(join(app_path, 'pom.xml')) and found_app("Java Maven") and check_requirements(['java', 'mvn']): elif exists(join(app_path, 'pom.xml')) and found_app("Java Maven") and check_requirements(['java', 'mvn']):
settings.update(deploy_java(app, deltas)) settings.update(deploy_java(app, deltas))
elif exists(join(app_path, 'build.gradle')) and found_app("Java Gradle") and check_requirements(['java', 'gradle']): elif exists(join(app_path, 'build.gradle')) and found_app("Java Gradle") and check_requirements(['java', 'gradle']):
settings.update(deploy_java(app, deltas)) settings.update(deploy_java(app, deltas))
elif (exists(join(app_path, 'Godeps')) or len(glob(join(app_path,'*.go')))) and found_app("Go") and check_requirements(['go']): elif (exists(join(app_path, 'Godeps')) or len(glob(join(app_path, '*.go')))) and found_app("Go") and check_requirements(['go']):
settings.update(deploy_go(app, deltas)) settings.update(deploy_go(app, deltas))
elif exists(join(app_path, 'project.clj')) and found_app("Clojure Lein") and check_requirements(['java', 'lein']): elif exists(join(app_path, 'project.clj')) and found_app("Clojure Lein") and check_requirements(['java', 'lein']):
settings.update(deploy_clojure(app, deltas)) settings.update(deploy_clojure(app, deltas))
@ -374,16 +382,16 @@ def do_deploy(app, deltas={}, newrev=None):
else: else:
echo("Error: app '{}' not found.".format(app), fg='red') echo("Error: app '{}' not found.".format(app), fg='red')
def deploy_gradle(app, deltas={}): def deploy_gradle(app, deltas={}):
"""Deploy a Java application using Gradle""" """Deploy a Java application using Gradle"""
java_path = join(ENV_ROOT, app) java_path = join(ENV_ROOT, app)
build_path = join(APP_ROOT, app, 'build') build_path = join(APP_ROOT, app, 'build')
env_file = join(APP_ROOT, app, 'ENV') env_file = join(APP_ROOT, app, 'ENV')
build = join(APP_ROOT, app, 'build.gradle')
env = { env = {
'VIRTUAL_ENV': java_path, 'VIRTUAL_ENV': java_path,
"PATH": ':'.join([join(java_path, "bin"), join(app, ".bin"),environ['PATH']]) "PATH": ':'.join([join(java_path, "bin"), join(app, ".bin"), environ['PATH']])
} }
if exists(env_file): if exists(env_file):
@ -403,6 +411,7 @@ def deploy_gradle(app, deltas={}):
return spawn_app(app, deltas) return spawn_app(app, deltas)
def deploy_java(app, deltas={}): def deploy_java(app, deltas={}):
"""Deploy a Java application using Maven""" """Deploy a Java application using Maven"""
# TODO: Use jenv to isolate Java Application environments # TODO: Use jenv to isolate Java Application environments
@ -410,11 +419,10 @@ def deploy_java(app, deltas={}):
java_path = join(ENV_ROOT, app) java_path = join(ENV_ROOT, app)
target_path = join(APP_ROOT, app, 'target') target_path = join(APP_ROOT, app, 'target')
env_file = join(APP_ROOT, app, 'ENV') env_file = join(APP_ROOT, app, 'ENV')
pom = join(APP_ROOT, app, 'pom.xml')
env = { env = {
'VIRTUAL_ENV': java_path, 'VIRTUAL_ENV': java_path,
"PATH": ':'.join([join(java_path, "bin"), join(app, ".bin"),environ['PATH']]) "PATH": ':'.join([join(java_path, "bin"), join(app, ".bin"), environ['PATH']])
} }
if exists(env_file): if exists(env_file):
@ -434,20 +442,20 @@ def deploy_java(app, deltas={}):
return spawn_app(app, deltas) return spawn_app(app, deltas)
def deploy_clojure(app, deltas={}): def deploy_clojure(app, deltas={}):
"""Deploy a Clojure Application""" """Deploy a Clojure Application"""
virtual = join(ENV_ROOT, app) virtual = join(ENV_ROOT, app)
target_path = join(APP_ROOT, app, 'target') target_path = join(APP_ROOT, app, 'target')
env_file = join(APP_ROOT, app, 'ENV') env_file = join(APP_ROOT, app, 'ENV')
projectfile = join(APP_ROOT, app, 'project.clj')
if not exists(target_path): if not exists(target_path):
makedirs(virtual) makedirs(virtual)
env = { env = {
'VIRTUAL_ENV': virtual, 'VIRTUAL_ENV': virtual,
"PATH": ':'.join([join(virtual, "bin"), join(app, ".bin"), environ['PATH']]), "PATH": ':'.join([join(virtual, "bin"), join(app, ".bin"), environ['PATH']]),
"LEIN_HOME": environ.get('LEIN_HOME', join(environ['HOME'],'.lein')), "LEIN_HOME": environ.get('LEIN_HOME', join(environ['HOME'], '.lein')),
} }
if exists(env_file): if exists(env_file):
env.update(parse_settings(env_file, env)) env.update(parse_settings(env_file, env))
@ -504,7 +512,7 @@ def deploy_node(app, deltas={}):
'VIRTUAL_ENV': virtualenv_path, 'VIRTUAL_ENV': virtualenv_path,
'NODE_PATH': node_path, 'NODE_PATH': node_path,
'NPM_CONFIG_PREFIX': abspath(join(node_path, "..")), 'NPM_CONFIG_PREFIX': abspath(join(node_path, "..")),
"PATH": ':'.join([join(virtualenv_path, "bin"), join(node_path, ".bin"),environ['PATH']]) "PATH": ':'.join([join(virtualenv_path, "bin"), join(node_path, ".bin"), environ['PATH']])
} }
if exists(env_file): if exists(env_file):
env.update(parse_settings(env_file, env)) env.update(parse_settings(env_file, env))
@ -514,7 +522,8 @@ def deploy_node(app, deltas={}):
version = env.get("NODE_VERSION") version = env.get("NODE_VERSION")
node_binary = join(virtualenv_path, "bin", "node") node_binary = join(virtualenv_path, "bin", "node")
installed = check_output("{} -v".format(node_binary), cwd=join(APP_ROOT, app), env=env, shell=True).decode("utf8").rstrip("\n") if exists(node_binary) else "" installed = check_output("{} -v".format(node_binary), cwd=join(APP_ROOT, app), env=env, shell=True).decode("utf8").rstrip(
"\n") if exists(node_binary) else ""
if version and check_requirements(['nodeenv']): if version and check_requirements(['nodeenv']):
if not installed.endswith(version): if not installed.endswith(version):
@ -523,7 +532,8 @@ def deploy_node(app, deltas={}):
echo("Warning: Can't update node with app running. Stop the app & retry.", fg='yellow') echo("Warning: Can't update node with app running. Stop the app & retry.", fg='yellow')
else: else:
echo("-----> Installing node version '{NODE_VERSION:s}' using nodeenv".format(**env), fg='green') echo("-----> Installing node version '{NODE_VERSION:s}' using nodeenv".format(**env), fg='green')
call("nodeenv --prebuilt --node={NODE_VERSION:s} --clean-src --force {VIRTUAL_ENV:s}".format(**env), cwd=virtualenv_path, env=env, shell=True) call("nodeenv --prebuilt --node={NODE_VERSION:s} --clean-src --force {VIRTUAL_ENV:s}".format(**env),
cwd=virtualenv_path, env=env, shell=True)
else: else:
echo("-----> Node is installed at {}.".format(version)) echo("-----> Node is installed at {}.".format(version))
@ -558,7 +568,7 @@ def deploy_python(app, deltas={}):
call('virtualenv --python=python{version:d} {app:s}'.format(**locals()), cwd=ENV_ROOT, shell=True) call('virtualenv --python=python{version:d} {app:s}'.format(**locals()), cwd=ENV_ROOT, shell=True)
first_time = True first_time = True
activation_script = join(virtualenv_path,'bin','activate_this.py') activation_script = join(virtualenv_path, 'bin', 'activate_this.py')
exec(open(activation_script).read(), dict(__file__=activation_script)) exec(open(activation_script).read(), dict(__file__=activation_script))
if first_time or getmtime(requirements) > getmtime(virtualenv_path): if first_time or getmtime(requirements) > getmtime(virtualenv_path):
@ -582,8 +592,8 @@ def spawn_app(app, deltas={}):
procfile = join(app_path, 'Procfile') procfile = join(app_path, 'Procfile')
workers = parse_procfile(procfile) workers = parse_procfile(procfile)
workers.pop("release", None) workers.pop("release", None)
ordinals = defaultdict(lambda:1) ordinals = defaultdict(lambda: 1)
worker_count = {k:1 for k in workers.keys()} worker_count = {k: 1 for k in workers.keys()}
# the Python virtualenv # the Python virtualenv
virtualenv_path = join(ENV_ROOT, app) virtualenv_path = join(ENV_ROOT, app)
@ -602,7 +612,7 @@ def spawn_app(app, deltas={}):
'LOG_ROOT': LOG_ROOT, 'LOG_ROOT': LOG_ROOT,
'HOME': environ['HOME'], 'HOME': environ['HOME'],
'USER': environ['USER'], 'USER': environ['USER'],
'PATH': ':'.join([join(virtualenv_path,'bin'),environ['PATH']]), 'PATH': ':'.join([join(virtualenv_path, 'bin'), environ['PATH']]),
'PWD': dirname(env_file), 'PWD': dirname(env_file),
'VIRTUAL_ENV': virtualenv_path, 'VIRTUAL_ENV': virtualenv_path,
} }
@ -617,7 +627,7 @@ def spawn_app(app, deltas={}):
node_path = join(virtualenv_path, "node_modules") node_path = join(virtualenv_path, "node_modules")
if exists(node_path): if exists(node_path):
env["NODE_PATH"] = node_path env["NODE_PATH"] = node_path
env["PATH"] = ':'.join([join(node_path, ".bin"),env['PATH']]) env["PATH"] = ':'.join([join(node_path, ".bin"), env['PATH']])
# Load environment variables shipped with repo (if any) # Load environment variables shipped with repo (if any)
if exists(env_file): if exists(env_file):
@ -647,7 +657,7 @@ def spawn_app(app, deltas={}):
nginx_ssl += " http2" nginx_ssl += " http2"
elif "--with-http_spdy_module" in nginx and "nginx/1.6.2" not in nginx: # avoid Raspbian bug elif "--with-http_spdy_module" in nginx and "nginx/1.6.2" not in nginx: # avoid Raspbian bug
nginx_ssl += " spdy" nginx_ssl += " spdy"
nginx_conf = join(NGINX_ROOT,"{}.conf".format(app)) nginx_conf = join(NGINX_ROOT, "{}.conf".format(app))
env.update({ env.update({
'NGINX_SSL': nginx_ssl, 'NGINX_SSL': nginx_ssl,
@ -667,9 +677,8 @@ def spawn_app(app, deltas={}):
env['NGINX_SOCKET'] = "{BIND_ADDRESS:s}:{PORT:s}".format(**env) env['NGINX_SOCKET'] = "{BIND_ADDRESS:s}:{PORT:s}".format(**env)
echo("-----> nginx will look for app '{}' on {}".format(app, env['NGINX_SOCKET'])) echo("-----> nginx will look for app '{}' on {}".format(app, env['NGINX_SOCKET']))
domain = env['NGINX_SERVER_NAME'].split()[0] domain = env['NGINX_SERVER_NAME'].split()[0]
key, crt = [join(NGINX_ROOT, "{}.{}".format(app,x)) for x in ['key','crt']] key, crt = [join(NGINX_ROOT, "{}.{}".format(app, x)) for x in ['key', 'crt']]
if exists(join(ACME_ROOT, "acme.sh")): if exists(join(ACME_ROOT, "acme.sh")):
acme = ACME_ROOT acme = ACME_ROOT
www = ACME_WWW www = ACME_WWW
@ -683,7 +692,8 @@ def spawn_app(app, deltas={}):
if not exists(key) or not exists(join(ACME_ROOT, domain, domain + ".key")): if not exists(key) or not exists(join(ACME_ROOT, domain, domain + ".key")):
echo("-----> getting letsencrypt certificate") echo("-----> getting letsencrypt certificate")
call('{acme:s}/acme.sh --issue -d {domain:s} -w {www:s}'.format(**locals()), shell=True) call('{acme:s}/acme.sh --issue -d {domain:s} -w {www:s}'.format(**locals()), shell=True)
call('{acme:s}/acme.sh --install-cert -d {domain:s} --key-file {key:s} --fullchain-file {crt:s}'.format(**locals()), shell=True) call('{acme:s}/acme.sh --install-cert -d {domain:s} --key-file {key:s} --fullchain-file {crt:s}'.format(
**locals()), shell=True)
if exists(join(ACME_ROOT, domain)) and not exists(join(ACME_WWW, app)): if exists(join(ACME_ROOT, domain)) and not exists(join(ACME_WWW, app)):
symlink(join(ACME_ROOT, domain), join(ACME_WWW, app)) symlink(join(ACME_ROOT, domain), join(ACME_WWW, app))
else: else:
@ -692,14 +702,16 @@ def spawn_app(app, deltas={}):
# fall back to creating self-signed certificate if acme failed # fall back to creating self-signed certificate if acme failed
if not exists(key) or stat(crt).st_size == 0: if not exists(key) or stat(crt).st_size == 0:
echo("-----> generating self-signed certificate") echo("-----> generating self-signed certificate")
call('openssl req -new -newkey rsa:4096 -days 365 -nodes -x509 -subj "/C=US/ST=NY/L=New York/O=Piku/OU=Self-Signed/CN={domain:s}" -keyout {key:s} -out {crt:s}'.format(**locals()), shell=True) call(
'openssl req -new -newkey rsa:4096 -days 365 -nodes -x509 -subj "/C=US/ST=NY/L=New York/O=Piku/OU=Self-Signed/CN={domain:s}" -keyout {key:s} -out {crt:s}'.format(
**locals()), shell=True)
# restrict access to server from CloudFlare IP addresses # restrict access to server from CloudFlare IP addresses
acl = [] acl = []
if env.get('NGINX_CLOUDFLARE_ACL', 'false').lower() == 'true': if env.get('NGINX_CLOUDFLARE_ACL', 'false').lower() == 'true':
try: try:
cf = loads(urlopen('https://api.cloudflare.com/client/v4/ips').read().decode("utf-8")) cf = loads(urlopen('https://api.cloudflare.com/client/v4/ips').read().decode("utf-8"))
if cf['success'] == True: if cf['success'] is True:
for i in cf['result']['ipv4_cidrs']: for i in cf['result']['ipv4_cidrs']:
acl.append("allow {};".format(i)) acl.append("allow {};".format(i))
for i in cf['result']['ipv6_cidrs']: for i in cf['result']['ipv6_cidrs']:
@ -709,7 +721,7 @@ def spawn_app(app, deltas={}):
remote_ip = environ['SSH_CLIENT'].split()[0] remote_ip = environ['SSH_CLIENT'].split()[0]
echo("-----> Adding your IP ({}) to nginx ACL".format(remote_ip)) echo("-----> Adding your IP ({}) to nginx ACL".format(remote_ip))
acl.append("allow {};".format(remote_ip)) acl.append("allow {};".format(remote_ip))
acl.extend(["allow 127.0.0.1;","deny all;"]) acl.extend(["allow 127.0.0.1;", "deny all;"])
except Exception: except Exception:
cf = defaultdict() cf = defaultdict()
echo("-----> Could not retrieve CloudFlare IP ranges: {}".format(format_exc()), fg="red") echo("-----> Could not retrieve CloudFlare IP ranges: {}".format(format_exc()), fg="red")
@ -721,7 +733,7 @@ def spawn_app(app, deltas={}):
env['INTERNAL_NGINX_STATIC_MAPPINGS'] = '' env['INTERNAL_NGINX_STATIC_MAPPINGS'] = ''
# Get a mapping of /url:path1,/url2:path2 # Get a mapping of /url:path1,/url2:path2
static_paths = env.get('NGINX_STATIC_PATHS','') static_paths = env.get('NGINX_STATIC_PATHS', '')
# prepend static worker path if present # prepend static worker path if present
if 'static' in workers: if 'static' in workers:
stripped = workers['static'].strip("/").rstrip("/") stripped = workers['static'].strip("/").rstrip("/")
@ -733,19 +745,21 @@ def spawn_app(app, deltas={}):
static_url, static_path = item.split(':') static_url, static_path = item.split(':')
if static_path[0] != '/': if static_path[0] != '/':
static_path = join(app_path, static_path) static_path = join(app_path, static_path)
env['INTERNAL_NGINX_STATIC_MAPPINGS'] = env['INTERNAL_NGINX_STATIC_MAPPINGS'] + expandvars(INTERNAL_NGINX_STATIC_MAPPING, locals()) env['INTERNAL_NGINX_STATIC_MAPPINGS'] = env['INTERNAL_NGINX_STATIC_MAPPINGS'] + expandvars(
INTERNAL_NGINX_STATIC_MAPPING, locals())
except Exception as e: except Exception as e:
echo("Error {} in static path spec: should be /url1:path1[,/url2:path2], ignoring.".format(e)) echo("Error {} in static path spec: should be /url1:path1[,/url2:path2], ignoring.".format(e))
env['INTERNAL_NGINX_STATIC_MAPPINGS'] = '' env['INTERNAL_NGINX_STATIC_MAPPINGS'] = ''
env['INTERNAL_NGINX_CUSTOM_CLAUSES'] = expandvars(open(join(app_path, env["NGINX_INCLUDE_FILE"])).read(), env) if env.get("NGINX_INCLUDE_FILE") else "" env['INTERNAL_NGINX_CUSTOM_CLAUSES'] = expandvars(open(join(app_path, env["NGINX_INCLUDE_FILE"])).read(),
env) if env.get("NGINX_INCLUDE_FILE") else ""
env['INTERNAL_NGINX_PORTMAP'] = "" env['INTERNAL_NGINX_PORTMAP'] = ""
if 'web' in workers or 'wsgi' in workers or 'jwsgi' in workers: if 'web' in workers or 'wsgi' in workers or 'jwsgi' in workers:
env['INTERNAL_NGINX_PORTMAP'] = expandvars(NGINX_PORTMAP_FRAGMENT, env) env['INTERNAL_NGINX_PORTMAP'] = expandvars(NGINX_PORTMAP_FRAGMENT, env)
env['INTERNAL_NGINX_COMMON'] = expandvars(NGINX_COMMON_FRAGMENT, env) env['INTERNAL_NGINX_COMMON'] = expandvars(NGINX_COMMON_FRAGMENT, env)
echo("-----> nginx will map app '{}' to hostname '{}'".format(app, env['NGINX_SERVER_NAME'])) echo("-----> nginx will map app '{}' to hostname '{}'".format(app, env['NGINX_SERVER_NAME']))
if('NGINX_HTTPS_ONLY' in env) or ('HTTPS_ONLY' in env): if ('NGINX_HTTPS_ONLY' in env) or ('HTTPS_ONLY' in env):
buffer = expandvars(NGINX_HTTPS_ONLY_TEMPLATE, env) buffer = expandvars(NGINX_HTTPS_ONLY_TEMPLATE, env)
echo("-----> nginx will redirect all requests to hostname '{}' to HTTPS".format(env['NGINX_SERVER_NAME'])) echo("-----> nginx will redirect all requests to hostname '{}' to HTTPS".format(env['NGINX_SERVER_NAME']))
else: else:
@ -755,7 +769,7 @@ def spawn_app(app, deltas={}):
# prevent broken config from breaking other deploys # prevent broken config from breaking other deploys
try: try:
nginx_config_test = str(check_output("nginx -t 2>&1 | grep {}".format(app), env=environ, shell=True)) nginx_config_test = str(check_output("nginx -t 2>&1 | grep {}".format(app), env=environ, shell=True))
except: except Exception:
nginx_config_test = None nginx_config_test = None
if nginx_config_test: if nginx_config_test:
echo("Error: [nginx config] {}".format(nginx_config_test), fg='red') echo("Error: [nginx config] {}".format(nginx_config_test), fg='red')
@ -764,17 +778,17 @@ def spawn_app(app, deltas={}):
# Configured worker count # Configured worker count
if exists(scaling): if exists(scaling):
worker_count.update({k: int(v) for k,v in parse_procfile(scaling).items() if k in workers}) worker_count.update({k: int(v) for k, v in parse_procfile(scaling).items() if k in workers})
to_create = {} to_create = {}
to_destroy = {} to_destroy = {}
for k, v in worker_count.items(): for k, v in worker_count.items():
to_create[k] = range(1,worker_count[k] + 1) to_create[k] = range(1, worker_count[k] + 1)
if k in deltas and deltas[k]: if k in deltas and deltas[k]:
to_create[k] = range(1, worker_count[k] + deltas[k] + 1) to_create[k] = range(1, worker_count[k] + deltas[k] + 1)
if deltas[k] < 0: if deltas[k] < 0:
to_destroy[k] = range(worker_count[k], worker_count[k] + deltas[k], -1) to_destroy[k] = range(worker_count[k], worker_count[k] + deltas[k], -1)
worker_count[k] = worker_count[k]+deltas[k] worker_count[k] = worker_count[k] + deltas[k]
# Cleanup env # Cleanup env
for k, v in list(env.items()): for k, v in list(env.items()):
@ -823,6 +837,8 @@ def spawn_worker(app, kind, command, env, ordinal=1):
settings = [ settings = [
('chdir', join(APP_ROOT, app)), ('chdir', join(APP_ROOT, app)),
('uid', getpwuid(getuid()).pw_name),
('gid', getgrgid(getgid()).gr_name),
('master', 'true'), ('master', 'true'),
('project', app), ('project', app),
('max-requests', env.get('UWSGI_MAX_REQUESTS', '1024')), ('max-requests', env.get('UWSGI_MAX_REQUESTS', '1024')),
@ -832,6 +848,8 @@ def spawn_worker(app, kind, command, env, ordinal=1):
('enable-threads', env.get('UWSGI_ENABLE_THREADS', 'true').lower()), ('enable-threads', env.get('UWSGI_ENABLE_THREADS', 'true').lower()),
('log-x-forwarded-for', env.get('UWSGI_LOG_X_FORWARDED_FOR', 'false').lower()), ('log-x-forwarded-for', env.get('UWSGI_LOG_X_FORWARDED_FOR', 'false').lower()),
('log-maxsize', env.get('UWSGI_LOG_MAXSIZE', UWSGI_LOG_MAXSIZE)), ('log-maxsize', env.get('UWSGI_LOG_MAXSIZE', UWSGI_LOG_MAXSIZE)),
('logfile-chown', '%s:%s' % (getpwuid(getuid()).pw_name, getgrgid(getgid()).gr_name)),
('logfile-chmod', '640'),
('logto', '{log_file:s}.{ordinal:d}.log'.format(**locals())), ('logto', '{log_file:s}.{ordinal:d}.log'.format(**locals())),
('log-backupname', '{log_file:s}.{ordinal:d}.log.old'.format(**locals())), ('log-backupname', '{log_file:s}.{ordinal:d}.log.old'.format(**locals())),
] ]
@ -840,20 +858,20 @@ def spawn_worker(app, kind, command, env, ordinal=1):
if exists(join(env_path, "bin", "activate_this.py")): if exists(join(env_path, "bin", "activate_this.py")):
settings.append(('virtualenv', env_path)) settings.append(('virtualenv', env_path))
if kind== 'jwsgi': if kind == 'jwsgi':
settings.extend([ settings.extend([
('module', command), ('module', command),
('threads', env.get('UWSGI_THREADS','4')), ('threads', env.get('UWSGI_THREADS', '4')),
('plugin', 'jvm'), ('plugin', 'jvm'),
('plugin', 'jwsgi') ('plugin', 'jwsgi')
]) ])
python_version = int(env.get('PYTHON_VERSION','3')) python_version = int(env.get('PYTHON_VERSION', '3'))
if kind == 'wsgi': if kind == 'wsgi':
settings.extend([ settings.extend([
('module', command), ('module', command),
('threads', env.get('UWSGI_THREADS','4')), ('threads', env.get('UWSGI_THREADS', '4')),
]) ])
if python_version == 2: if python_version == 2:
settings.extend([ settings.extend([
@ -877,7 +895,6 @@ def spawn_worker(app, kind, command, env, ordinal=1):
('plugin', 'asyncio_python3'), ('plugin', 'asyncio_python3'),
]) ])
# If running under nginx, don't expose a port at all # If running under nginx, don't expose a port at all
if 'NGINX_SERVER_NAME' in env: if 'NGINX_SERVER_NAME' in env:
sock = join(NGINX_ROOT, "{}.sock".format(app)) sock = join(NGINX_ROOT, "{}.sock".format(app))
@ -890,7 +907,7 @@ def spawn_worker(app, kind, command, env, ordinal=1):
echo("-----> nginx will talk to uWSGI via {BIND_ADDRESS:s}:{PORT:s}".format(**env), fg='yellow') echo("-----> nginx will talk to uWSGI via {BIND_ADDRESS:s}:{PORT:s}".format(**env), fg='yellow')
settings.extend([ settings.extend([
('http', '{BIND_ADDRESS:s}:{PORT:s}'.format(**env)), ('http', '{BIND_ADDRESS:s}:{PORT:s}'.format(**env)),
('http-socket', '{BIND_ADDRESS:s}:{PORT:s}'.format(**env)), ('http-use-socket', '{BIND_ADDRESS:s}:{PORT:s}'.format(**env)),
]) ])
elif kind == 'web': elif kind == 'web':
echo("-----> nginx will talk to the 'web' process via {BIND_ADDRESS:s}:{PORT:s}".format(**env), fg='yellow') echo("-----> nginx will talk to the 'web' process via {BIND_ADDRESS:s}:{PORT:s}".format(**env), fg='yellow')
@ -901,7 +918,8 @@ def spawn_worker(app, kind, command, env, ordinal=1):
settings.append(('attach-daemon', command)) settings.append(('attach-daemon', command))
if kind in ['wsgi', 'web']: if kind in ['wsgi', 'web']:
settings.append(('log-format','%%(addr) - %%(user) [%%(ltime)] "%%(method) %%(uri) %%(proto)" %%(status) %%(size) "%%(referer)" "%%(uagent)" %%(msecs)ms')) settings.append(('log-format',
'%%(addr) - %%(user) [%%(ltime)] "%%(method) %%(uri) %%(proto)" %%(status) %%(size) "%%(referer)" "%%(uagent)" %%(msecs)ms'))
# remove unnecessary variables from the env in nginx.ini # remove unnecessary variables from the env in nginx.ini
for k in ['NGINX_ACL']: for k in ['NGINX_ACL']:
@ -922,10 +940,13 @@ def spawn_worker(app, kind, command, env, ordinal=1):
copyfile(available, enabled) copyfile(available, enabled)
def do_restart(app): def do_restart(app):
"""Restarts a deployed app"""
config = glob(join(UWSGI_ENABLED, '{}*.ini'.format(app))) config = glob(join(UWSGI_ENABLED, '{}*.ini'.format(app)))
if len(config): if len(config) > 0:
echo("Restarting app '{}'...".format(app), fg='yellow') echo("Restarting app '{}'...".format(app), fg='yellow')
for c in config: for c in config:
remove(c) remove(c)
@ -969,9 +990,7 @@ def multi_tail(app, filenames, catch_up=20):
# Check for updates on every file # Check for updates on every file
for f in filenames: for f in filenames:
line = peek(files[f]) line = peek(files[f])
if not line: if line:
continue
else:
updated = True updated = True
yield "{} | {}".format(prefixes[f].ljust(longest), line) yield "{} | {}".format(prefixes[f].ljust(longest), line)
@ -990,6 +1009,8 @@ def multi_tail(app, filenames, catch_up=20):
# === CLI commands === # === CLI commands ===
CONTEXT_SETTINGS = dict(help_option_names=['-h', '--help']) CONTEXT_SETTINGS = dict(help_option_names=['-h', '--help'])
@group(context_settings=CONTEXT_SETTINGS) @group(context_settings=CONTEXT_SETTINGS)
def piku(): def piku():
"""The smallest PaaS you've ever seen""" """The smallest PaaS you've ever seen"""
@ -1059,7 +1080,7 @@ def cmd_config_set(app, settings):
k, v = map(lambda x: x.strip(), s.split("=", 1)) k, v = map(lambda x: x.strip(), s.split("=", 1))
env[k] = v env[k] = v
echo("Setting {k:s}={v} for '{app:s}'".format(**locals()), fg='white') echo("Setting {k:s}={v} for '{app:s}'".format(**locals()), fg='white')
except: except Exception:
echo("Error: malformed setting '{}'".format(s), fg='red') echo("Error: malformed setting '{}'".format(s), fg='red')
return return
write_config(config_file, env) write_config(config_file, env)
@ -1121,12 +1142,12 @@ def cmd_destroy(app):
for p in [join(x, '{}*.ini'.format(app)) for x in [UWSGI_AVAILABLE, UWSGI_ENABLED]]: for p in [join(x, '{}*.ini'.format(app)) for x in [UWSGI_AVAILABLE, UWSGI_ENABLED]]:
g = glob(p) g = glob(p)
if len(g): if len(g) > 0:
for f in g: for f in g:
echo("Removing file '{}'".format(f), fg='yellow') echo("Removing file '{}'".format(f), fg='yellow')
remove(f) remove(f)
nginx_files = [join(NGINX_ROOT, "{}.{}".format(app,x)) for x in ['conf','sock','key','crt']] nginx_files = [join(NGINX_ROOT, "{}.{}".format(app, x)) for x in ['conf', 'sock', 'key', 'crt']]
for f in nginx_files: for f in nginx_files:
if exists(f): if exists(f):
echo("Removing file '{}'".format(f), fg='yellow') echo("Removing file '{}'".format(f), fg='yellow')
@ -1150,7 +1171,7 @@ def cmd_logs(app, process):
app = exit_if_invalid(app) app = exit_if_invalid(app)
logfiles = glob(join(LOG_ROOT, app, process + '.*.log')) logfiles = glob(join(LOG_ROOT, app, process + '.*.log'))
if len(logfiles): if len(logfiles) > 0:
for line in multi_tail(app, logfiles): for line in multi_tail(app, logfiles):
echo(line.strip(), fg='white') echo(line.strip(), fg='white')
else: else:
@ -1180,7 +1201,7 @@ def cmd_ps_scale(app, settings):
app = exit_if_invalid(app) app = exit_if_invalid(app)
config_file = join(ENV_ROOT, app, 'SCALING') config_file = join(ENV_ROOT, app, 'SCALING')
worker_count = {k:int(v) for k, v in parse_procfile(config_file).items()} worker_count = {k: int(v) for k, v in parse_procfile(config_file).items()}
deltas = {} deltas = {}
for s in settings: for s in settings:
try: try:
@ -1193,7 +1214,7 @@ def cmd_ps_scale(app, settings):
echo("Error: worker type '{}' not present in '{}'".format(k, app), fg='red') echo("Error: worker type '{}' not present in '{}'".format(k, app), fg='red')
return return
deltas[k] = c - worker_count[k] deltas[k] = c - worker_count[k]
except: except Exception:
echo("Error: malformed setting '{}'".format(s), fg='red') echo("Error: malformed setting '{}'".format(s), fg='red')
return return
do_deploy(app, deltas) do_deploy(app, deltas)
@ -1212,9 +1233,10 @@ def cmd_run(app, cmd):
for f in [stdout, stderr]: for f in [stdout, stderr]:
fl = fcntl(f, F_GETFL) fl = fcntl(f, F_GETFL)
fcntl(f, F_SETFL, fl | O_NONBLOCK) fcntl(f, F_SETFL, fl | O_NONBLOCK)
p = Popen(' '.join(cmd), stdin=stdin, stdout=stdout, stderr=stderr, env=environ, cwd=join(APP_ROOT,app), shell=True) p = Popen(' '.join(cmd), stdin=stdin, stdout=stdout, stderr=stderr, env=environ, cwd=join(APP_ROOT, app), shell=True)
p.communicate() p.communicate()
@piku.command("restart") @piku.command("restart")
@argument('app') @argument('app')
def cmd_restart(app): def cmd_restart(app):
@ -1229,7 +1251,7 @@ def cmd_restart(app):
def cmd_setup(): def cmd_setup():
"""Initialize environment""" """Initialize environment"""
echo("Running in Python {}".format(".".join(map(str,version_info)))) echo("Running in Python {}".format(".".join(map(str, version_info))))
# Create required paths # Create required paths
for p in [APP_ROOT, GIT_ROOT, ENV_ROOT, UWSGI_ROOT, UWSGI_AVAILABLE, UWSGI_ENABLED, LOG_ROOT, NGINX_ROOT]: for p in [APP_ROOT, GIT_ROOT, ENV_ROOT, UWSGI_ROOT, UWSGI_AVAILABLE, UWSGI_ENABLED, LOG_ROOT, NGINX_ROOT]:
@ -1250,14 +1272,14 @@ def cmd_setup():
('enable-threads', 'true'), ('enable-threads', 'true'),
('threads', '{}'.format(cpu_count() * 2)), ('threads', '{}'.format(cpu_count() * 2)),
] ]
with open(join(UWSGI_ROOT,'uwsgi.ini'), 'w') as h: with open(join(UWSGI_ROOT, 'uwsgi.ini'), 'w') as h:
h.write('[uwsgi]\n') h.write('[uwsgi]\n')
# pylint: disable=unused-variable # pylint: disable=unused-variable
for k, v in settings: for k, v in settings:
h.write("{k:s} = {v}\n".format(**locals())) h.write("{k:s} = {v}\n".format(**locals()))
# mark this script as executable (in case we were invoked via interpreter) # mark this script as executable (in case we were invoked via interpreter)
if not(stat(PIKU_SCRIPT).st_mode & S_IXUSR): if not (stat(PIKU_SCRIPT).st_mode & S_IXUSR):
echo("Setting '{}' as executable.".format(PIKU_SCRIPT), fg='yellow') echo("Setting '{}' as executable.".format(PIKU_SCRIPT), fg='yellow')
chmod(PIKU_SCRIPT, stat(PIKU_SCRIPT).st_mode | S_IXUSR) chmod(PIKU_SCRIPT, stat(PIKU_SCRIPT).st_mode | S_IXUSR)
@ -1276,7 +1298,7 @@ def cmd_setup_ssh(public_key_file):
setup_authorized_keys(fingerprint, PIKU_SCRIPT, key) setup_authorized_keys(fingerprint, PIKU_SCRIPT, key)
except Exception: except Exception:
echo("Error: invalid public key file '{}': {}".format(key_file, format_exc()), fg='red') echo("Error: invalid public key file '{}': {}".format(key_file, format_exc()), fg='red')
elif '-' == public_key_file: elif public_key_file == '-':
buffer = "".join(stdin.readlines()) buffer = "".join(stdin.readlines())
with NamedTemporaryFile(mode="w") as f: with NamedTemporaryFile(mode="w") as f:
f.write(buffer) f.write(buffer)
@ -1296,7 +1318,7 @@ def cmd_stop(app):
app = exit_if_invalid(app) app = exit_if_invalid(app)
config = glob(join(UWSGI_ENABLED, '{}*.ini'.format(app))) config = glob(join(UWSGI_ENABLED, '{}*.ini'.format(app)))
if len(config): if len(config) > 0:
echo("Stopping app '{}'...".format(app), fg='yellow') echo("Stopping app '{}'...".format(app), fg='yellow')
for c in config: for c in config:
remove(c) remove(c)
@ -1352,7 +1374,7 @@ cat | PIKU_ROOT="{PIKU_ROOT:s}" {PIKU_SCRIPT:s} git-hook {app:s}""".format(**env
@piku.command("git-upload-pack", hidden=True) @piku.command("git-upload-pack", hidden=True)
@argument('app') @argument('app')
def cmd_git_receive_pack(app): def cmd_git_upload_pack(app):
"""INTERNAL: Handle git upload pack for an app""" """INTERNAL: Handle git upload pack for an app"""
app = sanitize_app_name(app) app = sanitize_app_name(app)
env = globals() env = globals()