diff --git a/INSTALL.md b/INSTALL.md index 0352947..c0cc75a 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -1,8 +1,12 @@ # Installation -_TODO: describe the system requirements and installation process._ +These installation notes should cover most Debian Linux variants (on any architecture). Very minor changes should be required to deploy on RHEL variants like CentOS, and there is specific emphasis on Raspbian because that's the typical deployment target. -## Setting up the `piku` user +You can, however, run `piku` on _any_ POSIX system where [uWSGI][uwsgi] and Python are available. + +_TODO: describe the overall installation process._ + +## Setting up the `piku` user (Debian Linux, any architecture) _TODO: describe the need for a separate user and why it's configured this way._ @@ -12,7 +16,7 @@ If you're impatient, you need to make sure you have a `~/.ssh/authorized_keys` f command="FINGERPRINT= NAME=default /home/piku/piku.py $SSH_ORIGINAL_COMMAND",no-agent-forwarding,no-user-rc,no-X11-forwarding,no-port-forwarding ``` -## uWSGI Installation +## uWSGI Installation (Debian Linux variants, any architecture) [uWSGI][uwsgi] can be installed in a variety of fashions. However, these instructions assume you're installing it from source, and as such may vary from system to system. @@ -34,16 +38,19 @@ sudo update-rc.d uwsgi-piku defaults sudo service uwsgi-piku start ``` -## Go Installation (on Raspberry Pi) +## Go Installation (Debian Linux variants, on Raspberry Pi) > This is **EXPERIMENTAL** and may not work at all. +### Raspbian + Since Raspbian's Go compiler is version 1.0.2, we need something more up-to-date. 1. Get an [ARM 6 binary tarball][goarm] 2. Unpack it under the `piku` user like such: ```bash +su - piku cd ~ tar -zxvf /tmp/go1.5.3.linux-arm.tar.gz ``` @@ -51,6 +58,7 @@ tar -zxvf /tmp/go1.5.3.linux-arm.tar.gz 3. Give it a temporary `GOPATH` and install `godep`: ```bash +su - piku cd ~ GOROOT=$HOME/go GOPATH=$HOME/golibs PATH=$PATH:$HOME/go/bin go get github.com/tools/godep ``` diff --git a/README.md b/README.md index 2406701..f339177 100644 --- a/README.md +++ b/README.md @@ -14,11 +14,12 @@ From the bottom up: - [ ] `chroot`/namespace isolation - [ ] Proxy deployments to other nodes (build on one box, deploy to many) - [ ] Support Clojure/Java deployments -- [ ] Support Go deployments -- [ ] Support barebones binary deployments - [ ] CLI command documentation - [ ] Complete installation instructions (see `INSTALL.md` for a working draft) - [ ] Installation helper/SSH key add +- [ ] Support barebones binary deployments +- [ ] Sample Go app +- [ ] Support Go deployments - [x] Worker scaling - [x] Remote CLI commands for changing/viewing applied/live settings - [x] Remote tailing of all logfiles for a single application @@ -27,7 +28,7 @@ From the bottom up: - [X] `Procfile` support (`wsgi` and `worker` processes for now, `web` processes being tested) - [x] Basic CLI commands to manage apps - [x] `virtualenv` isolation -- [x] Support Python deployments (currently hardcoded until `Procfile` is implemented) +- [x] Support Python deployments - [x] Repo creation upon first push - [x] Basic understanding of [how `dokku` works](http://off-the-stack.moorman.nu/2013-11-23-how-dokku-works.html) @@ -37,6 +38,7 @@ From the bottom up: * `git push paas master` your code * `piku` determines the runtime and installs the dependencies for your app (building whatever's required) * For Python, it segregates each app's dependencies into a `virtualenv` + * For Go, it defines a separate `GOPATH` for each app * It then looks at a `Procfile` and starts the relevant workers using [uWSGI][uwsgi] as a generic process manager Later on, I intend to do fancier `dokku`-like stuff like reconfiguring `nginx`, but a twist I'm planning on doing is having one `piku` machine act as a build box and deploy the finished product to another. @@ -61,6 +63,10 @@ I intend to support Python, Go, Node and Clojure (Java), but will be focusing on **A:** Partly because it's supposed to run on a [Pi][pi], because it's Japanese onomatopeia for 'twitch' or 'jolt', and because I know the name will annoy some of my friends. +**Q:** Why Python/why not Go? + +**A:** I actually thought about doing this in Go right off the bat, but [click][click] is so cool and I needed to have [uWSGI][uwsgi] running anyway, so I caved in. But I'm very likely to take something like [suture](https://github.com/thejerf/suture) and port this across, doing away with [uWSGI][uwsgi] altogether. + **Q:** Does it run under Python 3? **A:** It should. `click` goes a long way towards abstracting the simpler stuff, and I tried to avoid most obvious incompatibilities (other than a few differences in `subprocess.call` and the like). However, this targets Python 2.7 first, since that's the default on Raspbian. Pull requests are welcome. @@ -69,6 +75,7 @@ I intend to support Python, Go, Node and Clojure (Java), but will be focusing on **A:** I use `dokku` daily, and for most of my personal stuff. But the `dokku` stack relies on a number of `x64` containers that need to be completely rebuilt for ARM, and when I decided I needed something like this (March 2016) that was barely possible - `docker` itself is not fully baked for ARM yet, and people are still trying to get `herokuish` and `buildstep` to build on ARM. +[click]: http://click.poocoo.org [pi]: http://www.raspberrypi.org [dokku]: https://github.com/dokku/dokku [raspi-cluster]: https://github.com/rcarmo/raspi-cluster diff --git a/piku.py b/piku.py index 7585825..204dfb3 100644 --- a/piku.py +++ b/piku.py @@ -125,6 +125,8 @@ def do_deploy(app): if exists(join(app_path, 'requirements.txt')): echo("-----> Python app detected.", fg='green') deploy_python(app) + # if exists(join(app_path, 'Godeps')) or len(glob(join(app_path),'*.go')): + # Go deployment else: echo("-----> Could not detect runtime!", fg='red') # TODO: detect other runtimes @@ -174,6 +176,8 @@ def spawn_app(app, deltas={}): live = join(ENV_ROOT, app, 'LIVE_ENV') # Scaling scaling = join(ENV_ROOT, app, 'SCALING') + + # Bootstrap environment env = { 'PATH': os.environ['PATH'], 'VIRTUAL_ENV': virtualenv_path, @@ -184,6 +188,7 @@ def spawn_app(app, deltas={}): # Load environment variables shipped with repo (if any) if exists(env_file): env.update(parse_settings(env_file, env)) + # Override with custom settings (if any) if exists(settings): env.update(parse_settings(settings, env)) @@ -211,6 +216,7 @@ def spawn_app(app, deltas={}): for w in v: enabled = join(UWSGI_ENABLED, '%s_%s.%d.ini' % (app, k, w)) if not exists(enabled): + echo("-----> Spawning '%s:%s.%d'" % (app, kind, ordinal), fg='green') spawn_worker(app, k, workers[k], env, w) # Remove unnecessary workers (leave logfiles) @@ -260,15 +266,13 @@ def spawn_worker(app, kind, command, env, ordinal=1): for k, v in settings: h.write("%s = %s\n" % (k, v)) - if exists(enabled): - os.unlink(enabled) - echo("-----> Spawning '%s:%s.%d'" % (app, kind, ordinal), fg='green') shutil.copyfile(available, enabled) -def multi_tail(app, filenames): +def multi_tail(app, filenames, catch_up=20): """Tails multiple log files""" + # Seek helper def peek(handle): where = handle.tell() line = handle.readline() @@ -281,6 +285,7 @@ def multi_tail(app, filenames): files = {} prefixes = {} + # Set up current state for each log file for f in filenames: prefixes[f] = splitext(basename(f))[0] files[f] = open(f) @@ -288,12 +293,15 @@ def multi_tail(app, filenames): files[f].seek(0, 2) longest = max(map(len, prefixes.values())) + + # Grab a little history (if any) for f in filenames: - for line in deque(open(f), 20): + for line in deque(open(f), catch_up): yield "%s | %s" % (prefixes[f].ljust(longest), line) while True: updated = False + # Check for updates on every file for f in filenames: line = peek(files[f]) if not line: @@ -301,8 +309,10 @@ def multi_tail(app, filenames): else: updated = True yield "%s | %s" % (prefixes[f].ljust(longest), line) + if not updated: sleep(1) + # Check if logs rotated for f in filenames: if exists(f): if os.stat(f).st_ino != inodes[f]: @@ -316,7 +326,8 @@ def multi_tail(app, filenames): @group() def piku(): - """Initialize paths""" + """The smallest PaaS you've ever seen""" + # Initialize paths for p in [APP_ROOT, GIT_ROOT, ENV_ROOT, UWSGI_ROOT, UWSGI_AVAILABLE, UWSGI_ENABLED, LOG_ROOT]: if not exists(p): os.makedirs(p) @@ -363,7 +374,7 @@ def deploy_app(app, setting): @argument('app') @argument('settings', nargs=-1) def deploy_app(app, settings): - """Show application configuration""" + """Set a configuration setting""" app = sanitize_app_name(app) config_file = join(ENV_ROOT, app, 'ENV') @@ -384,7 +395,7 @@ def deploy_app(app, settings): @piku.command("config:live") @argument('app') def deploy_app(app): - """Show current application settings""" + """Show live configuration settings""" app = sanitize_app_name(app) live_config = join(ENV_ROOT, app, 'LIVE_ENV')