diff --git a/README.md b/README.md index a2ef34b..e48df44 100644 --- a/README.md +++ b/README.md @@ -3,9 +3,9 @@ ## Primary Supported Platforms +* Ubuntu 18.04LTS, Latest Updates * OSX Mojave with Homebrew, Latest Updates * Raspbian Stretch/Stretch-Lite, Latest Updates -* Ubuntu 16.04+, Latest Updates ## Maillist / Group @@ -14,62 +14,7 @@ General discussion mailllist: deployment, tweaks and tuning: * mailto:eotk-users+subscribe@googlegroups.com (via email) * https://groups.google.com/group/eotk-users/subscribe (via web) -NB: bugs are still best reported through `Issues`, above. - -## Changelog - -### HEAD (will become v1.4 after soak-testing) -* new features - * auto-generate (`eotk make-scripts`) wrapper scripts for: - * installation into "init" startup `eotk-init.sh` - * log rotation / housekeeping via installation into eotk-user cronjob (`eotk-housekeeping.sh`) - * stricter enforcement of onionification - * by default will drop compressed, non-onionifiable content with error code `520` - * if this is a problem (why?) use `set drop_unrewritable_content 0` to disable - * validate worker onion addresses upon creation - * works around [issue logged with tor](https://trac.torproject.org/projects/tor/ticket/29429) - * `eotk spotpush` now permits pushing explicitly named scripts from `$EOTK_HOME`, as well as hunting in `projects.d` - * `set hard_mode 1` is now **default** and onionifies both `foo.com` AND `foo\.com` (regexp-style) - * use `set hard_mode 2` to further onionify `foo\\.com` and `foo\\\.com` at slight performance cost - * `set ssl_mkcert 1` to use [mkcert](https://github.com/FiloSottile/mkcert) by @FiloSottile for certificate generation, if you have installed it - * refactor nonces used in rewriting preserved strings - * improvements to `set debug_trap pattern [...]` logging - * support for OpenResty LuaJIT in Raspbian build scripts - * update code version for Raspbian builds scripts - * tor HiddenServiceVersion nits - * dead code removal - * deprecate support for Raspbian Jessie - -### v1.3 -* new features - * "Runbook" has been [moved to the documentation directory](docs.d/RUNBOOK.md) - * tor2web has been blocked-by-default - * since the reason for EOTK is to provide Clearnet websites with an Onion presence, Tor2web is not necessary - * the `FORCE_HTTPS` feature has been added and made *default* - * if your site is 100% HTTPS then you do not need to do anything, - * however sites which require insecure `HTTP` may have to use `set force_https 0` in configurations. - -### v1.2 -* new features: - * optional blocks to methods other than GET/HEAD - * optional 403/Forbidden blocks for accesses to certain Locations or Hosts, including as regexps - * nb: all blocks/block-patterns are *global* and apply to all servers in a project - * optional time-based caching of static content for `N` seconds, with selectable cache size (def: 16Mb) -* new [How To Install](docs.d/HOW-TO-INSTALL.md) guide -* custom install process for Ubuntu, tested on Ubuntu Server 16.04.2-LTS -* renaming / factor-out of Raspbian install code -* fixes to onionbalance support - -### v1.1 -* first cut of onionbalance / softmap - -### v1.0 -* have declared a stable alpha release -* architecture images, at bottom of this page -* all of CSP, HSTS and HPKP are suppressed by default; onion networking mitigates much of this -* ["tunables"](docs.d/TEMPLATES.md) documentation for template content -* `troubleshooting` section near the bottom of this page -* See [project activity](https://github.com/alecmuffett/eotk/graphs/commit-activity) for information +NB: bugs should be reported through `Issues`, above. ### In the News @@ -85,22 +30,17 @@ NB: bugs are still best reported through `Issues`, above. ## Introduction -The goal of EOTK is to provide a tool for prototyping, and deploying -at scale, HTTP and HTTPS onion sites to provide official presence for -popular websites. +EOTK provides a tool for deploying HTTP and HTTPS onion sites to +provide official onion-networking presences for popular websites. -The results are essentially a "man in the middle" proxy; set them up -only for your own sites or for sites which do not require login -credentials of any kind. - -The resulting NGINX configs are probably not terribly well tuned; -please review for your own consideration. I shall try not to modify -the configuration file format. +The result is essentially a "man in the middle" proxy; you should set +them up only for your own sites, or for sites which do not require +login credentials of any kind. ## Important Note About Anonymity The presumed use-case of EOTK is that you have an already-public -website and that you wish to give it a corresponding Onion address. +website and you wish to give it a corresponding Onion address. A lot of people mistakenly believe that Tor Onion Networking is "all about anonymity" - which is incorrect, since it also includes: @@ -114,7 +54,7 @@ about anonymity" - which is incorrect, since it also includes: ...none of which are the same as "anonymity", but all of which are valuable qualities to add to communications. -Also: setting up an Onion address can provide less contention, more +Further: setting up an Onion address can provide less contention, more speed & more bandwidth to people accessing your site than they would get by using Tor "Exit Nodes". @@ -126,7 +66,7 @@ If you want to set up a server which includes anonymity **as well as** all of the aforementioned qualities, you [want to be reading an entirely different document, instead](https://github.com/alecmuffett/the-onion-diaries/blob/master/basic-production-onion-server.md). -## EOTK Usage Notes +## EOTK and HTTPS When connecting to the resulting onions over HTTP/SSL, you will be using wildcard self-signed SSL certificates - you *will* encounter @@ -147,336 +87,27 @@ In production, of course, one would expect to use an SSL EV certificate to provide identity and assurance to an onion site, rendering these issues moot. -## Command List - -### Flags - -* `--local`: ignore the presence of `eotk-workers.conf` and operate - upon local projects; used to administer projects running locally on - a machine which might *also* be running onionbalance. -* `--remote`: functionally the same as `--local` but denotes remote - execution on a worker; used to inhibit recursion and loops amongst - worker machines, of A calls B calls A calls B ... - -### Configuration - -* `eotk config [filename]` # default `onions.conf` - * *synonyms:* `conf`, `configure` - * parses the config file and sets up and populates the projects -* `eotk maps projectname ...` # or: `-a` for all - * prints which onions correspond to which dns domains - * for softmap, this list may not show until after `ob-config` and `ob-start` -* `eotk harvest projectname ...` # or: `-a` for all - * *synonyms:* `onions` - * prints list of onions used by projects - -### Onion Generation - -* `eotk genkey` - * *synonyms:* `gen` - * generate an onion key and stash it in `secrets.d` - -### Project Status & Debugging - -* `eotk status projectname ...` # or: `-a` for all - * active per-project status -* `eotk ps` - * do a basic grep for possibly-orphaned processes -* `eotk debugon projectname ...` # or: `-a` for all - * enable verbose tor logs -* `eotk debugoff projectname ...` # or: `-a` for all - * disable verbose tor logs - -### Starting & Stopping Projects - -* `eotk start projectname ...` # or: `-a` for all - * start projects -* `eotk stop projectname ...` # or: `-a` for all - * stop projects -* `eotk restart projectname ...` # or: `-a` for all - * *synonyms:* `bounce`, `reload` - * stop, and restart, projects -* `eotk nxreload projectname ...` # or: `-a` for all - * politely ask NGINX to reload its config files - - -### Starting & Stopping OnionBalance - -* `eotk ob-start projectname ...` # or: `-a` for all, if applicable - * *synonyms:* -* `eotk ob-restart projectname ...` # or: `-a` for all, if applicable - * *synonyms:* -* `eotk ob-stop` - * *synonyms:* -* `eotk ob-status` - * *synonyms:* - -### Configuring Remote Workers - -* `eotk-workers.conf` - * if not present, only `localhost` will be used - * if present, contains one hostname per line, no comments - * the label `localhost` is a hardcoded synonym for local activity - * other (remote) systems are accessed via `ssh`, `scp` & `rsync` -* `eotk ob-remote-nuke-and-push` - * *synonyms:* -* `eotk ob-nxpush` - * *synonyms:* -* `eotk ob-torpush` - * *synonyms:* -* `eotk ob-spotpush` - * *synonyms:* - -### Backing-Up Remote Workers - -* eotk `mirror` - * *synonyms:* -* eotk `backup` - * *synonyms:* - ## Installation -Please see [How To Install](docs.d/HOW-TO-INSTALL.md) guide +Please refer to the [How To Install](docs.d/HOW-TO-INSTALL.md) guide ## Video Demonstrations -These videos are instructive, but slightly dated. Commands may have -changed slightly, but not much, and I will try to help you understand -what has changed rather than break the command entirely. +**These videos are instructive, but dated.** + +Commands have changed - but not very much - but please check the +documentation rather than trust the videos; consider the videos to be +advisory rather than correct. * [Basic Introduction to EOTK](https://www.youtube.com/watch?v=ti_VkVmE3J4) * [Rough Edges: SSL Certificates & Strange Behaviour](https://www.youtube.com/watch?v=UieLTllLPlQ) * [Using OnionBalance](https://www.youtube.com/watch?v=HNJaMNVCb-U) [![Basic Introduction to EOTK](http://img.youtube.com/vi/ti_VkVmE3J4/0.jpg)](http://www.youtube.com/watch?v=ti_VkVmE3J4) - [![Rough Edges: SSL Certificates & Strange Behaviour](http://img.youtube.com/vi/UieLTllLPlQ/0.jpg)](http://www.youtube.com/watch?v=UieLTllLPlQ) - [![Using OnionBalance](http://img.youtube.com/vi/HNJaMNVCb-U/0.jpg)](http://www.youtube.com/watch?v=HNJaMNVCb-U) -## Experimenting - -After [installation](docs.d/HOW-TO-INSTALL.md), if you want to experiment -with some prefabricated projects, try this: - -* `./010-configure-demo.sh` - * creates working config files, and tor & nginx config files -* Follow the on-screen instructions to start your onions -* Connect to your onions -* Play exciting games of "SSL-Certificate-Acceptance-Whackamole" - -## I want to create my own project! - -Okay, there are two ways to create your own project: - -### EASY WITH AUTOMATIC ONIONS - -Create a config file with a `.tconf` suffix - we'll pretend it's -`foo.tconf` - and use this kind of syntax: - -``` -set project myproject -hardmap %NEW_ONION% foo.com -hardmap %NEW_ONION% foo.co.uk -hardmap %NEW_ONION% foo.de -``` - -...and then run - -`eotk config foo.tconf` - -...which will create the onions for you and will populate a `foo.conf` -for you, and it will then configure EOTK from *that*. You should -probably *delete* `foo.tconf` afterwards, since forcibly reusing it -would trash your existing onions. - -### SLIGHTLY HARDER WITH MANUAL ONIONS - -* Do `eotk genkey` - it will print the name of the onion it generates - * Do this as many times as you wish/need. - * Alternately get a tool like `scallion` or `shallot` and use that to "mine" a desirable onion address. - * https://github.com/katmagic/Shallot - in C, for CPUs - * Seems okay on Linux, not sure about other platforms - * https://github.com/lachesis/scallion - in C#, for CPUs & GPUs (GPU == very fast) - * Advertised as working on Windows, Linux; works well on OSX under "Mono" - * Be sure to store your mined private keys in `secrets.d` with a - filename like `a2s3c4d5e6f7g8h9.key` where `a2s3c4d5e6f7g8h9` is - the corresponding onion address. -* Create a config file with a `.conf` suffix - we'll pretend it's - `foo.conf` - and use this kind of syntax, substituting - `a2s3c4d5e6f7g8h9` for the onion address that you generated. - -``` -set project myproject -hardmap secrets.d/a2s3c4d5e6f7g8h9.key foo.com -``` - -...and then (IMPORTANT) run: - -`eotk config foo.conf` - -...which will configure EOTK. - -### THEN START YOUR PROJECT - -Like this: - -`eotk start myproject` - -## What if I have subdomains? - -When you are setting up the mappings in a config file, you may have to -accomodate "subdomains"; the general form of a internet hostname is -like this: - -* `hostname.domain.tld` # like: www.facebook.com or www.gov.uk - * or: `hostname.domain.sld.tld` # like: www.amazon.co.uk -* `hostname.subdom.domain.tld` # like: www.prod.facebook.com -* `hostname.subsubdom.subdom.domain.tld` # cdn.lhr.eu.foo.net -* `hostname.subsubsubdom.subsubdom.subdom.domain.tld` # ... - -...and so on, where: - -* tld = [top level domain](https://en.wikipedia.org/wiki/Top-level_domain) - * sld = [second level domain](https://en.wikipedia.org/wiki/.uk#Second-level_domains) -* domain = *generally the name of the organisation you are interested in* -* subdomain = *some kind of internal structure* -* hostname = *actual computer, or equivalent* - -When you are setting up mappings, generally the rules are: - -* you will **map one domain per onion** -* you will **ignore all hostnames** -* you will **append all possible subdomain stems** - -So if your browser tells you that you are fetching content from -`cdn7.dublin.ireland.europe.foo.co.jp`, you should add a line like: - -``` -hardmap %NEW_ONION% foo.co.jp europe ireland.europe dublin.ireland.europe -``` - -...and EOTK should do the rest. All this is necessary purely for -correctness of the self-signed SSL-Certificates - which are going to -be weird, anyway - and the rest of the HTML-rewriting code in EOTK -will be blind to subdomains. - -### Subdomain Summary - -Subdomains are supported like this, for `dev` as an example: - -``` -set project myproject -hardmap secrets.d/a2s3c4d5e6f7g8h9.key foo.com dev -``` - -...and if you have multiple subdomains: - -``` -hardmap secrets.d/a2s3c4d5e6f7g8h9.key foo.com dev blogs dev.blogs [...] -``` - -## My company has a lot of sites/domains! - -Example: - -* `www.foo.com.au` -* `www.syd.foo.com.au` -* `www.per.foo.com.au`, -* `www.cdn.foo.net` -* `www.foo.aws.amazon.com` -* ... - -Put them all in the same project as separate mappings, remembering to -avoid the actual "hostnames" as described above: - -``` -set project fooproj -hardmap %NEW_ONION% foo.com.au syd per -hardmap %NEW_ONION% foo.net cdn -hardmap %NEW_ONION% foo.aws.amazon.com -``` - -Onion mapping/translations will be applied for all sites in the same project. - -## Troubleshooting - -The logs for any given project will reside in -`projects.d/.d/logs.d/` - -If something is problematic, first try: - -* `git pull` and... -* `eotk config .conf` again, and then... -* `eotk bounce -a` - -### Lots of broken images, missing images, missing CSS - -This is probably an SSL/HTTPS thing. - -Because of the nature of SSL self-signed certificates, you have to -manually accept the certificate of each and every site for which a -certificate has been created. See the second of the YouTube videos for -some mention of this. - -In short: this is normal and expected behaviour. You can temporarily -fix this by: - -* right-clicking on the image for `Open In New Tab`, and accepting the - certificate -* or using `Inspect Element > Network` to find broken resources, and - doing the same -* or - if you know the list of domains in advance - visiting the - `/hello-onion/` URL for each of them, in advance, to accept - certificates. - -If you get an -[official SSL certificate for your onion site](https://blog.digicert.com/ordering-a-onion-certificate-from-digicert/) -then the problem will vanish. Until then, I am afraid that you will be -stuck playing certificate "whack-a-mole". - -### NGINX: Bad Gateway - -Generally this means that NGINX cannot connect to the remote website, -which usually happens because: - -* the site name in the config file, is wrong -* the nginx daemon tries to do a DNS resolution, which fails - -Check the NGINX logfiles in the directory cited above, for -confirmation. - -If DNS resolution is failing, *PROBABLY* the cause is probably lack -of access to Google DNS / 8.8.8.8; therefore in your config file -you should add a line like this - to use `localhost` as an example: - -``` -set nginx_resolver 127.0.0.1 -``` - -...and then do: - -``` -eotk stop -a -eotk config filename.conf -eotk start -a -``` - -If you need a local DNS resolver, I recommend `dnsmasq`. - -### I can't connect, it's just hanging! - -If your onion project has just started, it can take up to a few -minutes to connect for the first time; also sometimes TorBrowser -caches stale descriptors for older onions. Try restarting TorBrowser -(or use the `New Identity` menu item) and have a cup of tea. If it -persists, check the logfiles. - -### OnionBalance runs for a few days, and then just stops responding! - -Is the clock/time of day correct on all your machines? Are you running NTP? We are not sure but having an incorrect clock may be a contributory factor to this issue. - ### Help I'm Stuck! Ping @alecmuffett on Twitter, or log an `Issue`, above. @@ -529,4 +160,3 @@ the rest of the FB-over-Tor team. Hugs. ---- *eotk (c) 2017 Alec Muffett* - diff --git a/docs.d/CHANGELOG.md b/docs.d/CHANGELOG.md new file mode 100644 index 0000000..b1283fe --- /dev/null +++ b/docs.d/CHANGELOG.md @@ -0,0 +1,54 @@ +# Changelog + +## HEAD (will become v1.4 after soak-testing) +* new features + * auto-generate (`eotk make-scripts`) wrapper scripts for: + * installation into "init" startup `eotk-init.sh` + * log rotation / housekeeping via installation into eotk-user cronjob (`eotk-housekeeping.sh`) + * stricter enforcement of onionification + * by default will drop compressed, non-onionifiable content with error code `520` + * if this is a problem (why?) use `set drop_unrewritable_content 0` to disable + * validate worker onion addresses upon creation + * works around [issue logged with tor](https://trac.torproject.org/projects/tor/ticket/29429) + * `eotk spotpush` now permits pushing explicitly named scripts from `$EOTK_HOME`, as well as hunting in `projects.d` + * `set hard_mode 1` is now **default** and onionifies both `foo.com` AND `foo\.com` (regexp-style) + * use `set hard_mode 2` to further onionify `foo\\.com` and `foo\\\.com` at slight performance cost + * `set ssl_mkcert 1` to use [mkcert](https://github.com/FiloSottile/mkcert) by @FiloSottile for certificate generation, if you have installed it + * refactor nonces used in rewriting preserved strings + * improvements to `set debug_trap pattern [...]` logging + * support for OpenResty LuaJIT in Raspbian build scripts + * update code version for Raspbian builds scripts + * tor HiddenServiceVersion nits + * dead code removal + * deprecate support for Raspbian Jessie + +## v1.3 +* new features + * "Runbook" has been [moved to the documentation directory](docs.d/RUNBOOK.md) + * tor2web has been blocked-by-default + * since the reason for EOTK is to provide Clearnet websites with an Onion presence, Tor2web is not necessary + * the `FORCE_HTTPS` feature has been added and made *default* + * if your site is 100% HTTPS then you do not need to do anything, + * however sites which require insecure `HTTP` may have to use `set force_https 0` in configurations. + +## v1.2 +* new features: + * optional blocks to methods other than GET/HEAD + * optional 403/Forbidden blocks for accesses to certain Locations or Hosts, including as regexps + * nb: all blocks/block-patterns are *global* and apply to all servers in a project + * optional time-based caching of static content for `N` seconds, with selectable cache size (def: 16Mb) +* new [How To Install](docs.d/HOW-TO-INSTALL.md) guide +* custom install process for Ubuntu, tested on Ubuntu Server 16.04.2-LTS +* renaming / factor-out of Raspbian install code +* fixes to onionbalance support + +## v1.1 +* first cut of onionbalance / softmap + +## v1.0 +* have declared a stable alpha release +* architecture images, at bottom of this page +* all of CSP, HSTS and HPKP are suppressed by default; onion networking mitigates much of this +* ["tunables"](docs.d/TEMPLATES.md) documentation for template content +* `troubleshooting` section near the bottom of this page +* See [project activity](https://github.com/alecmuffett/eotk/graphs/commit-activity) for information diff --git a/docs.d/COMMANDS.md b/docs.d/COMMANDS.md new file mode 100644 index 0000000..6354e36 --- /dev/null +++ b/docs.d/COMMANDS.md @@ -0,0 +1,86 @@ +# Command List + +## Flags + +* `--local`: ignore the presence of `eotk-workers.conf` and operate + upon local projects; used to administer projects running locally on + a machine which might *also* be running onionbalance. +* `--remote`: functionally the same as `--local` but denotes remote + execution on a worker; used to inhibit recursion and loops amongst + worker machines, of A calls B calls A calls B ... + +## Configuration + +* `eotk config [filename]` # default `onions.conf` + * *synonyms:* `conf`, `configure` + * parses the config file and sets up and populates the projects +* `eotk maps projectname ...` # or: `-a` for all + * prints which onions correspond to which dns domains + * for softmap, this list may not show until after `ob-config` and `ob-start` +* `eotk harvest projectname ...` # or: `-a` for all + * *synonyms:* `onions` + * prints list of onions used by projects + +## Onion Generation + +* `eotk genkey` + * *synonyms:* `gen` + * generate an onion key and stash it in `secrets.d` + +## Project Status & Debugging + +* `eotk status projectname ...` # or: `-a` for all + * active per-project status +* `eotk ps` + * do a basic grep for possibly-orphaned processes +* `eotk debugon projectname ...` # or: `-a` for all + * enable verbose tor logs +* `eotk debugoff projectname ...` # or: `-a` for all + * disable verbose tor logs + +## Starting & Stopping Projects + +* `eotk start projectname ...` # or: `-a` for all + * start projects +* `eotk stop projectname ...` # or: `-a` for all + * stop projects +* `eotk restart projectname ...` # or: `-a` for all + * *synonyms:* `bounce`, `reload` + * stop, and restart, projects +* `eotk nxreload projectname ...` # or: `-a` for all + * politely ask NGINX to reload its config files + + +## Starting & Stopping OnionBalance + +* `eotk ob-start projectname ...` # or: `-a` for all, if applicable + * *synonyms:* +* `eotk ob-restart projectname ...` # or: `-a` for all, if applicable + * *synonyms:* +* `eotk ob-stop` + * *synonyms:* +* `eotk ob-status` + * *synonyms:* + +## Configuring Remote Workers + +* `eotk-workers.conf` + * if not present, only `localhost` will be used + * if present, contains one hostname per line, no comments + * the label `localhost` is a hardcoded synonym for local activity + * other (remote) systems are accessed via `ssh`, `scp` & `rsync` +* `eotk ob-remote-nuke-and-push` + * *synonyms:* +* `eotk ob-nxpush` + * *synonyms:* +* `eotk ob-torpush` + * *synonyms:* +* `eotk ob-spotpush` + * *synonyms:* + +## Backing-Up Remote Workers + +* eotk `mirror` + * *synonyms:* +* eotk `backup` + * *synonyms:* diff --git a/docs.d/HOW-TO-INSTALL.md b/docs.d/HOW-TO-INSTALL.md index b66672a..98e64a3 100644 --- a/docs.d/HOW-TO-INSTALL.md +++ b/docs.d/HOW-TO-INSTALL.md @@ -1,8 +1,6 @@ # Requirements -EOTK requires Tor 0.3.5.7+ - -EOTK requires recent `nginx` (1.15.8+) with the following modules/features enabled: +EOTK requires recent `nginx` with the following modules/features enabled: * `headers_more` * `ngx_http_substitutions_filter_module` @@ -10,25 +8,18 @@ EOTK requires recent `nginx` (1.15.8+) with the following modules/features enabl * `http_ssl` * Lua and/or LuaJIT (ideally from OpenResty) -# After the Installation +# Installation -Once you have installed EOTK (below) and configured and tested it -for your project, run: +Per-platform: -* `eotk make-scripts` +## Ubuntu 18.04LTS (prebuilt via tor and canonical) -This will create two files: +Install a `ubuntu-18.04.2-live-server-amd64.iso` server instance; then: -* `eotk-init.sh` - for installing on your system as a startup script -* `eotk-housekeeping.sh` - for cronjob log rotation and other cleanup work - -Please read the individual files for installation instructions; -local setup is intended to be pretty simple. - -# Per-Platform Installations - -Where you don't have Tor, NGINX or OnionBalance, -or much other stuff currently installed: +* `git clone https://github.com/alecmuffett/eotk.git` +* `cd eotk` +* `./opt.d/install-everything-on-ubuntu-18.04.sh` + * this includes a full update ## macOS Mojave (prebuilt via homebrew) @@ -37,25 +28,11 @@ or much other stuff currently installed: * `cd eotk` * `./opt.d/install-everything-on-macos.sh` -## Ubuntu 18.04LTS (prebuilt via tor and canonical) - -Install a base Ubuntu 18.04LTS server image (or better) and then do: - -* `git clone https://github.com/alecmuffett/eotk.git` -* `cd eotk` -* `./opt.d/install-everything-on-ubuntu-18.04.sh` - -You can then do: - -* `./eotk config demo.d/wikipedia.tconf` -* `./eotk start wikipedia` - ## Raspbian (manual builds) -Serially, this takes about 1h45m on a PiZero, or about 30m on a Pi3b. -These figures should improve when recent Tor updates sediment into Raspbian. - -Scripts are supplied for stretch +Serially, installation takes about 1h45m on a PiZero, or about 30m on +a Pi3b. These figures should improve when recent Tor updates sediment +into Raspbian; scripts are supplied for Raspbian "Stretch". * `sudo apt-get install -y git` * `git clone https://github.com/alecmuffett/eotk.git` @@ -64,42 +41,259 @@ Scripts are supplied for stretch * `./opt.d/build-tor-on-raspbian-stretch.sh` * `./opt.d/install-onionbalance-on-raspbian-stretch.sh` -# Piecemeal Installation Notes +# Dealing With OnionBalance And Load-Balancing -You only need this section if you have to do installation in bits -because pre-existing software: +*NEW FOR 2019:* -## Ubuntu Tor Installation +OnionBalance as-of June 2019 is an flaky piece of software which is +hard to run on modern Linux because an stale python crypto library; +more than 90% of Onion sites will not practically need it - or, not +initially, anyway - so I am *deprecating OnionBalance in EOTK* until +it is majorly overhauled and also supports v3 onion addressing. -In a browser elsewhere, retreive the instructions for installing Tor -from https://www.torproject.org/docs/debian.html.en +So, I recommend people to avoid OnionBalance and use hardmap (local) +for the moment, until Tor fix it. Consider OB if and only if your +system is under sustained high bandwidth and strongly demonstrates +extended choking on throughput. -* Set the menu options for: - * run *Ubuntu Xenial Xerus* - * and want *Tor* - * and version *stable* - * and read what is now on the page. -* Configure the APT repositories for Tor - * I recommend that you add the tor repositories into a new file - * Use: `/etc/apt/sources.list.d/tor.list` or similar -* Follow the instructions given: - * Do the documented `gpg` thing - * Do the documented `apt update` thing - * Do the documented `tor` installation thing +# Dealing With HTTPS Certificates -## Ubuntu NGINX Installation +TBD -Through `apt-get`; logfiles are tweaked to be writable by admin users. +# Demonstration And Testing -* `sudo apt-get install nginx-extras` -* `sudo find /var/log/nginx/ -type f -perm -0200 -print0 | sudo xargs -0 chmod g+w` +After installation, you can do: -## Ubuntu OnionBalance Installation +* `./eotk config demo.d/wikipedia.tconf` +* `./eotk start wikipedia` -Through `apt-get` and `pip`; using `pip` tends to mangle permissions, -hence the find/xargs-chmod commands. +# Configuring Start-On-Boot, And Logfile Compression -* `sudo apt-get install socat python-pip` -* `sudo pip install onionbalance` -* `sudo find /usr/local/bin /usr/local/lib -perm -0400 -print0 | sudo xargs -0 chmod a+r` -* `sudo find /usr/local/bin /usr/local/lib -perm -0100 -print0 | sudo xargs -0 chmod a+x` +Once you have installed EOTK (below) and configured and tested it +for your project, run: + +* `./eotk make-scripts` + +This will create two files: + +* `./eotk-init.sh` - for installing on your system as a startup script +* `./eotk-housekeeping.sh` - for cronjob log rotation and other cleanup work + +Please read the individual files for installation instructions; local +setup is intended to be pretty simple, let me know if anything is +confusing. + +# LEGACY DOCUMENTATION IN NEED OF REVIEW, BELOW THIS POINT + +## I Want To Create My Own Project! + +Okay, there are two ways to create your own project: + +### Easy With Automatic Onions + +Create a config file with a `.tconf` suffix - we'll pretend it's +`foo.tconf` - and use this kind of syntax: + +``` +set project myproject +hardmap %NEW_ONION% foo.com +hardmap %NEW_ONION% foo.co.uk +hardmap %NEW_ONION% foo.de +``` + +...and then run + +`eotk config foo.tconf` + +...which will create the onions for you and will populate a `foo.conf` +for you, and it will then configure EOTK from *that*. You should +probably *delete* `foo.tconf` afterwards, since forcibly reusing it +would trash your existing onions. + +### Slightly Harder With Manually-Mined Onions + +* Do `eotk genkey` - it will print the name of the onion it generates + * Do this as many times as you wish/need. + * Alternately get a tool like `scallion` or `shallot` and use that to "mine" a desirable onion address. + * https://github.com/katmagic/Shallot - in C, for CPUs + * Seems okay on Linux, not sure about other platforms + * https://github.com/lachesis/scallion - in C#, for CPUs & GPUs (GPU == very fast) + * Advertised as working on Windows, Linux; works well on OSX under "Mono" + * Be sure to store your mined private keys in `secrets.d` with a + filename like `a2s3c4d5e6f7g8h9.key` where `a2s3c4d5e6f7g8h9` is + the corresponding onion address. +* Create a config file with a `.conf` suffix - we'll pretend it's + `foo.conf` - and use this kind of syntax, substituting + `a2s3c4d5e6f7g8h9` for the onion address that you generated. + +``` +set project myproject +hardmap secrets.d/a2s3c4d5e6f7g8h9.key foo.com +``` + +...and then (IMPORTANT) run: + +`eotk config foo.conf` + +...which will configure EOTK. + +### Then Start Your Project + +Like this: + +`eotk start myproject` + +## What If I Have Subdomains? + +When you are setting up the mappings in a config file, you may have to +accomodate "subdomains"; the general form of a internet hostname is +like this: + +* `hostname.domain.tld` # like: www.facebook.com or www.gov.uk + * or: `hostname.domain.sld.tld` # like: www.amazon.co.uk +* `hostname.subdom.domain.tld` # like: www.prod.facebook.com +* `hostname.subsubdom.subdom.domain.tld` # cdn.lhr.eu.foo.net +* `hostname.subsubsubdom.subsubdom.subdom.domain.tld` # ... + +...and so on, where: + +* tld = [top level domain](https://en.wikipedia.org/wiki/Top-level_domain) + * sld = [second level domain](https://en.wikipedia.org/wiki/.uk#Second-level_domains) +* domain = *generally the name of the organisation you are interested in* +* subdomain = *some kind of internal structure* +* hostname = *actual computer, or equivalent* + +When you are setting up mappings, generally the rules are: + +* you will **map one domain per onion** +* you will **ignore all hostnames** +* you will **append all possible subdomain stems** + +So if your browser tells you that you are fetching content from +`cdn7.dublin.ireland.europe.foo.co.jp`, you should add a line like: + +``` +hardmap %NEW_ONION% foo.co.jp europe ireland.europe dublin.ireland.europe +``` + +...and EOTK should do the rest. All this is necessary purely for +correctness of the self-signed SSL-Certificates - which are going to +be weird, anyway - and the rest of the HTML-rewriting code in EOTK +will be blind to subdomains. + +### Subdomain Summary + +Subdomains are supported like this, for `dev` as an example: + +``` +set project myproject +hardmap secrets.d/a2s3c4d5e6f7g8h9.key foo.com dev +``` + +...and if you have multiple subdomains: + +``` +hardmap secrets.d/a2s3c4d5e6f7g8h9.key foo.com dev blogs dev.blogs [...] +``` + +## My Company Has A Lot Of Sites/Domains! + +Example: + +* `www.foo.com.au` +* `www.syd.foo.com.au` +* `www.per.foo.com.au`, +* `www.cdn.foo.net` +* `www.foo.aws.amazon.com` +* ... + +Put them all in the same project as separate mappings, remembering to +avoid the actual "hostnames" as described above: + +``` +set project fooproj +hardmap %NEW_ONION% foo.com.au syd per +hardmap %NEW_ONION% foo.net cdn +hardmap %NEW_ONION% foo.aws.amazon.com +``` + +Onion mapping/translations will be applied for all sites in the same project. + +## Troubleshooting + +The logs for any given project will reside in +`projects.d/.d/logs.d/` + +If something is problematic, first try: + +* `git pull` and... +* `eotk config .conf` again, and then... +* `eotk bounce -a` + +### Lots Of Broken Images, Missing Images, Missing CSS + +This is probably an SSL/HTTPS thing. + +Because of the nature of SSL self-signed certificates, you have to +manually accept the certificate of each and every site for which a +certificate has been created. See the second of the YouTube videos for +some mention of this. + +In short: this is normal and expected behaviour. You can temporarily +fix this by: + +* right-clicking on the image for `Open In New Tab`, and accepting the + certificate +* or using `Inspect Element > Network` to find broken resources, and + doing the same +* or - if you know the list of domains in advance - visiting the + `/hello-onion/` URL for each of them, in advance, to accept + certificates. + +If you get an +[official SSL certificate for your onion site](https://blog.digicert.com/ordering-a-onion-certificate-from-digicert/) +then the problem will vanish. Until then, I am afraid that you will be +stuck playing certificate "whack-a-mole". + +### NGINX: Bad Gateway + +Generally this means that NGINX cannot connect to the remote website, +which usually happens because: + +* the site name in the config file, is wrong +* the nginx daemon tries to do a DNS resolution, which fails + +Check the NGINX logfiles in the directory cited above, for +confirmation. + +If DNS resolution is failing, *PROBABLY* the cause is probably lack +of access to Google DNS / 8.8.8.8; therefore in your config file +you should add a line like this - to use `localhost` as an example: + +``` +set nginx_resolver 127.0.0.1 +``` + +...and then do: + +``` +eotk stop -a +eotk config filename.conf +eotk start -a +``` + +If you need a local DNS resolver, I recommend `dnsmasq`. + +### I Can't Connect, It's Just Hanging! + +If your onion project has just started, it can take up to a few +minutes to connect for the first time; also sometimes TorBrowser +caches stale descriptors for older onions. Try restarting TorBrowser +(or use the `New Identity` menu item) and have a cup of tea. If it +persists, check the logfiles. + +### OnionBalance Runs For A Few Days, And Then Just Stops Responding! + +Is the clock/time of day correct on all your machines? Are you +running NTP? We are not sure but having an incorrect clock may be a +contributory factor to this issue. diff --git a/docs.d/RUNBOOK.md b/docs.d/SOFTMAP-RUNBOOK.md similarity index 100% rename from docs.d/RUNBOOK.md rename to docs.d/SOFTMAP-RUNBOOK.md diff --git a/TIPS-FOR-MINING-ONIONS.md b/docs.d/TIPS-FOR-MINING-ONIONS.md similarity index 100% rename from TIPS-FOR-MINING-ONIONS.md rename to docs.d/TIPS-FOR-MINING-ONIONS.md diff --git a/docs.d/TODO.md b/docs.d/TODO.md index 9722fc7..439ff38 100644 --- a/docs.d/TODO.md +++ b/docs.d/TODO.md @@ -1,8 +1,6 @@ # Stuff To Consider / Implement -* maybe a setting is needed to drop extra headers into the outbound rewrites? compare: subsextra; add to origin_rewrites - -* 020-generate-init-script.sh +* maybe a setting is needed to drop extra headers into the outbound rewrites? compare: subsextra; add to origin rewrites * does onionbalance-tor use opt.d/tor in preference, or not? * eotk script: refactor so that there's a separate ob-start which does NOT call ob-gather * consider downgrade of RPi scripts to 3.0.x series Tor