Use the more traditional '.dist' extension for configuration templates
that are part of the MapOSMatic distribution. Update the installation
instructions accordingly.
Includes some style fixes in the INSTALL and template files, as well as
a fix to the wrapper script to correctly allow for an undefined log
location (defaults to stderr).
Signed-off-by: Maxime Petazzoni <maxime.petazzoni@bulix.org>
Bring back the forking mode to the MapOSMatic rendering daemon, used by
default. We have to do this because Mapnik keeps a cache of the
shapefiles loaded during a rendering, which is not suitable for the
long-running execution of a single-process rendering daemon.
Signed-off-by: Maxime Petazzoni <maxime.petazzoni@bulix.org>
Automatically send an email to the configured admins with a traceback of
the rendering exception when a rendering fails.
Signed-off-by: Maxime Petazzoni <maxime.petazzoni@bulix.org>
The settings_local.py file defines the RENDERING_RESULT_FORMATS which
contains the formats (PNG, SVGZ, PDF, CSV) in which the maps should be
rendered. However, depending on the selected rendering layout, some
formats may not be supported: typically, PNG and SVGZ are not
supported by the multi-page rendering layout.
Therefore, the daemon automatically renders the map only in the
formats that are the intersection of the formats choosen in
RENDERING_RESULT_FORMATS and the formats compatible with the rendering
layout.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@enix.org>
For multi-page renderings, the thumbnail was useless: it was the last
page of the index. Instead, we create with ImageMagick a small
thumbnail thats displays the cover page and overview page of the
multi-page PDF. We take into account landscape/portrait renderings to
combine those two images into a single thumbnail with a good ratio.
We directly use subprocess to run the ImageMagick commands, because
the Python ImageMagick API is so poorly documented that after spending
several hours trying to figure out which method to use, switching to
subprocess with ImageMagick commands turned out to be a more
reasonable solution.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@enix.org>
This patch makes sure that we can work on the exact same city envelopes
in maposmatic and ocitysmap: we now share ocitysmap's
get_geometry_from_osmid(). Previously, get_bbox_from_osmid crashed if
the osm ID was present multiple times in the polygon table, and
ignored the line table. This patch fixes both, by basing itself on
ocitysmap's 093b95bcf6f54e855b781a9ccd60ce8117b35976.
In d2-ocitysmap2, the zoom factor can be defined in the
stylesheet. This patch makes sure that the stylesheet is chosen before
proposing a list of possible paper sizes. Then it determines the
possible paper sizes from this stylesheet.
In settings_local.py, OCITYSMAP_CFG_PATH is set to None by default
and render.py put this value in a list before passing it to OCitySMap
constructor that compares it with None, not a list with a None item.
Signed-off-by: Gaël Utard <gael.utard@laposte.net>
Rework the MapOSMatic logging setup to initialize both the maposmatic
and ocitysmap logging targets, and do so only once to avoid duplicate
logging messages.
Signed-off-by: Maxime Petazzoni <maxime.petazzoni@bulix.org>
* The MapRenderingJob model has been modified: the paper size is no
longer stored as a string identifying a particular paper format,
but rather two IntegerFields are used to store the width/height in
millimeters of the paper. This allows to support "Best fit" paper
size.
* The MapRenderingJobForm is extended to offer a paper orientation
selection form, with portrait/landscape selection.
* The /apis/papersize/ service now returns all informations given by
OCitySMap on allowed paper sizes, and not only the name of the
allowed paper formats.
* As the "paper size" panel is now used to also select the paper
orientation, it is renamed to "step-paper" instead of
"step-paper-size".
* A bunch of Javascript code is used to update the orientation
selector depending on the selected paper format.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@enix.org>
Signed-off-by: Maxime Petazzoni <maxime.petazzoni@bulix.org>
The renderer threads did not handle exceptions happening during the
data preparation phase. This commit fixes this, avoiding the daemon to
vanish when asking a rendering that fails (most common cause is GEOS
intersect errors).
Signed-off-by: Maxime Petazzoni <maxime.petazzoni@bulix.org>
This dbck utility, run through the environment wrapper, checks and
cleans the MapOSMatic job database. This first version provides the
following checks/repairs capabilities:
* obsolete status
* mutually exclusive admin/bbox data
* missing map language
A --dry-run option is also available to only look at what operations
would be done.
Signed-off-by: Maxime Petazzoni <maxime.petazzoni@bulix.org>
The render module, originally a simple extract of the render_job
function from the old daemon, has been rewriten to expose a more
sensible API and to provide self-contained timeout support.
Two classes are now available, exposing the same public API for seamless
use of any of the two:
* JobRenderer is a simple, blocking job renderer. It can be threaded
if the caller decides to call start() instead of run().
* TimingOutJobRenderer, as its name suggests, is a timing out job
renderer that makes use of the threading capability of JobRenderer
to handle a timeout on the rendering process and kill the rendering
thread, whatever it is doing, if the given timeout is reached.
The render module also now exposes a few public RESULT_ constants that
can be used to identify the result of a job rendering. This is used in
the daemon to infer an appropriate resultmsg.
As a standalone process, the job renderer now takes an optional second
argument: the timeout, in seconds.
.../scripts/wrapper.py scripts/render.py <jobid> [timeout]
As of now, the daemon is fully usable in production with the same level
of functionality as before.
Signed-off-by: Maxime Petazzoni <maxime.petazzoni@bulix.org>
This is the first step of the rewrite of the MapOSMatic rendering
daemon. The long-term objective is to get a more flexible rendering
daemon that would support the rendering of several jobs in parallel, and
with an overall more Python-y and maintainable code.
This first shot brings a completely new, simpler MapOSMatic daemon with
the same level of functionality as the previous daemon in terms of
serial rendering of the job queue. Three major changes happen here:
1. the bash-based wrapper script has been removed, in favor of a more
clever Python wrapper. Cleaner, and more importantly more portable.
The wrapper still needs a bit of configuration, and after the
config.py-template has been tweaked into a valid config.py file,
the daemon can be started by:
.../scripts/wrapper.py scripts/daemon.py
2. the externalization of the rendering routine into a 'render'
module. This module does not access the database - only the daemon
does. The sole purpose of the render module is to encapsulate the
rendering process and its errors+exceptions handling.
It can also be used as a standalone, job-ID-based renderer:
.../scripts/wrapper.py scripts/render.py <jobid>
3. the cleanup mechanism now runs in a separate thread, in the
background.
Signed-off-by: Maxime Petazzoni <maxime.petazzoni@bulix.org>
Previously, when the rendering directory was over the defined threshold,
files where removed progressively, oldest first, to make up some space.
No information was kept about jobs whose files were removed, making it
harder to keep track of valid jobs with files available.
This change introduces two new things:
1. a new job status, 3, for jobs processed but now without files, or
"obsolete" jobs.
2. a new cleanup mechanism that considers jobs as the atomic unit of
cleaning instead of files, as this would leave with jobs without
all their renderings (which didn't make much sense).
The cleanup function underwent the following modifications:
* files are now sorted by content modification time and not last
metadata change (a simple chmod could mess up the order);
* thumbnails are excluded from the list of considered files for
removal (this is still is discussion, but for now let's keep them);
* when a file needs to be removed, all files from its parent job are
removed and the job's status is set to 3 (see
MapRenderingJob#remove_all_files).
* if no parent job can be found, it's an orphaned file and can be
safely removed. Files starting with a '.' are of course preserved;
* some logging improvements during the cleanup phase.
New 'job-done-obsolete' and 'job-error-obsolete' status icons are now
available, and the status icon filename is now inferred with a custom
template tag (this also led to some cleanup in extratags.py).
The file size of the renderings is also displayed next to each format in
the job information.
Signed-off-by: Maxime Petazzoni <maxime.petazzoni@bulix.org>
Following the API change in OCitySMap to improve paralellization of
OCitySMap rendering processes, we now need to provide the prefix to the
map_areas table name. We use here 'maposmaticd_PID_'.
Signed-off-by: Maxime Petazzoni <maxime.petazzoni@bulix.org>
Each time the daemon serviced a request, it would forget to close the
writing endpoint of the child-father pipe, causing it to refuse to
service more than ca. 1024 requests. This patch fixes this bug.
The configuration file for OCitySMap wasn't taken into account for
renderings based on administrative boundaries.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
By default, the maposmaticd daemon was reading the default ocitysmap
configuration (~/.ocitysmap.conf or /etc/...). This patch allows to
specify a path to another ocitysmap config file from within
settings_local.py.
Hartmut Holzgraefe