Skip to content

ohmg-dev/OldInsuranceMaps

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Online Historical Map Georeferencer (OHMG)

OHMG is a web application that facilitates public participation in the process of georeferencing and mosaicking historical maps. Currently, a single implementation of this software exists, at OldInsuranceMaps.net, a platform based around the Sanborn Map Collection at the Library of Congress (loc.gov/collections/sanborn-maps). More generic deployments are in the works.


Please don't hesitate to open a ticket if you have trouble with the site, find a bug, or have suggestions otherwise.


Software Details

OHMG uses the Django web framework for URL routing, auth, and the ORM, and Django Ninja to create an API. A newsletter is implemented with Django Newsletter.

The frontend is built (mostly) with Svelte, using OpenLayers for all map interfaces. OpenStreetMap and Mapbox are the basemap sources.

Other components include:

  • Postgres + PostGIS
    • Database
  • GDAL
    • A dependency of PostGIS, and also used directly for all warping/mosaicking operations.
  • Celery + RabbitMQ
    • Background task management (a handful of loading/splitting/warping processes run in the background)
  • TiTiler
    • Tileserver for COGs (Cloud Optimized GeoTIFFs)

Installation

Running the application requires a number of components to be installed and configured properly. The following commands assume a Debian-based Linux distribution.

Install system dependencies

You will need a few system packages.

sudo apt install build-essential gdal-bin python3-gdal libgeos-dev libgdal-dev

Extra dependencies helpful during development:

sudo apt install graphviz graphviz-dev pre-commit

Note

3.5 is the minimum GDAL version that the app requires, so the system gdal installation must be >= or higher than that, however, the version of the Python bindings must be <= to the system version. While pinning a specific Python version is easy, anticipating the exact system gdal across distros is trickier (Debian 13: 3.10.3, Debian 12: 3.6.2, Ubuntu 24: 3.8.4, etc.). The solution here is to install whatever GDAL comes with the distro, and pin the Python bindings in pyproject.toml very low (between 3.5 and 3.6), to ensure maximum liklihood of a smooth installation.

Get the source code

Clone the repo and enter the local directory

git clone https://github.com/ohmg-dev/OldInsuranceMaps && cd OldInsuranceMaps

Install with uv

First, install uv, an all-in-one Python version and package manager.

With uv installed, run this command inside the cloned repo:

uv sync --extra dev

This will:

  1. Install the proper version of Python
  2. Create a new virtual environment in .venv
  3. Install all Python dependencies into that environment.

Next, install pre-commit hook (if you will be writing code)

pre-commit install

Use the .env.example to create your .env file

cp .env.example .env

See environment variables for more information.

Initialize database

In your running Postgres instance, create a database like this

psql -U postgres -c "CREATE USER ohmg WITH ENCRYPTED PASSWORD '$DB_PASSWORD'"
psql -U postgres -c "CREATE DATABASE oldinsurancemaps WITH OWNER ohmg;"
psql -U postgres -d oldinsurancemaps -c "CREATE EXTENSION PostGIS;"

Run migrations and create admin user to access the Django admin panel (and login to the site once it's running)

python manage.py migrate
python manage.py createsuperuser

Load a few other fixtures with some default objects:

python manage.py loaddata default-region-categories
python manage.py loaddata default-layerset-categories
python manage.py loaddata default-navbar

Alternatively, if you have set all of the DATABASE_* variables in .env, you can use the included script to perform all of the actions described above:

source ./scripts/setup_database.sh

The superuser created by this script is username: admin, password: admin.

Load test data into database (optional)

source ./scripts/load_dev_data.sh

Build frontend

There are a few js and css plugins that must be downloaded to the local static directory:

python manage.py get-plugins

The frontend uses a suite of independently built svelte components. First install pnpm: https://pnpm.io/installation. Then:

cd ohmg/frontend/svelte_components
pnpm install

During development use the following command to auto-build the components and reload your browser.

pnpm run dev

In production, use the build command instead, and then Django's collectstatic to consolidate all static assets.

pnpm run build 
cd ../../..
python manage.py collectstatic --noinput

This bash script combines all steps into one:

source ./scripts/deploy_frontend.sh

Run Django dev server

You can now activate the virtual environment and then run the django dev server:

source .venv/bin/activate
python manage.py runserver

and view the site at http://localhost:8000.

However, a few more pieces need to be set up independently before the app will be fully functional. Complete the following sections and then restart the dev server so that any new .env values will be properly acquired.

Note

You can also skip the virtualenv activation and use uv to run management commands like this:

uv run manage.py runserver

Rabbit + Celery

In development, RabbitMQ can be run via Docker like so:

docker run --name rabbitmq --hostname my-rabbit \
  -p 5672:5672 \
  -p 15672:15672 \
  -e RABBITMQ_DEFAULT_USER=username \
  -e RABBITMQ_DEFAULT_PASS=password \
  --rm \
  rabbitmq:3-alpine

For convenience, this command is in the following script:

source ./scripts/rabbit_dev.sh

Once RabbitMQ is running, update .env with the RABBITMQ_DEFAULT_USER and RABBITMQ_DEFAULT_PASS credentials you used above when creating the container.

Now you are ready to run Celery in development with:

source ./scripts/celery_dev.sh

TiTiler

TiTiler can also be run via Docker, using a slightly modified version of the official container (it is only modified to include the WMS endpoint extension):

docker run --name titiler \
  -p 8008:8000 \
  -e PORT=8000 \
  -e MOSAIC_STRICT_ZOOM=False \
  -e WORKERS_PER_CORE=1 \
  --rm \
  -it \
  ghcr.io/mradamcox/titiler:0.26.0-ohmg

Or the same command is wrapped in:

source ./scripts/titiler_dev.sh

This will start a container running TiTiler and expose it to localhost:8008.

Make sure you have TITILER_HOST=http://localhost:8008 in .env (see environment variables).

Static file server

During development, a separate HTTP server must be used to supply TiTiler with COG endpoints, because the Django dev server does not serve HTTP range requests (more on this here and here). The easiest workaround is to use node's http-server.

From within the repository root (alongside the uploaded directory) run:

npx http-server .

All COGs will now be accessible at http://localhost:8080/uploaded/.


Make sure you have LOCAL_MEDIA_HOST=http://localhost:8080 in .env. LOCAL_MEDIA_HOST is a prefix that will be prepended to any uploaded media paths that are passed to TiTiler.

In production, you will already be using a webserver for static files so you will not need to use http-server. In this case, remove LOCAL_MEDIA_HOST from .env or set it to ''.

Running tests

All tests are stored in ohmg/tests. Make sure you have installed dev requirements, then run:

python manage.py test

To skip the tests that make external calls to the LOC API, use the following command. Keep in mind that coverage numbers will be lower when you skip tests.

python manage.py test --exclude-tag=loc

Install the Place scaffolding

Load all the place objects to create geography scaffolding (this will take a minute or two)

python manage.py place import-all

Environment variables

section in progress

About

A public space for georeferencing historical fire insurance maps.

Topics

Resources

License

Stars

Watchers

Forks

Contributors 2

  •  
  •