Elevation: DIY API with Open Topo Data

For the next iteration of ExploreHere, I’m working to give every square mile of the world a story that describes the land’s history, ecosystem, geology, flora/fauna, etc. By combining dozens of databases that describe a single aspect of this story, I’m hoping to generate a digital twin of the world.

One critical piece of this story is elevation.

Private Elevation API’s

Google’s API is expensive, $5 per 1000 requests. I estimate I’ll need about 1 million elevation points, so using Google would cost about $5k. That’s significantly more than I want to spend on elevation data.

Public Elevation Data

There are several techniques for generating elevation data; photogrammetry, LiDAR, GNSS, InSAR, and many more. Each technique has strengths and weaknesses, but there are a handful of public elevation datasets that cover various parts of the world in different resolutions.

A company, Mapzen, aggregated all these different datasets across the world into a single unified tile-set of various resolutions and open sourced the tileset. This tileset makes it possible to get the best data anywhere in the world.

Since this tileset covers the entire world, the files size is large; ~2+ TB of data. Since I only need the US, I deleted all tilesets outside of the bounding box of the US.

Running Open Topo Data Locally

Running the server locally was really straight forward; as the instructions say, you just install docker and git and then run the following commands.

$ git clone https://github.com/ajnisbet/opentopodata.git
$ cd opentopodata
$ make build
$ make run

If you’re on a mac with an M1 chip, you’ll want to run ‘make build-m1’.
Once its running, you can open http://localhost:5000/v1/test-dataset?locations=-10,120 and see a test result.

Deploying Open Topo Data to a Remote Server

Getting it running locally was a great start, but it would be really nice to have a service that I could spin up and use in production. I’ve heard great things about deploying on fly.io, so let’s get that set up!

Since opentopodata includes a Docker container, it’s pretty straightforward getting it deployed on a remote server.

Get the fly.io CLI set up on your local machine

Edit and Create Files

Update the config.yml in your local project to contain the following. Checkout the opentopodata docs to learn how to add other datasets. I scoped mine to a mapzen dataset with only the UnitedStates.

max_locations_per_request: 100
datasets:
  - name: mapzen
    path: data/mapzen/UnitedStates

Create a fly.toml file in the root that contains the following code. Make sure to swap out ‘app_name’ for the name you want the app to be called.

app = "app_name"
primary_region = "iad"

[build]
  dockerfile = "docker/Dockerfile"

[env]
  PORT = "5000"

[[mounts]]
  source = "data"
  destination = "/app/data"

[[services]]
  protocol = "tcp"
  internal_port = 5000

  [[services.ports]]
    port = 80
    handlers = ["http"]

  [[services.ports]]
    port = 443
    handlers = ["tls", "http"]
  [services.concurrency]
    hard_limit = 25
    soft_limit = 20

  [[services.tcp_checks]]
    interval = "10s"
    timeout = "2s"
    restart_limit = 0

Add a Procfile in the root of your project that contains the following:

web: gunicorn opentopodata:app

Deploy

We can now deploy the app to fly.io. First, log in to the fly.io CLI, then launch the application.

$ fly auth login
$ fly launch

We’ll use the S3 API to download our tileset into the new volume we created. Our new VM needs a bit more memory to run the download command, so let’s increase it a bit. This VM will cost $5.70/month

$ fly scale memory 1024

Since the application reads data from a local directory, we need to set up a fly.io volume to put the map tiles into.

$ fly volume create data

You’ll also need to allocate enough space on the VM to fit the tileset you’re looking to put on the volume. The US tileset I made is about 160GB. Fly.io charges $.15/GB, so this will cost $24/month. You can get the volume_id under the fly.io dashboard /volumes. Fly.io will allow you to upgrade volume space, but you can’t downgrade space after it’s been set. The only way would be to delete the volume and start with a new one.

$ fly volumes extend put_volume_id_here -s 160

We have the VM set up with a volume large enough to hold our dataset. We should be able to visit our project’s public URL and see an example response from the app. Now we want to get the real data onto our volume so our app can be used. First, let’s SSH into our VM.

$ fly ssh console

Let’s install the AWS CLI, as it’s the fastest way to get our tileset downloaded into our volume, then let’s configure the CLI.

$ pip install awscli
$ aws configure

Navigate to the volume from the root and download the data from S3. The path you use for the app/data needs to match the dataset’s path from the config.yml we deployed earlier.

$ cd app/data
$ mkdir mapzen
$ cd mapzen

$ aws s3 cp s3://your-s3-bucket/UnitedStates UnitedStates --recursive

Conclusion

That’s it! We have a publicly deployed opentopodata server on fly.io for less than $30/month!

For production loads, you might need to keep an eye on the CPU usage and adjust the VM size depending on the server load.

Moving forward, this will become the base server for elevation data in ExploreHere. I plan to pre-compute elevations for each hex index and hit the API in real-time when users request their elevation in the app. Huge thanks to Andrew Nisbet for all his work done on opentopodata. If managing your own instance isn’t a good fit for your use case, or if you want even higher resolution (1m LiDAR), check out Andrew’s side project GPXZ or reach out to andrew@opentopodata.org for more details!