Improving OpenStreetMap by leveraging open data

# Improving OpenStreetMap by leveraging open data

Ivan Habunek

DORS/CLUC 2024

---

## What is OpenStreetMap?

<div style="display: grid; grid-template-columns: auto 1fr; align-items: center; gap: 2rem;">
  <img src="images/osm_logo.svg" style="width: 15rem" />
  <div>
      <p>OpenStreetMap is a free, editable map of the whole world that is being built by volunteers largely from scratch and released with an open-content license.</p>
      <p><a href="https://openstreetmap.org/">https://openstreetmap.org/</a></p>
  </div>
</div>

???

Think Wikipedia, but maps.

---

## Who are we?

<div style="display: grid; grid-template-columns: auto 1fr; align-items: center; gap: 2rem;">
  <img src="images/osm_hr_logo.svg" style="width: 15rem" />
  <div>
      <h3>OSM Hrvatska</h3>
      <p>Croatian OSM user group</p>
      <p><a href="https://osm-hr.org/">https://osm-hr.org/</a></p>
  </div>
</div>

???

I'm a member of OSM Hrvatska, a group of mapping enthusiasts.

---

## Use case: Mapping public transport

???

I'd like to focus on mapping public transport such as tram and bus routes.

The state of public transport data was not the best.

To better explain this, here's a crash course in mapping public transport in OSM.

---

## OpenStreetMap elements

<p class="element"><img src="images/node.svg"> Node</p>
<p class="element"><img src="images/way.svg"> Way</p>
<p class="element"><img src="images/relation.svg"> Relation</p>

???

* node - defines a points in space
  * e.g. mountain peak, cape, tree, bench...
* way - ordered list of nodes
  * linear features, e.g. roads, rivers, boundaries
* relation - collection of nodes, ways and other relations
  * how members work together
  * e.g. turn restrictions, tram/bus routes
* each element has key-value tags!
* stored in a database

---

## How to map a bus route?

---

???

Step 1: Map platforms in the route

---

???

Step 2: Map the ways along the route

* roads for buses
* rails for trams

---

### JOSM<br />Editor 🡒

???

Step 3: Create a relation containing the platforms and the roads

---

### JOSM<br />Editor 🡒

???

Tags

---

### JOSM<br />Editor 🡒

---

### JOSM<br />Editor 🡒

---

## Problem: This is a lot of work

Zagreb alone has:

* 19 tram routes
* ~150 bus routes
* ~2700 platforms

---

## Problem: Local knowledge required

Need to know:

* platform locations
* bus/tram routes

???

* operator websites usually don't list exact locations of stops

---

???

Source:
https://autobusi.org/forum/index.php?topic=13108.0

---

## What we want

Create a tool which helps us:

1. Locate stops and routes
2. Assist with data entry
3. Find errors in OpenStreetMap
4. Detect route changes

---

## General Transit Feed Specification (GTFS)

https://www.zet.hr/odredbe/datoteke-u-gtfs-formatu/669

https://gtfs.org/

???

* around that time ZET started publishing it's schedule in GTFS format
* GTFS is a data format devised by google
* primarily used to import data into google maps
* fortunately, google is not the only one who can play this game

---

## GTFS files

* stops.txt
* routes.txt
* trips.txt
* stop_times.txt

https://gtfs.org/schedule/reference/

---

### GTFS: stops.txt

<pre class="codeblock" style="font-size: 1rem">
stop_id,stop_code,stop_name,stop_desc,stop_lat,stop_lon,zone_id,stop_url,location_type,parent_station
"98_1","1","Črnomerec",,45.815002,15.934932,,,0,98
"98_2","2","Črnomerec",,45.815398,15.933924,,,0,98
"98_11","11","Črnomerec",,45.815127,15.934536,,,0,98
"98",,"Črnomerec",,45.815175,15.934464,,,1,
"99_23","23","Črnomerec",,45.815318,15.933231,,,0,99
"99_26","26","Črnomerec",,45.814933,15.933877,,,0,99
"99_31","31","Črnomerec",,45.816087,15.933219,,,0,99
"99_32","32","Črnomerec",,45.816059,15.933327,,,0,99
"99_36","36","Črnomerec",,45.814943,15.934140,,,0,99
"99_42","42","Črnomerec",,45.815739,15.933910,,,0,99
"99_52","52","Črnomerec",,45.815800,15.934131,,,0,99
"99_62","62","Črnomerec",,45.815837,15.934329,,,0,99
"99_75","75","Črnomerec",,45.815703,15.933606,,,0,99
"99_82","82","Črnomerec",,45.815622,15.933777,,,0,99
"99_85","85","Črnomerec",,45.815799,15.933997,,,0,99
"99",,"Črnomerec",,45.815622,15.933777,,,1,
"100_1","1","Sveti Duh",,45.813297,15.943123,,,0,100
"100_2","2","Sveti Duh",,45.813639,15.941471,,,0,100
"100",,"Sveti Duh",,45.813468,15.942297,,,1,
"101_1","1","Mandaličina",,45.812427,15.948326,,,0,101
"101_2","2","Mandaličina",,45.812493,15.948860,,,0,101
</pre>

???

* contains stop locations for tram/bus/funicular stops
* each stop has a unique ID (this will matter later)
* I will be using "stop" and "platform" interchangably

---

> #### 2.1. I would like to import data into the OSM map, can I just go ahead?
> There are strict rules about what can and cannot be imported into the OSM map and these rules depend upon the source of ownership and license terms relating to such data.

https://osmfoundation.org/wiki/Licence/Licence_and_Legal_FAQ

---

## Open License

> This Open License grants the Reuser any reuse of the Information subject to the License, including a spatially and temporally unrestricted, free-of-charge, non-exclusive and personal right to reuse the Information under this License.<br/>
https://prod-data.gov.hr/en/open-license

Compatible with OSM<br />
https://www.openstreetmap.org/copyright

---

<div class="row">
    <img src="images/geofabrik_map.jpg" style="width: 100%"/>
    <div>
      <h3>Geofabrik download server</h3>
      <blockquote>
        This server has data extracts from the OpenStreetMap project which are normally updated every day.
      </blockquote>
      <p style="font-size: 90%"><a href="http://download.geofabrik.de/europe/croatia.html">http://download.geofabrik.de/europe/croatia.html</a></p>
    </div>
</div>

???

* OSM is a database
* Loading the whole planet file is not an option (74 GB)
* Geofabrik provides data by country
* Also provides daily diffs

---

## Load data

???

* CSV is easy to parse, this is done in code
* Osmosis is an open source tool which supports loading OSM data from PBF to a custom PostgreSQL schema
* Now we have both data sets in PostgreSQL and the next task is to compare them

---

## Data loaded

Now we can compare GTFS and OSM data using SQL!

---

## Matching platforms

Problem: How to match GTFS stops with corresponding OSM nodes?

<p>Solution: Use the <code>gtfs:stop_id</code> tag</p>

https://wiki.openstreetmap.org/wiki/Key:gtfs:stop_id

<div class="row">
  <div>
    GTFS stops.txt
    <img src="images/stops_txt_sample_marked_2.png" style="width: 100%"/>
  </div>
  <div>
  </div>
</div>

???

* the first problem encountered was how to match platforms in GTFS with platforms in OSM
* first idea - by proximity - turned out to be inexact
* we solved this by adding a `gtfs_id`

---

## Matching platforms

Problem: How to match platforms in GTFS data with corresponding OSM nodes?

<p>Solution: Use the <code>gtfs:stop_id</code> tag</p>

https://wiki.openstreetmap.org/wiki/Key:gtfs:stop_id

<div class="row">
  <div>
    GTFS stops.txt
    <img src="images/stops_txt_sample_marked_2.png" style="width: 100%"/>
  </div>
  <div>
    OSM node tags
    <img src="images/platform_tags_marked.png" style="width: 100%"/>
  </div>
</div>

???

* the first problem encountered was how to match platforms in GTFS with platforms in OSM
* first idea - by proximity - turned out to be inexact
* we solved this by adding a `gtfs_id`

---

## Matching platforms

Problem: how to set `gtfs:stop_id` tag on all existing platforms?

<p>Solution: <b>OSM Conflator</b> by Ilya Zverev</p>

https://wiki.openstreetmap.org/wiki/OSM_Conflator

???

* a python script to merge a third-party dataset with coordinates and OpenStreetMap data.
* Ilya Zverev is a name which crops up a lot in the OSM world

---

## Matching platforms

Problem: Imperfect GTFS data

* one platform with multiple `gtfs:stop_id`

* multiple platforms with the same `gtfs:stop_id`

???

Ideally we would have 1 GTFS stop 1 OSM platform

---

#### https://triglav.bezdomni.net/

Logo by Nela Dunato

---

???

We can display locations of all stops

---

???

We can display public transport routes

---

## What we want

<span class="check">☑</span> Locate stops and routes

<span class="gray"><span class="check">☐</span> Assist with data entry</span>

<span class="gray"><span class="check">☐</span> Find errors in OpenStreetMap</span>

<span class="gray"><span class="check">☐</span> Detect route changes</span>

---

### JOSM<br/>remote<br />control

???

JOSM can be controlled via HTTP.
This allows Triglav to load platforms, zoom to platforms.

---

### JOSM<br/>remote<br />control

---

### JOSM<br/>remote<br />control

---

### JOSM<br/>remote<br />control

???

Remote control can also add missing stops.

---

## What we want

<span class="check">☑</span> Locate stops and routes

<span class="check">☑</span> Assist with data entry</span>

<span class="gray"><span class="check">☐</span> Find errors in OpenStreetMap</span>

<span class="gray"><span class="check">☐</span> Detect route changes</span>

---

## Some errors we can detect

* Missing route relations
* Incorrect stops
* Broken routes
* One-way roads traversed in wrong way
* Platforms too far from route

---

???

We can detect and display errors per route

---

## What we want

<span class="check">☑</span> Locate stops and routes

<span class="check">☑</span> Assist with data entry

<span class="check">☑</span> Find errors in OpenStreetMap

<span class="gray"><span class="check">☐</span> Detect route changes</span>

---

## Nightly update

* OSM data from Geofabrik
* GTFS data from ZET
* regenerate errors

<pre class="codeblock" style="margin-top: 10rem; color: gray">
$ crontab -l
40 * * * * /home/ihabunek/projects/triglav/osmosis_update
0 5 * * * /home/ihabunek/projects/triglav/pt_update
</pre>

---

---

## What we want

<span class="check">☑</span> Locate stops and routes

<span class="check">☑</span> Assist with data entry

<span class="check">☑</span> Find errors in OpenStreetMap

<span class="check">☑</span> Detect route changes

---

<div class="row">
  <div>
    <h3>OsmAnd</h3>
    <p>Offline Maps and Navigation</p>
    <a href="https://osmand.net/">https://osmand.net/</a>
  </div>
  <img src="images/osmand_1.png" style="height: 80vh"/>
  <img src="images/osmand_2.png" style="height: 80vh"/>
</div>

---

---

---

## What next?

---

---

---

### OSM Hrvatska

https://osm-hr.org/

https://community.openstreetmap.org/

Telegram: @osmhr

### Ivan Habunek

https://ivan.habunek.com/

Mastodon: @ihabunek@mastodon.social

Twitter: @ihabunek