40 Cities OSMnx/Networkx Dijkstra Adjacency Matrices for Traffic Prediction Graph Convolutional Networks
About
Adjacency matrices were calculated for 40 cities using sensor data from https://doi.org/10.1038/s41598-019-51539-5 (ETH Zürich). I personally rented a Google Cloud server for roughly 500€ to calculate these matrices because we have so few of them in the traffic prediction Data Science community. Since most models in the community are being trained on just these few datasets there’s risk for overfitting.
Use remove_missing_sensors.py to remove sensors that aren’t in both the matrix and data.
Repository can be found here: https://github.com/ThomasAFink/osmnx_adjacency_matrix_for_graph_convolutional_networks
Terms of Use and Data Refrences
I’m NOT affiliated with ETH Zürich nor with the Dataset UTD19. To use or publish this data, you must register and follow the terms of use, which can be found at the bottom of the dataset website UTD19 https://utd19.ethz.ch/. The website UTD19 https://utd19.ethz.ch/ and paper must be referenced “Understanding traffic capacity of urban networks” https://doi.org/10.1038/s41598-019-51539-5. It’s best to reference the school as well, Eidgenössische Technische Hochschule Zürich (ETH Zürich). 6GB data collected from the sensors is also available when signing up for UTD19 https://utd19.ethz.ch/.
Loder, A., L. Ambühl, M. Menendez and K.W. Axhausen (2019) Understanding traffic capacity of urban networks, Scientific Reports, 9 (1) 16283. https://doi.org/10.1038/s41598-019-51539-5
OpenStreetMap https://www.openstreetmap.org/ must also be referenced because the matrices where calculated using OpenStreetMap. If you use any of the Maps you must reference both OpenStreetMap and Mapbox https://www.mapbox.com/ in addition to UTD19.
Furthermore it’s probably best practice to also reference cities in publications. Additional data can often be received upon opendata requests from cities.
The code for mapping and calculating the matrices on the other hand is MIT licence.
Tutorials
Tutorials for the code have been provided on Medium by Thomas A. Fink.
Creating an Adjacency Matrix Using the Dijkstra Algorithm for Graph Convolutional Networks GCNs https://thomasafink.medium.com/creating-an-adjacency-matrix-using-the-dijkstra-algorithm-for-graph-convolutional-networks-gcns-cc84c37e297 https://github.com/ThomasAFink/osmnx_adjacency_matrix_for_graph_convolutional_networks
Plotting the Optimal Route for Data Scientists in Python using the Dijkstra Algorithm https://thomasafink.medium.com/plotting-the-optimal-route-for-data-scientists-in-python-using-the-dijkstra-algorithm-14e3e9383a0a https://github.com/ThomasAFink/optimal_path_dijkstra_for_data_science
The Graph Networks
The optimal Dijkstra distances between each sensor with every other sensor was calculated using OSMnx and NetworkX. Each city includes atleast one normalized and one original matrice with the raw values. Two example Graph Networks visualized:
The Data
The data can be used for machine learning, but not all cities have enough data. Variables include speed flow and occupancy. I have only tested the Los Angeles dataset with an LSTM-GCN model so far. Accuracy, R2 and Variance were over 90% on par with other datasets such as METR-LA and PEMS-BAY.
More Data Sources
Source: https://github.com/graphhopper/open-traffic-collection
Global
https://wiki.openstreetmap.org/wiki/TMC#Available_datasets https://unece.org/transport/transport-statistics/traffic-census-2020 https://telraam.net/#9/48.1497/11.5850 https://www.graphhopper.com/open-source/
Australia
https://data-exchange.vicroads.vic.gov.au/ https://data-exchange.vicroads.vic.gov.au/docs/services https://data-exchange.vicroads.vic.gov.au/data-exchange-platform-goes-live
Canada
British Columbia https://www.th.gov.bc.ca/trafficData/
Europe
https://data.europa.eu/data/datasets?locale=en&query=traffic&page=1 https://www.datex2.eu/
Austria:
https://contentportal.asfinag.at/data
Belgium:
https://datastore.brussels/web/- https://www.verkeerscentrum.be/uitwisseling/datex2full-
Catalonia:
http://www.gencat.cat/transit/opendata/incidenciesGML.xml
Czechia:
https://registr.dopravniinfo.cz/en/
Estonia:
https://tarktee.mnt.ee/#/en/datex
Finland:
https://www.digitraffic.fi/en/road-traffic/ https://aineistot.vayla.fi/roadworks/roadworks_d2.xml https://aineistot.vayla.fi/roadworks/roadworks_infoxml.xml https://aineistot.vayla.fi/painorajoitukset/painorajoitukset_d2.xml
France:
https://data.nasdaq.com/data/INSEE-national-institute-of-statistics-and-economic-studies-france https://opendata.paris.fr/explore/dataset/comptages-routiers-permanents/information/?disjunctive.libelle&disjunctive.etat_trafic&disjunctive.libelle_nd_amont&disjunctive.libelle_nd_aval
Germany:
https://opendata.muenchen.de/dataset?tags=Fahrrad https://www.offenedaten-koeln.de/search/type/dataset https://opendata.duesseldorf.de/dataset/verkehrsmeldungen-mobilit%C3%A4tsdaten https://opendata.jena.de/group/mobilitat https://darmstadt.ui-traffic.de/faces/TrafficData.xhtml https://suche.transparenz.hamburg.de/dataset/geo-online-portal-hamburg https://geodienste.hamburg.de/HH_WFS_Verkehr_opendata?REQUEST=GetCapabilities&SERVICE=WFS https://open.nrw/dataset/verkehrszentrale-verkehrsinformationen-der-viz-nrw-fuer-nordrhein-westfalen-1476687235163 https://open.nrw/dataset/verkehrszentrale-verkehrslage-los-1476688071631 https://www.mcloud.de/web/guest/suche/-/results/detail/verkehrsdatenautomatischedauerzhlstellen https://www.bast.de/DE/Verkehrstechnik/Fachthemen/v2-verkehrszaehlung/Daten/2017_1/Jawe2017.html?nn=1819490 https://mobilithek.info/ https://www.mdm-portal.de/migration/ https://autobahn.api.bund.dev/ https://www.mdm-portal.de/
Italy:
https://github.com/noi-techpark/BZtraffic
Lithuania:
http://restrictions.eismoinfo.lt/ https://eismoinfo.lt/#!/
Luxembourg:
https://www.cita.lu/info_trafic/datex/situationrecord
Netherlands:
http://opendata.ndw.nu/ https://gitlab.com/traffxml/vild2ltef
Norway:
Poland:
https://gitlab.com/traffxml/traff-gddkia https://kpd.gddkia.gov.pl/index.php/en/homepage/
Slovenia:
https://www.promet.si/en/plugins-for-developers
Spain:
Sweden:
https://statistik.tkgbg.se// https://www.trafikverket.se/tjanster/Oppna_data/oppna-data-vi-erbjuder/
Switzerland:
https://www.astra.admin.ch/astra/de/home/dokumentation/daten-informationsprodukte/verkehrsdaten.html
UK:
https://www.trafficengland.com/services-info https://www.traffic.gov.scot/datex/ https://www.gov.uk/traffic-counts https://www.data.gov.uk/dataset/dc18f7d5-2669-490f-b2b5-77f27ec133ad/highways-agency-network-journey-time-and-traffic-flow-data
USA
Several entries are take from this stackexchange answer
- nationwide
- Alabama
- Arizona
- Arlington
- California and https://koordinates.com/layer/109326-california-traffic-volumes-aadt/
- Chicago or one specific set
- Colorado
- Delaware Valley
- Florida
- Indiana
- Maine
- Massachusetts
- Michigan (seems to be no longer available)
- Minnesota
- New York City and New York
- North Carolina
- Ohio
- Oregon
- South Carolina
- Tennessee (seems to be no longer available)
- Washington
- Wisconsin
New York City: https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page