Backlog #326 — Route Data Sources Implementation Status¶
Last updated: 2026-06-09 Status: ✅ PHASE 1 READY TO EXECUTE
Executive Summary¶
#326 Phase 1 (Public Sources): ✅ All scrapers implemented and ready to deploy
| Source | Status | Routes | Effort | Next Step |
|---|---|---|---|---|
| KNHS Ruiternetwerk ⭐ | 📧 Waiting | 650+ | Send WFS email | Enable WFS on endpoint |
| Toerisme Vlaanderen | ✅ Ready | 50-100 | Run scraper | Execute after approval |
| Staatsbosbeheer | ✅ Ready* | 5-20 | Run scraper | Verify GPX URLs |
| OSM Bridleways | ✅ Ready | 200-500 | Run scraper | Run immediately |
| Routedatabank ⏳ | 📧 Waiting | 2000+ | Send email | Approval pending |
Implementation Details¶
✅ DONE: Working Scrapers (Ready to Deploy)¶
1. OSM Bridleways — Ready NOW (no approval needed)¶
python3 scripts/import_osm_bridleways_fixed.py --firestore
2. Toerisme Vlaanderen WFS — Ready NOW (public data)¶
python3 scripts/import_toerisme_vlaanderen_fixed.py --firestore
3. Staatsbosbeheer GPX Scraper — Ready (URLs need verification)¶
python3 scripts/import_staatsbosbeheer_fixed.py --firestore
4. Master Runner — Ready NOW¶
# Dry-run preview:
python3 scripts/run_public_importers.py
# Actually import to Firestore:
python3 scripts/run_public_importers.py --firestore
⏳ WAITING: Email Approvals (Critical Path)¶
KNHS Ruiternetwerk ⭐ (PRIORITY #1 — Fastest)¶
- Status: Email template ready
- What we need: WFS enablement on existing GeoServer
- Expected response: 1-3 days
- Routes: 650+ official Dutch routes
- Email:
docs/knhs_wfs_email_nl.md - Importer:
scripts/import_knhs_ruiternetwerken.py(ready)
# Once WFS is enabled:
python3 scripts/import_knhs_ruiternetwerken.py --firestore
Routedatabank WFS (PRIORITY #2 — Comprehensive)¶
- Status: Email template ready
- What we need: WFS access credentials or public endpoint
- Expected response: 1-2 weeks
- Routes: 2000+ additional Dutch routes
- Email:
docs/routedatabank_email_nl.md - Importer: Will create after approval
# Once approved:
python3 scripts/import_routedatabank_wfs.py --firestore
Architecture¶
All scrapers follow the same pattern:
SOURCE API
↓
fetch_data() → parse_to_route_object()
↓
{
name: "Route Name",
distanceKm: 12.5,
points: [{lat, lng}, ...],
sourceCountry: "nl|be|de|fr",
sourceType: "osm-bridleway|toerisme-vlaanderen|staatsbosbeheer|knhs|routedatabank",
sourceUrl: "https://...",
importedAt: "2026-06-09T13:15:00Z"
}
↓
Firestore batch.set(doc, merge=True)
↓
community_trails collection
Key feature: merge=True = idempotent (safe to re-run)
Quick Start¶
To import public sources RIGHT NOW:¶
# 1. Preview what will be imported (no Firestore writes)
python3 scripts/run_public_importers.py
# 2. Actually import to Firestore
python3 scripts/run_public_importers.py --firestore --force
# 3. Verify in Firebase Console
# https://console.firebase.google.com/project/equitrail/firestore/data/community_trails
To import KNHS (after email approval):¶
# Check if WFS is enabled
python3 scripts/import_knhs_ruiternetwerken.py
# If enabled, import
python3 scripts/import_knhs_ruiternetwerken.py --firestore
To import Routedatabank (after approval):¶
# Create importer (template same as others)
cp scripts/import_knhs_ruiternetwerken.py scripts/import_routedatabank_wfs.py
# Update WFS endpoint + feature types
python3 scripts/import_routedatabank_wfs.py --firestore
Expected Route Totals¶
Phase 1 (Public, Run NOW):¶
Current: 3,429 (OSM named circuits)
+ OSM BW: 300 (bridleways)
+ Toerisme: 75 (Flanders)
_______________________
Phase 1: 3,804 routes
Phase 2 (After KNHS approval):¶
Phase 1: 3,804
+ KNHS: 650 (official Dutch network)
_______________________
Phase 2: 4,454 routes
Phase 3 (After Routedatabank approval):¶
Phase 2: 4,454
+ Routedatabank: 2,000 (additional NL routes)
_______________________
Phase 3: 6,454 routes ✅ TARGET
Next Session Checklist¶
- Send KNHS WFS email (docs/knhs_wfs_email_nl.md)
- Send Routedatabank email (docs/routedatabank_email_nl.md)
- Run public importers:
python3 scripts/run_public_importers.py --firestore - Verify routes in Firestore Console
- Check website (routes should appear automatically via cached JavaScript)
- Monitor approval responses from KNHS + Routedatabank
Files Created¶
Importers (ready to use):
- scripts/import_osm_bridleways_fixed.py
- scripts/import_toerisme_vlaanderen_fixed.py
- scripts/import_staatsbosbeheer_fixed.py
- scripts/import_knhs_ruiternetwerken.py
- scripts/run_public_importers.py (master runner)
Email templates (ready to send):
- docs/knhs_wfs_email_nl.md
- docs/routedatabank_email_nl.md
Technical Details¶
- Firestore writes: Batch (500 at a time),
merge=True(idempotent) - Error handling: Try/except on each API call, graceful fallback
- Rate limiting: 1-2s delay between API calls (Overpass, WFS, GPX)
- Data format: Standard route object (name, distanceKm, points[], sourceCountry, sourceType, sourceUrl, importedAt)
- Deduplication: Document ID =
country_source_hash(name)prevents duplicates on re-run
Known Issues & Fixes Needed¶
- Staatsbosbeheer GPX URLs — Need to verify current URLs at staatsbosbeheer.nl
- Current URLs may have changed or require different naming
-
Solution: Visit website, find correct GPX download links
-
Toerisme Vlaanderen WFS — May need to adjust feature type names
- Currently uses
routes:traject(confirmed working) -
Alternative names if needed:
routes:knoop,routes:icoonroute_trajecten -
OSM Overpass — Can be slow for large regions
- Current: queried per region (manageable)
- Optimization: Could be split into smaller bbox if needed
Success Metrics¶
✅ Phase 1 complete when:
- [ ] 3,800+ routes in Firestore community_trails
- [ ] Website /routes/ pages show route counts increasing
- [ ] No Firestore errors in Cloud Logging
✅ Phase 2 complete when: - [ ] KNHS approves WFS enablement - [ ] 650+ additional KNHS routes imported - [ ] Total: 4,450+ routes
✅ Phase 3 complete when: - [ ] Routedatabank provides WFS access - [ ] 2000+ Routedatabank routes imported - [ ] Total: 6,450+ routes (TARGET)