Skip to content

Latest commit

 

History

History
159 lines (118 loc) · 7.56 KB

learning_track.md

File metadata and controls

159 lines (118 loc) · 7.56 KB
Raw

This guide tracks useful steps to learn how to maintain and modify this system.

Discover the PAN public website

Discover the PAN admin dashboard

  • Ask the team to get access to the admin dashboard
  • Expect a bit of delay / cache refresh period
  • Get into the dashboard and go around (via the "Administration" top link)

Run the PAN site locally (simple version)

  • Install the required tooling (Elixir/Erlang/Node/Postgres) - see readme
  • Restore a production database - see readme
  • Do not attempt to install the "validator" yet, nor to access the admin backoffice

Run the test suite locally

  • Make sure to run ChromeDriver in a way or another
  • Run the test suite with mix test
  • Learn how to run a single test (see readme), as this is very useful for debugging
  • ⚠️ All the tests should pass locally! If they don't, file an issue

Different techniques to debug stuff

(More will come here later, especially with Elixir 1.15+)

  • IO.puts(x)
  • IO.inspect(x, IEx.inspect_opts)
  • dbg + iex -S mix phx.server
  • find apps/transport | entr -c mix run my_script.exs
  • @tag :focus and find debug.exs apps/transport/{lib,test} | entr -c mix cmd --app transport mix test --color --only focus
  • mix test apps/transport/test/transport/import_data_test.exs --only focus
  • doctest ImportData, import: true, tags: [:focus]
  • elixir --sname node -S mix phx.server and iex --sname console --remsh node (https://github.com/etalab/transport-site/pull/2960/files) to connect to a running node and make evaluations (useful to inspect ETS state for instance)
  • LiveBook in non-standalone mode (create a notebook then switch from standalone to connected in the settings)

Understand the "stats" page

  • Check out the /stats page, entry point for bizdev questions on data quality
  • Look at _maps.html.eex and map.js
  • Search the code responsible for quality_features_query

Run a manual GTFS validation (on the server)

Discover the HTTP routes served by the application

  • Run mix phx.routes TransportWeb.Router locally
    • Examine the listed routes
    • Check-out apps/transport/lib/transport_web/router.ex where they are defined
  • Check-out apps/transport/lib/transport_web/plugs/router.ex (/api, /gbfs & the rest)
    • This top-level router is referred to in apps/transport/lib/transport_web/endpoint.ex
  • Run mix phx.routes TransportWeb.API.Router (this will list all the /api sub-routes)
  • Run mix phx.routes GBFS.Router (same for /gbfs)
  • In short: the "endpoint" includes a main router, which in turn includes 3 sub-routers

Read the logs from the production database

  • The site is deployed on CleverCloud
  • Install clever-tools
  • clever login
  • clever --help
  • Go to your local transport-site git clone
  • clever link $$REPLACE_BY_APP_ID$$ (pick app_id in the CleverCloud dashboard for transport-site)
  • clever status
  • clever logs --help
  • clever logs to stream the current logs
  • clever logs --addon $$REPLACE_BY_PG_ADDON_ID$$ (pick addon_id at top-right of CC dashboard for transport-site-postgresql "Information" tab)

Learn how to deploy the Elixir app on staging (aka "prochainement")

  • Use a force push of your branch, e.g. git push <remote> <branche>:prochainement -f (so if your branch is some-feature, this will usually be: git push origin some-feature:prochainement -f)
  • This will trigger a redeploy. Redeploy process can be monitored from the CleverCloud dashboard and takes roughly 5 to 10 minutes.
  • If you see errors in the CC app logs due to Ecto migrations (due to divergence of branches), you'll want to reset the staging database (see below)

Learn how to reset the staging (aka "prochainement") database

  • Go to the CleverCloud dashboard for the production Postgres database and download it locally
  • Read the restore_db.sh script
  • Go to the CleverCloud dashboard for the staging Postgres database, and run restore_db.sh with proper parameters

Learn how to connect via SSH

  • Make sure to link the correct app (production or staging) with clever link $$REPLACE_BY_APP_ID$$ (as displayed in the staging/production app CC dashboards)
  • Verify the linking status with clever applications
  • Log with the app alias: clever ssh --alias transport-prochainement

Explore the GBFS conversion proxy

  • Check out the code under apps/gbfs
  • Run the app locally with mix phx.server
  • Discover the available converted feeds via http://localhost:5000/gbfs
  • Note that this does not contain all the GBFS feeds, only the ones we convert

Learn about the GTFS and GTFS-RT specifications

Explore the GTFS validator locally

Run the GTFS validator via the Elixir app locally

  • Compile the validator project with cargo build --release
  • Run it as a server with ./target/release/main
  • Override in dev.secret.exs the gtfs_validator_url defined in dev.exs with the displayed host & port (e.g. http://127.0.0.1:7878)
  • Run the site with mix phx.server
  • Go to http://localhost:5000/validate
  • Upload a GTFS file
  • Verify that it goes through the Elixir apps logs, then the validator logs

Debug the GTFS validator locally

  • Make sure to have Rust Analyzer installed
  • If you use VSCode, install the CodeLLDB extension to help with debugging
  • Run "Debug unit tests in library 'validator'"
  • Add a breakpoint and verify that it is effective
  • Read this article for more information

Run the realtime API locally

Install the GTFS to NETEX converter locally

IDEAS for the next steps

  • Understanding the code behind #1373
  • Overall architecture (diagram by Francis)
  • Structure of the Umbrella app (apps)
  • Measuring code coverage
  • Discovering the database structure
  • Clever Cloud deployment and operations (Sentry, UptimeRobot)
  • Running import jobs (locally)
  • Diving into import jobs (locally)
  • What is GBFS (workshop, slides)
  • How to upgrade Elixir, Erlang and Node
  • How to launch linters etc (like CI does)