Org Web Adapter

Editing decks/patha-presentation.org

#+OPTIONS: num:nil toc:nil
#+REVEAL_TRANS: Fade
#+REVEAL_THEME: Black
#+Title: About Colin
#+Author: colin@onec.me | @secsate@mastodon.technology
#+Email: colin@onec.me

* Me
** Family
Live in Castine, ME

Four acre farm; chickens, ducks, sheep, dairy goats

Four kids (GBBG); ages 8 to 9 month
** Fun & community
Endurance runner & swimmer

Summer league ultimate

Selectboard member in town

Volunteer @ county jail library

Reading
** Career
Newspaper CMS (local Maine newspaper)

Agency life @ Five Q

Transportation + devops @ Ridecell

Electronic health records @ Elation Health
** Fit @ PathAI
Passion for improving healthcare, esp. in rural areas

Driven by delivering value to users

Endlessly curious
* Challenge
** Enabling features considered hazerdous?
** Django REST Framework's
BrowseableAPIRenderer

/Allows user-friendly in-browser viewing and interaction of a DRF API/
** One day Elation's app went down
Peaking at CloudWatch, it was clear the database was taking a beating
** Weekly release had gone out days earlier
Because of weekly release cadence, the release itself didn't seem like the problem

Took a while to decide to rollback

** Roll back the release
Production back and stable, load on MySQL relaxes
** Five whys
Scheduled next day to figure out how to get release out
** Release enabled the DRF BrowsableAPIRenderer
Change made to allow internal API exploration of new API by support team.

Decision was code reviewed by peer, and, as with all settings.py changes, cleared by at least two lead engineers.
** Traffic logs provide evidence
Looking at traffic logs, I discovered a specific API endpoint response time blowing up

Correlated the API calls with the outage time and speculated on the causation
** Turns out ...
Many healthcare providers are not great API users
** A single provider was hitting our endpoint every 5s
Regardless of whether anything had changed!
** But that shouldn't have caused the issue ...
So long as they were sending JSON Accept-header ...
** But they weren't ...
They were using Accept-header: text/html  :(
** But why did HTML kill our production app?
Deep in the bowels of MySQL's slow queries log, we discovered a clue

DRF's Browsable API contains a number of auto-generated widgets for connecting objects

Massive SQL queries from complex joins to build HTML dropdowns for relating models
** Solutions
1. Ask API abuser to cease and desist (or at least, for God's sake send a proper
   Accept header)
2. Disable BrowsableAPIRenderer for all time
3. Force certain APIs used by vendors (not our frontend) from responding with
   JSON regardless of the request
** Force JSON response
Solution worked because we had a limited set of APIs used by external vendors

BrowseableAPIRenderer could still be used by support team

Disabling BrowseableAPIRenderer could not stop us from making same mistake again

API abusers were companies many times larger than us with tech teams less able
to be responsive, and so were unlikely to change their behavior.
** Thank you!