#+OPTIONS: num:nil toc:nil
#+REVEAL_TRANS: Fade
#+REVEAL_THEME: Black
About Colin
#+Author: colin@onec.me | @secsate@mastodon.technology
#+Email: colin@onec.me
Me
Family
Live in Castine, ME
Four acre farm; chickens, ducks, sheep, dairy goats
Four kids (GBBG); ages 8 to 9 month
Fun & community
Endurance runner & swimmer
Summer league ultimate
Selectboard member in town
Volunteer @ county jail library
Reading
Career
Newspaper CMS (local Maine newspaper)
Agency life @ Five Q
Transportation + devops @ Ridecell
Electronic health records @ Elation Health
Fit @ PathAI
Passion for improving healthcare, esp. in rural areas
Driven by delivering value to users
Endlessly curious
Challenge
Enabling features considered hazerdous?
Django REST Framework's
BrowseableAPIRenderer
/Allows user-friendly in-browser viewing and interaction of a DRF API/
One day Elation's app went down
Peaking at CloudWatch, it was clear the database was taking a beating
Weekly release had gone out days earlier
Because of weekly release cadence, the release itself didn't seem like the problem
Took a while to decide to rollback
Roll back the release
Production back and stable, load on MySQL relaxes
Five whys
Scheduled next day to figure out how to get release out
Release enabled the DRF BrowsableAPIRenderer
Change made to allow internal API exploration of new API by support team.
Decision was code reviewed by peer, and, as with all settings.py changes, cleared by at least two lead engineers.
Traffic logs provide evidence
Looking at traffic logs, I discovered a specific API endpoint response time blowing up
Correlated the API calls with the outage time and speculated on the causation
Turns out ...
Many healthcare providers are not great API users
A single provider was hitting our endpoint every 5s
Regardless of whether anything had changed!
But that shouldn't have caused the issue ...
So long as they were sending JSON Accept-header ...
But they weren't ...
They were using Accept-header: text/html :(
But why did HTML kill our production app?
Deep in the bowels of MySQL's slow queries log, we discovered a clue
DRF's Browsable API contains a number of auto-generated widgets for connecting objects
Massive SQL queries from complex joins to build HTML dropdowns for relating models
Solutions
1. Ask API abuser to cease and desist (or at least, for God's sake send a proper
Accept header)
2. Disable BrowsableAPIRenderer for all time
3. Force certain APIs used by vendors (not our frontend) from responding with
JSON regardless of the request
Force JSON response
Solution worked because we had a limited set of APIs used by external vendors
BrowseableAPIRenderer could still be used by support team
Disabling BrowseableAPIRenderer could not stop us from making same mistake again
API abusers were companies many times larger than us with tech teams less able
to be responsive, and so were unlikely to change their behavior.