Narrative

Exam Submission: 8 Seconds → Under 1 Second

During exam windows, the submission endpoint was buckling. Under sustained concurrent load, it would slow down to 8 seconds per submission — which in an exam context is a real problem. Students would retry, making it worse.

PostgreSQLPythonPerformanceBackend

← Narrative Hub

// the problem

What Was Broken

// solution

How It Was Built

I profiled the request path under load. The issue was clear — every submission was doing a synchronous write to PostgreSQL, holding the connection open until the write confirmed. Under high concurrency this created a queue that compounded quickly. I reworked the flow to use batched async inserts — submissions would acknowledge immediately, and the actual DB write happened asynchronously in controlled batches. I also added a lightweight queue to buffer the burst.

// results

What Changed

Submission response time dropped from 8 seconds to under 1 second. No more retries, no more cascading slowdowns. Exams ran cleanly from that point on.

// deep dive q&a

Common Questions

The queue was persistent — if the worker failed, submissions stayed in the queue and would be retried. We also had reconciliation logic that ran after each exam window to verify submission counts matched what the queue processed.

PostgreSQL. The async batching was handled in the application layer — we used Python's async capabilities to process inserts in controlled batch sizes rather than one-by-one.