CASE STUDY
Bulk enrichment moved onto a thread pool, with the deadlocks found and the shared state made thread-safe.
Role
Backend Developer — concurrency and data layer
Project
Product information management system
Throughput increase
40%
Query latency reduction
60%
Execution
ExecutorService · Callable · @Async
Shared state
AtomicInteger · guarded sections
CONTEXT
Product records arrive in bulk and need enriching before they are useful downstream. The read path is heavy and constant, so the database has to stay responsive while enrichment is running.
THE PROBLEM
Enriching records one at a time left the service waiting on I/O with cores idle. The first parallel implementation introduced deadlocks between threads contending for the same resources in different orders, and shared counters were being updated without synchronisation.
HOW IT WORKS
Enrichment work is dispatched across a pool of workers. Shared counters use AtomicInteger, and the sections that genuinely need mutual exclusion are the only ones that take a lock.
TRADE-OFFS
What was chosen, what it was chosen over, and why.
ConsideredA thread per task, or parallel streams
WhyEnrichment is I/O-bound, so parallelism helps — but an unbounded thread count competes for the same database connections and makes things slower. A bounded pool sized against the connection pool keeps the two in proportion.
ConsideredSynchronising every access to shared state
WhyAtomics handle the common case — a counter incremented from many threads — without the contention of a lock. Reserving synchronised blocks for the sections that genuinely need mutual exclusion keeps the fast path fast.
ConsideredLock timeouts with retries
WhyA timeout turns a deadlock into an intermittent failure and hides the design problem. Acquiring shared resources in one agreed order makes the deadlock impossible rather than recoverable.
ConsideredAdding a caching layer in front of PostgreSQL
WhyProfiling put the cost in query plans and connection churn, not in repeated identical reads. Fixing those cut latency ~60% without adding a component to operate or an invalidation problem to get wrong.
IN HINDSIGHT