The bulk messaging problem
National programmes routinely need to broadcast advisories to tens of thousands of recipients in a single operation. A climate alert, a voucher distribution, a programme registration window, a market price update.
The naive implementation, a synchronous loop that calls the messaging gateway once per recipient, fails almost immediately. Web requests time out. Gateway rate limits trigger. Partial sends leave the operator unsure which messages went out and which did not. Retries duplicate messages and erode trust with the very users the programme is trying to reach.
The pattern
The correct pattern is a queue and worker architecture. Simple to describe and unforgiving in the details.
- When an operator triggers a broadcast, the application does not call the gateway. It writes one row per recipient into a queue table with status pending. The web request returns immediately with a queue identifier.
- A separate background worker, scheduled to run every minute, pulls a small batch of pending rows, locks them as processing, calls the gateway, and updates each row to sent or failed based on the gateway response.
- Delivery reports arrive asynchronously through webhooks and update the rows again to delivered, failed, or unreachable, with the carrier and cost captured for analytics.
The details that matter
The pattern is simple in outline and unforgiving in execution. The differences between a system that works on day one and one that holds up for years come down to a small set of decisions.
- Batch size. Tuned to the gateway rate limit, with back pressure when the limit is approached.
- Locking. Survives worker crashes. A row stuck in processing must be reclaimable by another worker after a timeout, without duplicating the send.
- Idempotency keys. So retries do not duplicate messages, even when the network fails between the gateway and the worker.
- Dead letter queues. For permanent failures, with a clear operator surface to inspect and resolve them.
- Idempotent webhook handlers. Providers retry, sometimes aggressively. The handler must accept the same event multiple times without corrupting state.
- Operator dashboards. Showing pending, processing, sent, delivered, and failed counts in real time, so programme staff know exactly where a broadcast stands.
What we deployed for FAO Zambia
PANEOTECH implemented this pattern in the Integrated Stakeholder Engagement Platform delivered to FAO Zambia under the Sustainable Intensification of Smallholder Farming Systems programme, in partnership with the European Union, the Ministry of Agriculture, and the International Maize and Wheat Improvement Centre.
The system handles bulk broadcasts of up to 50,000 recipients per operation across MTN, Airtel, and Zamtel, with consolidated delivery reporting that includes carrier identification and cost per message. The same pattern serves the WhatsApp and email channels, with channel specific worker logic that respects each provider's rate and template constraints.
The operational benefit
For institutional programmes the operational benefit is decisive. Programme staff can fire a 50,000 recipient advisory before lunch, watch delivery progress through the dashboard, and have full audit data for the donor report by the end of the week. The architecture does the boring work, reliably, every time.
