Streaming large result sets

The first version of an export endpoint used a database cursor but still accumulated encoded rows in an application buffer. It looked like streaming in a code review and behaved like batching in production.

Follow the slowest consumer

The database fetch size controls round trips, not total memory. The encoder and HTTP writer also need bounded buffers. When the socket blocks, the producer must stop reading rather than queue another chunk.

for rows.Next() {
    item := scan(rows)
    if err := encoder.Encode(item); err != nil {
        return err
    }
}

This plain loop works when the response writer applies backpressure synchronously. Extra worker pools often make the bound harder to see and rarely improve a single ordered export.

Measure the whole path

I test with a deliberately slow client, record resident memory, and cancel halfway through. Cancellation should release the cursor quickly; otherwise the endpoint keeps expensive database work alive for a client that has gone away.

For resumable exports, keyset pagination is usually clearer than holding a long transaction. For one continuous download, a read-only cursor is fine if its timeout and cleanup behavior are explicit.