Zaqar/bp/buffer-messages
< Zaqar
To buffer or to stream; that is the question.
Buffering
messages = list(msgs) {bring them all in at once}
Pros:
- Easier input validation
- Potentially more performant (iteratior protocol introduces overhead)
- Limits are known - back of the envelope calculations are available
- pymongo already forces this on us
Cons:
- More memory use
Streaming
message = next(msg) {one at a time}
Pros:
- Lower memory use
- You never have more in memory than you need
Cons:
- Makes some implementation details more difficult
- We'd have to patch pymongo to actually benefit from the pros (as I understand)
