Difference between revisions of "Zaqar/bp/buffer-messages"
< Zaqar
m (→Streaming) |
m (Malini moved page Marconi/bp/buffer-messages to Zaqar/bp/buffer-messages: Project Rename) |
(No difference)
|
Latest revision as of 18:42, 7 August 2014
To buffer or to stream; that is the question.
Buffering
messages = list(msgs) {bring them all in at once}
Pros:
- Easier input validation
- Potentially more performant (iteratior protocol introduces overhead)
- Limits are known - back of the envelope calculations are available
- pymongo already forces this on us
Cons:
- More memory use
Streaming
message = next(msg) {one at a time}
Pros:
- Lower memory use
- You never have more in memory than you need
Cons:
- Makes some implementation details more difficult
- We'd have to patch pymongo to actually benefit from the pros (as I understand)