Difference between revisions of "Zaqar/bp/buffer-messages"
< Zaqar
m (→Buffering) |
m (→Streaming) |
||
Line 25: | Line 25: | ||
* Lower memory use | * Lower memory use | ||
* You never have more in memory than you need | * You never have more in memory than you need | ||
+ | |||
Cons: | Cons: |
Revision as of 14:45, 31 July 2013
To buffer or to stream; that is the question.
Buffering
messages = list(msgs) {bring them all in at once}
Pros:
- Easier input validation
- Potentially more performant (iteratior protocol introduces overhead)
- Limits are known - back of the envelope calculations are available
- pymongo already forces this on us
Cons:
- More memory use
Streaming
message = next(msg) {one at a time}
Pros:
- Lower memory use
- You never have more in memory than you need
Cons:
- Makes some implementation details more difficult
- We'd have to patch pymongo to actually benefit from the pros (as I understand)