Difference between revisions of "Zaqar/bp/buffer-messages"
< Zaqar
(Created page with "To buffer or to stream; that is the question. === Buffering === messages = list(msgs) {bring them all in at once} Pros: * Easier input validation * Potentially more perfo...") |
m (Malini moved page Marconi/bp/buffer-messages to Zaqar/bp/buffer-messages: Project Rename) |
||
| (2 intermediate revisions by one other user not shown) | |||
| Line 11: | Line 11: | ||
* Limits are known - back of the envelope calculations are available | * Limits are known - back of the envelope calculations are available | ||
* pymongo already forces this on us | * pymongo already forces this on us | ||
| + | |||
Cons: | Cons: | ||
| Line 24: | Line 25: | ||
* Lower memory use | * Lower memory use | ||
* You never have more in memory than you need | * You never have more in memory than you need | ||
| + | |||
Cons: | Cons: | ||
Latest revision as of 18:42, 7 August 2014
To buffer or to stream; that is the question.
Buffering
messages = list(msgs) {bring them all in at once}
Pros:
- Easier input validation
- Potentially more performant (iteratior protocol introduces overhead)
- Limits are known - back of the envelope calculations are available
- pymongo already forces this on us
Cons:
- More memory use
Streaming
message = next(msg) {one at a time}
Pros:
- Lower memory use
- You never have more in memory than you need
Cons:
- Makes some implementation details more difficult
- We'd have to patch pymongo to actually benefit from the pros (as I understand)
