Jump to: navigation, search

Zaqar/bp/buffer-messages

To buffer or to stream; that is the question.

Buffering

messages = list(msgs) {bring them all in at once}

Pros:

  • Easier input validation
  • Potentially more performant (iteratior protocol introduces overhead)
  • Limits are known - back of the envelope calculations are available
  • pymongo already forces this on us


Cons:

  • More memory use

Streaming

message = next(msg) {one at a time}

Pros:

  • Lower memory use
  • You never have more in memory than you need


Cons:

  • Makes some implementation details more difficult
  • We'd have to patch pymongo to actually benefit from the pros (as I understand)