Skip to content

Performance degradation on python producer.send with large number of records #258

Description

@AlexMikhalev

slack thread: https://infinyon.slack.com/archives/CQ8UX6SBH/p1684349700779719

simple producer in python:

#!/usr/bin/env python3
from datetime import datetime
from fluvio import Fluvio
import time 

PARTITION = 0

if __name__ == "__main__":
   # Connect to cluster
   fluvio = Fluvio.connect()

   producer = fluvio.topic_producer("simple")

   start_time = time.time()

   for x in range(10000):
       producer.send_string("{}: timestamp: {}".format(x, datetime.now()))

   # Flush the last entry
   producer.flush()
   print("completed in  {} seconds", (time.time()-start_time))

10K records pushed in 1.2153611183166504 seconds, when if you increase the range to 100000
python3 producer100K.py
The same code takes 64.67606687545776 to complete.
Rust producer doesn't have such behaviour.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions