Python websocket multithread/ multiprocess storing the received JSON data in one thread questions

tobywong

For my development, I’m trying to have two threads.

One threads is for receiving real time data and put them into a list/dictionary(which one should I use to store the JSON I received?)

And the other thread is to loop through the list/dictionary and print out the data in the JSON object/string we need.(not all the fields are needed in the received JSON data)

But the second thread doesn’t start.

The reason to use multithread is because we don't want to miss out on updated data when the program is printing the data(eventually, the "print" action will be replaced by updated our database)

What is the problem?

I could provide my code if it's needed.

Thank you.

nick.zincone

Hi @tobywong

You can probably search on the internet for multithreaded examples in Python that suit your requirements. When I do this, I explicitly search for an example to achieve multi-threaded processing.

For example:

# Example threading using the thread-safe 'queue.Queue' class which allows you to put
# data into the queue on one thread and retrieve it on another - asynchronously

import threading
import queue

# Create the queue
data_queue = queue.Queue()

In thread 1, when you receive your data, simply store it within the queue

def receive_data_cb(data):
   data_queue.put(data)

In thread 2, process the data when it becomes available

def process_data():
   while True:
      data = data_queue.get()    # Block until data arrives
      print(data)                # This is where you can store in a DB

To put it all together, create 2 threads and start each:

thread1 = threading.Thread(target=receive_data_cb)
thread2 = threading.Thread(target=process_data)

thread1.start()
thread2.start()

The above is the general algorithm I found by researching. However, I would personally experiment and try some simple tests. For example, you may not need to explicitly create 2 distinct threads here but instead just one which handle your processing of the data. The main thread, where you start your application, could handle the data coming from the WebSocket server.