Hi @tobywong

You can probably search on the internet for multithreaded examples in Python that suit your requirements. When I do this, I explicitly search for an example to achieve multi-threaded processing.

For example:

# Example threading using the thread-safe 'queue.Queue' class which allows you to put
# data into the queue on one thread and retrieve it on another - asynchronously

import threading
import queue

# Create the queue
data_queue = queue.Queue()

In thread 1, when you receive your data, simply store it within the queue

def receive_data_cb(data):
   data_queue.put(data)

In thread 2, process the data when it becomes available

def process_data():
   while True:
      data = data_queue.get()    # Block until data arrives
      print(data)                # This is where you can store in a DB

To put it all together, create 2 threads and start each:

thread1 = threading.Thread(target=receive_data_cb)
thread2 = threading.Thread(target=process_data)

thread1.start()
thread2.start()

The above is the general algorithm I found by researching. However, I would personally experiment and try some simple tests. For example, you may not need to explicitly create 2 distinct threads here but instead just one which handle your processing of the data. The main thread, where you start your application, could handle the data coming from the WebSocket server.

Using queue.Queue works! Thank you so much. But why doesn't list work?

Hi @tobywong

Queue class is much more complex than list and designed to operate efficiently in multi-thread context.
This article compares list and Queue: https://www.troyfawkes.com/learn-python-multithreading-queues-basics/

Behind the scene, Queue is managing a deque or a list, but if you check Queue.put() and Queue.get(), you'll see that these functions are using:

• a Semaphoe to protect simultaneous call to both functions
• wait() function to let the GIL to suspend the current thread and let others restart

With your implementation, if you don't use async function, the thread that manages the list can't be suspended by the GIL so others threads are waiting infinitely to run