I am trying to extract large amounts of market depth data from TRTH API, and I use python as my API client. After the data is extracted, I am streaming the data (in an effort to conserve memory), but during the stream, I often get the Connection reset by peer error. I assume this is because the server closes the connection. Because of this, I am unable to actually extract and parse larger amounts of data.
Is there a workaround for this? What would be the best way to extract large amounts of data from the TRTH API? Should I create multiple jobs and split the API call into more API jobs with smaller time intervals?
Thank you in advance!