Why do I keep getting timeout errors when pulling the full list of US stocks? Are there any solution

cond = 'IN(TR.ExchangeCountryCode,US)'
each_req = "SCREEN(U(IN(Equity(active,public)))/UNV:Public/, " + cond + ", NOT_IN(TR.InstrumentTypeCode,BONDFUT,BONDSPREAD,CEF,ETF,ETFA,ETFB,ETFC,ETFE,ETFM,ETFO,ETFX,ETMF,FUN,GROWUNT,HDG,INS,OPF,OPTRTS,PAIDSUBRTS,PREFERRED,PRF,RTS,SUBSRTS),CURN=USD)"
fields = ["TR.OrganizationID", "TR.CompanyName", "TR.ExchangeName", "TR.IsPrimaryInstrument", "TR.ISIN","TR.InstrumentType", "TR.ExchangeCountry"]
df_stock = ld.get_data(each_req, fields)

Find more posts tagged with

python

screener

time-out

#content

Accepted answers

All comments

jason.ramchandani01

@rice So this query works for me (I removed the /UNV:Public/) as you have already specified it:

cond = 'IN(TR.ExchangeCountryCode,US)'

each_req = "SCREEN(U(IN(Equity(active,public)))," + cond + ", NOT_IN(TR.InstrumentTypeCode,BONDFUT,BONDSPREAD,CEF,ETF,ETFA,ETFB,ETFC,ETFE,ETFM,ETFO,ETFX,ETMF,FUN,GROWUNT,HDG,INS,OPF,OPTRTS,PAIDSUBRTS,PREFERRED,PRF,RTS,SUBSRTS),CURN=USD)"

fields = ["TR.OrganizationID", "TR.CompanyName", "TR.ExchangeName", "TR.IsPrimaryInstrument", "TR.ISIN","TR.InstrumentType", "TR.ExchangeCountry"]


df_stock = rd.get_data(each_req, fields)

Does this work for you? It is also a large universe being returned - the indicated limit per API call is 10K datapoints per API call - this is over 100k (ie 10x the recommended amount). It is not surprising that you are getting timed out - particularly if you are running it during periods of high server load. An idea to return smaller chunks of the universe would be to split by market cap band, or by company name beginning with a letter of alphabet. The benefit of this with large universes is less timeouts. I hope this can help.

Kindest regards,

rice

https://community.developers.refinitiv.com/discussion/comment/127357#Comment_127357

Thank you for the suggestion. I’d like to try splitting the queries by market cap bands and company name initials. Could you provide a code example for this?

Jirapongse

@rice

You can contact the helpdesk team via LSEG Support and ask for the screener formula which can be used to get the required item.

Then, you can apply the formula to the Python code. For more information, please refer to this Find Your Right Companies with SCREENER | the Data Library (Python) article.

Jirapongse

@rice

You can try to increase the HTTP client timeout by running the following code before opening a session.

ld.get_config()["http.request-timeout"] = 600

You can refer to the answer on this discussion.

However, the request can also be timed out by the server side so you may still need to split the query by adding other filters.

EXPLORE OUR SITES