For a deeper look into our DataScope Select REST API, look into:

Overview |  Quickstart |  Documentation |  Downloads |  Tutorials

question

Upvotes
3 0 0 3

Searched for Futures and Options unsuccessfully with status code 502 in between of multiple long runs

DSS Rest API

Domain - https://selectapi.datascope.refinitiv.com/RestApi/v1/

URL - https://selectapi.datascope.refinitiv.com/RestApi/v1/Search/FuturesAndOptionsSearch

Total instruments for 1 day data - 418788

Details of issue -
As more then 400 instruments were giving server busy error, tried querying this above URL in chunks with instruments = 400 and 'respond-async;odata.maxpagesize=1000'.


After running for 5 loops(instrument chunks), it failed and raised error -

Processing instruments bucket No.5 for searchFuturesOptions with len:400 Instrument Range BLSR.TA<-->RKH.L

ERROR Searched for Futures and Options unsuccessfully with status code 502


msg: 'Content of request: b\'{"error":{"message":"An error occurred on the server. If this problem persists, contact Support (referenci' (str)

response: '<Response [502]>' (Response)


Could you please guide what is the good way to query large amount of instruments, as if its fails in between, time and processing both gets wasted.

#technologydss-rest-api
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
80.5k 261 52 75

@trisha.gehlot

Thank you for reaching out to us.

According to the DSS Status Codes page, the 502 is Bad Gateway.

502 Bad Gateway indicates that the server, while acting as a gateway or proxy, received an invalid response from the upstream server it accessed in attempting to fulfill the request. The application can retry.

What do you mean by "Total instruments for 1 day data - 418788"?

Please share the request message that you are using. I will try it on my machine.


icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
3 0 0 3

@Jirapongse Thank you for your response!!


As it was a Bad Gateway, it might not arise again, but could please suggest best way to deal with large amount of instruments.

418788 this is the total number of instruments to be searched.

Request body-

searchRequest = {"IdentifierType": identifierType, "PreferredIdentifierType": preferredIdentifierType}

if underlyingRIC:

searchRequest["UnderlyingRic"] = identifiers

requestBody = json.dumps({"SearchRequest": searchRequest})

return requestBody

Request Body -

{"SearchRequest": {"IdentifierType": "Ric", "PreferredIdentifierType": "Ric", "UnderlyingRic": "AAL.O,ABT.N,AMD.O,APD.N,AEP.O,AXP.N,AMGN.O,AAPL.O,ADM.N,ADSK.O,ADP.O,AVY.N,BLL.N,BSET.O,BAX.N,BDX.N,HRB.N,BA.N,BGGSQ.PK,BMY.N,BFb.N,BC.N,CNA.N,CSX.O,CPB.N,CAT.N,CLX.N,KO.N,CL.N,CAG.N,CTB.N,GLW.N,CR.N,CMI.N,DE.N,DLX.N,DDS.N,DIS.N,RRD.N,DOV.N,ETN.N,ECL.N,EMR.N,ETR.N,FMC.N,NEE.N,FMCC.PK,FNMA.PK,FLTWQold.PK,TGNA.N,GPS.N,GD.N,GE.N,GIS.N,GCO.N,GPC.N,GT.O,GRA.N,GWW.N,HAL.N,LHX.N,HAS.O,HP.N,HSY.N,HPQ.N,HD.N,HUM.N,ITW.N,INTC.O,IBM.N,IFF.N,IP.N,JNJ.N,K.N,KMB.N,KR.N,LLY.N,LB.N,LNC.N,LPX.N,LOW.N,LUB.N,MMC.N,MAS.N,MAT.O,MDRIQ.PK,MCD.N,SPGI.N,MDT.N,CVS.N,MRK.N,MDP.N,MSI.N,NL.N,NC.N,THC.N,NAV.N,NYT.N,NEM.N,NKE.N,JWN.N,NSC.N,NOC.N,NUE.N,OXY.N,OKE.N,ORCL.N,PHM.N,PNC.N,PPG.N,PCAR.O,PCG.N,PLLdum.N,PH.N,CPPRQ.PK,PEP.O,PFE.N,PBI.N,PG.N,PEG.N,RAD.N,ROK.N,R.N,EIX.N,SPXC.N,SLB.N,SCI.N,SHW.N,SKY.N,SNA.N,SO.N,SWK.N,STI.N,SYY.N,TJX.N,TXN.O,TXT.N,TKR.N,GL.N,UALAQold.OB,UN.N,UNP.N,RTX.N,VFC.N,WMT.N,WBA.O,WLBAQ.PK,WY.N,WHR.N,WMB.N,XRXold.N,ADBE.O,AIN.N,MATX.N,APC.N,AONOLD.N,APAold.O,AMAT.O,AVT.O,BKH.N,CMS.N,CBT.N,CDNS.O,CAH.N,CRS.N,CHKAQ.PK,CRUS.O,CTAS.O,CMA.N,DBD.N,EBF.N,EFX.N,FSS.N,FISV.O,BEN.N,GATX.N,AJRD.N,HOG.N,HSC.N,HE.N,IDA.N,IPG.N,KELYA.O,LANC.O,LEG.N,MKC.N,MU.O,MLHR.O,MUR.N,MYL.O,NI.N,UVV.N,CSCO.O,DG.N,FITB.O,.SPX,EAT.N,KEY.N,NFG.N,NPK.N,NDSN.O,ES.N,NTRS.O,ODPold.O,OGE.N,OLN.N,OMC.N,PAYX.O,PNR.N,PNW.N,PGR.N,SCHW.N,SEE.N,SON.N,LUV.N,STT.N,TFX.N,TDS.N,TER.O,TMO.N,TIF.N,TRN.N,TSN.N,UFI.N,VLO.N,GHC.N,WEC.N,XLNX.O,AGCO.N,AES.N,RAMP.N,AEM.N,Y.N,AMSC.O,AP.N,APH.N,ARCB.O,ARW.N,ASTE.O,ATO.N,AZO.N,BBBY.O,BBY.N,BIO.N,BSX.N,CACI.N,ELY.N,CHH.N,FUN.N,CAKE.O,CINF.O,CTG.O,CRVL.O,CACC.O,XRAY.O,DGII.O,DXYN.O,DY.N,EA.O,VALPQ.PK,ENZN.PK,FAST.O,GILD.O,HUBB.N,JBHT.O,IDXX.O,AEGN.O,IONS.O,KLAC.O,KEM.N,LSCC.O,MTX.N,PRCP.O,SKYW.O,SWN.N,SHYF.O,SPLS.O,PSA.N,SNPS.O,SNV.N,TCO.N,TECD.O,TPLold.N,TR.N,TSS.N,UGI.N,UHS.N,VSH.N,VNO.N,WRI.N,WMK.N,WDC.O,WSM.N,WWW.N,ZBRA.O,YPF.N,L.N,EQC.N,HRL.N,KIM.N,KSS.N,LEE.N,JEF.N,MBI.N,MDC.N,MTG.N,MGM.N,MAN.N,NJR.N,ORI.N,OMI.N,RHI.N,HST.O,MSFT.O,PRGO.N,EMN.N,SBUX.O,TV.N,RYN.N,MGA.N,QCOM.O,SPY.P,ALB.N,INTU.O,TEF.N,LRCX.O,SPG.N,KOF.N,TOT.N,DRE.N,IMO.A,.RUI,.RUT,ALL.N,MLM.N,HBAN.O,MXIM.O,VLY.O,ENIA.N,JJSF.O,MCHP.O,NWN.N,REGN.O,MEOH.O,TEVA.N,PZZA.O,ADTN.O,GBX.N,COF.N,STRT.O,HIW.N,LMT.N,DRI.N,AFG.N,INCY.O,BWA.N,RHHBY.PK,FCX.N,AKS.N,TGS.N,UL.N,CPSS.O,AEG.N,SWM.N,IT.N,ITT.N,HIG.N,TUP.N,DOFSQ.PK,FMS.N,ATGE.N,CBRI.PK,AGLJ.J,SASY.PA,TDC.CO,ALSO.PA,CNPP.PA,PUBP.PA,AGFB.BR,SAN.MC,WHLJ.J,ADEN.S,ITUJ.J,ABBN.S,NELES.HE,IDR.MC,HSBA.L,INCH.L,NTGY.MC,HUDN.AS,LKOH.MM,MSNG.MM,RTKM_p.MM,ALB.MC,BMPS.MI,OML.L,TEF.MC,RTKM.MM,ABB.ST,SAF.PA"}}'


response = requests.post(

DataScopeURL.SEARCH_FUTURES_OPTIONS,

requestBody,

headers=populateHttpRequestHeaders(self.loginDetails, self.proxyServers, prefer='respond-async;odata.maxpagesize=1000', passToken=True),

proxies=self.proxyServers,

)

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

@trisha.gehlot

I found the answer in this discussion.

Upvotes
3 0 0 3

@Jirapongse Thank you for this paging logic, will try to increase page size!!

But the issue above is, when I send 418788 instruments all together, the server give exception.

And when I sent instrument Chunks for eg. 2000 instruments is taking 12 hours in querying this Rest API

I need to check, what is the best practice to deal with this, while sending the query to FuturesAndOptionsSearch


*Also if you can confirm, how many concurrent queries they support for our account?

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

@trisha.gehlot

I think 418788 instruments are too much for search endpoint.

Please contact your LSEG account team or sales team directly to verify this use case.

We may be able to suggest other products for this use case.

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.