For a deeper look into our DataScope Select REST API, look into:

Overview |  Quickstart |  Documentation |  Downloads |  Tutorials

question

Upvotes
Accepted
3 1 1 3

DSS REST API - Error 400 malformed request payload


Hello,

I'm trying to extract data (End of day, Time Series,...) using the DSS REST API with Python 3.5.2.

First, I connect to the DSS server as described in REST API Tutorial 1, without any problem

header = {'Prefer': 'respond-async', 'Content-Type': 'application/json; odata.metadata=minimal'}
urlGetToken = 'https://hosted.datascopeapi.reuters.com/RestApi/v1/Authentication/RequestToken'
loginData = json.dumps({'Credentials':{'Password':Password,'Username':Login}})
resp = requests.post(urlGetToken, loginData, headers=header)


I can also get the available list of fields without any error

header2 = {'Prefer': 'respond-async, wait=5', 'Content-Type': 'application/json; odata.metadata=minimal', 'Authorization': myToken}
urlGetListFields = "https://hosted.datascopeapi.reuters.com/RestApi/v1/Extractions/GetValidContentFieldTypes(ReportTemplateType=ThomsonReuters.Dss.Api.Extractions.ReportTemplates.ReportTemplateTypes'TimeSeriesPricing')"
resp2 = requests.get(urlGetListFields,headers=header2)

Finally, when I try to extract some data:

# From users input
listIdentifiers = [{"Identifier":"BE0003592038","IdentifierType":"Isin"},{"Identifier":"BE0003593044","IdentifierType":"Isin"}]
fromdate = "2016-06-01"
todate = "2016-06-30"
header3 = {'Content-Type': 'application/json; odata.metadata=minimal; charset=utf-8', 'Authorization': myToken}
urlGetData = 'https://hosted.datascopeapi.reuters.com/RestApi/v1/Extractions/Extract'
odataType = "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.TimeSeriesExtractionRequest"
condition = {"LastPriceOnly": False,"StartDate": fromdate + "T00:00:00.000Z","EndDate": todate + "T00:00:00.000Z"}
# in practice listFields is bigger 
listFields = ["Accrued Interest","Close Price","ISIN","RIC"]
requestData = {"ExtractionRequest": {"@odata.type": odataType,"ContentFieldNames": listFields,"IdentifierList": {"@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.InstrumentIdentifierList","InstrumentIdentifiers": listIdentifiers},"Condition":condition}}
resp3 = requests.post(urlGetData, data=json.dumps(requestData), headers=header3)


Sometimes, it works perfectly well as I get resp3.status_code = 200 or resp3.status_code = 202, but most of the time I get resp3.status_code = 400 with one of the following messages for the exact same request:

400{'error': {'message': 'Malformed request payload: For the property name "StartDate" in the JSON request the value could not be parsed successfully. Please check the casing or spelling of the property.'}}
400{'error': {'message': 'Malformed request payload: Syntax error at Line 1, Char 64: Invalid array element type for property \'ContentFieldNames\'.  Expected element type of \'ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.ExtractionRequestBase\', but found \'System.String\'. {"ExtractionRequest": {"ContentFieldNames": ["Accrued Interes'}}
400{'error': {'message': 'Malformed request payload: For the property name "EndDate" in the JSON request the value could not be parsed successfully. Please check the casing or spelling of the property.'}}
400{'error': {'message': 'Malformed request payload: For the property name "IdentifierList" in the JSON request the value could not be parsed successfully. Please check the casing or spelling of the property.'}}
400{'error': {'message': 'Malformed request payload: For the property name "Identifier" in the JSON request the value could not be parsed successfully. Please check the casing or spelling of the property.'}}


I tried to solve the problem by including charset=utf-8 in the header, using the json parameter from requests, ... but it seems that the only effective hack is to delete then include again a field in listFields.

Do you have any idea that could help me?

pythondss-rest-apidssdatascope-selecterrorerror-400json-payload
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
Accepted
1.1k 4 3 5

Unfortunately, the order of the @odata.type line is crucial (we do intend to fix that in a future release):

You sent in:

  {
    "ExtractionRequest": {
        "ContentFieldNames": ["Accrued Interest",
        "ClosePrice",
        "ISIN",
        "RIC"],
        "@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.TimeSeriesExtractionRequest",
        "IdentifierList": {
            "@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.InstrumentIdentifierList",
            "InstrumentIdentifiers": [{
                "IdentifierType": "Isin",
                "Identifier": "BE0003592038"
            },
            {
                "IdentifierType": "Isin",
                "Identifier": "BE0003593044"
            }]
        },
        "Condition": {
            "LastPriceOnly": false,
            "EndDate": "2016-06-30T00:00:00.000Z",
            "StartDate": "2016-06-01T00:00:00.000Z"
        }
    }
}

but when edited to this, it worked (note: I also change "ClosePrice" to "Close Price"):

{
    "ExtractionRequest": {
        "@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.TimeSeriesExtractionRequest",
        "ContentFieldNames": ["Accrued Interest",
        "Close Price",
        "ISIN",
        "RIC"],
        "IdentifierList": {
            "@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.InstrumentIdentifierList",
            "InstrumentIdentifiers": [{
                "IdentifierType": "Isin",
                "Identifier": "BE0003592038"
            },
            {
                "IdentifierType": "Isin",
                "Identifier": "BE0003593044"
            }]
        },
        "Condition": {
            "LastPriceOnly": false,
            "EndDate": "2016-06-30T00:00:00.000Z",
            "StartDate": "2016-06-01T00:00:00.000Z"
        }
    }
}
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
1.1k 4 3 5

Is there any chance you can capture the full request and response actually sent on the wire?

I should be able to tell you what was wrong with any 1 request that causes a 400 reply as long as I can see the raw data you sent in.

We currently do not record every bad request, but we are considering doing just that so we can be more supportive on these types of requests.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
3 1 1 3

Hello @Rick Weyrauch, thank you for your quick answer. Here is what I get (I replaced the token by MyToken):

>>> requests.post(urlGetData, data=json.dumps(requestData), headers=header3)

INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): hosted.datascopeapi.reuters.com

send: b'POST /RestApi/v1/Extractions/Extract HTTP/1.1\r\nHost: hosted.datascopeapi.reuters.com\r\nAccept-Encoding: gzip, deflate\r\nUser-Agent: python-requests/2.11.1\r\nConnection: keep-alive\r\nAccept: */*\r\nContent-Type: application/json; odata.metadata=minimal; charset=utf-8\r\nAuthorization: myToken\r\nContent-Length: 577\r\n\r\n'send: b'{"ExtractionRequest": {"ContentFieldNames": ["Accrued Interest", "ClosePrice", "ISIN", "RIC"], "@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.TimeSeriesExtractionRequest", "IdentifierList": {"@odata.type": "#ThomsonReuters.Dss.Api.Extractions.ExtractionRequests.InstrumentIdentifierList", "InstrumentIdentifiers": [{"IdentifierType": "Isin", "Identifier": "BE0003592038"}, {"IdentifierType": "Isin", "Identifier": "BE0003593044"}]}, "Condition": {"LastPriceOnly": false, "EndDate": "2016-06-30T00:00:00.000Z", "StartDate": "2016-06-01T00:00:00.000Z"}}}'

reply: 'HTTP/1.1 400 Bad Request\r\n'

DEBUG:requests.packages.urllib3.connectionpool:"POST /RestApi/v1/Extractions/Extract HTTP/1.1" 400 343

header: Set-Cookie header: Cache-Control header: Pragma header: Content-Type header: Expires header: Server header: X-Request-Execution-Correlation-Id header: X-App-Id header: X-App-Version header: Date header: Content-Length <Response [400]>
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
3 1 1 3

@Rick Weyrauch Thank you very much, your answer helped me solve the problem using OrderedDict from collections.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvote
20 2 1 5

@nvico and @Rick Weyrauch thank you for this post. I was experiencing the same problem using Python and use of the OrderedDict type solved it, i.e. the seemingly random nature of the error when using a Python dictionary to create the payload for an On Demand POST request to DSS: sometimes the very same request worked and sometimes not.

I would like to add something that I found: the 'randomness' occurred when I was using the dictionary Python 3.5 but when I made requests from another machine using Python 3.6.1, with the same dictionary structure and script, the request worked every time. I subsequently upgraded my Python 3.5 to 3.6.1 and now [using the dictionary, not OrderedDict] it works reliably too.

However, for what I am assuming is good practice, and compatibility with earlier versions of Python, I am using OrderedDict.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.