Difference in bahaviour if instruments are validated before before populating a request

atappis

Hi,

I am generating TickHistoryTimeAndSales on demand extractions and have noted a difference in the bahaviour of requests, depending on whether instruments get pre-validated or not.

If I do not pre-validate the instrument list, then invalid instruments get ignored and there is no record of them at all in the resulting notes file. When i do pre-validate, the notes file correctly shows permission errors associated with the instruments.

I use the following code to generate requests.

First generate a collection of InstrumentIdentifier objects from a DataTable

var reutersInstruments = from DataRow r in instrumentTable.AsEnumerable()
                         select new InstrumentIdentifier()
                         {
                             Identifier = (string)r["Identifier"],
                             IdentifierType = IdentifierType.Ric
                         };

The following code for validating the list

var validateInstruments = context.InstrumentListOperations.ValidateIdentifiers(reutersInstruments, false);

The request is then populated in one of 2 possible ways

var request = new TickHistoryTimeAndSalesExtractionRequest
              {
                  Condition = new TickHistoryTimeAndSalesCondition
                  {
                      ReportDateRangeType = reportDateRangeType,
                      TimeRangeMode = timeRangeMode,
                      QueryStartDate = queryStartDate,
                      QueryEndDate = queryEndDate,
                      ApplyCorrectionsAndCancellations = applyCorrectionsAndCancellations,
                      ExtractBy = extractBy,
                      MessageTimeStampIn = messageTimeStampIn,
                      SortBy = sortBy,
                      DisplaySourceRIC = displaySourceRIC
                  },
                  ContentFieldNames = fields.ToArray(),
                  IdentifierList = new InstrumentIdentifierList
                  {
                      // InstrumentIdentifiers = validateInstruments.ValidatedInstruments.ToArray()
                      InstrumentIdentifiers = reutersInstruments.ToArray()
                  },
              };

The resulting requests (in JSON) and associated notes files are attached. requests.zip

I would expect the notes file to be the same in both cases.

veerapath.rungruengrayubkul

@atappis

I have tried more testing and found that the result depends on User Preferences. There are two interfaces for pre-validate; ValidateIdentifiers and ValidateIdentifiersWithOptions. From my understanding, the ValidateIdentifiers uses validation options set in the User Preferences, but the ValidateIdentifersWithOptions uses the validation options passed in the options parameter.

This means that the result of pre-validate you found should be based on the User Preferences. I guess your User Preferences should have "AllowOpenAccessInstruments" checked.

For the non pre-validate, application can set the "UseUserPreferencesForValidationOptions" to use validation options set in the User Preferences or set specific options with "InstrumentValidationOptions".

IdentifierList = new InstrumentIdentifierList() { InstrumentIdentifiers = reutersInstruments.ToArray(), UseUserPreferencesForValidationOptions = true },

So in this case, you should get the same result using the "UseUserPreferencesForValidationOptions" = true.

Christiaan Meihsl

@atappis, I have escalated your query to the development team.

veerapath.rungruengrayubkul

Hi @atappis

I'm able to make the notes file to be the same in both case by adding the "AllowOpenAccessInstruments" = true in the InstrumentValidationOptions. It seems like the pre-validate allows Open Access Instrument.

Below is the snippet code.

IdentifierList = new InstrumentIdentifierList() { InstrumentIdentifiers = reutersInstruments.ToArray(), ValidationOptions = new InstrumentValidationOptions() { AllowOpenAccessInstruments = true} }, 

atappis

This sounds like a workaround. Can you provide a more detailed explanation please? What does this mean? Is the difference in behaviour valid?

atappis

I see, thank you.

I believe that this option should be defaulted to true if not specified, to ensure consistency in behaviour. Naturally this can be overridden in both cases to explicitly define the required behaviour.

UseUserPreferencesForValidationOptions = true

In my example, am i correct in assuming that, if I manually pre-validate the instruments and populate the request with validated instruments, then a validation step is not performed again as part of report execution on your side?

And in the 2nd case where I just populate an unvalidated list of rics into the report this list would first get validated internally?

If so, it would make sense for the results of the validation to be included in the notes file, regardless of the validation options?

Is the information in the notes file documented anywhere? I realize it's unstructured and more of a log file, but there is useful status/error information in there that I might want to report at the RIC level and right now I'm considering scraping the notes file.

Christiaan Meihsl

https://community.developers.refinitiv.com/discussion/comment/35590#Comment_35590

FYI: DSS GUI User Guide p36 extract:

Allow Import of Open Access Instruments from Real-Time Feed

• True: allow import & intraday extractions of Open Access Instruments (OAI)
• False: allow only instruments that can be validated, ignore OAIs

OAI can't be validated. They include but are not limited to: illiquid bonds, contributed forex/money markets, some hybrid instruments, cash commodities etc.

Some RICs are restricted from Open Access functionality: those that require 3rd-party content permissioning, and unsupported content (news headlines / stories, navigation RICs, ICW information, pages, etc.).

atappis

https://community.developers.refinitiv.com/discussion/comment/35600#Comment_35600

Thank you for the additional detail. Much appreciated.

veerapath.rungruengrayubkul

@atappis

Your assumption is correct. If you populate an unvalidated list of rics into the report this list, it will get validated internally.

I have tried the same request of non pre-validate in Postman and get the validation result in the IdentifierValidationErrors field. This can be accessed using the RawExtractionResult.IdentifierValidationErrors attribute of DSS .NET SDK.

{
    "@odata.context": "https://hosted.datascopeapi.reuters.com/RestApi/v1/$metadata#RawExtractionResults/$entity",
    "JobId": "0x066faab0e960230b",
    "Notes": [
        "Extraction Services Version 12.2.39833 ..."
    ],
    "IdentifierValidationErrors": [
        {
            "Identifier": {
                "@odata.type": "#ThomsonReuters.Dss.Api.Content.InstrumentIdentifier",
                "Identifier": "AUDECB10Y=ICAP",
                "IdentifierType": "Ric",
                "Source": ""
            },
            "Message": "Not found"
        },
        {
            "Identifier": {
                "@odata.type": "#ThomsonReuters.Dss.Api.Content.InstrumentIdentifier",
                "Identifier": "AUDECB10Y=TPI",
                "IdentifierType": "Ric",
                "Source": ""
            },
            "Message": "Not found"
        }
    ]
}