Not clearing cache on REFRESH?

I try to find out how to handle refresh messages in Market By
Price (level 2) data. I was told by the support help desk that there are refresh messages on
which we should clear the whole cache (our constructed order book) and
refresh messages which do not. How are we suppose to distinguish these
two types of refresh messages?

For example in the underlying TRTHv2 data snippet we took from DataScope, if we
would clear our cache and order book on the refresh message then in the
subsequent update we have to update an order which does not exist
anymore:

7203.T,Market By Price,2018-01-15T01:06:37.281977617Z,+9,Raw,REFRESH,,,,,4742,0,58927,0
,,,,Summary,,,,,,,,,20
,,,,FID,1,,PROD_PERM,4742,
,,,,FID,3,,DSPLY_NAME,TOYOTA MOTOR CO,
,,,,FID,15,,CURRENCY,392,JPY
,,,,FID,17,,ACTIV_DATE,2018-01-15,
,,,,FID,259,,RECORDTYPE,113,
,,,,FID,1080,,PREF_DISP,0,
,,,,FID,1709,,RDN_EXCHD2,106,TYO
,,,,FID,3422,,PROV_SYMB,7203,
,,,,FID,3423,,PR_RNK_RUL,1,NOR
,,,,FID,3425,,OR_RNK_RUL,2,"PTS "
,,,,FID,3694,,MNEMONIC,7203,
,,,,FID,4148,,TIMACT_MS,3996735,
,,,,FID,5357,,CONTEXT_ID,2125,
,,,,FID,6401,,DDS_DSO_ID,4098,
,,,,FID,6480,,SPS_SP_RIC,.[SPSTFFL2,
,,,,FID,6516,,BOOK_STATE,3,X
,,,,FID,6517,,HALT_REASN," ",
,,,,FID,6519,,MKT_OR_RUL,1,I
,,,,FID,6614,,TRD_STATUS,1,"N "
,,,,FID,14269,,TIMACT_NS,01:06:36.735317000,
7203.T,Market By Price,2018-01-15T01:06:38.298915012Z,+9,Raw,UPDATE,UNSPECIFIED,,,,4742,,58928,0
,,,,Summary,,,,,,,,,6
,,,,FID,17,,ACTIV_DATE,2018-01-15,
,,,,FID,4148,,TIMACT_MS,3998230,
,,,,FID,6516,,BOOK_STATE,3,X
,,,,FID,6517,,HALT_REASN," ",
,,,,FID,6614,,TRD_STATUS,1,"N "
,,,,FID,14269,,TIMACT_NS,01:06:38.230599000,
,,,,MapEntry,,UPDATE,,,,,,76530000A,8 <===========FAULTY UPDATE
,,,,FID,3427,,ORDER_PRC,7653,
,,,,FID,3428,,ORDER_SIDE,2,ASK
,,,,FID,4356,,ACC_SIZE,1500,
,,,,FID,3430,,NO_ORD,12,
,,,,FID,6527,,LV_TIM_MS,3998230,
,,,,FID,6529,,LV_DATE,2018-01-15,
,,,,FID,3886,,ORDER_TONE," ",
,,,,FID,14268,,LV_TIM_NS,01:06:38.230599000,

Find more posts tagged with

elektron-sdk

tick-history-rest-api

Accepted answers

Christiaan Meihsl

@vic, a refresh message presents the current state, i.e. a snapshot at one point in time.

I don't see why one should clear the cache when receiving a refresh message, but maybe I'm missing something, I'm not a data expert.

For data content query like this one, the best and speediest way to receive an answer is to open a content-related enquiry via My Account or to call the Thomson Reuters Help Desk directly, making sure you specify that you are working with TRTH Market By Price (level 2) data.

All comments

nick.zincone

Hi @vic,

Assuming you are using an Elektron API, EMA for example, to process your MBP data, it is possible to receive an unsolicited Refresh message to not clear your cache. That is, this section describes how a Provider would indicate to a consume what to do:

When you receive a refresh, you can capture the above settings within the Refresh Indication flag:

Hope this helps.

vic

@nick.zincone.1

Thanks for your reply. This question is about Thomson Reuters Tick History data I download from DataScope Select not from the Elektron API. It's historical data downloaded by Thomson Reuters with the Elektron API and saved in historical files. But there's no way to tell if we should clear cache or not on the Refresh messages because it's not specified in the files.

nick.zincone

Hi @vic,

Given you included the "elektron api" as a group within your question, I thought you may have been using one of those APIs to access the data. That being said, I don't know whether TRTH can include these details within the files. I've solicited some help to see if that is possible.

Christiaan Meihsl

@vic, a refresh message presents the current state, i.e. a snapshot at one point in time.

I don't see why one should clear the cache when receiving a refresh message, but maybe I'm missing something, I'm not a data expert.

zoya faberov

Hello @vic,

I agree with @Christiaan Meihsl,

The requirement to clear and repopulate application cache, and the helpful clearCache indications found in refresh messages are designed for processing of market data in realtime, on subscription to realtime data feeds.

The indications are not relevant for historical data, that can be retrieved from Datascope and designed for non-realtime, historical data analytics.

vic

@Christiaan Meihsl @zoya faberov

So if a refresh message presents the current state, and that state we save in our applications cache (for example, we save the state of the order book). Then on a subsequent refresh message we need to have clearCache or not clearCache to distinguish whether we should clear and repopulate the order book or just add the values of the refresh message to it.

This does not differ between real-time data directly from real-time feeds or historical data from Datascope. For historical data I would like to feed the historical data in a program which simulates real-time trading (creating and maintaining the order book over time), so it also needs to distinguish whether it need to clear or not clear the cache (saved data).

nick.zincone

@vic,

Because you are not using our streaming APIs, the data you request for will always be up to date and you will need to completely replace your application order book. When you explicitly request for data, the refresh will always instruct you to clear the cache.

The Indication to not clear the cache will never apply to applications that are trying to simulate real time because you will never get this indication when you request for the data. It only applies to our streaming APIs and it is a rare asynchronous event.

Hope this is clear.

Nick

vic

@nick.zincone If we would replace the complete order book in my example at the start of this topic with the data of the REFRESH, then there would be nothing in the order book afterwards. Because the REFRESH is empty of prices. This can not be right, as this is during trading hours. Also the many updates messages that comes afterwards indicates that something is not right if we clear the order book there.

nick.zincone

@vic,

Are you saying that when you request for an order book, you initially get a REFRESH with the contents of that order book, then later request for the same data during market hours, you get a REFRESH that is empty? If so, please post a new question showing how you are doing this.

Thanks.

vic

@nick.zincone I did not request the data myself, these are real-time API (ELEKTRON) historical recordings Thomson Reuters provides as a product. It's just a recording of the stream, saved into files which you can download from a website.

zoya faberov

Hello @vic,

Agree with you and with @nick.zincone,

As Datascope does not contain indications to clear cache (is designed to archive content only)

The way I would proceed, in order to back-test, on REFRESH for an instrument, I would clear the book for that instrument and recreate from the refresh.

Does this approach make sense?

vic

@zoya faberov But as in my example, some refreshes are empty of order book data. If I clear there and recreate I end up with an empty order book, which is wrong.

zoya faberov

Hi @vic,

I can see your point. This goes back to Datascope being designed to hold content, and not expressly designed for back-testing of applications, by storing the way in which it was published.

I would make a judgement call on refresh, and if you see an empty one, discard it.

EXPLORE OUR SITES