question

Upvotes
Accepted
vic avatar image
1 ●3 ●1 ●7

Not clearing cache on REFRESH?

I try to find out how to handle refresh messages in Market By Price (level 2) data. I was told by the support help desk that there are refresh messages on which we should clear the whole cache (our constructed order book) and refresh messages which do not. How are we suppose to distinguish these two types of refresh messages?

For example in the underlying TRTHv2 data snippet we took from DataScope, if we would clear our cache and order book on the refresh message then in the subsequent update we have to update an order which does not exist anymore:

7203.T,Market By Price,2018-01-15T01:06:37.281977617Z,+9,Raw,REFRESH,,,,,4742,0,58927,0
,,,,Summary,,,,,,,,,20
,,,,FID,1,,PROD_PERM,4742,
,,,,FID,3,,DSPLY_NAME,TOYOTA MOTOR CO,
,,,,FID,15,,CURRENCY,392,JPY
,,,,FID,17,,ACTIV_DATE,2018-01-15,
,,,,FID,259,,RECORDTYPE,113,
,,,,FID,1080,,PREF_DISP,0,
,,,,FID,1709,,RDN_EXCHD2,106,TYO
,,,,FID,3422,,PROV_SYMB,7203,
,,,,FID,3423,,PR_RNK_RUL,1,NOR
,,,,FID,3425,,OR_RNK_RUL,2,"PTS "
,,,,FID,3694,,MNEMONIC,7203,
,,,,FID,4148,,TIMACT_MS,3996735,
,,,,FID,5357,,CONTEXT_ID,2125,
,,,,FID,6401,,DDS_DSO_ID,4098,
,,,,FID,6480,,SPS_SP_RIC,.[SPSTFFL2,
,,,,FID,6516,,BOOK_STATE,3,X
,,,,FID,6517,,HALT_REASN," ",
,,,,FID,6519,,MKT_OR_RUL,1,I
,,,,FID,6614,,TRD_STATUS,1,"N "
,,,,FID,14269,,TIMACT_NS,01:06:36.735317000,
7203.T,Market By Price,2018-01-15T01:06:38.298915012Z,+9,Raw,UPDATE,UNSPECIFIED,,,,4742,,58928,0
,,,,Summary,,,,,,,,,6
,,,,FID,17,,ACTIV_DATE,2018-01-15,
,,,,FID,4148,,TIMACT_MS,3998230,
,,,,FID,6516,,BOOK_STATE,3,X
,,,,FID,6517,,HALT_REASN," ",
,,,,FID,6614,,TRD_STATUS,1,"N "
,,,,FID,14269,,TIMACT_NS,01:06:38.230599000,
,,,,MapEntry,,UPDATE,,,,,,76530000A,8 <===========FAULTY UPDATE
,,,,FID,3427,,ORDER_PRC,7653,
,,,,FID,3428,,ORDER_SIDE,2,ASK
,,,,FID,4356,,ACC_SIZE,1500,
,,,,FID,3430,,NO_ORD,12,
,,,,FID,6527,,LV_TIM_MS,3998230,
,,,,FID,6529,,LV_DATE,2018-01-15,
,,,,FID,3886,,ORDER_TONE," ",
,,,,FID,14268,,LV_TIM_NS,01:06:38.230599000,


elektron-sdktick-history-rest-api
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

zoya faberov avatar image   zoya faberov

Hello @vic,

Thank you for your participation in the forum.

Is one of the replies below satisfactory in resolving your query? If yes please click the 'Accept' text next to the reply. This will guide all community members who have a similar question. Otherwise please post again offering further insight into your question.

Thanks,

AHS

pimchaya.wongrukun01 avatar image   pimchaya.wongrukun01

Hello @vic

Thank you for your participation in the forum.

Is one of the replies below satisfactory in resolving your query? If yes please click the 'Accept' text next to the reply. This will guide all community members who have a similar question. Otherwise please post again offering further insight into your question.

Thanks,

AHS

wasin.w avatar image   wasin.w

Please be informed that a reply has been verified as correct in answering the question, and has been marked as such.

Thanks,

AHS

8 Answers

· Write an Answer
Upvotes
Accepted
Christiaan Meihsl avatar image
13.7k ●26 ●8 ●12

@vic, a refresh message presents the current state, i.e. a snapshot at one point in time.

I don't see why one should clear the cache when receiving a refresh message, but maybe I'm missing something, I'm not a data expert.

For data content query like this one, the best and speediest way to receive an answer is to open a content-related enquiry via My Account or to call the Thomson Reuters Help Desk directly, making sure you specify that you are working with TRTH Market By Price (level 2) data.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
nick.zincone avatar image
17.1k ●80 ●39 ●63

Hi @vic,

Assuming you are using an Elektron API, EMA for example, to process your MBP data, it is possible to receive an unsolicited Refresh message to not clear your cache. That is, this section describes how a Provider would indicate to a consume what to do:

When you receive a refresh, you can capture the above settings within the Refresh Indication flag:

Hope this helps.


ahs2.png (36.5 KiB)
ahs.png (46.6 KiB)
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
vic avatar image
1 ●3 ●1 ●7

@nick.zincone.1

Thanks for your reply. This question is about Thomson Reuters Tick History data I download from DataScope Select not from the Elektron API. It's historical data downloaded by Thomson Reuters with the Elektron API and saved in historical files. But there's no way to tell if we should clear cache or not on the Refresh messages because it's not specified in the files.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

nick.zincone avatar image nick.zincone

Hi @vic,

Given you included the "elektron api" as a group within your question, I thought you may have been using one of those APIs to access the data. That being said, I don't know whether TRTH can include these details within the files. I've solicited some help to see if that is possible.

Gurpreet avatar image   Gurpreet nick.zincone

Asking @Christiaan Meihsl if he knows or we should direct the user to contact the content helpdesk.

Upvotes
zoya faberov avatar image
32.2k ●40 ●11 ●20

Hello @vic,

I agree with @Christiaan Meihsl,

The requirement to clear and repopulate application cache, and the helpful clearCache indications found in refresh messages are designed for processing of market data in realtime, on subscription to realtime data feeds.

The indications are not relevant for historical data, that can be retrieved from Datascope and designed for non-realtime, historical data analytics.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
vic avatar image
1 ●3 ●1 ●7

@Christiaan Meihsl @zoya.farberov

So if a refresh message presents the current state, and that state we save in our applications cache (for example, we save the state of the order book). Then on a subsequent refresh message we need to have clearCache or not clearCache to distinguish whether we should clear and repopulate the order book or just add the values of the refresh message to it.

This does not differ between real-time data directly from real-time feeds or historical data from Datascope. For historical data I would like to feed the historical data in a program which simulates real-time trading (creating and maintaining the order book over time), so it also needs to distinguish whether it need to clear or not clear the cache (saved data).

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
nick.zincone avatar image
17.1k ●80 ●39 ●63

@vic,

Because you are not using our streaming APIs, the data you request for will always be up to date and you will need to completely replace your application order book. When you explicitly request for data, the refresh will always instruct you to clear the cache.

The Indication to not clear the cache will never apply to applications that are trying to simulate real time because you will never get this indication when you request for the data. It only applies to our streaming APIs and it is a rare asynchronous event.

Hope this is clear.

Nick

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
vic avatar image
1 ●3 ●1 ●7

@nick.zincone.1 If we would replace the complete order book in my example at the start of this topic with the data of the REFRESH, then there would be nothing in the order book afterwards. Because the REFRESH is empty of prices. This can not be right, as this is during trading hours. Also the many updates messages that comes afterwards indicates that something is not right if we clear the order book there.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

nick.zincone avatar image nick.zincone

@vic,

Are you saying that when you request for an order book, you initially get a REFRESH with the contents of that order book, then later request for the same data during market hours, you get a REFRESH that is empty? If so, please post a new question showing how you are doing this.

Thanks.

vic avatar image vic

@nick.zincone.1 I did not request the data myself, these are real-time API (ELEKTRON) historical recordings Thomson Reuters provides as a product. It's just a recording of the stream, saved into files which you can download from a website.

Upvotes
zoya faberov avatar image
32.2k ●40 ●11 ●20

Hello @vic,

Agree with you and with @nick.zincone.1,

As Datascope does not contain indications to clear cache (is designed to archive content only)

The way I would proceed, in order to back-test, on REFRESH for an instrument, I would clear the book for that instrument and recreate from the refresh.

Does this approach make sense?

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

vic avatar image vic

@zoya.farberov But as in my example, some refreshes are empty of order book data. If I clear there and recreate I end up with an empty order book, which is wrong.

zoya faberov avatar image zoya faberov

Hi @vic,

I can see your point. This goes back to Datascope being designed to hold content, and not expressly designed for back-testing of applications, by storing the way in which it was published.

I would make a judgement call on refresh, and if you see an empty one, discard it.

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.