How to extract metadata using Real-Time SDK

Y_Intercept

The raw dump files downloaded from S3 VDB has some useful metadata, such as the timestamp and type of a message (pls check the attachment if you are not sure what I am referring to).

We want to extract the same from Real-Time SDK, when the updates are streamed to use in real-time.

How can we achieve this?

Jirapongse

@Y_Intercept

Thank you for reaching out to us.

You need to run another simple application (such as testclient, or cons100) that subscribes to one RIC from the same server to verify what the problem is. If you still see a high latency in another simple applicaiton, the problem could be on the server side. Otherwise, the problem could be on the application side.

To verify a problem in the application, you need to contact the API support team via Contact Premium Support. However, you need to have a Refinitiv Developer Connect (RDC) named user in order to submit a query to the API suppor team. For more information regarding RDC, please contact your LSEG account team or sales team.

Jirapongse

@Y_Intercept

Thank you for reaching out to us.

The timestamp is an application's timestamp so it is not available in the update message.

In EMA C++, you can get the update type from the UpdateMsg::getUpdateTypeNum() method. It returns the following types.

Y_Intercept

Hi @Jirapongse ,

We are using C#, is the same enum also defined in the C# version of the Real-Time SDK?

• If yes, can you show me the Enum name in C# SDK?
• If not, is the above doxygen doc available to the public? If C# enum is not available, we want to keep a reference to these values in our codebase.

Jirapongse

https://community.developers.refinitiv.com/discussion/comment/122941#Comment_122941

Please check the reference guide in the package.

The enums are defined in the LSEG.Ema.Rdm.EmaRdm class.

The method is UpdateMsg.UpdateTypeNum().

Y_Intercept

Got it, guess this UpdateMsg::getUpdateTypeNum() wont help as the sample RIC we are checking have all the updates marked as UNSPECIFIED:

301372.SZ,Market Price,2024-11-18T01:59:22.810483189Z,+8,rawMP,UPDATE,UNSPECIFIED,,,,5601,,26672,44
,,,,FID,25,,ASK,22.65,
,,,,FID,730,,BEST_BSIZ1,300,
,,,,FID,30,,BIDSIZE,300,
,,,,FID,77,,NUM_MOVES,1607,
,,,,FID,735,,BEST_ASIZ1,300,
,,,,FID,22,,BID,22.59,
,,,,FID,445,,BEST_ASK5,22.76,
,,,,FID,56,,PCTCHNG,-1.778,
,,,,FID,3854,,SALTIM_MS,7161000,
,,,,FID,11,,NETCHNG_1,-0.41,
,,,,FID,379,,SALTIM,01:59:21.000000000,
,,,,FID,32741,,TRNOVR_UNS,11629163,
,,,,FID,734,,BEST_BSIZ5,100,
,,,,FID,13432,,TR_TRD_FLG,1,"OB   "
,,,,FID,178,,TRDVOL_1,300,
,,,,FID,3404,,VWAP,22.731,
,,,,FID,443,,BEST_ASK3,22.69,
,,,,FID,31,,ASKSIZE,300,
,,,,FID,32743,,ACVOL_UNS,511600,
,,,,FID,736,,BEST_ASIZ2,1900,
,,,,FID,100,,TURNOVER,11629163,
,,,,FID,6,,TRDPRC_1,22.65,
,,,,FID,438,,BEST_BID3,22.56,
,,,,FID,1496,,WEIGHTING,22.731,
,,,,FID,439,,BEST_BID4,22.55,
,,,,FID,440,,BEST_BID5,22.47,
,,,,FID,739,,BEST_ASIZ5,1000,
,,,,FID,732,,BEST_BSIZ3,500,
,,,,FID,733,,BEST_BSIZ4,9900,
,,,,FID,444,,BEST_ASK4,22.74,
,,,,FID,442,,BEST_ASK2,22.66,
,,,,FID,5,,TIMACT,01:59:22.000000000,
,,,,FID,373,,IRGVOL,511600,
,,,,FID,731,,BEST_BSIZ2,300,
,,,,FID,436,,BEST_BID1,22.59,
,,,,FID,738,,BEST_ASIZ4,11300,
,,,,FID,1067,,EXCHTIM,01:59:21.000000000,
,,,,FID,737,,BEST_ASIZ3,1100,
,,,,FID,32,,ACVOL_1,511600,
,,,,FID,18,,TRDTIM_1,01:59:21.000000000,
,,,,FID,441,,BEST_ASK1,22.65,
,,,,FID,437,,BEST_BID2,22.58,
,,,,FID,14,,PRCTCK_1,2,#FE#
,,,,FID,3855,,QUOTIM_MS,7161000,

Now the question we are facing is this, in the dump file, multiple timestamps, such as

FID,18,,TRDTIM_1,01:59:21.000000000,
FID,3854,,SALTIM_MS,7161000,
FID,1067,,EXCHTIM,01:59:21.000000000,

are consistent, and the metadata timestamp is 2024-11-18T01:59:22.810483189Zmeaning that it takes ~2.81 seconds for the updates to reach your application. By our standard this is not too bad.

However the poblem is, from our feed handler, we stamp the time immediately after our feed handler receives the update, our FH's timestamp is 2024-11-18 01:59:40.332, meaning that it takes almost 20 seconds to reach our feed handler.

Now the problem is, without your metadata timestamp (i.e., 2024-11-18T01:59:22.810483189Zfrom the dump csv file) available in realtime, which component is to be blamed? Usually how will you advise the clients to investigate this kind of latency issue?

Y_Intercept

Okay, and I just dug into the data, seems this UpdateTypeNum() is not that useful. For the RIC we are currently investigating, all the updates are marked as UNSPECIFIED

301372.SZ,Market Price,2024-11-18T01:59:22.810483189Z,+8,rawMP,UPDATE,UNSPECIFIED,,,,5601,,26672,44
,,,,FID,25,,ASK,22.65,
,,,,FID,730,,BEST_BSIZ1,300,
,,,,FID,30,,BIDSIZE,300,
,,,,FID,77,,NUM_MOVES,1607,
,,,,FID,735,,BEST_ASIZ1,300,
,,,,FID,22,,BID,22.59,
,,,,FID,445,,BEST_ASK5,22.76,
,,,,FID,56,,PCTCHNG,-1.778,
,,,,FID,3854,,SALTIM_MS,7161000,
,,,,FID,11,,NETCHNG_1,-0.41,
,,,,FID,379,,SALTIM,01:59:21.000000000,
,,,,FID,32741,,TRNOVR_UNS,11629163,
,,,,FID,734,,BEST_BSIZ5,100,
,,,,FID,13432,,TR_TRD_FLG,1,"OB   "
,,,,FID,178,,TRDVOL_1,300,
,,,,FID,3404,,VWAP,22.731,
,,,,FID,443,,BEST_ASK3,22.69,
,,,,FID,31,,ASKSIZE,300,
,,,,FID,32743,,ACVOL_UNS,511600,
,,,,FID,736,,BEST_ASIZ2,1900,
,,,,FID,100,,TURNOVER,11629163,
,,,,FID,6,,TRDPRC_1,22.65,
,,,,FID,438,,BEST_BID3,22.56,
,,,,FID,1496,,WEIGHTING,22.731,
,,,,FID,439,,BEST_BID4,22.55,
,,,,FID,440,,BEST_BID5,22.47,
,,,,FID,739,,BEST_ASIZ5,1000,
,,,,FID,732,,BEST_BSIZ3,500,
,,,,FID,733,,BEST_BSIZ4,9900,
,,,,FID,444,,BEST_ASK4,22.74,
,,,,FID,442,,BEST_ASK2,22.66,
,,,,FID,5,,TIMACT,01:59:22.000000000,
,,,,FID,373,,IRGVOL,511600,
,,,,FID,731,,BEST_BSIZ2,300,
,,,,FID,436,,BEST_BID1,22.59,
,,,,FID,738,,BEST_ASIZ4,11300,
,,,,FID,1067,,EXCHTIM,01:59:21.000000000,
,,,,FID,737,,BEST_ASIZ3,1100,
,,,,FID,32,,ACVOL_1,511600,
,,,,FID,18,,TRDTIM_1,01:59:21.000000000,
,,,,FID,441,,BEST_ASK1,22.65,
,,,,FID,437,,BEST_BID2,22.58,
,,,,FID,14,,PRCTCK_1,2,#FE#
,,,,FID,3855,,QUOTIM_MS,7161000,

Now our problem is this, using the above update from a dump csv file as a example, many timestamp field, such as:

301372.SZ,Market Price,2024-11-18T01:59:22.810483189Z,+8,rawMP,UPDATE,UNSPECIFIED,,,,5601,,26672,44
,,,,FID,379,,SALTIM,01:59:21.000000000,
,,,,FID,5,,TIMACT,01:59:22.000000000,
,,,,FID,1067,,EXCHTIM,01:59:21.000000000,
,,,,FID,18,,TRDTIM_1,01:59:21.000000000
,,,,,FID,3855,,QUOTIM_MS,7161000

are consistent, showing that such update happened at 2024-11-18T01:59:21Z. And the metadata timestamp 2024-11-18T01:59:22.810483189Zimplies that it takes ~2.8 seconds for the dumping application to receive this update. By our standard, this is not too bad. The real problem is this: our real-time feed handler also stamp a time immiedately after such update is received from a call back, accroding to our record, this update was received by our FH at 2024-11-18 01:59:40.332meaning that it takes almost 20 seconds for our FH to receive it.

The question is, which component(s) is to be blamed? Is it that our FH just too slow or is it that the network is just not stable? Without a metadata timestamp, we cant be sure now. For this kind of question, usually how will you advise your clients to do?

Thanks,

Y_Intercept

Okay, and I just dug into the data, seems this UpdateTypeNum() is not that useful. For the RIC we are currently investigating, all the updates are marked as UNSPECIFIED

301372.SZ,Market Price,2024-11-18T01:59:22.810483189Z,+8,rawMP,UPDATE,UNSPECIFIED,,,,5601,,26672,44,,,,FID,25,,ASK,22.65,,,,,FID,730,,BEST_BSIZ1,300,,,,,FID,30,,BIDSIZE,300,,,,,FID,77,,NUM_MOVES,1607,,,,,FID,735,,BEST_ASIZ1,300,,,,,FID,22,,BID,22.59,,,,,FID,445,,BEST_ASK5,22.76,,,,,FID,56,,PCTCHNG,-1.778,,,,,FID,3854,,SALTIM_MS,7161000,,,,,FID,11,,NETCHNG_1,-0.41,,,,,FID,379,,SALTIM,01:59:21.000000000,,,,,FID,32741,,TRNOVR_UNS,11629163,,,,,FID,734,,BEST_BSIZ5,100,,,,,FID,13432,,TR_TRD_FLG,1,"OB   ",,,,FID,178,,TRDVOL_1,300,,,,,FID,3404,,VWAP,22.731,,,,,FID,443,,BEST_ASK3,22.69,,,,,FID,31,,ASKSIZE,300,,,,,FID,32743,,ACVOL_UNS,511600,,,,,FID,736,,BEST_ASIZ2,1900,,,,,FID,100,,TURNOVER,11629163,,,,,FID,6,,TRDPRC_1,22.65,,,,,FID,438,,BEST_BID3,22.56,,,,,FID,1496,,WEIGHTING,22.731,,,,,FID,439,,BEST_BID4,22.55,,,,,FID,440,,BEST_BID5,22.47,,,,,FID,739,,BEST_ASIZ5,1000,,,,,FID,732,,BEST_BSIZ3,500,,,,,FID,733,,BEST_BSIZ4,9900,,,,,FID,444,,BEST_ASK4,22.74,,,,,FID,442,,BEST_ASK2,22.66,,,,,FID,5,,TIMACT,01:59:22.000000000,,,,,FID,373,,IRGVOL,511600,,,,,FID,731,,BEST_BSIZ2,300,,,,,FID,436,,BEST_BID1,22.59,,,,,FID,738,,BEST_ASIZ4,11300,,,,,FID,1067,,EXCHTIM,01:59:21.000000000,,,,,FID,737,,BEST_ASIZ3,1100,,,,,FID,32,,ACVOL_1,511600,,,,,FID,18,,TRDTIM_1,01:59:21.000000000,,,,,FID,441,,BEST_ASK1,22.65,,,,,FID,437,,BEST_BID2,22.58,,,,,FID,14,,PRCTCK_1,2,#FE#,,,,FID,3855,,QUOTIM_MS,7161000,

Now our problem is this, using the above update from a dump csv file as a example, many timestamp field, such as:

301372.SZ,Market Price,2024-11-18T01:59:22.810483189Z,+8,rawMP,UPDATE,UNSPECIFIED,,,,5601,,26672,44,,,,FID,379,,SALTIM,01:59:21.000000000,,,,,FID,5,,TIMACT,01:59:22.000000000,,,,,FID,1067,,EXCHTIM,01:59:21.000000000,,,,,FID,18,,TRDTIM_1,01:59:21.000000000,,,,,FID,3855,,QUOTIM_MS,7161000

are consistent, showing that such update happened at 2024-11-18T01:59:21Z. And the metadata timestamp 2024-11-18T01:59:22.810483189Zimplies that it takes ~2.8 seconds for the dumping application to receive this update. By our standard, this is not too bad. The real problem is this: our real-time feed handler also stamp a time immiedately after such update is received from a call back, accroding to our record, this update was received by our FH at 2024-11-18 01:59:40.332meaning that it takes almost 20 seconds for our FH to receive it.

The question is, which component(s) is to be blamed? Is it that our FH just too slow or is it that the network is just not stable? Without a metadata timestamp, we cant be sure now. For this kind of question, usually how will you advise your clients to do?

Thanks,