Discover Refinitiv
MyRefinitiv Refinitiv Perspectives Careers
Created with Sketch.
All APIs Questions & Answers  Register |  Login
Ask a question
  • Questions
  • Tags
  • Badges
  • Unanswered
Search:
  • Home /
  • TREP APIs /
  • RFA /
avatar image
Question by RG1 · Oct 05, 2020 at 01:47 PM · rfardm symbol list

RFA RSSL recommended size of fragment for large RDM symbol list

RFA RSSL OMMConsumer gets RSSL_RET_INCOMPLETE_DATA in class MapReadIterator in method forth() exception when processing large RDM Symbol List . Consumer RFA API has does not define numInputBuffers parameter, assuming default 5.

1. What is recommended size of fragment for OMM provider, publishing large symbol list refresh to split into multiple fragments, considering consumer uses default numInputBuffers.

2. What is exact meaning of "input buffer size" in RFA Configuration guide? The guide mentions [numInputBuffers * input buffer size bytes] is a number of bytes for a single read socket call.


Thank you

People who like this

0 Show 0
Comment
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

10 Replies

  • Sort: 
avatar image
REFINITIV
Best Answer
Answer by moragodkrit.chumsri_1 · Nov 04, 2020 at 08:41 AM

The case 09235979 is now closed. It appeared to be caused by the misconfiguration on the infra, truncating the list.

Comment

People who like this

0 Show 2 · Share
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

avatar image
REFINITIV
umer.nalla ♦♦ · Nov 04, 2020 at 09:59 AM 0
Share

Hi @moragodkrit.chumsri_1

Can you please provide more details on the misconfiguration - might be useful for future reference?

Thanks.

avatar image
REFINITIV
moragodkrit.chumsri_1 ♦♦ umer.nalla ♦♦ · Nov 04, 2020 at 10:10 AM 0
Share

@umer.nalla

It has no problem with a direct connection between Provider and Consumer; therefore, the market data team re-configures the infra, and the problem does not occur. Then they just ask veerapath close the case. So we don't know what exactly causes the issue on the infra side.

avatar image
REFINITIV
Answer by moragodkrit.chumsri_1 · Oct 06, 2020 at 05:31 AM

@RG1

Hi Rimma,

What version of RFA C++ you are using?

As far as I understand the numinputbuffer is RSSL Channel Parameter. It's the number of input buffers available to an RSSL channel. This is the amount of data RSSL will attempt to read off of the wire with each socket read call. But the problem you are experiencing is the exception when calling MapReadIterator::forth. Normally, if the entry at this position is blank or with Invalid / Incomplete data an 'InvalidUsageException' is thrown.


  • Does the problem always occur when your provider publishes the same symbolist? And does it occurs when it decodes the map from the same/specific fragment?

I think you can turn on the RSSL tracing log on both the Provider and Consumer side. So you can compare the data when the problem occurs.

With the information I have, for now, I still not quite sure why the problem may relate to the default input buffer. If the Provider can encode the fragments without any problem, the consumer should be able to handle the messages as well in my opinion.

Comment

People who like this

0 Show 0 · Share
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

avatar image
Answer by RG1 · Oct 06, 2020 at 03:19 PM

Hi Moragodkrit,

The problem happens when provider publishes large symbol list ( map of about 6K items) . Enabled RFA tracing on both sides. RDM symbol list message is truncated on consumer side. What can be explained by a small read buffer, I suspect. Consumer is on RFA C++ version 8.1.0.L1.

Comment

People who like this

0 Show 0 · Share
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

avatar image
Answer by RG1 · Oct 06, 2020 at 03:40 PM

Moragodkrit, in addition, smaller fragments work, consumer is able to process the full list. Question - what's the limit on size of fragment. It is clearly driven by the consumer read ability.

Regards

Rimma

Comment

People who like this

0 Show 0 · Share
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

avatar image
REFINITIV
Answer by umer.nalla · Oct 06, 2020 at 04:19 PM

Hi @RG1

Whilst I am not sure about a limit on the Fragment size from an OMM SymbolList Encoding/decoding point of view, my understanding is that whilst you can use payloads larger than 64k - the TREP infrastructure components i.e. ADS and ADH are optimised for 64k message sizes.

Therefore, purely from an ADS+ADH performance point of view, I would recommend you limit any multi fragment messages to 64k size max for each fragment.

I am also aware that making batch requests, there is a limit of 64K on the batch request message payload as well - so the same may apply to Multipart message fragments too.

Hopefully, my colleague will be able to provide a definitive answer.

Comment

People who like this

0 Show 0 · Share
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

avatar image
Answer by RG1 · Oct 06, 2020 at 04:49 PM

Hi @umer.nalla ,

Our observation on the infra side: such long message/map is not a problem for ADH/ADS. The issue is observed when symbol list is requested for the first time, request goes to the provider ( interactive) . Infra passes through the response to the consumer, which fails to process it. When subscribing again, the symbol list comes back from ADH cache fragmented, and consumer able to process all fragments of the list correctly. Obviously, ADH does fragmentation, I suspect that ADH/ADS must have larger read buffer(s) , than the regular consumer, for better throughput, and able to swallow such payload.


My second question was if the issue relates to the read system buffer, as according to RFA configuration guide,

"RSSL Channel Parameter. Number of input buffers available to an RSSL channel. This is the amount of data RSSL will attempt to read off of the wire with each socket read call. This will be [numInputBuffers * input buffer size bytes] in length. "

Regards


Comment

People who like this

0 Show 1 · Share
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

avatar image
REFINITIV
umer.nalla ♦♦ · Oct 07, 2020 at 08:44 AM 1
Share

Hi @RG1

Yes, you are correct the ADS can indeed handle larger messages as mentioned in my reply 'you can use payloads larger than 64k'. However, the point I was trying to make is that the TREP components are optimised for 64K messages.


avatar image
REFINITIV
Answer by moragodkrit.chumsri_1 · Oct 07, 2020 at 05:03 AM

@RG1

I assume that the symbolist Map contains around 6k map entry count(6k items). According to RFA Dev Guide, in theory, a Map currently has a maximum entry count of 65,535 but a recommend overall fragment message seems to be around 64k as Umer said.

  • Is there any problem when connecting the consumer app to the provider app directly? Just want to make sure that it does not relate to the infra.

I did a quick test by modifying Provider_Interactive example to publish symbol list RIC which contains around 6k items/map entry. I have added key name item<count> where count starts from 1-6k. Then connecting StartConsumer_SymbolList(from RFA C++ 8.1.3 package) to the provider app. The consumer can process 6k items without the exception error. The Message size from the trace is around 178k(below is sample trace). That is a reason that I can't confirm that the problem actually relates to the default input buffer.

<refreshMsg domainType="RSSL_DMT_SYMBOL_LIST" streamId="3" containerType="RSSL_DT_MAP" flags="0x11E8 (RSSL_RFMF_HAS_MSG_KEY|RSSL_RFMF_SOLICITED|RSSL_RFMF_REFRESH_COMPLETE|RSSL_RFMF_HAS_QOS|RSSL_RFMF_CLEAR_CACHE|RSSL_RFMF_HAS_PART_NUM)" groupId="0" partNum="0" qosDynamic="0" qosRate="1" qosTimeliness="1" dataState="RSSL_DATA_OK" streamState="RSSL_STREAM_OPEN" code="RSSL_SC_NONE" text="MessageComplete"  dataSize="178910">
    <key  flags="0x7 (RSSL_MKF_HAS_SERVICE_ID|RSSL_MKF_HAS_NAME|RSSL_MKF_HAS_NAME_TYPE)"  serviceId="1" name="0#ITEMS" nameType="1"/>
    <dataBody>
        <map flags="0x1 (RSSL_MPF_HAS_SET_DEFS)" countHint="0" keyPrimitiveType="RSSL_DT_ASCII_STRING" containerType="RSSL_DT_FIELD_LIST" >
            <fieldSetDefs>
                <fieldSetDef setId="0">
                    <fieldSetDefEntry fieldId="3422" dataType="RSSL_DT_RMTES_STRING" />
                    <fieldSetDefEntry fieldId="1" dataType="RSSL_DT_UINT" />
                </fieldSetDef>
            </fieldSetDefs>
            <mapEntry flags="0x0" action="RSSL_MPEA_ADD_ENTRY" key="item1" >
                <fieldList flags="0x3 (RSSL_FLF_HAS_FIELD_LIST_INFO|RSSL_FLF_HAS_SET_DATA)" fieldListNum="0" dictionaryId="1">
                    <fieldEntry fieldId="3422" dataType="RSSL_DT_RMTES_STRING" data="appSymbol2"/>
                    <fieldEntry fieldId="1" dataType="RSSL_DT_UINT" data="3056"/>
                </fieldList>
            </mapEntry>
            <mapEntry flags="0x0" action="RSSL_MPEA_ADD_ENTRY" key="item2" >
                <fieldList flags="0x3 (RSSL_FLF_HAS_FIELD_LIST_INFO|RSSL_FLF_HAS_SET_DATA)" fieldListNum="0" dictionaryId="1">
                    <fieldEntry fieldId="3422" dataType="RSSL_DT_RMTES_STRING" data="appSymbol1"/>
                    <fieldEntry fieldId="1" dataType="RSSL_DT_UINT" data="3056"/>
                </fieldList>
            </mapEntry>
            ...
            <mapEntry flags="0x0" action="RSSL_MPEA_ADD_ENTRY" key="item5999" >
                <fieldList flags="0x3 (RSSL_FLF_HAS_FIELD_LIST_INFO|RSSL_FLF_HAS_SET_DATA)" fieldListNum="0" dictionaryId="1">
                    <fieldEntry fieldId="3422" dataType="RSSL_DT_RMTES_STRING" data="appSymbol2"/>
                    <fieldEntry fieldId="1" dataType="RSSL_DT_UINT" data="3056"/>
                </fieldList>
            </mapEntry>
            <mapEntry flags="0x0" action="RSSL_MPEA_ADD_ENTRY" key="item6000" >
                <fieldList flags="0x3 (RSSL_FLF_HAS_FIELD_LIST_INFO|RSSL_FLF_HAS_SET_DATA)" fieldListNum="0" dictionaryId="1">
                    <fieldEntry fieldId="3422" dataType="RSSL_DT_RMTES_STRING" data="appSymbol1"/>
                    <fieldEntry fieldId="1" dataType="RSSL_DT_UINT" data="3056"/>
                </fieldList>
            </mapEntry>
        </map>
    </dataBody>
</refreshMsg>


Since this case requires a further investigation to confirm the issue.

Can you please open a new support ticket for this issue?

You can open a new support case by going to RFA Section and click Contact Premium Support



Comment

People who like this

0 Show 0 · Share
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

avatar image
REFINITIV
Answer by umer.nalla · Oct 07, 2020 at 09:18 AM

Hi @RG1

Parameters such as numInputBuffers relate to the transport/network layer whereas the OMM encoding/decoding of messages and payloads is happening at the API level - therefore the changes to numinputBuffers should not impact the ability to encode and decode larger OMM messages.

Just to be clear numInputBuffers specifies the number of sequential input buffers to allocate for reading data into. This controls the maximum number of bytes that can be handled with a single network read operation. Changes to this value will affect how fast we can pull in data from a full OS receive buffer - not the size of any OMMMessag payloads being transported on the network.

Comment

People who like this

0 Show 0 · Share
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

avatar image
Answer by RG1 · Oct 07, 2020 at 05:44 PM

Tested direct consumer-publisher connection - no issue with the large symbol list. Infra seems to be a culprit. ADH/ADS version trep3.3.0.L1

I tried to open premium support ticket, several times I get "an error occurred, Please try later.." on Submit. Please fix the service.

Thank you


Comment

People who like this

0 Show 0 · Share
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

avatar image
REFINITIV
Answer by umer.nalla · Oct 08, 2020 at 10:53 AM

Hi @RG1

Apologies for the problem you are facing raising a ticket - we are having some issues with the new developer portal.

I have created a Support Ticket in your name and email address ref 09235979 - you should get an email shortly. If not please post back here.


Comment

People who like this

0 Show 0 · Share
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Watch this question

Add to watch list
Add to your watch list to receive emailed updates for this question. Too many emails? Change your settings >
10 People are following this question.

Related Questions

RFA login request

RSSL buffer failed

RFA8_NET120_x64.dll upgrade frequency

Invalid ILA login(E0011), USER_NOT_PERMIT

RFA suspect state

  • Feedback
  • Copyright
  • Cookie Policy
  • Privacy Statement
  • Terms of Use
  • Careers
  • Anonymous
  • Sign in
  • Create
  • Ask a question
  • Spaces
  • Alpha
  • App Studio
  • Block Chain
  • Bot Platform
  • Calais
  • Connected Risk APIs
  • DSS
  • Data Fusion
  • Data Model Discovery
  • Datastream
  • Eikon COM
  • Eikon Data APIs
  • Elektron
    • EMA
    • ETA
    • WebSocket API
  • Legal One
  • Messenger Bot
  • Messenger Side by Side
  • ONESOURCE
    • Indirect Tax
  • Open PermID
    • Entity Search
  • Org ID
  • PAM
    • PAM - Logging
  • ProView
  • ProView Internal
  • Product Insight
  • Project Tracking
  • Refinitiv Data Platform
    • Refinitiv Data Platform Libraries
  • Rose's Space
  • Screening
    • Qual-ID API
    • Screening Deployed
    • Screening Online
    • World-Check One
    • World-Check One Zero Footprint
  • Side by Side Integration API
  • TR Knowledge Graph
  • TREP APIs
    • CAT
    • DACS Station
    • Open DACS
    • RFA
    • UPA
  • TREP Infrastructure
  • TRIT
  • TRKD
  • TRTH
  • Thomson One Smart
  • Transactions
    • REDI API
  • Velocity Analytics
  • Wealth Management Web Services
  • World-Check Data File
  • Explore
  • Tags
  • Questions
  • Badges