EMA - only partially processing batch request when using a view simultaneously

Hello,

For context, I am using Kotlin, so if I specify any class names it is the Java counterpart.

I have a list of RICs (2098 to be exact, which I get from iterating through the ASX equities chain) which I add to an OmmArray.

I then create a view-related OmmArray with the following field IDs: 22, 25, 30, 31, 1025, 3386. These are BID, ASK, BIDSIZE, ASKSIZE, QUOTIM, QUOTE_DATE respectively.

I add both these arrays to an ElementList, along with the specifying the view type to be FIDs.

I then add this ElementList to a message request, and register this message request to a single OmmConsumerClient - this is all pretty standard stuff.

What happens next is: 856 RICs are successfully requested. The remaining RICs are never seen in ANY messages. When I play around with different FIDs in the View, this number changes. Sometimes it's more and sometimes it's less.

Even narrowing the view scope down to only include FID 22 (BID), the full list of RICs is still not processed.

How can I go about batch requesting these RICs with the aforementioned view, and guarantee subbing to every RIC in the list? Also for my understanding, why is this failing?

After more debugging: even requesting each RIC individually with a view is not subbing to all RICs despite definitely sending the request off. The only way to successfully sub to the RICs I want is to remove the View. It seems like some sort of block on your side.

NOTE: RICs are requested sequentially (even in the batch on your end, it seems), and only a sequential subset of the RICs are successfully subbed to. This means all RICs beyond a certain RIC are not subbed to, and all RICs before a certain RIC are subbed to. That is why I assume some sort of block.

Help is appreciated. Thank you.

Find more posts tagged with

java

batch

ema-api

batch-request

Accepted answers

wasin.w

Hello @red

By default, the EMA API sends all requests registered by the application to the server directly. Your workaround is creating the application layer logic to throttle requests before sending them to EMA which is the suggested way when requesting a lot of items.

Do you mean the problem does not occur if you subscribe to all fields request, but if you use View, the problem occurs?

If so, I recommend you contact the API support team via the "Contact premium support" button on the https://developers.refinitiv.com/en/api-catalog/refinitiv-real-time-opnsrc/rt-sdk-java page.

Alternatively, you may submit this issue to the RTSDK Development team directly via RTSDK - GitHub Issues page.

Hope this helps.

All comments

wasin.w

Hello @red

What EMA Java (or RTSDK) version that you are using? Could you please share the snippet code that constructs the View, Batch, and the request message?

Please also enable the XML trace messages. The XML trace contains incoming and outgoing messages between the API and the server, so the messages can help us to verify if the API receives data from the server. You can enable the XML trace messages by setting the XmlTraceToStdOut parameter in the EmaConfig.xml to "1".

You can find more detail on the "EMA Java Logging" section of the Enterprise Message API (EMA) - Configuration Overview article.

red

hi @wasin.w, thanks for your reply.

I am using the following:

implementation("com.refinitiv.ema:ema:3.6.4.0")

Minimal example code block where I construct a request:

class MarketDataConsumerClient : OmmConsumerClient {
   // NOTE THAT THE CONSUMER NEEDS TO BE CONSTRUCTED BEFORE THIS CODE WORKS
    private var consumer: OmmConsumer? = null
    private val service = "ERT_FD3_LF1"

    private val fieldIds = mapOf(
        "BID" to 22L,
        "ASK" to 25L,
        "BIDSIZE" to 30L,
        "ASKSIZE" to 31L,
        "QUOTIM" to 1025L,
        "QUOTE_DATE" to 3386L
    )


    private fun requestBatchRics(rics: List<String>) {
        val ricArray = EmaFactory.createOmmArray().apply {
            rics.forEach { add(EmaFactory.createOmmArrayEntry().ascii(it)) }
        }
        // The view array allows us to customize what data we are interested in (reduces amount of data received)
        val viewDataArray = EmaFactory.createOmmArray().apply {
            fieldIds.values.forEach { fieldValue -> add(EmaFactory.createOmmArrayEntry().intValue(fieldValue)) }
        }

        val batchList = EmaFactory.createElementList().apply {
            add(EmaFactory.createElementEntry().array(EmaRdm.ENAME_BATCH_ITEM_LIST, ricArray))
            add(EmaFactory.createElementEntry().uintValue(EmaRdm.ENAME_VIEW_TYPE, 1))
            add(EmaFactory.createElementEntry().array(EmaRdm.ENAME_VIEW_DATA, viewDataArray))
        }

        val requestMessage = EmaFactory
            .createReqMsg()
            .serviceName(service)
            .payload(batchList)
        consumer!!.registerClient(requestMessage, this)
    }
}

A trace for a request for a RIC that doesn't sub/send any message:

<!-- rwfMajorVer="14" rwfMinorVer="1" -->
<REQUEST domainType="MARKET_PRICE" streamId="1226" containerType="ELEMENT_LIST" flags="0x446 (HAS_PRIORITY|STREAMING|HAS_QOS|HAS_VIEW)" Qos: Realtime/ConflatedByRateInfo/Static - timeInfo: 0 - rateInfo: 3000 priorityClass="1" priorityCount="1" dataSize="46">
    <key flags="0x03 (HAS_SERVICE_ID|HAS_NAME)" serviceId="253" name="KFM.AX"/>
    <dataBody>
        <elementList flags="0x08 (HAS_STANDARD_DATA)">
            <elementEntry name=":ViewType" dataType="UINT" data="1"/>
            <elementEntry name=":ViewData" dataType="ARRAY">
                <array itemLength="2" primitiveType="INT">
                    <arrayEntry data="22"/>
                    <arrayEntry data="25"/>
                    <arrayEntry data="30"/>
                    <arrayEntry data="31"/>
                    <arrayEntry data="1025"/>
                    <arrayEntry data="3386"/>
                </array>
            </elementEntry>
        </elementList>
    </dataBody>
</REQUEST>

EDIT: the above was just processed - let me make sure I capture output of one that definitely doesn't work.

EDIT 2: This is now working? I have no explanation other than the additional logging is slowing down the requests due to file IO. Your explanation is much appreciated.

Is there a limit on outstanding requests that Refinitiv can process on your end before it throws additional requests away? I added a Thread.sleep(1) between individual RIC requests and everything's working. This is definitely hacky and I do not want to do this.

EDIT 3: The following code block works. Please note the Thread.sleep(1). This is not a viable solution for me, so your help is appreciated. Something is dropping requests/responses.

private fun requestRicsIndividually(rics: List<String>) {
        val viewDataArray = EmaFactory.createOmmArray().apply {
            fieldIds.values.forEach { fieldValue -> add(EmaFactory.createOmmArrayEntry().intValue(fieldValue)) }
        }
        val batchList = EmaFactory.createElementList().apply {
            add(EmaFactory.createElementEntry().uintValue(EmaRdm.ENAME_VIEW_TYPE, 1))
            add(EmaFactory.createElementEntry().array(EmaRdm.ENAME_VIEW_DATA, viewDataArray))
        }
        rics.forEach {
            Thread.sleep(1)
            logger.debug("Requesting ric[$it]")

            val requestMessage = EmaFactory
                .createReqMsg()
                .serviceName(service)
                .payload(batchList)
                .name(it)
            consumer!!.registerClient(requestMessage, this)
        }
    }

Thank you!

red

@wasin.w My most viable solution thus far is to dynamically create a new OmmConsumer every 500 RICs on the same channel. This seems to work well. Let me know what you think.

Still - I am curious as to why a view affects RIC subscriptions, as it works flawlessly without a view. With a view, it is returning a consistent subset of the RICs requested (unless delaying subscriptions or spreading them out between multiple OmmConsumers).

Red.

wasin.w

Hello @red

Do you mean the problem does not occur if you subscribe to all fields request, but if you use View, the problem occurs?

If so, I recommend you contact the API support team via the "Contact premium support" button on the https://developers.refinitiv.com/en/api-catalog/refinitiv-real-time-opnsrc/rt-sdk-java page.

Alternatively, you may submit this issue to the RTSDK Development team directly via RTSDK - GitHub Issues page.

Hope this helps.

red

Do you recommend throttling, as opposed to creating new consumer(s)?

You are correct - the problem does not occur if I subscribe to all field requests, but if I use the view, suddenly there is a (reproducible and consistent) problem of only a subscribing to a subset of RICs.

wasin.w

Hello @red

I am sorry for the unclear answer. Your workaround that separates the list of items into multiple OmmConsumer objects is fine.

However, there are a lot of ways to make the application not subscribe all items to the API at once. It is based on your preference whether the throttling, or separate items to multiple consumers, or mix of them, etc. that match your development requirement.

EXPLORE OUR SITES