Generic Message Payload Size?
Right next question. Now I've got a .Net ValueAdd Provider and Consumer exchanging XML data using OMM generic messages. Is there a limit to the payload size? Some of our documents are about 40 Mbyte. Is that too big? Should I consider zipping them up first?
Hi @RicknBaker
As mentioned above RSSL can handle large payloads.
However, when I was looking at large payloads a while back, I was recommended that in order optimise the data distribution across the TREP infrastructure, I should ideally limit the size to no greater then 6k bytes. This is the size of the RSSL message buffer in the infrastructure. Larger messages mean the infrastructure has to break up and re-assemble at the expense of extra resources.
You mentioned in your other posts that data wont be sent too often, so you may not be overly concerned about performance - but you should discuss with your Market Data team about possible impact on other users of your TREP infrastructure.
Based, on the above, zipping the data would appear to be a good idea - however, you should do some testing to determine for yourself the suitability of doing so.
If you take a look at our tutorial on Decoding Machine Readable News you will note that we zip and split our larger news payloads into smaller fragments for distribution. The consumer then has to unzip and re-assemble the fragments to get the full payload.
According to the specification of the underlying API (ETA), the size of payload in the message can be up to 4,294,967,295 bytes. This payload length is typically limited by the contained type’s specification.
However, if the XML is in the container's entry, the size is limited to 65,535 bytes.
OK, thanks I'll take a look at compression. Things seem to blow up spectacularly with big payloads .
Are there any samples showing how to send a multi fragment generic message?
Thanks guys, I appreciate the help.
I can't find any samples for this.
However, to send multi-part generic messages, the application needs to set the GenericMsg.IndicationMaskFlag.MessageComplete flag on the final part.
genericMsg.IndicationMask = GenericMsg.IndicationMaskFlag.MessageComplete
For example, if you would like to send four generic messages for a XML file. The first, second, and third generic messages will not have the GenericMsg.IndicationMaskFlag.MessageComplete flag. Only the forth (final) part will have the GenericMsg.IndicationMaskFlag.MessageComplete flag.
MessageComplete flag is used to indicate whether this generic message is the final portion of the message.
Moreover, you can also set GenericMsg.SeqNum and GenericMsg.SecondarySeqNum to specify the order of messages.
genericMsg.SeqNum = 1; genericMsg.SecondarySeqNum = 2;
Thanks. I now have an implementation that breaks big payloads down and submits them as individual messages Presumably, there's no chance of these getting out of order or dropped is there?
I did discover that a huge payload, around 2Mb of compressed XML caused the ADH server I connect to to hang. Last message seen in the log was
<xxxxx.1.adh.rrcpTransport.1.srcSide.rrcp.send.transmissionBus: Warning: Tue Feb 13 12:54:22 2018> RRCP STATUS MSG: RRCP_CONG_BEGIN: possible congestion sets in<END>
<xxx Warning: Tue Feb 13 12:54:23 2018> RRCP STATUS MSG: RRCP_CONG_END: congestion ends<END>
It uses TCP/IP to send messages. Therefore, if the application sends messages in order, there is no chance to get out of order or dropped messages.
Moreover, TREP also uses reliable UDP protocol so messages couldn't be out of order.
However, you can use GenericMsg.SeqNum and GenericMsg.SecondarySeqNum to specify the order of messages.
For the error messages on TREP, please direct contact TREP support team via Contact Us.