Retrieving TS1 data encoded in FIDs ROW64_1-14 from Elektron (RTO)

mauryhill

There are two problems with the way that RTO is encoding TS1 data in UTF-8

• One character is mapped to a different character in the same range
• Several characters are expanded out into two characters and given the way it's done, I'm not sure it's possible to reverse the expansion 100% correctly in all cases

I've also captured the websocket traffic in Fiddler and it looks like this is coming from upstream and we're not distorting it any way in our "app". I've attached this rto-202105181200-ts1.zip showing this binary data in FIDs ROW64_1-14 before it reaches our app. Rename the filename from .zip -> .saz to open in Fiddler.

Also attached is "ts1-rto character mappings.txt" which shows the IDN characters and how they are being mapped to RTO. We arrived at this by making the same request through the LPC (Legacy Protocol Converter) 1.2 which connects to Elektron (RTO) and then comparing the binary data in the FIDs.

Can anyone shed an light on this decoding process? Or whom to have look at this?

Maybe our request in the Fiddler SAZ is just wrong? Its pretty basic.

{
  "ID":2,"Streaming":false
  "Key":{
    "Name":["dBOc1d"],
    "Service":"ELEKTRON_DD"
  }
}

Either the LPC is dealing with these different encodings or it is making a slightly different request that causes the data to come back encoded differently.

christopher.dugan1

Hello Maury,

Per my colleague, there are UTF-8 issues when using websockets to retrieve data. I presume this applies to all data, not just TS1. Unfortunately, I don't have any ticket numbers available describing the issue of websockets handling UTF-8.

LPC comes with the testclient application for sample connections. It has the ability to output the data to a file or to STDOUT. Here is an example command line to use:

#> cd /opt/refinitiv/SOFTWARE/lpc1.2.0/linux7_x86_64/demo

#> export LD_LIBRARY_PATH=`pwd`

#> ./testclient -S ELEKTRON_DD -ct rssl -p 14002 -il dIBMd -o 512 -I 1 -O -u yourAppUserName -l stdout -dfile ../../etc/RDMFieldDictionary

The "stdout" option is using a lower case 'L'. The dictionary file is needed to decode the FIDs.

mauryhill

I created a MyRefinitiv ticket and was given the following response.

"We have received your query regarding problems with the way that RTO is encoding ts1realtime.txt in UTF-8 you saw.

We have tested on our side with ts1LPC.txt as well as with the real-time service but both cannot reproduce your mentioned issue. Please find our testing result in the attachment."

We are now comparing the above two file against our raw request with out .NET app. It is a bit more work to get what the LPC is doing since it communicates with https. We are investigating the usage of a proxy like Fiddler to capture the data for review.

mauryhill

Thank you. I did also make a My Refinitiv ticket and received two files that we are in the process of comparing. But we still don't know if we are doing something wrong.

mauryhill

christopher.dugan1

May 20, 2021

Hello Maury,

Per my colleague, there are UTF-8 issues when using websockets to retrieve data. I presume this applies to all data, not just TS1. Unfortunately, I don't have any ticket numbers available describing the issue of websockets handling UTF-8.

LPC comes with the testclient application for sample connections. It has the ability to output the data to a file or to STDOUT. Here is an example command line to use:

#> cd /opt/refinitiv/SOFTWARE/lpc1.2.0/linux7_x86_64/demo

#> export LD_LIBRARY_PATH=`pwd`

#> ./testclient -S ELEKTRON_DD -ct rssl -p 14002 -il dIBMd -o 512 -I 1 -O -u yourAppUserName -l stdout -dfile ../../etc/RDMFieldDictionary

The "stdout" option is using a lower case 'L'. The dictionary file is needed to decode the FIDs.

Thanks. I know it works through the LPC and am aware of the dictionary. I think the problem is as you have indicated, the handling of the data using websockets.

Is there somewhere I can raise a ticket?