High CPU usage with sink_driven_src.

nikita.ou · October 2018

High CPU usage with sink_driven_src. I use sink_driven_src to pllayback a data file (200M). The CPU reaches 100% when i run the command. Would you please advise how I can resolve this?

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

44294 radmin 20 0 2504m 2.4g 3956 R 100.0 3.9 6:01.37 sink_driven_src

zoya faberov · October 2018

Hello @nikita.ou ,

This information changes the situation. I understand now that you are able to record and replay ACTIV service, but over 20M file size you run into a potential capacity issue.

You have an option to record a smaller XML file and work with it.

You can record and replay the file as binary, not XML.

(Infra tools do not work with zipped files)

Replay Service can potentially handle higher performance use cases, then Infra Tools, as it uses a custom, highly efficient proprietary storage.

Hope one of these works for you. In an unfortunate case if it does not, troubleshooting a potential capacity issue in a client's environment is beyond API forum intention, and is better handled via Get Support Refinitiv channel.

zoya faberov · October 2018

It is very feasible to achieve 100% CPU utilization with sink_driven_src tool.

The tool was originally built for market data engineers, rather then developers, and stressing the system to capacity was one of the intended uses.

I would experiment with start_rate and max_rate to allow to better understand what update rates you are comfortable with for your organization's application testing requirements.

Another replay option is ReplayService for TREP from Tradeweb. It will allow you to replay the data at the original speed, i.e. intervals between the updates will be preserved. Or times the original speed, i.e. speed-up to requirement.

nikita.ou · October 2018

https://community.developers.refinitiv.com/discussion/comment/33085#Comment_33085

Hi Zoya, sorry that I didn't make my question clear.

I use "./sink_driven_src -S ACTIV -C 100000 -E rssl -N 14003 -U 1 -K -Q /reuters/ads/demo/active.out" to run the data replay. After I run the command, it shows "Initialized RSSL Library" and the CPU utilization jumps to 100% after a few seconds.

It seems the process hangs in the initialize status and the data replay doesn't get started. Any suggestion on how to resolve it? thank you

zoya faberov · October 2018

https://community.developers.refinitiv.com/discussion/comment/33085#Comment_33085

Hi @nikita.ou,

Well, the next statement after "Initialized" should have been "Loaded ...".

Does not look like the tool is loading your XML. Your service, ACTIV, is a custom publisher. Depending on how conformant to standards the publisher is, there may be issues with replaying a custom publisher data.

I am attaching a small elektron capture file, zipped, try to play that, service is ELEKTRON_EDGE

curr-capture.zip

If this file appears to replay, test consuming with rmdstestclient, instrument "JPY="

nikita.ou · October 2018

https://community.developers.refinitiv.com/discussion/comment/33141#Comment_33141

Thanks Zoya, I am able the reply the same custom feed with a 20M data file. But when the data file gets bigger, i.e 200M, the tool doesn't load the XML. is it a resource issue?

is there a way that I can record the data, zip it and then play the zipped file? Would you please advise the command?

nikita.ou · October 2018

https://community.developers.refinitiv.com/discussion/comment/33143#Comment_33143

Thanks Zoya, are the below command the right one that I should use?

rmdstestclient –S serviceName -f ItemfileName -h serverName -p 14002 –ct rssl -u username –rf 8 –I 1 –m –obf binary_data_file

sink_driven_src –S serviceName -c -I 1 -K -N 14002 -tps 1000 -mutp 0 -sndbuf 2048 -ik -bdf binary_data_file

zoya faberov · October 2018

https://community.developers.refinitiv.com/discussion/comment/33143#Comment_33143

@nikita.ou,

Does not seem to record for me. Does it record for you?

Yes, it should work.

However, I have been using Infra tools XML recordings for demo captures and Replay Service for almost all other, i.e. stress testing, error condition testing, I have not used infra tools in binary mode for a very long time now. There may have been changes I am not aware of.

I will try to find out more from our engineers.