High CPU usage with sink_driven_src.

High CPU usage with sink_driven_src. I use sink_driven_src to pllayback a data file (200M). The CPU reaches 100% when i run the command. Would you please advise how I can resolve this?

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

44294 radmin 20 0 2504m 2.4g 3956 R 100.0 3.9 6:01.37 sink_driven_src

Find more posts tagged with

trep

infrastructure

data

elektron

refinitiv-realtime

Accepted answers

zoya faberov

Hello @nikita.ou ,

This information changes the situation. I understand now that you are able to record and replay ACTIV service, but over 20M file size you run into a potential capacity issue.

You have an option to record a smaller XML file and work with it.

You can record and replay the file as binary, not XML.

(Infra tools do not work with zipped files)

Replay Service can potentially handle higher performance use cases, then Infra Tools, as it uses a custom, highly efficient proprietary storage.

Hope one of these works for you. In an unfortunate case if it does not, troubleshooting a potential capacity issue in a client's environment is beyond API forum intention, and is better handled via Get Support Refinitiv channel.

All comments

zoya faberov

It is very feasible to achieve 100% CPU utilization with sink_driven_src tool.

The tool was originally built for market data engineers, rather then developers, and stressing the system to capacity was one of the intended uses.

I would experiment with start_rate and max_rate to allow to better understand what update rates you are comfortable with for your organization's application testing requirements.

Another replay option is ReplayService for TREP from Tradeweb. It will allow you to replay the data at the original speed, i.e. intervals between the updates will be preserved. Or times the original speed, i.e. speed-up to requirement.

nikita.ou

Hi Zoya, sorry that I didn't make my question clear.

I use "./sink_driven_src -S ACTIV -C 100000 -E rssl -N 14003 -U 1 -K -Q /reuters/ads/demo/active.out" to run the data replay. After I run the command, it shows "Initialized RSSL Library" and the CPU utilization jumps to 100% after a few seconds.

It seems the process hangs in the initialize status and the data replay doesn't get started. Any suggestion on how to resolve it? thank you

zoya faberov

Hi @nikita.ou,

Well, the next statement after "Initialized" should have been "Loaded ...".

Does not look like the tool is loading your XML. Your service, ACTIV, is a custom publisher. Depending on how conformant to standards the publisher is, there may be issues with replaying a custom publisher data.

I am attaching a small elektron capture file, zipped, try to play that, service is ELEKTRON_EDGE

curr-capture.zip

If this file appears to replay, test consuming with rmdstestclient, instrument "JPY="

nikita.ou

Thanks Zoya, I am able the reply the same custom feed with a 20M data file. But when the data file gets bigger, i.e 200M, the tool doesn't load the XML. is it a resource issue?

is there a way that I can record the data, zip it and then play the zipped file? Would you please advise the command?

zoya faberov

Hello @nikita.ou ,

This information changes the situation. I understand now that you are able to record and replay ACTIV service, but over 20M file size you run into a potential capacity issue.

You have an option to record a smaller XML file and work with it.

You can record and replay the file as binary, not XML.

(Infra tools do not work with zipped files)

Replay Service can potentially handle higher performance use cases, then Infra Tools, as it uses a custom, highly efficient proprietary storage.

nikita.ou

Thanks Zoya, are the below command the right one that I should use?

rmdstestclient –S serviceName -f ItemfileName -h serverName -p 14002 –ct rssl -u username –rf 8 –I 1 –m –obf binary_data_file

sink_driven_src –S serviceName -c -I 1 -K -N 14002 -tps 1000 -mutp 0 -sndbuf 2048 -ik -bdf binary_data_file

zoya faberov

@nikita.ou,

Does not seem to record for me. Does it record for you?

Yes, it should work.

However, I have been using Infra tools XML recordings for demo captures and Replay Service for almost all other, i.e. stress testing, error condition testing, I have not used infra tools in binary mode for a very long time now. There may have been changes I am not aware of.

I will try to find out more from our engineers.

zoya faberov

Hello @nikita.ou ,

Rmdstestclient from Infra Tools 0.0.6, windows, does not seem to capture binary data.

I have tested the same command using rmdstestclient included with infra (ADS) on Linux, and was able to capture binary data with no issues.

I have opened product support ticket #06997310 to investigate this.

Are you able to capture on Linux or ask your market data team to help you do it?

nikita.ou

Thanks Zoya, it is ok. I can do the data recording on Linux.

zoya faberov

Hi @nikita.ou

The issue manifested was local to my test machine. There is not issue recording binary files on windows in general.

EXPLORE OUR SITES