Discover Refinitiv
MyRefinitiv Refinitiv Perspectives Careers
Created with Sketch.
All APIs Questions & Answers  Register |  Login
Ask a question
  • Questions
  • Tags
  • Badges
  • Unanswered
Search:
  • Home /
  • TREP APIs /
  • RFA /
avatar image
Question by jan.krivanek · May 17, 2016 at 12:51 PM · rfalogging

Serialize/Deserialize RespMsgs in RFA.NET

Is there a way how to (de)serialize entire RespMsg in RFA.NET so that those can be efficiently (means without generating GC garbage and without unnecessary processing) stored and then decoded later?

The use case behind this question is logging. We need to log each incoming message (along with our receiving and processing time stamps). Logs are not expected to be human readable. But they are expected to contain all needed information for investigations - means we need to be able to retrieve same info that the application did.

Manual transforming of the message to string (similar to example app) is a very inefficient way how to achieve this.

One solution to this seems to be by using

\Connections\<connectionName>\traceMsgToFile = true

This however seems to log messages in very verbose format, plus we are unable to add our application timestamps

Steven McCoy

People who like this

1 Show 0
Comment
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

2 Replies

  • Sort: 
avatar image
REFINITIV
Best Answer
Answer by Steven McCoy · May 17, 2016 at 02:50 PM

For incoming data you can grab the encoded buffer backing the RespMsg with getEncodedBuffer() then insert that into a binary log, e.g. GZIP-enabled Protobufs works well at 100k msg/s+ rates.

This buffer then would be independent of transforms and dictionary issues and can be re-decoded post-facto to replay any issues.

RFA/C++ example with SSL/Marketfeed into a Protobuf coded stream.

Comment

People who like this

0 Show 1 · Share
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

avatar image
jan.krivanek · May 18, 2016 at 09:01 AM 0
Share

Great answer!

Before marking this as answer - do you have any suggestion how to get content of the ThomsonReuters.RFA.Common.Buffer without incurring memory allocation?

I'm missing some some 'CopyTo(byte[] preallocatedArray)' method.

The only workarounds I could think off involves accessing the private _Buffer member with reflection or pointers casting and then accessing it in order to manually copy to preallocated buffer.

avatar image
Answer by jan.krivanek · May 18, 2016 at 09:53 AM

I'm posting as separate answer since comment legth restriction won't let me post the code sample. The main credit still goes to Steven McCoy.

Here is the hack code that I stitched together to copy the RFA Buffer without memory allocations. It relies on knowledge of internals of the Buffer implementation, however until there is some 'CopyTo' method, I guess that that's the only option.

public static class RfaBufferUtils
{
	public static unsafe void CopyRfaBufferToBuffer(ThomsonReuters.RFA.Common.Buffer rfaBuffer,
		byte[] destination, out int length)
	{
		length = 0;




		if(destination.Length < rfaBuffer.Size)
			throw new ArgumentException("Insufficient buffer");




		FieldInfo fi = typeof (ThomsonReuters.RFA.Common.Buffer).GetField("_Buffer",
			BindingFlags.NonPublic | BindingFlags.Instance);
		var bufferPtr = fi.GetValue(rfaBuffer);




		//rfa.common.Buffer* bufferPtr
		long pointrValue = *(long*) IntPtr.Add(((IntPtr) Pointer.Unbox(bufferPtr)), 8);
		if (pointrValue == 0L)
		{
			destination[0] = 0;
			return;
		}
		length = *(int*) IntPtr.Add(((IntPtr) Pointer.Unbox(bufferPtr)), 20);
		Marshal.Copy(new IntPtr((void*) pointrValue), destination, 0, length);
	}
}

This is how it can be called:

public void ProcessEvent(Event evt)
{
	RespMsg respMsg = (RespMsg)((OMMItemEvent)evt).Msg;
	long before = GC.GetTotalMemory(false);
	long length;
	RfaBufferUtils.CopyRfaBufferToBuffer(respMsg.EncodedBuffer, _preallocatedByteArray, out length);
	long after = GC.GetTotalMemory(false);
	Debug.Assert(after - before == 0);
	
	// Store the _preallocatedByteArray content as needed
	
	//...
}
Comment

People who like this

0 Show 0 · Share
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Watch this question

Add to watch list
Add to your watch list to receive emailed updates for this question. Too many emails? Change your settings >
5 People are following this question.

Related Questions

RFA verbose logging on SSLED connections

Thomson Reuters RFA EventQueue - Log Queue depth - How to log the current queue depth periodically

RFA log file rotation

Windows Event Logging Question (how to disable)

Activate full message logs for RSSL connections in RFA C++

  • Feedback
  • Copyright
  • Cookie Policy
  • Privacy Statement
  • Terms of Use
  • Careers
  • Anonymous
  • Sign in
  • Create
  • Ask a question
  • Spaces
  • Alpha
  • App Studio
  • Block Chain
  • Bot Platform
  • Calais
  • Connected Risk APIs
  • DSS
  • Data Fusion
  • Data Model Discovery
  • Datastream
  • Eikon COM
  • Eikon Data APIs
  • Elektron
    • EMA
    • ETA
    • WebSocket API
  • Legal One
  • Messenger Bot
  • Messenger Side by Side
  • ONESOURCE
    • Indirect Tax
  • Open PermID
    • Entity Search
  • Org ID
  • PAM
    • PAM - Logging
  • ProView
  • ProView Internal
  • Product Insight
  • Project Tracking
  • Refinitiv Data Platform
    • Refinitiv Data Platform Libraries
  • Rose's Space
  • Screening
    • Qual-ID API
    • Screening Deployed
    • Screening Online
    • World-Check One
    • World-Check One Zero Footprint
  • Side by Side Integration API
  • TR Knowledge Graph
  • TREP APIs
    • CAT
    • DACS Station
    • Open DACS
    • RFA
    • UPA
  • TREP Infrastructure
  • TRIT
  • TRKD
  • TRTH
  • Thomson One Smart
  • Transactions
    • REDI API
  • Velocity Analytics
  • Wealth Management Web Services
  • World-Check Data File
  • Explore
  • Tags
  • Questions
  • Badges