question

Upvote
Accepted
jan.krivanek avatar image
20 ●1 ●0 ●5

Serialize/Deserialize RespMsgs in RFA.NET

Is there a way how to (de)serialize entire RespMsg in RFA.NET so that those can be efficiently (means without generating GC garbage and without unnecessary processing) stored and then decoded later?

The use case behind this question is logging. We need to log each incoming message (along with our receiving and processing time stamps). Logs are not expected to be human readable. But they are expected to contain all needed information for investigations - means we need to be able to retrieve same info that the application did.

Manual transforming of the message to string (similar to example app) is a very inefficient way how to achieve this.

One solution to this seems to be by using

\Connections\<connectionName>\traceMsgToFile = true

This however seems to log messages in very verbose format, plus we are unable to add our application timestamps

treprfarfa-apilogging
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

2 Answers

Upvotes
Accepted
Steven McCoy avatar image
1.2k ●23 ●28 ●42

For incoming data you can grab the encoded buffer backing the RespMsg with getEncodedBuffer() then insert that into a binary log, e.g. GZIP-enabled Protobufs works well at 100k msg/s+ rates.

This buffer then would be independent of transforms and dictionary issues and can be re-decoded post-facto to replay any issues.

RFA/C++ example with SSL/Marketfeed into a Protobuf coded stream.

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

jan.krivanek avatar image jan.krivanek

Great answer!

Before marking this as answer - do you have any suggestion how to get content of the ThomsonReuters.RFA.Common.Buffer without incurring memory allocation?

I'm missing some some 'CopyTo(byte[] preallocatedArray)' method.

The only workarounds I could think off involves accessing the private _Buffer member with reflection or pointers casting and then accessing it in order to manually copy to preallocated buffer.

Upvotes
jan.krivanek avatar image
20 ●1 ●0 ●5

I'm posting as separate answer since comment legth restriction won't let me post the code sample. The main credit still goes to Steven McCoy.

Here is the hack code that I stitched together to copy the RFA Buffer without memory allocations. It relies on knowledge of internals of the Buffer implementation, however until there is some 'CopyTo' method, I guess that that's the only option.

public static class RfaBufferUtils
{
	public static unsafe void CopyRfaBufferToBuffer(ThomsonReuters.RFA.Common.Buffer rfaBuffer,
		byte[] destination, out int length)
	{
		length = 0;




		if(destination.Length < rfaBuffer.Size)
			throw new ArgumentException("Insufficient buffer");




		FieldInfo fi = typeof (ThomsonReuters.RFA.Common.Buffer).GetField("_Buffer",
			BindingFlags.NonPublic | BindingFlags.Instance);
		var bufferPtr = fi.GetValue(rfaBuffer);




		//rfa.common.Buffer* bufferPtr
		long pointrValue = *(long*) IntPtr.Add(((IntPtr) Pointer.Unbox(bufferPtr)), 8);
		if (pointrValue == 0L)
		{
			destination[0] = 0;
			return;
		}
		length = *(int*) IntPtr.Add(((IntPtr) Pointer.Unbox(bufferPtr)), 20);
		Marshal.Copy(new IntPtr((void*) pointrValue), destination, 0, length);
	}
}

This is how it can be called:

public void ProcessEvent(Event evt)
{
	RespMsg respMsg = (RespMsg)((OMMItemEvent)evt).Msg;
	long before = GC.GetTotalMemory(false);
	long length;
	RfaBufferUtils.CopyRfaBufferToBuffer(respMsg.EncodedBuffer, _preallocatedByteArray, out length);
	long after = GC.GetTotalMemory(false);
	Debug.Assert(after - before == 0);
	
	// Store the _preallocatedByteArray content as needed
	
	//...
}
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.