How to recover from encodeSummaryDataComplete failure?

If when encoding a map or a vector you call encodeSummaryDataComplete(encodeIterator,true) and you receive a BUFFER_TOO_SMALL the encodeIterator becomes 'broken' and you cannot complete the encoding.

Our application expects to be able to encode a message in a single pass. In order to avoid unnecessary garbage collection we allocate a fixed number of buffers at application start up. If a message cannot fit within a buffer we expect to get a CodecReturnCode.BUFFER_TOO_SMALL, when we get that return code we start calling encodeComplete(encodeIterator,true) on all the nested containers, we remember where we are in the published data so we can start a continuation message re encode the nested containers and continue processing the data. Also If we get BUFFER_TOO_SMALL on creating a container we can roll it back with encodeComplete(encodeIterator,false).

BUT we can also get BUFFER_TOO_SMALL when we call encodeSummaryDataComplete(encodeIterator,true). After we receive this result code the encodeIterator seems to be 'broken', that is we cant callEncodeComplete to produce a message, The summaryData container has been encodeCompleted(true) so cant be undone either.

If you are encoding a Map/Vector with Map/Vector entries it gets more serious because failing to complete a summarydata on one of the Entry payloads causes the whole message to fail, including all the previous entriess, which in our case we dont have any more.

Find more posts tagged with

elektron

encoding

elektron-sdk

elektron-transport-api

eta-api

rrt

Accepted answers

Jirapongse

In UPA C, when rsslEncodeMapSummaryDataComplete(encIter, RSSL_TRUE) returns RSSL_RET_BUFFER_TOO_SMALL, the application should calls rsslEncodeMapComplete(encIter, RSSL_FALSE) in order to roll back to the last previously successful encoded point in the contents.

if ((retVal = rsslEncodeMapSummaryDataComplete(encIter, RSSL_TRUE)) < RSSL_RET_SUCCESS)
{
printf("Error %s (%d) encountered with rsslEncodeMapSummaryDataComplete(). Error Text: %s\n",
rsslRetCodeToString(retVal), retVal, rsslRetCodeInfo(retVal));
if (retVal == RSSL_RET_BUFFER_TOO_SMALL)
{
retVal = rsslEncodeMapComplete(encIter, RSSL_FALSE);
…
…

Then, you can use rsslRealignEncodeIteratorBuffer to assign a new larger buffer.

//newBuffer contains a new larger buffer
retVal = rsslRealignEncodeIteratorBuffer(encIter, &newBuffer);

After that, you need to call rsslEncodeMapInit() to start encoding map and summary data again.

if ((retVal = rsslEncodeMapInit(encIter, &rsslMap, 0, 0 )) < RSSL_RET_SUCCESS)
{
/* error condition - switch our success value to false so we can roll back */
success = RSSL_FALSE;
/* print out message with return value string, value, and text */
printf("Error %s (%d) encountered with rsslEncodeMapInit(). Error Text: %s\n",
rsslRetCodeToString(retVal), retVal, rsslRetCodeInfo(retVal));
}

All comments

kim.valentine

OK make it public, Just to note that we get the same problem with both the C and Java UPA encoders

Jirapongse

The question has been escalated to TRDC.

Jirapongse

if ((retVal = rsslEncodeMapSummaryDataComplete(encIter, RSSL_TRUE)) < RSSL_RET_SUCCESS)
{
printf("Error %s (%d) encountered with rsslEncodeMapSummaryDataComplete(). Error Text: %s\n",
rsslRetCodeToString(retVal), retVal, rsslRetCodeInfo(retVal));
if (retVal == RSSL_RET_BUFFER_TOO_SMALL)
{
retVal = rsslEncodeMapComplete(encIter, RSSL_FALSE);
…
…

Then, you can use rsslRealignEncodeIteratorBuffer to assign a new larger buffer.

//newBuffer contains a new larger buffer
retVal = rsslRealignEncodeIteratorBuffer(encIter, &newBuffer);

After that, you need to call rsslEncodeMapInit() to start encoding map and summary data again.

if ((retVal = rsslEncodeMapInit(encIter, &rsslMap, 0, 0 )) < RSSL_RET_SUCCESS)
{
/* error condition - switch our success value to false so we can roll back */
success = RSSL_FALSE;
/* print out message with return value string, value, and text */
printf("Error %s (%d) encountered with rsslEncodeMapInit(). Error Text: %s\n",
rsslRetCodeToString(retVal), retVal, rsslRetCodeInfo(retVal));
}

kim.valentine

@Jirapongse

In java we allocate all the buffers up front to avoid garbage collection so it is not really possible to meaningfully allocate a larger buffer.

Also our contract with the publishing application is that we only have one pass through the data.

After we have successfully encoded something we always expect to be able to perform an encodeComplete(true) on all the enclosing containers. When EncodeInit or Encode fails we

call encodeComplete(false) for that object, but then call encodeComplete(true) on all enclosing containers. We can then publish the buffer and then encode the containers and continue for the next buffer. EncodeComplete(true) should not really make the message longer because it is just populating the length of the encoded object in the message. Since we have to set the hasSummaryData flag in the Map/Vector before calling encodeInit the encoder has enough information to reserve the space up front.

Jirapongse

@kim.valentine

From my test, Map.encodeSummaryDataComplete(true) requires few bytes to complete. For example, before calling Map.encodeSummaryDataComplete(true), buffer position in EncodeIterator is 143. After calling Map.encodeSummaryDataComplete(true), buffer position is moved to 146.

For this reason, Map.encodeSummaryDataComplete(true) can return BUFFER_TOO_SMALL.

Dhanakrit.Kim.Suanmalai

I would like to add the answer for UPA Java for this question.

When you receive the return
code -21 (Buffer Too Small) after executing encodeSummaryDataComplete(iterator, true), you should

Roll back the
iterator to the last complete encoded position by calling encodeComplete(iterator,
false)
Realign the new
larger buffer to the iterator by calling realignBuffer function.
It is mandatory to
reinitialize the iterator again. With that, the iterator should be now ready
and can perform encoding from the last completed point.

Please
find the snippet below.

/* complete encoding of summary data.
If any field list encoding failed, success is false */\

if ((retVal = vector.encodeSummaryDataComplete(encIter, success)) < CodecReturnCodes.SUCCESS) 
{
	/* error condition - switch our success value to false so we can roll back */
	success = false; 
	System.out.printf("Error %s (%d)
	encountered with EncodeVectorSummaryDataComplete.  Error Text: %s\n", CodecReturnCodes.toString(retVal), retVal, CodecReturnCodes.info(retVal)); 

	System.out.printf("Rolling back the summary data.....\n");
	if ((retVal = vector. encodeComplete(encIter, success)) < CodecReturnCodes.SUCCESS) // Rollback with the success = false  
	{ 
		System.out.printf("Error %s (%d) encountered with EncodeVectorSummaryDataComplete.  Error Text: %s\n", CodecReturnCodes.toString(retVal), retVal, CodecReturnCodes.info(retVal)); 
	} else 
	{
		// Realign buffer 
		// Creating a new larger buffer 
		Buffer newBuff = CodecFactory.createBuffer(); 
		newBuff.data(ByteBuffer.allocate(1000)); 
  		if ((retVal = encIter.realignBuffer(newBuff)) < CodecReturnCodes.SUCCESS) 
  		{
			System.out.printf("Error %s (%d) encountered with realignBuffer.  Error Text: %s\n", CodecReturnCodes.toString(retVal), retVal, CodecReturnCodes.info(retVal));  
		} else 
  		{  
			System.out.printf("SUCCESS: realignBuffer");
			// From here the iterator is now larger with the previous data being copied as well. 
			// You can continue your operations from here. 
  		} 
  	}

With this approach, you will need to re-encode the entire data over again since it rolls back to the very beginning of the vector you have initialized. That means the raw data should be reusable at least until the entire vector data will be encoded completely.

EXPLORE OUR SITES