Many times, in a single fragment message, the expected size (fid="32480" name="TOT_SIZE") has smaller value than buffer size (fid="32641" name="FRAGMENT"), is it normal?
where fieldEntry is a FieldEntry object with fid 32641.
ntaBuffer = fieldEntry.buffer().bufer().array();
example
Expected total buffer size: 1457 current size: 1944
Hello @jim.yu
Thank you for your participation in the forum.
Is the reply below satisfactory in resolving your query? If yes please click the 'Accept' text next to the reply. This will guide all community members who have a similar question. Otherwise please post again offering further insight into your question.
Thanks,
AHS
Hello @jim.yu
Thank you for your participation in the forum.
Is the reply below satisfactory in resolving your query? If yes please click the 'Accept' text next to the reply. This will guide all community members who have a similar question. Otherwise please post again offering further insight into your question.
Thanks,
AHS
Please be informed that a reply has been verified as correct in answering the question, and has been marked as such. Thanks, AHS
Please be updated:
The API development team found the fix for this issue. They suggest you use the fieldEntry.buffer().buffer().limit() instead of fieldEntry.buffer().buffer().array().length to get a proper size of full used OMMBuffer amount.
Example:
From:
if((fieldEntry.buffer().buffer().array().length
> totalSize) && (totalSize > 0)){
To:
if((fieldEntry.buffer().buffer().limit() >
totalSize) && (totalSize > 0)){
The development team confirms this is API usage issues in the example application. In between 1.0.2 and 1.0.5, EMA was enhanced to internally pool created objects to avoid garbage collection. The current EMA implementation is reusing pooled OmmBuffer objects. The OmmBuffer.buffer().array().length returns the full capacity length instead of the full used amount while the OmmBuffer.buffer().limit() returns the used size value.
In addition, when reassembling the fragments back, please also be mindful of the limit, and use:
...
ByteArrayOutputStream fragOutputStream = new
ByteArrayOutputStream( );
for(int i = 0; i < alFrags.size(); i++) {
try {
fragOutputStream.write( alFrags.get(i).array(),0, alFrags.get(i).limit());
} catch(Exception e) {}
}
strFlatFrag+= unzipPayload(fragOutputStream.toByteArray());
...
For an updated complete example please see Streaming Console Example
This manifestation is most likely a result of an issue with reassembly of the fragments.
Please verify that the fragments reassembled are unique, FRAG_NUM is increased by one, each fragment is only appended once?
GUID is only unique per RIC.
Hope this helps, if not, could you please provide a log of the concatenated fragments?
This is ONE FRAGMENT message. I have posted one example below.
You can run your tutorial sample 6, where buffer is getting from ntaBuffer = fieldEntry.buffer().bufer().array();
expectedSize = 822 buffer size = 870
UpdateMsg
streamId="5"
domain="NewsTextAnalytics Domain"
updateTypeNum="0"
permissionData="03 01 01 10 00 2a 10 01 0a 10 02 0b 10 02 1a 10 04 0b 10 04 2b 10 04 3b 10 04 6a 10 10 1a 10 11 9b 10 12 1a 10 23 0b 10 26 3b 10 28 3b 10 30 3b 10 30 7c"
name="MRN_TRNA"
serviceId="257"
serviceName="ELEKTRON_DD"
Payload dataType="FieldList"
FieldList
FieldEntry fid="4148" name="TIMACT_MS" dataType="UInt" value="76061644"
FieldEntry fid="17" name="ACTIV_DATE" dataType="Date" value="27 JUN 2017 "
FieldEntry fid="8593" name="MRN_TYPE" dataType="Rmtes" value="TRNA"
FieldEntry fid="8506" name="MRN_V_MAJ" dataType="Rmtes" value="2"
FieldEntry fid="11787" name="MRN_V_MIN" dataType="Rmtes" value="10"
FieldEntry fid="32480" name="TOT_SIZE" dataType="UInt" value="822"
FieldEntry fid="32479" name="FRAG_NUM" dataType="UInt" value="1"
FieldEntry fid="4271" name="GUID" dataType="Rmtes" value="tr:FWN1JN0B1_1706271rEtWPnZDSSd8K7YKiyXpD0902PY2vhdeC7yfV"
FieldEntry fid="12215" name="MRN_SRC" dataType="Rmtes" value="C291MKSA1AUK1-4.0.6_TRNA-C291MKSA1AUK1"
FieldEntry fid="32641" name="FRAGMENT" dataType="Buffer"
1F8B 0800 0000 0000 00FF CD55 5D6F AB46 ...........U]o.F10FD 2B68 D547 2762 B9B6 B1B7 5525 07E3 ..+h.G'b....U%..64EF 3504 B138 6E52 55D5 0636 CE36 18DC d.5..8nRU..6.6..059C D2AB FCF7 CE00 C98D 1F6E 2AF5 A93C ...........n*..<30C3 9CD9 8F39 BB67 F84A 6421 F3B6 D669 0....9.g.Jd!...i45D8 D76F 1F22 2D8D 82D0 AF10 AB2A 557B E..o."-......*U{3918 C288 1744 E12D 190D C132 EB72 48C4 9....D.-...2.rH.C6CE 7C32 A774 EE52 0063 16F2 F390 FC36 ..|2.t.R.c.....6A4F1 0C06 9E24 74E1 50EE 1500 A116 6563 .....$t.P.....ec5265 F122 05E8 DE94 4FCA 2CD2 5A97 05A0 Re."....O.,.Z...3CE4 095F 243E 200F DA54 75A0 0A44 0418 <.._$> ..Tu..D..55A4 309E 8E48 AE8B 2795 F10C B702 6B16 U.0..H..'.....k.E551 E575 EB95 4D51 F715 E85A EDBB 4FC2 .Q.u..MQ...Z..O.EC11 79D6 4556 3EC3 DCD4 B922 2FA3 EFC2 ..y.EV>...."/...CEF8 43F8 D3F2 2374 F221 EA02 0A3B 3D18 ..C...#t.!...;=.9DAA 449A 1D70 5464 3A95 7569 4E6B 362A ..D..pTd:.uiNk6*5747 D917 7A6E E333 2215 52B0 87D7 7026 WG..zn.3".R...p&67F4 5D2C 543B 59EB 23E4 DBE7 333A B3A9 g.],T;Y.#...3:..7B02 36B5 9139 62D4 99CC E6CE 3B2C 2A2B {.6..9b.....;,*+FD3A D09E 4CED F9C9 C86D 69B2 A10E 3A1D .:..L....mi...:.9163 9937 7BF5 BF67 1839 2ED4 73C5 2107 .c.7{..g.9..s.!.6FF7 7D99 B542 FFAD BAD4 B4DC 1F64 D1BE o.}..B.......d..5635 22EA AFF4 5116 3BF5 76F7 36E1 D25F V5"...Q.;.v.6.._F1D0 5FC2 413C 2A99 C13D 83D3 DA01 0281 .._.A<*..=......BD34 4FA8 803D D223 4D4B D883 CC2B D573 .4O..=.#MK...+.s8657 7398 1838 7E7E 4FDE 0B64 B415 6CE8 .Ws..8~~O..d..l.4699 AA5F 2711 6C6C DB53 DB9E 7D82 9288 F.._'.ll.S..}...46B9 D486 ADB6 21FD 1CDA 17F4 77EA DA53 F.....!.....w..SC7A5 C6AF B751 71B7 1422 9B7D 716F BFE8 .....Qq..".}qo..F697 C3D2 9EDB 4E74 EB1C 1F33 E5B9 EDC3 ......Nt...3....0D39 2D39 93B5 4CDA 4327 3408 A38C 94CA .9-9..L.C'4.....5672 AFF3 16E5 DB2D F5AE 3ECC E3E2 7A13 Vr.....-..>...z.7BBE C543 CFFA 0995 FC33 B3BC D85F F2C4 {..C.....3..._..121B 2E84 6F2D 84D8 04BE B0BC EB1B 3F5E ....o-........?^5CFA D696 2757 16F2 1547 7EBC BA8E 032B \...'W...G~....+5E24 3CBC FCD1 8A62 0E53 258B F8D2 4FAC ^$<....b.S%...O.1F9C 493F 21AC 9803 D78D DCE1 8AAA 4042 ..I?!.........@B552D 71B7 5D1B CAFB A651 BC71 804D A3C9 U-q.]....Q.q.M..3412 DB75 9C30 62A8 0F34 EBB0 7756 C1B6 4..u.0b..4..wV..77AE 7B13 C5EB DED9 6027 C2B2 13B8 CB55 w.{.....`'.....U2DF7 07BC 7B20 8D33 7B7A E6B8 8943 99ED -...{ .3{z...C..B231 3D1F D3F9 DD6B A3F1 8C92 B5CA BE93 .1=....k........0942 C44C 5D2D 4CFA D8A9 6638 FD5A 3E29 .B.L]-L...f8.Z>)A1FE 6C86 E6F4 8222 2F8F 3A53 28EC 50B0 ..l...."/.:S(.P.3889 050C ACBA 96D7 D5F8 9F8F B99F E3DF 8...............6A1A 765A 35F7 7FA8 B453 2BB9 6074 8684 j.vZ5....S+.`t..A29D 83F5 1912 75C9 C6DD 7BFA B933 0804 ......u...{..3..ACEB F501 BBFB D6D4 8151 872D 02BF 2B03 .........Q.-..+.DC8B 75DC 3BDE 9A8B 840E FEF0 9700 37C4 ..u.;.........7.DCDE 8DBD ABDE D90C 6337 095F BF79 0178 ........c7._.y.x27FF 1138 B406 5A63 91B6 48E4 CB3F 729A '..8..Zc..H..?r.16B3 AA06 0000 ......
FieldListEnd
PayloadEnd
UpdateMsgEnd
Hello @jim.yu ,
So far I am not seeing this behavior. I am just watching MRN_TRNA stream.
Would be helpful to know if you have seen it once,
or more then once?
Thanks,
-Zoya
Hello @jim.yu,
I see the issue.
I was not using Tutorial 6, as am accustomed to testing with another MRN tester example, to see the data.
I will look into fixing the tutorial,
In the meantime, to avoid the issue now, please download
NewsAnalytics Streaming Example (MRN)
Thanks for reporting this
-AHS
I am not so familiar with your tutorial codes so can you please just tell me what's the change. And just want to confirm with you that ema and eta jar files are good.
Hello @jim.yu,
The jars with the latest download of SDK, EMA and ETA, both are good to use.
From the description of the behavior, the fragment is not processed right, the new message arrives before the previous one is fully processed, and you get the behavior that you observe, the incorrect size.
Please copy from either this simple example MRN Streaming Console example
Or the updated EMA MRN Tutorial to do it correctly.
@zoya.farberov can you tell me what is changed and why it is not the bug on the library? Or are you saying buffer size is bigger than tot_size in message is normal case.
To replicate the issue is easy
1. get total size via field id 32480
2. get buffer via field id 32641
3. compare total size with buffer size.
There is NO manipulation on data at all, how could this be a problem on the tutorial code? This MUST be a bug on the library.
I just don't know who is reviewing the answer. This IS a bug on library and IS confirmed by Zoya. This developer community is good to discuss but not a good way to ask tech support.
quoted:
Opened SR with library dev,
Your service request number is : 05682050
Dear all,
There is a confirmed issue with buffer size in MRN stream reported incorrectly, the issue started with version 1.0.5, the product development team is aware of the issue and it is being fixed.
We will update this post when we have more information.
We apologize for any confusion on this,
-AHS
It does not solve the issue at all.
onUpdateMsg
update msg data type 132
Decode fieldList:
FIELD_ENTRY: 4148/TIMACT_MS: 75720467 loadType: 4
FIELD_ENTRY: 17/ACTIV_DATE: 9 OCT 2017 loadType: 9
FIELD_ENTRY: 8593/MRN_TYPE: TRNA loadType: 19
FIELD_ENTRY: 8506/MRN_V_MAJ: 2 loadType: 19
FIELD_ENTRY: 11787/MRN_V_MIN: 10 loadType: 19
FIELD_ENTRY: 32480/TOT_SIZE: 824 loadType: 4
FIELD_ENTRY: 32479/FRAG_NUM: 1 loadType: 4
FIELD_ENTRY: 4271/GUID: tr:BwnP2MMa__1710092C2R7KY8KIGgDqPaYmbsczO9B18G8YuMr0KqwS loadType: 19
FIELD_ENTRY: 12215/MRN_SRC: C383QUJA1ADTC-4.0.6_TRNA-C383QUJA1ADTC loadType: 19
FIELD_ENTRY: 32641/FRAGMENT: 1F8B 0800 0000 0000 00FF CD55 5173 DA38 ...........UQs.810FE 2B1E 3F53 46B2 0D06 BF81 A129 97E2 ..+.?SF......)..3A98 5E26 BDB9 E928 F686 A8B1 A544 92A1 :.^&...(.....D..6E87 FF7E 2BDB 34A4 D3E6 66EE E978 D02E n..~+.4...f..x..FBED 4ADA 4FBB EBEF 2E13 AC6C 0CCF B51B ..J.O......l....7D7F FE93 E552 019A FE42 9BD6 60E2 1285 }....R...B..`...1BB9 F13A 4D6E DC41 6F94 45EB E3A6 51E0 ...:Mn.Ao.E...Q.4D47 5312 8C27 0182 9B28 CDE6 C3C4 FDBB MGS..'...(......F75B 1518 F9C2 A335 27AC 0204 D2CC 99D7 .[.....5'.......9A0B D0DA 4999 7AD0 CE4A E4E8 73AB E403 ....I.z..J..s...A859 6EB8 14E8 F631 592C DFAE 92E5 02A1 .Yn....1Y,......3BAE B459 83B0 5086 0244 8E3B D181 5B72 ;..Y..P..D.;..[rF100 C5AA B0B7 C2D3 85DC 4369 9A58 D6C2 ..........Ci.X..74C9 7003 55FB D78D C8C0 3D70 51C8 036E t.p.U.....=pQ..n4EBD 77EE 71F0 5BD8 0B5E 85FD C56B E8E8 N.w.q.[..^...k..5534 4414 6FFA A878 0E5B A676 C896 2878 U4D.o..x.[.v..(xCE8C 543F 25AD A084 3DEB 321D 12FB 1BB8 ..T?%...=.2.....DA72 50E1 D2BF 0F3D 3325 B063 86EF D19D .rP....=3%.c....0C29 9950 3A7E 01D6 46B1 D262 FEC4 F349 .).P:~..F..b...I7886 A552 F353 E088 4C47 E179 E0B5 5445 x..R.S..LG.y..TE9F86 174E 06EE 5E96 7505 FF7B 862D C702 ...N..^.u..{.-..0E7A 853E B6D0 6F65 D164 FC9B E572 EAA3 .z.>..oe.d...r..772E AB47 269A 3E1C 6984 AFF9 3D13 3BF8 w..G&.>.i...=.;.65FD DD03 2BB0 D4F0 C176 88A0 A1C2 BAB5 e...+....v......FD50 598A 986A DCE8 8E95 1A3A DE6C 759E .PY..j.....:.lu.3646 C60E CF0C FA7E 7044 9F46 E3AD FE04 6F.....~pD.F....A5BB 93B6 5914 1032 2664 1262 5E2E B7DD ....Y..2&d.b^...6354 343F 88D4 5BAF D9E7 CF34 A484 4CBD cT4?..[....4..L.D8DB 8497 3793 CBD5 C56E F194 B29B EA56 ....7....n.....VE7DF 3E4C E774 7231 B9A9 D78A 5C3E 1D32 ..>L.tr1....\>.2F765 DE05 336C DB3C DABE 4BD0 6C7B 09A0 .e..3l.<..K.l{..78CB 2A5E 36B6 9DDB A3CE 32FC 557F 0E6C x.*^6.....2.U..l830E 1D23 9D0D 9624 D3E0 6CEF B92A 9CAB ...#...$..l..*..9A29 03CA F108 0D9D 2553 828B 9D76 9828 .)......%S...v.(9C77 529B 135C 364E 2CC5 1D28 4B8B 13B3 .wR..\6N,..(K...B2C4 E34A A4BA 663B 7B1C 08CB 2718 66AF ...J..f;{...'.f.DACE A4B2 1B20 E244 809D 1F75 C16D 7C3B ..... .D...u.m|;7D92 349A 67D7 6845 254E 3676 ECD8 A4B6 }.4.g.hE%N6v....58AF DAB0 EAD1 9617 5EE9 0D25 6FC8 74EB X.......^..%o.t.D188 7811 2143 2FA4 9F4E B324 56C0 0C14 ..x.!C/..N.$V...BFF3 F489 F5E4 7AA6 F2FB B631 FAD7 35EC ......z....1..5.0132 78AA FBF9 73B4 7D2C F7BC 00DB BB49 .2x...s.},.....ID65F 4BCB 5AE5 D0E6 F09F DFB0 DBE3 5F53 ._K.Z........._SEA2E AAEB DB2F 90B7 FDE8 CE23 EA8D D086 ...../.....#....32A0 BD0C 3A39 0EA7 ADE2 D171 27C7 D661 2...:9.....q'..a1975 EB05 AE17 51D0 AEE3 3F5A 61DD D711 .u....Q...?Za...BD5A B4D2 F75A F1E9 6CE0 E313 78D1 6CBD .Z...Z..l...x.l.DC64 9D3A 9FC5 DB4E 3B7D 3750 C526 CA9E .d.:...N;}7P.&..3532 E9F4 F7CB 3E3C B1F1 BDBA BCEE B5CD 52....><........72F6 FEA4 ADE2 678D FE50 B727 ADDF FBA3 r.....g..P.'....952F BE4B 5816 35CE 5791 6377 FAC7 E33F ./.KX.5.W.cw...?793A BCF2 FA06 0000 y:...... loadType: 16
=>FRAGMENT JSON zipped SIZE: 833
=>Total Size : 824
Hi @jim.yu
I have tested the NewsAnalytics__Streaming_Console_20171009.zip and compared it with the old MRNConsumer.java. The new NewsAnalytics__Streaming_Console_20171009.zip can consume and parse MRN news data successfully while the old MRNConsumer.java encounters error for the same news.
The fixed is in the application source code level, not the API library files. Did you test with the new NewsAnalytics__Streaming_Console_20171009.zip (new MRNConsumer.java version file)?
Result from NewsAnalytics__Streaming_Console_20171009.zip:
onUpdateMsg
Item Name: MRN_TRNA
Service Name: ELEKTRON_DD
Decode fieldList:
FIELD_ENTRY: 4148/TIMACT_MS: 33218671 loadType: 18
FIELD_ENTRY: 17/ACTIV_DATE: 10 OCT 2017 loadType: 22
FIELD_ENTRY: 8593/MRN_TYPE: TRNA loadType: 31
FIELD_ENTRY: 8506/MRN_V_MAJ: 2 loadType: 31
FIELD_ENTRY: 11787/MRN_V_MIN: 10 loadType: 31
FIELD_ENTRY: 32480/TOT_SIZE: 909 loadType: 18
FIELD_ENTRY: 32479/FRAG_NUM: 1 loadType: 18
FIELD_ENTRY: 4271/GUID: tr:H9N1MA04O_1710101W+ptXLPPAic561vneZhdeklf6QaI2fcpMLFMj loadType: 31
FIELD_ENTRY: 12215/MRN_SRC: C383QUJA1ADTC-4.0.6_TRNA-C383QUJA1ADTC loadType: 31
FIELD_ENTRY: 32641/FRAGMENT: 1F8B 0800 0000 0000 00FF ED56 6D6F 9B48 ...........Vmo.H10FE 2B68 3FF4 4BDD 6897 37DB E874 12C1 ..+h?.K.h.7..t..4E8C 6A30 1748 D3E6 74AA 36B0 B1B7 E6C5 N.j0.H..t.6.....85C5 2917 E5BF DF2C E0D8 6E1B 57CA A7FB ..)....,..n.W...501B 31C3 3CB3 2F33 FBCC C023 A239 4D1B ...=>FRAGMENT JSON zipped SIZE: 909 totalSize=909
=>FRAGMENT JSON STRING: {"analytics":{"analyticsScores":[{"assetClass":"CMPNY","assetCodes":["P:5035965288","R:002632.SZ"],"assetId":"5035965288","assetName":"Daoming Optics&Chemical Co Ltd","brokerAction":"UNDEFINED",
Result from the old MRNConsumer.java:
onUpdateMsg Item Name: MRN_TRNA Service Name: ELEKTRON_DD Decode fieldList: FIELD_ENTRY: 4148/TIMACT_MS: 33218671 loadType: 4 FIELD_ENTRY: 17/ACTIV_DATE: 10 OCT 2017 loadType: 9 FIELD_ENTRY: 8593/MRN_TYPE: TRNA loadType: 19 FIELD_ENTRY: 8506/MRN_V_MAJ: 2 loadType: 19 FIELD_ENTRY: 11787/MRN_V_MIN: 10 loadType: 19 FIELD_ENTRY: 32480/TOT_SIZE: 909 loadType: 4 GUID incoming TOT_SIZE is 909 FIELD_ENTRY: 32479/FRAG_NUM: 1 loadType: 4 FIELD_ENTRY: 4271/GUID: tr:H9N1MA04O_1710101W+ptXLPPAic561vneZhdeklf6QaI2fcpMLFMj loadType: 19 FIELD_ENTRY: 12215/MRN_SRC: C383QUJA1ADTC-4.0.6_TRNA-C383QUJA1ADTC loadType: 19 FIELD_ENTRY: 32641/FRAGMENT: 1F8B 0800 0000 0000 00FF ED56 6D6F 9B48 ...........Vmo.H10FE 2B68 3FF4 4BDD 6897 37DB E874 12C1 ..+h?.K.h.7..t..4E8C 6A30 1748 D3E6 74AA 36B0 B1B7 E6C5 N.j0.H..t.6.....85C5 2917 E5BF DF2C E0D8 6E1B 57CA A7FB ..)....,..n.W...501B 31C3 3CB3 2F33 FBCC C023 A239 4D1B ...=>FRAGMENT JSON zipped SIZE: 1027 totalSize=909 <<<<TOTSIZE=909>>>>>
ok, sorry, I forgot to update my classpath, so my test was still on old app jar file. With new app jar file, seems good so far.
