question

Upvotes
Accepted
chinelo.okoli avatar image
1 ●1 ●1 ●3

Inconsistent result from DSS TRTH v2

Hi there,@manigandan.r

When i request the extraction result from a Time and Sales report schedule - i get different results using exactly the same request.

I see the file size is exactly the same but my call to response.getEntity().getContent() appears to return inconsistent results. E.g. the last 2 calls i made to the API, T&S returned 69937 vs 101031 lines.

This is how i set up my client:

private CloseableHttpClient httpclient = HttpClientBuilder.create().build();

.....

String urlGet = this.url + "/Extractions/ReportExtractions('"+reportExtractionId+"')/Files";

HttpGet request = new HttpGet(urlGet);

request.addHeader("Authorization", "Token "+this.sessionToken);

HttpResponse response = this.httpclient.execute(request);

BufferedReader rd = new BufferedReader(new InputStreamReader(response.getEntity().getContent()));

StringBuffer result = new StringBuffer();

String line = "";

while ((line = rd.readLine()) != null) { result.append(line); }

JSONObject jsonGetResponse = new JSONObject(result.toString());

JSONArray valueJArray = jsonGetResponse.getJSONArray("value");

String ExtractedFileId = ""; String ExtractedFileName = ""; String FileType = ""; boolean success = false;

for (int i = 0; i < valueJArray.length(); i++)

{

ExtractedFileId = valueJArray.getJSONObject(i).getString("ExtractedFileId");

......

//FileType="Full"

urlGet = this.url + "/Extractions/ExtractedFiles('"+ExtractedFileId+"')/$value";

request = new HttpGet(urlGet);

request.addHeader("Authorization", "Token "+this.sessionToken);

response = this.httpclient.execute(request);

HttpEntity entity = response.getEntity();

rd = new BufferedReader(new InputStreamReader(entity.getContent()));

......

while ((line = rd.readLine()) != null) {...}

......

}

The same code works fine for Depth extraction which also has a gzipped file output.

Am i missing something here please?

Thanks

tick-history-rest-api
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Nipat Kunvutipongsak avatar image   Nipat Kunvutipongsak

Hello @chinelo.okoli,

Thank you for your participation in the forum. Is the reply below satisfactory in resolving your query?

If yes please click the 'Accept' text next to the reply. This will guide all community members who have a similar question. Otherwise please post again offering further insight into your question.

Thanks,

AHS

Alex Putkov.1 avatar image   Alex Putkov.1

@chinelo.okoli
Please be informed that a reply has been verified as correct in answering the question, and has been marked as such.
Thanks,
AHS

3 Answers

· Write an Answer
Upvotes
Accepted
Christiaan Meihsl avatar image
13.7k ●26 ●8 ●12

@chinelo.okoli

Sorry this took some time, but it was a complex nut to crack. The support team found a solution, kudos go to them.

The key problem is the merged GZip files cause issues for the default decompression methods. For Java, the Apache Commons Compress https://commons.apache.org/proper/commons-compress/index.html library seems like a safe option. However, it needs to be implemented correctly.

GZIPStream from the java.util.zip package will work if it is done on the local machine, but may fail if used while reading directly from the HTTP Stream.

Key factors for Apache Commons Compress implementation:

Disable Content Compression on the HTTP client

private CloseableHttpClienthttpclient = HttpClients.create().disableContentCompression().build();

Set decompressConcatenated = true when initializing the stream

GzipCompressorInputStream gis =new GzipCompressorInputStream(
myURLConnection.getInputStream(), true);

If these are set, you can use In-Stream Decompression and process the stream while downloading the file.

In a nutshell, here are the code details:

    Add the Apache Commons Compress library:

    import
org.apache.commons.compress.compressors.gzip.GzipCompressorInputStream;

    Use this setting when declaring the httpclient:

    private CloseableHttpClient httpclient = HttpClientBuilder.create().disableContentCompression().build();

    Full code of a method to extract and display on the fly is attached.solutioncode.txt.

    I will update all our Java samples, in the next version they will use this method.


    solutioncode.txt (1.6 KiB)
    icon clock
    10 |1500

    Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

    Christiaan Meihsl avatar image Christiaan Meihsl

    All the TRTH REST API Java samples have been checked, and updated where required to implement the correction. The new package, which also includes new samples and enhanced functionality, is available for download.

    Upvotes
    Christiaan Meihsl avatar image
    13.7k ●26 ●8 ●12

    @chinelo.okoli, the code snippets you posted look ok.

    Could you share your entire code so I could test it ? Could you also share your report template and instrument list ? If you do not want to post these on the forum you can also send them to me directly.

    Added notes:

    Our sample code DSS2ImmediateScheduleTicksTRTHClient2 (available under the downloads tab of TRTH REST) was tested for extractions up to 14MB gziped (99MB unzipped) without issue. Could you try that one with your template and instrument list to see if it exhibits the same issue ?

    If you are using a pre-defined instrument list and report template on the server, I can send you a variant of DSS2ImmediateScheduleTicksTRTHClient2 that would use them which you could use for testing.

    When you get 2 files of different size, what does the last line of data in the file look like ? Is it truncated ?

    icon clock
    10 |1500

    Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

    Christiaan Meihsl avatar image   Christiaan Meihsl

    Customer sent me his code directly. Issue reproduced and currently under investigation.

    Christiaan Meihsl avatar image   Christiaan Meihsl

    Issue reproduced with DSS2ImmediateScheduleTicksTRTHClient2 and a larger data set (~1.1MB gziped = ~100k data lines).

    Saving data to file (method extractAndSaveFile) is ok.

    When treating the data stream on the fly (method extractFileAndDisplayContents) with this code

    BufferedReader rd = new BufferedReader(new InputStreamReader(response.getEntity().getContent()));
    while ((line = rd.readLine()) != null) {
                   
      …
    }

    the result set is sometimes truncated (last lines are missing). Issue appears randomly, and not with small data sets. Under investigation.

    Current workaround: save to file, read back from file.

    Christiaan Meihsl avatar image   Christiaan Meihsl

    Same code seems to retrieve data ok from On Demand but has issues when from Schedule (they use different end points). Escalated to DSS SWAT.

    Christiaan Meihsl avatar image   Christiaan Meihsl

    I will soon publish a solution, thanks to some great help from DSS support.

    Upvotes
    chinelo.okoli avatar image
    1 ●1 ●1 ●3

    Thanks a lot Christiaan

    icon clock
    10 |1500

    Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

    Write an Answer

    Hint: Notify or tag a user in this post by typing @username.

    Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.