Request Pagination issue with Standard Extractions venue files list

acampbell

When making a request to:

UserPackageDeliveryGetUserPackageDeliveriesByDateRange

with the arguments

(SubscriptionId='0x0400dc1d24a00cb4',FromDate=2018-02-02T12:00:00+00:00,ToDate=2018-02-04T12:00:00+00:00)

The next-link chain continues infinitely. When the response containing the last of the unique results comes, a nextlink is also returned which leads to a response with previously returned values. This in turn contains another nextlink to another such response. This continues without end.

Handling this locally complicates our interface to the TRTHv2 REST API. Is this behaviour expected, and, if not, will it be addressed?

Christiaan Meihsl

@acampbell,

I tried to reproduce your issue, but data of 2 Feb was no longer available. I therefore used more recent dates. To test if paging works properly I varied the maxpagesize setting in the header:

Prefer:odata.maxpagesize=20000;respond-async

Test 1

Following what you mentioned about multiples of 7 days, I made the same request as you, but for 9-11 march, which is exactly 5 weeks later as your original request.

Here is the GET request I used:

https://hosted.datascopeapi.reuters.com/RestApi/v1/StandardExtractions/UserPackageDeliveryGetUserPackageDeliveriesByDateRange(SubscriptionId='0x0400dc1d24a00cb4',FromDate=2018-03-09T12:00:00+00:00,ToDate=2018-03-11T12:00:00+00:00)

Test 1 results:

• maxpagesize=20000 => 1 set of 14240 records
  
• maxpagesize=10000 => 1 set of 10000 records with a nextlink that delivered 4240 records and no additional next link.
  
• maxpagesize=5000 => 2 sets of 5000 records, each with a nextlink; the last nextlink delivered 4240 records and no additional next link.

The total number of records is consistent, this is the expected behavior.

Test 2

I also tried reducing the range to 24 hours, to test smaller maxpagesize values. Here is the GET request I used:

https://hosted.datascopeapi.reuters.com/RestApi/v1/StandardExtractions/UserPackageDeliveryGetUserPackageDeliveriesByDateRange(SubscriptionId='0x0400dc1d24a00cb4',FromDate=2018-03-13T12:00:00+00:00,ToDate=2018-03-14T12:00:00+00:00)

Test 2 results:

• maxpagesize=10000 => 1 set of 6300 records
• maxpagesize=4000 => 1 set of 4000 records with a nextlink that delivered 2300 records and no additional next link.
• maxpagesize=1000 => 6 sets of 1000 records, each with a nextlink; the last nextlink delivered 300 records and no additional next link.

Again, the total number of records is consistent, this is the expected behavior.

Comments:

• The default maxpagesize is 250, you will observe a very long list of next links.
• The total number of records received for a specific request depends on your data permissioning, so running the same test as I did might give you a different total record count, but the behavior in function of the maxpagesize should remain consistent.

Conclusion:

If you are you still observing infinitely looping results sets, please send us the IDs as instructed by Veerapath in his answer, as well as the exact request parameters you used.

acampbell

By the way, this happens for datetime ranges exactly 7 and 14 days later, also. It did not occur 7 days earlier, nor any other multiple of 7 days earlier.

veerapath.rungruengrayubkul

@acampbell,

This is not expected behavior. If there are no more rows of result , the nextlink will not be returned.

Could you please provide sample of datetime ranges and reproduction steps which can be used for the issue's replication?

Also, can you log the "Request-Execution-Correlation-Id" on the response header of the last result's message? The "Request-Execution-Correlation-Id" is useful for further investigation on the server side. For more information about how to retrieve the header, please see this Diagnostics page.

Nipat Kunvutipongsak

https://community.developers.refinitiv.com/discussion/comment/24304#Comment_24304

veerapath.rungruengrayubkul

Feb 20, 2018

@acampbell,

This is not expected behavior. If there are no more rows of result , the nextlink will not be returned.

Could you please provide sample of datetime ranges and reproduction steps which can be used for the issue's replication?

Also, can you log the "Request-Execution-Correlation-Id" on the response header of the last result's message? The "Request-Execution-Correlation-Id" is useful for further investigation on the server side. For more information about how to retrieve the header, please see this Diagnostics page.

@acampbell, can you provide more information as suggested by @veerapath.rungruengrayubkul?