question

Upvotes
Accepted
Edward Richards avatar image
1 ●0 ●0 ●2

Refinitiv Data Platform - CFS - S3 native download

We are using CFS via the Refinitiv Data Platform to download a custom Tick History extraction. We can access the files through api.refinitiv.com and providing the token as a header, as in the [Python examples](https://github.com/Refinitiv-API-Samples/Article.RDP.Python.IntroToCFS/blob/main/CFSIntro.ipynb).


However we wish to download the files natively in AWS, so that we can transfer them directly to our own S3 bucket without going out to the web. E.g. `aws s3 cp s3://refinitiv-bucket/.../file.csv.gz s3://our-bucket/file.csv.gz`


The file API exposes the url for each file, such as

`https://s3-us-east-1.amazonaws.com/a.../..../normalised/..../data/merged/merged-Report.csv.gz`

Can we access this file directly in AWS?

#technologyrdp-apirdpaws
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Jirapongse avatar image   Jirapongse

Contacted the CFS product owner (Janakiraman, Manivannan Kaveripakkam).

wasin.w avatar image   wasin.w

Hello @Edward Richards

Thank you for your participation in the forum. Is the reply below satisfactory in resolving your query?


If so please can you click the 'Accept' text next to the appropriate reply? This will guide all community members who have a similar question.

Thanks,


AHS

raksina.samasiri avatar image   raksina.samasiri

Please be informed that a reply has been verified as correct in answering the question, and has been marked as such.

Thanks,
AHS

4 Answers

· Write an Answer
Upvotes
Accepted
Jirapongse avatar image
79.2k ●251 ●52 ●74

@Edward Richards

I got the response from the product team that the direct access to the file in the bucket is not allowed.

Howover, you can contact the helpdesk team directly via MyRefinitiv to confirm this.


icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
wasin.w avatar image
24.7k ●54 ●17 ●14

Hello @Edward Richards

Unfortunately, the RDP Bulk file does not expose the S3 location. However, you can download the API from RDP API /file-store/v1/files/{id}/stream service directly by calling the API without "?doNotRedirect=true" parameter

Example:

https://api.refinitiv.com/file-store/v1/files/{fileId}/stream?doNotRedirect=true

Result: Return the expose public HTTP URL Like "https://s3-us-east-1.amazonaws.com/a.../..../normalised/..../data/merged/merged-Report.csv.gz"

https://api.refinitiv.com/file-store/v1/files/{fileId}/stream

Result: Download the file directly.

I hope this information helps.


icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
Jirapongse avatar image
79.2k ●251 ●52 ●74

@Edward Richards

I think the Refinitiv Data Platform - CFS returns a presigned URL. You can use it to download the file.

However, according to this discussion, you can't copy it directly to another S3 bucket.



icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
Edward Richards avatar image
1 ●0 ●0 ●2

Thank you both.

This is surprising, why does the RDP CFS files API (https://api.refinitiv.com/file-store/v1/files?filesetId=) give information about the S3 location, such as S3 URL and AWS ARN, if accessing it natively in AWS isn't supported?

1694525617047.png


1694525617047.png (35.0 KiB)
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.