Discover Refinitiv
MyRefinitiv Refinitiv Perspectives Careers
Created with Sketch.
All APIs Questions & Answers  Register |  Login
Ask a question
  • Questions
  • Tags
  • Badges
  • Unanswered
Search:
  • Home /
  • TRTH /
avatar image
REFINITIV
Question by Janik.Zikovsky · Feb 11, 2019 at 12:26 PM · datascope selecttrth v2s3

TRTH - direct copy from one AWS S3 bucket to another S3 bucket

@Christiaan Meihsl - I have seen your documentation on downloading directly from your AWS S3 bucket at https://developers.thomsonreuters.com/article/boost-trth-downloads-aws

I am wondering if it possible to go one step further and issue a direct S3 to S3 copy command. Such commands are normally much faster because the client does not actually download the data, it is copied by AWS on their back-end. See for example: https://docs.aws.amazon.com/AmazonS3/latest/dev/CopyingObjectUsingJava.html

Do you have any advice on how to authenticate the copy with AWS?

CC - @Raghavender.Bommoju

People who like this

0 Show 0
Comment
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

5 Replies

  • Sort: 
avatar image
REFINITIV
Best Answer
Answer by Janik.Zikovsky · Feb 19, 2019 at 12:36 PM

Answering my own question: As far as I can tell, this is not currently possible. DSS would have to provide a signed URL authorizing this request.

Comment

People who like this

0 Show 0 · Share
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

avatar image
REFINITIV
Answer by jirapongse.phuriphanvichai · Feb 12, 2019 at 04:32 AM

When downloading TRTH data from AWS, you get the pre-signed URL. The question could be how to copy an object from the pre-signed URL. I found a few answers via the stack overflow.

  • How do I copy an object with presigned URL?

Someone has mentioned about a pre-signed PUT URL with the x-amz-copy-source header.

Comment

People who like this

0 Show 1 · Share
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

avatar image
REFINITIV
Janik.Zikovsky · Feb 12, 2019 at 09:45 AM 0
Share

Thanks - but I think the accepted answer (that it is not possible) is probably correct. I don't see a way you can use the signature when the query has to change from a download to a copy. The signature is made of a hash that includes the request parameters, any change would invalidate the signature.

avatar image
Answer by pankaj · Jul 10, 2020 at 03:50 AM

Can someone please update the latest situation in this regards?

What is the Best Method for DIRECTLY COPYING the VBD "Venue by Day" Data Files from Refinitiv AWS S3 Bucket to the Client's AWS S3 Bucket, without having to first download and then again upload these files individually from the client machine? What are the best practices for this particular type of AWS S3 direct copy method?

Please provide links to the example codes to accomplish the same.

Regards

@Christiaan Meihsl @Janik.Zikovsky @jirapongse.phuriphanvichai @veerapath.rungruengrayubkul

Comment

People who like this

0 Show 0 · Share
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

avatar image
Answer by janik.zikovsky1 · Jul 10, 2020 at 03:20 PM

Hi Pankaj,


I have been able to do this partially - technically you are still downloading to the local machine but by passing the streams around you do it in a streaming way, from one inputStream to another, and avoid having to create a local file to then upload. In this code snippet awsURI is the signed URL returned by DSS :


    URLConnection urlConnection = awsURI.toURL().openConnection();
    // In some cases (reposted files) data as reported by DSS is incorrect. Happened on May 11, 2020.
    // Need to rely on HTTP headers instead
    long contentLength = urlConnection.getContentLengthLong();
    if (contentLength <= 0) {
      // But as a failure scenario, use the DSS data.
      contentLength = task.getFileSize();
    }
    // Same for MD5 sum
    String md5sum = urlConnection.getHeaderField("x-amz-meta-md5sum");
    if (StringUtils.isBlank(md5sum)) {
      md5sum = task.getContentMd5();
    }
    LOGGER.info("Starting download of {} with content length {}", awsURI.toURL(), contentLength);
    try (InputStream stream = urlConnection.getInputStream()) {
      String filename = FilenameUtils.getName(awsURI.toURL().getPath());
      ObjectMetadata metadata = new ObjectMetadata();
      metadata.setContentLength(contentLength);

      // Convert hex-encoded MD5 to raw bytes, then base64-encode
      byte[] binaryMd5 = DatatypeConverter.parseHexBinary(md5sum);
      String base64md5 = Base64.encodeBase64String(binaryMd5);
      metadata.setContentMD5(base64md5);
      // Save a simple MD5 sum because the ETag cannot be used in multi-part uploads.
      // Metadata has to start with "x-amz-meta-" to be accepted
      metadata.setHeader("x-amz-meta-" + VbdUtility.CHECKSUM_METADATA_NAME, task.getContentMd5());

      Date date = new Date();
      metadata.setLastModified(date);

      PutObjectRequest request = new PutObjectRequest(config.downloadBucket(), destKey, stream, metadata);
      request.setSdkClientExecutionTimeout(-1);  // disabled

      // Start the upload in the background, but then wait for completion
      Upload upload = transferManager.upload(request);
      upload.waitForCompletion();

      TransferState transferState = upload.getState();
      LOGGER.info("Transfer status of {} is {} and upload.isDone: {}", filename, transferState, upload.isDone());
      return transferState;
    }
  }





Comment
pankaj

People who like this

1 Show 1 · Share
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

avatar image
pankaj · Jul 11, 2020 at 11:19 AM 0
Share

Thanks Janik for sharing the solution that you are using.

Are you able to verify the data integrity of the uploaded VBD files on your AWS account by comparing the MD5 checksum? Does this streaming method works perfectly every time or you notice the incidents when the MD5 does not matches and you have to re-upload the files again?

And does Refinitiv staff have any alternative methods for achieving the same results? Will some support staff please comment?

If AWS S3 direct upload is not supported, then even some other cloud direct upload will also be helpful, maybe like uploading on Google Cloud Platform GCP instead.

The only main requirement is that the upload should be directly from Refinitiv AWS Link to the client Cloud Platform, without having the need to first download and then re-upload these big files.

Any help is appreciated.

Best Regards

avatar image
Answer by janik.zikovsky1 · Jul 13, 2020 at 12:49 AM

Hi Pankaj,

Yes, the upload's MD5 is validated - the AWS S3 SDK does this for you because of the "metadata.setContentMD5(base64md5);" call - see AWS S3 API docs for more details. I don't believe I've seen a case where the MD5 validation failed but there is a retry mechanism implemented so I may not have noticed.

Sincerely,

Janik Zikovsky


Comment
pankaj

People who like this

1 Show 1 · Share
10 |1500 characters needed characters left characters exceeded
▼
  • Viewable by all users
  • Viewable by moderators
  • Viewable by moderators and the original poster
  • Advanced visibility
Viewable by all users

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

avatar image
pankaj · Jul 14, 2020 at 09:34 AM 0
Share

Thank you so much Janik.

Watch this question

Add to watch list
Add to your watch list to receive emailed updates for this question. Too many emails? Change your settings >
13 People are following this question.

Related Questions

Which is the best way to get EOD prices for equity options?

TRTH v2 trade qualifiers in a TimeAndSales request

Tick direction in TimeandSales

Why the url’s structure is different in the 202 Status case TRTHv2 REST API

Quota Usage Details TRTH V2 by User (not global)

  • Feedback
  • Copyright
  • Cookie Policy
  • Privacy Statement
  • Terms of Use
  • Careers
  • Anonymous
  • Sign in
  • Create
  • Ask a question
  • Spaces
  • Alpha
  • App Studio
  • Block Chain
  • Bot Platform
  • Calais
  • Connected Risk APIs
  • DSS
  • Data Fusion
  • Data Model Discovery
  • Datastream
  • Eikon COM
  • Eikon Data APIs
  • Elektron
    • EMA
    • ETA
    • WebSocket API
  • Legal One
  • Messenger Bot
  • Messenger Side by Side
  • ONESOURCE
    • Indirect Tax
  • Open PermID
    • Entity Search
  • Org ID
  • PAM
    • PAM - Logging
  • ProView
  • ProView Internal
  • Product Insight
  • Project Tracking
  • Refinitiv Data Platform
    • Refinitiv Data Platform Libraries
  • Rose's Space
  • Screening
    • Qual-ID API
    • Screening Deployed
    • Screening Online
    • World-Check One
    • World-Check One Zero Footprint
  • Side by Side Integration API
  • TR Knowledge Graph
  • TREP APIs
    • CAT
    • DACS Station
    • Open DACS
    • RFA
    • UPA
  • TREP Infrastructure
  • TRIT
  • TRKD
  • TRTH
  • Thomson One Smart
  • Transactions
    • REDI API
  • Velocity Analytics
  • Wealth Management Web Services
  • World-Check Data File
  • Explore
  • Tags
  • Questions
  • Badges