Hi
After making some minor changes (log traces, error handling, URL replacement, a return value is now provided) on the awsRetrieveDataAndSaveToFile method found on DSSOnDemandIntradayBarsRTH.java Java sample file, my method looks like the way is shown below:
public boolean extractAndSaveFileFromAWSServer( String filename, String extractedFileId) throws DataAccessException {
boolean success = false;
CloseableHttpClient httpclient = HttpClientBuilder.create().disableContentCompression().build();
logger.info("Retrieve compressed data directly from the AWS server on local file '"+ filename+"'");
//final String url = getConfig().getUrl() + "/Extractions/RawExtractionResults('"+extractedFileId+"')/$value";
final String url = getConfig().getUrl() + "/Extractions/ExtractedFiles('"+extractedFileId+"')/$value";
logger.info("Url for extraction file retrieval: "+url);
HttpGet requestGet = new HttpGet(url);
requestGet.addHeader("Authorization", "Token "+ getSessionToken());
requestGet.addHeader("Prefer", "respond-async");
//The next header asks for the download to occur through AWS:
requestGet.addHeader("X-Direct-Download", "true");
try {
HttpClientContext context = HttpClientContext.create();
HttpResponse responseGet = httpclient.execute(requestGet, context);
//Handle the redirection:
//Initialise awsURI to the initial URI (i.e. the same value as that of url):
URI awsURI = requestGet.getURI();
//Retrieve the last Location, which is the AWS URL:
List<URI> locations = context.getRedirectLocations();
if (locations != null) {
awsURI = locations.get(locations.size() - 1);
logger.info("AWS URI for file '"+filename+"' ('"+extractedFileId+"') is '"+awsURI+"'");
} else {
throw new IOException("ERROR: could not retrieve the AWS URI");
}
URL myURL = awsURI.toURL();
//To request the file from AWS we do not use the session Token (which is for the RTH server).
//Authentication is done through the AWS URL, which is a self signed URL.
HttpURLConnection myURLConnection = (HttpURLConnection)myURL.openConnection();
//This works both with compressed and non compressed files
try( DataInputStream readerIS = new DataInputStream( myURLConnection.getInputStream())) {
Files.copy (readerIS, Paths.get (filename));
}
logger.info("File '"+filename+"'was saved to disk (AWS).");
//Closing the http client
httpclient.close();
} catch (ClientProtocolException e) {
throw new DataAccessException("Error when reading/saving the extraction file "+extractedFileId+"!",e);
} catch (IOException e) {
throw new DataAccessException("Error when reading/saving the extraction file "+extractedFileId+"!",e);
}
success=true;
return success;
}
Unfortunately, we are getting and IOException exception (ERROR: could not retrieve the AWS URI )
2022-03-01 08:50:27,274 ERROR [main] - com.apama.iaf.transport.reuters.trth2.test.TestClientDepth: AdapterException occured! com.apama.iaf.transport.reuters.trth2.exception.DataAccessException: Error when reading/saving the extraction file VjF8MHgwN2ViNWRhNjM1MWRhMjIyfA! at com.apama.iaf.transport.reuters.trth2.api.ApiHelper.extractAndSaveFileFromAWS(ApiHelper.java:909) at com.apama.iaf.transport.reuters.trth2.api.ApiHelper.saveAndReadReportExtractionsFiles(ApiHelper.java:778) at com.apama.iaf.transport.reuters.trth2.handler.AbstractRequestHandlerImpl.process(AbstractRequestHandlerImpl.java:167) at com.apama.iaf.transport.reuters.trth2.handler.AbstractRequestHandlerImpl.processRequest(AbstractRequestHandlerImpl.java:115) at com.apama.iaf.transport.reuters.trth2.test.TestClientDepth.main(TestClientDepth.java:129) Caused by: java.io.IOException: ERROR: could not retrieve the AWS URI at com.apama.iaf.transport.reuters.trth2.api.ApiHelper.extractAndSaveFileFromAWS(ApiHelper.java:888) ... 4 more
which is raised where redirection's location availability is checked, as it's shown in the following code excerpt:
if (locations != null) {
awsURI = locations.get(locations.size() - 1);
logger.info("AWS URI for file '"+filename+"' ('"+extractedFileId+"') is '"+awsURI+"'");
} else {
throw new IOException("ERROR: could not retrieve the AWS URI");
}
Any idea @Christiaan Meihsl Meihsl about what's causing the issue?
Thanks in advance for your help.
