question

Upvotes
Accepted
4 0 0 2

ERROR: processing did not complete successfully !

Trying to download ZFU8 I get the following response

Chain Identifier: ChainRIC 0#ZFU8, Constituents: 2
RIC      1FVU8^0,
Start date       29.08.2008 0:00:00,
Change Date      31.05.2014 0:00:00,
Expiration Date  30.09.2008 0:00:00,
RIC      1FVU8^0,
Start date       29.08.2008 0:00:00,
Change Date      01.10.2016 0:00:00,
Expiration Date  30.09.2008 0:00:00,
RIC      1FVU8^0,
Start date       29.08.2008 0:00:00,
Change Date      31.08.2017 0:00:00,
Expiration Date  30.09.2008 0:00:00,
RIC      ZFU8,
Start date       ,
Change Date      02.07.2006 0:00:00,
Expiration Date  19.09.2008 0:00:00,
RIC      ZFU8,
Start date       ,
Change Date      06.07.2006 0:00:00,
Expiration Date  19.09.2008 0:00:00,
RIC      ZFU8,
Start date       29.08.2008 0:00:00,
Change Date      02.06.2007 0:00:00,
Expiration Date  19.09.2008 0:00:00,
RIC      ZFU8,
Start date       ,
Change Date      15.06.2007 0:00:00,
Expiration Date  ,
RIC      ZFU8,
Start date       ,
Change Date      01.07.2007 0:00:00,
Expiration Date  30.09.2008 0:00:00,
RIC      ZFU8,
Start date       ,
Change Date      07.07.2007 0:00:00,
Expiration Date  30.09.2008 0:00:00,
RIC      ZFU8,
Start date       ,
Change Date      26.07.2007 0:00:00,
Expiration Date  30.09.2008 0:00:00,
RIC      ZFU8,
Start date       ,
Change Date      04.08.2007 0:00:00,
Expiration Date  30.09.2008 0:00:00,
RIC      ZFU8,
Start date       ,
Change Date      17.08.2007 0:00:00,
Expiration Date  30.09.2008 0:00:00,
RIC      ZFU8,
Start date       29.08.2008 0:00:00,
Change Date      24.08.2007 0:00:00,
Expiration Date  30.09.2008 0:00:00,
RIC      ZFU8,
Start date       29.08.2008 0:00:00,
Change Date      06.10.2007 0:00:00,
Expiration Date  30.09.2008 0:00:00,
RIC      ZFU8^0,
Start date       29.08.2008 0:00:00,
Change Date      04.10.2008 0:00:00,
Expiration Date  30.09.2008 0:00:00,
RIC      ZFU8^0,
Start date       29.08.2008 0:00:00,
Change Date      06.12.2008 0:00:00,
Expiration Date  30.09.2008 0:00:00,
RIC      ZFU8^0,
Start date       29.08.2008 0:00:00,
Change Date      04.07.2009 0:00:00,
Expiration Date  30.09.2008 0:00:00,
RIC      ZFU8^0,
Start date       29.08.2008 0:00:00,
Change Date      23.11.2009 0:00:00,
Expiration Date  30.09.2008 0:00:00,
RIC      ZFU8^0,
Start date       29.08.2008 0:00:00,
Change Date      26.11.2011 0:00:00,
Expiration Date  30.09.2008 0:00:00,
ZFU8,    Ric,    03.07.2008 0:00:00 +00:00,      30.09.2008 0:00:00 +00:00,      6
ExtractionID: 2000000665078075
Waiting for file size...
Waiting for file size...
Waiting for file size...
Waiting for file size...
Waiting for file size...
ERROR: processing did not complete successfully !

WARNING messages:
Waiting for file size...
Waiting for file size...
Waiting for file size...
Waiting for file size...
Waiting for file size...

Press Enter to exit

Here is the code I use to access historical data

using System;
using System.Linq;
using System.Diagnostics;
using System.IO;
using DataScope.Select.Api.Extractions;
using DataScope.Select.Api.Extractions.ExtractionRequests;
using DataScope.Select.Api.Extractions.ReportExtractions;
using DataScope.Select.Api.Extractions.ReportTemplates;
using DataScope.Select.Api.Extractions.Schedules;
using DataScope.Select.Api.Extractions.SubjectLists;
using DataScope.Select.Api.Content;
using DataScope.Select.Core.RestApi;
using DataScope.Select.Api.Search;
using DataScope.Select.Api;
using System.Text;
using ICSharpCode.SharpZipLib.GZip;
using System.Text.RegularExpressions;

namespace RefinitivHistoryTesting
{

    class Program
    {
        public static string getExtractionID(string note)
        {
            string pattern = @"Extraction ID: ([0-9]+)";
            string extractionID = "";
            Regex rgx = new Regex(pattern);
            Match match = rgx.Match(note);
            if (match.Success)
            {
                string[] words = match.Value.Split(':');
                extractionID = words[1].ToString().Trim();
                Console.WriteLine("ExtractionID: {0}", extractionID);
            }
            return extractionID;
        }
        static void Main(string[] args)
        {
            //-----------------------------------------------------------------
            //Connect and authenticate to the DSS server:
            //-----------------------------------------------------------------
            Uri dssUri = new Uri("https://selectapi.datascope.refinitiv.com/RestApi/v1/");
            string dssUserName = "****";
            string dssUserPassword = "****";

            //var out_dir = @".\Storage\";
            //var out_dir = @"E:\TimeTick\";
            var out_dir = @"C:\Users\Manuchar\Refinitiv_history\";

            var SearchContext = new SearchContext(dssUri, dssUserName, dssUserPassword);

            var ExtractionsContext = new ExtractionsContext(dssUri, dssUserName, dssUserPassword);
            ExtractionsContext.Options.AutomaticDecompression = false;
            try
            {
                ExtractionsContext.DefaultRequestHeaders.Add("x -direct-download", "true");
            }
            catch (Exception)
            { }

            var availableMbPFields = ExtractionsContext.GetValidContentFieldTypes(ReportTemplateTypes.TickHistoryRaw);
            var availableTnSFields = ExtractionsContext.GetValidContentFieldTypes(ReportTemplateTypes.TickHistoryTimeAndSales);


            var dateRange = DateTimeRange.Create(
                new DateTimeOffset(2000, 08, 01, 00, 00, 0, TimeSpan.Zero),
                new DateTimeOffset(2026, 02, 20, 00, 00, 0, TimeSpan.Zero));

            var request = new HistoricalChainResolutionRequest
            {
                ChainRics = new[] { "0#ZFU8" },
                Range = dateRange
            };

            var chain_results = SearchContext.HistoricalChainResolution(request);
            foreach (var chain in chain_results)
            {
                Console.WriteLine("Chain Identifier: {0} {1}, Constituents: {2}", chain.IdentifierType, chain.Identifier, chain.Constituents.Count());
                foreach (var constituent in chain.Constituents)
                {
                    var searchResults = SearchContext.HistoricalSearch(
                        HistoricalSearchRequest.Create(
                            constituent.Identifier, constituent.IdentifierType, dateRange, HistoricalResultsBy.Ric, null));

                    var result = searchResults.FirstOrDefault(pos => pos.Identifier.Equals("ZFU8"));
                    if (result != null)
                    {

                        var extractionDatesResult = ExtractionsContext.Extract(
                            new HistoricalReferenceExtractionRequest
                            {
                                ContentFieldNames = new DssCollection<string>()
                                {
                                    "RIC", "Start date", "Change Date", "Expiration Date"
                                },

                                Condition = new HistoricalReferenceCondition
                                {
                                    StartDate = result.FirstDate,
                                    EndDate = result.LastDate
                                },

                                IdentifierList = new InstrumentIdentifierList
                                {
                                    InstrumentIdentifiers = new[]
                                    {
                                        InstrumentIdentifier.Create(result.IdentifierType, result.Identifier)
                                    },
                                    ValidationOptions = new InstrumentValidationOptions
                                    {
                                        AllowHistoricalInstruments = true
                                    },
                                    UseUserPreferencesForValidationOptions = false
                                }
                            });

                        dynamic srcLastDate = null;
                        foreach (var content in extractionDatesResult)
                            foreach (var prop in content.DynamicProperties)
                            {
                                Console.WriteLine("{0}\t {1},\r", prop.Key, prop.Value);
                                if (srcLastDate == null && prop.Key == "Expiration Date" && prop.Value != null) srcLastDate = prop.Value;
                            }

                        dynamic lastDate = new DateTimeOffset(2008, 09, 30, 00, 00, 00, TimeSpan.Zero);//result.LastDate;
                        dynamic startDate = new DateTimeOffset(2008, 07, 03, 00, 00, 00, TimeSpan.Zero); //result.FirstDate;

                        Console.WriteLine("{0},\t {1},\t {2},\t {3},\t {4}", result.Identifier, result.IdentifierType, startDate, lastDate, result.DomainCode);

                        var extractionResult = ExtractionsContext.ExtractRaw(
                             
                             new TickHistoryRawExtractionRequest
                             {
                                 Condition = new TickHistoryRawCondition
                                 {
                                     ReportDateRangeType = ReportDateRangeType.Range,
                                     QueryStartDate = startDate,// new DateTimeOffset(2023, 03, 21, 20, 0, 0, TimeSpan.Zero), 
                                     QueryEndDate = lastDate,// new DateTimeOffset(2023, 03, 21, 19, 59, 59, TimeSpan.Zero), 
                                     ExtractBy = TickHistoryExtractByMode.Ric,
                                     MessageTimeStampIn = TickHistoryTimeOptions.LocalExchangeTime,
                                     SortBy = TickHistorySort.SingleByRic,
                                     DomainCode = TickHistoryRawDomain.MarketPrice,
                                     DisplaySourceRIC = true
                                 },
                                 IdentifierList = new InstrumentIdentifierList
                                 {
                                     InstrumentIdentifiers = new[]
                                            {
                                                InstrumentIdentifier.Create(result.IdentifierType, result.Identifier)
                                            },

                                     ValidationOptions = new InstrumentValidationOptions
                                     {
                                         AllowHistoricalInstruments = true
                                     },
                                     UseUserPreferencesForValidationOptions = false
                                 },
                             });

                        string extractionID = getExtractionID(extractionResult.Notes[0]);
                        ReportExtraction resultsize = null;
                        long fileSize = 0;

                        Boolean success = false;
                        string errorMsgs = "";
                        string warningMsgs = "";
                        string permissionMsgs = "";

                        int attempts = 0; 

                        while ((resultsize == null || resultsize.FullFile == null) && attempts < 5)
                        {
                            System.Threading.Thread.Sleep(5000); 
                            try
                            {
                                resultsize = ExtractionsContext.ReportExtractionOperations.Get(extractionID);
                                ExtractionsContext.LoadProperty(resultsize, "FullFile");
                                if (resultsize.FullFile != null)
                                {
                                    fileSize = resultsize.FullFile.Size;
                                    Console.WriteLine("File Size: {0}", fileSize);
                                    success = true;
                                }
                                else
                                {
                                    Console.WriteLine("Waiting for file size...");
                                    warningMsgs += "Waiting for file size...\n";
                                }
                            }
                            catch (NullReferenceException ex)
                            {
                                
                                Console.WriteLine("FullFile is not yet available, continuing to wait...");
                                errorMsgs += "FullFile is not yet available, continuing to wait...\n";
                                errorMsgs += "Exception message: " + ex.Message + "\n";
                            }
                            catch (Exception ex)
                            {
                                
                                Console.WriteLine("An error occurred: " + ex.Message);
                                errorMsgs += "An error occurred: " + ex.Message + "\n";
                            }

                            attempts++; 
                        }

                        if (!success)
                        {
                            Console.WriteLine("ERROR: processing did not complete successfully !\n" + errorMsgs);
                        }
                        if (warningMsgs != "")
                        {
                            Console.WriteLine("WARNING messages:\n" + warningMsgs);
                        }
                        if (permissionMsgs != "")
                        {
                            Console.WriteLine("PERMISSION ISSUES messages:\n" + permissionMsgs);
                        }

                     var fileName = out_dir + "_" + result.Identifier + "_" + startDate.DateTime.ToString("dd.MM.yyyy") + "_" + lastDate.DateTime.ToString("dd.MM.yyyy") + "_RawMarketPrice.gzip";
                            using (var response = ExtractionsContext.RawExtractionResultOperations.GetReadStream(extractionResult))
                            using (var fileStream = File.Create(fileName))
                                response.Stream.CopyTo(fileStream);
                        }
                    }
                }
                try
                {
                    ExtractionsContext.DefaultRequestHeaders.Remove("x-direct-download");
                }
                catch (Exception)
                { }
                Console.WriteLine("Press Enter to exit");
                Console.ReadLine();
            }
        }
    }

Can anyone help me with the problem without sending me to the A documentation?

I've read everything and tried different options.

I have a problem with the following files

ZFU8
startDate = new DateTimeOffset(2008, 07, 03, 00, 00, 00, TimeSpan.Zero)
lastDate = new DateTimeOffset(2008, 09, 30, 00, 00, 00, TimeSpan.Zero)

ZFZ8
startDate = new DateTimeOffset(2008, 08, 16, 00, 00, 00, TimeSpan.Zero)
lastDate = new DateTimeOffset(2008, 12, 31, 00, 00, 00, TimeSpan.Zero)

ZFU9
startDate = new DateTimeOffset(2009, 05, 23, 00, 00, 00, TimeSpan.Zero)
lastDate = new DateTimeOffset(2009, 09, 30, 00, 00, 00, TimeSpan.Zero)

ZFU1
startDate = new DateTimeOffset(2011, 05, 21, 00, 00, 00, TimeSpan.Zero)
lastDate = new DateTimeOffset(2011, 09, 30, 00, 00, 00, TimeSpan.Zero)

ZFM2
startDate = new DateTimeOffset(2012, 02, 18, 00, 00, 00, TimeSpan.Zero)
lastDate = new DateTimeOffset(2012, 06, 29, 00, 00, 00, TimeSpan.Zero)

ZFU3
startDate = new DateTimeOffset(2013, 05, 18, 00, 00, 00, TimeSpan.Zero)
lastDate = new DateTimeOffset(2013, 09, 30, 00, 00, 00, TimeSpan.Zero) 

Who can, please give some good code advice.

What could be the problem

I know for sure that the code is 100% functional

With this code I have downloaded all chainRIC: ZF except those with error:

ERROR: processing did not complete successfully !


#technology#producttick-history-rest-apiricserror.net
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Hello @manuchar_chagiashvili

Thank you for your participation in the forum. Is the reply below satisfactory in resolving your query?


If so please can you click the 'Accept' text next to the appropriate reply? This will guide all community members who have a similar question.

Thanks,


AHS

Upvotes
Accepted
78.8k 249 52 74

@manuchar_chagiashvili

Thank you reaching out to us.

I can run the code properly.

1704426832115.png

According to the code, the FullFile is not available. You need to check the extraction notes and report's status.

        public static string getExtractionID(string note)
        {
            Console.WriteLine("### Notes ###\n" + note );
            string pattern = @"Extraction ID: ([0-9]+)";
            string extractionID = "";
            Regex rgx = new Regex(pattern);
            Match match = rgx.Match(note);
            if (match.Success)
            {
                string[] words = match.Value.Split(':');
                extractionID = words[1].ToString().Trim();
                Console.WriteLine("ExtractionID: {0}", extractionID);
            }
            return extractionID;
        }
...
while ((resultsize == null || resultsize.FullFile == null) && attempts < 5)
                        {
                            System.Threading.Thread.Sleep(5000);
                            try
                            {
                                resultsize = ExtractionsContext.ReportExtractionOperations.Get(extractionID);
                                Console.WriteLine("Status: "+resultsize.Status);
                                Console.WriteLine("Detalied Status: "+resultsize.DetailedStatus);
                                ExtractionsContext.LoadProperty(resultsize, "FullFile");
                                if (resultsize.FullFile != null)
                                {
                                    fileSize = resultsize.FullFile.Size;
                                    Console.WriteLine("File Size: {0}", fileSize);
                                    success = true;
                                }
...




1704426832115.png (26.2 KiB)
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Thank you.

Here's what I got:

ExtractionID: 2000000665353495
Status: Completed
Detailed Status: Done
FullFile is not yet available, continuing to wait...
Status: Completed
Detailed Status: Done
FullFile is not yet available, continuing to wait...
Status: Completed
Detailed Status: Done
FullFile is not yet available, continuing to wait...
Status: Completed
Detailed Status: Done
FullFile is not yet available, continuing to wait...
Status: Completed
Detailed Status: Done
FullFile is not yet available, continuing to wait...
ERROR: processing did not complete successfully !
 
Press Enter to exit

Can you tell me what else I can modify in the code to download the necessary RICs?

P.S. Was writing a comment, but it got caught in moderation. I'll repeat it here.

Jirapongse avatar image Jirapongse manuchar_chagiashvili

Please share the extraction note.

 public static string getExtractionID(string note)
        {
            Console.WriteLine("### Notes ###\n" + note );
            string pattern = @"Extraction ID: ([0-9]+)";
            string extractionID = "";
            Regex rgx = new Regex(pattern);
            Match match = rgx.Match(note);
            if (match.Success)
            {
                string[] words = match.Value.Split(':');
                extractionID = words[1].ToString().Trim();
                Console.WriteLine("ExtractionID: {0}", extractionID);
            }
            return extractionID;
        }
Chain Identifier: ChainRIC 0#ZF:, Constituents: 53
Start date       29.08.2008 0:00:00,
Change Date      26.11.2011 0:00:00,
Expiration Date  30.09.2008 0:00:00,
ZFU8,    Ric,    03.07.2008 0:00:00 +00:00,      30.09.2008 0:00:00 +00:00,      6
### Notes ###
Extraction Services Version 17.4.1.46677 (972294413835), Built Oct 25 2023 19:05:27
User ID: 9034323
Extraction ID: 2000000665387197
Correlation ID: CiD/9034323/0x0000000000000000/RESTAPI-I/EXT.2000000665387197
Schedule: 0x08c46b418b396db4 (ID = 0x0000000000000000)
Input List (1 items):  (ID = 0x08c46b418b396db4) Created: 01/05/2024 07:05:59 Last Modified: 01/05/2024 07:05:59
Report Template: _OnD_0x08c46b418b396db4 (ID = 0x08c46b418b496db4) Created: 01/05/2024 07:03:24 Last Modified: 01/05/2024 07:03:24
Schedule dispatched via message queue (0x08c46b418b396db4), Data source identifier (CB206CFAEABC47BD868EB681D8CD60F5)
Schedule Time: 01/05/2024 07:03:25
Processing started at 01/05/2024 07:03:25
Processing completed successfully at 01/05/2024 07:06:00
Extraction finished at 01/05/2024 07:06:00 UTC, with servers: tmc28nu8xn03, TRTH (149.979 secs)
Historical Instrument <RIC,ZFU8> expanded to 1 RIC: ZFU8.
Total instruments after instrument expansion = 1


Show more comments
Upvotes
4 0 0 2

Thank you.

Here's what I got:

Chain Identifier: ChainRIC 0#ZF:, Constituents: 53
RIC      1FVU8^0,
Start date       29.08.2008 0:00:00,
Change Date      31.05.2014 0:00:00,
Expiration Date  30.09.2008 0:00:00,
RIC      1FVU8^0,
Start date       29.08.2008 0:00:00,
Change Date      01.10.2016 0:00:00,
Expiration Date  30.09.2008 0:00:00,
RIC      1FVU8^0,
Start date       29.08.2008 0:00:00,
Change Date      31.08.2017 0:00:00,
Expiration Date  30.09.2008 0:00:00,
RIC      ZFU8,
Start date       ,
Change Date      02.07.2006 0:00:00,
Expiration Date  19.09.2008 0:00:00,
RIC      ZFU8,
Start date       ,
Change Date      06.07.2006 0:00:00,
Expiration Date  19.09.2008 0:00:00,
RIC      ZFU8,
Start date       29.08.2008 0:00:00,
Change Date      02.06.2007 0:00:00,
Expiration Date  19.09.2008 0:00:00,
RIC      ZFU8,
Start date       ,
Change Date      15.06.2007 0:00:00,
Expiration Date  ,
RIC      ZFU8,
Start date       ,
Change Date      01.07.2007 0:00:00,
Expiration Date  30.09.2008 0:00:00,
RIC      ZFU8,
Start date       ,
Change Date      07.07.2007 0:00:00,
Expiration Date  30.09.2008 0:00:00,
RIC      ZFU8,
Start date       ,
Change Date      26.07.2007 0:00:00,
Expiration Date  30.09.2008 0:00:00,
RIC      ZFU8,
Start date       ,
Change Date      04.08.2007 0:00:00,
Expiration Date  30.09.2008 0:00:00,
RIC      ZFU8,
Start date       ,
Change Date      17.08.2007 0:00:00,
Expiration Date  30.09.2008 0:00:00,
RIC      ZFU8,
Start date       29.08.2008 0:00:00,
Change Date      24.08.2007 0:00:00,
Expiration Date  30.09.2008 0:00:00,
RIC      ZFU8,
Start date       29.08.2008 0:00:00,
Change Date      06.10.2007 0:00:00,
Expiration Date  30.09.2008 0:00:00,
RIC      ZFU8^0,
Start date       29.08.2008 0:00:00,
Change Date      04.10.2008 0:00:00,
Expiration Date  30.09.2008 0:00:00,
RIC      ZFU8^0,
Start date       29.08.2008 0:00:00,
Change Date      06.12.2008 0:00:00,
Expiration Date  30.09.2008 0:00:00,
RIC      ZFU8^0,
Start date       29.08.2008 0:00:00,
Change Date      04.07.2009 0:00:00,
Expiration Date  30.09.2008 0:00:00,
RIC      ZFU8^0,
Start date       29.08.2008 0:00:00,
Change Date      23.11.2009 0:00:00,
Expiration Date  30.09.2008 0:00:00,
RIC      ZFU8^0,
Start date       29.08.2008 0:00:00,
Change Date      26.11.2011 0:00:00,
Expiration Date  30.09.2008 0:00:00,
ZFU8,    Ric,    03.07.2008 0:00:00 +00:00,      30.09.2008 0:00:00 +00:00,      6
ExtractionID: 2000000665353495
Status: Completed
Detailed Status: Done
FullFile is not yet available, continuing to wait...
Status: Completed
Detailed Status: Done
FullFile is not yet available, continuing to wait...
Status: Completed
Detailed Status: Done
FullFile is not yet available, continuing to wait...
Status: Completed
Detailed Status: Done
FullFile is not yet available, continuing to wait...
Status: Completed
Detailed Status: Done
FullFile is not yet available, continuing to wait...
ERROR: processing did not complete successfully !

Press Enter to exit

Can you tell me what else I can modify in the code to download the necessary RICs?

icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Upvotes
78.8k 249 52 74

@manuchar_chagiashvili

In the note, you should have this line.

1704439788937.png

According to note, the file wasn't created due to the Quota Limit.


1704439788937.png (26.7 KiB)
icon clock
10 |1500

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

According to note, the file wasn't created due to the Quota Limit.

Jirapongse, thanks!!!

I don't see a button to close the question, since I got the answer and it's clear to me what to do.

Please close the question as answered.

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.