curl: (18) transfer closed with 803828224 bytes remaining to read

Ayan

While trying to download files ( in GB size), we are getting the above message. We are using curl command to retrive the data. Anyone faced same issue earlier? Is there any solution for the same?

Command is as below.

curl -k -X GET -H 'Authorization: Token xxxxxxxxxxxxxxxxxxxxxxx' -o NAS-2017-07-25-NORMALIZEDMP-Data-1-of-1.csv.gz https://10.192.6.221/RestApi/v1/StandardExtractions/UserPackageDeliveries('0x05cf06a3fedb3026')/$value

veerapath.rungruengrayubkul

@Ayan

It seems like the connection has been closed, while the file has been downloading. It might be connection issue between your machine and the firewall.

Curl provides an option to resume download. This option may help you continue the download, once the issue occurs for huge file. Below is the sample command. It can check download progress from the existing file name, and then continue downloading the data.

curl -k -H 'Authorization:  Token xxxxxxxxxxxxxxxxxxxxxxx' -o NAS-2017-07-25-NORMALIZEDMP-Data-1-of-1.csv.gz -C - https://10.192.6.221/RestApi/v1/StandardExtractions/UserPackageDeliveries('0x05cf06a3fedb3026')/$value

Hope this helps.

Ayan

Hi, Thanks for the reply, but it actually doesnot resolve my problem. Your suggestion is to download the file with -C option which will resume the download from the same place where it left. Now I have a few questions here.

1. In worst case scenario, what is the guarantee that it will not terminate the download again after resuming it?

2. If it terminates too, then how many times we must try to complete the download?

3. How will we know that the entire files got downloaded?

Hope I am able to elaborate my problem. Please help.

veerapath.rungruengrayubkul

@Ayan

Yes. The curl option should not solve the issue. It is a workaround once the issue occurs, so you do not need to re-download the file from begining. The issue can reoccur again on this command, once there is a connection closed issue. If the entire file gets the download, the command will exit without any error message.

I am able to download the same file with the curl command in my environment, so I suspect that the issue is in the network and firewall in your environment. Could you contact your administrator to verify your network and firewall?

Ayan

https://community.developers.refinitiv.com/discussion/comment/16734#Comment_16734

Thanks for the reply, I will contact my network team on the same.

Jirapongse

@Ayan

I got the same error when using netsh command to do port forwarding.

C:\netsh interface portproxy add v4tov4 listenport=443 listenaddress=127.0.0.1 connectport=443 connectaddress=hosted.datascopeapi.reuters.com

Then, I used this curl command on Windows to download a VBD file.

curl -k -X GET -H "Authorization: Token <token>" -o NAS-2017-07-25-NORMALIZEDMP-Data-1-of-1.csv.gz https://127.0.0.1/RestApi/v1/StandardExtractions/UserPackageDeliveries('0x05cf06a3fedb3026')/$value

After that, I cut the connection between netsh and datascope.

The curl command showed the following error:

curl: (18) transfer closed with 1789333843 bytes remaining to read

Like Veerapath mentioned, you need to contact the firewall admin to verify the connection and settings.

To remove the netsh port forwarding, I used the following command.

C:\netsh interface portproxy delete v4tov4 listenport=443 listenaddress=127.0.0.1

Ayan

We will use -C option in Curl to continue download if "Connection Close" occurs.