How to handle a download link / dropbox
Requested datasets can be quite large, depending on the type of experiment.
Given a download link, one can use the tool cURL to download a file through the command line interface instead. Another tool is wget.
Warning
Always ensure you understand the commands you execute to avoid unintended behaviour.
Use the manual - man curl - or use for example tools like explainshell
First, click on the desired dataset.
You will be presented with additional support (see How-to for different platforms with typical tools). Make sure to replace the values in curly brackets ({}). These should typically be replaced with the location on your system.
Raw URL: shows the download link if you'd need it for downloading it in the way you prefer. An additional button is present to download it straight to your current system within your browser session.Linux&Windows: these are the download commands you can copy and paste in your terminal.
Next, go to the command line interface of your machine and follow the steps for your platform with the tool of choice. Additional tools for Windows are listed in the next section.
Finally, please check the Troubleshooting section if you encounter any issues.
How-to for different platforms with typical tools
Please replace the values in curly brackets {} (also remove the brackets)
# curl, md5 and unzip should be preinstalled
# Downloading
curl -o {/My/Location/some_file.zip} '{DOWNLOAD_LINK}'
# Verify MD5 hash to ensure no corruption occured
md5 {/My/Locaction/some_file.zip}
# Extracting
unzip {/My/Location/some_file.zip}
If not on your machine, please install through for example homebrew
# md5 and unzip should be preinstalled
# Downloading
wget -O {/My/Location/some_file.zip} '{DOWNLOAD_LINK}'
# Verify MD5 hash to ensure no corruption occured
md5 {/My/Locaction/some_file.zip}
# Extracting
unzip {/My/Location/some_file.zip}
# Update package sources list
apt update
# Install curl
apt -y install curl
# Install zip and unzip
apt install zip unzip
# Downloading
curl -o {/My/Location/some_file.zip} '{DOWNLOAD_LINK}'
# Verify MD5 hash to ensure no corruption occured
md5sum {/My/Location/some_file.zip}
# Extracting
unzip {/My/Location/some_file.zip}
# Update package sources list
apt update
# Install curl
apt-get install wget
# Install zip and unzip
apt install zip unzip
# Downloading
wget -O {/My/Location/some_file.zip} '{DOWNLOAD_LINK}'
# Verify MD5 hash to ensure no corruption occured
md5sum {/My/Location/some_file.zip}
# Extracting
unzip {/My/Location/some_file.zip}
If not on your machine, please install through the official curl website
# Verify
Get-Command curl.exe
# Downloadling
curl.exe -o {C:\My\Location\some_file.zip} '{DOWNLOAD_LINK}'
# Verify MD5 hash to ensure no corruption occured
Get-FileHash {C:\My\Location\some_file.zip} -Algorithm MD5
# Extracting
Expand-Archive {C:\My\Location\some_file.zip} -DestinationPath {C:\My\Location\Unzipedfolder} -Verbose
# Source file location
$source = '{DOWNLOAD_LINK}'
# Destination to save the file
$destination = '{C:\My\Location\some_file.zip}'
# Downloading
Invoke-WebRequest -Uri $source -OutFile $destination
# Verify MD5 hash to ensure no corruption occured
Get-FileHash {C:\My\Location\some_file.zip} -Algorithm MD5
# Extracting
Expand-Archive {C:\My\Location\some_file.zip} -DestinationPath {C:\My\Location\Unzipedfolder} -Verbose
# Source file location
$source = '{DOWNLOAD_LINK}'
# Destination to save the file
$destination = '{C:\My\Location\some_file.zip}'
# Downloading
Start-BitsTransfer -Source $source -Destination $destination
# Verify MD5 hash to ensure no corruption occured
Get-FileHash {C:\My\Location\some_file.zip} -Algorithm MD5
# Extracting
Expand-Archive {C:\My\Location\some_file.zip} -DestinationPath {C:\My\Location\Unzipedfolder} -Verbose
# Load the 7zip package
module load 7zip
# Downloading
curl -o {/My/Location/some_file.zip} '{DOWNLOAD_LINK}'
# Verify MD5 hash to ensure no corruption occured
md5sum {/My/Location/some_file.zip}
# Extracting
7z x {/My/Location/some_file.zip}
# Load the 7zip package
module load 7zip
# Downloading
wget -O {/My/Location/some_file.zip} '{DOWNLOAD_LINK}'
# Verify MD5 hash to ensure no corruption occured
md5sum {/My/Location/some_file.zip}
# Extracting
7z x {/My/Location/some_file.zip}
Troubleshooting
Resume interrupted download
Some commands have options to resume the download from where it was interrupted. Here's how to resume.
curl -o {/My/Location/some_file.zip} -C - '{DOWNLOAD_LINK}'
wget -c -O {/My/Location/some_file.zip} '{DOWNLOAD_LINK}'
Invoke-WebRequest -Uri '{DOWNLOAD_LINK}' -OutFile '{C:\My\Location\some_file.zip}' -Resume
Info
The tool screen enables you to create a terminal session on your server that is shielded from your other terminal operations. This is particulary useful when you connect to your server over ssh. If you logout, your ssh session ends. However, by downloading in a screen session on your server, you can safely disconnect from your server while your download operation remains.
Downloadling does not work
Make sure to use the correct code and symbols. Do not omit the single ticks ' around the download url.
403 Forbidden
If you have an error of the likes
HTTP request sent, awaiting response... 403 Forbidden
2023-02-20 13:35:31 ERROR 403: Forbidden.
This can mean that you:
- did not use the single ticks around the download link
- have the wrong download link
- accidentally added additional characters to the download link
- accidentally added leading or trailing whitespace to the link
- accidentally removed characters from the download link
- are trying to download an expired link.
If you have verified any of these issues and your issue is not resolved, please contact info@genomicscore.be
Overlap / Potential ZIP bomb
In some cases, an error will show up of the format
error: invalid zip file with overlapped components (possible zip bomb)
If the MD5 sum checks out, there is no reason to suspect a zip bomb and it is safe to continue. This can be done with for example 7-zip:
7z x {some_file.zip}
Installing on the VSC (Vlaamse Super Computer)
See Using the module system
module load 7zip