Automate Cloud File Transfers with Google Colab

This Notebook script enables you to transfer files between different cloud services without downloading to your PC, saving on internet charges 😃.
I often faced the challenge of downloading files from one cloud to my local PC and then re-uploading them to another Cloud. This becomes a nightmare with slow internet speeds with Huge Files. Online tools that offer similar services charge exorbitant fees. So, I created this solution to automate the process and maximize internet speed.

Let’s begin the journey

1. Setting up the Environment

First of all, let’s make sure we have the necessary libraries installed. We’ll need the requests library for making HTTP requests and python-magic for detecting file types.

!pip install requests
!pip install python-magic

2. Writing the Colab Python Script

Our Python script will use the requests library to interact with cloud services. We'll define a function get_content_type to fetch the content type of a file from its URL.

# Importing necessary libraries
import requests
from requests.auth import HTTPBasicAuth
from tqdm import tqdm
import os
from urllib.parse import urlparse
from datetime import datetime
import magic

# Function to get content type
def get_content_type(url):
    try:
        response = requests.head(url)
        response.raise_for_status()
        content_type = response.headers.get('content-type')
        return content_type.split(';')[0] if content_type else None
    except requests.exceptions.RequestException as e:
        print(f"Failed to fetch content type. Error: {e}")
        return None
# Define the URLs and credentials
download_url = input("Enter the URL of the file you want to download: ")
upload_url = "" // upload WebDav URL
username = ""   //Upload cloud username
password = ""   //Upload cloud password
chunk_size = 1024 * 1024  # 1 MB
# Check if the WebDAV server is reachable
try:
    response = requests.head(upload_url, auth=HTTPBasicAuth(username, password))
    response.raise_for_status()
    print("WebDAV server is reachable. Starting upload...")
except requests.exceptions.RequestException as e:
    print(f"Failed to connect to WebDAV server. Error: {e}")
    exit(1)
# Get the content type of the file
content_type = get_content_type(download_url)
if not content_type:
    print("Failed to determine content type. Please check the URL.")
    exit(1)
# Download the file in chunks and display progress
temp_file_path = "temp_file"
with requests.get(download_url, stream=True) as download_response:
    total_size = int(download_response.headers.get('content-length', 0))
    with tqdm(total=total_size, unit='B', unit_scale=True, desc='Downloading file', ascii=True) as pbar:
        with open(temp_file_path, 'wb') as temp_file:
            for chunk in download_response.iter_content(chunk_size=chunk_size):
                if chunk:
                    temp_file.write(chunk)
                    pbar.update(len(chunk))
# Use python-magic to detect the file type
mime = magic.Magic(mime=True)
file_type = mime.from_file(temp_file_path)
# Modify the upload URL to include a timestamp for uniqueness
timestamp = datetime.now().strftime("%Y%m%d%H%M%S")
upload_url = f"{upload_url}file_{timestamp}.{file_type.split('/')[1]}"
# Upload the file in chunks and display progress
with open(temp_file_path, 'rb') as temp_file:
    with tqdm(total=os.path.getsize(temp_file_path), unit='B', unit_scale=True, desc='Uploading file', ascii=True) as pbar:
        temp_file.seek(0)  # Seek back to the beginning of the file
        with requests.put(
            upload_url,
            auth=HTTPBasicAuth(username, password),
            data=temp_file,
            headers={"Content-Type": content_type},
        ) as upload_response:
            if upload_response.status_code == 201:
                print("File uploaded successfully")
            else:
                print(f"Failed to upload file. Status code: {upload_response.status_code}")
# Remove the temporary file
os.remove(temp_file_path)

Conclusion

The Cloud Transfer Tool is a tool for saving time and avoiding unnecessary internet charges. However, it’s important to use this tool responsibly and ethically. Avoid using it for transferring copyrighted or unethical content. Always ensure that your use of the tool complies with the policies of the cloud services and platforms you are interacting with. Additionally, familiarize yourself with Google Colab’s policies and guidelines to ensure that your usage complies. By using the Cloud Transfer Tool responsibly, you can make the most of its benefits while staying in line with ethical and legal standards.

Resources

GitHub Link — https://github.com/madhurajayashanka/Cloud-Transfer-Script