How to Upload Files to Facebook Zip File
Uploading and Downloading Zip Files In GCP Cloud Storage Using Python
GCP (Google Cloud Platform) cloud storage is the object storage service provided by Google for storing many data formats from PNG files to zipped source code for web apps and cloud functions. The data is stored in a flat, key/value-like data structure where the key is your storage object's proper noun and the value is your data.
Object storage is great for storing massive amounts of data every bit a unmarried entity, data that will later be accessed all at once as opposed to data that will be read and written in small subsets as is the case with relational and non-relational databases.
If you're looking to store a collection of files as a single unit, either to annal a large number of log files for time to come audits or to parcel and shop code as a office of an automated deployment cycle, it's likely you volition do so by packing all of information technology together as a zip file.
Using an application to automate the process of creating, altering, or unzipping a zilch file in memory is a useful skill to have however working with retention streams and bytes rather than integers, strings, and objects can be daunting when it is unfamiliar territory.
Whether yous are specifically looking to upload and download nada files to GCP cloud storage or you merely have an interest in learning how to piece of work with zip files in memory, this post volition walk you through the process of creating a new zip file from files on your local machine and uploading them to cloud storage as well as downloading an existing zip file in deject storage and unzipping it to a local directory.
Establishing Credentials
Before y'all tin can begin uploading and downloading local files to cloud storage every bit zip files, you lot volition need to create the customer object used in your Python code to communicate with your project's cloud storage resources in GCP.
There are various ways to establish credentials that will grant the client object access to a cloud storage saucepan, the most common of which is to create a service account and assign information technology to your application in ane of two ways.
The starting time selection is to assign the service business relationship to a detail resource upon deployment. For example, if your code is existence deployed as a GCP cloud part, you lot would attach the service business relationship to the application upon deployment using either the gcloud sdk:
# using powershell and the gcloud sdk to deploy a python deject function gcloud functions deploy my-cloud-function ` --entry-bespeak my_function_name ` --runtime python38 ` --service-account my-cloud-function @ my-project-id.iam.gserviceaccount.com ` --trigger-http
Or by using an IAC (infrastructure as lawmaking) solution similar Terraform:
resource "google_service_account" "my_cloud_func_sa" { account_id = "my-cloud-function" display_name = "Cloud Function Service Account" } resource "google_project_iam_binding" "cloud_storage_user" { project = "my-project-id" role = "roles/storage.objectAdmin" members = [ "serviceAccount: ${ google_service_account . my_cloud_func_sa . electronic mail } " , ] } resource "google_cloud_functions_function" "my_cloud_func" { name = "my-cloud-function" entry_point = "my_function_name" runtime = "python38" service_account_email = google_service_account . my_cloud_func_sa . email trigger_http = true }
Note that the service account as defined in Terraform is besides being referenced in a google_project_iam_binding resource as a member that will be assigned the role of storage.objectAdmin. Yous will need to assign a like role (or ideally one with the minimal permissions required for your lawmaking to perform its tasks) if you lot choose to create a service account using the GCP panel.
For lawmaking beingness deployed with an assigned service business relationship, creating the GCP cloud storage client in Python requires only the project id be passed as an statement to the customer constructor.
from google.cloud import storage client = storage . Client ( project = GCP_PROJECT_ID , )
All the same if yous would like to upload and download to cloud storage using a CLI application or to exam your cloud office before deploying it, yous volition want to use a locally stored JSON credentials file.
To create the file, open the GCP console and select IAM & Admin from the Navigation carte du jour, accessed through the hamburger carte icon in the superlative left corner.
From the IAM & Admin menu, select the Service Accounts page and either create a new service account or click on the link of an existing one, plant under the Electronic mail column of the service accounts table.
At the bottom of the Details folio for the selected service account, click Add Key > Create New Key and select the JSON pick.
This will download the JSON credentials file to your car.
Anyone with access to this file will have the credentials necessary to brand changes to your deject resources according to the permissions of this service account. Store it in a secure place and do not bank check this file into source control. If you do, immediately delete the central from the same bill of fare used to create it and remove the JSON file from source control.
To allow your client object to apply these credentials and access GCP deject storage, initializing the customer will require a few extra steps. You will demand to create a credentials object using the from_service_account_file method on the service_account.Credentials form of the google.oauth2 library. The just required argument for this method is the absolute or relative file path to your JSON credentials file.
This credentials object volition be passed as a second argument to the storage.Client class constructor.
from google.cloud import storage from google.oauth2 import service_account credentials = service_account . Credentials . from_service_account_file ( SERVICE_ACCOUNT_FILE ) client = storage . Client ( project = GCP_PROJECT_ID , credentials = credentials )
Uploading Local Files to Cloud Storage every bit a Nix File
At present that your client object has the required permissions to access cloud storage you can brainstorm uploading local files as a zip file.
Assuming that the files y'all intend to upload are all in the aforementioned directory and are not already zipped, y'all will upload the files to GCP deject storage as a zip file by creating a zip archive in retentivity and uploading it every bit bytes.
from google.deject import storage from zipfile import ZipFile , ZipInfo def upload (): source_dir = pathlib . Path ( SOURCE_DIRECTORY ) archive = io . BytesIO () with ZipFile ( annal , 'w' ) as zip_archive : for file_path in source_dir . iterdir (): with open ( file_path , 'r' ) as file : zip_entry_name = file_path . proper noun zip_file = ZipInfo ( zip_entry_name ) zip_archive . writestr ( zip_file , file . read ()) annal . seek ( 0 ) object_name = 'super-of import-data-v1' bucket = customer . bucket ( BUCKET_NAME ) blob = storage . Blob ( object_name , saucepan ) blob . upload_from_file ( archive , content_type = 'application/nil' )
io.BytesIO() creates an in retentiveness binary stream used past the ZipFile object to store all the data from your local files equally bytes.
The files in the source directory are iterated over and for each one a ZipInfo object is created and written to the ZipFile object along with the contents of the source file. The ZipInfo object corresponds to an private file entry inside a zip file and will be labeled with whatever file name and extension yous use in the constructor to instantiate the ZipInfo object. Using zip_entry_name = file_path.name every bit in the example above will gear up the file proper noun and extension in the zip file to lucifer the name and extension of the local file.
The in memory binary stream (the archive variable) is what you will be uploading to GCP deject storage, yet a prerequisite for uploading an in memory stream is that the stream position be set to the starting time of the stream. Without moving the position of the stream back to cypher with archive.seek(0) yous will get an mistake from the Google API when you endeavor to upload the data.
With the in retentiveness binary stream set to exist delivered, the remaining lines of code create a new Bucket object for the specified bucket and a Blob object for the storage object. The zipped files are so uploaded to cloud storage and tin later retrieved using the storage object proper noun you used to create the Hulk instance.
A saucepan in cloud storage is a user divers partition for the logical separation of data and a hulk (equally the Python grade is called) is another name for a storage object.
Downloading a Zip File Hulk in Deject Storage to a Local Directory
To download a nil file storage object and unzip it into a local directory, you will need to reverse the process past first creating a bucket object and a blob object in order to download the zip file equally bytes.
def download (): target_dir = pathlib . Path ( TARGET_DIRECTORY ) object_name = 'super-important-information-v1' bucket = client . bucket ( BUCKET_NAME ) blob = storage . Blob ( object_name , bucket ) object_bytes = blob . download_as_bytes () archive = io . BytesIO () archive . write ( object_bytes ) with ZipFile ( annal , 'due west' ) as zip_archive : zip_archive . extractall ( target_dir )
In one case downloaded, the bytes tin be written to an in retentiveness stream which will in plough be used to create a ZipFile object in order to extract the files to your target directory. io.BytesIO() is once more used to create the in memory binary stream and the write method on the BytesIO object is used to write the downloaded bytes to the stream. The ZipFile object has a method for extracting all of its contents to a specified directory, making the concluding pace a elementary one.
With these two functions and the appropriate credentials y'all should take everything yous need to start uploading and downloading your own zip files into cloud storage using Python.
And if you lot'd like to see all the Python lawmaking in one place, you lot can find it here every bit a Gist on my Github account.
Source: https://dev.to/jakewitcher/uploading-and-downloading-zip-files-in-gcp-cloud-storage-using-python-2l1b
Post a Comment for "How to Upload Files to Facebook Zip File"