René's URL Explorer Experiment


Title: Add storage.cloud documentation and example scripts by Sazwanismail · Pull Request #241 · Visual-Studio-Code/.github · GitHub

Open Graph Title: Add storage.cloud documentation and example scripts by Sazwanismail · Pull Request #241 · Visual-Studio-Code/.github

X Title: Add storage.cloud documentation and example scripts by Sazwanismail · Pull Request #241 · Visual-Studio-Code/.github

Description: Added documentation and example scripts for Google Cloud Storage usage, including quickstart guides, merging data, and CORS configuration.````markdown name=README.md # storage.cloud — Google Cloud Storage (GCS) docs & quickstart storage.cloud is a compact documentation and example repo with copy-pastable commands and small scripts for common Google Cloud Storage tasks: Authentication (gcloud ADC, service accounts, access tokens) Browser access vs programmatic access (storage.cloud.google.com vs API) Signed URLs and sharing CORS configuration for browser clients Practical patterns to merge many objects (CSV/text) in GCS Load patterns for BigQuery and recommendations for large datasets Repository layout index.html — simple landing page / site overview docs/ quickstart.md — commands & short how-tos merge-data.md — Malay quick guide for combining files signed-urls.md — signing approaches and examples examples/ merge_csv_gcs.py — small Python script to merge CSVs in a GCS prefix cors.json — CORS example LICENSE — suggested license (MIT) Quickstart highlights Authenticate locally (ADC): gcloud auth application-default login Get an access token for Authorization header: gcloud auth print-access-token List buckets: gsutil ls gs:// Copy to/from GCS: gsutil cp ./local-file gs://BUCKET/OBJECT gsutil cp gs://BUCKET/OBJECT ./local-file Create a signed URL: gsutil signurl -d 1h /path/to/key.json gs://BUCKET/OBJECT Security reminders Use service accounts with least privilege in production. Signed URLs grant access to anyone with the URL until expiry. Avoid making sensitive objects public; prefer signed URLs or IAM roles. Contributing Send PRs or issues with improvements. Keep examples minimal and targeted. Include tests or usage examples when adding scripts. If you want, I can: Expand any doc into a full tutorial (step-by-step with screenshots). Add a shell script to compose >32 objects safely. Create a Dataflow/Beam starter pipeline for large-scale merges. ```html name=index.html storage.cloud — Google Cloud Storage quickstart

storage.cloud

Quick reference and starter content for Google Cloud Storage (GCS)

Overview

storage.cloud collects short guides, command snippets, and small reusable scripts for common GCS workflows:

  • Authenticated access (gcloud, service accounts, access tokens)
  • Signed URLs for temporary public access
  • CORS configuration for browser-based apps
  • Merging objects (CSV/text) and loading to BigQuery

Helpful links

Get started

Open docs/quickstart.md for commands and brief examples you can run locally. For combining files, see docs/merge-data.md and examples/merge_csv_gcs.py.

``` ````markdown name=docs/quickstart.md # Quickstart — Google Cloud Storage (GCS) This page collects the most-used commands and short examples for getting started with GCS. Prerequisites - Install Google Cloud SDK (gcloud, gsutil) - For programmatic examples, install the relevant client libraries (Python: google-cloud-storage) Authentication - Local developer (Application Default Credentials - ADC): gcloud auth application-default login - Service account (recommended for server apps): 1. Create: gcloud iam service-accounts create my-sa --display-name="My SA" 2. Grant roles (example: objectViewer): gcloud projects add-iam-policy-binding PROJECT_ID \ --member="serviceAccount:my-sa@PROJECT_ID.iam.gserviceaccount.com" \ --role="roles/storage.objectViewer" 3. (Optional) Create key for local use: gcloud iam service-accounts keys create key.json \ --iam-account=my-sa@PROJECT_ID.iam.gserviceaccount.com Access tokens - Print an access token for HTTP Authorization: gcloud auth print-access-token - Use as header: Authorization: Bearer Common gsutil commands - List buckets: gsutil ls gs:// - List objects in a bucket/prefix: gsutil ls gs://BUCKET/PREFIX/ - Download object: gsutil cp gs://BUCKET/OBJECT ./local-file - Upload file: gsutil cp ./local-file gs://BUCKET/OBJECT - Make object public (use sparingly): gsutil acl ch -u AllUsers:R gs://BUCKET/OBJECT Signed URLs - Short way to create an expiring URL using a service account key: gsutil signurl -d 1h /path/to/key.json gs://BUCKET/OBJECT - Notes: - V4 signed URLs are supported; maximum expiry is 7 days. - Signed URLs allow access without a Google account. CORS (for browser clients) - Example file: cors.json (in repo) - Apply: gsutil cors set cors.json gs://BUCKET BigQuery ingestion - BigQuery accepts wildcards — you can load many CSVs without pre-merging: bq load --autodetect --source_format=CSV dataset.table gs://BUCKET/PATH/*.csv Troubleshooting - Permission denied: check IAM roles (roles/storage.objectViewer or a custom role). - Invalid credentials: re-run `gcloud auth application-default login` or refresh service account keys. - CORS errors: ensure bucket CORS includes your domain and required methods/headers. Security recommendations - Use service accounts with least privilege. - Prefer IAM + uniform bucket-level access over ACLs. - Do not embed long-lived keys in client-side code; use signed URLs for browser access. Further reading - GCS docs: https://cloud.google.com/storage/docs - Signed URLs: https://cloud.google.com/storage/docs/access-control/signed-urls - gsutil reference: https://cloud.google.com/storage/docs/gsutil # Gabungkan Semua Data (Combine all data) Dokumen ringkas (Bahasa Melayu) untuk menggabungkan fail yang tersimpan di Google Cloud Storage. Sebelum mula - Pastikan anda mempunyai akses ke bucket (roles/storage.objectViewer atau storage.objectAdmin untuk penulisan). - Jika dataset besar (GB/TB), pertimbangkan Dataflow/Dataproc atau import terus ke BigQuery. Pilihan umum 1) Quick merge (fail kecil/sederhana) - Jika saiz kecil supaya muat dalam memori: gsutil cat gs://BUCKET/PATH/*.csv | gsutil cp - gs://BUCKET/PATH/combined.csv - Risiko: memori dan rangkaian. Pastikan hanya digunakan untuk set kecil. 2) gsutil compose (gabungkan objek di GCS tanpa muat turun) - Gabungkan hingga 32 objek per operasi: gsutil compose gs://BUCKET/part1.csv gs://BUCKET/part2.csv gs://BUCKET/combined.csv - Untuk >32 objek, buat compose berperingkat (tree compose). - Nota: compose menggabungkan byte; pastikan setiap objek mempunyai newline yang sesuai dan tidak menggabungkan header berganda. 3) Load terus ke BigQuery (disarankan untuk analisis) - BigQuery boleh baca wildcard CSV: bq load --autodetect --source_format=CSV dataset.table gs://BUCKET/PATH/*.csv - Kelebihan: skala, tiada pra-gabungan, schema handling. 4) Skrip tersuai (contoh Python) - Kelebihan: kawalan penuh (buang header berganda, normalisasi). - Lihat `examples/merge_csv_gcs.py` untuk contoh. Contoh strategi untuk >32 objek menggunakan gsutil compose (pattern) - Bahagikan senarai objek kepada kumpulan 32, compose setiap kumpulan ke objek sementara, kemudian compose hasil sementara (ulang sehingga satu objek terakhir). - Atau gunakan pipeline Dataflow untuk penstriman dan penulisan semula. Amalan terbaik - Untuk output akhir yang akan dianalitik: lebih baik tulis ke BigQuery atau Parquet (kolumnar) untuk kos dan prestasi. - Pastikan logging dan versi objek dihidupkan jika data penting (object versioning). - Gunakan encryption keys (CMEK) jika diperlukan. Contoh ringkas penggunaan skrip - Jika anda mahu, berikan: - Bucket name (contoh: gs://my-bucket/data/) - Prefix dan jenis fail (.csv) - Saiz anggaran (GB/TB) Saya akan hasilkan skrip automatik yang sesuai (bash atau Python) untuk anda. # Signed URLs — Quick reference Signed URLs grant temporary access to an object without requiring the requester to have a Google account. Key points - Signed URLs include authentication in query parameters. - Anyone with the URL can access the object until expiry. - Maximum expiry for V4 signed URLs is 7 days. Create with gsutil (using service account key file) gsutil signurl -d 1h /path/to/key.json gs://BUCKET/OBJECT Create programmatically (Python example sketch) - Use google-auth library to sign and generate a V4 URL, or use google-cloud-storage's blob.generate_signed_url method. - Prefer service account credentials with least privilege. Security tips - Monitor signed-URL usage where possible (Cloud Storage logs). - Shorter expiry reduces risk. - Do not embed long-lived private keys in public repositories. When to use - Temporary downloads for users without Google accounts. - Browser uploads (PUT) when combined with appropriate CORS settings. #!/usr/bin/env python3 """ examples/merge_csv_gcs.py Merge CSV files in a GCS prefix into one CSV while keeping only the first header. Requirements: pip install google-cloud-storage Usage: export GOOGLE_APPLICATION_CREDENTIALS="/path/to/key.json" python3 examples/merge_csv_gcs.py my-bucket data/prefix/ output/combined.csv """ import sys import csv from io import StringIO from google.cloud import storage def merge_csvs(bucket_name, prefix, output_blob_name): client = storage.Client() bucket = client.bucket(bucket_name) blobs = sorted([b for b in bucket.list_blobs(prefix=prefix) if b.name.endswith('.csv')]) if not blobs: print("No CSV files found with prefix:", prefix) return out_buf = StringIO() writer = None first = True for blob in blobs: print("Reading:", blob.name) # Use streaming download if files are large: download_as_text() may load entire object data = blob.download_as_text() reader = csv.reader(StringIO(data)) header = next(reader, None) if first: writer = csv.writer(out_buf) if header: writer.writerow(header) for row in reader: writer.writerow(row) first = False else: # skip header for subsequent files for row in reader: writer.writerow(row) out_blob = bucket.blob(output_blob_name) out_blob.upload_from_string(out_buf.getvalue(), content_type='text/csv') print(f'Uploaded gs://{bucket_name}/{output_blob_name}') if __name__ == "__main__": if len(sys.argv) != 4: print("Usage: merge_csv_gcs.py BUCKET PREFIX OUTPUT_BLOB") print("Example: merge_csv_gcs.py my-bucket data/ output/combined.csv") sys.exit(1) merge_csvs(sys.argv[1], sys.argv[2], sys.argv[3]) [ { "origin": ["https://example.com"], "method": ["GET", "HEAD", "PUT", "POST"], "responseHeader": ["Content-Type", "x-goog-meta-custom"], "maxAgeSeconds": 3600 } ] MIT License Copyright (c) 2025 Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction... (You can replace with your full MIT text or preferred license.) Would you like me to: generate a shell helper to compose >32 objects safely (tree compose), or produce a Dataflow/Beam starter for large merges, or create a translated Malay full-site version? Tell me which follow-up and I will produce the exact script or pipeline.

Open Graph Description: Added documentation and example scripts for Google Cloud Storage usage, including quickstart guides, merging data, and CORS configuration.````markdown name=README.md # storage.cloud — Google Cloud ...

X Description: Added documentation and example scripts for Google Cloud Storage usage, including quickstart guides, merging data, and CORS configuration.````markdown name=README.md # storage.cloud — Google Cloud ...

Opengraph URL: https://github.com/Visual-Studio-Code/.github/pull/241

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:4b057c21-5434-cd7a-c817-c0f9e9a9ce45
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-id9F02:2652DC:CD7B0B:1099445:6B00F070
html-safe-noncef8c7e164ccdca89de5a83651d39b69593384f60bd376de8d53a3e2c159e6931e
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5RjAyOjI2NTJEQzpDRDdCMEI6MTA5OTQ0NTo2QjAwRjA3MCIsInZpc2l0b3JfaWQiOiI3MDIyMzczNzI0MTg0MjA4NDgiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ==
visitor-hmac80d0d325a7642a582ae08d746e110f916cf6c11c21f7a84165f2b188e6a40cd3
hovercard-subject-tagpull_request:3448084127
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/Visual-Studio-Code/.github/pull/241/files
twitter:imagehttps://avatars.githubusercontent.com/u/208310745?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/208310745?s=400&v=4
og:image:altAdded documentation and example scripts for Google Cloud Storage usage, including quickstart guides, merging data, and CORS configuration.````markdown name=README.md # storage.cloud — Google Cloud ...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
None4ef498cbabcbe3c50befc7a5b743e7072ee770b4daf0e15e54801180d1fa91fc
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/Visual-Studio-Code/.github git https://github.com/Visual-Studio-Code/.github.git
octolytics-dimension-user_id61069792
octolytics-dimension-user_loginVisual-Studio-Code
octolytics-dimension-repository_id550489658
octolytics-dimension-repository_nwoVisual-Studio-Code/.github
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id550489658
octolytics-dimension-repository_network_root_nwoVisual-Studio-Code/.github
turbo-body-classeslogged-out env-production page-responsive
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release9983dd5da06943f833c971dd5b6bc988e2858be5
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/Visual-Studio-Code/.github/pull/241/files#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2FVisual-Studio-Code%2F.github%2Fpull%2F241%2Ffiles
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2FVisual-Studio-Code%2F.github%2Fpull%2F241%2Ffiles
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=Visual-Studio-Code%2F.github
Reloadhttps://github.com/Visual-Studio-Code/.github/pull/241/files
Reloadhttps://github.com/Visual-Studio-Code/.github/pull/241/files
Reloadhttps://github.com/Visual-Studio-Code/.github/pull/241/files
Visual-Studio-Codehttps://github.com/Visual-Studio-Code
.githubhttps://github.com/Visual-Studio-Code/.github
Codehttps://github.com/Visual-Studio-Code/.github
Issues119 (119)https://github.com/Visual-Studio-Code/.github/issues
Pull requests103 (103)https://github.com/Visual-Studio-Code/.github/pulls
Actionshttps://github.com/Visual-Studio-Code/.github/actions
Projectshttps://github.com/Visual-Studio-Code/.github/projects
Security and qualityhttps://github.com/Visual-Studio-Code/.github/security
Insightshttps://github.com/Visual-Studio-Code/.github/insights
Sign up for GitHub https://github.com/signup?return_to=%2FVisual-Studio-Code%2F.github%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2FVisual-Studio-Code%2F.github%2Fissues%2Fnew%2Fchoose
Muhamad Sazwan Bin Ismail (Sazwanismail)https://github.com/Sazwanismail
Visual-Studio-Code:mainhttps://github.com/Visual-Studio-Code/.github/tree/main
Htpps-console-firebase-google-com:Sazwanismail-patch-1https://github.com/Htpps-console-firebase-google-com/.github/tree/Sazwanismail-patch-1
Conversation 0 https://github.com/Visual-Studio-Code/.github/pull/241
Commits 1 https://github.com/Visual-Studio-Code/.github/pull/241/commits
Checks 0 https://github.com/Visual-Studio-Code/.github/pull/241/checks
Files changed https://github.com/Visual-Studio-Code/.github/pull/241/files
Please reload this pagehttps://github.com/Visual-Studio-Code/.github/pull/241/files
Add storage.cloud documentation and example scripts https://github.com/Visual-Studio-Code/.github/pull/241/files#top
Show all changes 1 commit https://github.com/Visual-Studio-Code/.github/pull/241/files
84825be Add storage.cloud documentation and example scripts Sazwanismail Nov 6, 2025 https://github.com/Visual-Studio-Code/.github/pull/241/commits/84825bee15337958d5942be895501e336c57e95a
Clear filters https://github.com/Visual-Studio-Code/.github/pull/241/files
Please reload this pagehttps://github.com/Visual-Studio-Code/.github/pull/241/files
Please reload this pagehttps://github.com/Visual-Studio-Code/.github/pull/241/files
storage.cloudhttps://github.com/Visual-Studio-Code/.github/pull/241/files#diff-cff3e873f467c86f5bc30712d9fb7e803dfdfff12b250c5e12c0b5668e133935
View file https://github.com/Visual-Studio-Code/.github/blob/84825bee15337958d5942be895501e336c57e95a/storage.cloud
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/Visual-Studio-Code/.github/pull/241/{{ revealButtonHref }}
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.