René's URL Explorer Experiment
Title: Add storage.cloud documentation and example scripts by Sazwanismail · Pull Request #241 · Visual-Studio-Code/.github · GitHub
Open Graph Title: Add storage.cloud documentation and example scripts by Sazwanismail · Pull Request #241 · Visual-Studio-Code/.github
X Title: Add storage.cloud documentation and example scripts by Sazwanismail · Pull Request #241 · Visual-Studio-Code/.github
Description: Added documentation and example scripts for Google Cloud Storage usage, including quickstart guides, merging data, and CORS configuration.````markdown name=README.md # storage.cloud — Google Cloud Storage (GCS) docs & quickstart
storage.cloud is a compact documentation and example repo with copy-pastable commands and small scripts for common Google Cloud Storage tasks:
Authentication (gcloud ADC, service accounts, access tokens)
Browser access vs programmatic access (storage.cloud.google.com vs API)
Signed URLs and sharing
CORS configuration for browser clients
Practical patterns to merge many objects (CSV/text) in GCS
Load patterns for BigQuery and recommendations for large datasets
Repository layout
index.html — simple landing page / site overview
docs/
quickstart.md — commands & short how-tos
merge-data.md — Malay quick guide for combining files
signed-urls.md — signing approaches and examples
examples/
merge_csv_gcs.py — small Python script to merge CSVs in a GCS prefix
cors.json — CORS example
LICENSE — suggested license (MIT)
Quickstart highlights
Authenticate locally (ADC): gcloud auth application-default login
Get an access token for Authorization header: gcloud auth print-access-token
List buckets: gsutil ls gs://
Copy to/from GCS: gsutil cp ./local-file gs://BUCKET/OBJECT gsutil cp gs://BUCKET/OBJECT ./local-file
Create a signed URL: gsutil signurl -d 1h /path/to/key.json gs://BUCKET/OBJECT
Security reminders
Use service accounts with least privilege in production.
Signed URLs grant access to anyone with the URL until expiry.
Avoid making sensitive objects public; prefer signed URLs or IAM roles.
Contributing
Send PRs or issues with improvements.
Keep examples minimal and targeted.
Include tests or usage examples when adding scripts.
If you want, I can:
Expand any doc into a full tutorial (step-by-step with screenshots).
Add a shell script to compose >32 objects safely.
Create a Dataflow/Beam starter pipeline for large-scale merges.
```html name=index.html
storage.cloud — Google Cloud Storage quickstart
Overview
storage.cloud collects short guides, command snippets, and small reusable scripts for common GCS workflows:
- Authenticated access (gcloud, service accounts, access tokens)
- Signed URLs for temporary public access
- CORS configuration for browser-based apps
- Merging objects (CSV/text) and loading to BigQuery
Get started
Open docs/quickstart.md for commands and brief examples you can run locally. For combining files, see docs/merge-data.md and examples/merge_csv_gcs.py.
```
````markdown name=docs/quickstart.md
# Quickstart — Google Cloud Storage (GCS)
This page collects the most-used commands and short examples for getting started with GCS.
Prerequisites
- Install Google Cloud SDK (gcloud, gsutil)
- For programmatic examples, install the relevant client libraries (Python: google-cloud-storage)
Authentication
- Local developer (Application Default Credentials - ADC): gcloud auth application-default login
- Service account (recommended for server apps):
1. Create: gcloud iam service-accounts create my-sa --display-name="My SA"
2. Grant roles (example: objectViewer): gcloud projects add-iam-policy-binding PROJECT_ID \ --member="serviceAccount:my-sa@PROJECT_ID.iam.gserviceaccount.com" \ --role="roles/storage.objectViewer"
3. (Optional) Create key for local use: gcloud iam service-accounts keys create key.json \ --iam-account=my-sa@PROJECT_ID.iam.gserviceaccount.com
Access tokens
- Print an access token for HTTP Authorization: gcloud auth print-access-token
- Use as header: Authorization: Bearer
Common gsutil commands
- List buckets: gsutil ls gs://
- List objects in a bucket/prefix: gsutil ls gs://BUCKET/PREFIX/
- Download object: gsutil cp gs://BUCKET/OBJECT ./local-file
- Upload file: gsutil cp ./local-file gs://BUCKET/OBJECT
- Make object public (use sparingly): gsutil acl ch -u AllUsers:R gs://BUCKET/OBJECT
Signed URLs
- Short way to create an expiring URL using a service account key: gsutil signurl -d 1h /path/to/key.json gs://BUCKET/OBJECT
- Notes:
- V4 signed URLs are supported; maximum expiry is 7 days.
- Signed URLs allow access without a Google account.
CORS (for browser clients)
- Example file: cors.json (in repo)
- Apply: gsutil cors set cors.json gs://BUCKET
BigQuery ingestion
- BigQuery accepts wildcards — you can load many CSVs without pre-merging: bq load --autodetect --source_format=CSV dataset.table gs://BUCKET/PATH/*.csv
Troubleshooting
- Permission denied: check IAM roles (roles/storage.objectViewer or a custom role).
- Invalid credentials: re-run `gcloud auth application-default login` or refresh service account keys.
- CORS errors: ensure bucket CORS includes your domain and required methods/headers.
Security recommendations
- Use service accounts with least privilege.
- Prefer IAM + uniform bucket-level access over ACLs.
- Do not embed long-lived keys in client-side code; use signed URLs for browser access.
Further reading
- GCS docs: https://cloud.google.com/storage/docs
- Signed URLs: https://cloud.google.com/storage/docs/access-control/signed-urls
- gsutil reference: https://cloud.google.com/storage/docs/gsutil
# Gabungkan Semua Data (Combine all data)
Dokumen ringkas (Bahasa Melayu) untuk menggabungkan fail yang tersimpan di Google Cloud Storage.
Sebelum mula
- Pastikan anda mempunyai akses ke bucket (roles/storage.objectViewer atau storage.objectAdmin untuk penulisan).
- Jika dataset besar (GB/TB), pertimbangkan Dataflow/Dataproc atau import terus ke BigQuery.
Pilihan umum
1) Quick merge (fail kecil/sederhana)
- Jika saiz kecil supaya muat dalam memori: gsutil cat gs://BUCKET/PATH/*.csv | gsutil cp - gs://BUCKET/PATH/combined.csv
- Risiko: memori dan rangkaian. Pastikan hanya digunakan untuk set kecil.
2) gsutil compose (gabungkan objek di GCS tanpa muat turun)
- Gabungkan hingga 32 objek per operasi: gsutil compose gs://BUCKET/part1.csv gs://BUCKET/part2.csv gs://BUCKET/combined.csv
- Untuk >32 objek, buat compose berperingkat (tree compose).
- Nota: compose menggabungkan byte; pastikan setiap objek mempunyai newline yang sesuai dan tidak menggabungkan header berganda.
3) Load terus ke BigQuery (disarankan untuk analisis)
- BigQuery boleh baca wildcard CSV: bq load --autodetect --source_format=CSV dataset.table gs://BUCKET/PATH/*.csv
- Kelebihan: skala, tiada pra-gabungan, schema handling.
4) Skrip tersuai (contoh Python)
- Kelebihan: kawalan penuh (buang header berganda, normalisasi).
- Lihat `examples/merge_csv_gcs.py` untuk contoh.
Contoh strategi untuk >32 objek menggunakan gsutil compose (pattern)
- Bahagikan senarai objek kepada kumpulan 32, compose setiap kumpulan ke objek sementara, kemudian compose hasil sementara (ulang sehingga satu objek terakhir).
- Atau gunakan pipeline Dataflow untuk penstriman dan penulisan semula.
Amalan terbaik
- Untuk output akhir yang akan dianalitik: lebih baik tulis ke BigQuery atau Parquet (kolumnar) untuk kos dan prestasi.
- Pastikan logging dan versi objek dihidupkan jika data penting (object versioning).
- Gunakan encryption keys (CMEK) jika diperlukan.
Contoh ringkas penggunaan skrip
- Jika anda mahu, berikan:
- Bucket name (contoh: gs://my-bucket/data/)
- Prefix dan jenis fail (.csv)
- Saiz anggaran (GB/TB) Saya akan hasilkan skrip automatik yang sesuai (bash atau Python) untuk anda.
# Signed URLs — Quick reference
Signed URLs grant temporary access to an object without requiring the requester to have a Google account.
Key points
- Signed URLs include authentication in query parameters.
- Anyone with the URL can access the object until expiry.
- Maximum expiry for V4 signed URLs is 7 days.
Create with gsutil (using service account key file)
gsutil signurl -d 1h /path/to/key.json gs://BUCKET/OBJECT
Create programmatically (Python example sketch)
- Use google-auth library to sign and generate a V4 URL, or use google-cloud-storage's blob.generate_signed_url method.
- Prefer service account credentials with least privilege.
Security tips
- Monitor signed-URL usage where possible (Cloud Storage logs).
- Shorter expiry reduces risk.
- Do not embed long-lived private keys in public repositories.
When to use
- Temporary downloads for users without Google accounts.
- Browser uploads (PUT) when combined with appropriate CORS settings.
#!/usr/bin/env python3
"""
examples/merge_csv_gcs.py
Merge CSV files in a GCS prefix into one CSV while keeping only the first header.
Requirements:
pip install google-cloud-storage
Usage:
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/key.json"
python3 examples/merge_csv_gcs.py my-bucket data/prefix/ output/combined.csv
"""
import sys
import csv
from io import StringIO
from google.cloud import storage
def merge_csvs(bucket_name, prefix, output_blob_name):
client = storage.Client()
bucket = client.bucket(bucket_name)
blobs = sorted([b for b in bucket.list_blobs(prefix=prefix) if b.name.endswith('.csv')])
if not blobs:
print("No CSV files found with prefix:", prefix)
return
out_buf = StringIO()
writer = None
first = True
for blob in blobs:
print("Reading:", blob.name)
# Use streaming download if files are large: download_as_text() may load entire object
data = blob.download_as_text()
reader = csv.reader(StringIO(data))
header = next(reader, None)
if first:
writer = csv.writer(out_buf)
if header:
writer.writerow(header)
for row in reader:
writer.writerow(row)
first = False
else:
# skip header for subsequent files
for row in reader:
writer.writerow(row)
out_blob = bucket.blob(output_blob_name)
out_blob.upload_from_string(out_buf.getvalue(), content_type='text/csv')
print(f'Uploaded gs://{bucket_name}/{output_blob_name}')
if __name__ == "__main__":
if len(sys.argv) != 4:
print("Usage: merge_csv_gcs.py BUCKET PREFIX OUTPUT_BLOB")
print("Example: merge_csv_gcs.py my-bucket data/ output/combined.csv")
sys.exit(1)
merge_csvs(sys.argv[1], sys.argv[2], sys.argv[3])
[
{
"origin": ["https://example.com"],
"method": ["GET", "HEAD", "PUT", "POST"],
"responseHeader": ["Content-Type", "x-goog-meta-custom"],
"maxAgeSeconds": 3600
}
]
MIT License
Copyright (c) 2025
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction...
(You can replace with your full MIT text or preferred license.)
Would you like me to:
generate a shell helper to compose >32 objects safely (tree compose), or
produce a Dataflow/Beam starter for large merges, or
create a translated Malay full-site version?
Tell me which follow-up and I will produce the exact script or pipeline.
Open Graph Description: Added documentation and example scripts for Google Cloud Storage usage, including quickstart guides, merging data, and CORS configuration.````markdown name=README.md # storage.cloud — Google Cloud ...
X Description: Added documentation and example scripts for Google Cloud Storage usage, including quickstart guides, merging data, and CORS configuration.````markdown name=README.md # storage.cloud — Google Cloud ...
Opengraph URL: https://github.com/Visual-Studio-Code/.github/pull/241
X: @github
direct link
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:4b057c21-5434-cd7a-c817-c0f9e9a9ce45 |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | 9F02:2652DC:CD7B0B:1099445:6B00F070 |
| html-safe-nonce | f8c7e164ccdca89de5a83651d39b69593384f60bd376de8d53a3e2c159e6931e |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5RjAyOjI2NTJEQzpDRDdCMEI6MTA5OTQ0NTo2QjAwRjA3MCIsInZpc2l0b3JfaWQiOiI3MDIyMzczNzI0MTg0MjA4NDgiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | 80d0d325a7642a582ae08d746e110f916cf6c11c21f7a84165f2b188e6a40cd3 |
| hovercard-subject-tag | pull_request:3448084127 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | ///pull_requests/show/files |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/Visual-Studio-Code/.github/pull/241/files |
| twitter:image | https://avatars.githubusercontent.com/u/208310745?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/208310745?s=400&v=4 |
| og:image:alt | Added documentation and example scripts for Google Cloud Storage usage, including quickstart guides, merging data, and CORS configuration.````markdown name=README.md # storage.cloud — Google Cloud ... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 4ef498cbabcbe3c50befc7a5b743e7072ee770b4daf0e15e54801180d1fa91fc |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/Visual-Studio-Code/.github git https://github.com/Visual-Studio-Code/.github.git |
| octolytics-dimension-user_id | 61069792 |
| octolytics-dimension-user_login | Visual-Studio-Code |
| octolytics-dimension-repository_id | 550489658 |
| octolytics-dimension-repository_nwo | Visual-Studio-Code/.github |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 550489658 |
| octolytics-dimension-repository_network_root_nwo | Visual-Studio-Code/.github |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 9983dd5da06943f833c971dd5b6bc988e2858be5 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
| Skip to content | https://github.com/Visual-Studio-Code/.github/pull/241/files#start-of-content |
| https://github.com/ |
| Sign in | https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2FVisual-Studio-Code%2F.github%2Fpull%2F241%2Ffiles |
| GitHubRepository hosting, code review, and collaboration | https://github.com/platform |
| CopilotAI coding agent for tasks, projects, and automated work | https://github.com/features/copilot |
| ActionsCI/CD and automation | https://github.com/features/actions |
| Code QualityQuality analysis, fixes, and enforcement | https://github.com/features/code-quality |
| CodespacesCloud development environments | https://github.com/features/codespaces |
| Code SecurityVulnerability detection and remediation | https://github.com/security/advanced-security/code-security |
| Secret ProtectionSecret scanning and leak prevention | https://github.com/security/advanced-security/secret-protection |
| All Features | https://github.com/features |
| Documentation | https://docs.github.com/en |
| Training | https://learn.github.com/skills |
| Certifications | https://learn.github.com/credentials |
| Build with Copilot | https://github.com/features/ai |
| Customize Copilot | https://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview |
| Integrations | https://github.com/marketplace |
| Trending repos | https://github.com/trending |
| GitHub Sponsors | https://github.com/open-source/sponsors |
| Developer articles | https://github.com/resources/articles |
| ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place. | https://github.blog/changelog/ |
| Enterprises | https://github.com/enterprise |
| Team | https://github.com/team |
| Startups | https://github.com/enterprise/startups |
| Nonprofits | https://github.com/solutions/industry/nonprofits |
| Healthcare | https://github.com/solutions/industry/healthcare |
| Financial services | https://github.com/solutions/industry/financial-services |
| Manufacturing | https://github.com/solutions/industry/manufacturing |
| Government | https://github.com/solutions/industry/government |
| See all industries | https://github.com/solutions/industry |
| Why GitHub | https://github.com/why-github |
| Customer Stories | https://github.com/customer-stories |
| Business insights | https://github.com/solutions/executive-insights |
| Trust Center | https://github.com/trust-center |
| GitHub Advanced SecurityEnterprise-grade code security | https://github.com/security/advanced-security |
| Copilot for businessEnterprise-grade AI features | https://github.com/features/copilot/copilot-business |
| Premium SupportEnterprise-grade 24/7 support | https://github.com/enterprise/premium-support |
| Documentation | https://docs.github.com/en |
| Resource center | https://github.com/resources |
| Blog | https://github.blog/ |
| Events & webinars | https://github.com/resources/events |
| Community Forum | https://github.com/orgs/community/discussions |
| Customer Stories | https://github.com/customer-stories |
| Integrations | https://github.com/marketplace |
| Customer support | https://help.github.com/ |
| Training | https://learn.github.com/skills |
| GitHub Status | https://www.githubstatus.com/ |
| About | https://github.com/about |
| Partners | https://github.com/partners |
| Changelog | https://github.blog/changelog/ |
| GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub. | https://githubuniverse.com/ |
| Pricing | https://github.com/pricing |
| Sign in | https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2FVisual-Studio-Code%2F.github%2Fpull%2F241%2Ffiles |
| Sign up | https://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=Visual-Studio-Code%2F.github |
| Reload | https://github.com/Visual-Studio-Code/.github/pull/241/files |
| Reload | https://github.com/Visual-Studio-Code/.github/pull/241/files |
| Reload | https://github.com/Visual-Studio-Code/.github/pull/241/files |
| Visual-Studio-Code | https://github.com/Visual-Studio-Code |
| .github | https://github.com/Visual-Studio-Code/.github |
| Code | https://github.com/Visual-Studio-Code/.github |
| Issues119 (119) | https://github.com/Visual-Studio-Code/.github/issues |
| Pull requests103 (103) | https://github.com/Visual-Studio-Code/.github/pulls |
| Actions | https://github.com/Visual-Studio-Code/.github/actions |
| Projects | https://github.com/Visual-Studio-Code/.github/projects |
| Security and quality | https://github.com/Visual-Studio-Code/.github/security |
| Insights | https://github.com/Visual-Studio-Code/.github/insights |
| Sign up for GitHub
| https://github.com/signup?return_to=%2FVisual-Studio-Code%2F.github%2Fissues%2Fnew%2Fchoose |
| terms of service | https://docs.github.com/terms |
| privacy statement | https://docs.github.com/privacy |
| Sign in | https://github.com/login?return_to=%2FVisual-Studio-Code%2F.github%2Fissues%2Fnew%2Fchoose |
| Muhamad Sazwan Bin Ismail (Sazwanismail) | https://github.com/Sazwanismail |
| Visual-Studio-Code:main | https://github.com/Visual-Studio-Code/.github/tree/main |
| Htpps-console-firebase-google-com:Sazwanismail-patch-1 | https://github.com/Htpps-console-firebase-google-com/.github/tree/Sazwanismail-patch-1 |
|
Conversation
0
| https://github.com/Visual-Studio-Code/.github/pull/241 |
|
Commits
1
| https://github.com/Visual-Studio-Code/.github/pull/241/commits |
|
Checks
0
| https://github.com/Visual-Studio-Code/.github/pull/241/checks |
|
Files changed
| https://github.com/Visual-Studio-Code/.github/pull/241/files |
| Please reload this page | https://github.com/Visual-Studio-Code/.github/pull/241/files |
|
Add storage.cloud documentation and example scripts
| https://github.com/Visual-Studio-Code/.github/pull/241/files#top |
|
Show all changes
1 commit
| https://github.com/Visual-Studio-Code/.github/pull/241/files |
|
84825be
Add storage.cloud documentation and example scripts
Sazwanismail Nov 6, 2025
| https://github.com/Visual-Studio-Code/.github/pull/241/commits/84825bee15337958d5942be895501e336c57e95a |
|
Clear filters
| https://github.com/Visual-Studio-Code/.github/pull/241/files |
| Please reload this page | https://github.com/Visual-Studio-Code/.github/pull/241/files |
| Please reload this page | https://github.com/Visual-Studio-Code/.github/pull/241/files |
| storage.cloud | https://github.com/Visual-Studio-Code/.github/pull/241/files#diff-cff3e873f467c86f5bc30712d9fb7e803dfdfff12b250c5e12c0b5668e133935 |
|
View file
| https://github.com/Visual-Studio-Code/.github/blob/84825bee15337958d5942be895501e336c57e95a/storage.cloud |
|
Open in desktop
| https://desktop.github.com |
| https://github.co/hiddenchars |
| https://github.com/Visual-Studio-Code/.github/pull/241/{{ revealButtonHref }} |
|
| https://github.com |
| Terms | https://docs.github.com/site-policy/github-terms/github-terms-of-service |
| Privacy | https://docs.github.com/site-policy/privacy-policies/github-privacy-statement |
| Security | https://github.com/security |
| Status | https://www.githubstatus.com/ |
| Community | https://github.community/ |
| Docs | https://docs.github.com/ |
| Contact | https://support.github.com?tags=dotcom-footer |
Viewport: width=device-width
URLs of crawlers that visited me.