René's URL Explorer Experiment

Title: GitHub - ToBeAGoodProgrammer/StudySpark: 学习 Spark 的一个小项目，以及其中各种调优的笔记 · GitHub

Open Graph Title: GitHub - ToBeAGoodProgrammer/StudySpark: 学习 Spark 的一个小项目，以及其中各种调优的笔记

X Title: GitHub - ToBeAGoodProgrammer/StudySpark: 学习 Spark 的一个小项目，以及其中各种调优的笔记

Description: 学习 Spark 的一个小项目，以及其中各种调优的笔记. Contribute to ToBeAGoodProgrammer/StudySpark development by creating an account on GitHub.

Open Graph Description: 学习 Spark 的一个小项目，以及其中各种调优的笔记. Contribute to ToBeAGoodProgrammer/StudySpark development by creating an account on GitHub.

X Description: 学习 Spark 的一个小项目，以及其中各种调优的笔记. Contribute to ToBeAGoodProgrammer/StudySpark development by creating an account on GitHub.

Opengraph URL: https://github.com/ToBeAGoodProgrammer/StudySpark

X: @github

direct link

Domain: github.com

route-pattern	/:user_id/:repository
route-controller	files
route-action	disambiguate
fetch-nonce	v2:fff60970-b6b6-2171-ab85-4905ae83b875
current-catalog-service-hash	f3abb0cc802f3d7b95fc8762b94bdcb13bf39634c40c357301c4aa1d67a256fb
request-id	B5FC:10BAD9:4C92F0:6FF132:6A588CD8
html-safe-nonce	3cab14773048910f81718153f634afc98f544b65514b9270d2e23c6cc72d2c6a
visitor-payload	eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJCNUZDOjEwQkFEOTo0QzkyRjA6NkZGMTMyOjZBNTg4Q0Q4IiwidmlzaXRvcl9pZCI6IjU2NDYxNjA3NzkyOTAzODE1MjgiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ==
visitor-hmac	14ef0adb9263a5d8c766d3d433df7836d77f32b6a2f5522f7fa7637ae6f838a8
hovercard-subject-tag	repository:232254444
github-keyboard-shortcuts	repository,copilot
google-site-verification	Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-url	https://collector.github.com/github/collect
analytics-location	//
fb:app_id	1401488693436528
apple-itunes-app	app-id=1477376905, app-argument=https://github.com/ToBeAGoodProgrammer/StudySpark
twitter:image	https://opengraph.githubassets.com/cbaaef5d05ecaf8336eeb22058d1c864b4e9fb1f3126c9d5505629a336d02354/ToBeAGoodProgrammer/StudySpark
twitter:card	summary_large_image
og:image	https://opengraph.githubassets.com/cbaaef5d05ecaf8336eeb22058d1c864b4e9fb1f3126c9d5505629a336d02354/ToBeAGoodProgrammer/StudySpark
og:image:alt	学习 Spark 的一个小项目，以及其中各种调优的笔记. Contribute to ToBeAGoodProgrammer/StudySpark development by creating an account on GitHub.
og:image:width	1200
og:image:height	600
og:site_name	GitHub
og:type	object
hostname	github.com
expected-hostname	github.com
None	5f2a0c7865178af3d91dd9f13b0cdfc3c73a2529c873d2780bb4c01271a57ec6
turbo-cache-control	no-cache
go-import	github.com/ToBeAGoodProgrammer/StudySpark git https://github.com/ToBeAGoodProgrammer/StudySpark.git
octolytics-dimension-user_id	43293242
octolytics-dimension-user_login	ToBeAGoodProgrammer
octolytics-dimension-repository_id	232254444
octolytics-dimension-repository_nwo	ToBeAGoodProgrammer/StudySpark
octolytics-dimension-repository_public	true
octolytics-dimension-repository_is_fork	true
octolytics-dimension-repository_parent_id	90385041
octolytics-dimension-repository_parent_nwo	yangtong123/StudySpark
octolytics-dimension-repository_network_root_id	90385041
octolytics-dimension-repository_network_root_nwo	yangtong123/StudySpark
turbo-body-classes	logged-out env-production page-responsive
disable-turbo	false
browser-stats-url	https://api.github.com/_private/browser/stats
browser-errors-url	https://api.github.com/_private/browser/errors
release	8aae7b8d6caacacf5c66eaeb2702d8dc88d85b4a
ui-target	full
theme-color	#1e2327
color-scheme	light dark

Links:

Skip to content	https://github.com/ToBeAGoodProgrammer/StudySpark#start-of-content
	https://github.com/
Sign in	https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2FToBeAGoodProgrammer%2FStudySpark
GitHub CopilotWrite better code with AI	https://github.com/features/copilot
GitHub Copilot appDirect agents from issue to merge	https://github.com/features/ai/github-app
MCP RegistryNewIntegrate external tools	https://github.com/mcp
ActionsAutomate any workflow	https://github.com/features/actions
CodespacesInstant dev environments	https://github.com/features/codespaces
IssuesPlan and track work	https://github.com/features/issues
Code ReviewManage code changes	https://github.com/features/code-review
GitHub Advanced SecurityFind and fix vulnerabilities	https://github.com/security/advanced-security
Code securitySecure your code as you build	https://github.com/security/advanced-security/code-security
Secret protectionStop leaks before they start	https://github.com/security/advanced-security/secret-protection
Why GitHub	https://github.com/why-github
Documentation	https://docs.github.com
Blog	https://github.blog
Changelog	https://github.blog/changelog
Marketplace	https://github.com/marketplace
View all features	https://github.com/features
Enterprises	https://github.com/enterprise
Small and medium teams	https://github.com/team
Startups	https://github.com/enterprise/startups
Nonprofits	https://github.com/solutions/industry/nonprofits
App Modernization	https://github.com/solutions/use-case/app-modernization
DevSecOps	https://github.com/solutions/use-case/devsecops
DevOps	https://github.com/solutions/use-case/devops
CI/CD	https://github.com/solutions/use-case/ci-cd
View all use cases	https://github.com/solutions/use-case
Healthcare	https://github.com/solutions/industry/healthcare
Financial services	https://github.com/solutions/industry/financial-services
Manufacturing	https://github.com/solutions/industry/manufacturing
Government	https://github.com/solutions/industry/government
View all industries	https://github.com/solutions/industry
View all solutions	https://github.com/solutions
AI	https://github.com/resources/articles?topic=ai
Software Development	https://github.com/resources/articles?topic=software-development
DevOps	https://github.com/resources/articles?topic=devops
Security	https://github.com/resources/articles?topic=security
View all topics	https://github.com/resources/articles
Customer stories	https://github.com/customer-stories
Events & webinars	https://github.com/resources/events
Ebooks & reports	https://github.com/resources/whitepapers
Business insights	https://github.com/solutions/executive-insights
GitHub Skills	https://skills.github.com
Documentation	https://docs.github.com
Customer support	https://support.github.com
Community forum	https://github.com/orgs/community/discussions
Trust center	https://github.com/trust-center
Partners	https://github.com/partners
View all resources	https://github.com/resources
GitHub SponsorsFund open source developers	https://github.com/open-source/sponsors
Security Lab	https://securitylab.github.com
Maintainer Community	https://maintainers.github.com
Accelerator	https://github.com/open-source/accelerator
GitHub Stars	https://stars.github.com
Archive Program	https://archiveprogram.github.com
Topics	https://github.com/topics
Trending	https://github.com/trending
Collections	https://github.com/collections
Enterprise platformAI-powered developer platform	https://github.com/enterprise
GitHub Advanced SecurityEnterprise-grade security features	https://github.com/security/advanced-security
Copilot for BusinessEnterprise-grade AI features	https://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 support	https://github.com/enterprise/premium-support
Pricing	https://github.com/pricing
Search syntax tips	https://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
documentation	https://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
Sign in	https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2FToBeAGoodProgrammer%2FStudySpark
Sign up	https://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E&source=header-repo&source_repo=ToBeAGoodProgrammer%2FStudySpark
Reload	https://github.com/ToBeAGoodProgrammer/StudySpark
Reload	https://github.com/ToBeAGoodProgrammer/StudySpark
Reload	https://github.com/ToBeAGoodProgrammer/StudySpark
ToBeAGoodProgrammer	https://github.com/ToBeAGoodProgrammer
StudySpark	https://github.com/ToBeAGoodProgrammer/StudySpark
yangtong123/StudySpark	https://github.com/yangtong123/StudySpark
Notifications	https://github.com/login?return_to=%2FToBeAGoodProgrammer%2FStudySpark
Fork 0	https://github.com/login?return_to=%2FToBeAGoodProgrammer%2FStudySpark
Star 0	https://github.com/login?return_to=%2FToBeAGoodProgrammer%2FStudySpark
Code	https://github.com/ToBeAGoodProgrammer/StudySpark
Pull requests 0	https://github.com/ToBeAGoodProgrammer/StudySpark/pulls
Actions	https://github.com/ToBeAGoodProgrammer/StudySpark/actions
Projects	https://github.com/ToBeAGoodProgrammer/StudySpark/projects
Security and quality 0	https://github.com/ToBeAGoodProgrammer/StudySpark/security
Insights	https://github.com/ToBeAGoodProgrammer/StudySpark/pulse
Code	https://github.com/ToBeAGoodProgrammer/StudySpark
Pull requests	https://github.com/ToBeAGoodProgrammer/StudySpark/pulls
Actions	https://github.com/ToBeAGoodProgrammer/StudySpark/actions
Projects	https://github.com/ToBeAGoodProgrammer/StudySpark/projects
Security and quality	https://github.com/ToBeAGoodProgrammer/StudySpark/security
Insights	https://github.com/ToBeAGoodProgrammer/StudySpark/pulse
	https://github.com/ToBeAGoodProgrammer/StudySpark
Branches	https://github.com/ToBeAGoodProgrammer/StudySpark/branches
Tags	https://github.com/ToBeAGoodProgrammer/StudySpark/tags
	https://github.com/ToBeAGoodProgrammer/StudySpark/branches
	https://github.com/ToBeAGoodProgrammer/StudySpark/tags
30 Commits	https://github.com/ToBeAGoodProgrammer/StudySpark/commits/master/
	https://github.com/ToBeAGoodProgrammer/StudySpark/commits/master/
.idea	https://github.com/ToBeAGoodProgrammer/StudySpark/tree/master/.idea
.idea	https://github.com/ToBeAGoodProgrammer/StudySpark/tree/master/.idea
pic	https://github.com/ToBeAGoodProgrammer/StudySpark/tree/master/pic
pic	https://github.com/ToBeAGoodProgrammer/StudySpark/tree/master/pic
src	https://github.com/ToBeAGoodProgrammer/StudySpark/tree/master/src
src	https://github.com/ToBeAGoodProgrammer/StudySpark/tree/master/src
README.md	https://github.com/ToBeAGoodProgrammer/StudySpark/blob/master/README.md
README.md	https://github.com/ToBeAGoodProgrammer/StudySpark/blob/master/README.md
StudySpark.iml	https://github.com/ToBeAGoodProgrammer/StudySpark/blob/master/StudySpark.iml
StudySpark.iml	https://github.com/ToBeAGoodProgrammer/StudySpark/blob/master/StudySpark.iml
pom.xml	https://github.com/ToBeAGoodProgrammer/StudySpark/blob/master/pom.xml
pom.xml	https://github.com/ToBeAGoodProgrammer/StudySpark/blob/master/pom.xml
README	https://github.com/ToBeAGoodProgrammer/StudySpark
	https://github.com/ToBeAGoodProgrammer/StudySpark#studyspark
	https://github.com/ToBeAGoodProgrammer/StudySpark#目录
项目内容	https://github.com/ToBeAGoodProgrammer/StudySpark#%E9%A1%B9%E7%9B%AE%E5%86%85%E5%AE%B9
学习笔记	https://github.com/ToBeAGoodProgrammer/StudySpark#%E5%AD%A6%E4%B9%A0%E7%AC%94%E8%AE%B0
一些操作	https://github.com/ToBeAGoodProgrammer/StudySpark#%E4%B8%80%E4%BA%9B%E6%93%8D%E4%BD%9C
性能调优	https://github.com/ToBeAGoodProgrammer/StudySpark#%E6%80%A7%E8%83%BD%E8%B0%83%E4%BC%98%E8%BF%98%E6%9C%89%E8%BF%99%E7%A7%8D%E6%93%8D%E4%BD%9Cboom
调节并行度	https://github.com/ToBeAGoodProgrammer/StudySpark#%E8%B0%83%E8%8A%82%E5%B9%B6%E8%A1%8C%E5%BA%A6
重构RDD与持久化	https://github.com/ToBeAGoodProgrammer/StudySpark#%E9%87%8D%E6%9E%84rdd%E4%B8%8E%E6%8C%81%E4%B9%85%E5%8C%96
广播大变量	https://github.com/ToBeAGoodProgrammer/StudySpark#%E5%B9%BF%E6%92%AD%E5%A4%A7%E5%8F%98%E9%87%8F
使用Kryo序列化	https://github.com/ToBeAGoodProgrammer/StudySpark#%E4%BD%BF%E7%94%A8kryo%E5%BA%8F%E5%88%97%E5%8C%96
使用fastutil优化数据格式	https://github.com/ToBeAGoodProgrammer/StudySpark#%E4%BD%BF%E7%94%A8fastutil%E4%BC%98%E5%8C%96%E6%95%B0%E6%8D%AE%E6%A0%BC%E5%BC%8F
调节数据本地化等待时长	https://github.com/ToBeAGoodProgrammer/StudySpark#%E8%B0%83%E8%8A%82%E6%95%B0%E6%8D%AE%E6%9C%AC%E5%9C%B0%E5%8C%96%E7%AD%89%E5%BE%85%E6%97%B6%E9%95%BF
JVM调优之降低cache操作的内存占比	https://github.com/ToBeAGoodProgrammer/StudySpark#jvm%E8%B0%83%E4%BC%98%E4%B9%8B%E9%99%8D%E4%BD%8Ecache%E6%93%8D%E4%BD%9C%E7%9A%84%E5%86%85%E5%AD%98%E5%8D%A0%E6%AF%94
JVM调优之调节Executor堆外内存与连接等待时长	https://github.com/ToBeAGoodProgrammer/StudySpark#jvm%E8%B0%83%E4%BC%98%E4%B9%8B%E8%B0%83%E8%8A%82executor%E5%A0%86%E5%A4%96%E5%86%85%E5%AD%98%E4%B8%8E%E8%BF%9E%E6%8E%A5%E7%AD%89%E5%BE%85%E6%97%B6%E9%95%BF
Shuffle调优	https://github.com/ToBeAGoodProgrammer/StudySpark#shuffle%E8%B0%83%E4%BC%98
调节Map端内存缓冲与Reduce端内存占比	https://github.com/ToBeAGoodProgrammer/StudySpark#%E8%B0%83%E8%8A%82map%E7%AB%AF%E5%86%85%E5%AD%98%E7%BC%93%E5%86%B2%E4%B8%8Ereduce%E7%AB%AF%E5%86%85%E5%AD%98%E5%8D%A0%E6%AF%94
spark.shuffle.sort.bypassMergeThreshold	https://github.com/ToBeAGoodProgrammer/StudySpark#sparkshufflesortbypassmergethreshold
算子调优	https://github.com/ToBeAGoodProgrammer/StudySpark#%E7%AE%97%E5%AD%90%E8%B0%83%E4%BC%98
map与mapPartitions	https://github.com/ToBeAGoodProgrammer/StudySpark#map%E4%B8%8Emappartitions
filter过后使用coalesce	https://github.com/ToBeAGoodProgrammer/StudySpark#filter%E8%BF%87%E5%90%8E%E4%BD%BF%E7%94%A8coalesce
foreachPartition优化写数据库	https://github.com/ToBeAGoodProgrammer/StudySpark#foreachpartition%E4%BC%98%E5%8C%96%E5%86%99%E6%95%B0%E6%8D%AE%E5%BA%93
repartition解决Spark SQL并行度过低	https://github.com/ToBeAGoodProgrammer/StudySpark#repartition%E8%A7%A3%E5%86%B3spark-sql%E5%B9%B6%E8%A1%8C%E5%BA%A6%E8%BF%87%E4%BD%8E
reduceByKey的Map端本地聚合	https://github.com/ToBeAGoodProgrammer/StudySpark#reducebykey%E7%9A%84map%E7%AB%AF%E6%9C%AC%E5%9C%B0%E8%81%9A%E5%90%88
troubleShooting	https://github.com/ToBeAGoodProgrammer/StudySpark#troubleshooting
控制shuffle reduce端缓冲大小以避免OOM	https://github.com/ToBeAGoodProgrammer/StudySpark#%E6%8E%A7%E5%88%B6shuffle-reduce%E7%AB%AF%E7%BC%93%E5%86%B2%E5%A4%A7%E5%B0%8F%E4%BB%A5%E9%81%BF%E5%85%8Doom
JVM GC导致shuffle文件拉取失败	https://github.com/ToBeAGoodProgrammer/StudySpark#jvm-gc%E5%AF%BC%E8%87%B4shuffle%E6%96%87%E4%BB%B6%E6%8B%89%E5%8F%96%E5%A4%B1%E8%B4%A5
Yarn队列资源不足导致Application直接失败	https://github.com/ToBeAGoodProgrammer/StudySpark#yarn%E9%98%9F%E5%88%97%E8%B5%84%E6%BA%90%E4%B8%8D%E8%B6%B3%E5%AF%BC%E8%87%B4application%E7%9B%B4%E6%8E%A5%E5%A4%B1%E8%B4%A5
各种序列化导致的报错	https://github.com/ToBeAGoodProgrammer/StudySpark#%E5%90%84%E7%A7%8D%E5%BA%8F%E5%88%97%E5%8C%96%E5%AF%BC%E8%87%B4%E7%9A%84%E6%8A%A5%E9%94%99
算子函数返回NULL导致的问题	https://github.com/ToBeAGoodProgrammer/StudySpark#%E7%AE%97%E5%AD%90%E5%87%BD%E6%95%B0%E8%BF%94%E5%9B%9Enull%E5%AF%BC%E8%87%B4%E7%9A%84%E9%97%AE%E9%A2%98
Yarn-cluster模式JVM内存溢出无法执行	https://github.com/ToBeAGoodProgrammer/StudySpark#yarn-cluster%E6%A8%A1%E5%BC%8Fjvm%E5%86%85%E5%AD%98%E6%BA%A2%E5%87%BA%E6%97%A0%E6%B3%95%E6%89%A7%E8%A1%8C
数据倾斜解决方案	https://github.com/ToBeAGoodProgrammer/StudySpark#%E6%95%B0%E6%8D%AE%E5%80%BE%E6%96%9C%E8%A7%A3%E5%86%B3%E6%96%B9%E6%A1%88
聚合源数据	https://github.com/ToBeAGoodProgrammer/StudySpark#%E8%81%9A%E5%90%88%E6%BA%90%E6%95%B0%E6%8D%AE
提高shuffle操作的reduce并行度	https://github.com/ToBeAGoodProgrammer/StudySpark#%E6%8F%90%E9%AB%98shuffle%E6%93%8D%E4%BD%9C%E7%9A%84reduce%E5%B9%B6%E8%A1%8C%E5%BA%A6
使用随机key实现双重聚合	https://github.com/ToBeAGoodProgrammer/StudySpark#%E4%BD%BF%E7%94%A8%E9%9A%8F%E6%9C%BAkey%E5%AE%9E%E7%8E%B0%E5%8F%8C%E9%87%8D%E8%81%9A%E5%90%88
将reduce join转为map join	https://github.com/ToBeAGoodProgrammer/StudySpark#%E5%B0%86reduce-join%E8%BD%AC%E4%B8%BAmap-join
sample采样倾斜key进行两次join	https://github.com/ToBeAGoodProgrammer/StudySpark#sample%E9%87%87%E6%A0%B7%E5%80%BE%E6%96%9Ckey%E8%BF%9B%E8%A1%8C%E4%B8%A4%E6%AC%A1join
使用随机数以及扩容表进行join	https://github.com/ToBeAGoodProgrammer/StudySpark#%E4%BD%BF%E7%94%A8%E9%9A%8F%E6%9C%BA%E6%95%B0%E4%BB%A5%E5%8F%8A%E6%89%A9%E5%AE%B9%E8%A1%A8%E8%BF%9B%E8%A1%8Cjoin
Spark SQL数据倾斜解决方案	https://github.com/ToBeAGoodProgrammer/StudySpark#spark-sql%E6%95%B0%E6%8D%AE%E5%80%BE%E6%96%9C%E8%A7%A3%E5%86%B3%E6%96%B9%E6%A1%88
Spark Streaming	https://github.com/ToBeAGoodProgrammer/StudySpark#spark-streaming
Spark Streaming高可用	https://github.com/ToBeAGoodProgrammer/StudySpark#spark-streaming%E9%AB%98%E5%8F%AF%E7%94%A8
Spark Streaming性能调优	https://github.com/ToBeAGoodProgrammer/StudySpark#spark-streaming%E6%80%A7%E8%83%BD%E8%B0%83%E4%BC%98
	https://github.com/ToBeAGoodProgrammer/StudySpark#项目内容
UserVisitSessionAnalyzeSpark	https://github.com/ToBeAGoodProgrammer/StudySpark/blob/master/src/main/java/com/yt/spark/spark/session/UserVisitSessionAnalyzeSpark.java
PageOneStepConvertRateSpark	https://github.com/ToBeAGoodProgrammer/StudySpark/blob/master/src/main/java/com/yt/spark/spark/page/PageOneStepConvertRateSpark.java
AreaTop3ProductSpark	https://github.com/ToBeAGoodProgrammer/StudySpark/blob/master/src/main/java/com/yt/spark/spark/product/AreaTop3ProductSpark.java
AdClickRealTimeStatSpark	https://github.com/ToBeAGoodProgrammer/StudySpark/blob/master/src/main/java/com/yt/spark/spark/ad/AdClickRealTimeStatSpark.java
	https://github.com/ToBeAGoodProgrammer/StudySpark#学习笔记
	https://github.com/ToBeAGoodProgrammer/StudySpark#一些操作
JDBCHelper	https://github.com/ToBeAGoodProgrammer/StudySpark/blob/master/src/main/java/com/yt/spark/jdbc/JDBCHelper.java
SessionAggrStatAccumulator	https://github.com/ToBeAGoodProgrammer/StudySpark/blob/master/src/main/java/com/yt/spark/spark/session/SessionAggrStatAccumulator.java
CategorySortKey	https://github.com/ToBeAGoodProgrammer/StudySpark/blob/master/src/main/java/com/yt/spark/spark/session/CategorySortKey.java
	https://github.com/ToBeAGoodProgrammer/StudySpark#性能调优还有这种操作boom
	https://github.com/ToBeAGoodProgrammer/StudySpark#调节并行度
	https://github.com/ToBeAGoodProgrammer/StudySpark#重构rdd与持久化
	https://github.com/ToBeAGoodProgrammer/StudySpark#广播大变量
	https://github.com/ToBeAGoodProgrammer/StudySpark#使用kryo序列化
	https://github.com/ToBeAGoodProgrammer/StudySpark#使用fastutil优化数据格式
UserVisitSessionAnalyzeSpark.java	https://github.com/ToBeAGoodProgrammer/StudySpark/blob/master/src/main/java/com/yt/spark/spark/session/UserVisitSessionAnalyzeSpark.java
	https://github.com/ToBeAGoodProgrammer/StudySpark#调节数据本地化等待时长
	https://github.com/ToBeAGoodProgrammer/StudySpark#jvm调优之降低cache操作的内存占比
	https://github.com/ToBeAGoodProgrammer/StudySpark#jvm调优之调节executor堆外内存与连接等待时长
	https://github.com/ToBeAGoodProgrammer/StudySpark#shuffle调优
	https://github.com/ToBeAGoodProgrammer/StudySpark#调节map端内存缓冲与reduce端内存占比
	https://github.com/ToBeAGoodProgrammer/StudySpark#sparkshufflesortbypassmergethreshold
	https://github.com/ToBeAGoodProgrammer/StudySpark#算子调优
	https://github.com/ToBeAGoodProgrammer/StudySpark#map与mappartitions
	https://github.com/ToBeAGoodProgrammer/StudySpark#filter过后使用coalesce
	https://github.com/ToBeAGoodProgrammer/StudySpark#foreachpartition优化写数据库
	https://github.com/ToBeAGoodProgrammer/StudySpark#repartition解决spark-sql并行度过低
调节并行度	https://github.com/ToBeAGoodProgrammer/StudySpark#%E8%B0%83%E8%8A%82%E5%B9%B6%E8%A1%8C%E5%BA%A6
	https://github.com/ToBeAGoodProgrammer/StudySpark#reducebykey的map端本地聚合
	https://github.com/ToBeAGoodProgrammer/StudySpark#troubleshooting
	https://github.com/ToBeAGoodProgrammer/StudySpark#控制shuffle-reduce端缓冲大小以避免oom
	https://github.com/ToBeAGoodProgrammer/StudySpark#jvm-gc导致shuffle文件拉取失败
JVM调优之调节Executor堆外内存与连接等待时长	https://github.com/ToBeAGoodProgrammer/StudySpark#jvm%E8%B0%83%E4%BC%98%E4%B9%8B%E8%B0%83%E8%8A%82executor%E5%A0%86%E5%A4%96%E5%86%85%E5%AD%98%E4%B8%8E%E8%BF%9E%E6%8E%A5%E7%AD%89%E5%BE%85%E6%97%B6%E9%95%BF
	https://github.com/ToBeAGoodProgrammer/StudySpark#yarn队列资源不足导致application直接失败
	https://github.com/ToBeAGoodProgrammer/StudySpark#各种序列化导致的报错
	https://github.com/ToBeAGoodProgrammer/StudySpark#算子函数返回null导致的问题
	https://github.com/ToBeAGoodProgrammer/StudySpark#yarn-cluster模式jvm内存溢出无法执行
	https://github.com/ToBeAGoodProgrammer/StudySpark#数据倾斜解决方案
	https://github.com/ToBeAGoodProgrammer/StudySpark#聚合源数据
	https://github.com/ToBeAGoodProgrammer/StudySpark#提高shuffle操作的reduce并行度
	https://github.com/ToBeAGoodProgrammer/StudySpark#使用随机key实现双重聚合
	https://github.com/ToBeAGoodProgrammer/StudySpark/blob/master/pic/随机key实现双重聚合.png
	https://github.com/ToBeAGoodProgrammer/StudySpark#将reduce-join转为map-join
	https://github.com/ToBeAGoodProgrammer/StudySpark/blob/master/pic/reduce join转为map join.png
	https://github.com/ToBeAGoodProgrammer/StudySpark#sample采样倾斜key进行两次join
	https://github.com/ToBeAGoodProgrammer/StudySpark/blob/master/pic/sample采样倾斜key进行两次join.png
	https://github.com/ToBeAGoodProgrammer/StudySpark#使用随机数以及扩容表进行join
	https://github.com/ToBeAGoodProgrammer/StudySpark#spark-sql数据倾斜解决方案
	https://github.com/ToBeAGoodProgrammer/StudySpark#spark-streaming
	https://github.com/ToBeAGoodProgrammer/StudySpark#spark-streaming高可用
	https://github.com/ToBeAGoodProgrammer/StudySpark#spark-streaming性能调优
Readme	https://github.com/ToBeAGoodProgrammer/StudySpark#readme-ov-file
Please reload this page	https://github.com/ToBeAGoodProgrammer/StudySpark
Activity	https://github.com/ToBeAGoodProgrammer/StudySpark/activity
0 forks	https://github.com/ToBeAGoodProgrammer/StudySpark/forks
Report repository	https://github.com/contact/report-content?content_url=https%3A%2F%2Fgithub.com%2FToBeAGoodProgrammer%2FStudySpark&report=ToBeAGoodProgrammer+%28user%29
Releases	https://github.com/ToBeAGoodProgrammer/StudySpark/releases
Packages 0	https://github.com/users/ToBeAGoodProgrammer/packages?repo_name=StudySpark
Please reload this page	https://github.com/ToBeAGoodProgrammer/StudySpark
Contributors	https://github.com/ToBeAGoodProgrammer/StudySpark/graphs/contributors
Please reload this page	https://github.com/ToBeAGoodProgrammer/StudySpark
	https://github.com
Terms	https://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacy	https://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Security	https://github.com/security
Status	https://www.githubstatus.com/
Community	https://github.community/
Docs	https://docs.github.com/
Contact	https://support.github.com?tags=dotcom-footer

Viewport: width=device-width

URLs of crawlers that visited me.