| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /:user_id/:repository |
| route-controller | files |
| route-action | disambiguate |
| fetch-nonce | v2:5ed6d78f-9feb-62f0-6194-f63ef2f2e103 |
| current-catalog-service-hash | f3abb0cc802f3d7b95fc8762b94bdcb13bf39634c40c357301c4aa1d67a256fb |
| request-id | A2B8:15289F:24ACA97:3085880:6B036C54 |
| html-safe-nonce | 3664dd95656014f1310a86aa24c844169da4acfec8ae9859472469e3efd546ea |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJBMkI4OjE1Mjg5RjoyNEFDQTk3OjMwODU4ODA6NkIwMzZDNTQiLCJ2aXNpdG9yX2lkIjoiNzQwNjA1Mzc0MDg1MzI5MjExNiIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | 71d06b3299c3240437cb9a59ac29c54354b218c093e41774251aa454e35bd3c4 |
| hovercard-subject-tag | repository:155800823 |
| github-keyboard-shortcuts | repository,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | // |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/Machine-Learning-Research/bert |
| twitter:image | https://opengraph.githubassets.com/ff1c61d561f3dd24807f47232b25015e99a417546e5148bcb4dfb40419d3ecf0/Machine-Learning-Research/bert |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/ff1c61d561f3dd24807f47232b25015e99a417546e5148bcb4dfb40419d3ecf0/Machine-Learning-Research/bert |
| og:image:alt | TensorFlow code and pre-trained models for BERT. Contribute to Machine-Learning-Research/bert development by creating an account on GitHub. |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456 |
| turbo-cache-control | no-cache |
| go-import | github.com/Machine-Learning-Research/bert git https://github.com/Machine-Learning-Research/bert.git |
| octolytics-dimension-user_id | 44436925 |
| octolytics-dimension-user_login | Machine-Learning-Research |
| octolytics-dimension-repository_id | 155800823 |
| octolytics-dimension-repository_nwo | Machine-Learning-Research/bert |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | true |
| octolytics-dimension-repository_parent_id | 154747577 |
| octolytics-dimension-repository_parent_nwo | google-research/bert |
| octolytics-dimension-repository_network_root_id | 154747577 |
| octolytics-dimension-repository_network_root_nwo | google-research/bert |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 423dbb54c5e7bfecc838eb387c1435eb3d0e760d |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| Skip to content | https://github.com/Machine-Learning-Research/bert#start-of-content |
| https://github.com/ |
| Sign in | https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2FMachine-Learning-Research%2Fbert |
| GitHubRepository hosting, code review, and collaboration | https://github.com/platform |
| CopilotAI coding agent for tasks, projects, and automated work | https://github.com/features/copilot |
| ActionsCI/CD and automation | https://github.com/features/actions |
| Code QualityQuality analysis, fixes, and enforcement | https://github.com/features/code-quality |
| CodespacesCloud development environments | https://github.com/features/codespaces |
| Code SecurityVulnerability detection and remediation | https://github.com/security/advanced-security/code-security |
| Secret ProtectionSecret scanning and leak prevention | https://github.com/security/advanced-security/secret-protection |
| All Features | https://github.com/features |
| Documentation | https://docs.github.com/en |
| Training | https://learn.github.com/skills |
| Certifications | https://learn.github.com/credentials |
| Build with Copilot | https://github.com/features/ai |
| Customize Copilot | https://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview |
| Integrations | https://github.com/marketplace |
| Trending repos | https://github.com/trending |
| GitHub Sponsors | https://github.com/open-source/sponsors |
| Developer articles | https://github.com/resources/articles |
| ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place. | https://github.blog/changelog/ |
| Enterprises | https://github.com/enterprise |
| Team | https://github.com/team |
| Startups | https://github.com/enterprise/startups |
| Nonprofits | https://github.com/solutions/industry/nonprofits |
| Healthcare | https://github.com/solutions/industry/healthcare |
| Financial services | https://github.com/solutions/industry/financial-services |
| Manufacturing | https://github.com/solutions/industry/manufacturing |
| Government | https://github.com/solutions/industry/government |
| See all industries | https://github.com/solutions/industry |
| Why GitHub | https://github.com/why-github |
| Customer Stories | https://github.com/customer-stories |
| Business insights | https://github.com/solutions/executive-insights |
| Trust Center | https://github.com/trust-center |
| GitHub Advanced SecurityEnterprise-grade code security | https://github.com/security/advanced-security |
| Copilot for businessEnterprise-grade AI features | https://github.com/features/copilot/copilot-business |
| Premium SupportEnterprise-grade 24/7 support | https://github.com/enterprise/premium-support |
| Documentation | https://docs.github.com/en |
| Resource center | https://github.com/resources |
| Blog | https://github.blog/ |
| Events & webinars | https://github.com/resources/events |
| Community Forum | https://github.com/orgs/community/discussions |
| Customer Stories | https://github.com/customer-stories |
| Integrations | https://github.com/marketplace |
| Customer support | https://help.github.com/ |
| Training | https://learn.github.com/skills |
| GitHub Status | https://www.githubstatus.com/ |
| About | https://github.com/about |
| Partners | https://github.com/partners |
| Changelog | https://github.blog/changelog/ |
| GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub. | https://githubuniverse.com/ |
| Pricing | https://github.com/pricing |
| Sign in | https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2FMachine-Learning-Research%2Fbert |
| Sign up | https://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E&source=header-repo&source_repo=Machine-Learning-Research%2Fbert |
| Reload | https://github.com/Machine-Learning-Research/bert |
| Reload | https://github.com/Machine-Learning-Research/bert |
| Reload | https://github.com/Machine-Learning-Research/bert |
| Machine-Learning-Research | https://github.com/Machine-Learning-Research |
| bert | https://github.com/Machine-Learning-Research/bert |
| Code | https://github.com/Machine-Learning-Research/bert |
| Pull requests | https://github.com/Machine-Learning-Research/bert/pulls |
| Actions | https://github.com/Machine-Learning-Research/bert/actions |
| Projects | https://github.com/Machine-Learning-Research/bert/projects |
| Wiki | https://github.com/Machine-Learning-Research/bert/wiki |
| Security and quality | https://github.com/Machine-Learning-Research/bert/security |
| Insights | https://github.com/Machine-Learning-Research/bert/insights |
| https://github.com/Machine-Learning-Research/bert |
| bert | https://github.com/Machine-Learning-Research/bert |
| google-research/bert | https://github.com/google-research/bert |
| Notifications | https://github.com/login?return_to=%2FMachine-Learning-Research%2Fbert |
| Fork0 (0) | https://github.com/login?return_to=%2FMachine-Learning-Research%2Fbert |
| Star0 (0) | https://github.com/login?return_to=%2FMachine-Learning-Research%2Fbert |
| arxiv.org/abs/1810.04805 | https://arxiv.org/abs/1810.04805 |
| Readme | https://github.com/Machine-Learning-Research/bert#readme-ov-file |
| Apache-2.0 license | https://github.com/Machine-Learning-Research/bert#Apache-2.0-1-ov-file |
| Contributing | https://github.com/Machine-Learning-Research/bert#contributing-ov-file |
| Activity | https://github.com/Machine-Learning-Research/bert/activity |
| Custom properties | https://github.com/Machine-Learning-Research/bert/custom-properties |
| 0 forks | https://github.com/Machine-Learning-Research/bert/forks |
| Report repository | https://github.com/contact/report-content?content_url=https%3A%2F%2Fgithub.com%2FMachine-Learning-Research%2Fbert&report=Machine-Learning-Research+%28user%29 |
| Branches | https://github.com/Machine-Learning-Research/bert/branches |
| Tags | https://github.com/Machine-Learning-Research/bert/tags |
| https://github.com/Machine-Learning-Research/bert/branches |
| https://github.com/Machine-Learning-Research/bert/tags |
| 16 Commits | https://github.com/Machine-Learning-Research/bert/commits/master/ |
| https://github.com/Machine-Learning-Research/bert/commits/master/ |
| .gitignore | https://github.com/Machine-Learning-Research/bert/blob/master/.gitignore |
| .gitignore | https://github.com/Machine-Learning-Research/bert/blob/master/.gitignore |
| CONTRIBUTING.md | https://github.com/Machine-Learning-Research/bert/blob/master/CONTRIBUTING.md |
| CONTRIBUTING.md | https://github.com/Machine-Learning-Research/bert/blob/master/CONTRIBUTING.md |
| LICENSE | https://github.com/Machine-Learning-Research/bert/blob/master/LICENSE |
| LICENSE | https://github.com/Machine-Learning-Research/bert/blob/master/LICENSE |
| README.md | https://github.com/Machine-Learning-Research/bert/blob/master/README.md |
| README.md | https://github.com/Machine-Learning-Research/bert/blob/master/README.md |
| __init__.py | https://github.com/Machine-Learning-Research/bert/blob/master/__init__.py |
| __init__.py | https://github.com/Machine-Learning-Research/bert/blob/master/__init__.py |
| create_pretraining_data.py | https://github.com/Machine-Learning-Research/bert/blob/master/create_pretraining_data.py |
| create_pretraining_data.py | https://github.com/Machine-Learning-Research/bert/blob/master/create_pretraining_data.py |
| extract_features.py | https://github.com/Machine-Learning-Research/bert/blob/master/extract_features.py |
| extract_features.py | https://github.com/Machine-Learning-Research/bert/blob/master/extract_features.py |
| modeling.py | https://github.com/Machine-Learning-Research/bert/blob/master/modeling.py |
| modeling.py | https://github.com/Machine-Learning-Research/bert/blob/master/modeling.py |
| modeling_test.py | https://github.com/Machine-Learning-Research/bert/blob/master/modeling_test.py |
| modeling_test.py | https://github.com/Machine-Learning-Research/bert/blob/master/modeling_test.py |
| optimization.py | https://github.com/Machine-Learning-Research/bert/blob/master/optimization.py |
| optimization.py | https://github.com/Machine-Learning-Research/bert/blob/master/optimization.py |
| optimization_test.py | https://github.com/Machine-Learning-Research/bert/blob/master/optimization_test.py |
| optimization_test.py | https://github.com/Machine-Learning-Research/bert/blob/master/optimization_test.py |
| requirements.txt | https://github.com/Machine-Learning-Research/bert/blob/master/requirements.txt |
| requirements.txt | https://github.com/Machine-Learning-Research/bert/blob/master/requirements.txt |
| run_classifier.py | https://github.com/Machine-Learning-Research/bert/blob/master/run_classifier.py |
| run_classifier.py | https://github.com/Machine-Learning-Research/bert/blob/master/run_classifier.py |
| run_pretraining.py | https://github.com/Machine-Learning-Research/bert/blob/master/run_pretraining.py |
| run_pretraining.py | https://github.com/Machine-Learning-Research/bert/blob/master/run_pretraining.py |
| run_squad.py | https://github.com/Machine-Learning-Research/bert/blob/master/run_squad.py |
| run_squad.py | https://github.com/Machine-Learning-Research/bert/blob/master/run_squad.py |
| sample_text.txt | https://github.com/Machine-Learning-Research/bert/blob/master/sample_text.txt |
| sample_text.txt | https://github.com/Machine-Learning-Research/bert/blob/master/sample_text.txt |
| tokenization.py | https://github.com/Machine-Learning-Research/bert/blob/master/tokenization.py |
| tokenization.py | https://github.com/Machine-Learning-Research/bert/blob/master/tokenization.py |
| tokenization_test.py | https://github.com/Machine-Learning-Research/bert/blob/master/tokenization_test.py |
| tokenization_test.py | https://github.com/Machine-Learning-Research/bert/blob/master/tokenization_test.py |
| README | https://github.com/Machine-Learning-Research/bert |
| Contributing | https://github.com/Machine-Learning-Research/bert |
| Apache-2.0 license | https://github.com/Machine-Learning-Research/bert |
| https://github.com/Machine-Learning-Research/bert#bert |
| https://github.com/Machine-Learning-Research/bert#introduction |
| https://arxiv.org/abs/1810.04805 | https://arxiv.org/abs/1810.04805 |
| SQuAD v1.1 | https://rajpurkar.github.io/SQuAD-explorer/ |
| download the pre-trained models | https://github.com/Machine-Learning-Research/bert#pre-trained-models |
| run a state-of-the-art fine-tuning | https://github.com/Machine-Learning-Research/bert#fine-tuning-with-bert |
| https://github.com/Machine-Learning-Research/bert#what-is-bert |
| word2vec | https://www.tensorflow.org/tutorials/representation/word2vec |
| GloVe | https://nlp.stanford.edu/projects/glove/ |
| Semi-supervised Sequence Learning | https://arxiv.org/abs/1511.01432 |
| Generative Pre-Training | https://blog.openai.com/language-unsupervised/ |
| ELMo | https://allennlp.org/elmo |
| ULMFit | http://nlp.fast.ai/classification/2018/05/15/introducting-ulmfit.html |
| Transformer | https://arxiv.org/abs/1706.03762 |
| BookCorpus | http://yknzhu.wixsite.com/mbweb |
| https://github.com/Machine-Learning-Research/bert#what-has-been-released-in-this-repository |
| Transformer | https://arxiv.org/abs/1706.03762 |
| https://github.com/Machine-Learning-Research/bert#pre-trained-models |
| BERT-Base, Uncased | https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-12_H-768_A-12.zip |
| BERT-Large, Uncased | https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-24_H-1024_A-16.zip |
| BERT-Base, Cased | https://storage.googleapis.com/bert_models/2018_10_18/cased_L-12_H-768_A-12.zip |
| https://github.com/Machine-Learning-Research/bert#fine-tuning-with-bert |
| out-of-memory issues | https://github.com/Machine-Learning-Research/bert#out-of-memory-issues |
| https://github.com/Machine-Learning-Research/bert#fine-tuning-with-cloud-tpus |
| Google Cloud TPU tutorial | https://cloud.google.com/tpu/docs/tutorials/mnist |
| BERT FineTuning with Cloud TPUs | https://colab.research.google.com/github/tensorflow/tpu/blob/master/tools/colab/bert_finetuning_with_cloud_tpus.ipynb |
| https://github.com/Machine-Learning-Research/bert#sentence-and-sentence-pair-classification-tasks |
| GLUE data | https://gluebenchmark.com/tasks |
| this script | https://gist.github.com/W4ngatang/60c2bdb54d156a41194446737ce03e2e |
| https://github.com/Machine-Learning-Research/bert#squad |
| SQuAD website | https://rajpurkar.github.io/SQuAD-explorer/ |
| train-v1.1.json | https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v1.1.json |
| dev-v1.1.json | https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v1.1.json |
| evaluate-v1.1.py | https://github.com/allenai/bi-att-flow/blob/master/squad/evaluate-v1.1.py |
| TriviaQA | http://nlp.cs.washington.edu/triviaqa/ |
| https://github.com/Machine-Learning-Research/bert#out-of-memory-issues |
| Gradient checkpointing | https://github.com/openai/gradient-checkpointing |
| https://github.com/Machine-Learning-Research/bert#using-bert-to-extract-fixed-feature-vectors-like-elmo |
| Tokenization | https://github.com/Machine-Learning-Research/bert#tokenization |
| https://github.com/Machine-Learning-Research/bert#tokenization |
| WordPiece | https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/data_generators/text_encoder.py |
| https://github.com/Machine-Learning-Research/bert#pre-training-with-bert |
| spaCy | https://spacy.io/ |
| https://github.com/Machine-Learning-Research/bert#pre-training-tips-and-caveats |
| preemptible Cloud TPU v2 | https://cloud.google.com/tpu/docs/pricing |
| https://github.com/Machine-Learning-Research/bert#pre-training-data |
| the latest dump | https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2 |
| WikiExtractor.py | https://github.com/attardi/wikiextractor |
| BookCorpus | http://yknzhu.wixsite.com/mbweb |
| Project Guttenberg Dataset | https://web.eecs.umich.edu/~lahiri/gutenberg_dataset.html |
| Common Crawl | http://commoncrawl.org/ |
| https://github.com/Machine-Learning-Research/bert#learning-a-new-wordpiece-vocabulary |
| Google's SentencePiece library | https://github.com/google/sentencepiece |
| tensor2tensor's WordPiece generation script | https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/data_generators/text_encoder_build_subword.py |
| Rico Sennrich's Byte Pair Encoding library | https://github.com/rsennrich/subword-nmt |
| https://github.com/Machine-Learning-Research/bert#using-bert-in-colab |
| Colab | https://colab.research.google.com |
| BERT FineTuning with Cloud TPUs | https://colab.research.google.com/github/tensorflow/tpu/blob/master/tools/colab/bert_finetuning_with_cloud_tpus.ipynb |
| https://github.com/Machine-Learning-Research/bert#faq |
| https://github.com/Machine-Learning-Research/bert#is-this-code-compatible-with-cloud-tpus-what-about-gpus |
| https://github.com/Machine-Learning-Research/bert#i-am-getting-out-of-memory-errors-what-is-wrong |
| out-of-memory issues | https://github.com/Machine-Learning-Research/bert#out-of-memory-issues |
| https://github.com/Machine-Learning-Research/bert#is-there-a-pytorch-version-available |
| https://github.com/Machine-Learning-Research/bert#will-models-in-other-languages-be-released |
| https://github.com/Machine-Learning-Research/bert#will-models-larger-than-bert-large-be-released |
| https://github.com/Machine-Learning-Research/bert#what-license-is-this-library-released-under |
| https://github.com/Machine-Learning-Research/bert#how-do-i-cite-bert |
| the Arxiv paper | https://arxiv.org/abs/1810.04805 |
| https://github.com/Machine-Learning-Research/bert#disclaimer |
| https://github.com/Machine-Learning-Research/bert#contact-information |
| arxiv.org/abs/1810.04805 | https://arxiv.org/abs/1810.04805 |
| Readme | https://github.com/Machine-Learning-Research/bert#readme-ov-file |
| Apache-2.0 license | https://github.com/Machine-Learning-Research/bert#Apache-2.0-1-ov-file |
| Contributing | https://github.com/Machine-Learning-Research/bert#contributing-ov-file |
| Activity | https://github.com/Machine-Learning-Research/bert/activity |
| Custom properties | https://github.com/Machine-Learning-Research/bert/custom-properties |
| 0 forks | https://github.com/Machine-Learning-Research/bert/forks |
| Report repository | https://github.com/contact/report-content?content_url=https%3A%2F%2Fgithub.com%2FMachine-Learning-Research%2Fbert&report=Machine-Learning-Research+%28user%29 |
|
| https://github.com |
| Terms | https://docs.github.com/site-policy/github-terms/github-terms-of-service |
| Privacy | https://docs.github.com/site-policy/privacy-policies/github-privacy-statement |
| Security | https://github.com/security |
| Status | https://www.githubstatus.com/ |
| Community | https://github.community/ |
| Docs | https://docs.github.com/ |
| Contact | https://support.github.com?tags=dotcom-footer |