René's URL Explorer Experiment

    {
       "@context": "https://schema.org",
       "@type": "Article",
       "name": "\uae30\ucd08\ubd80\ud130 \uc2dc\uc791\ud558\ub294 NLP: Sequence to Sequence \ub124\ud2b8\uc6cc\ud06c\uc640 Attention\uc744 \uc774\uc6a9\ud55c \ubc88\uc5ed",
       "headline": "\uae30\ucd08\ubd80\ud130 \uc2dc\uc791\ud558\ub294 NLP: Sequence to Sequence \ub124\ud2b8\uc6cc\ud06c\uc640 Attention\uc744 \uc774\uc6a9\ud55c \ubc88\uc5ed",
       "description": "PyTorch Documentation. Explore PyTorch, an open-source machine learning library that accelerates the path from research prototyping to production deployment. Discover tutorials, API references, and guides to help you build and deploy deep learning models efficiently.",
       "url": "/intermediate/seq2seq_translation_tutorial.html",
       "articleBody": "\ucc38\uace0 Go to the end to download the full example code. \uae30\ucd08\ubd80\ud130 \uc2dc\uc791\ud558\ub294 NLP: Sequence to Sequence \ub124\ud2b8\uc6cc\ud06c\uc640 Attention\uc744 \uc774\uc6a9\ud55c \ubc88\uc5ed# Author: Sean Robertson\ubc88\uc5ed: \ud669\uc131\uc218 \uc774 \ud29c\ud1a0\ub9ac\uc5bc\uc740 3\ubd80\ub85c \uad6c\uc131\ub41c \uc2dc\ub9ac\uc988\uc758 \uc77c\ubd80\uc785\ub2c8\ub2e4: \uae30\ucd08\ubd80\ud130 \uc2dc\uc791\ud558\ub294 NLP: \ubb38\uc790-\ub2e8\uc704 RNN\uc73c\ub85c \uc774\ub984 \ubd84\ub958\ud558\uae30 \uae30\ucd08\ubd80\ud130 \uc2dc\uc791\ud558\ub294 NLP: \ubb38\uc790-\ub2e8\uc704 RNN\uc73c\ub85c \uc774\ub984 \uc0dd\uc131\ud558\uae30 \uae30\ucd08\ubd80\ud130 \uc2dc\uc791\ud558\ub294 NLP: Sequence to Sequence \ub124\ud2b8\uc6cc\ud06c\uc640 Attention\uc744 \uc774\uc6a9\ud55c \ubc88\uc5ed \uc774 \ud29c\ud1a0\ub9ac\uc5bc\uc740 \u201c\uae30\ucd08\ubd80\ud130 \uc2dc\uc791\ud558\ub294 NLP\u201d\uc758 \uc138\ubc88\uc9f8\uc774\uc790 \ub9c8\uc9c0\ub9c9 \ud3b8\uc73c\ub85c, NLP \ubaa8\ub378\ub9c1 \uc791\uc5c5\uc744 \uc704\ud55c \ub370\uc774\ud130 \uc804\ucc98\ub9ac\uc5d0 \uc0ac\uc6a9\ud560 \uc790\uccb4 \ud074\ub798\uc2a4\uc640 \ud568\uc218\ub4e4\uc744 \uc791\uc131\ud574\ubcf4\uaca0\uc2b5\ub2c8\ub2e4. \uc774 \ud504\ub85c\uc81d\ud2b8\uc5d0\uc11c\ub294 \uc2e0\uacbd\ub9dd\uc774 \ubd88\uc5b4\ub97c \uc601\uc5b4\ub85c \ubc88\uc5ed\ud558\ub3c4\ub85d \uac00\ub974\uce60 \uc608\uc815\uc785\ub2c8\ub2e4. [KEY: \u003e input, = target, \u003c output] \u003e il est en train de peindre un tableau . = he is painting a picture . \u003c he is painting a picture . \u003e pourquoi ne pas essayer ce vin delicieux ? = why not try that delicious wine ? \u003c why not try that delicious wine ? \u003e elle n est pas poete mais romanciere . = she is not a poet but a novelist . \u003c she not not a poet but a novelist . \u003e vous etes trop maigre . = you re too skinny . \u003c you re all alone . \u2026 \uc131\uacf5\uc728\uc740 \ub2ec\ub77c\uc9c8 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \ud558\ub098\uc758 \uc2dc\ud000\uc2a4\ub97c \ub2e4\ub978 \uc2dc\ud000\uc2a4\ub85c \ubc14\uafb8\ub294 \ub450 \uac1c\uc758 RNN\uc774 \ud568\uaed8 \ub3d9\uc791\ud558\ub294 sequence to sequence network \uc758 \uac04\ub2e8\ud558\uc9c0\ub9cc \uac15\ub825\ud55c \uc544\uc774\ub514\uc5b4\uac00 \uc774\uac83(\ubc88\uc5ed)\uc744 \uac00\ub2a5\ud558\uac8c \ud569\ub2c8\ub2e4. \uc778\ucf54\ub354 \ub124\ud2b8\uc6cc\ud06c\ub294 \uc785\ub825 \uc2dc\ud000\uc2a4\ub97c \ubca1\ud130\ub85c \uc555\ucd95\ud558\uace0, \ub514\ucf54\ub354 \ub124\ud2b8\uc6cc\ud06c\ub294 \ud574\ub2f9 \ubca1\ud130\ub97c \uc0c8\ub85c\uc6b4 \uc2dc\ud000\uc2a4\ub85c \ud3bc\uce69\ub2c8\ub2e4. \uc774 \ubaa8\ub378\uc744 \uac1c\uc120\ud558\uae30 \uc704\ud574 Attention Mechanism \uc744 \uc0ac\uc6a9\ud558\uba74 \ub514\ucf54\ub354\uac00 \uc785\ub825 \uc2dc\ud000\uc2a4\uc758 \ud2b9\uc815 \ubc94\uc704\uc5d0 \uc9d1\uc911\ud560 \uc218 \uc788\ub3c4\ub85d \ud569\ub2c8\ub2e4. \ucd94\ucc9c \uc790\ub8cc: \ucd5c\uc18c\ud55c Pytorch\ub97c \uc124\uce58\ud588\uace0, Python\uc744 \uc54c\uace0, Tensor\ub97c \uc774\ud574\ud55c\ub2e4\uace0 \uac00\uc815\ud569\ub2c8\ub2e4.: http://pytorch.org/ \uc124\uce58 \uc548\ub0b4\ub97c \uc704\ud55c \uc790\ub8cc PyTorch\ub85c \ub525\ub7ec\ub2dd\ud558\uae30: 60\ubd84\ub9cc\uc5d0 \ub05d\uc7a5\ub0b4\uae30 \uc77c\ubc18\uc801\uc778 PyTorch \uc2dc\uc791\uc744 \uc704\ud55c \uc790\ub8cc \uc608\uc81c\ub85c \ubc30\uc6b0\ub294 \ud30c\uc774\ud1a0\uce58(PyTorch) \ub113\uace0 \uae4a\uc740 \ud1b5\ucc30\uc744 \uc704\ud55c \uc790\ub8cc Torch \uc0ac\uc6a9\uc790\ub97c \uc704\ud55c PyTorch \uc774\uc804 Lua Torch \uc0ac\uc6a9\uc790\ub97c \uc704\ud55c \uc790\ub8cc Sequence to Sequence \ub124\ud2b8\uc6cc\ud06c\uc640 \ub3d9\uc791 \ubc29\ubc95\uc5d0 \uad00\ud574\uc11c \uc544\ub294 \uac83\uc740 \uc720\uc6a9\ud569\ub2c8\ub2e4: Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation Sequence to Sequence Learning with Neural Networks Neural Machine Translation by Jointly Learning to Align and Translate A Neural Conversational Model \uc774\uc804 \ud29c\ud1a0\ub9ac\uc5bc\uc5d0 \uc788\ub294 \uae30\ucd08\ubd80\ud130 \uc2dc\uc791\ud558\ub294 NLP: \ubb38\uc790-\ub2e8\uc704 RNN\uc73c\ub85c \uc774\ub984 \ubd84\ub958\ud558\uae30 \uc640 \uae30\ucd08\ubd80\ud130 \uc2dc\uc791\ud558\ub294 NLP: \ubb38\uc790-\ub2e8\uc704 RNN\uc73c\ub85c \uc774\ub984 \uc0dd\uc131\ud558\uae30 \ub294 \uac01\uac01 \uc778\ucf54\ub354, \ub514\ucf54\ub354 \ubaa8\ub378\uacfc \ube44\uc2b7\ud55c \ucee8\uc13c\uc744 \uac00\uc9c0\uae30 \ub54c\ubb38\uc5d0 \ub3c4\uc6c0\uc774 \ub429\ub2c8\ub2e4. \uc694\uad6c \uc0ac\ud56d from __future__ import unicode_literals, print_function, division from io import open import unicodedata import re import random import torch import torch.nn as nn from torch import optim import torch.nn.functional as F import numpy as np from torch.utils.data import TensorDataset, DataLoader, RandomSampler device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\") \ub370\uc774\ud130 \ud30c\uc77c \ubd88\ub7ec\uc624\uae30# \uc774 \ud504\ub85c\uc81d\ud2b8\uc758 \ub370\uc774\ud130\ub294 \uc218\ucc9c \uac1c\uc758 \uc601\uc5b4-\ud504\ub791\uc2a4\uc5b4 \ubc88\uc5ed \uc30d\uc785\ub2c8\ub2e4. Open Data Stack Exchange \uc5d0 \uad00\ud55c \uc774 \uc9c8\ubb38\uc740 https://tatoeba.org/eng/downloads \uc5d0\uc11c \ub2e4\uc6b4 \ub85c\ub4dc\uac00 \uac00\ub2a5\ud55c \uacf5\uac1c \ubc88\uc5ed \uc0ac\uc774\ud2b8 https://tatoeba.org/ \ub97c \uc54c\ub824 \uc8fc\uc5c8\uc2b5\ub2c8\ub2e4. \ub354 \ub098\uc740 \ubc29\ubc95\uc73c\ub85c \uc5b8\uc5b4 \uc30d\uc744 \uac1c\ubcc4 \ud14d\uc2a4\ud2b8 \ud30c\uc77c\ub85c \ubd84\ud560\ud558\ub294 \ucd94\uac00 \uc791\uc5c5\uc744 \uc218\ud589\ud55c https://www.manythings.org/anki/ \uac00 \uc788\uc2b5\ub2c8\ub2e4: \uc601\uc5b4-\ud504\ub791\uc2a4\uc5b4 \uc30d\uc774 \ub108\ubb34 \ucee4\uc11c \uc800\uc7a5\uc18c\uc5d0 \ud3ec\ud568 \ud560 \uc218 \uc5c6\uae30 \ub54c\ubb38\uc5d0 \uacc4\uc18d\ud558\uae30 \uc804\uc5d0 data/eng-fra.txt \ub85c \ub2e4\uc6b4\ub85c\ub4dc\ud558\uc2ed\uc2dc\uc624. \uc774 \ud30c\uc77c\uc740 \ud0ed\uc73c\ub85c \uad6c\ubd84\ub41c \ubc88\uc5ed \uc30d \ubaa9\ub85d\uc785\ub2c8\ub2e4: I am cold. J\u0027ai froid. \ucc38\uace0 \uc5ec\uae30 \uc5d0\uc11c \ub370\uc774\ud130\ub97c \ub2e4\uc6b4 \ubc1b\uace0 \ud604\uc7ac \ub514\ub809\ud1a0\ub9ac\uc5d0 \uc555\ucd95\uc744 \ud478\uc2ed\uc2dc\uc624. \ubb38\uc790 \ub2e8\uc704 RNN \ud29c\ud1a0\ub9ac\uc5bc\uc5d0\uc11c \uc0ac\uc6a9\ub41c \ubb38\uc790 \uc778\ucf54\ub529\uacfc \uc720\uc0ac\ud558\uac8c, \uc5b8\uc5b4\uc758 \uac01 \ub2e8\uc5b4\ub4e4\uc744 One-Hot \ubca1\ud130 \ub610\ub294 \uadf8 \ub2e8\uc5b4\uc758 \uc8fc\uc18c\uc5d0\ub9cc \ub2e8 \ud558\ub098\uc758 1\uc744 \uc81c\uc678\ud558\uace0 \ubaa8\ub450 0\uc778 \ud070 \ubca1\ud130\ub85c \ud45c\ud604\ud569\ub2c8\ub2e4. \ud55c \uac00\uc9c0 \uc5b8\uc5b4\uc5d0 \uc788\ub294 \uc218\uc2ed \uac1c\uc758 \ubb38\uc790\uc640 \ub2ec\ub9ac \ubc88\uc5ed\uc5d0\ub294 \uc544\uc8fc \ub9ce\uc740 \ub2e8\uc5b4\ub4e4\uc774 \uc788\uae30 \ub54c\ubb38\uc5d0 \uc778\ucf54\ub529 \ubca1\ud130\ub294 \ub9e4\uc6b0 \ub354 \ud07d\ub2c8\ub2e4. \uadf8\ub7ec\ub098 \uc6b0\ub9ac\ub294 \uc57d\uac04\uc758 \ud2b8\ub9ad\ub97c \uc368\uc11c \uc5b8\uc5b4 \ub2f9 \uc218\ucc9c \ub2e8\uc5b4 \ub9cc \uc0ac\uc6a9\ud558\ub3c4\ub85d \ub370\uc774\ud130\ub97c \ub2e4\ub4ec\uc744 \uac83\uc785\ub2c8\ub2e4. \ub098\uc911\uc5d0 \ub124\ud2b8\uc6cc\ud06c\uc758 \uc785\ub825 \ubc0f \ubaa9\ud45c\ub85c \uc0ac\uc6a9\ud558\ub824\uba74 \ub2e8\uc5b4 \ub2f9 \uace0\uc720 \ubc88\ud638\uac00 \ud544\uc694\ud569\ub2c8\ub2e4. \uc774 \ubaa8\ub4e0 \uac83\uc744 \ucd94\uc801\ud558\uae30 \uc704\ud574 \uc6b0\ub9ac\ub294 \ub2e8\uc5b4\u2192\uc0c9\uc778(word2index)\uacfc \uc0c9\uc778\u2192\ub2e8\uc5b4(index2word) \uc0ac\uc804, \uadf8\ub9ac\uace0 \ub098\uc911\uc5d0 \ud76c\uadc0 \ub2e8\uc5b4\ub97c \ub300\uccb4\ud558\ub294\ub370 \uc0ac\uc6a9\ud560 \uac01 \ub2e8\uc5b4\uc758 \ube48\ub3c4 word2count \ub97c \uac00\uc9c4 Lang \uc774\ub77c\ub294 \ud5ec\ud37c \ud074\ub798\uc2a4\ub97c \uc0ac\uc6a9\ud569\ub2c8\ub2e4. SOS_token = 0 EOS_token = 1 class Lang: def __init__(self, name): self.name = name self.word2index = {} self.word2count = {} self.index2word = {0: \"SOS\", 1: \"EOS\"} self.n_words = 2 # SOS \uc640 EOS \ud3ec\ud568 def addSentence(self, sentence): for word in sentence.split(\u0027 \u0027): self.addWord(word) def addWord(self, word): if word not in self.word2index: self.word2index[word] = self.n_words self.word2count[word] = 1 self.index2word[self.n_words] = word self.n_words += 1 else: self.word2count[word] += 1 \ud30c\uc77c\uc740 \ubaa8\ub450 \uc720\ub2c8 \ucf54\ub4dc\ub85c \ub418\uc5b4\uc788\uc5b4 \uac04\ub2e8\ud558\uac8c \ud558\uae30 \uc704\ud574 \uc720\ub2c8 \ucf54\ub4dc \ubb38\uc790\ub97c ASCII\ub85c \ubcc0\ud658\ud558\uace0, \ubaa8\ub4e0 \ubb38\uc790\ub97c \uc18c\ubb38\uc790\ub85c \ub9cc\ub4e4\uace0, \ub300\ubd80\ubd84\uc758 \uad6c\ub450\uc810\uc744 \uc9c0\uc6cc\uc90d\ub2c8\ub2e4. # \uc720\ub2c8 \ucf54\ub4dc \ubb38\uc790\uc5f4\uc744 \uc77c\ubc18 ASCII\ub85c \ubcc0\ud658\ud558\uc2ed\uc2dc\uc624. # https://stackoverflow.com/a/518232/2809427 def unicodeToAscii(s): return \u0027\u0027.join( c for c in unicodedata.normalize(\u0027NFD\u0027, s) if unicodedata.category(c) != \u0027Mn\u0027 ) # \uc18c\ubb38\uc790, \ub2e4\ub4ec\uae30, \uadf8\ub9ac\uace0 \ubb38\uc790\uac00 \uc544\ub2cc \ubb38\uc790 \uc81c\uac70 def normalizeString(s): s = unicodeToAscii(s.lower().strip()) s = re.sub(r\"([.!?])\", r\" \\1\", s) s = re.sub(r\"[^a-zA-Z!?]+\", r\" \", s) return s.strip() To read the data file we will split the file into lines, and then split lines into pairs. The files are all English \u2192 Other Language, so if we want to translate from Other Language \u2192 English I added the reverse flag to reverse the pairs. def readLangs(lang1, lang2, reverse=False): print(\"Reading lines...\") # \ud30c\uc77c\uc744 \uc77d\uace0 \uc904\ub85c \ubd84\ub9ac lines = open(\u0027data/%s-%s.txt\u0027 % (lang1, lang2), encoding=\u0027utf-8\u0027).\\ read().strip().split(\u0027\\n\u0027) # \ubaa8\ub4e0 \uc904\uc744 \uc30d\uc73c\ub85c \ubd84\ub9ac\ud558\uace0 \uc815\uaddc\ud654 pairs = [[normalizeString(s) for s in l.split(\u0027\\t\u0027)] for l in lines] # \uc30d\uc744 \ub4a4\uc9d1\uace0, Lang \uc778\uc2a4\ud134\uc2a4 \uc0dd\uc131 if reverse: pairs = [list(reversed(p)) for p in pairs] input_lang = Lang(lang2) output_lang = Lang(lang1) else: input_lang = Lang(lang1) output_lang = Lang(lang2) return input_lang, output_lang, pairs \ub9ce\uc740 \uc608\uc81c \ubb38\uc7a5\uc774 \uc788\uace0 \uc2e0\uc18d\ud558\uac8c \ud559\uc2b5\ud558\uae30\ub97c \uc6d0\ud558\uae30 \ub54c\ubb38\uc5d0 \ube44\uad50\uc801 \uc9e7\uace0 \uac04\ub2e8\ud55c \ubb38\uc7a5\uc73c\ub85c\ub9cc \ub370\uc774\ud130 \uc14b\uc744 \uc815\ub9ac\ud560 \uac83\uc785\ub2c8\ub2e4. \uc5ec\uae30\uc11c \ucd5c\ub300 \uae38\uc774\ub294 10 \ub2e8\uc5b4 (\uc885\ub8cc \ubb38\uc7a5 \ubd80\ud638 \ud3ec\ud568)\uc774\uba70 \u201cI am\u201d \ub610\ub294 \u201cHe is\u201d \ub4f1\uc758 \ud615\ud0dc\ub85c \ubc88\uc5ed\ub418\ub294 \ubb38\uc7a5\uc73c\ub85c \ud544\ud130\ub9c1\ub429\ub2c8\ub2e4.(\uc774\uc804\uc5d0 \uc544\ud3ec\uc2a4\ud2b8\ub85c\ud53c\ub294 \ub300\uccb4 \ub428) MAX_LENGTH = 10 eng_prefixes = ( \"i am \", \"i m \", \"he is\", \"he s \", \"she is\", \"she s \", \"you are\", \"you re \", \"we are\", \"we re \", \"they are\", \"they re \" ) def filterPair(p): return len(p[0].split(\u0027 \u0027)) \u003c MAX_LENGTH and \\ len(p[1].split(\u0027 \u0027)) \u003c MAX_LENGTH and \\ p[1].startswith(eng_prefixes) def filterPairs(pairs): return [pair for pair in pairs if filterPair(pair)] \ub370\uc774\ud130 \uc900\ube44\ub97c \uc704\ud55c \uc804\uccb4 \uacfc\uc815: \ud14d\uc2a4\ud2b8 \ud30c\uc77c\uc744 \uc77d\uace0 \uc904\ub85c \ubd84\ub9ac\ud558\uace0, \uc904\uc744 \uc30d\uc73c\ub85c \ubd84\ub9ac\ud569\ub2c8\ub2e4. \ud14d\uc2a4\ud2b8\ub97c \uc815\uaddc\ud654 \ud558\uace0 \uae38\uc774\uc640 \ub0b4\uc6a9\uc73c\ub85c \ud544\ud130\ub9c1 \ud569\ub2c8\ub2e4. \uc30d\uc744 \uc774\ub8ec \ubb38\uc7a5\ub4e4\ub85c \ub2e8\uc5b4 \ub9ac\uc2a4\ud2b8\ub97c \uc0dd\uc131\ud569\ub2c8\ub2e4. def prepareData(lang1, lang2, reverse=False): input_lang, output_lang, pairs = readLangs(lang1, lang2, reverse) print(\"Read %s sentence pairs\" % len(pairs)) pairs = filterPairs(pairs) print(\"Trimmed to %s sentence pairs\" % len(pairs)) print(\"Counting words...\") for pair in pairs: input_lang.addSentence(pair[0]) output_lang.addSentence(pair[1]) print(\"Counted words:\") print(input_lang.name, input_lang.n_words) print(output_lang.name, output_lang.n_words) return input_lang, output_lang, pairs input_lang, output_lang, pairs = prepareData(\u0027eng\u0027, \u0027fra\u0027, True) print(random.choice(pairs)) Reading lines... Read 135842 sentence pairs Trimmed to 11445 sentence pairs Counting words... Counted words: fra 4601 eng 2991 [\u0027elle vient de france\u0027, \u0027she is from france\u0027] Seq2Seq \ubaa8\ub378# Recurrent Neural Network(RNN)\ub294 \uc2dc\ud000\uc2a4\uc5d0\uc11c \uc791\ub3d9\ud558\uace0 \ub2e4\uc74c \ub2e8\uacc4\uc758 \uc785\ub825\uc73c\ub85c \uc790\uc2e0\uc758 \ucd9c\ub825\uc744 \uc0ac\uc6a9\ud558\ub294 \ub124\ud2b8\uc6cc\ud06c\uc785\ub2c8\ub2e4. Sequence to Sequence network, \ub610\ub294 Seq2Seq \ub124\ud2b8\uc6cc\ud06c, \ub610\ub294 Encoder Decoder network \ub294 \uc778\ucf54\ub354 \ubc0f \ub514\ucf54\ub354\ub77c\uace0 \ud558\ub294 \ub450 \uac1c\uc758 RNN\uc73c\ub85c \uad6c\uc131\ub41c \ubaa8\ub378\uc785\ub2c8\ub2e4. \uc778\ucf54\ub354\ub294 \uc785\ub825 \uc2dc\ud000\uc2a4\ub97c \uc77d\uace0 \ub2e8\uc77c \ubca1\ud130\ub97c \ucd9c\ub825\ud558\uace0, \ub514\ucf54\ub354\ub294 \ud574\ub2f9 \ubca1\ud130\ub97c \uc77d\uc5b4 \ucd9c\ub825 \uc2dc\ud000\uc2a4\ub97c \uc0dd\uc131\ud569\ub2c8\ub2e4. \ubaa8\ub4e0 \uc785\ub825\uc5d0 \ud574\ub2f9\ud558\ub294 \ucd9c\ub825\uc774 \uc788\ub294 \ub2e8\uc77c RNN\uc758 \uc2dc\ud000\uc2a4 \uc608\uce21\uacfc \ub2ec\ub9ac Seq2Seq \ubaa8\ub378\uc740 \uc2dc\ud000\uc2a4 \uae38\uc774\uc640 \uc21c\uc11c\ub97c \uc790\uc720\ub86d\uac8c\ud558\uae30 \ub54c\ubb38\uc5d0 \ub450 \uc5b8\uc5b4 \uc0ac\uc774\uc758 \ubc88\uc5ed\uc5d0 \uc774\uc0c1\uc801\uc785\ub2c8\ub2e4. \ub2e4\uc74c \ubb38\uc7a5 Je ne suis pas le chat noir \u2192 I am not the black cat \ub97c \uc0b4\ud3b4 \ubd05\uc2dc\ub2e4. \uc785\ub825 \ubb38\uc7a5\uc758 \ub2e8\uc5b4 \ub300\ubd80\ubd84\uc740 \ucd9c\ub825 \ubb38\uc7a5\uc5d0\uc11c \uc9c1\uc5ed(chat noir \uc640 black cat)\ub418\uc9c0\ub9cc \uc57d\uac04 \ub2e4\ub978 \uc21c\uc11c\ub3c4 \uc788\uc2b5\ub2c8\ub2e4. ne/pas \uad6c\uc870\ub85c \uc778\ud574 \uc785\ub825 \ubb38\uc7a5\uc5d0 \ub2e8\uc5b4\uac00 \ud558\ub098 \ub354 \uc788\uc2b5\ub2c8\ub2e4. \uc785\ub825 \ub2e8\uc5b4\uc758 \uc2dc\ud000\uc2a4\ub97c \uc9c1\uc5ed\ud574\uc11c \uc815\ud655\ud55c \ubc88\uc5ed\uc744 \ub9cc\ub4dc\ub294 \uac83\uc740 \uc5b4\ub824\uc6b8 \uac83\uc785\ub2c8\ub2e4. Seq2Seq \ubaa8\ub378\uc744 \uc0ac\uc6a9\ud558\uba74 \uc778\ucf54\ub354\ub294 \ud558\ub098\uc758 \ubca1\ud130\ub97c \uc0dd\uc131\ud569\ub2c8\ub2e4. \uc774\uc0c1\uc801\uc778 \uacbd\uc6b0\uc5d0 \uc785\ub825 \uc2dc\ud000\uc2a4\uc758 \u201c\uc758\ubbf8\u201d\ub97c \ubb38\uc7a5\uc758 N \ucc28\uc6d0 \uacf5\uac04\uc5d0 \uc788\ub294 \ub2e8\uc77c \uc9c0\uc810\uc778 \ub2e8\uc77c \ubca1\ud130\uc73c\ub85c \uc778\ucf54\ub529\ud569\ub2c8\ub2e4. \uc778\ucf54\ub354# Seq2Seq \ub124\ud2b8\uc6cc\ud06c\uc758 \uc778\ucf54\ub354\ub294 \uc785\ub825 \ubb38\uc7a5\uc758 \ubaa8\ub4e0 \ub2e8\uc5b4\uc5d0 \ub300\ud574 \uc5b4\ub5a4 \uac12\uc744 \ucd9c\ub825\ud558\ub294 RNN\uc785\ub2c8\ub2e4. \ubaa8\ub4e0 \uc785\ub825 \ub2e8\uc5b4\uc5d0 \ub300\ud574 \uc778\ucf54\ub354\ub294 \ubca1\ud130\uc640 \uc740\ub2c9 \uc0c1\ud0dc\ub97c \ucd9c\ub825\ud558\uace0 \ub2e4\uc74c \uc785\ub825 \ub2e8\uc5b4\ub97c \uc704\ud574 \uadf8 \uc740\ub2c9 \uc0c1\ud0dc\ub97c \uc0ac\uc6a9\ud569\ub2c8\ub2e4. class EncoderRNN(nn.Module): def __init__(self, input_size, hidden_size, dropout_p=0.1): super(EncoderRNN, self).__init__() self.hidden_size = hidden_size self.embedding = nn.Embedding(input_size, hidden_size) self.gru = nn.GRU(hidden_size, hidden_size, batch_first=True) self.dropout = nn.Dropout(dropout_p) def forward(self, input): embedded = self.dropout(self.embedding(input)) output, hidden = self.gru(embedded) return output, hidden \ub514\ucf54\ub354# \ub514\ucf54\ub354\ub294 \uc778\ucf54\ub354 \ucd9c\ub825 \ubca1\ud130\ub97c \ubc1b\uc544\uc11c \ubc88\uc5ed\uc744 \uc0dd\uc131\ud558\uae30 \uc704\ud55c \ub2e8\uc5b4 \uc2dc\ud000\uc2a4\ub97c \ucd9c\ub825\ud569\ub2c8\ub2e4. \uac04\ub2e8\ud55c \ub514\ucf54\ub354# \uac00\uc7a5 \uac04\ub2e8\ud55c Seq2Seq \ub514\ucf54\ub354\ub294 \uc778\ucf54\ub354\uc758 \ub9c8\uc9c0\ub9c9 \ucd9c\ub825\ub9cc\uc744 \uc774\uc6a9\ud569\ub2c8\ub2e4. \uc774 \ub9c8\uc9c0\ub9c9 \ucd9c\ub825\uc740 \uc804\uccb4 \uc2dc\ud000\uc2a4\uc5d0\uc11c \ubb38\ub9e5\uc744 \uc778\ucf54\ub4dc\ud558\uae30 \ub54c\ubb38\uc5d0 \ubb38\ub9e5 \ubca1\ud130(context vector) \ub85c \ubd88\ub9bd\ub2c8\ub2e4. \uc774 \ubb38\ub9e5 \ubca1\ud130\ub294 \ub514\ucf54\ub354\uc758 \ucd08\uae30 \uc740\ub2c9 \uc0c1\ud0dc\ub85c \uc0ac\uc6a9 \ub429\ub2c8\ub2e4. \ub514\ucf54\ub529\uc758 \ub9e4 \ub2e8\uacc4\uc5d0\uc11c \ub514\ucf54\ub354\uc5d0\uac8c \uc785\ub825 \ud1a0\ud070\uacfc \uc740\ub2c9 \uc0c1\ud0dc\uac00 \uc8fc\uc5b4\uc9d1\ub2c8\ub2e4. \ucd08\uae30 \uc785\ub825 \ud1a0\ud070\uc740 \ubb38\uc790\uc5f4-\uc2dc\uc791 (start-of-string) \u003cSOS\u003e \ud1a0\ud070\uc774\uace0, \uccab \uc740\ub2c9 \uc0c1\ud0dc\ub294 \ubb38\ub9e5 \ubca1\ud130(\uc778\ucf54\ub354\uc758 \ub9c8\uc9c0\ub9c9 \uc740\ub2c9 \uc0c1\ud0dc) \uc785\ub2c8\ub2e4. class DecoderRNN(nn.Module): def __init__(self, hidden_size, output_size): super(DecoderRNN, self).__init__() self.embedding = nn.Embedding(output_size, hidden_size) self.gru = nn.GRU(hidden_size, hidden_size, batch_first=True) self.out = nn.Linear(hidden_size, output_size) def forward(self, encoder_outputs, encoder_hidden, target_tensor=None): batch_size = encoder_outputs.size(0) decoder_input = torch.empty(batch_size, 1, dtype=torch.long, device=device).fill_(SOS_token) decoder_hidden = encoder_hidden decoder_outputs = [] for i in range(MAX_LENGTH): decoder_output, decoder_hidden = self.forward_step(decoder_input, decoder_hidden) decoder_outputs.append(decoder_output) if target_tensor is not None: # Teacher forcing \ud3ec\ud568: \ubaa9\ud45c\ub97c \ub2e4\uc74c \uc785\ub825\uc73c\ub85c \uc804\ub2ec decoder_input = target_tensor[:, i].unsqueeze(1) # Teacher forcing else: # Teacher forcing \ubbf8\ud3ec\ud568: \uc790\uc2e0\uc758 \uc608\uce21\uc744 \ub2e4\uc74c \uc785\ub825\uc73c\ub85c \uc0ac\uc6a9 _, topi = decoder_output.topk(1) decoder_input = topi.squeeze(-1).detach() # \uc785\ub825\uc73c\ub85c \uc0ac\uc6a9\ud560 \ubd80\ubd84\uc744 \ud788\uc2a4\ud1a0\ub9ac\uc5d0\uc11c \ubd84\ub9ac decoder_outputs = torch.cat(decoder_outputs, dim=1) decoder_outputs = F.log_softmax(decoder_outputs, dim=-1) return decoder_outputs, decoder_hidden, None # \ud559\uc2b5 \ub8e8\ud504\uc758 \uc77c\uad00\uc131 \uc720\uc9c0\ub97c \uc704\ud574 `None` \uc744 \ucd94\uac00\ub85c \ubc18\ud658 def forward_step(self, input, hidden): output = self.embedding(input) output = F.relu(output) output, hidden = self.gru(output, hidden) output = self.out(output) return output, hidden \uc774 \ubaa8\ub378\uc758 \uacb0\uacfc\ub97c \ud559\uc2b5\ud558\uace0 \uad00\ucc30\ud558\ub294 \uac83\uc744 \uad8c\uc7a5\ud558\uc9c0\ub9cc, \uacf5\uac04\uc744 \uc808\uc57d\ud558\uae30 \uc704\ud574 \ucd5c\uc885 \ubaa9\uc801\uc9c0\ub85c \ubc14\ub85c \uc774\ub3d9\ud574\uc11c Attention \uba54\ucee4\ub2c8\uc998\uc744 \uc18c\uac1c \ud560 \uac83\uc785\ub2c8\ub2e4. Attention \ub514\ucf54\ub354# \ubb38\ub9e5 \ubca1\ud130\ub9cc \uc778\ucf54\ub354\uc640 \ub514\ucf54\ub354 \uc0ac\uc774\ub85c \uc804\ub2ec \ub41c\ub2e4\uba74, \ub2e8\uc77c \ubca1\ud130\uac00 \uc804\uccb4 \ubb38\uc7a5\uc744 \uc778\ucf54\ub529 \ud574\uc57c\ud558\ub294 \ubd80\ub2f4\uc744 \uac00\uc9c0\uac8c \ub429\ub2c8\ub2e4. Attention\uc740 \ub514\ucf54\ub354 \ub124\ud2b8\uc6cc\ud06c\uac00 \uc790\uae30 \ucd9c\ub825\uc758 \ubaa8\ub4e0 \ub2e8\uacc4\uc5d0\uc11c \uc778\ucf54\ub354 \ucd9c\ub825\uc758 \ub2e4\ub978 \ubd80\ubd84\uc5d0 \u201c\uc9d1\uc911\u201d \ud560 \uc218 \uc788\uac8c \ud569\ub2c8\ub2e4. \uccab\uc9f8 Attention \uac00\uc911\uce58 \uc758 \uc138\ud2b8\ub97c \uacc4\uc0b0\ud569\ub2c8\ub2e4. \uc774\uac83\uc740 \uac00\uc911\uce58 \uc870\ud569\uc744 \ub9cc\ub4e4\uae30 \uc704\ud574\uc11c \uc778\ucf54\ub354 \ucd9c\ub825 \ubca1\ud130\uc640 \uacf1\ud574\uc9d1\ub2c8\ub2e4. \uadf8 \uacb0\uacfc(\ucf54\ub4dc\uc5d0\uc11c attn_applied)\ub294 \uc785\ub825 \uc2dc\ud000\uc2a4\uc758 \ud2b9\uc815 \ubd80\ubd84\uc5d0 \uad00\ud55c \uc815\ubcf4\ub97c \ud3ec\ud568\ud574\uc57c\ud558\uace0 \ub530\ub77c\uc11c \ub514\ucf54\ub354\uac00 \uc54c\ub9de\uc740 \ucd9c\ub825 \ub2e8\uc5b4\ub97c \uc120\ud0dd\ud558\ub294 \uac83\uc744 \ub3c4\uc640\uc90d\ub2c8\ub2e4. \uc5b4\ud150\uc158 \uac00\uc911\uce58 \uacc4\uc0b0\uc740 \ub514\ucf54\ub354\uc758 \uc785\ub825 \ubc0f \uc740\ub2c9 \uc0c1\ud0dc\ub97c \uc785\ub825\uc73c\ub85c \uc0ac\uc6a9\ud558\ub294 \ub2e4\ub978 feed-forwad \uacc4\uce35\uc778 attn \uc73c\ub85c \uc218\ud589\ub429\ub2c8\ub2e4. \ud559\uc2b5 \ub370\uc774\ud130\uc5d0\ub294 \ubaa8\ub4e0 \ud06c\uae30\uc758 \ubb38\uc7a5\uc774 \uc788\uae30 \ub54c\ubb38\uc5d0 \uc774 \uacc4\uce35\uc744 \uc2e4\uc81c\ub85c \ub9cc\ub4e4\uace0 \ud559\uc2b5\uc2dc\ud0a4\ub824\uba74 \uc801\uc6a9 \ud560 \uc218 \uc788\ub294 \ucd5c\ub300 \ubb38\uc7a5 \uae38\uc774 (\uc778\ucf54\ub354 \ucd9c\ub825\uc744 \uc704\ud55c \uc785\ub825 \uae38\uc774)\ub97c \uc120\ud0dd\ud574\uc57c \ud569\ub2c8\ub2e4. \ucd5c\ub300 \uae38\uc774\uc758 \ubb38\uc7a5\uc740 \ubaa8\ub4e0 Attention \uac00\uc911\uce58\ub97c \uc0ac\uc6a9\ud558\uc9c0\ub9cc \ub354 \uc9e7\uc740 \ubb38\uc7a5\uc740 \ucc98\uc74c \uba87 \uac1c\ub9cc \uc0ac\uc6a9\ud569\ub2c8\ub2e4. \ubd80\uac00\uc801 \uc5b4\ud150\uc158(Additive Attention)\uc774\ub77c\uace0\ub3c4 \uc54c\ub824\uc9c4 \ubc14\ub2e4\ub098\uc6b0 \uc5b4\ud150\uc158(Bahdanau Attention)\uc740 \uae30\uacc4 \ubc88\uc5ed \uc791\uc5c5\uacfc \uac19\uc740 \uc2dc\ud000\uc2a4-\ud22c-\uc2dc\ud000\uc2a4 \ubaa8\ub378\uc5d0\uc11c \uc77c\ubc18\uc801\uc73c\ub85c \uc0ac\uc6a9\ud558\ub294 \uc5b4\ud150\uc158 \uae30\ubc95(mechanism)\uc785\ub2c8\ub2e4. \uc774 \uc5b4\ud150\uc158 \uae30\ubc95\uc740 Bahdanau et al.\uc758 \ub17c\ubb38\uc778 Neural Machine Translation by Jointly Learning to Align and Translate \uc5d0\uc11c \uc18c\uac1c\ub418\uc5c8\uc2b5\ub2c8\ub2e4. \uc774 \uc5b4\ud150\uc158 \uae30\ubc95\uc740 \ud559\uc2b5\ub41c \uc815\ub82c \ubaa8\ub378(learned alignment model)\uc744 \uc0ac\uc6a9\ud558\uc5ec \uc778\ucf54\ub354\uc640 \ub514\ucf54\ub354\uc758 \uc740\ub2c9 \uc0c1\ud0dc(hidden state) \uac04\uc758 \uc5b4\ud150\uc158 \uc810\uc218\ub97c \uacc4\uc0b0\ud569\ub2c8\ub2e4. \uc774\ub294 \uc815\ub82c\ub41c \uc5b4\ud150\uc158 \uc810\uc218\ub97c \uacc4\uc0b0\ud558\uae30 \uc704\ud574 feed-forward \uc2e0\uacbd\ub9dd\uc744 \uc0ac\uc6a9\ud569\ub2c8\ub2e4. \ub610\ub294, \ub514\ucf54\ub354\uc758 \uc740\ub2c9 \uc0c1\ud0dc\uc640 \uc778\ucf54\ub354\uc758 \uc740\ub2c9 \uc0c1\ud0dc \uc0ac\uc774\uc758 \uc5b4\ud150\uc158 \uc810\uc218\ub97c Dot-Product\ub85c \uacc4\uc0b0\ud558\ub294 \ub8e8\uc639 \uc5b4\ud150\uc158(Luong Attention)\uacfc \uac19\uc740 \ub2e4\ub978 \uc5b4\ud150\uc158 \uae30\ubc95\ub4e4\uc744 \uc0ac\uc6a9\ud560 \uc218\ub3c4 \uc788\uc2b5\ub2c8\ub2e4. \uc774\ub294 \ubc14\ub2e4\ub098\uc6b0 \uc5b4\ud150\uc158(Bahdanau Attention)\uc5d0\uc11c \uc0ac\uc6a9\ud558\ub294 \ube44\uc120\ud615 \ubcc0\ud658(non-linear transformation)\uc744 \uc0ac\uc6a9\ud558\uc9c0\ub294 \uc54a\uc2b5\ub2c8\ub2e4. \uc774 \ud29c\ud1a0\ub9ac\uc5bc\uc5d0\uc11c\ub294 \ubc14\ub2e4\ub098\uc6b0 \uc5b4\ud150\uc158(Bahdanau Attention)\uc744 \uc0ac\uc6a9\ud560 \uac83\uc785\ub2c8\ub2e4. \ud558\uc9c0\ub9cc \uc774\ub97c \ub8e8\uc639 \uc5b4\ud150\uc158(Luong Attention) \uae30\ubc95\uc73c\ub85c \ubcc0\uacbd\ud574\ubcf4\ub294 \uac83\ub3c4 \uc88b\uc740 \uc5f0\uc2b5\uc774 \ub420 \uac83\uc785\ub2c8\ub2e4. class BahdanauAttention(nn.Module): def __init__(self, hidden_size): super(BahdanauAttention, self).__init__() self.Wa = nn.Linear(hidden_size, hidden_size) self.Ua = nn.Linear(hidden_size, hidden_size) self.Va = nn.Linear(hidden_size, 1) def forward(self, query, keys): scores = self.Va(torch.tanh(self.Wa(query) + self.Ua(keys))) scores = scores.squeeze(2).unsqueeze(1) weights = F.softmax(scores, dim=-1) context = torch.bmm(weights, keys) return context, weights class AttnDecoderRNN(nn.Module): def __init__(self, hidden_size, output_size, dropout_p=0.1): super(AttnDecoderRNN, self).__init__() self.embedding = nn.Embedding(output_size, hidden_size) self.attention = BahdanauAttention(hidden_size) self.gru = nn.GRU(2 * hidden_size, hidden_size, batch_first=True) self.out = nn.Linear(hidden_size, output_size) self.dropout = nn.Dropout(dropout_p) def forward(self, encoder_outputs, encoder_hidden, target_tensor=None): batch_size = encoder_outputs.size(0) decoder_input = torch.empty(batch_size, 1, dtype=torch.long, device=device).fill_(SOS_token) decoder_hidden = encoder_hidden decoder_outputs = [] attentions = [] for i in range(MAX_LENGTH): decoder_output, decoder_hidden, attn_weights = self.forward_step( decoder_input, decoder_hidden, encoder_outputs ) decoder_outputs.append(decoder_output) attentions.append(attn_weights) if target_tensor is not None: # Teacher forcing \ud3ec\ud568: \ubaa9\ud45c\ub97c \ub2e4\uc74c \uc785\ub825\uc73c\ub85c \uc804\ub2ec decoder_input = target_tensor[:, i].unsqueeze(1) # Teacher forcing else: # Teacher forcing \ubbf8\ud3ec\ud568: \uc790\uc2e0\uc758 \uc608\uce21\uc744 \ub2e4\uc74c \uc785\ub825\uc73c\ub85c \uc0ac\uc6a9 _, topi = decoder_output.topk(1) decoder_input = topi.squeeze(-1).detach() # \uc785\ub825\uc73c\ub85c \uc0ac\uc6a9\ud560 \ubd80\ubd84\uc744 \ud788\uc2a4\ud1a0\ub9ac\uc5d0\uc11c \ubd84\ub9ac decoder_outputs = torch.cat(decoder_outputs, dim=1) decoder_outputs = F.log_softmax(decoder_outputs, dim=-1) attentions = torch.cat(attentions, dim=1) return decoder_outputs, decoder_hidden, attentions def forward_step(self, input, hidden, encoder_outputs): embedded = self.dropout(self.embedding(input)) query = hidden.permute(1, 0, 2) context, attn_weights = self.attention(query, encoder_outputs) input_gru = torch.cat((embedded, context), dim=2) output, hidden = self.gru(input_gru, hidden) output = self.out(output) return output, hidden, attn_weights \ucc38\uace0 \uae38\uc774 \uc81c\ud55c\uc744 \ud574\uacb0\ud558\uae30 \uc704\ud574 \uc0c1\ub300\uc801 \uc704\uce58 \uc811\uadfc(relative position approach) \ubc29\uc2dd\uc744 \uc0ac\uc6a9\ud558\ub294 \ub2e4\ub978 \ud615\ud0dc\uc758 \uc5b4\ud150\uc158 \ubc29\uc2dd\ub4e4\ub3c4 \uc788\uc2b5\ub2c8\ub2e4. Effective Approaches to Attention-based Neural Machine Translation \uc5d0\uc11c \u201clocal attention\u201d \uc5d0 \ub300\ud574 \uc77d\uc5b4\ubcf4\uc138\uc694. \ud559\uc2b5# \ud559\uc2b5 \ub370\uc774\ud130 \uc900\ube44# \ud559\uc2b5\uc744 \uc704\ud574\uc11c, \uac01 \uc30d\ub9c8\ub2e4 \uc785\ub825 Tensor(\uc785\ub825 \ubb38\uc7a5\uc758 \ub2e8\uc5b4 \uc8fc\uc18c)\uc640 \ubaa9\ud45c Tensor(\ubaa9\ud45c \ubb38\uc7a5\uc758 \ub2e8\uc5b4 \uc8fc\uc18c)\uac00 \ud544\uc694\ud569\ub2c8\ub2e4. \uc774 \ubca1\ud130\ub4e4\uc744 \uc0dd\uc131\ud558\ub294 \ub3d9\uc548 \ub450 \uc2dc\ud000\uc2a4\uc5d0 EOS \ud1a0\ud070\uc744 \ucd94\uac00 \ud569\ub2c8\ub2e4. def indexesFromSentence(lang, sentence): return [lang.word2index[word] for word in sentence.split(\u0027 \u0027)] def tensorFromSentence(lang, sentence): indexes = indexesFromSentence(lang, sentence) indexes.append(EOS_token) return torch.tensor(indexes, dtype=torch.long, device=device).view(1, -1) def tensorsFromPair(pair): input_tensor = tensorFromSentence(input_lang, pair[0]) target_tensor = tensorFromSentence(output_lang, pair[1]) return (input_tensor, target_tensor) def get_dataloader(batch_size): input_lang, output_lang, pairs = prepareData(\u0027eng\u0027, \u0027fra\u0027, True) n = len(pairs) input_ids = np.zeros((n, MAX_LENGTH), dtype=np.int32) target_ids = np.zeros((n, MAX_LENGTH), dtype=np.int32) for idx, (inp, tgt) in enumerate(pairs): inp_ids = indexesFromSentence(input_lang, inp) tgt_ids = indexesFromSentence(output_lang, tgt) inp_ids.append(EOS_token) tgt_ids.append(EOS_token) input_ids[idx, :len(inp_ids)] = inp_ids target_ids[idx, :len(tgt_ids)] = tgt_ids train_data = TensorDataset(torch.LongTensor(input_ids).to(device), torch.LongTensor(target_ids).to(device)) train_sampler = RandomSampler(train_data) train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=batch_size) return input_lang, output_lang, train_dataloader \ubaa8\ub378 \ud559\uc2b5# \ud559\uc2b5\uc744 \uc704\ud574\uc11c \uc778\ucf54\ub354\uc5d0 \uc785\ub825 \ubb38\uc7a5\uc744 \ub123\uace0 \ubaa8\ub4e0 \ucd9c\ub825\uacfc \ucd5c\uc2e0 \uc740\ub2c9 \uc0c1\ud0dc\ub97c \ucd94\uc801\ud569\ub2c8\ub2e4. \uadf8\ub7f0 \ub2e4\uc74c \ub514\ucf54\ub354\uc5d0 \uccab \ubc88\uc9f8 \uc785\ub825\uc73c\ub85c \u003cSOS\u003e \ud1a0\ud070\uacfc \uc778\ucf54\ub354\uc758 \ub9c8\uc9c0\ub9c9 \uc740\ub2c9 \uc0c1\ud0dc\uac00 \uccab \ubc88\uc9f8 \uc740\ub2c9 \uc0c1\ud0dc\ub85c \uc81c\uacf5\ub429\ub2c8\ub2e4. \u201cTeacher forcing\u201d\uc740 \ub2e4\uc74c \uc785\ub825\uc73c\ub85c \ub514\ucf54\ub354\uc758 \uc608\uce21\uc744 \uc0ac\uc6a9\ud558\ub294 \ub300\uc2e0 \uc2e4\uc81c \ubaa9\ud45c \ucd9c\ub825\uc744 \ub2e4\uc74c \uc785\ub825\uc73c\ub85c \uc0ac\uc6a9\ud558\ub294 \ucee8\uc149\uc785\ub2c8\ub2e4. \u201cTeacher forcing\u201d\uc744 \uc0ac\uc6a9\ud558\uba74 \uc218\ub834\uc774 \ube68\ub9ac\ub418\uc9c0\ub9cc \ud559\uc2b5\ub41c \ub124\ud2b8\uc6cc\ud06c\uac00 \uc798\ubabb \uc0ac\uc6a9\ub420 \ub54c \ubd88\uc548\uc815\uc131\uc744 \ubcf4\uc785\ub2c8\ub2e4.. Teacher-forced \ub124\ud2b8\uc6cc\ud06c\uc758 \ucd9c\ub825\uc774 \uc77c\uad00\ub41c \ubb38\ubc95\uc73c\ub85c \uc77d\uc9c0\ub9cc \uc815\ud655\ud55c \ubc88\uc5ed\uacfc\ub294 \uac70\ub9ac\uac00 \uba40\ub2e4\ub294 \uac83\uc744 \ubcfc \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uc9c1\uad00\uc801\uc73c\ub85c \ucd9c\ub825 \ubb38\ubc95\uc744 \ud45c\ud604\ud558\ub294 \ubc95\uc744 \ubc30\uc6b0\uace0 \uad50\uc0ac\uac00 \ucc98\uc74c \uba87 \ub2e8\uc5b4\ub97c \ub9d0\ud558\uba74 \uc758\ubbf8\ub97c \u201c\uc120\ud0dd\u201d \ud560 \uc218 \uc788\uc9c0\ub9cc, \ubc88\uc5ed\uc5d0\uc11c \ucc98\uc74c\uc73c\ub85c \ubb38\uc7a5\uc744 \ub9cc\ub4dc\ub294 \ubc95\uc740 \uc798 \ubc30\uc6b0\uc9c0 \ubabb\ud569\ub2c8\ub2e4. PyTorch\uc758 autograd \uac00 \uc81c\uacf5\ud558\ub294 \uc790\uc720 \ub355\ubd84\uc5d0 \uac04\ub2e8\ud55c If \ubb38\uc73c\ub85c Teacher Forcing\uc744 \uc0ac\uc6a9\ud560\uc9c0 \uc544\ub2c8\uba74 \uc0ac\uc6a9\ud558\uc9c0 \uc54a\uc744\uc9c0\ub97c \uc120\ud0dd\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \ub354 \ub9ce\uc774 \uc0ac\uc6a9\ud558\ub824\uba74 teacher_forcing_ratio \ub97c \ud655\uc778\ud558\uc2ed\uc2dc\uc624. def train_epoch(dataloader, encoder, decoder, encoder_optimizer, decoder_optimizer, criterion): total_loss = 0 for data in dataloader: input_tensor, target_tensor = data encoder_optimizer.zero_grad() decoder_optimizer.zero_grad() encoder_outputs, encoder_hidden = encoder(input_tensor) decoder_outputs, _, _ = decoder(encoder_outputs, encoder_hidden, target_tensor) loss = criterion( decoder_outputs.view(-1, decoder_outputs.size(-1)), target_tensor.view(-1) ) loss.backward() encoder_optimizer.step() decoder_optimizer.step() total_loss += loss.item() return total_loss / len(dataloader) \uc774\uac83\uc740 \ud604\uc7ac \uc2dc\uac04\uacfc \uc9c4\ud589\ub960%\uc744 \uace0\ub824\ud574 \uacbd\uacfc\ub41c \uc2dc\uac04\uacfc \ub0a8\uc740 \uc608\uc0c1 \uc2dc\uac04\uc744 \ucd9c\ub825\ud558\ub294 \ud5ec\ud37c \ud568\uc218\uc785\ub2c8\ub2e4. import time import math def asMinutes(s): m = math.floor(s / 60) s -= m * 60 return \u0027%dm %ds\u0027 % (m, s) def timeSince(since, percent): now = time.time() s = now - since es = s / (percent) rs = es - s return \u0027%s (- %s)\u0027 % (asMinutes(s), asMinutes(rs)) \uc804\uccb4 \ud559\uc2b5 \uacfc\uc815\uc740 \ub2e4\uc74c\uacfc \uac19\uc2b5\ub2c8\ub2e4: \ud0c0\uc774\uba38 \uc2dc\uc791 optimizers\uc640 criterion \ucd08\uae30\ud654 \ud559\uc2b5 \uc30d\uc758 \uc138\ud2b8 \uc0dd\uc131 \ub3c4\uc2dd\ud654\ub97c \uc704\ud55c \ube48 \uc190\uc2e4 \ubc30\uc5f4 \uc2dc\uc791 \uadf8\ub7f0 \ub2e4\uc74c \uc6b0\ub9ac\ub294 \uc5ec\ub7ec \ubc88 train \uc744 \ud638\ucd9c\ud558\uba70 \ub54c\ub85c\ub294 \uc9c4\ud589\ub960 (\uc608\uc81c\uc758 %, \ud604\uc7ac\uae4c\uc9c0\uc758 \uc608\uc0c1 \uc2dc\uac04)\uacfc \ud3c9\uade0 \uc190\uc2e4\uc744 \ucd9c\ub825\ud569\ub2c8\ub2e4. def train(train_dataloader, encoder, decoder, n_epochs, learning_rate=0.001, print_every=100, plot_every=100): start = time.time() plot_losses = [] print_loss_total = 0 # Reset every print_every plot_loss_total = 0 # Reset every plot_every encoder_optimizer = optim.Adam(encoder.parameters(), lr=learning_rate) decoder_optimizer = optim.Adam(decoder.parameters(), lr=learning_rate) criterion = nn.NLLLoss() for epoch in range(1, n_epochs + 1): loss = train_epoch(train_dataloader, encoder, decoder, encoder_optimizer, decoder_optimizer, criterion) print_loss_total += loss plot_loss_total += loss if epoch % print_every == 0: print_loss_avg = print_loss_total / print_every print_loss_total = 0 print(\u0027%s (%d %d%%) %.4f\u0027 % (timeSince(start, epoch / n_epochs), epoch, epoch / n_epochs * 100, print_loss_avg)) if epoch % plot_every == 0: plot_loss_avg = plot_loss_total / plot_every plot_losses.append(plot_loss_avg) plot_loss_total = 0 showPlot(plot_losses) \uacb0\uacfc \ub3c4\uc2dd\ud654# matplotlib\ub85c \ud559\uc2b5 \uc911\uc5d0 \uc800\uc7a5\ub41c \uc190\uc2e4 \uac12 plot_losses \uc758 \ubc30\uc5f4\uc744 \uc0ac\uc6a9\ud558\uc5ec \ub3c4\uc2dd\ud654\ud569\ub2c8\ub2e4. import matplotlib.pyplot as plt plt.switch_backend(\u0027agg\u0027) import matplotlib.ticker as ticker import numpy as np def showPlot(points): plt.figure() fig, ax = plt.subplots() # \uc8fc\uae30\uc801\uc778 \uac04\uaca9\uc73c\ub85c \uc774 locator\uac00 tick\uc744 \uc124\uc815 loc = ticker.MultipleLocator(base=0.2) ax.yaxis.set_major_locator(loc) plt.plot(points) \ud3c9\uac00# \ud3c9\uac00\ub294 \ub300\ubd80\ubd84 \ud559\uc2b5\uacfc \ub3d9\uc77c\ud558\uc9c0\ub9cc \ubaa9\ud45c\uac00 \uc5c6\uc73c\ubbc0\ub85c \uac01 \ub2e8\uacc4\ub9c8\ub2e4 \ub514\ucf54\ub354\uc758 \uc608\uce21\uc744 \ub418\ub3cc\ub824 \uc804\ub2ec\ud569\ub2c8\ub2e4. \ub2e8\uc5b4\ub97c \uc608\uce21\ud560 \ub54c\ub9c8\ub2e4 \uadf8 \ub2e8\uc5b4\ub97c \ucd9c\ub825 \ubb38\uc790\uc5f4\uc5d0 \ucd94\uac00\ud569\ub2c8\ub2e4. \ub9cc\uc57d EOS \ud1a0\ud070\uc744 \uc608\uce21\ud558\uba74 \uac70\uae30\uc5d0\uc11c \uba48\ucda5\ub2c8\ub2e4. \ub098\uc911\uc5d0 \ub3c4\uc2dd\ud654\ub97c \uc704\ud574\uc11c \ub514\ucf54\ub354\uc758 Attention \ucd9c\ub825\uc744 \uc800\uc7a5\ud569\ub2c8\ub2e4. def evaluate(encoder, decoder, sentence, input_lang, output_lang): with torch.no_grad(): input_tensor = tensorFromSentence(input_lang, sentence) encoder_outputs, encoder_hidden = encoder(input_tensor) decoder_outputs, decoder_hidden, decoder_attn = decoder(encoder_outputs, encoder_hidden) _, topi = decoder_outputs.topk(1) decoded_ids = topi.squeeze() decoded_words = [] for idx in decoded_ids: if idx.item() == EOS_token: decoded_words.append(\u0027\u003cEOS\u003e\u0027) break decoded_words.append(output_lang.index2word[idx.item()]) return decoded_words, decoder_attn \ud559\uc2b5 \uc138\ud2b8\uc5d0 \uc788\ub294 \uc784\uc758\uc758 \ubb38\uc7a5\uc73c\ub85c \ud3c9\uac00\ud55c \ub2e4\uc74c, \uc785\ub825(input), \ubaa9\ud45c(target) \ubc0f \ucd9c\ub825(output) \uac12\ub4e4\uc744 \ud45c\uc2dc\ud558\uc5ec \uc8fc\uad00\uc801\uc73c\ub85c \ud488\uc9c8\uc5d0 \ub300\ud574 \ud310\ub2e8\ud574\ubcfc \uc218 \uc788\uc2b5\ub2c8\ub2e4: def evaluateRandomly(encoder, decoder, n=10): for i in range(n): pair = random.choice(pairs) print(\u0027\u003e\u0027, pair[0]) print(\u0027=\u0027, pair[1]) output_words, _ = evaluate(encoder, decoder, pair[0], input_lang, output_lang) output_sentence = \u0027 \u0027.join(output_words) print(\u0027\u003c\u0027, output_sentence) print(\u0027\u0027) \ud559\uc2b5\uacfc \ud3c9\uac00# \uc774\ub7ec\ud55c \ubaa8\ub4e0 \ud5ec\ud37c \ud568\uc218\ub97c \uc774\uc6a9\ud574\uc11c (\ucd94\uac00 \uc791\uc5c5\ucc98\ub7fc \ubcf4\uc774\uc9c0\ub9cc \uc5ec\ub7ec \uc2e4\ud5d8\uc744 \ub354 \uc27d\uac8c \uc218\ud589 \ud560 \uc218 \uc788\uc74c) \uc2e4\uc81c\ub85c \ub124\ud2b8\uc6cc\ud06c\ub97c \ucd08\uae30\ud654\ud558\uace0 \ud559\uc2b5\uc744 \uc2dc\uc791\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uc785\ub825 \ubb38\uc7a5\uc774 \ub9ce\uc774 \ud544\ud130\ub9c1\ub418\uc5c8\uc74c\uc744 \uae30\uc5b5\ud558\uc2ed\uc2dc\uc624. \uc774 \uc791\uc740 \ub370\uc774\ud130 \uc138\ud2b8\uc758 \uacbd\uc6b0 256 \ud06c\uae30\uc758 \uc740\ub2c9 \ub178\ub4dc(hidden node)\uc640 \ub2e8\uc77c GRU \uacc4\uce35 \uac19\uc740 \uc0c1\ub300\uc801\uc73c\ub85c \uc791\uc740 \ub124\ud2b8\uc6cc\ud06c\ub97c \uc0ac\uc6a9\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4. MacBook CPU\uc5d0\uc11c \uc57d 40\ubd84 \ud6c4\uc5d0 \ud569\ub9ac\uc801\uc778 \uacb0\uacfc\ub97c \uc5bb\uc744 \uac83\uc785\ub2c8\ub2e4. \ucc38\uace0 \uc774 \ub178\ud2b8\ubd81\uc744 \uc2e4\ud589\ud558\uba74 \ud559\uc2b5, \ucee4\ub110 \uc911\ub2e8, \ud3c9\uac00\ub97c \ud560 \uc218 \uc788\uace0 \ub098\uc911\uc5d0 \uc774\uc5b4\uc11c \ud559\uc2b5\uc744 \ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uc778\ucf54\ub354\uc640 \ub514\ucf54\ub354\uac00 \ucd08\uae30\ud654 \ub41c \ud589\uc744 \uc8fc\uc11d \ucc98\ub9ac\ud558\uace0 trainIters \ub97c \ub2e4\uc2dc \uc2e4\ud589\ud558\uc2ed\uc2dc\uc624. hidden_size = 128 batch_size = 32 input_lang, output_lang, train_dataloader = get_dataloader(batch_size) encoder = EncoderRNN(input_lang.n_words, hidden_size).to(device) decoder = AttnDecoderRNN(hidden_size, output_lang.n_words).to(device) train(train_dataloader, encoder, decoder, 80, print_every=5, plot_every=5) Reading lines... Read 135842 sentence pairs Trimmed to 11445 sentence pairs Counting words... Counted words: fra 4601 eng 2991 0m 34s (- 8m 42s) (5 6%) 1.5170 1m 23s (- 9m 44s) (10 12%) 0.6699 2m 10s (- 9m 25s) (15 18%) 0.3511 2m 57s (- 8m 51s) (20 25%) 0.1969 3m 46s (- 8m 17s) (25 31%) 0.1227 4m 35s (- 7m 39s) (30 37%) 0.0856 6m 2s (- 7m 45s) (35 43%) 0.0657 6m 36s (- 6m 36s) (40 50%) 0.0525 7m 9s (- 5m 34s) (45 56%) 0.0459 7m 45s (- 4m 39s) (50 62%) 0.0409 8m 28s (- 3m 51s) (55 68%) 0.0372 9m 17s (- 3m 5s) (60 75%) 0.0347 10m 7s (- 2m 20s) (65 81%) 0.0329 10m 54s (- 1m 33s) (70 87%) 0.0314 11m 43s (- 0m 46s) (75 93%) 0.0306 12m 32s (- 0m 0s) (80 100%) 0.0287 \ub4dc\ub86d\uc544\uc6c3(dropout) \ub808\uc774\uc5b4\ub4e4\uc744 \ud3c9\uac00 (eval) \ubaa8\ub4dc\ub85c \uc124\uc815\ud569\ub2c8\ub2e4. encoder.eval() decoder.eval() evaluateRandomly(encoder, decoder) \u003e il est a la maison aujourd hui = he is at home today \u003c he is at home today \u003cEOS\u003e \u003e ce sont celles qui veulent y aller = they are the ones who want to go \u003c they are the ones who want to go \u003cEOS\u003e \u003e desole si je vous ai fait peur = i m sorry if i scared you \u003c i m sorry if i scared you \u003cEOS\u003e \u003e je suis heureux de te voir ici = i m happy to see you here \u003c i m glad to see you here \u003cEOS\u003e \u003e je prends mon apres midi demain = i m taking tomorrow afternoon off \u003c i m taking tomorrow afternoon off \u003cEOS\u003e \u003e tu es tres intelligente = you re very smart \u003c you re very intelligent \u003cEOS\u003e \u003e tu es malin = you re clever \u003c you re clever \u003cEOS\u003e \u003e je ne vais pas abandonner maintenant = i m not quitting now \u003c i m not quitting now \u003cEOS\u003e \u003e nous sommes sans emploi = we re unemployed \u003c we re unemployed \u003cEOS\u003e \u003e il est canadien = he is canadian \u003c he is canadian \u003cEOS\u003e Attention \uc2dc\uac01\ud654# Attention \uba54\ucee4\ub2c8\uc998\uc758 \uc720\uc6a9\ud55c \uc18d\uc131\uc740 \ud558\ub098\ub294 \ud574\uc11d \uac00\ub2a5\uc131\uc774 \ub192\uc740 \ucd9c\ub825\uc785\ub2c8\ub2e4. \uc785\ub825 \uc2dc\ud000\uc2a4\uc758 \ud2b9\uc815 \uc778\ucf54\ub354 \ucd9c\ub825\uc5d0 \uac00\uc911\uce58\ub97c \ubd80\uc5ec\ud558\ub294 \ub370 \uc0ac\uc6a9\ub418\ubbc0\ub85c \uac01 \uc2dc\uac04 \ub2e8\uacc4\uc5d0\uc11c \ub124\ud2b8\uc6cc\ud06c\uac00 \uac00\uc7a5 \uc9d1\uc911\ub418\ub294 \uc704\uce58\ub97c \ud30c\uc545\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4. Attention \ucd9c\ub825\uc744 \ud589\ub82c\ub85c \ud45c\uc2dc\ud558\uae30 \uc704\ud574\uc11c\ub294 plt.matshow(attentions) \uc744 \uadf8\ub0e5 \uc2e4\ud589\ud574\ub3c4 \ub429\ub2c8\ub2e4. \ud558\uc9c0\ub9cc \uc880 \ub354 \ub098\uc740 \uc2dc\uac01\ud654\ub97c \uc704\ud574 \ucd95(axis)\uacfc \ub77c\ubca8(label)\uc744 \ucd94\uac00\ud558\ub294 \uc57d\uac04\uc758 \uc791\uc5c5\uc744 \ub354 \ud574\ubcf4\uaca0\uc2b5\ub2c8\ub2e4: def showAttention(input_sentence, output_words, attentions): fig = plt.figure() ax = fig.add_subplot(111) cax = ax.matshow(attentions.cpu().numpy(), cmap=\u0027bone\u0027) fig.colorbar(cax) # \ucd95 \uc124\uc815 ax.set_xticklabels([\u0027\u0027] + input_sentence.split(\u0027 \u0027) + [\u0027\u003cEOS\u003e\u0027], rotation=90) ax.set_yticklabels([\u0027\u0027] + output_words) # \ub9e4 \ud2f1\ub9c8\ub2e4 \ub77c\ubca8 \ubcf4\uc5ec\uc8fc\uae30 ax.xaxis.set_major_locator(ticker.MultipleLocator(1)) ax.yaxis.set_major_locator(ticker.MultipleLocator(1)) plt.show() def evaluateAndShowAttention(input_sentence): output_words, attentions = evaluate(encoder, decoder, input_sentence, input_lang, output_lang) print(\u0027input =\u0027, input_sentence) print(\u0027output =\u0027, \u0027 \u0027.join(output_words)) showAttention(input_sentence, output_words, attentions[0, :len(output_words), :]) evaluateAndShowAttention(\u0027il n est pas aussi grand que son pere\u0027) evaluateAndShowAttention(\u0027je suis trop fatigue pour conduire\u0027) evaluateAndShowAttention(\u0027je suis desole si c est une question idiote\u0027) evaluateAndShowAttention(\u0027je suis reellement fiere de vous\u0027) input = il n est pas aussi grand que son pere output = he is not as tall as his father \u003cEOS\u003e /workspace/tutorials-kr/intermediate_source/seq2seq_translation_tutorial.py:822: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator. /workspace/tutorials-kr/intermediate_source/seq2seq_translation_tutorial.py:824: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator. input = je suis trop fatigue pour conduire output = i am too tired to drive away \u003cEOS\u003e /workspace/tutorials-kr/intermediate_source/seq2seq_translation_tutorial.py:822: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator. /workspace/tutorials-kr/intermediate_source/seq2seq_translation_tutorial.py:824: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator. input = je suis desole si c est une question idiote output = i m sorry if this is a stupid question \u003cEOS\u003e /workspace/tutorials-kr/intermediate_source/seq2seq_translation_tutorial.py:822: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator. /workspace/tutorials-kr/intermediate_source/seq2seq_translation_tutorial.py:824: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator. input = je suis reellement fiere de vous output = i m really proud of you \u003cEOS\u003e /workspace/tutorials-kr/intermediate_source/seq2seq_translation_tutorial.py:822: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator. /workspace/tutorials-kr/intermediate_source/seq2seq_translation_tutorial.py:824: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator. \uc5f0\uc2b5# \ub2e4\ub978 \ub370\uc774\ud130 \uc14b\uc744 \uc2dc\ub3c4\ud574 \ubcf4\uc2ed\uc2dc\uc624 \ub2e4\ub978 \uc5b8\uc5b4\uc30d \uc0ac\ub78c \u2192 \uae30\uacc4 (e.g. IOT \uba85\ub839\uc5b4) \ucc44\ud305 \u2192 \uc751\ub2f5 \uc9c8\ubb38 \u2192 \ub2f5\ubcc0 word2vec \ub610\ub294 GloVe \uac19\uc740 \ubbf8\ub9ac \ud559\uc2b5\ub41c word embedding \uc73c\ub85c embedding \uc744 \uad50\uccb4\ud558\uc2ed\uc2dc\uc624 \ub354 \ub9ce\uc740 \ub808\uc774\uc5b4, \uc740\ub2c9 \uc720\ub2db, \ub354 \ub9ce\uc740 \ubb38\uc7a5\uc744 \uc0ac\uc6a9\ud558\uc2ed\uc2dc\uc624. \ud559\uc2b5 \uc2dc\uac04\uacfc \uacb0\uacfc\ub97c \ube44\uad50\ud574 \ubcf4\uc2ed\uc2dc\uc624 \ub9cc\uc57d \uac19\uc740 \uad6c\ubb38 \ub450\uac1c\uc758 \uc30d\uc73c\ub85c \ub41c \ubc88\uc5ed \ud30c\uc77c\uc744 \uc774\uc6a9\ud55c\ub2e4\uba74, (I am test \\t I am test), \uc774\uac83\uc744 \uc624\ud1a0\uc778\ucf54\ub354\ub85c \uc0ac\uc6a9\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4. \uc774\uac83\uc744 \uc2dc\ub3c4\ud574 \ubcf4\uc2ed\uc2dc\uc624: \uc624\ud1a0\uc778\ucf54\ub354 \ud559\uc2b5 \uc778\ucf54\ub354 \ub124\ud2b8\uc6cc\ud06c \uc800\uc7a5\ud558\uae30 \uadf8 \uc0c1\ud0dc\uc5d0\uc11c \ubc88\uc5ed\uc744 \uc704\ud55c \uc0c8\ub85c\uc6b4 \ub514\ucf54\ub354 \ud559\uc2b5 Total running time of the script: (12 minutes 42.958 seconds) Download Jupyter notebook: seq2seq_translation_tutorial.ipynb Download Python source code: seq2seq_translation_tutorial.py Download zipped: seq2seq_translation_tutorial.zip",
       "author": {
         "@type": "Organization",
         "name": "PyTorch Contributors",
         "url": "https://pytorch.org"
       },
       "image": "../_static/img/pytorch_seo.png",
       "mainEntityOfPage": {
         "@type": "WebPage",
         "@id": "/intermediate/seq2seq_translation_tutorial.html"
       },
       "datePublished": "2023-01-01T00:00:00Z",
       "dateModified": "2023-01-01T00:00:00Z"
     }