{"id":3975,"date":"2022-04-06T12:09:37","date_gmt":"2022-04-06T03:09:37","guid":{"rendered":"https:\/\/agirobots.com\/?p=3975"},"modified":"2025-07-30T17:34:03","modified_gmt":"2025-07-30T08:34:03","slug":"attention-mechanism-transformer","status":"publish","type":"post","link":"https:\/\/developers.agirobots.com\/jp\/attention-mechanism-transformer\/","title":{"rendered":"\u3010\u8a73\u8aac\u3011Attention\u6a5f\u69cb\u306e\u8d77\u6e90\u304b\u3089\u5b66\u3076Transformer"},"content":{"rendered":"\n<p>\u307f\u306a\u3055\u3093\u306f\u3001Transformer\u306b\u3064\u3044\u3066\u3069\u306e\u3088\u3046\u306a\u30a4\u30e1\u30fc\u30b8\u3092\u6301\u3063\u3066\u3044\u307e\u3059\u304b\uff1f<br>\u6700\u8fd1\u306f\u3001BERT\u3001GPT\u306a\u3069\u306eTransformer\u30d9\u30fc\u30b9\u306e\u30e2\u30c7\u30eb\u304c\u76ee\u3092\u898b\u5f35\u308b\u3088\u3046\u306a\u6210\u679c\u3092\u3060\u3057\u3066\u3044\u308b\u306e\u3067\u3001\u305d\u308c\u3089\u3092\u60f3\u50cf\u3059\u308b\u65b9\u304c\u591a\u3044\u304b\u3068\u601d\u3044\u307e\u3059\u3002\u3053\u308c\u3089\u306fTransformer\u306e\u767a\u5c55\u5f62\u3067\u3059\u304c\u3001Transformer\u306e\u8d77\u6e90\u306e\u307b\u3046\u3001\u5373\u3061Attention\u3092\u60f3\u50cf\u3055\u308c\u305f\u65b9\u3082\u3044\u308b\u304b\u3082\u3057\u308c\u307e\u305b\u3093\u3002\u3053\u306e\u8a18\u4e8b\u3067\u8aac\u660e\u3057\u3066\u3044\u304f\u306e\u306f\u3001Transformer\u306e\u8d77\u6e90\u306eAttention\u6a5f\u69cb\u306b\u3064\u3044\u3066\u3067\u3059\u3002BERT\u3084GPT\u306b\u3064\u3044\u3066\u306f\u3001\u5225\u306e\u8a18\u4e8b\u3067\u89e3\u8aac\u3067\u304d\u308c\u3070\u3068\u601d\u3044\u307e\u3059\u3002<\/p>\n\n\n\n<p>Transformer\u306e\u8ad6\u6587\u30bf\u30a4\u30c8\u30eb\u300c<a rel=\"noreferrer noopener\" href=\"https:\/\/arxiv.org\/abs\/1706.03762\" target=\"_blank\">Attention Is All You Need<\/a>\u300d\u304b\u3089Transformer\u306e\u6210\u529f\u306fAttention\u6a5f\u69cb\u306b\u3042\u308b\u3053\u3068\u304c\u63a8\u6e2c\u3067\u304d\u308b\u3068\u601d\u3044\u307e\u3059\u304c\u3001\u305d\u306e\u901a\u308a\u3067\u3001Attention\u6a5f\u69cb\u306a\u3057\u306bTransformer\u3092\u8a9e\u308b\u306e\u306f\u96e3\u3057\u3044\u3067\u3059\u3002\u672c\u8a18\u4e8b\u3067\u306f\u3001Attention\u6a5f\u69cb\u306b\u7126\u70b9\u3092\u5f53\u3066\u3001\u53e4\u304f\u304b\u3089\u884c\u308f\u308c\u3066\u3044\u308b\u8a8d\u77e5\u79d1\u5b66\u306b\u57fa\u3065\u304f\u7814\u7a76\u306a\u3069\u3082\u7d39\u4ecb\u3057\u306a\u304c\u3089\u8aac\u660e\u3057\u3066\u3044\u304d\u307e\u3059\u3002Transformer\u306e\u89e3\u8aac\u3060\u3051\u3067\u3042\u308c\u3070\u3001\u3082\u3063\u3068\u77ed\u304f\u3067\u304d\u308b\u306e\u3067\u3059\u304c\u3001Attention\u306b\u7126\u70b9\u3092\u5f53\u3066\u305f\u306e\u3067\u3001\u8a18\u4e8b\u304c\u9577\u304f\u306a\u3063\u3066\u3057\u307e\u3044\u307e\u3057\u305f\u3002\u7d14\u7c8b\u306bAttention\u6a5f\u69cb\u306b\u3064\u3044\u3066\u8208\u5473\u304c\u3042\u308a\u307e\u3057\u305f\u3089\u3001\u6700\u5f8c\u307e\u3067\u304a\u8aad\u307f\u3044\u305f\u3060\u3051\u308b\u3068\u5b09\u3057\u3044\u3067\u3059\u3002YouTube\u306b\u89e3\u8aac\u8a18\u4e8b\u3092\u30a2\u30c3\u30d7\u3057\u3066\u3044\u308b\u306e\u3067\u3001\u52d5\u753b\u3067\u5b66\u3073\u305f\u3044\u65b9\u306f\u305c\u3072\u3054\u89a7\u4e0b\u3055\u3044\u3002<\/p>\n\n\n\n<p>\u5c11\u3057\u3067\u3082\u65e9\u304fTransformer\u306b\u3064\u3044\u3066\u8981\u70b9\u3092\u77e5\u308a\u305f\u3044\u3068\u3044\u3046\u65b9\u5411\u3051\u306b\u77ed\u3081\u306e\u8a18\u4e8b\u3082\u51fa\u3057\u3066\u3044\u307e\u3059\u306e\u3067\u3001\u77e5\u308a\u305f\u3044\u5185\u5bb9\u304cAttention\u6a5f\u69cb\u306b\u3064\u3044\u3066\u306a\u3089\u3001\u4eca\u958b\u3044\u3066\u3044\u308b\u3053\u306e\u8a18\u4e8b\u3092\u3001Transformer\u306b\u3064\u3044\u3066\u306a\u3089\u4ee5\u4e0b\u306e\u30ea\u30f3\u30af\u306e\u8a18\u4e8b\u3092\u304a\u52e7\u3081\u3057\u307e\u3059\u3002<\/p>\n\n\n\n<figure class=\"wp-block-embed aligncenter is-type-wp-embed is-provider-agirobots-blog wp-block-embed-agirobots-blog\"><div class=\"wp-block-embed__wrapper\">\n<div><a href=\"https:\/\/developers.agirobots.com\/jp\/essence-transformer\/\" class=\"st-cardlink st-embed-cardlink\"><div class=\"kanren st-cardbox\"><dl class=\"clearfix\"><dt class=\"st-card-img\"><img decoding=\"async\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2023\/02\/CD5DF6F6-E0AD-4577-8654-A8FE3F602C48-150x150.jpeg\" alt=\"\" width=\"100\" height=\"100\" \/><\/dt><dd><p class=\"st-cardbox-t\">Transformer\u306b\u3064\u3044\u3066\u672c\u8cea\u3092\u5206\u304b\u308a\u3084\u3059\u304f\u89e3\u8aac\uff01<\/p><div class=\"st-card-excerpt smanone\"><p>\u7686\u3055\u3093\u3053\u3093\u306b\u3061\u306f\uff01 \u53bb\u5e74\uff082022\u5e7411\u6708\uff09\u306b\u767a\u8868\u3055\u308c\u305fChatGPT\u306e\u8a71\u984c\u304c\u5c3d\u304d\u306a\u3044\u4eca\u65e5\u3053\u306e\u9803\u3001ChatGPT\u3067\u4f7f\u308f\u308c\u3066\u3044\u308b\u91cd\u8981\u306a\u6280\u8853\u306e\uff11\u3064\u3067\u3042\u308bTransformer\u306b\u3064\u3044\u3066\u3001\u8208\u5473\u3092\u6301\u3063\u305f\u65b9\u306f\u6ca2\u5c71 ... <\/p><\/div><\/dd><\/dl><\/div><\/a><\/div>\n<\/div><\/figure>\n\n\n\n<p>\u672c\u8a18\u4e8b\u306e\u8aac\u660e\u52d5\u753b\u3082\u516c\u958b\u3057\u3066\u3044\u307e\u3059\uff08\u4ee5\u4e0b\uff09\u3002\u305c\u3072\u3054\u6d3b\u7528\u3044\u305f\u3060\u3051\u308c\u3070\u3068\u601d\u3044\u307e\u3059\u3002<\/p>\n\n\n\n<p>\u3053\u306e\u5185\u5bb9\u306b\u95a2\u3057\u3066\u3001\u89e3\u8aac\u30b9\u30e9\u30a4\u30c9\u3082\u516c\u958b\u3057\u3066\u3044\u308b\u306e\u3067\u3001\u6709\u52b9\u6d3b\u7528\u3057\u3066\u3044\u305f\u3060\u3051\u308c\u3070\u3068\u601d\u3044\u307e\u3059\u3002<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<p>\u672c\u8a18\u4e8b\u306e\u69cb\u6210\u306b\u3064\u3044\u3066\u8aac\u660e\u3057\u307e\u3059\u3002<\/p>\n\n\n\n<p>\u307e\u305a\u521d\u3081\u306b\u3001\u8a8d\u77e5\u79d1\u5b66\u306e\u7814\u7a76\u3068\u7d61\u3081\u3066Attention\u3068\u3044\u3046\u8003\u3048\u65b9\u304c\u77e5\u80fd\u306e\u5b9f\u73fe\u306b\u3069\u3046\u95a2\u308f\u3063\u3066\u3044\u308b\u306e\u304b\u3092\u8aac\u660e\u3057\u307e\u3059\u3002\u3053\u306e\u5185\u5bb9\u3092\u7406\u89e3\u3059\u308b\u3053\u3068\u3067\u3001Transformer\u3067\u7a81\u5982\u3068\u3057\u3066\u77e5\u308c\u6e21\u3063\u305fAttention\u6a5f\u69cb\u304c\u3001\u5929\u624d\u305f\u3061\u306b\u3088\u308b100%\u306e\u7a81\u767a\u7684\u306a\u767a\u60f3\u3068\u3044\u3046\u308f\u3051\u3067\u306f\u306a\u3044\u3053\u3068\u3092\u3054\u7406\u89e3\u3044\u305f\u3060\u3051\u308c\u3070\u3068\u601d\u3044\u307e\u3059\u3002<\/p>\n\n\n\n<p>\u6b21\u306b\u3001Transformer\u304c\u767b\u5834\u3059\u308b\u4ee5\u524d\u306eAttention\u6a5f\u69cb\u306b\u3064\u3044\u3066\u3001\u518d\u5e30\u578b\u30cb\u30e5\u30fc\u30e9\u30eb\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\uff08RNN\uff09\u3084\u7573\u307f\u8fbc\u307f\u30cb\u30e5\u30fc\u30e9\u30eb\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\uff08CNN\uff09\u3067\u898b\u3089\u308c\u305f\u65e2\u5b58\u30e2\u30c7\u30eb\u3092\u7d39\u4ecb\u3057\u307e\u3059\u3002\u3053\u3053\u3067\u306f\u3001Transformer\u306e\u8ad6\u6587\u300cAttention Is All You Need\u300d\u3067Attention\u6a5f\u69cb\u304c\u4e00\u822c\u5316\u3055\u308c\u308b\u4ee5\u524d\u304b\u3089\u3001Attention\u6a5f\u69cb\u306e\u8003\u3048\u65b9\u304c\u53d6\u308a\u5165\u308c\u3089\u308c\u3066\u3044\u305f\u3053\u3068\u3092\u3054\u7406\u89e3\u3044\u305f\u3060\u3051\u308c\u3070\u3068\u601d\u3044\u307e\u3059\u3002<\/p>\n\n\n\n<p>\u305d\u3057\u3066\u3001\u3053\u3053\u307e\u3067\u5185\u5bb9\u3092\u30d9\u30fc\u30b9\u3068\u3057\u3066Transformer\u3067\u4f7f\u7528\u3055\u308c\u3066\u3044\u308bAttention\u6a5f\u69cb\u306b\u3064\u3044\u3066\u8aac\u660e\u3092\u3057\u307e\u3059\u3002<\/p>\n\n\n\n<p>\u672c\u8a18\u4e8b\u3092\u901a\u3058\u3066\u3001\u53e4\u304f\u304b\u3089\u8003\u3048\u3089\u308c\u3066\u304d\u305fAttention\u304c\u3001\u8fd1\u5e74\u5927\u304d\u306a\u6210\u529f\u3092\u53ce\u3081\u308b\u306b\u81f3\u308b\u307e\u3067\u306e\u5909\u9077\u3092\u3054\u7406\u89e3\u3044\u305f\u3060\u3051\u308b\u306e\u3067\u306f\u306a\u3044\u304b\u3068\u601d\u3044\u307e\u3059\u3002<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">\u8a8d\u77e5\u79d1\u5b66\u3068\u6ce8\u610f\uff08Attention\uff09<\/h2>\n\n\n\n<p>\u30cb\u30e5\u30fc\u30e9\u30eb\u30cd\u30c3\u30c8\u306e\u5fdc\u7528\u306b\u8208\u5473\u304c\u3042\u308b\u65b9\u3060\u3068\u3001\u8a8d\u77e5\u79d1\u5b66\u306b\u99b4\u67d3\u307f\u304c\u306a\u3044\u65b9\u304c\u591a\u3044\u304b\u3082\u3057\u308c\u307e\u305b\u3093\u3002\u3067\u3059\u304c\u3001Attention\u6a5f\u69cb\u304c\u6628\u4eca\u306e\u6df1\u5c64\u5b66\u7fd2\u3067\u9ad8\u7cbe\u5ea6\u3092\u53e9\u304d\u51fa\u3059\u4e2d\u3001\u672c\u6765\u3001\u6ce8\u610f\uff08Attention\uff09\u3068\u306f\u4f55\u304b\u3001\u306b\u3064\u3044\u3066\u79c1\u305f\u3061\u4eba\u9593\u306e\u77e5\u306e\u30e1\u30ab\u30cb\u30ba\u30e0\u304b\u3089\u8003\u3048\u308b\u8a8d\u77e5\u79d1\u5b66\u7684\u306a\u30a2\u30d7\u30ed\u30fc\u30c1\u306f\u3001\u6df1\u5c64\u5b66\u7fd2\u306b\u304a\u3044\u3066Attention\u6a5f\u69cb\u304c\u306a\u305c\u91cd\u8981\u306a\u306e\u304b\u3001\u306a\u305c\u9ad8\u7cbe\u5ea6\u3092\u51fa\u305b\u308b\u306e\u304b\u3092\u76f4\u611f\u7684\u306b\u7406\u89e3\u3059\u308b\u4e0a\u3067\u6709\u76ca\u306a\u3082\u306e\u3068\u306a\u308b\u306f\u305a\u3067\u3059\u3002<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">\u8a8d\u77e5\u79d1\u5b66<\/h3>\n\n\n\n<p>\u8a8d\u77e5\u79d1\u5b66\u3068\u306f\u4eba\u9593\u306e\u8133\u3084\u5fc3\u3068\u3044\u3063\u305f\u77e5\u306e\u50cd\u304d\u3068\u6027\u8cea\u3092\u7406\u89e3\u3057\u3088\u3046\u3068\u3059\u308b\u7814\u7a76\u5206\u91ce\u3067\u3001\u5fc3\u7406\u5b66\u3084\u4eba\u5de5\u77e5\u80fd\u3001\u795e\u7d4c\u79d1\u5b66\u3001\u54f2\u5b66\u306a\u3069\u3001\u69d8\u3005\u306a\u5b66\u554f\u3092\u307e\u305f\u3050\u5b66\u969b\u7684\u306a\u69cb\u9020\u3092\u3057\u3066\u3044\u307e\u3059\u3002\u306a\u304b\u3067\u3082\u3001\u4eba\u5de5\u77e5\u80fd\u5206\u91ce\u306f\u77e5\u80fd\u3092\u4f5c\u308b\u30a2\u30d7\u30ed\u30fc\u30c1\u304b\u3089\u53e4\u304f\u304b\u3089\u8a8d\u77e5\u79d1\u5b66\u3068\u95a2\u308f\u308a\u3092\u6301\u3063\u3066\u767a\u5c55\u3057\u3066\u3044\u307e\u3059\u3002<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large is-resized\"><img decoding=\"async\" width=\"1024\" height=\"634\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-18-1024x634.jpg\" alt=\"\" class=\"wp-image-4212\" style=\"width:768px;height:476px\" srcset=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-18-1024x634.jpg 1024w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-18-300x186.jpg 300w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-18-768x475.jpg 768w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-18.jpg 1512w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure><\/div>\n\n\n<h3 class=\"wp-block-heading\">\u30a6\u30a3\u30b1\u30f3\u30ba\u306e\u60c5\u5831\u51e6\u7406\u30e2\u30c7\u30eb<\/h3>\n\n\n\n<p>\u8a8d\u77e5\u79d1\u5b66\u3067\u306f\u3001\u77e5\u306e\u30e1\u30ab\u30cb\u30ba\u30e0\u3092\u8ad6\u7406\u7684\u304b\u3064\u6bb5\u968e\u7684\u306b\u8868\u73fe\u3059\u308b\u305f\u3081\u306b\u3001\u305d\u306e\u8a8d\u77e5\u904e\u7a0b\u3092\u30e2\u30c7\u30eb\u5316\u3057\u307e\u3059\u3002\u30e2\u30c7\u30eb\u5316\u306e\u7bc4\u56f2\u306f\u3001\u611f\u899a\u5165\u529b\u304b\u3089\u904b\u52d5\u51fa\u529b\u306b\u81f3\u308b\u307e\u3067\u306e\u77e5\u7684\u30b7\u30b9\u30c6\u30e0\u306e\u30d7\u30ed\u30bb\u30b9\u3001\u30a2\u30eb\u30b4\u30ea\u30ba\u30e0\u3001\u904e\u7a0b\u306b\u81f3\u308b\u3053\u3068\u304c\u4e00\u822c\u7684\u3067\u3059\u3002<\/p>\n\n\n\n<p>\u3053\u3053\u3067\u306f\u3001\u5fc3\u7406\u5b66\u8005\u306e\u30a6\u30a3\u30b1\u30f3\u30ba\u304c\u63d0\u6848\u3057\u305f\u3001\u30a6\u30a3\u30b1\u30f3\u30ba\u306e\u60c5\u5831\u51e6\u7406\u30e2\u30c7\u30eb\uff08\u4e0b\u56f3\uff09\u306b\u3064\u3044\u3066\u7d39\u4ecb\u3057\u307e\u3059\u3002\u3053\u308c\u306f\u591a\u304f\u306e\u652f\u6301\u3092\u5f97\u3066\u3044\u308b\u8a8d\u77e5\u30e2\u30c7\u30eb\u3067\u3001\u307e\u305f\u3001Attention\u6a5f\u69cb\u3068\u3082\u5927\u304d\u306a\u95a2\u9023\u304c\u3042\u308a\u307e\u3059\u3002<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" width=\"1024\" height=\"547\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-1024x547.jpg\" alt=\"\" class=\"wp-image-4108\" srcset=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-1024x547.jpg 1024w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-300x160.jpg 300w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-768x411.jpg 768w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-1536x821.jpg 1536w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image.jpg 1736w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure><\/div>\n\n\n<p>\u30a6\u30a3\u30b1\u30f3\u30ba\u306e\u60c5\u5831\u51e6\u7406\u30e2\u30c7\u30eb\u306f\u3001\u611f\u899a\u5165\u529b\u304b\u3089\u904b\u52d5\u51fa\u529b\u306b\u81f3\u308b\u307e\u3067\u306e\u30d7\u30ed\u30bb\u30b9\u3092\u3001\u5de6\u304b\u3089\u9806\u756a\u306b\u3001\u611f\u899a\u51e6\u7406\u3001\u77e5\u899a\u3001\u8a18\u61b6\u30fb\u8a8d\u77e5\u3001\u53cd\u5fdc\u9078\u629e\u3001\u53cd\u5fdc\u5b9f\u884c\u3068\u3044\u3046\u6bb5\u968e\u306b\u5206\u3051\u3001\u5404\u3005\u306b\u6ce8\u610f\u8cc7\u6e90\u304b\u3089\u4f55\u3089\u304b\u306e\u51e6\u7406\u3092\u53d7\u3051\u308b\u69cb\u9020\u3092\u3057\u3066\u3044\u307e\u3059\u3002<\/p>\n\n\n\n<p>\u7279\u306b\u3001\u6ce8\u610f\u8cc7\u6e90\u306b\u6ce8\u76ee\u3057\u3066\u304f\u3060\u3055\u3044\u3002\u305d\u308c\u305e\u308c\u306e\u9805\u76ee\u306b\u4f55\u3089\u304b\u306e\u6ce8\u610f\uff08Attention\uff09\u306e\u77e2\u5370\u3092\u5411\u3051\u3066\u3044\u307e\u3059\u304c\u3001\u306a\u305c\u6ce8\u610f\u3092\u5c0e\u5165\u3057\u305f\u3068\u601d\u3044\u307e\u3059\u304b\uff1f<\/p>\n\n\n\n<p>\u30a6\u30a3\u30b1\u30f3\u30ba\u306e\u60c5\u5831\u51e6\u7406\u30e2\u30c7\u30eb\u306b\u304a\u3044\u3066\u3001\u6ce8\u610f\u3068\u306f\u300c\u7279\u5b9a\u306e\u5bfe\u8c61\u306b\u611f\u899a\u3084\u610f\u8b58\u3092\u96c6\u4e2d\u3055\u305b\u308b\u3053\u3068\u3067\u3001\u4e0d\u5fc5\u8981\u306a\u60c5\u5831\u3092\u6392\u9664\u3001\u5fc5\u8981\u306a\u60c5\u5831\u3092\u9078\u629e\u3057\u3001\u9650\u308a\u3042\u308b\u77e5\u7684\u8cc7\u6e90\u3092\u52b9\u679c\u7684\u306b\u914d\u5206\u3059\u308b\u3082\u306e\u300d\u3068\u3044\u3046\u4f4d\u7f6e\u4ed8\u3051\u306b\u306a\u3063\u3066\u3044\u307e\u3059\u3002<\/p>\n\n\n\n<p>\u79c1\u306e\u89e3\u91c8\u3067\u3059\u304c\u3001\u6211\u3005\u306e\u77e5\u306b\u306f\u7a7a\u9593\u30fb\u6642\u9593\u7684\u306b\u9650\u308a\u304c\u3042\u308b\u306e\u3067\u3001\u9650\u308a\u3042\u308b\u77e5\u3092\u52b9\u7387\u7684\u304b\u3064\u5408\u7406\u7684\u306b\u5229\u7528\u3059\u308b\u305f\u3081\u306b\u5c0e\u5165\u3057\u305f\u306e\u3067\u306f\u306a\u3044\u304b\u3068\u8003\u3048\u3066\u3044\u307e\u3059\u3002<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">\u7121\u610f\u8b58\u7684\u306a\u6ce8\u610f\uff08Attention\uff09<\/h3>\n\n\n\n<p>\u65e5\u672c\u8a9e\u306b\u3001\u300c\u6ce8\u610f\u3092\u5411\u3051\u308b\u300d\u3068\u3044\u3046\u8868\u73fe\u304c\u3042\u308a\u307e\u3059\u3002\u3053\u308c\u306f\u3001\u7279\u5b9a\u306e\u7269\u4e8b\u306b\u6ce8\u76ee\u3092\u3059\u308b\u3053\u3068\u3067\u3059\u3002\u305d\u308c\u306f\u307e\u3055\u306b\u6ce8\u610f\u306e\u50cd\u304d\u3092\u610f\u56f3\u7684\u306b\u5229\u7528\u3057\u305f\u3082\u306e\u3067\u3059\u3002\u4e00\u65b9\u3067\u3001\u6c17\u3065\u3044\u305f\u3089\u7279\u5b9a\u306e\u7269\u4e8b\u3092\u6ce8\u8996\u3057\u3066\u3044\u305f\u3001\u307f\u305f\u3044\u306b\u300c\u6ce8\u610f\u300d\u306f\u7121\u610f\u8b58\u7684\u306b\u8abf\u7bc0\u3055\u308c\u308b\u306e\u304c\u4e00\u822c\u7684\u3067\u3059\u3002\u3053\u3053\u3067\u306f\u3001\u7121\u610f\u8b58\u7684\u306a\u6ce8\u610f\u3068\u3057\u3066<span data-color=\"#fffd6b\" style=\"background: linear-gradient(transparent 60%,rgba(255, 253, 107, 0.7) 0);\" class=\"vk_highlighter\">\u8996\u899a\u7684\u6ce8\u610f<\/span>\u3068<span data-color=\"#fffd6b\" style=\"background: linear-gradient(transparent 60%,rgba(255, 253, 107, 0.7) 0);\" class=\"vk_highlighter\">\u8074\u899a\u7684\u6ce8\u610f<\/span>\u3092\u7d39\u4ecb\u3057\u307e\u3059\u3002<\/p>\n\n\n\n<p><strong>[\u8996\u899a\u7684\u6ce8\u610f]<\/strong><br>\u8857\u3092\u6b69\u3044\u3066\u3044\u3066\u770b\u677f\u306b\u66f8\u304b\u308c\u305f\u6587\u5b57\u3092\u8aad\u3080\u884c\u70ba\u304c\u53ef\u80fd\u306a\u306e\u306f\u8996\u899a\u7684\u6ce8\u610f\u306e\u70ba\u3067\u3059\u3002\u307e\u305a\u8996\u899a\u60c5\u5831\u306e\u4e2d\u304b\u3089\u3001\u770b\u677f\u306b\u96c6\u4e2d\u3092\u5411\u3051\u3066\u3001\u305d\u306e\u4e2d\u306e\u6587\u5b57\u3092\u8a8d\u8b58\u3057\u7406\u89e3\u3059\u308b\u4e00\u9023\u306e\u30d7\u30ed\u30bb\u30b9\u3067\u3059\u304c\u3001\u8ab0\u3067\u3082\u3067\u304d\u307e\u3059\u3088\u306d\u3002<\/p>\n\n\n\n<p><strong>[\u8074\u899a\u7684\u6ce8\u610f]<\/strong><br>\u30ab\u30af\u30c6\u30eb\u30d1\u30fc\u30c6\u30a3\u52b9\u679c\u3092\u3054\u5b58\u3058\u3067\u3057\u3087\u3046\u304b\uff1f\u3053\u3067\u306f\u3001\u9a12\u304c\u3057\u3044\u74b0\u5883\u4e0b\u3067\u3082\u81ea\u5206\u3068\u4f1a\u8a71\u3057\u3066\u3044\u308b\u76f8\u624b\u306e\u58f0\u3092\u8a8d\u8b58\u3057\u3001\u7406\u89e3\u3059\u308b\u3053\u3068\u304c\u3067\u304d\u308b\u3068\u3044\u3046\u3082\u306e\u3067\u3059\u3002\u4f1a\u8a71\u76f8\u624b\u306e\u58f0\u3092\u9078\u629e\u7684\u306b\u9078\u3076\u3053\u3068\u3067\u3001\u3053\u306e\u80fd\u529b\u304c\u5b9f\u73fe\u3057\u3066\u3044\u308b\u3082\u306e\u3068\u8003\u3048\u3089\u308c\u307e\u3059\u3002<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">\u6ce8\u610f\uff08Attention\uff09\u306e\u53ef\u80fd\u6027<\/h3>\n\n\n\n<p>\u8a8d\u77e5\u79d1\u5b66\u3068\u6ce8\u610f\u306e\u8aac\u660e\u306e\u6700\u5f8c\u306b\u3001\u9ad8\u5ea6\u306a\u77e5\u80fd\u3068\u6ce8\u610f\u306e\u4f8b\u3092\u8003\u3048\u3066\u307f\u307e\u3057\u3087\u3046\u3002\u6211\u3005\u306e\u9ad8\u5ea6\u306a\u77e5\u80fd\u3068\u8a00\u3046\u3068\u3001\u79c1\u306f\u771f\u3063\u5148\u306b\u4f1a\u8a71\u3092\u601d\u3044\u6d6e\u304b\u3079\u307e\u3059\u3002\u4f1a\u8a71\u306f\u3001\u81ea\u5206\u306e\u8003\u3048\u3066\u3044\u308b\u3053\u3068\u3092\u76f8\u624b\u306b\u4f1d\u3048\u308b\u305f\u3081\u306e\u91cd\u8981\u306a\u30a4\u30f3\u30bf\u30d5\u30a7\u30fc\u30b9\u3067\u3059\u3002\u81ea\u5206\u306e\u8003\u3048\u3066\u3044\u308b\u3053\u3068\u3092\u8a00\u8449\u306b\u3059\u308b\u969b\u306b\u3001\u982d\u306e\u4e2d\u306b\u3042\u308b\u3042\u3089\u3086\u308b\u601d\u8003\u3092\u6ce8\u610f\u8cc7\u6e90\u3092\u3082\u3061\u3044\u3066\u3001\u53d6\u6368\u9078\u629e\u3068\u9806\u4f4d\u4ed8\u3051\u3057\u3066\u3044\u307e\u3059\u3088\u306d\uff1f<\/p>\n\n\n\n<p>\u3042\u308c\uff1f\u4f1a\u8a71\u306b\u9650\u3089\u305a\u3001\u79c1\u305f\u3061\u306e\u601d\u8003\u305d\u306e\u3082\u306e\u304c\u6ce8\u610f\u306b\u3088\u3063\u3066\u6210\u308a\u7acb\u3063\u3066\u3044\u308b\u306e\u3067\u306f\uff1f\u3068\u601d\u3063\u305f\u65b9\uff01\u92ed\u3044\u3068\u601d\u3044\u307e\u3059\u3002<\/p>\n\n\n\n<p>\u3053\u306e\u3088\u3046\u306b\u3001\u6ce8\u610f\u306f\u9ad8\u5ea6\u306a\u77e5\u80fd\u306e\u672c\u8cea\u3067\u306f\u306a\u3044\u3067\u3057\u3087\u3046\u304b\uff1f<\/p>\n\n\n\n<p class=\"is-style-st-paragraph-check\"><span class=\"hutoaka\">\u3008\u3053\u308c\u3060\u3051\u306f\u899a\u3048\u3066\u304a\u3053\u3046\u3009<\/span><br>\u8a8d\u77e5\u79d1\u5b66\u306b\u304a\u3051\u308b\u30a6\u30a3\u30b1\u30f3\u30ba\u306e\u60c5\u5831\u51e6\u7406\u30e2\u30c7\u30eb\u306b\u4ee3\u8868\u3055\u308c\u308b\u3088\u3046\u306b\u3001\u6ce8\u610f\uff08Attention\uff09\u306f\u53e4\u304f\u304b\u3089\u91cd\u8981\u306a\u6a5f\u80fd\u3092\u62c5\u3063\u3066\u3044\u308b\u3068\u8003\u3048\u3089\u308c\u3066\u3044\u307e\u3059\u3002<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">\u6df1\u5c64\u5b66\u7fd2\u3068\u6ce8\u610f\uff08Attention\uff09<\/h2>\n\n\n\n<p>\u6ce8\u610f\uff08Attention\uff09\u304c\u53e4\u304f\u304b\u3089\u91cd\u8981\u306a\u6982\u5ff5\u3068\u3057\u3066\u8003\u3048\u3089\u308c\u3066\u304d\u305f\u3053\u3068\u3092\u3054\u7406\u89e3\u3044\u305f\u3060\u3044\u305f\u3068\u3053\u308d\u3067\u3001\u3053\u3053\u304b\u3089\u306f\u3001\u6df1\u5c64\u5b66\u7fd2\u306b\u304a\u3051\u308b\u6ce8\u610f\u306b\u8a71\u3092\u30b7\u30d5\u30c8\u3055\u305b\u3066\u3044\u304d\u307e\u3059\u3002<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">\u6df1\u5c64\u5b66\u7fd2<\/h3>\n\n\n\n<p>\u6df1\u5c64\u5b66\u7fd2\u3068\u306f\u6df1\u5c64\u30cb\u30e5\u30fc\u30e9\u30eb\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\uff08Deep Neural Network\uff1aDNN\uff09\u306e\u5b66\u7fd2\u3092\u6271\u3046\u3082\u306e\u3067\u3001\u30c7\u30a3\u30fc\u30d7\u30e9\u30fc\u30cb\u30f3\u30b0\u3068\u3082\u547c\u3070\u308c\u307e\u3059\u3002\u6df1\u5c64\u5b66\u7fd2\u306f\u3001\u8fd1\u5e74\u3001\u9a5a\u7570\u7684\u306a\u30b9\u30d4\u30fc\u30c9\u3067\u6210\u9577\u3057\u3001\u6a5f\u68b0\u5b66\u7fd2\u306e\u4e3b\u6d41\u3068\u306a\u3063\u3066\u3044\u307e\u3059\u3002<\/p>\n\n\n\n<p>\u6df1\u5c64\u5b66\u7fd2\u306f\u4f55\u6bb5\u968e\u3082\u306e\u30cb\u30e5\u30fc\u30ed\u30f3\u5c64\uff08Layer\uff09\u3092\u91cd\u306d\u308b\u3053\u3068\u3067\u968e\u5c64\u7684\u306b\u7279\u5fb4\u62bd\u51fa\u3092\u3057\u3066\u304a\u308a\u3001\u5c64\u6570\u306f\u591a\u3044\u3082\u306e\u3060\u30681000\u5c64\u4ee5\u4e0a\u306b\u306a\u308a\u307e\u3059\u3002\u8907\u96d1\u306a\u3082\u306e\u3060\u3068\u3001\u8907\u6570\u306e\u5c64\u3092\u307e\u3068\u3081\u3066\u30e2\u30b8\u30e5\u30fc\u30eb\u3092\u69cb\u6210\u3057\u3001\u305d\u306e\u30e2\u30b8\u30e5\u30fc\u30eb\u3092\u4f55\u5c64\u3082\u91cd\u306d\u3066\u3044\u308b\u3082\u306e\u3082\u3042\u308a\u307e\u3059\u3002\u5177\u4f53\u4f8b\u3068\u3057\u3066\u3001GoogLeNet\u306e\u69cb\u9020\u3092\u5982\u4f55\u306b\u793a\u3057\u307e\u3059\u3002<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" width=\"1024\" height=\"233\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/02\/image-12-edited-1024x233.jpg\" alt=\"\" class=\"wp-image-3608\" srcset=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/02\/image-12-edited-1024x233.jpg 1024w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/02\/image-12-edited-300x68.jpg 300w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/02\/image-12-edited-768x175.jpg 768w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/02\/image-12-edited-1536x349.jpg 1536w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/02\/image-12-edited-2048x466.jpg 2048w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><figcaption class=\"wp-element-caption\">GoogLeNet\u306e\u8ad6\u6587\u300cGoing Deeper with Convolutions\u300d\u3088\u308a\u5f15\u7528<\/figcaption><\/figure>\n\n\n\n<p>\u304b\u306a\u308a\u5f37\u5f15\u3067\u3059\u304c\u3001\u30a6\u30a3\u30b1\u30f3\u30ba\u306e\u60c5\u5831\u51e6\u7406\u30e2\u30c7\u30eb\u3092\u6df1\u5c64\u30cb\u30e5\u30fc\u30e9\u30eb\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u30e2\u30c7\u30eb\u6d41\u306e\u63cf\u304d\u65b9\u306b\u5247\u308a\u63cf\u304d\u304b\u3048\u308b\u3068\u4e0b\u56f3\u306b\u306a\u308b\u3068\u601d\u3044\u307e\u3059\u3002<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" width=\"1024\" height=\"395\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-1-1024x395.jpg\" alt=\"\" class=\"wp-image-4112\" srcset=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-1-1024x395.jpg 1024w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-1-300x116.jpg 300w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-1-768x296.jpg 768w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-1-1.jpg 1058w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure><\/div>\n\n\n<p>\u203b\u3042\u304f\u307e\u3067\u30a4\u30e1\u30fc\u30b8\u3067\u3059\u3002\u5b9f\u969b\u306eAttention\u5c64\u306e\u9069\u7528\u65b9\u6cd5\u306f\u69d8\u3005\u3042\u308a\u3001\u5fc5\u305a\u3057\u3082\u3053\u306e\u5f62\u3092\u53d6\u3089\u306a\u3051\u308c\u3070\u306a\u3089\u306a\u3044\u308f\u3051\u3067\u306f\u3042\u308a\u307e\u305b\u3093\u3057\u3001\u3080\u3057\u308d\u3001\u3053\u306e\u3088\u3046\u306a\u69cb\u9020\u3092\u3068\u308b\u5834\u5408\u306f\u6ec5\u591a\u306b\u306a\u3044\u3067\u3057\u3087\u3046\u3002<\/p>\n\n\n\n<p>\u3053\u306e\u56f3\u3067\u306fAttention\u6a5f\u69cb\u306b\u5165\u529b\u304c\u3042\u308a\u307e\u305b\u3093\u304c\u3001\u5b9f\u969b\u306b\u6df1\u5c64\u5b66\u7fd2\u3067\u4f7f\u7528\u3059\u308b\u969b\u306f\u3001\u3055\u307e\u3056\u307e\u306a\u5165\u529b\u3084\u5b66\u7fd2\u30d1\u30e9\u30e1\u30fc\u30bf\u3092\u3082\u3064\u6a5f\u69cb\u3068\u3057\u3066\u5b9f\u73fe\u3055\u308c\u307e\u3059\u3002<\/p>\n\n\n\n<p>\u4e0a\u56f3\u306e\u3088\u3046\u306aAttention\u6a5f\u69cb\u3092\u6301\u3064\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u3092\u8003\u3048\u305f\u5834\u5408\u3001\u3053\u306eAttention\u6a5f\u69cb\u306e\u6a5f\u80fd\u306f\u3069\u3046\u3044\u3063\u305f\u3082\u306e\u306b\u306a\u308b\u3067\u3057\u3087\u3046\u304b\uff1f<br>\u5c11\u3057\u8003\u3048\u3066\u307f\u307e\u3057\u3087\u3046\uff01<\/p>\n\n\n\n<p>\u3055\u307e\u3056\u307e\u8003\u3048\u3089\u308c\u308b\u3068\u601d\u3044\u307e\u3059\u304c\u3001\u30a6\u30a3\u30b1\u30f3\u30ba\u306e\u60c5\u5831\u51e6\u7406\u30e2\u30c7\u30eb\u306e\u6ce8\u610f\u306e\u5f79\u5272\u306b\u5247\u308b\u3068\u3001\u524d\u306e\u5c64\u304c\u51fa\u529b\u3057\u305f\u60c5\u5831\u306b\u304a\u3044\u3066\u3001\u6ce8\u76ee\u3059\u3079\u304d\u60c5\u5831\u3092\u901a\u904e\u3055\u305b\u3001\u6ce8\u76ee\u3059\u3079\u304d\u3067\u306a\u3044\u60c5\u5831\u3092\u5830\u304d\u6b62\u3081\u308b\u3088\u3046\u306a\u3001\u30b2\u30fc\u30c8\u7684\u5f79\u5272\u3092\u62c5\u3044\u3001\u6b21\u306e\u5c64\u306b\u5165\u529b\u3055\u308c\u308b\u60c5\u5831\u306b\u5909\u63db\u3057\u3066\u3044\u308b\u304b\u3082\u3057\u308c\u307e\u305b\u3093\u3002<br>\u307e\u305f\u3001\u305d\u308c\u305e\u308c\u306e\u5c64\u3092\u4fef\u77b0\u3057\u3066\u3001\u3088\u308a\u5408\u7406\u7684\u306b\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u304c\u6a5f\u80fd\u3059\u308b\u3088\u3046\u6307\u793a\u3057\u3066\u3044\u308b\u304b\u3082\u3057\u308c\u307e\u305b\u3093\u3002<\/p>\n\n\n\n<p>\u3053\u306e\u3088\u3046\u306a\u6a5f\u80fd\u3092\u5b9f\u73fe\u3057\u3066\u3044\u308b\u3068\u3057\u305f\u3089\u3001\u6df1\u5c64\u5b66\u7fd2\u306e\u6027\u80fd\u304c\u30b0\u30c3\u3068\u4e0a\u304c\u308b\u304b\u3082\uff01\u3063\u3066\u601d\u3044\u307e\u305b\u3093\u304b\uff1f<\/p>\n\n\n\n<p>\u5b9f\u969b\u306b\u3001Attention\u6a5f\u69cb\u306b\u3088\u3063\u3066\u6df1\u5c64\u5b66\u7fd2\u306e\u7cbe\u5ea6\u306f\u5927\u304d\u304f\u4e0a\u304c\u308a\u307e\u3059\u3002\u4ee5\u4e0b\u306b\u6df1\u5c64\u5b66\u7fd2\u306b\u304a\u3051\u308bAttention\u6a5f\u69cb\u306e\u6982\u8981\u3092\u7b87\u6761\u66f8\u304d\u3067\u793a\u3057\u307e\u3059\u3002<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>\u4eca\u307e\u3067\u6df1\u5c64\u5b66\u7fd2\u3067\u4e2d\u5fc3\u7684\u5b58\u5728\u3092\u793a\u3057\u3066\u304d\u305fCNN\u3084RNN\u306b\u4e26\u30763\u3064\u76ee\u306e\u753b\u671f\u7684\u306a\u30a2\u30fc\u30ad\u30c6\u30af\u30c1\u30e3\u3002\u4e3b\u306a\u4f7f\u3044\u65b9\u306f\u4ee5\u4e0b\u306e2\u3064\u3002\n<ul class=\"wp-block-list\">\n<li>CNN\u3084RNN\u306bAttention\u6a5f\u69cb\u3092\u5c0e\u5165\u3059\u308b<\/li>\n\n\n\n<li>\u5b8c\u5168\u306bAttention\u30d9\u30fc\u30b9\u306e\u30e2\u30c7\u30eb\u306b\u3059\u308b\uff08Transformer\u306a\u3069\uff09<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>\u30b3\u30f3\u30d4\u30e5\u30fc\u30bf\u30d3\u30b8\u30e7\u30f3\u3084\u81ea\u7136\u8a00\u8a9e\u51e6\u7406\u306a\u3069\u306e\u591a\u7a2e\u591a\u69d8\u306a\u30bf\u30b9\u30af\u306b\u5bfe\u3057\u3066\u6700\u5148\u7aef\u306e\u6027\u80fd\u3092\u793a\u3057\u305f\uff08SOTA\u3092\u9054\u6210\uff01\u3068\u3088\u304f\u8a00\u308f\u308c\u308b\uff09\n<ul class=\"wp-block-list\">\n<li>\u8ffd\u3005\u8aac\u660e\u3059\u308bTransformer\u304c\u5927\u3044\u306b\u95a2\u4fc2\u3057\u3066\u3044\u308b<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>\u89e3\u91c8\u53ef\u80fd\u6027\u306e\u5411\u4e0a\n<ul class=\"wp-block-list\">\n<li>\u3053\u308c\u306f\u7d50\u69cb\u91cd\u8981\u3067\u3001\u8fd1\u5e74\u91cd\u8981\u8996\u3055\u308c\u3066\u3044\u308b\u89e3\u91c8\u53ef\u80fd\u306aAI\uff08XAI\uff09\u306b\u95a2\u9023\u3042\u308a<\/li>\n\n\n\n<li>\u4f55\u3089\u304b\u306e\u65b9\u6cd5\u3067Attention\u3092\u53ef\u8996\u5316\u3059\u308b\u3053\u3068\u3067\u30cb\u30e5\u30fc\u30e9\u30eb\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u304c\u4f55\u51e6\u306b\u6ce8\u76ee\u3057\u3066\u3044\u308b\u304b\u3001\u306a\u305c\u3001\u305d\u306e\u51fa\u529b\u7d50\u679c\u3068\u306a\u308b\u306e\u304b\u306b\u95a2\u3057\u3066\u60c5\u5831\u3092\u4e0e\u3048\u3066\u304f\u308c\u308b<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<p>\u4ee5\u964d\u306e\u7ae0\u3067\u306f\u3001\u3053\u308c\u3089\u306b\u3064\u3044\u3066\u8aac\u660e\u3057\u3066\u3044\u304d\u307e\u3059\u3002<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Attention\u6a5f\u69cb\u306e\u8d77\u6e90<\/h2>\n\n\n\n<p>Attention\u6a5f\u69cb\u3068\u3044\u3046\u306e\u306f\u3001CNN\u3084RNN\u306b\u4e26\u30763\u3064\u76ee\u306e\u6a5f\u69cb\u3068\u3057\u3066\u7d39\u4ecb\u3055\u308c\u308b\u3053\u3068\u304c\u591a\u3044\u3067\u3059\u304c\u3001\u305d\u306e\u8d77\u6e90\u306fCNN\u3084RNN\u3068\u5f37\u3044\u95a2\u308f\u308a\u3092\u6301\u3063\u3066\u3044\u307e\u3059\u3002\u3068\u3044\u3046\u306e\u3082\u3001Attention\u6a5f\u69cb\u3068\u3057\u3066\u4e00\u822c\u5316\u3055\u308c\u308b\u4ee5\u524d\u306e\u3001Attention\u6a5f\u69cb\u306b\u4f3c\u305f\u30e1\u30ab\u30cb\u30ba\u30e0\u306f\u3001CNN\u304a\u3088\u3073RNN\u306e\u4e21\u5206\u91ce\u3067\u307f\u3089\u308c\u307e\u3059\u3002\u5177\u4f53\u7684\u306b\u306f\u3001CNN\u5206\u91ce\u306b\u304a\u3044\u3066\u306fSENet\u3001RNN\u5206\u91ce\u306b\u304a\u3044\u3066\u306f\u8a9e\u9806\u304c\u8003\u616e\u3067\u304d\u308bSeq2Seq\u3067\u3059\u3002\u3053\u308c\u306f\u3001Appendix\u7684\u306a\u5185\u5bb9\u3067\u3059\u304c\u3001Memory Network\u3068\u3044\u3046\u3082\u306e\u3082\u3042\u3063\u305f\u308a\u3057\u307e\u3059\u3002<\/p>\n\n\n\n<p>\u305d\u3053\u3067\u3001\u672c\u7ae0\u3067\u306f\u3001RNN\u304a\u3088\u3073CNN\u306e\u57fa\u790e\u7684\u306a\u5185\u5bb9\u304b\u3089\u8aac\u660e\u3057\u3001\u8a9e\u9806\u304c\u8003\u616e\u3067\u304d\u308bSeq2Seq\uff08\u4ee5\u4e0b\uff1aSeq2Seq+Attention\uff09\u3068SENet\u3001Memory Network\u306b\u3064\u3044\u3066\u89e3\u8aac\u3057\u3066\u3044\u304d\u307e\u3059\u3002<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">RNN\u306e\u767a\u5c55\u3068Attention\u306e\u8d77\u6e90<\/h3>\n\n\n\n<p>Attention\u6a5f\u69cb\u306e\u8d77\u6e90\u3092\u8aac\u660e\u3059\u308b\u3068\u304d\u3001RNN\u3068CNN\u306e\u3069\u3061\u3089\u304b\u3089\u8aac\u660e\u3059\u308b\u304b\u8ff7\u3044\u307e\u3057\u305f\u304c\u3001RNN\u304b\u3089\u8aac\u660e\u3059\u308b\u3053\u3068\u306b\u3057\u307e\u3059\u3002<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">RNN\u306e\u57fa\u790e<\/h4>\n\n\n\n<p>RNN\uff08Recurrent Neural Network: RNN\uff09\u3068\u306f\u3001\u5185\u90e8\u306b\u518d\u5e30\u7684\u306a\u69cb\u9020\u3092\u6301\u3064\u30cb\u30e5\u30fc\u30e9\u30eb\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u3067\u3001\u6c17\u6e29\u5909\u52d5\u3001\u8a00\u8a9e\u51e6\u7406\u3068\u3044\u3063\u305f\u7cfb\u5217\u7684\u306a\u60c5\u5831\u306e\u8b58\u5225\u3001\u63a8\u8ad6\u306b\u4f7f\u7528\u3055\u308c\u307e\u3059\u3002\u518d\u5e30\u69cb\u9020\u3092\u6301\u305f\u306a\u3044\u30cb\u30e5\u30fc\u30e9\u30eb\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u3067\u7cfb\u5217\u30c7\u30fc\u30bf\u3092\u6271\u304a\u3046\u3068\u3059\u308b\u3068\u3001\u6642\u523bt\u2212n\u304b\u3089\u6642\u523bt\u307e\u3067\u306e\u5165\u529b\u60c5\u5831\u3092\u3059\u3079\u3066\u4e00\u62ec\u3067\u4e0e\u3048\u308b\u5fc5\u8981\u304c\u3042\u308b\u306a\u3069\u6271\u3044\u306b\u304f\u304f\u306a\u308a\u307e\u3059\u304c\u3001RNN\u3067\u306f\u5185\u90e8\u306e\u72b6\u614b\u3092\u6b21\u6642\u523b\u306b\u4f1d\u64ad\u3055\u305b\u308b\u518d\u5e30\u69cb\u9020\u306b\u3088\u308a\u3001\u9010\u6b21\u7684\u306a\u5165\u529b\u74b0\u5883\u306b\u304a\u3044\u3066\u3001\u6642\u9593\u4f9d\u5b58\u306e\u7279\u5fb4\u3092\u5b66\u7fd2\u3067\u304d\u307e\u3059\u3002<\/p>\n\n\n\n<p>\u3053\u3053\u3067\u306f\u3001Seq2Seq\u3068\u547c\u3070\u308c\u308b\u7cfb\u5217\u30e2\u30c7\u30eb\u306bAttention\u6a5f\u69cb\u3092\u8ffd\u52a0\u3057\u305f\u30e2\u30c7\u30eb\u306b\u3064\u3044\u3066\u7406\u89e3\u3067\u304d\u308b\u3088\u3046\u306b\u3001\u305d\u306e\u57fa\u790e\u77e5\u8b58\u3068\u3057\u3066\u3001RNN\u306e\u8868\u8a18\u3001RNN\u306e\u7a2e\u985e\u3001Encoder-Decoder\u306b\u3064\u3044\u3066\u89e3\u8aac\u3057\u307e\u3059\u3002<\/p>\n\n\n\n<h5 class=\"wp-block-heading\">RNN\u306e\u8868\u8a18<\/h5>\n\n\n\n<p>RNN\u3092\u56f3\u793a\u3057\u305f\u56f3\u306e\u8868\u8a18\u306f\u3001\u6642\u9593\u8ef8\u3067\u5c55\u958b\u3057\u306a\u3044\u5834\u5408\uff08\u4e0b\u56f3\u5de6\uff09\u3068\u5c55\u958b\u3057\u305f\u5834\u5408\uff08\u4e0b\u56f3\u53f3\uff09\u306e2\u7a2e\u985e\u304c\u5b58\u5728\u3057\u307e\u3059\u3002\u6642\u9593\u8ef8\u3067\u5c55\u958b\u3057\u306a\u3044\u5834\u5408\u3001\u5165\u529b\u4f4d\u7f6e\u3068\u51fa\u529b\u958b\u59cb\u4f4d\u7f6e\u304c\u7570\u306a\u308b\u3088\u3046\u306a\u30e2\u30c7\u30eb\u3092\u4e0a\u624b\u304f\u66f8\u304d\u8868\u305b\u306a\u3044\u306e\u3067\u3001\u4ee5\u964d\u3067\u306f\u53f3\u306e\u56f3\u306e\u3088\u3046\u306a\u6642\u9593\u8ef8\u3067\u5c55\u958b\u3057\u305f\u56f3\u3092\u4f7f\u7528\u3057\u3066\u3044\u304d\u307e\u3059\u3002<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" width=\"613\" height=\"270\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-3.png\" alt=\"\" class=\"wp-image-4123\" srcset=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-3.png 613w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-3-300x132.png 300w\" sizes=\"(max-width: 613px) 100vw, 613px\" \/><\/figure><\/div>\n\n\n<p>RNN\u3067\u306f\u3001\u4e2d\u9593\u5c64\u306b\u4f7f\u7528\u3055\u308c\u3066\u3044\u308b\u518d\u5e30\u5c64\u3068\u3057\u3066\u4f55\u3092\u4f7f\u3046\u304b\u306b\u3088\u3063\u3066\u3001\u8868\u73fe\u80fd\u529b\u304c\u5909\u308f\u308a\u307e\u3059\u3002\u6700\u3082\u5358\u7d14\u306a\u3082\u306e\u3060\u3068\u3001\u5165\u529b\u5c64\u3084\u51fa\u529b\u5c64\u3067\u4f7f\u7528\u3059\u308b\u3088\u3046\u306a\u30cb\u30e5\u30fc\u30ed\u30f3\u5c64\u3092\u4f7f\u3044\u307e\u3059\u3057\u3001\u8907\u96d1\u306a\u3082\u306e\u3060\u3068LSTM\uff08Long Short Term Memory\uff09\u3001GRU\uff08Gated Recurrent Unit\uff09\u3001SRU\uff08Simple Recurrent Unit\uff09\u306a\u3069\u3092\u4f7f\u7528\u3059\u308b\u3053\u3068\u3082\u3042\u308a\u307e\u3059\u3002<\/p>\n\n\n\n<p>\u5c11\u3057\u4f59\u8ac7\u3067\u3059\u304c\u3001LSTM\u5c64\u3092\u6642\u9593\u8ef8\u3067\u5c55\u958b\u3057\u305f\u5834\u5408\u3068\u5c55\u958b\u3057\u306a\u3044\u5834\u5408\u306e\u56f3\u306f\u4ee5\u4e0b\u306e\u3088\u3046\u306b\u306a\u308a\u307e\u3059\u3002<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" width=\"1024\" height=\"340\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-4-1024x340.jpg\" alt=\"\" class=\"wp-image-4136\" srcset=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-4-1024x340.jpg 1024w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-4-300x99.jpg 300w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-4-768x255.jpg 768w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-4.jpg 1164w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure><\/div>\n\n\n<p>\u8a73\u3057\u304f\u77e5\u308a\u305f\u3044\u65b9\u306f\u3001\u4ee5\u4e0b\u306e\u8a18\u4e8b\u3092\u3054\u53c2\u8003\u304f\u3060\u3055\u3044\u3002<\/p>\n\n\n\n<figure class=\"wp-block-embed is-type-wp-embed is-provider-agirobots-blog wp-block-embed-agirobots-blog\"><div class=\"wp-block-embed__wrapper\">\n<div><a href=\"https:\/\/developers.agirobots.com\/jp\/lstmgruentrance-noformula\/\" class=\"st-cardlink st-embed-cardlink\"><div class=\"kanren st-cardbox\"><dl class=\"clearfix\"><dt class=\"st-card-img\"><img decoding=\"async\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2020\/07\/246b34fd7c71050c39d5575a56f8909e-150x150.png\" alt=\"\" width=\"100\" height=\"100\" \/><\/dt><dd><p class=\"st-cardbox-t\">\u76f4\u611f\u3067\u7406\u89e3\u3059\u308bLSTM\u30fbGRU\u5165\u9580 - \u6a5f\u68b0\u5b66\u7fd2\u306e\u57fa\u790e\u3092\u30de\u30b9\u30bf\u30fc\u3057\u3088\u3046\uff01<\/p><div class=\"st-card-excerpt smanone\"><p>\u3053\u308c\u3060\u3051\u306f\uff08\u30dd\u30a4\u30f3\u30c8\uff09 \u672c\u8a18\u4e8b\u3067\u8a71\u3059\u5185\u5bb9\u306e\u4e2d\u3067\u3001\u91cd\u8981\u306a\u90e8\u5206\u3092\u30d4\u30c3\u30af\u30a2\u30c3\u30d7\u3057\u3066\u7b87\u6761\u66f8\u304d\u306b\u3057\u307e\u3057\u305f\u3002 \u9806\u4f1d\u64ad\u578b\u30cb\u30e5\u30fc\u30e9\u30eb\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\uff08FNN\uff09 \u9806\u4f1d\u64ad\u578b\u30cb\u30e5\u30fc\u30e9\u30eb\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\uff08Feedforward Ne ... <\/p><\/div><\/dd><\/dl><\/div><\/a><\/div>\n<\/div><\/figure>\n\n\n\n<p>\u52d5\u753b\u3067\u3082\u8aac\u660e\u3057\u3066\u3044\u307e\u3059\u3002<\/p>\n\n\n\n<h5 class=\"wp-block-heading\">RNN\u306e\u7a2e\u985e<\/h5>\n\n\n\n<p>\u3053\u3053\u3067\u306f\u3001\u5165\u51fa\u529b\u5f62\u5f0f\u306b\u6ce8\u76ee\u3057\u305f\u3068\u304d\u306eRNN\u306e\u7a2e\u985e\u306f\u4e0b\u56f3\u306e\u3088\u3046\u306b4\u3064\u306b\u5206\u3051\u308b\u3053\u3068\u304c\u3067\u304d\u307e\u3059\u3002<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Simple RNN\uff1a\u7cfb\u5217\uff08\u30b7\u30fc\u30b1\u30f3\u30b9\uff09\u3092\u5165\u529b\u3068\u3057\u3066\u53d7\u3051\u7d9a\u3051\u3001\u540c\u6642\u306b\u3001\u51fa\u529b\u3068\u3057\u3066\u5909\u63db\u5f8c\u306e\u7cfb\u5217\uff08\u30b7\u30fc\u30b1\u30f3\u30b9\uff09\u3092\u51fa\u529b\u3057\u7d9a\u3051\u308b<\/li>\n\n\n\n<li>Seq2Vec\uff1a\u6709\u9650\u9577\u306e\u7cfb\u5217\uff08\u30b7\u30fc\u30b1\u30f3\u30b9\uff09\u3092\u53d7\u3051\u53d6\u308a\u6700\u5f8c\u306b\u30d9\u30af\u30c8\u30eb\u3092\u51fa\u529b\u3059\u308b<\/li>\n\n\n\n<li>Vec2Seq\uff1a\u30d9\u30af\u30c8\u30eb\u3092\u5165\u529b\u3068\u3057\u3066\u53d7\u3051\u53d6\u3063\u305f\u76f4\u5f8c\u304b\u3089\u3001\u6709\u9650\u9577\u306e\u7cfb\u5217\uff08\u30b7\u30fc\u30b1\u30f3\u30b9\uff09\u3092\u51fa\u529b\u3059\u308b<\/li>\n\n\n\n<li>Seq2Seq\uff1a\u6709\u9650\u9577\u306e\u7cfb\u5217\uff08\u30b7\u30fc\u30b1\u30f3\u30b9\uff09\u3092\u53d7\u3051\u53d6\u3063\u3066\u6709\u9650\u9577\u306e\u7cfb\u5217\uff08\u30b7\u30fc\u30b1\u30f3\u30b9\uff09\u3092\u51fa\u529b\u3059\u308b<\/li>\n<\/ul>\n\n\n\n<p>\u3069\u306e\u7a2e\u985e\u306eRNN\u3092\u4f7f\u7528\u3059\u308b\u304b\u306f\u3001\u6271\u3046\u5bfe\u8c61\u306b\u4f9d\u5b58\u3057\u307e\u3059\u3002\u4f8b\u3048\u3070\u3001\u6c17\u6e29\u60c5\u5831\u3092\u30ea\u30a2\u30eb\u30bf\u30a4\u30e0\u306b\u53d7\u3051\u53d6\u308a\u3001\u6b21\u306e\u6642\u523b\u306e\u6c17\u6e29\u3092\u4e88\u6e2c\u3059\u308b\u5834\u5408\u306fSimple RNN\u578b\u304c\u76f8\u5fdc\u3057\u3044\u3067\u3059\u3057\u3001\u4eca\u65e5\u4e00\u65e5\u306e\u6c17\u6e29\u5909\u5316\u3092\u53d7\u3051\u53d6\u3063\u3066\u5feb\u9069\u5ea6\u3092\u51fa\u529b\u3055\u305b\u305f\u3044\u5834\u5408\u306fSeq2Vec\u578b\u304c\u76f8\u5fdc\u3057\u3044\u3068\u601d\u3044\u307e\u3059\u3002Vec2Seq\u306a\u3089\u5feb\u9069\u3055\u306e\u6307\u6a19\u3092\u53d7\u3051\u53d6\u3063\u3066\u3001\u76f8\u5fdc\u3057\u3044\u6c17\u6e29\u8fba\u5909\u5316\u3092\u51fa\u529b\u3055\u305b\u308b\u3053\u3068\u304c\u3067\u304d\u308b\u304b\u3082\u3057\u308c\u307e\u305b\u3093\u3002Seq2Seq\u306f\u6a5f\u68b0\u7ffb\u8a33\u3092\u4e2d\u5fc3\u306b\u4f7f\u7528\u3055\u308c\u308b\u30e2\u30c7\u30eb\u3067\u3001\u65e5\u672c\u8a9e\u3092\u5165\u529b\u3057\u3066\u82f1\u8a9e\u3092\u51fa\u529b\u3059\u308b\u3068\u3044\u3063\u305f\u4f7f\u3044\u65b9\u304c\u53ef\u80fd\u3067\u3059\u3002<\/p>\n\n\n\n<p>\u4ee5\u964d\u3067\u306f\u3001Seq2Seq\u30e2\u30c7\u30eb\u306b\u7126\u70b9\u3092\u5f53\u3066\u3066\u89e3\u8aac\u3057\u3066\u3044\u304d\u307e\u3059\u3002<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" width=\"1024\" height=\"539\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-7-1024x539.png\" alt=\"\" class=\"wp-image-4139\" srcset=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-7-1024x539.png 1024w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-7-300x158.png 300w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-7-768x404.png 768w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-7.png 1114w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure><\/div>\n\n\n<h5 class=\"wp-block-heading\">Encoder-Decoder\u30e2\u30c7\u30eb<\/h5>\n\n\n\n<p>Seq2Seq\u306e\u8a73\u3057\u3044\u5185\u5bb9\u306b\u79fb\u308b\u524d\u306b\u3001\u3061\u3087\u3063\u3068\u3057\u305f\u7d39\u4ecb\u3067\u3059\u304c\u3001Seq2Seq\u30e2\u30c7\u30eb\u306e\u8a2d\u8a08\u306fEncoder-Decoder\u3068\u3044\u3046\u8a2d\u8a08\u601d\u60f3\u306b\u57fa\u3065\u3044\u3066\u3044\u307e\u3059\u3002\u3053\u306e\u3001Encoder-Decoder\u3068\u3044\u3046\u8a2d\u8a08\u601d\u60f3\u306b\u57fa\u3065\u3044\u305f\u3082\u306e\u3068\u3057\u3066\u3001\u4ed6\u306bAutoEncoder\u304c\u3042\u308a\u307e\u3059\u3002<\/p>\n\n\n\n<p>Encoder-Decoder\u30e2\u30c7\u30eb\u3068\u306f\u3001\u304b\u306a\u308a\u62bd\u8c61\u5ea6\u306e\u9ad8\u3044\u8a2d\u8a08\u601d\u60f3\u3067\u3059\u304c\u3001\u524d\u534a\u306eEncoder\u3067\u4e00\u822c\u7684\u306b\u306f\u5165\u529b\u3088\u308a\u5c0f\u3055\u3044\u7279\u5fb4\u7a7a\u9593\u306e\u56fa\u5b9a\u9577\u30d9\u30af\u30c8\u30eb\u306b\u5727\u7e2e\u3057\u3001\u5f8c\u534a\u306eDecoder\u3067\u305d\u306e\u56fa\u5b9a\u9577\u30d9\u30af\u30c8\u30eb\u3092\u5165\u529b\u3068\u3057\u3066\u53d7\u3051\u3066\u5fa9\u5143\u3055\u305b\u308b\u4f7f\u3044\u65b9\u3092\u3057\u307e\u3059\u3002<\/p>\n\n\n\n<p>\u30ad\u30fc\u3068\u306a\u308b\u306e\u306f\u5165\u529b\u6b21\u5143\u3088\u308a\u5c0f\u3055\u306a\u7279\u5fb4\u7a7a\u9593\u3078\u5199\u50cf\u3059\u308b\u969b\u306b\u3001\u5fc5\u8981\u306e\u306a\u3044\u60c5\u5831\u3092\u6368\u3066\u308b\u3053\u3068\u3092\u5b66\u3076\u70b9\u3067\u3059\u3002<\/p>\n\n\n\n<p>\u4f8b\u3048\u3070\u3001AutoEncoder\u3092\u4f7f\u3063\u3066\u3001\u51fa\u529b\u753b\u50cf\u3092\u5165\u529b\u753b\u50cf\u306b\u8fd1\u3065\u3051\u308b\u3088\u3046\u306b\u5b66\u7fd2\u3055\u305b\u305f\u5834\u5408\u3001\u91cd\u8981\u5ea6\u306e\u4f4e\u3044\u30ce\u30a4\u30ba\u60c5\u5831\u306f\u5207\u308a\u6368\u3066\u308b\u3088\u3046\u306b\u5b66\u7fd2\u304c\u884c\u308f\u308c\u3001\u30ce\u30a4\u30ba\u9664\u53bb\u306b\u4f7f\u3048\u308b\u30e2\u30c7\u30eb\u3092\u5f97\u308b\u3053\u3068\u304c\u3067\u304d\u307e\u3059\u3002<\/p>\n\n\n\n<p>Encoder-Decoder\u306e\u91cd\u8981\u306a\u30dd\u30a4\u30f3\u30c8\u306f\u3001\u5165\u529b\u3088\u308a\u5c0f\u3055\u3044\u7279\u5fb4\u7a7a\u9593\u306e\u56fa\u5b9a\u9577\u30d9\u30af\u30c8\u30eb\u306b\u5727\u7e2e\u3059\u308b\u3068\u304d\u306b\u3001\u672c\u8cea\u7684\u3067\u306a\u3044\u60c5\u5831\u3092\u6368\u3066\u308b\u5b66\u7fd2\u3092\u3059\u308b\u3053\u3068\u3067\u3059\u3002<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>\u9577\u6240\uff1a\u672c\u8cea\u7684\u306a\u60c5\u5831\u3092\u62bd\u51fa\u3059\u308b\u306e\u304c\u5f97\u610f<\/li>\n\n\n\n<li>\u77ed\u6240\uff1a\u5177\u4f53\u7684\u306a\u5165\u529b\u306b\u5bfe\u3057\u3066\u3001\u5177\u4f53\u7684\u306a\u51fa\u529b\u3092\u5909\u3048\u308b\u3053\u3068\u304c\u82e6\u624b<\/li>\n<\/ul>\n\n\n\n<p>\u3053\u306e\u6a5f\u80fd\u306f\u3001\u7279\u5fb4\u62bd\u51fa\u3068\u3044\u3046\u5074\u9762\u3067\u306f\u512a\u308c\u3066\u3044\u308b\u304b\u3082\u3057\u308c\u307e\u305b\u3093\u304c\u3001\u69d8\u3005\u306a\u8a00\u3044\u56de\u3057\u304c\u53ef\u80fd\u306a\u6587\u7ae0\u8868\u73fe\u306a\u3069\u3001\u672c\u8cea\u304c\u540c\u3058\uff08=\u7b26\u53f7\u5316\u3057\u305f\u30d9\u30af\u30c8\u30eb\u304c\u985e\u4f3c\uff09\u3067\u3082\u3001\u5165\u529b\u3055\u308c\u305f\u6587\u7ae0\u8868\u73fe\u306b\u3088\u3063\u3066\u3001\u7ffb\u8a33\u3092\u5909\u5316\u3055\u305b\u305f\u3044\u3068\u3044\u3063\u305f\u30bf\u30b9\u30af\u306f\u5b9f\u73fe\u304c\u96e3\u3057\u304f\u306a\u308b\u3068\u8003\u3048\u3089\u308c\u307e\u3059\u3002<\/p>\n\n\n\n<p>\u3053\u306e\u3088\u3046\u306a\u7406\u7531\u304b\u3089\u3001Attention\u3092\u5c0e\u5165\u3059\u308b\u52d5\u6a5f\u304c\u751f\u307e\u308c\u305f\u3068\u3044\u3048\u307e\u3059\u3002\u307e\u3055\u306b\u3001\u5177\u4f53\u7684\u306a\u6587\u7ae0\u5165\u529b\u304b\u3089\u672c\u8cea\u7684\u306a\u7279\u5fb4\u3092\u62bd\u51fa\u3059\u308b\u3053\u3068\u304c\u5f97\u610f\u306aSeq2Seq\u306b\u3001\u3055\u307e\u3056\u307e\u306a\u8a00\u3044\u56de\u3057\u306a\u3069\u5177\u4f53\u7684\u306a\u8868\u73fe\u306e\u6587\u7ae0\u3092\u51fa\u529b\u3067\u304d\u308b\u3088\u3046\u3001Attention\u3063\u307d\u3044\u30e1\u30ab\u30cb\u30ba\u30e0\u3092\u52a0\u3048\u308b\u3068\u3044\u3046\u8a71\u306b\u7e4b\u304c\u308a\u307e\u3059\u3002\u8a73\u3057\u3044\u8a71\u306f\u4ee5\u964d\u306e\u30bb\u30af\u30b7\u30e7\u30f3\u3067\u8aac\u660e\u3057\u307e\u3059\u3002<\/p>\n\n\n\n<p>Encoder-Decoder\u306e\u8ab2\u984c\u70b9\u306f2\u3064\u3067\u3001<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>\u5168\u3066\u306e\u5165\u529b\u30921\u3064\u306e\u56fa\u5b9a\u9577\u30d9\u30af\u30c8\u30eb\u306b\u5199\u50cf\u3059\u308b\u305f\u3081\u3001\u5165\u51fa\u529b\u9593\u306e\u4e26\u3073\u65b9\u306b\u3064\u3044\u3066\u30eb\u30fc\u30eb\u3092\u5b66\u3076\u3053\u3068\u304c\u3067\u304d\u306a\u3044<\/li>\n<\/ul>\n\n\n\n<p>\u3068\u3044\u3046\u70b9\u3067\u3059\u30021\u3064\u306e\u56fa\u5b9a\u9577\u30d9\u30af\u30c8\u30eb\u306f\u5165\u529b\u306e\u672c\u8cea\u60c5\u5831\uff08what\uff09\u3067\u3042\u308a\u3001\u305d\u306e\u5165\u529b\u3001\u5373\u3061\u6587\u8868\u73fe\u3084\u5199\u771f\u8868\u73fe\u3068\u3044\u3063\u305f\uff08How to\uff09\u3092\u5b66\u3079\u306a\u3044\u306e\u3067\u3001How to\u3092\u5b66\u3076\u306e\u304cAttention\u3068\u3044\u3046\u89e3\u91c8\u3082\u53ef\u80fd\u3067\u3059\u3002<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" width=\"651\" height=\"475\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-8.png\" alt=\"\" class=\"wp-image-4140\" srcset=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-8.png 651w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-8-300x219.png 300w\" sizes=\"(max-width: 651px) 100vw, 651px\" \/><\/figure><\/div>\n\n\n<h4 class=\"wp-block-heading\">Seq2Seq<\/h4>\n\n\n\n<p>seq2seq\u306fEncoder-Decoder\u30e2\u30c7\u30eb\u306e1\u3064\u3067\u3001\u81ea\u7136\u8a00\u8a9e\u3084\u97f3\u58f0\u306a\u3069\u306e\u7cfb\u5217\u30c7\u30fc\u30bf\u3092\u5bfe\u8c61\u3068\u3057\u3066\u3001\u5165\u529b\u3055\u308c\u305f\u7cfb\u5217\u60c5\u5831\u3092\u5225\u306e\u7cfb\u5217\u60c5\u5831\u306b\u5909\u63db\u3059\u308b\u30e2\u30c7\u30eb\u3067\u3059\u3002\u4ee3\u8868\u7684\u306a\u7528\u9014\u306f\u6a5f\u68b0\u7ffb\u8a33\u3067\u3059\u3002\u6a5f\u68b0\u7ffb\u8a33\u3068\u3044\u3046\u306e\u306f\u3001\u300cThis is a pen.\u300d\u3092\u300c\u3053\u308c\u306f\u30da\u30f3\u3067\u3059\u3002\u300d\u306b\u7ffb\u8a33\u3059\u308b\u3053\u3068\u3067\u3059\u3002\u3053\u3053\u3067\u306f\u3001\u4eca\u7d39\u4ecb\u3057\u305f\u82f1\u8a9e\u304b\u3089\u65e5\u672c\u8a9e\u3078\u306e\u7ffb\u8a33\u3092\u4f8b\u306b\u8aac\u660e\u3057\u3066\u3044\u304d\u307e\u3059\u3002<\/p>\n\n\n\n<h5 class=\"wp-block-heading\">Seq2Seq\u306b\u3088\u308b\u6a5f\u68b0\u7ffb\u8a33\u3068\u305d\u306e\u8ab2\u984c<\/h5>\n\n\n\n<p>\u4e0b\u56f3\u306fSeq2Seq\u306e\u5165\u529b\u306b\u300cthis is a pen.\u300d\u3092\u4e0e\u3048\u305f\u969b\u3001\u300c\u3053\u308c\u306f\u30da\u30f3\u3067\u3059\u3002\u300d\u304c\u51fa\u529b\u3055\u308c\u308b\u5834\u5408\u3092\u793a\u3057\u3066\u3044\u307e\u3059\u3002\u3053\u3053\u3067\u3001this\u306et\u304c\u5927\u6587\u5b57\u3067\u306a\u3044\u7406\u7531\u3092\u8aac\u660e\u3057\u3066\u304a\u304d\u307e\u3059\u3002\u5b9f\u969b\u306b\u30e2\u30c7\u30eb\u306b\u5358\u8a9e\u3092\u5165\u529b\u3059\u308b\u969b\u3001\u57cb\u3081\u8fbc\u307f\u5c64\u3092\u4f7f\u7528\u3057\u3066\u5358\u8a9e\u3092\u57cb\u3081\u8fbc\u307f\u30d9\u30af\u30c8\u30eb\u306b\u5909\u63db\u3057\u307e\u3059\u304c\u3001\u57cb\u3081\u8fbc\u307f\u5c64\u304c\u5bfe\u5fdc\u3067\u304d\u308b\u5358\u8a9e\u5f62\u306b\u5909\u63db\u3059\u308b\u30d7\u30ed\u30bb\u30b9\u304c\u5fc5\u8981\u3067\u3001\u305d\u306e\u30d7\u30ed\u30bb\u30b9\u3067\u5927\u6587\u5b57\u304c\u5c0f\u6587\u5b57\u306b\u5909\u63db\u3055\u308c\u305f\u3068\u89e3\u91c8\u3057\u3066\u304f\u3060\u3055\u3044\u3002\u3082\u3057\u3001\u5927\u6587\u5b57\u5165\u529b\u3092\u8a8d\u3081\u308b\u3088\u3046\u306b\u8a2d\u8a08\u3059\u308c\u3070\u3001\u5927\u6587\u5b57\u3092\u5c0f\u6587\u5b57\u306b\u5909\u63db\u3059\u308b\u5fc5\u8981\u306f\u3042\u308a\u307e\u305b\u3093\u304c\u3001\u4eca\u56de\u306e\u4f8b\u3067\u306f\u3001\u305d\u308c\u3092\u8a8d\u3081\u306a\u3044\u3053\u3068\u3068\u3057\u305f\u3060\u3051\u3067\u3059\u3002<\/p>\n\n\n\n<p>\u56f3\u306e\u4e2d\u306b\u3042\u308b\u3001\u3008BOS\u3009\u306fbeginning of the sentence\u306e\u7565\u3067\u7ffb\u8a33\u958b\u59cb\u3092\u8868\u3057\u3066\u3044\u307e\u3059\u3002\u3008EOS\u3009\u306fend of sentence\u306e\u7565\u3067\u7ffb\u8a33\u7d42\u4e86\u3092\u8868\u3057\u307e\u3059\u3002\u3064\u307e\u308a\u3001\u3008BOS\u3009\u3092\u53d7\u3051\u53d6\u308b\u3068\u76f4\u3061\u306b\u7ffb\u8a33\u3092\u958b\u59cb\u3057\u3001\u7ffb\u8a33\u304c\u7d42\u4e86\u3059\u308b\u3068\u3008EOS\u3009\u3092\u51fa\u529b\u3057\u307e\u3059\u3002<\/p>\n\n\n\n<p>\u3053\u3053\u3067\u3001Seq2Seq\u306b\u306f\u554f\u984c\u70b9\u304c\u3042\u308a\u307e\u3059\u3002\u305d\u308c\u306f\u3001\u30a8\u30f3\u30b3\u30fc\u30c0\u3067\u3001\u5165\u529b\u7cfb\u5217\u306e\u30b5\u30a4\u30ba\u306b\u95a2\u308f\u3089\u305a\u3001\u56fa\u5b9a\u9577\u306e\u7279\u5fb4\u30d9\u30af\u30c8\u30eb\u306b\u7b26\u53f7\u5316\u3055\u308c\u308b\u3053\u3068\u3067\u3059\u3002\u5165\u529b\u30b5\u30a4\u30ba\u304c\u7570\u306a\u308b\u6587\u7ae0\u3067\u3042\u3063\u3066\u3082\u3001\u540c\u3058\u7279\u5fb4\u7a7a\u9593\u8868\u73fe\u306b\u7b26\u53f7\u5316\u3059\u308b\u305f\u3081\u3001\u9069\u5207\u306a\u5909\u63db\u3092\u65bd\u305b\u308b\u3068\u306f\u9650\u308a\u307e\u305b\u3093\u3002\u7279\u306b\u3001\u9577\u3044\u6587\u66f8\u306f\u6271\u3044\u3065\u3089\u304f\u306a\u308a\u307e\u3059\u3002\u307e\u305f\u3001Encoder\u306e\u51fa\u529b\u306fDecoder\u306e\u6700\u521d\u306e\u5c64\u306b\u306e\u307f\u6e21\u3055\u308c\u308b\u305f\u3081\u3001\u7ffb\u8a33\u304c\u9032\u3080\u306b\u3064\u308c\u6700\u521d\u306bDecoder\u304c\u53d7\u3051\u53d6\u3063\u305f\u60c5\u5831\u306f\u8584\u308c\u3066\u3044\u304d\u307e\u3059\u3002<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" width=\"1024\" height=\"197\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-23-1024x197.png\" alt=\"\" class=\"wp-image-4157\" srcset=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-23-1024x197.png 1024w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-23-300x58.png 300w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-23-768x148.png 768w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-23.png 1527w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure><\/div>\n\n\n<h5 class=\"wp-block-heading\">\u8997\u304d\u7a74\uff08Peeky\uff09\u3092\u6301\u3064Seq2Seq<\/h5>\n\n\n\n<p>\u7ffb\u8a33\u304c\u9032\u3080\u306b\u3064\u308c\u3066\u3001Decoder\u304c\u6700\u521d\u306b\u53d7\u3051\u53d6\u3063\u305fEncoder\u306e\u51fa\u529b\u60c5\u5831\u304c\u8584\u308c\u3066\u3057\u307e\u3046\u3068\u3044\u3046\u554f\u984c\u306f\u3001\u8997\u304d\u7a74\uff08Peeky\uff09\u69cb\u9020\u3092\u7528\u610f\u3059\u308b\u3053\u3068\u3067\u5927\u4f53\u3001\u89e3\u6c7a\u3067\u304d\u307e\u3059\u3002\u8997\u304d\u7a74\uff08Peeky\uff09\u306f\u3001\u4e00\u822c\u7684\u306b\u306f\u898b\u3048\u306a\u3044\u306f\u305a\u306e\u60c5\u5831\u3092\u8997\u304f\u3053\u3068\u304c\u3067\u304d\u308b\u4ed5\u7d44\u307f\u3067\u3001LSTM\u3067\u306f\u30d4\u30fc\u30d7\u30db\u30fc\u30eb\u7d50\u5408\uff08\u8997\u304d\u7a74\u7d50\u5408\uff09\u3068\u3057\u3066\u77e5\u3089\u308c\u3066\u3044\u307e\u3059\u3002\u8208\u5473\u304c\u3042\u308c\u3070\u3001\u4ee5\u4e0b\u306e\u8a18\u4e8b\u3092\u8aad\u3093\u3067\u307f\u3066\u304f\u3060\u3055\u3044\u3002<\/p>\n\n\n\n<figure class=\"wp-block-embed is-type-wp-embed is-provider-agirobots-blog wp-block-embed-agirobots-blog\"><div class=\"wp-block-embed__wrapper\">\n<div><a href=\"https:\/\/developers.agirobots.com\/jp\/lstmgruentrance-noformula\/\" class=\"st-cardlink st-embed-cardlink\"><div class=\"kanren st-cardbox\"><dl class=\"clearfix\"><dt class=\"st-card-img\"><img decoding=\"async\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2020\/07\/246b34fd7c71050c39d5575a56f8909e-150x150.png\" alt=\"\" width=\"100\" height=\"100\" \/><\/dt><dd><p class=\"st-cardbox-t\">\u76f4\u611f\u3067\u7406\u89e3\u3059\u308bLSTM\u30fbGRU\u5165\u9580 - \u6a5f\u68b0\u5b66\u7fd2\u306e\u57fa\u790e\u3092\u30de\u30b9\u30bf\u30fc\u3057\u3088\u3046\uff01<\/p><div class=\"st-card-excerpt smanone\"><p>\u3053\u308c\u3060\u3051\u306f\uff08\u30dd\u30a4\u30f3\u30c8\uff09 \u672c\u8a18\u4e8b\u3067\u8a71\u3059\u5185\u5bb9\u306e\u4e2d\u3067\u3001\u91cd\u8981\u306a\u90e8\u5206\u3092\u30d4\u30c3\u30af\u30a2\u30c3\u30d7\u3057\u3066\u7b87\u6761\u66f8\u304d\u306b\u3057\u307e\u3057\u305f\u3002 \u9806\u4f1d\u64ad\u578b\u30cb\u30e5\u30fc\u30e9\u30eb\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\uff08FNN\uff09 \u9806\u4f1d\u64ad\u578b\u30cb\u30e5\u30fc\u30e9\u30eb\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\uff08Feedforward Ne ... <\/p><\/div><\/dd><\/dl><\/div><\/a><\/div>\n<\/div><\/figure>\n\n\n\n<p>\u5148\u307b\u3069\u306e\u3001Seq2Seq\u30e2\u30c7\u30eb\u306b\u304a\u3044\u3066\u3001Decoder\u304b\u3089\u3044\u3064\u3044\u3067\u3082Encoder\u306e\u51fa\u529b\u3092\u898b\u308b\u3053\u3068\u304c\u3067\u304d\u308b\u8997\u304d\u7a74\u7d50\u5408\u3092\u65bd\u3057\u305f\u5834\u5408\u306e\u3082\u30e2\u30c7\u30eb\u306e\u56f3\u306f\u4ee5\u4e0b\u306e\u3088\u3046\u306b\u306a\u308a\u307e\u3059\u3002<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" width=\"1024\" height=\"212\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-24-1024x212.png\" alt=\"\" class=\"wp-image-4158\" srcset=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-24-1024x212.png 1024w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-24-300x62.png 300w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-24-768x159.png 768w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-24-1536x317.png 1536w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-24.png 1544w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure><\/div>\n\n\n<p>\u3053\u306e\u8997\u304d\u7a74\u7d50\u5408\u306b\u3088\u308a\u3001Decoder\u5074\u304c\u4f55\u3092\u7ffb\u8a33\u3057\u305f\u3044\u306e\u304b\u3068\u3044\u3046\u672c\u8cea\u7684\u306a\u60c5\u5831\u3092\u5e38\u306b\u7247\u624b\u306b\u6301\u3061\u3064\u3064\u7ffb\u8a33\u3059\u308b\u3053\u3068\u304c\u53ef\u80fd\u306b\u306a\u308a\u307e\u3059\u3002\u7ffb\u8a33\u304c\u9032\u3093\u3067\u3082\u8ef8\u304c\u30d6\u30ec\u306b\u304f\u304f\u3001\u9014\u4e2d\u3067\u4f55\u304c\u8a00\u3044\u305f\u304b\u3063\u305f\u3093\u3060\u3063\u3051\uff1f\uff1f\u306a\u72b6\u6cc1\u3092\u9632\u3052\u308b\u30a4\u30e1\u30fc\u30b8\u3067\u3059\u3002<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Seq2Seq + Attention\u6a5f\u69cb<\/h4>\n\n\n\n<p>Seq2Seq\u306fEncoder-Decoder\u30e2\u30c7\u30eb\u306a\u306e\u3067\u3001\u7cfb\u5217\u30c7\u30fc\u30bf\u30921\u3064\u306e\u56fa\u5b9a\u9577\u30d9\u30af\u30c8\u30eb\u306b\u5199\u50cf\u3057\u305d\u308c\u3092\u3001\u518d\u5ea6\u3001\u7cfb\u5217\u30c7\u30fc\u30bf\u306b\u5fa9\u53f7\u3057\u307e\u3059\u3002\u5b66\u7fd2\u5bfe\u8c61\u306e\u6027\u8cea\u4e0a\u3001\u6587\u7ae0\u8868\u73fe\u306a\u3069\u306e\u7cfb\u5217\u30c7\u30fc\u30bf\u306f\u9806\u5e8f\u304c\u91cd\u8981\u306b\u306a\u308b\u3053\u3068\u304c\u591a\u3044\u3067\u3059\u3002<\/p>\n\n\n\n<p>\u4f8b\u3092\u4e0a\u3052\u308b\u3068\u3001\u300c\u660e\u65e5\u306e\u5929\u6c17\u306f\u6674\u308c\u3067\u3059\u300d\u3068\u300c\u660e\u65e5\u306f\u6674\u308c\u306e\u4e88\u5831\u3067\u3059\u300d\u306f\u672c\u8cea\u7684\u306b\u306f\u540c\u3058\u610f\u5473\u3067\u3059\u304c\u3001\u6587\u7ae0\u8868\u73fe\u3068\u3057\u3066\u306f\u7570\u306a\u308a\u307e\u3059\u3002\u3057\u304b\u3057\u3001\u56fa\u5b9a\u9577\u30d9\u30af\u30c8\u30eb\u306b\u30a8\u30f3\u30b3\u30fc\u30c9\u3059\u308b\u3068\u3001\u4f3c\u901a\u3063\u305f\u3082\u306e\u306b\u306a\u308b\u3068\u8003\u3048\u3089\u308c\u307e\u3059\u3002\u3064\u307e\u308a\u3001\u672c\u8cea\u7684\u5185\u5bb9\u304c\u540c\u3058\u3067\u3042\u308c\u3070\u3001\u8a9e\u53e5\u306e\u4e26\u3073\u3084\u8868\u73fe\u306f\u30ce\u30a4\u30ba\u3068\u3057\u3066\u6368\u3066\u3089\u308c\u308b\u305f\u3081\u540c\u3058\u3088\u3046\u306a\u5185\u90e8\u8868\u73fe\uff08\u56fa\u5b9a\u9577\u30d9\u30af\u30c8\u30eb\uff09\u306b\u306a\u308b\u308f\u3051\u3067\u3059\u3002\u3053\u308c\u3067\u306f\u3001\u5165\u529b\u6587\u7ae0\u306e\u6587\u8108\u306b\u95a2\u4fc2\u306a\u304f\u540c\u3058\u3088\u3046\u306a\u7ffb\u8a33\u7d50\u679c\u306b\u306a\u308b\u53ef\u80fd\u6027\u304c\u3042\u308a\u307e\u3059\u3002\u3059\u306a\u308f\u3061\u3001\u72ec\u7279\u306a\u8a00\u3044\u56de\u3057\u306a\u3069\u306e\u9806\u5e8f\u60c5\u5831\u3092\u7ffb\u8a33\u7d50\u679c\u306b\u53cd\u6620\u3055\u305b\u308b\u306b\u306f\u3069\u3046\u3057\u305f\u3089\u3044\u3044\u304b\u3092\u8003\u3048\u3066\u3044\u304d\u305f\u3044\u308f\u3051\u3067\u3059\u3002\u305d\u3053\u3067\u3001Attention\u6a5f\u69cb\u3092\u5c0e\u5165\u3057\u307e\u3059\u3002<\/p>\n\n\n\n<p>\u203b\u3053\u306e\u7ae0\u3067\u7d39\u4ecb\u3059\u308bAttention\u6a5f\u69cb\u306f\u3001\u8ad6\u6587\u306e\u65b9\u3067\u306fAttention\u3068\u3044\u3046\u540d\u524d\u304c\u4f7f\u7528\u3055\u308c\u3066\u3044\u306a\u3044\u3053\u3068\u3068\u3001\u5f8c\u3067\u8a71\u3059Attention\u306e\u4e00\u822c\u7cfb\u306b\u306f\u5f53\u3066\u306f\u307e\u3089\u306a\u3044\u306e\u3067\u3001\u53b3\u5bc6\u306b\u8a00\u3046\u3068Attention\u6a5f\u69cb\u3067\u306f\u3042\u308a\u307e\u305b\u3093\u304c\u3001\u521d\u4ee3Attention\u6a5f\u69cb\u3068\u3044\u3046\u3053\u3068\u3067\u6349\u3048\u3066\u3082\u3089\u3048\u308c\u3070\u3068\u601d\u3044\u307e\u3059\u3002<\/p>\n\n\n\n<p>\u307e\u305a\u3001Seq2Seq\u306bAttention\u3092\u5c0e\u5165\u3059\u308b\u306b\u3042\u305f\u308a\u3001\u7ffb\u8a33\u524d\u306e\u8a9e\u53e5\u3068\u7ffb\u8a33\u5f8c\u306e\u8a9e\u53e5\u306e\u4e26\u3073\u306b\u306f\u4f55\u3089\u304b\u306e\u95a2\u4fc2\u304c\u3042\u308b\u3068\u4eee\u5b9a\u3057\u307e\u3059\u3002\u4f8b\u3048\u3070\u3001\u82f1\u8a9e\u3092\u65e5\u672c\u8a9e\u306b\u8a33\u3059\u3068\u304d\u306f\u5f8c\u308d\u304b\u3089\u8a33\u3059\u3068\u826f\u3044\u3067\u3059\u3088\u306d\u3002\u3053\u308c\u3092\u56f3\u793a\u3059\u308b\u3068\u4ee5\u4e0b\u306e\u3088\u3046\u306b\u306a\u308a\u307e\u3059\u3002\u4e0b\u56f3\u306f\u5148\u307b\u3069\u793a\u3057\u305fSeq2Seq\u306e\u56f3\u306b\u304a\u3044\u3066\u3001Encoder\u3068Decoder\u3092\u7e26\u306b\u914d\u7f6e\u3057\u3001\u5165\u529b\u8a9e\u53e5\u3068\u51fa\u529b\u8a9e\u53e5\u9593\u306e\u95a2\u9023\u6027\u3092\u8d64\u77e2\u5370\u3067\u7d50\u3093\u3060\u3082\u306e\u3067\u3059\u3002\u8d64\u77e2\u5370\u306e\u59cb\u70b9\u306f\u30a8\u30f3\u30b3\u30fc\u30c0\u306e\u96a0\u308c\u5c64\u3067\u3001\u51fa\u529b\u3067\u7d42\u70b9\u306f\u30c7\u30b3\u30fc\u30c0\u306e\u96a0\u308c\u5c64\u306e\u5165\u529b\u3067\u3059\u3002<\/p>\n\n\n\n<p>\u300cthis\u300d\u306f\u300c\u3053\u308c\u300d\u3068\u300c\u306f\u300d\u306b\u3001\u300ca\u300d\u3068\u300cpen\u300d\u306f\u300c\u30da\u30f3\u300d\u3068\u95a2\u9023\u304c\u3042\u308b\u3053\u3068\u3092\u8868\u3057\u3066\u3044\u307e\u3059\u3002<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" width=\"856\" height=\"639\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-26.png\" alt=\"\" class=\"wp-image-4160\" srcset=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-26.png 856w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-26-300x224.png 300w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-26-768x573.png 768w\" sizes=\"(max-width: 856px) 100vw, 856px\" \/><\/figure><\/div>\n\n\n<p>\u4ee5\u964d\u3067\u306f\u3053\u306e\u8003\u3048\u65b9\u3092\u6570\u5f0f\u3067\u4e00\u822c\u5316\u3057\u305f\u3044\u3068\u601d\u3044\u307e\u3059\u3002\u4e0a\u56f3\u306b\u304a\u3044\u3066\u77e2\u5370\u306e\u63a5\u7d9a\u95a2\u4fc2\u306f\u6709\u304b\u7121\u304b\u306e2\u30d1\u30bf\u30fc\u30f3\u306b\u306a\u3063\u3066\u3044\u308b\u306e\u3067\u3001\u3053\u308c\u3092\u5b66\u7fd2\u30d1\u30e9\u30e1\u30fc\u30bf\u3092\u6301\u3064\u7dda\u5f62\u30e2\u30c7\u30eb\u3067\u8868\u3057\u305f\u3044\u3068\u601d\u3044\u307e\u3059\u3002\u4ee5\u4e0b\u3067\u306f\u300c\u30da\u30f3\u300d\u3092\u51fa\u529b\u3059\u308b\u6642\u523b\u306e\u51fa\u529b\u5c64\\(\\boldsymbol{h}^{(3)}_{dec}\\)\u306eAttention\u3092\u56f3\u793a\u3057\u305f\u3082\u306e\u3067\u3059\u3002\u30a8\u30f3\u30b3\u30fc\u30c0\u306e\u5404\u30b9\u30c6\u30c3\u30d7\u306b\u304a\u3051\u308b\u4e2d\u9593\u5c64\u306e\u51fa\u529b\\(\\boldsymbol{h}^{(1)}_{enc}~\\boldsymbol{h}^{(5)}_{enc}\\)\u3068\u305d\u308c\u306b\u5bfe\u5fdc\u3059\u308b\u5b66\u7fd2\u91cd\u307f\u30d1\u30e9\u30e1\u30fc\u30bf\u3092\\(\\alpha^{(3)}_1~\\alpha^{(3)}_5\\)\u3001\u6700\u7d42\u7684\u306aAttention\u306e\u5024\u3092\\(\\boldsymbol{c}^{(3)}\\)\u3068\u3057\u3066\u3044\u307e\u3059\u3002<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" width=\"862\" height=\"641\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-28.png\" alt=\"\" class=\"wp-image-4162\" srcset=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-28.png 862w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-28-300x223.png 300w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-28-768x571.png 768w\" sizes=\"(max-width: 862px) 100vw, 862px\" \/><\/figure><\/div>\n\n\n<p>\u3053\u306e\u3068\u304d\u3001\u8d64\u7dda\u90e8\u5206\u3092\u6570\u5f0f\u3067\u8868\u3059\u3068\u4ee5\u4e0b\u306e\u3088\u3046\u306b\u306a\u308a\u307e\u3059\u3002\u5f0f\u3092\u898b\u308b\u3068\u308f\u304b\u308b\u3068\u601d\u3044\u307e\u3059\u304c\u3001\u30a8\u30f3\u30b3\u30fc\u30c0\u306e\u5404\u30bf\u30a4\u30e0\u30b9\u30c6\u30c3\u30d7\u306b\u304a\u3051\u308b\u4e2d\u9593\u5c64\u51fa\u529b\u306e\u7dda\u5f62\u7d50\u5408\u3067\u8868\u3057\u3066\u3044\u307e\u3059\u3002<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" width=\"610\" height=\"399\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-18.png\" alt=\"\" class=\"wp-image-4150\" srcset=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-18.png 610w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-18-300x196.png 300w\" sizes=\"(max-width: 610px) 100vw, 610px\" \/><\/figure><\/div>\n\n\n<!--\n$$\\begin{eqnarray}\n\\boldsymbol{c}^{(3)}&=&\\left[\\boldsymbol{h}_{enc}^{(1)} \\boldsymbol{h}_{enc}^{(2)} \\boldsymbol{h}_{enc}^{(3)} \\boldsymbol{h}_{enc}^{(4)} \\boldsymbol{h}_{enc}^{(5)}\\right]\n\\left[\\begin{matrix}\n\\alpha_1^{(3)}\\\\\n\\alpha_2^{(3)}\\\\\n\\alpha_3^{(3)}\\\\\n\\alpha_4^{(3)}\\\\\n\\alpha_5^{(3)}\n\\end{matrix}\\right]\\\\\n&=& \\alpha_1^{(3)}\\boldsymbol{h}_{enc}^{(1)}+\\alpha_2^{(3)}\\boldsymbol{h}_{enc}^{(2)}+\\alpha_3^{(3)}\\boldsymbol{h}_{enc}^{(3)}+\\alpha_4^{(3)} \\boldsymbol{h}_{enc}^{(4)}+\\alpha_5^{(3)}\\boldsymbol{h}_{enc}^{(5)}\n\\end{eqnarray}$$\n-->\n\n\n\n<p>\u3053\u308c\u3092\u3001\u30c7\u30b3\u30fc\u30c0\u306e\u5168\u3066\u306e\u4e2d\u9593\u30ce\u30fc\u30c9\u306b\u5bfe\u3057\u3066\u793a\u3059\u3068\u4e0b\u56f3\u306e\u3088\u3046\u306aAttention\u5c64\u3068\u3057\u3066\u8868\u305b\u308b\u3053\u3068\u304c\u5206\u304b\u308a\u307e\u3059\u3002<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" width=\"868\" height=\"640\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-30.png\" alt=\"\" class=\"wp-image-4164\" srcset=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-30.png 868w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-30-300x221.png 300w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-30-768x566.png 768w\" sizes=\"(max-width: 868px) 100vw, 868px\" \/><\/figure><\/div>\n\n\n<p>\u3053\u3053\u307e\u3067\u306e\u8a71\u304c\u3001RNN\u306b\u5c0e\u5165\u3055\u308c\u305f\u521d\u4ee3Attention\u6a5f\u69cb\u306b\u306a\u308a\u307e\u3059\u3002<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Memory Network\u3068Attention<\/h4>\n\n\n\n<p>\u3053\u3053\u3067\u306f\u3001Seq2Seq\u306eAttention\u3092\u66f4\u306b\u62e1\u5f35\u3057\u305f\u3082\u306e\u3068\u3057\u3066\u89e3\u91c8\u53ef\u80fd\u306aMemory Network\u306b\u3064\u3044\u3066\u89e3\u8aac\u3057\u305f\u3044\u3068\u601d\u3044\u307e\u3059\u3002<br>\uff08\u4eca\u5f8c\u3001\u8ffd\u8a18\u4e88\u5b9a\uff09<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">CNN\u306e\u8d77\u6e90\u3068Attention\u306e\u8d77\u6e90<\/h3>\n\n\n\n<p>\u3053\u3053\u304b\u3089\u306f\u3001CNN\u306b\u5c0e\u5165\u3055\u308c\u305f\u521d\u4ee3Attention\u6a5f\u69cb\u306b\u3064\u3044\u3066\u89e3\u8aac\u3057\u3066\u3044\u304d\u305f\u3044\u3068\u601d\u3044\u307e\u3059\u3002<\/p>\n\n\n\n<p>\u3053\u306e\u89e3\u8aac\u306b\u3042\u305f\u308aCNN\u306e\u57fa\u790e\u77e5\u8b58\u306b\u3064\u3044\u3066\u6700\u521d\u306b\u89e3\u8aac\u3057\u3066\u3044\u304d\u307e\u3059\u3002CNN\u306b\u3064\u3044\u3066\u3042\u308b\u7a0b\u5ea6\u306e\u77e5\u8b58\u304c\u3042\u308b\u65b9\u306f\u3001CNN\u306e\u57fa\u790e\u306f\u98db\u3070\u3057\u3066SENet\u3068Attention\u6a5f\u69cb\u304b\u3089\u8aad\u307f\u9032\u3081\u3066\u3082\u3089\u3063\u3066\u5927\u4e08\u592b\u3067\u3059\u3002<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">CNN\u306e\u57fa\u790e<\/h4>\n\n\n\n<p>CNN\u306e\u57fa\u790e\u3067\u306f\u3001CNN\u306e\u5143\u3068\u306a\u3063\u305f\u30cd\u30aa\u30b3\u30b0\u30cb\u30c8\u30ed\u30f3\u304b\u3089\u89e3\u8aac\u3057\u3001\u7573\u307f\u8fbc\u307f\u5c64\u3001\u30d7\u30fc\u30ea\u30f3\u30b0\u5c64\u306e\u4ed5\u7d44\u307f\u3068\u8a08\u7b97\u65b9\u6cd5\u3001\u753b\u50cf\u8a8d\u8b58\u30b3\u30f3\u30da\u30c6\u30a3\u30b7\u30e7\u30f3\u306eILSVRC\u3067\u767b\u5834\u3057\u305fCNN\u30e2\u30c7\u30eb\u306b\u3064\u3044\u3066\u7d39\u4ecb\u3057\u305f\u3044\u3068\u601d\u3044\u307e\u3059\u3002<\/p>\n\n\n\n<h5 class=\"wp-block-heading\">\u30cd\u30aa\u30b3\u30b0\u30cb\u30c8\u30ed\u30f3\u3068CNN<\/h5>\n\n\n\n<p>\u30cd\u30aa\u30b3\u30b0\u30cb\u30c8\u30ed\u30f3\uff081980\u5e74\u4ee3\uff09\u306fCNN\u306e\u539f\u578b\u3067\u3001\u65e5\u672c\u4eba\u306e\u798f\u5cf6\u6c0f\u306b\u3088\u308a\u63d0\u6848\u3055\u308c\u307e\u3057\u305f\u3002\u91cd\u8981\u306a\u30dd\u30a4\u30f3\u30c8\u306f\u3001\u751f\u4f53\u306e\u8996\u899a\u91ce\u3092\u3082\u3068\u306b\u30e2\u30c7\u30eb\u5316\u3055\u308c\u3066\u3044\u3066\u3001CNN\u3067\u3044\u3046\u7573\u307f\u8fbc\u307f\u5c64\u3084\u30d7\u30fc\u30ea\u30f3\u30b0\u5c64\u306b\u5bfe\u5fdc\u3059\u308b\u3001\u5358\u7d14\u578b\u7d30\u80de\u5c64\uff08S\u7d30\u80de\u5c64\uff09\u3068\u8907\u96d1\u578b\u7d30\u80de\u5c64\uff08C\u7d30\u80de\u5c64\uff09\u3092\u7e70\u308a\u8fd4\u3057\u9069\u7528\u3057\u3001\u6700\u5f8c\u306b\u5168\u7d50\u5408\u5c64\u306b\u6e21\u3057\u3066\u3044\u308b\u70b9\u3067\u3059\u3002\u3053\u308c\u306f\u3001\u73fe\u5728\u306eCNN\u306e\u5f62\u306b\u306a\u3063\u305f\u6700\u521d\u306e\u30e2\u30c7\u30eb\u3068\u8003\u3048\u308b\u3053\u3068\u304c\u3067\u304d\u307e\u3059\u3002\u4e00\u65b9\u3067\u3001\u5b66\u7fd2\u306b\u306fAdd-if-silent\u3068\u547c\u3070\u308c\u308b\u81ea\u5df1\u7d44\u7e54\u5316\u624b\u6cd5\u304c\u4f7f\u308f\u308c\u3066\u304a\u308a\u3001\u8aa4\u5dee\u9006\u4f1d\u64ad\u6cd5\u3092\u4f7f\u7528\u3059\u308b\u73fe\u5728\u306eCNN\u3068\u306f\u7570\u306a\u308a\u307e\u3059\u3002<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" width=\"1024\" height=\"630\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2019\/11\/\u30cd\u30aa\u30b3\u30b0\u30cb\u30c8\u30ed\u30f3-1024x630.png\" alt=\"\" class=\"wp-image-571\" srcset=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2019\/11\/\u30cd\u30aa\u30b3\u30b0\u30cb\u30c8\u30ed\u30f3-1024x630.png 1024w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2019\/11\/\u30cd\u30aa\u30b3\u30b0\u30cb\u30c8\u30ed\u30f3-300x185.png 300w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2019\/11\/\u30cd\u30aa\u30b3\u30b0\u30cb\u30c8\u30ed\u30f3-768x472.png 768w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2019\/11\/\u30cd\u30aa\u30b3\u30b0\u30cb\u30c8\u30ed\u30f3.png 1247w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure><\/div>\n\n\n<p>\u6b21\u306b\u3001\u8aa4\u5dee\u9006\u4f1d\u64ad\u6cd5\u3092\u4f7f\u7528\u3057\u3066\u5b66\u7fd2\u3059\u308b\u521d\u3081\u3066\u306eCNN\u30e2\u30c7\u30eb\u3067\u3042\u308bLeNet\uff081989\u5e74\uff09\u306b\u3064\u3044\u3066\u7d39\u4ecb\u3057\u307e\u3059\u3002LeNet\u3068\u306f\u3001\u73fe\u5728\u306e\u6df1\u5c64\u5b66\u7fd2\u6280\u8853\u306e\u767a\u5c55\u306b\u5927\u304d\u306a\u8ca2\u732e\u3092\u3057\u3066\u3044\u308b\u30e4\u30f3\u30fb\u30eb\u30ab\u30f3\u6c0f\u306b\u3088\u308a\u63d0\u6848\u3055\u308c\u307e\u3057\u305f\u3002LeNet\u306f\u7573\u307f\u8fbc\u307f\u5c64\u3068\u30b5\u30d6\u30b5\u30f3\u30d7\u30ea\u30f3\u30b0\u5c64\uff08\uff1d\u30d7\u30fc\u30ea\u30f3\u30b0\u5c64\uff09\u304c\u4ea4\u4e92\u306b\u9069\u7528\u3055\u308c\u3001\u6700\u5f8c\u306b\u5168\u7d50\u5408\u5c64\u3078\u7e4b\u304c\u308c\u3066\u304a\u308a\u3001\u30c7\u30a3\u30fc\u30d7\u30e9\u30fc\u30cb\u30f3\u30b0\u30d6\u30fc\u30e0\u3092\u5dfb\u304d\u8d77\u3053\u3057\u305fILSVRC2012\u512a\u52dd\u30e2\u30c7\u30eb\u306eAlexNet\u3068\u6bd4\u3079\u3066\u3082\u304b\u306a\u308a\u8fd1\u3044\u69cb\u9020\u3092\u3057\u3066\u3044\u308b\u3053\u3068\u304c\u5206\u304b\u308a\u307e\u3059\u3002<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" width=\"1024\" height=\"290\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2020\/07\/image-7-1024x290.png\" alt=\"\" class=\"wp-image-1749\" srcset=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2020\/07\/image-7-1024x290.png 1024w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2020\/07\/image-7-300x85.png 300w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2020\/07\/image-7-768x218.png 768w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2020\/07\/image-7.png 1280w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><figcaption class=\"wp-element-caption\">LeNet-5<\/figcaption><\/figure><\/div>\n\n\n<p>\u4ee5\u4e0b\u306bAlexNet\u3092\u793a\u3057\u307e\u3059\u3002AlexNet\u306f\u30d1\u30e9\u30e1\u30fc\u30bf\u6570\u304c1\u3064\u306eGPU\u4e0a\u306b\u4e57\u3089\u306a\u304b\u3063\u305f\u3053\u3068\u306b\u8d77\u56e0\u3057\u30662\u30d1\u30b9\u306b\u5206\u304b\u308c\u3066\u3044\u307e\u3059\u304c\u3001\u69cb\u9020\u306f\u5927\u5909\u4f3c\u3066\u3044\u308b\u3053\u3068\u3092\u63b4\u3093\u3067\u3044\u305f\u3060\u304f\u3053\u3068\u304c\u3067\u304d\u308b\u3068\u601d\u3044\u307e\u3059\u3002<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" width=\"1024\" height=\"577\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/02\/image-15-1024x577.jpg\" alt=\"alexnet\" class=\"wp-image-3625\" srcset=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/02\/image-15-1024x577.jpg 1024w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/02\/image-15-300x169.jpg 300w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/02\/image-15-768x433.jpg 768w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/02\/image-15-1536x866.jpg 1536w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/02\/image-15-2048x1155.jpg 2048w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><figcaption class=\"wp-element-caption\">AlexNet<\/figcaption><\/figure>\n\n\n\n<p>\u7573\u307f\u8fbc\u307f\u30cb\u30e5\u30fc\u30e9\u30eb\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u306b\u3064\u3044\u3066\u3088\u308a\u8a73\u3057\u304f\u77e5\u308a\u305f\u3044\u65b9\u3084\u3001LeNet-5\u306e\u5b9f\u76f8\u3092\u3057\u3066\u307f\u305f\u3044\u65b9\u306f\u4ee5\u4e0b\u306e\u8a18\u4e8b\u3092\u53c2\u7167\u304f\u3060\u3055\u3044\u3002<\/p>\n\n\n\n<figure class=\"wp-block-embed is-type-wp-embed is-provider-agirobots-blog wp-block-embed-agirobots-blog\"><div class=\"wp-block-embed__wrapper\">\n<div><a href=\"https:\/\/developers.agirobots.com\/jp\/cnn-theory-pytorch-programming\/\" class=\"st-cardlink st-embed-cardlink\"><div class=\"kanren st-cardbox\"><dl class=\"clearfix\"><dt class=\"st-card-img\"><img decoding=\"async\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2020\/07\/image-7-150x150.png\" alt=\"\" width=\"100\" height=\"100\" \/><\/dt><dd><p class=\"st-cardbox-t\">\u7573\u307f\u8fbc\u307f\u30cb\u30e5\u30fc\u30e9\u30eb\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u306e\u7406\u8ad6\u3068PyTorch\u306b\u3088\u308bLeNet\u306e\u5b9f\u88c5<\/p><div class=\"st-card-excerpt smanone\"><p>\u5f53\u8a18\u4e8b\u3067\u306f\u7573\u307f\u8fbc\u307f\u30cb\u30e5\u30fc\u30e9\u30eb\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\uff08Convolutional Neural Network;CNN\uff09\u306e\u57fa\u672c\u7684\u306a\u7406\u8ad6\u3068PyToch\u3092\u4f7f\u3063\u305f\u5b9f\u88c5\u306b\u3064\u3044\u3066\u89e3\u8aac\u3057\u3066\u3044\u304d\u307e\u3059\u3002 \u304a\u304a\u307e\u304b\u306a\u69cb\u6210\u306f\u3001\u7573\u307f\u8fbc ... <\/p><\/div><\/dd><\/dl><\/div><\/a><\/div>\n<\/div><\/figure>\n\n\n\n<p>\u307e\u305f\u3001AlexNet\u306b\u3064\u3044\u3066\u8a73\u3057\u304f\u77e5\u308a\u305f\u3044\u65b9\u306f\u4ee5\u4e0b\u306e\u8a18\u4e8b\u3092\u53c2\u7167\u304f\u3060\u3055\u3044\u3002<\/p>\n\n\n\n<h5 class=\"wp-block-heading\">\u7573\u307f\u8fbc\u307f\u5c64<\/h5>\n\n\n\n<p>\u7573\u307f\u8fbc\u307f\u5c64\u306f\u30cd\u30aa\u30b3\u30b0\u30cb\u30c8\u30ed\u30f3\u306b\u304a\u3051\u308bS\u7d30\u80de\u5c64\u306b\u5bfe\u5fdc\u3057\u3001\u30e1\u30a4\u30f3\u3068\u306a\u308b\u50cd\u304d\u306f\u30a8\u30c3\u30b8\u62bd\u51fa\u3092\u884c\u3046\u3053\u3068\u3067\u3059\u3002\u3053\u306e\u6a5f\u80fd\u306f\u30d5\u30a3\u30eb\u30bf\u3092\u4f7f\u7528\u3057\u3066\u5b9f\u73fe\u3055\u305b\u308b\u3053\u3068\u304c\u3067\u304d\u307e\u3059\u3002\u4e0b\u56f3(a)\u306f3x3\u306e\u30d5\u30a3\u30eb\u30bf\u3092\u5199\u771f\u306b\u9069\u7528\u3057\u305f\u5834\u5408\u306e\u56f3\u3067\u3001(b)\u306f5x5\u306e\u30d5\u30a3\u30eb\u30bf\u3092\u5199\u771f\u306b\u9069\u7528\u3057\u305f\u5834\u5408\u306e\u56f3\u306b\u306a\u308a\u307e\u3059\u3002<\/p>\n\n\n\n<p>\u305d\u308c\u305e\u308c\u306e\u5834\u5408\u306b\u304a\u3044\u3066\u3001\u30a8\u30c3\u30b8\u304c\u62bd\u51fa\u3067\u304d\u3066\u3044\u308b\u3053\u3068\u304c\u5206\u304b\u308a\u307e\u3059\u3002<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" width=\"1024\" height=\"372\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2020\/07\/image-13-1024x372.jpg\" alt=\"\" class=\"wp-image-1761\" srcset=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2020\/07\/image-13-1024x372.jpg 1024w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2020\/07\/image-13-300x109.jpg 300w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2020\/07\/image-13-768x279.jpg 768w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2020\/07\/image-13-1536x558.jpg 1536w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2020\/07\/image-13.jpg 1824w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>\u7573\u307f\u8fbc\u307f\u5c64\u306e\u8a08\u7b97\u65b9\u6cd5\u306f\u3001\u4ee5\u4e0b\u306e\u56f3\u306e\u3088\u3046\u306b\u306a\u308a\u307e\u3059\u3002\u307e\u305a\u3001\u5165\u529b\u7279\u5fb4\u30de\u30c3\u30d7\u306b\u304a\u3044\u3066\u3001\u7573\u307f\u8fbc\u307f\u30d5\u30a3\u30eb\u30bf\u306e\u30ab\u30fc\u30cd\u30eb\u30b5\u30a4\u30ba\u3068\u7b49\u3057\u3044\u30d1\u30c3\u30c1\u306b\u5206\u3051\u3066\u3001\u305d\u308c\u305e\u308c\u306e\u30d4\u30af\u30bb\u30eb\u3054\u3068\u306b\u5185\u7a4d\u3092\u3068\u308a\u307e\u3059\u3002<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" width=\"882\" height=\"498\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2020\/07\/image-30.png\" alt=\"\" class=\"wp-image-1789\" srcset=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2020\/07\/image-30.png 882w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2020\/07\/image-30-300x169.png 300w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2020\/07\/image-30-768x434.png 768w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2020\/07\/image-30-320x180.png 320w\" sizes=\"(max-width: 882px) 100vw, 882px\" \/><\/figure><\/div>\n\n\n<p>\u203b\u4e0b\u56f3\u306e\u8a08\u7b97\u65b9\u6cd5\u306b\u304a\u3044\u3066\u6d3b\u6027\u5316\u95a2\u6570\u306f\u7701\u7565\u3057\u3066\u3044\u307e\u3059\u3002<br>\u4f59\u8ac7\u3067\u3059\u304c\u3001\u7573\u307f\u8fbc\u307f\u30d5\u30a3\u30eb\u30bf\u306f\u7dda\u5f62\u30e2\u30c7\u30eb\u3067\u3042\u308b\u3053\u3068\u304b\u3089\u3001\u7279\u5fb4\u304c\u7dda\u5f62\u30e2\u30c7\u30eb\u3067\u62bd\u51fa\u53ef\u80fd\u3067\u3042\u308b\u3053\u3068\u3092\u4eee\u5b9a\u3057\u3066\u3044\u307e\u3059\u3002\u305d\u306e\u305f\u3081\u3001\u30de\u30a4\u30af\u30ed\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u3092\u4f7f\u7528\u3057\u3066\u975e\u7dda\u5f62\u306e\u7573\u307f\u8fbc\u307f\u30d5\u30a3\u30eb\u30bf\u3092\u69cb\u7bc9\u3057\u305fNetwork in Network\u3068\u3044\u3046\u3082\u306e\u3082\u3042\u308a\u307e\u3059\u3002<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" width=\"1024\" height=\"710\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-6-1024x710.jpg\" alt=\"\" class=\"wp-image-4191\" srcset=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-6-1024x710.jpg 1024w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-6-300x208.jpg 300w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-6-768x532.jpg 768w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-6-1536x1065.jpg 1536w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-6.jpg 1538w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure><\/div>\n\n\n<h5 class=\"wp-block-heading\">\u30d7\u30fc\u30ea\u30f3\u30b0\u5c64<\/h5>\n\n\n\n<p>\u30d7\u30fc\u30ea\u30f3\u30b0\u5c64\u306f\u30cd\u30aa\u30b3\u30b0\u30cb\u30c8\u30ed\u30f3\u306b\u304a\u3044\u3066C\u7d30\u80de\u306b\u5bfe\u5fdc\u3057\u3001\u5165\u529b\u7279\u5fb4\u91cf\u5185\u90e8\u306e\u7279\u5fb4\u306e\u4f4d\u7f6e\u9023\u308c\u3092\u8a31\u5bb9\u3059\u308b\u50cd\u304d\u304c\u3042\u308a\u307e\u3059\u3002\u4f8b\u3048\u3070\u3001\u4e0b\u56f3\u306e\u3088\u3046\u306b\u30ab\u30fc\u30cd\u30eb\u30b5\u30a4\u30ba\u304c2\u306b\u3088\u308b\u30de\u30c3\u30af\u30b9\u30d7\u30fc\u30ea\u30f3\u30b0\u306e\u5834\u5408\u306e\u4f8b\u3092\u793a\u3057\u307e\u3059\u3002<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" width=\"704\" height=\"371\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2020\/07\/image-32.png\" alt=\"\" class=\"wp-image-1791\" srcset=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2020\/07\/image-32.png 704w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2020\/07\/image-32-300x158.png 300w\" sizes=\"(max-width: 704px) 100vw, 704px\" \/><\/figure><\/div>\n\n\n<p>\u4ee5\u4e0b\u306b\u6700\u5927\u5024\u30d7\u30fc\u30ea\u30f3\u30b0\u53ca\u3073\u5e73\u5747\u5024\u30d7\u30fc\u30ea\u30f3\u30b0\u3092\u3001\u30ab\u30fc\u30cd\u30eb\u30b5\u30a4\u30ba2\u53ca\u30734\u3067\u9069\u7528\u3059\u308b\u5834\u5408\u306e\u4f8b\u3092\u793a\u3057\u307e\u3059\u3002<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" width=\"1024\" height=\"686\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2020\/07\/image-23-1024x686.png\" alt=\"\" class=\"wp-image-1780\" srcset=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2020\/07\/image-23-1024x686.png 1024w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2020\/07\/image-23-300x201.png 300w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2020\/07\/image-23-768x515.png 768w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2020\/07\/image-23-272x182.png 272w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2020\/07\/image-23.png 1222w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure><\/div>\n\n\n<p>\u6700\u5927\u5024\u30d7\u30fc\u30ea\u30f3\u30b0\u306e\u30ab\u30fc\u30cd\u30eb\u30b5\u30a4\u30ba\u304c4\u306e\u3068\u304d\u3001\u30a8\u30c3\u30b8\u304c\u51dd\u7e2e\u3055\u308c\u3066\u6fc3\u304f\u306a\u3063\u3066\u3044\u308b\u3053\u3068\u304c\u5206\u304b\u308a\u307e\u3059\u3002<\/p>\n\n\n\n<h5 class=\"wp-block-heading\">ILSVRC\u3068CNN<\/h5>\n\n\n\n<p>CNN\u306e\u767a\u5c55\u3068ILSVRC\u306f\u5207\u3063\u3066\u3082\u5207\u308a\u96e2\u305b\u306a\u3044\u95a2\u4fc2\u3067\u3059\u3002ILSVRC\u306f\u5e7e\u3064\u304b\u306e\u30bf\u30b9\u30af\u304b\u3089\u306a\u308b\u753b\u50cf\u8a8d\u8b58\u30b3\u30f3\u30da\u3067\u3001\u753b\u50cf\u5206\u985e\u30bf\u30b9\u30af\u306b\u304a\u3051\u308b\u30a8\u30e9\u30fc\u7387\u306b\u3064\u3044\u30662010\u5e74\u304b\u30892017\u5e74\u307e\u3067\u793a\u3057\u307e\u3059\u3002ILSVRC\u306e2010\u5e74\u30682011\u5e74\u306e\u512a\u52dd\u30e2\u30c7\u30eb\u306f\u6df1\u5c64\u5b66\u7fd2\u3067\u306f\u306a\u3044\u306e\u3067\u3001\u91cd\u8981\u306a\u306e\u306f2012\u5e74\u306eCNN\u306b\u3088\u308b\u512a\u52dd\u30e2\u30c7\u30eb\u306eAlexNet\u4ee5\u964d\u306e\u30e2\u30c7\u30eb\u3067\u3059\u3002\u8a73\u3057\u3044\u5185\u5bb9\u306f\u7701\u7565\u3057\u307e\u3059\u304cCNN\u3068Attention\u3068\u3044\u3046\u89b3\u70b9\u3067\u91cd\u8981\u306a2017\u5e74\u306e\u512a\u52dd\u30e2\u30c7\u30eb\u3067\u3042\u308bSENet\u3064\u3044\u3066\u7d39\u4ecb\u3057\u305f\u3044\u3068\u601d\u3044\u307e\u3059\u3002<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" width=\"1024\" height=\"483\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-1-1024x483.jpg\" alt=\"\" class=\"wp-image-4177\" srcset=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-1-1024x483.jpg 1024w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-1-300x141.jpg 300w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-1-768x362.jpg 768w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-1-1536x724.jpg 1536w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-1.jpg 1866w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure><\/div>\n\n\n<h4 class=\"wp-block-heading\">SENet\u3068Attention\u6a5f\u69cb<\/h4>\n\n\n\n<p>SENet\u306b\u3064\u3044\u3066\u89e3\u8aac\u3057\u307e\u3059\u3002SENet\u306fSqueeze and Excitation Network\u306e\u7565\u3067\u3001Squeeze\u306b\u306f\u5727\u7e2e\u3001Excitation\u306b\u306f\u52b1\u8d77\u306e\u610f\u5473\u304c\u3042\u308a\u307e\u3059\u3002SENet\u3068\u3044\u3046\u3068\u3001\u305d\u3046\u3044\u3063\u305f\u540d\u524d\u306e\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u304c\u3042\u308b\u306e\u304b\u3068\u601d\u308f\u308c\u308b\u304b\u3082\u3057\u308c\u307e\u305b\u3093\u304c\u3001\u305d\u3046\u3067\u306f\u306a\u304f\u3066\u3001\u4ee5\u4e0b\u3067\u8aac\u660e\u3059\u308bSE Block\u3068\u3044\u3046\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u306e\u30d6\u30ed\u30c3\u30af\u3092\u8ffd\u52a0\u3057\u305f\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u306e\u7dcf\u79f0\u3067\u3059\u3002\u4f8b\u3048\u3070\u3001ResNet\u306bSE Block\u3092\u8ffd\u52a0\u3057\u305fSE-ResNet\u3068\u3044\u3046\u3082\u306e\u3084\u3001Inception\u306bSE Block\u3092\u8ffd\u52a0\u3057\u305fSE-Inception\u3068\u3044\u3046\u3082\u306e\u304c\u3042\u308a\u307e\u3059\u3002\u3067\u306f\u3001SE Block\u3068\u306f\u4f55\u306a\u306e\u304b\u5177\u4f53\u7684\u306b\u89e3\u8aac\u3057\u3066\u3044\u304d\u307e\u3059\u3002<\/p>\n\n\n\n<p>SE Block\u306fGAP\u2192FC+ReLU\u2192FC+SIgmoid\u306e3\u5c64\u304b\u3089\u69cb\u6210\u3055\u308c\u3001\u7279\u5fb4\u30de\u30c3\u30d7\u306e\u30c1\u30e3\u30f3\u30cd\u30eb\u306b\u95a2\u3057\u3066\u3001\u6ce8\u76ee\u3059\u3079\u304d\u3068\u8003\u3048\u3089\u308c\u308b\u30de\u30c3\u30d7\u306e\u60c5\u5831\u306f\u591a\u304f\u901a\u904e\u3057\u3001\u6ce8\u76ee\u3057\u3079\u304d\u3067\u306a\u3044\u3068\u8003\u3048\u3089\u308c\u308b\u30de\u30c3\u30d7\u306e\u60c5\u5831\u306f\u3042\u307e\u308a\u901a\u904e\u3057\u306a\u3044\u3088\u3046\u306b\u30b2\u30fc\u30c6\u30a3\u30f3\u30b0\u3092\u884c\u3044\u307e\u3059\u3002\u3053\u308c\u306f\u3001\u7279\u5fb4\u30de\u30c3\u30d7\u306e\u30c1\u30e3\u30f3\u30cd\u30eb\u306b\u3064\u3044\u3066Attention\u3092\u9069\u7528\u3057\u3066\u3044\u308b\u3068\u89e3\u91c8\u3059\u308b\u3053\u3068\u304c\u3067\u304d\u307e\u3059\u3002<\/p>\n\n\n\n<p>GAP\u306b\u3088\u308a\u3001\u30de\u30c3\u30d7\u65b9\u5411\u306e\u5e73\u5747\u7279\u5fb4\u91cf\u3092\u6c42\u3081\u30012\u56de\u306e\u5909\u63db\u3092\u65bd\u3057\u3066\u3001\u30b7\u30b0\u30e2\u30a4\u30c9\u95a2\u6570\u306b\u3088\u308b0~1\u306e\u5024\u306b\u5909\u63db\u3057\u3001\u30de\u30c3\u30d7\u306b\u639b\u3051\u5408\u308f\u305b\u308b\u3053\u3068\u3067\u30b2\u30fc\u30c8\u6a5f\u69cb\u3092\u5b9f\u73fe\u3057\u3066\u3044\u307e\u3059\u3002<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" width=\"1010\" height=\"999\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-5.jpg\" alt=\"\" class=\"wp-image-4183\" srcset=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-5.jpg 1010w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-5-300x297.jpg 300w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-5-768x760.jpg 768w\" sizes=\"(max-width: 1010px) 100vw, 1010px\" \/><\/figure><\/div>\n\n\n<h2 class=\"wp-block-heading\">Attention\u6a5f\u69cb\u306e\u7a2e\u985e<\/h2>\n\n\n\n<p>\u3053\u3053\u307e\u3067\u3001\u5927\u5909\u9577\u304b\u3063\u305f\u3067\u3059\u304cAttention\u306e\u8d77\u6e90\u3068\u3082\u3044\u3048\u308b\u30e2\u30c7\u30eb\u306b\u3064\u3044\u3066\u7d39\u4ecb\u3057\u3066\u304d\u307e\u3057\u305f\u3002Attention\u6a5f\u69cb\u306f\u30b5\u30d6\u7684\u306a\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u3092\u7528\u610f\u3057\u3066\u6ce8\u610f\u3092\u5b9f\u73fe\u3057\u305f\u3082\u306e\u3067\u3042\u308b\u3053\u3068\u3092\u3054\u7406\u89e3\u3044\u305f\u3060\u3051\u305f\u3068\u601d\u3044\u307e\u3059\u3002\u3053\u3053\u307e\u3067\u306e\u8a71\u3057\u3067\u306fAttention\u6a5f\u69cb\u306e\u69cb\u9020\u304c\u4e00\u822c\u5316\u3055\u308c\u3066\u304a\u3089\u305a\u3001\u6df7\u4e71\u3057\u3066\u3044\u308b\u65b9\u3082\u591a\u3044\u3068\u601d\u3046\u306e\u3067\u3001\u73fe\u5728\u3001\u4e00\u822c\u5316\u3055\u308c\u3066\u3044\u308bAttention\u6a5f\u69cb\u306e\u57fa\u672c\u5f62\u72b6\u3092\u8aac\u660e\u3057\u3066\u3044\u304d\u305f\u3044\u3068\u601d\u3044\u307e\u3059\u3002<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Attention\u6a5f\u69cb\u306e\u57fa\u672c\u5f62<\/h3>\n\n\n\n<p>\u307e\u305a\u6700\u521d\u306b\u3001Attention\u6a5f\u69cb\u306e\u5165\u529b\u3092Query\u3001Key\u3001Value\u3092\u3082\u3064block\u3068\u3057\u3066\u4e00\u822c\u5316\u3057\u307e\u3059\u3002\u4ee5\u964d\u3067\u306f\u3001\u3053\u306e\u5f62\u3067\u4e00\u822c\u5316\u3057\u305f\u3082\u306e\u3092Attention\u6a5f\u69cb\u3068\u547c\u3076\u3053\u3068\u306b\u3057\u307e\u3059\u3002<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" width=\"355\" height=\"293\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/09\/image-2.png\" alt=\"\" class=\"wp-image-4226\" srcset=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/09\/image-2.png 355w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/09\/image-2-300x248.png 300w\" sizes=\"(max-width: 355px) 100vw, 355px\" \/><\/figure><\/div>\n\n\n<p>\u3053\u306e\u3068\u304d\u3001Attention\u6a5f\u69cb\u306f\u3001\u5165\u529b\u5f62\u5f0f\u3068Attention\u51fa\u529b\u306e\u9069\u7528\u65b9\u6cd5\u306e\u9055\u3044\u306b\u3088\u308a\u30014\u30d1\u30bf\u30fc\u30f3\u306b\u5206\u3051\u308b\u3053\u3068\u304c\u3067\u304d\u307e\u3059\u3002\u307e\u305a\u3001\u5165\u529b\u5f62\u5f0f\u306e\u9055\u3044\u306b\u3088\u308b\u5206\u985e\u304b\u3089\u8aac\u660e\u3057\u307e\u3059\u3002\u5165\u529b\u5f62\u5f0f\u306e\u9055\u3044\u306b\u3088\u308b\u5206\u985e\u306f\u3001Q\u3001K\u3001V\u306e\u5168\u3066\u306b\u540c\u3058\u30c7\u30fc\u30bf\u3092\u5165\u529b\u3059\u308bSelf-Attention\u3068\u3001Q\u306b\u5165\u529b\u3055\u308c\u308b\u30c7\u30fc\u30bf\u3068K\u3068V\u306b\u5165\u529b\u3055\u308c\u308b\u30c7\u30fc\u30bf\u304c\u7570\u306a\u308bSourceTarget-Attention\u306e2\u7a2e\u985e\u3067\u3059\u3002<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" width=\"730\" height=\"261\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/09\/image-4.png\" alt=\"\" class=\"wp-image-4237\" srcset=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/09\/image-4.png 730w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/09\/image-4-300x107.png 300w\" sizes=\"(max-width: 730px) 100vw, 730px\" \/><\/figure><\/div>\n\n\n<p>\u6b21\u306b\u3001Attention\u51fa\u529b\u306e\u9069\u7528\u65b9\u6cd5\u306e\u9055\u3044\u306b\u3088\u308b\u5206\u985e\u306b\u3064\u3044\u3066\u89e3\u8aac\u3057\u307e\u3059\u3002Attention\u51fa\u529b\u306e\u9069\u7528\u65b9\u6cd5\u306f\u52a0\u7b97\u578b\u3068\u4e57\u7b97\u578b\u306e2\u7a2e\u985e\u306b\u5206\u3051\u3089\u308c\u3001\u305d\u308c\u305e\u308c\u3001Additive Attention\u3001Multiplicative Attention\u3068\u547c\u3070\u308c\u307e\u3059\u3002<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" width=\"794\" height=\"299\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/09\/image-5.png\" alt=\"\" class=\"wp-image-4238\" srcset=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/09\/image-5.png 794w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/09\/image-5-300x113.png 300w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/09\/image-5-768x289.png 768w\" sizes=\"(max-width: 794px) 100vw, 794px\" \/><\/figure><\/div>\n\n\n<h2 class=\"wp-block-heading\">Transformer<\/h2>\n\n\n\n<p>\u3053\u3053\u307e\u3067\u3001Attention\u6a5f\u69cb\u306b\u3064\u3044\u3066\u8aac\u660e\u3057\u3066\u304d\u307e\u3057\u305f\u3002\u3084\u3063\u3068\u3001Transformer\u306e\u8aac\u660e\u3092\u884c\u3046\u3053\u3068\u304c\u3067\u304d\u307e\u3059\u3002<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">\u6982\u8981<\/h3>\n\n\n\n<p>\u3053\u3053\u3067\u306f\u3001Transformer\u306e\u6982\u7565\u306b\u3064\u3044\u3066\u89e3\u8aac\u3057\u3066\u3044\u304d\u307e\u3059\u3002\u6982\u7565\u3092\u4ee5\u4e0b\u306b\u56f3\u793a\u3057\u305f\u306e\u3067\u5fc5\u8981\u306b\u5fdc\u3058\u3066\u53c2\u7167\u3057\u306a\u304c\u3089\u8aad\u307f\u9032\u3081\u3066\u3044\u305f\u3060\u3051\u308c\u3070\u3068\u601d\u3044\u307e\u3059\u3002<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img decoding=\"async\" width=\"1798\" height=\"912\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/06\/image.jpg\" alt=\"\" class=\"wp-image-3978\" srcset=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/06\/image.jpg 1798w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/06\/image-300x152.jpg 300w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/06\/image-1024x519.jpg 1024w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/06\/image-768x390.jpg 768w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/06\/image-1536x779.jpg 1536w\" sizes=\"(max-width: 1798px) 100vw, 1798px\" \/><figcaption class=\"wp-element-caption\">Transformer\u306e\u6982\u7565\u56f3<\/figcaption><\/figure><\/div>\n\n\n<p>Attention is all you need\u3067\u793a\u3055\u308c\u305f\u521d\u4ee3Transformer\u306fEncoder\u3068Decoder\u304b\u3089\u69cb\u6210\u3055\u308c\u308b\u3001Encoder-Decoder\u30e2\u30c7\u30eb\u3067\u3059\u3002\u4ee3\u8868\u7684\u306aEncoder-Decoder\u30e2\u30c7\u30eb\u306b\u306f\u3001\u4e00\u822c\u7684\u306b\u6559\u5e2b\u306a\u3057\u3067\u6b21\u5143\u5727\u7e2e\u3059\u308b\u969b\u306b\u4f7f\u308f\u308c\u308bAutoEncoder\u3084\u81ea\u52d5\u7ffb\u8a33\u306e\u969b\u306b\u4f7f\u308f\u308c\u308bSeq2Seq\u306a\u3069\u304c\u3042\u308a\u3001\u4e21\u30e2\u30c7\u30eb\u306eEncoder\u3084Decoder\u306e\u7d30\u90e8\u69cb\u9020\u306b\u6ce8\u76ee\u3059\u308b\u3068\u5168\u304f\u7570\u306a\u308a\u307e\u3059\u304c\u3001\u3069\u3061\u3089\u3082Encoder\u3068Decoder\u304b\u3089\u69cb\u6210\u3055\u308c\u308b\u3068\u3044\u3046\u62bd\u8c61\u7684\u306a\u9818\u57df\u306b\u304a\u3044\u3066\u5171\u901a\u70b9\u3092\u6301\u3061\u3001\u672c\u8a18\u4e8b\u306e\u30e1\u30a4\u30f3\u30c6\u30fc\u30de\u3067\u3042\u308bTransformer\u3082\u540c\u69d8\u306e\u5171\u901a\u70b9\u3092\u6301\u3063\u3066\u3044\u307e\u3059\u3002<\/p>\n\n\n\n<p>\u305d\u308c\u3067\u306f\u3001Transformer\u306eEncoder\u90e8\u5206\u3068Decoder\u90e8\u5206\u306e\u5185\u90e8\u306b\u3064\u3044\u3066\u9806\u306b\u7d39\u4ecb\u3057\u3066\u3044\u304d\u307e\u3059\u3002Transformer\u306eEncoder\u3068Decoder\u306e\u30a2\u30fc\u30ad\u30c6\u30af\u30c1\u30e3\u306e\u5171\u901a\u4e8b\u9805\u3068\u3057\u3066\u306f\u3001\u5165\u529b\u3055\u308c\u305f\u30d9\u30af\u30c8\u30eb\u3092\u57cb\u3081\u8fbc\u307f\u8868\u73fe\u306b\u5909\u63db\u3059\u308bEmbedding\u5c64\u3001\u5358\u8a9e\u306e\u4f4d\u7f6e\u95a2\u4fc2\u3092\u7279\u5fb4\u56f3\u3051\u308bPositional Encoding\u5c64\u3001\u304a\u3088\u3073N\u56de\u7e70\u308a\u8fd4\u3055\u308c\u308bNx\u5c64\u3092\u6301\u3064\u70b9\u3067\u3059\uff08Nx\u306ex\u306f\u7a4d\u3092\u8868\u3057\u3066\u3044\u308b\u3068\u8003\u3048\u3089\u308c\u308b\uff09\u3002<\/p>\n\n\n\n<p>\u305d\u3057\u3066\u3001Nx\u5c64\u306f\u3001Encoder\u3068Decoder\u3067\u7570\u306a\u308a\u307e\u3059\u3002Transformer\u306e\u30a2\u30fc\u30ad\u30c6\u30af\u30c1\u30e3\u3092\u89e3\u8aac\u3059\u308b\u4e2d\u3067\u6700\u3082\u6838\u3068\u306a\u308b\u90e8\u5206\u3067\u3059\u3002<\/p>\n\n\n\n<p>\u56f3\u306b\u304a\u3044\u3066\u3001Encoder Layer (Nx)\u3068\u3057\u3066\u793a\u3055\u308c\u3066\u3044\u308b\u90e8\u5206\u306f\u3001Multi-Head Attention\u3068\u547c\u3070\u308c\u308bAttention\u6a5f\u69cb\u3068\u3001Feed Forward\u5c64\u304b\u3089\u306a\u308a\u307e\u3059\u3002\u307e\u305f\u3001Decoder Layer (Nx)\u3068\u3057\u3066\u793a\u3055\u308c\u3066\u3044\u308b\u90e8\u5206\u306f\u3001Masked Multi-Head Attention\u3068\u547c\u3070\u308c\u308bAttention\u6a5f\u69cb\u3001Multi-Head Attention\u3001Position-wise Feed Forward\u5c64\u304b\u3089\u306a\u308a\u307e\u3059\u3002<\/p>\n\n\n\n<p>\u3053\u306e\u3088\u3046\u306b\u3001Transformer\u306f\u5b8c\u5168\u306b\u3001Attention\u6a5f\u69cb\u3092\u30d9\u30fc\u30b9\u3068\u3057\u305f\u30e2\u30c7\u30eb\u3068\u306a\u3063\u3066\u3044\u307e\u3059\u3002<\/p>\n\n\n\n<p>\u4e0a\u306e\u6982\u7565\u56f3\u3067\u306f\u3001\u6982\u8981\u3092\u6349\u3048\u3084\u3059\u304f\u3059\u308b\u305f\u3081\u306bTransformer\u3092\u69cb\u6210\u3059\u308b\u8981\u7d20\u9593\u306e\u60c5\u5831\u306e\u6d41\u308c\u3092\u7701\u7565\u3057\u3066\u3044\u307e\u3057\u305f\u304c\u3001\u3053\u3053\u307e\u3067\u306e\u8aac\u660e\u3067\u3001\u5927\u307e\u304b\u306b\u69cb\u6210\u8981\u7d20\u3092\u7406\u89e3\u3044\u305f\u3060\u3051\u305f\u3068\u601d\u3046\u306e\u3067\u3001\u5404\u8981\u7d20\u9593\u306e\u7e4b\u304c\u308a\u3084\u60c5\u5831\u306e\u6d41\u308c\u306e\u77e2\u5370\u3092\u8ffd\u52a0\u3057\u305f\u3082\u306e\u3092\u4e0b\u56f3\u306b\u793a\u3057\u307e\u3059\u3002<\/p>\n\n\n\n<p>\u6570\u306f\u3001Transformer\u3068\u8a00\u3063\u305f\u6642\u306b\u4e00\u822c\u7684\u306b\u793a\u3055\u308c\u308b\u56f3\u3068\u307b\u307c\u540c\u3058\u3067\u3059\u3002<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" width=\"1024\" height=\"917\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-8-1024x917.jpg\" alt=\"\" class=\"wp-image-4199\" srcset=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-8-1024x917.jpg 1024w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-8-300x269.jpg 300w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-8-768x687.jpg 768w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-8.jpg 1354w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure><\/div>\n\n\n<p>\u4e0a\u624b\u304b\u3089\u308f\u304b\u308b\u3053\u3068\u3092\u8a18\u8f09\u3057\u307e\u3059\u3002\u307e\u305a\u3001Attention\u304a\u3088\u3073Feed Forward\u306e\u51fa\u529b\u306f\u3001\u305d\u308c\u305e\u308cResidual connection(\u6b8b\u5dee\u7d50\u5408)\u3055\u308c\u3001\u76f4\u5f8c\u306b\u6b63\u898f\u5316\u3055\u308c\u307e\u3059\u3002Encoder\u306eMulti-Head Attention\u306fSelf-Attention\u578b\u304c\u3001Decoder\u306eMulti-Head Attention\u306fSourceTarget-Attention\u304c\u63a1\u7528\u3055\u308c\u3066\u3044\u307e\u3059\u3002Decoder\u306b\u306f\u3001Multi-Head Attention\u306e\u4ed6\u306bMasked Multi-Head Attention\u304c\u3042\u308a\u307e\u3059\u304c\u3001\u3053\u306eMask\u3068\u3044\u3046\u306e\u306f\u3001\u5165\u529b\u304b\u3089padding\u3092\u9664\u5916\u3057\u305f\u308a\u3001\u5148\u8aad\u307f\u3092\u9632\u6b62\u3059\u308b\u305f\u3081\u306b\u4f7f\u7528\u3055\u308c\u3066\u3044\u307e\u3059\u3002<\/p>\n\n\n\n<p>\u4ee5\u964d\u3067\u306f\u3001Transformer\u3092\u69cb\u6210\u3059\u308b\u91cd\u8981\u306a\u4ed5\u7d44\u307f\u3067\u3042\u308b\u3001Embedding\u5c64\u3001Positional Encoding\u3001Multi~Head Attention\u3001Scaled Dot-Product Attention\u3001Feed-Forward\u306b\u3064\u3044\u3066\u9806\u756a\u306b\u8aac\u660e\u3057\u3066\u304d\u307e\u3059\u3002<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Embedding\u5c64<\/h3>\n\n\n\n<p>Embedding\u5c64\u306f\u3001Transformer\u3067\u81ea\u7136\u8a00\u8a9e\u304c\u6271\u3048\u308b\u3088\u3046\u306b\u3001\u5358\u8a9e\u3092\u7279\u5fb4\u30d9\u30af\u30c8\u30eb\u7a7a\u9593\u5185\u306b\u57cb\u3081\u8fbc\u3080\u3068\u304d\u306b\u4f7f\u7528\u3059\u308b\u5c64\u3067\u3001\u5206\u6563\u8868\u73fe\u3068\u547c\u3070\u308c\u308b\u8868\u73fe\u5f62\u5f0f\u3092\u6271\u3046\u3053\u3068\u304c\u3067\u304d\u307e\u3059\u3002<\/p>\n\n\n\n<p>\u5206\u6563\u8868\u73fe\u3068\u306f\u3001\u5358\u8a9e\u306a\u3069\u306e\u30c8\u30fc\u30af\u30f3\u3092\u7dda\u578b\u7a7a\u9593\u4e0a\u306e1\u70b9\u306b\u5bfe\u5fdc\u3055\u305b\u308b\u624b\u6cd5\u3067\u3059\u3002\u57cb\u3081\u8fbc\u307f\u5c64\u3067\u306f\u3001\u4efb\u610f\u306e\u30c8\u30fc\u30af\u30f3\u6bce\u306b\u4e0e\u3048\u3089\u308c\u3066\u3044\u308b\u552f\u4e00\u306e\u30b9\u30ab\u30e9\u30fc\u5024\u3092\u5165\u529b\u3068\u3057\u3066\u53d7\u3051\u53d6\u308a\u3001\u305d\u306e\u5024\u3092OneHot\u30d9\u30af\u30c8\u30eb\u5316\uff06\u7dda\u5f62\u5909\u63db\u3067\u56fa\u5b9a\u9577\u57cb\u3081\u8fbc\u307f\u30d9\u30af\u30c8\u30eb\u306b\u5bfe\u5fdc\u4ed8\u3051\u308b\u50cd\u304d\u3092\u3057\u307e\u3059\u3002<\/p>\n\n\n\n<p>\u5177\u4f53\u4f8b\u3092\u6319\u3052\u3066\u8aac\u660e\u3057\u307e\u3059\u3002\u4f8b\u3048\u3070\u3001weather\u3068\u3044\u3046\u82f1\u5358\u8a9e\u306b256\u3068\u3044\u3046\u30b9\u30ab\u30e9\u30fc\u5024\u304c\u4e0e\u3048\u308c\u3089\u308c\u3066\u3044\u308b\u3068\u3057\u307e\u3059\u3002\u3053\u306e\u3068\u304d\u3001Embedding\u5c64\u306f256\u3092\u53d7\u3051\u53d6\u308b\u3068\u3001256\u756a\u306e\u307f\u304c1\u3067\u305d\u308c\u4ee5\u5916\u304c0\u3067\u3042\u308bOneHot\u30d9\u30af\u30c8\u30eb\u3092\u751f\u6210\u3057\u307e\u3059\u3002\u3053\u308c\u3092\u5358\u7d14\u306a\u30cb\u30e5\u30fc\u30e9\u30eb\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u306b\u5165\u529b\u3057\u3001\u76ee\u7684\u306e\u7279\u5fb4\u7a7a\u9593\u306e1\u70b9\u306b\u5199\u50cf\u3057\u307e\u3059\u3002<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" width=\"1024\" height=\"492\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2022\/04\/IMG_7C503BDC1724-1-1024x492.jpeg\" alt=\"\" class=\"wp-image-4701\" srcset=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2022\/04\/IMG_7C503BDC1724-1-1024x492.jpeg 1024w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2022\/04\/IMG_7C503BDC1724-1-300x144.jpeg 300w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2022\/04\/IMG_7C503BDC1724-1-768x369.jpeg 768w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2022\/04\/IMG_7C503BDC1724-1.jpeg 1297w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure><\/div>\n\n\n<p>\u3067\u306f\u3001\u6587\u66f8\u304c\u5165\u529b\u3055\u308c\u308b\u5834\u5408\u306f\u3001\u3069\u306e\u3088\u3046\u306b\u306a\u308b\u3067\u3057\u3087\u3046\u304b\uff1f<\/p>\n\n\n\n<p>\u4f8b\u3068\u3057\u3066\u3001\u300c\u4eca\u65e5\u306f\u3044\u3044\u5929\u6c17\u3060\u300d\u3092\u8003\u3048\u3066\u307f\u307e\u3059\u3002\u5358\u8a9e\u5206\u5272\u51e6\u7406\u3092\u304a\u3053\u306a\u3046\u3068\u300c\u4eca\u65e5  \u306f  \u3044\u3044  \u5929\u6c17  \u3060\u300d\u3068\u306a\u308a\u307e\u3059\u3002\u305d\u308c\u305e\u308c\u5358\u8a9e\u306b\u4e0e\u3048\u3089\u308c\u305f\u30b9\u30ab\u30e9\u30fc\u5024\u306b\u5909\u63db\u3057\u305f\u3068\u304d\u3001\u300c21  2  32  90  15\u300d\u3068\u306a\u3063\u305f\u3068\u3057\u307e\u3059\u3002\u305d\u308c\u305e\u308c\u3092OneHot\u5f62\u5f0f\u306e\u5217\u30d9\u30af\u30c8\u30eb\u306b\u5909\u63db\u3057\u5408\u4f53\u3055\u305b\u308b\u3068\u3001\u4ee5\u4e0b\u306e\u3088\u3046\u306a\u884c\u5217\u304c\u3067\u304d\u307e\u3059\u3002\u3061\u306a\u307f\u306b\u3001\u884c\u5217\u30b5\u30a4\u30ba\u306f(\u5165\u529b\u6587\u7ae0\u306e\u5358\u8a9e\u6570\u00d7OneHot\u30d9\u30af\u30c8\u30eb\u30b5\u30a4\u30ba)\u3067\u3059\u3002<br><\/p>\n\n\n\n<p>$$<br>\\begin{bmatrix}<br>0&amp;0&amp;\\cdots&amp;0\\\\<br>0&amp;1&amp;\\cdots&amp;0\\\\<br>0&amp;0&amp;\\cdots&amp;0\\\\<br>0&amp;0&amp;\\cdots&amp;0\\\\<br>0&amp;0&amp;\\cdots&amp;0\\\\<br>\\end{bmatrix}<br>$$<\/p>\n\n\n\n<p>\u4e0a\u306e\u884c\u5217\u3092Embedding\u5c64\u306e\u30cb\u30e5\u30fc\u30e9\u30eb\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u3067\u7dda\u5f62\u5909\u63db\u3057\u305f\u3068\u3053\u308d\u3001\u4ee5\u4e0b\u306e\u884c\u5217\u304c\u51fa\u529b\u3055\u308c\u305f\u3068\u3057\u307e\u3059\u3002\u3053\u306e\u884c\u5217\u306e\u30b5\u30a4\u30ba\u306f(\u5165\u529b\u6587\u7ae0\u306e\u5358\u8a9e\u6570\u00d7\u5206\u6563\u8868\u73fe\u6b21\u5143\u6570)\u3067\u3059\u3002<\/p>\n\n\n\n<p>$$<br>\\begin{bmatrix}<br>0.2&amp;0.1&amp;\\cdots&amp;0.9\\\\<br>0.5&amp;0.3&amp;\\cdots&amp;0.8\\\\<br>0.3&amp;0.5&amp;\\cdots&amp;0.3\\\\<br>0.6&amp;0.2&amp;\\cdots&amp;0.2\\\\<br>0.8&amp;0.3&amp;\\cdots&amp;0.1\\\\<br>\\end{bmatrix}<br>$$<\/p>\n\n\n\n<p>\u3053\u306e\u3088\u3046\u306b\u3001\u4eba\u9593\u304c\u6271\u3046\u81ea\u7136\u8a00\u8a9e\u3092\u6a5f\u68b0\u3067\u6271\u3048\u308b\u3088\u3046\u306a\u8868\u73fe\u306b\u5909\u63db\u3059\u308b\u306e\u304cEmbedding\u5c64\u3067\u3059\u3002<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" width=\"1024\" height=\"366\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-10-1024x366.jpg\" alt=\"\" class=\"wp-image-4201\" srcset=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-10-1024x366.jpg 1024w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-10-300x107.jpg 300w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-10-768x274.jpg 768w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-10-1536x549.jpg 1536w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-10-2048x732.jpg 2048w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure><\/div>\n\n\n<h3 class=\"wp-block-heading\">Positional Encoding<\/h3>\n\n\n\n<p>\u5148\u307b\u3069\u3001Embedding\u5c64\u306b\u3088\u308a\u3001\u6587\u7ae0\u3092\u884c\u5217\u3067\u8868\u305b\u308b\u3053\u3068\u3092\u8aac\u660e\u3057\u307e\u3057\u305f\u3002\u6587\u7ae0\u3068\u3044\u3046\u306e\u306f\u3001\u6587\u5b57\u3092\u8aad\u3080\u65b9\u5411\u304c\u91cd\u8981\u3067\u3059\u3002\u3057\u304b\u3057\u3001\u884c\u5217\u3068\u3057\u3066\u8868\u3055\u308c\u3001\u304b\u3064\u3001\u4e00\u62ec\u3067\u51e6\u7406\u3059\u308b\u5834\u5408\u3001\u6587\u5b57\u306e\u9806\u756a\u306e\u6982\u5ff5\u304c\u306a\u304f\u306a\u3063\u3066\u3057\u307e\u3044\u307e\u3059\u3002\u3053\u308c\u306f\u3001\u6587\u7ae0\u3092\u6b63\u3057\u304f\u6271\u3048\u308b\u304b\u5206\u304b\u308a\u307e\u305b\u3093\u3002\u305d\u3053\u3067\u3001Embedding\u5c64\u304b\u3089\u306e\u884c\u5217\u306b\u4f4d\u7f6e\u60c5\u5831\u3092\u542b\u3093\u3060\u884c\u5217\u3092\u8db3\u3057\u5408\u308f\u305b\u308b\u3053\u3068\u3067\u3001\u9806\u756a\u306e\u6982\u5ff5\u3092\u6271\u3048\u308b\u3088\u3046\u306b\u3057\u307e\u3059\u3002\u3053\u308c\u3092\u53ef\u80fd\u306b\u3059\u308b\u306e\u304cPositional Encoding\u3067\u3059\u3002<\/p>\n\n\n\n<p>Positional Encoding\u306f\u3001\u4ee5\u4e0b\u306e\u6b63\u5f26\u95a2\u6570\u3068\u4f59\u5f26\u95a2\u6570\u3092\u7528\u3044\u3066\u3001\u4f4d\u7f6e\u60c5\u5831\u3092\u57cb\u3081\u8fbc\u3093\u3060\u884c\u5217\u3092\u751f\u6210\u3057\u307e\u3059\u3002<\/p>\n\n\n\n<p>$$\\begin{eqnarray}<br>\\boldsymbol{PE}_{(pos, 2i)} &amp;=&amp; \\sin\\left(\\frac{pos}{10000^{\\frac{2i}{d_{model}}}}\\right)\\\\<br>\\boldsymbol{PE}_{(pos, 2i+1)} &amp;=&amp; \\cos\\left(\\frac{pos}{10000^{\\frac{2i}{d_{model}}}}\\right)<br>\\end{eqnarray}$$<\/p>\n\n\n\n<p>\u4f8b\u3048\u3070\u3001\u5165\u529b\u6587\u7ae0\u306e\u5358\u8a9e\u6570\u304c50\u500b\u307e\u3067\u6271\u3048\u3066\u3001Embedding\u5c64\u306e\u57cb\u3081\u8fbc\u307f\u6b21\u5143\u6570\u304c128\u6b21\u5143\u306e\u5834\u5408\u3001Positional Encoding\u304c\u751f\u6210\u3059\u308b\u884c\u5217\u3092\u53ef\u8996\u5316\u3059\u308b\u3068\u3001\u4e0b\u56f3\u53f3\u306e\u3088\u3046\u306b\u306a\u308a\u307e\u3059\u3002\u4f55\u304b\u306e\u67c4\u306e\u3088\u3046\u306b\u306a\u3063\u3066\u3044\u307e\u3059\u304c\u3001128\u6b21\u5143\u306e\u884c\u30d9\u30af\u30c8\u30eb\u304c\u7e26\u306b50\u500b\u4e26\u3079\u3089\u308c\u3066\u3044\u307e\u3059\u3002\u305d\u308c\u305e\u308c\u306e\u884c\u30d9\u30af\u30c8\u30eb\u306f\u3001\u7d76\u5bfe\u306b\u540c\u3058\u3082\u306e\u306b\u306f\u306a\u308a\u307e\u305b\u3093\u3002\u3060\u304b\u3089\u4f4d\u7f6e\u304c\u8868\u305b\u308b\u3093\u3067\u3059\u3088\u306d\u3002<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" width=\"1024\" height=\"399\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-12-1024x399.jpg\" alt=\"\" class=\"wp-image-4203\" srcset=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-12-1024x399.jpg 1024w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-12-300x117.jpg 300w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-12-768x299.jpg 768w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-12-1536x598.jpg 1536w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-12-2048x798.jpg 2048w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure><\/div>\n\n\n<p>\u3053\u306e\u3088\u3046\u306b\u3001\u4f4d\u7f6e\u60c5\u5831\u3092\u3082\u3064\u884c\u5217\u3092Embedding\u5c64\u304b\u3089\u5f97\u3089\u308c\u305f\u884c\u5217\u306b\u8db3\u3057\u5408\u308f\u305b\u307e\u3059\u3002<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" width=\"1024\" height=\"575\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2022\/04\/IMG_CA3FC30228D9-1-1024x575.jpeg\" alt=\"\" class=\"wp-image-4704\" srcset=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2022\/04\/IMG_CA3FC30228D9-1-1024x575.jpeg 1024w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2022\/04\/IMG_CA3FC30228D9-1-300x169.jpeg 300w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2022\/04\/IMG_CA3FC30228D9-1-768x431.jpeg 768w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2022\/04\/IMG_CA3FC30228D9-1.jpeg 1271w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure><\/div>\n\n\n<h3 class=\"wp-block-heading\">Multi-Head Attention<\/h3>\n\n\n\n<p>\u3084\u3063\u3068\u3001Transformer\u3092\u69cb\u6210\u3059\u308b\u6700\u3082\u9769\u65b0\u7684\u306a\u8981\u7d20\u3067\u3042\u308bMulti-Head Attention\u306e\u8aac\u660e\u306b\u305f\u3069\u308a\u7740\u304d\u307e\u3057\u305f\u3002<\/p>\n\n\n\n<p>Multi-Head Attention\u306f\u4e0b\u56f3\u306b\u793a\u3057\u305f\u3088\u3046\u306a\u69cb\u9020\u3092\u3057\u3066\u3044\u307e\u3059\u3002<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" width=\"1024\" height=\"831\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-15-1024x831.jpg\" alt=\"\" class=\"wp-image-4206\" srcset=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-15-1024x831.jpg 1024w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-15-300x244.jpg 300w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-15-768x623.jpg 768w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-15.jpg 1328w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure><\/div>\n\n\n<p>\u307e\u305a\u3001\u6700\u3082\u5927\u304d\u306a\u7279\u5fb4\u306f\u3001\u5185\u90e8\u306b\u6301\u3063\u3066\u3044\u308bScaled Dot-Product Attention\u3092\u30a2\u30f3\u30b5\u30f3\u30d6\u30eb\u3057\u3066\u3044\u308b\u70b9\u3067\u3059\u3002Scaled Dot-Product Attention\u306b\u3064\u3044\u3066\u306f\u3001\u6b21\u306b\u8aac\u660e\u3057\u307e\u3059\u304c\u3001Scaled Dot-Product Attention\u306e3\u3064\u306e\u5165\u529b\u306f\u3001Multi-Head Attention\u306e3\u3064\u306e\u5165\u529b\uff08Value\u3001key\u3001Query\uff09\u3092\u7dda\u5f62\u5c64\u306b\u3088\u308a\u5909\u63db\u3057\u305f\u3082\u306e\u3068\u306a\u3063\u3066\u3044\u307e\u3059\u3002\u305d\u308c\u305e\u308c\u306eScaled Dot-Product Attention\u306e\u76f4\u524d\u306b\u7f6e\u304b\u308c\u3066\u3044\u308b\u7dda\u5f62\u5c64\u306f\u7570\u306a\u308b\u3082\u306e\u3068\u306a\u3063\u3066\u3044\u307e\u3059\u3002\u3053\u308c\u306f\u3001\u305d\u308c\u305e\u308c\u7570\u306a\u308b\u7dda\u5f62\u5909\u63db\u3092\u65bd\u3057\u305f\u5834\u5408\u306b\u304a\u3044\u3066\u6027\u80fd\u304c\u9ad8\u3044\u3053\u3068\u304c\u5b9f\u9a13\u7684\u306b\u78ba\u304b\u3081\u3089\u308c\u305f\u305f\u3081\u3067\u3059\u3002\u3061\u306a\u307f\u306b\u3001\u7dda\u5f62\u5c64\u306f\u5b66\u7fd2\u30d1\u30e9\u30e1\u30fc\u30bf\u3092\u6301\u3063\u3066\u3044\u307e\u3059\u3002<\/p>\n\n\n\n<p>Multi-Head Attention\u3092\u6570\u5f0f\u3067\u8868\u3059\u3068\u3001\u4e0b\u306e\u3088\u3046\u306b\u306a\u308a\u307e\u3059\u3002<\/p>\n\n\n\n<p>$$\\begin{eqnarray}<br>\\rm{MultiHead Attention}(\\boldsymbol{Q}, \\boldsymbol{K}, \\boldsymbol{V}) &amp;=&amp; \\rm{Concat}(head_1, head_2, \\cdots, head_h)\\boldsymbol{W}_o\\\\<br>\\rm{where}\\ head_i &amp;=&amp; \\rm{ScaledDotProductAttention}(\\boldsymbol{QW}^Q_i, \\boldsymbol{KW}^K_i, \\boldsymbol{VW}^V_i)<br>\\end{eqnarray}$$<\/p>\n\n\n\n<p>Multi-Head Attention\u306b\u3064\u3044\u3066\u306f\u3001\u4ee5\u4e0b\u306e\u8a18\u4e8b\u3067\u8a73\u3057\u304f\u8aac\u660e\u3057\u3066\u3044\u307e\u3059\u3002\u3088\u308a\u8a73\u7d30\u3092\u77e5\u308a\u305f\u3044\u65b9\u306f\u3001\u4ee5\u4e0b\u306e\u8a18\u4e8b\u3092\u304a\u8aad\u307f\u304a\u8aad\u307f\u3044\u305f\u3060\u3051\u308c\u3070\u3068\u601d\u3044\u307e\u3059\u3002<\/p>\n\n\n\n<figure class=\"wp-block-embed aligncenter is-type-wp-embed is-provider-agirobots-blog wp-block-embed-agirobots-blog\"><div class=\"wp-block-embed__wrapper\">\n<div><a href=\"https:\/\/developers.agirobots.com\/jp\/multi-head-attention\/\" class=\"st-cardlink st-embed-cardlink\"><div class=\"kanren st-cardbox\"><dl class=\"clearfix\"><dt class=\"st-card-img\"><img decoding=\"async\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2023\/02\/0E0D472B-78CA-45C4-AB6F-39BBD367B8AC-150x150.jpeg\" alt=\"\" width=\"100\" height=\"100\" \/><\/dt><dd><p class=\"st-cardbox-t\">\u3010Transformer\u306e\u57fa\u790e\u3011Multi-Head Attention\u306e\u4ed5\u7d44\u307f<\/p><div class=\"st-card-excerpt smanone\"><p>\u672c\u8a18\u4e8b\u3067\u306f\u3001Transformer\u306e\u57fa\u790e\u3068\u3057\u3066\u3001Multi-Head Attention\u306e\u4ed5\u7d44\u307f\u3092\u5206\u304b\u308a\u3084\u3059\u304f\u89e3\u8aac\u3057\u307e\u3059\u3002 \u672c\u8a18\u4e8b\u306e\u69cb\u6210\u306f\u3001\u306f\u3058\u3081\u306bTransformer\u304a\u3088\u3073Transformer  ... <\/p><\/div><\/dd><\/dl><\/div><\/a><\/div>\n<\/div><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\">Scaled Dot-Product Attention<\/h3>\n\n\n\n<p>\u6b21\u306b\u3001Multi-Head Attention\u5185\u3067\u4f7f\u7528\u3055\u308c\u3066\u308b\u3001Scaled Dot-Product Attention\u306b\u3064\u3044\u3066\u8aac\u660e\u3057\u307e\u3059\u3002<\/p>\n\n\n\n<p>Scaled Dot-Product Attention\u306f\u3001\u4e0b\u56f3\u306b\u793a\u3059\u3088\u3046\u306a\u69cb\u9020\u3092\u3057\u3066\u3044\u307e\u3059\u3002Scaled Dot-Product Attention\u306e\u50cd\u304d\u3092\u4e00\u8a00\u3067\u8868\u3059\u3068\u3001Query\u3068Key\u306e\u985e\u4f3c\u5ea6\u304b\u3089\u3001Value\u306e\u3069\u306e\u5024\u306b\u6ce8\u610f\u3092\u5411\u3051\u308b\u3079\u304d\u304b\u3092\u8a08\u7b97\u3057\u3066\u3044\u307e\u3059\u3002\u985e\u4f3c\u5ea6\u306f\u5185\u7a4d\u306b\u3088\u3063\u3066\u6c42\u3081\u3089\u308c\u307e\u3059\u3002\u305d\u306e\u305f\u3081\u3001Dot-Product\u3068\u3044\u3046\u5358\u8a9e\u304c\u540d\u524d\u306e\u4e2d\u306b\u4f7f\u7528\u3055\u308c\u3066\u3044\u308b\u306e\u3067\u3059\u3002<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large is-resized\"><img decoding=\"async\" width=\"905\" height=\"1024\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-13-905x1024.jpg\" alt=\"\" class=\"wp-image-4204\" style=\"width:453px;height:512px\" srcset=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-13-905x1024.jpg 905w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-13-265x300.jpg 265w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-13-768x869.jpg 768w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-13.jpg 956w\" sizes=\"(max-width: 905px) 100vw, 905px\" \/><\/figure><\/div>\n\n\n<p>Scaled Dot-Product Attention\u306f\u3001\u5185\u7a4d\u3057\u305f\u5024\u306b\u5bfe\u3057\u3066\u3001\u30b9\u30b1\u30fc\u30eb\u5316\u3092\u9069\u7528\u3057\u3066\u3044\u307e\u3059\u3002\u30b9\u30b1\u30fc\u30eb\u5316\u3059\u308b\u7406\u7531\u306f\u3001Query\u3068Key\u306e\u6b21\u5143\u6570\u306b\u4f9d\u5b58\u3057\u3066\u5024\u304c\u5909\u5316\u3059\u308b\u3053\u3068\u3092\u9632\u3050\u305f\u3081\u3067\u3059\u3002\u6b21\u5143\u6570\u304c\u5927\u304d\u3044\u30d9\u30af\u30c8\u30eb\u540c\u58eb\u306e\u5185\u7a4d\u306e\u65b9\u304c\u3001\u6b21\u5143\u6570\u304c\u5c11\u306a\u3044\u30d9\u30af\u30c8\u30eb\u540c\u58eb\u306e\u5185\u7a4d\u3088\u308a\u3082\u5024\u304c\u5927\u304d\u304f\u306a\u308a\u3084\u3059\u3044\u3053\u3068\u306f\u5bb9\u6613\u306b\u60f3\u50cf\u3067\u304d\u308b\u3067\u3057\u3087\u3046\u3002\u3053\u308c\u306f\u3001\u9069\u5207\u306a\u51e6\u7406\u304c\u884c\u3048\u306a\u3044\u305f\u3081\u3001\u6b21\u5143\u6570\u306e\u6839\u53f7\u3067\u5185\u7a4d\u7d50\u679c\u3092\u30b9\u30b1\u30fc\u30ea\u30f3\u30b0\u3057\u307e\u3059\u3002<\/p>\n\n\n\n<p>\u8a08\u7b97\u5f0f\u306f\u4ee5\u4e0b\u306e\u3088\u3046\u306b\u306a\u308a\u307e\u3059\u3002<\/p>\n\n\n\n<p>$$<br>\\rm{ScaledDotProductAttention}(\\boldsymbol{Q}, \\boldsymbol{K}, \\boldsymbol{V}) = \\rm{softmax}\\left(\\frac{\\boldsymbol{QK}^T}{\\sqrt{d_k}}\\right)\\boldsymbol{V}<br>$$<\/p>\n\n\n\n<p>\u3061\u306a\u307f\u306b\u3001Scaled Dot-Product Attention\u306f\u5b66\u7fd2\u30d1\u30e9\u30e1\u30fc\u30bf\u3092\u6301\u3061\u307e\u305b\u3093\u3002<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Feed- Forward<\/h3>\n\n\n\n<p>\u305d\u308c\u3067\u306f\u3001Transformer\u306e\u69cb\u6210\u8981\u7d20\u306e\u8aac\u660e\u306e\u6700\u5f8c\u306b\u306a\u308a\u307e\u3059\u304c\u3001Feed-Forward\u306b\u3064\u3044\u3066\u8aac\u660e\u3057\u307e\u3059\u3002\u53b3\u5bc6\u306b\u306f\u3001Position-wise Feed-Forward\u3068\u547c\u3070\u308c\u3066\u3044\u307e\u3059\u3002position-wise\u3068\u3044\u3046\u306e\u306f\u3001\u300c\u4f4d\u7f6e\u3054\u3068\u306b\u300d\u3092\u610f\u5473\u3057\u3001\u3053\u3053\u3067\u306f\u3001\u5165\u529b\u3055\u308c\u305f\u5404\u5358\u8a9e\u3054\u3068\u306b\u9069\u7528\u3055\u308c\u308b\u3068\u3044\u3046\u3053\u3068\u3092\u8868\u3057\u3066\u3044\u307e\u3059\u3002\u96e3\u3057\u3044\u3088\u3046\u306b\u611f\u3058\u307e\u3059\u304c\u3001\u5b9f\u969b\u306e\u8a08\u7b97\u5f0f\u306f\u5358\u7d14\u3067\u3059\u3002<\/p>\n\n\n\n<p>\u4f7f\u7528\u3055\u308c\u3066\u3044\u308bPosition-wise Feed-Forward\u306f2\u5c64\u3067\u30011\u5c64\u76ee\u306f\u91cd\u307f\u304c\\(\\boldsymbol{W_1}\\)\u3001\u30d0\u30a4\u30a2\u30b9\u304c\\(\\boldsymbol{b_1}\\)\u3001\u6d3b\u6027\u5316\u95a2\u6570\u306fReLU\u30012\u5c64\u306f\u91cd\u307f\u304c\\(\\boldsymbol{W_2}\\)\u3001\u30d0\u30a4\u30a2\u30b9\u304c\\(\\boldsymbol{b_2}\\)\u3001\u6d3b\u6027\u5316\u95a2\u6570\u306f\u6052\u7b49\u95a2\u6570\u3068\u3059\u308b\u3068\u3001\u4ee5\u4e0b\u306e\u3088\u3046\u306a\u5f0f\u3067\u8868\u3055\u308c\u308b\u8a08\u7b97\u3092\u884c\u3044\u307e\u3059\u3002<\/p>\n\n\n\n<p>$$<br>FNN(\\boldsymbol{x}) = \\max (0, \\boldsymbol{xW_1} + \\boldsymbol{b_1})\\boldsymbol{W_2} + \\boldsymbol{b_2}<br>$$<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-medium\"><img decoding=\"async\" width=\"300\" height=\"295\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2022\/04\/IMG_5F600EB1E533-1-300x295.jpeg\" alt=\"\" class=\"wp-image-4710\" srcset=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2022\/04\/IMG_5F600EB1E533-1-300x295.jpeg 300w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2022\/04\/IMG_5F600EB1E533-1.jpeg 597w\" sizes=\"(max-width: 300px) 100vw, 300px\" \/><\/figure><\/div>\n\n\n<h2 class=\"wp-block-heading\">Transformer<\/h2>\n\n\n\n<p>\u6700\u5f8c\u306b\u3001\u3053\u3053\u307e\u3067\u306e\u8a71\u3092\u307e\u3068\u3081\u307e\u3059\u3002<\/p>\n\n\n\n<p>Transformer\u672c\u4f53\u306f\u3001Multi-Head Attention\u3068Feed Forward\u306e\u7d44\u307f\u5408\u308f\u305b\u3092\u57fa\u672c\u69cb\u9020\u3068\u3057\u3001\u305d\u308c\u3092\u30b9\u30bf\u30c3\u30af\u3059\u308b\u3053\u3068\u3067\u69cb\u6210\u3055\u308c\u3066\u3044\u307e\u3059\u3002Transformer\u3067\u6587\u7ae0\u3092\u6271\u3046\u305f\u3081\u306b\u3001\u6587\u7ae0\u3092Embedding\u5c64\u3092\u7528\u3044\u3066\u5206\u6563\u8868\u73fe\u306b\u3057\u3001Positional Encoder\u3067\u4f4d\u7f6e\u60c5\u5831\u3092\u57cb\u3081\u8fbc\u3093\u3060\u3082\u306e\u3092\u5165\u529b\u3068\u3057\u3066\u53d7\u3051\u307e\u3059\u3002<\/p>\n\n\n\n<p>Multi-Head Attention\u306f\u3001Scaled Dot-Product Attention\u3092\u30a2\u30f3\u30b5\u30f3\u30d6\u30eb\u3057\u305f\u69cb\u9020\u3092\u3057\u3066\u3044\u307e\u3059\u3002\u305d\u308c\u304c\u3001Multi-Head\u306e\u6240\u4ee5\u3060\u3063\u305f\u308a\u3057\u307e\u3059\u3002Scaled Dot-Product Attention\u3078\u306e\u5165\u529b\u306f\u3001\u305d\u308c\u305e\u308c\u5b66\u7fd2\u30d1\u30e9\u30e1\u30fc\u30bf\u3092\u3082\u3064\u7dda\u5f62\u5c64\u3092\u304b\u307e\u3057\u305f\u3082\u306e\u3068\u306a\u3063\u3066\u3044\u307e\u3059\u3002<\/p>\n\n\n\n<p>Scaled Dot-Product Attention\u306f\u3001\u5b66\u7fd2\u30d1\u30e9\u30e1\u30fc\u30bf\u3092\u6301\u305f\u305a\u3001\u5165\u529b\u3055\u308c\u305fQuery\u3001Key\u306e\u985e\u4f3c\u5ea6\u304b\u3089\u3001Value\u306b\u542b\u307e\u308c\u3066\u3044\u308b\u60c5\u5831\u3092\u9069\u5207\u306b\u62bd\u51fa\u3059\u308b\u4ed5\u7d44\u307f\u3068\u306a\u3063\u3066\u3044\u307e\u3059\u3002<\/p>\n\n\n\n<p>\u3053\u308c\u304c\u3001\u8fd1\u5e74\u3001\u6ce8\u76ee\u3055\u308c\u3066\u3044\u308bTransformer\u306e\u57fa\u672c\u6982\u8981\u3068\u306a\u308a\u307e\u3059\uff08\u4e0b\u56f3\uff09\u3002<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img decoding=\"async\" width=\"1024\" height=\"510\" src=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-17-1024x510.jpg\" alt=\"\" class=\"wp-image-4208\" srcset=\"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-17-1024x510.jpg 1024w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-17-300x150.jpg 300w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-17-768x383.jpg 768w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-17-1536x765.jpg 1536w, https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2021\/08\/image-32-17.jpg 1826w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure><\/div>\n\n\n<h2 class=\"wp-block-heading\">\u3055\u3044\u3054\u306b<\/h2>\n\n\n\n<p>\u672c\u8a18\u4e8b\u3067\u306f\u3001Transformer\u306b\u3064\u3044\u3066\u3001Attention\u306e\u8d77\u6e90\u304b\u3089\u9806\u756a\u306b\u8aac\u660e\u3057\u3066\u304d\u307e\u3057\u305f\u3002Transformer\u3092\u5229\u7528\u3057\u305fBERT\u3084\u3001GPT\u306a\u3069\u306b\u306f\u89e6\u308c\u3066\u3044\u307e\u305b\u3093\u304c\u3001\u4eca\u5f8c\u306e\u8a18\u4e8b\u3067\u7d39\u4ecb\u3057\u3066\u3044\u3051\u308c\u3070\u3068\u601d\u3044\u307e\u3059\u3002<\/p>\n\n\n\n<p>\u6700\u5f8c\u307e\u3067\u304a\u8aad\u307f\u3044\u305f\u3060\u304d\u3001\u3042\u308a\u304c\u3068\u3046\u3054\u3056\u3044\u307e\u3057\u305f\u3002<\/p>\n","protected":false},"excerpt":{"rendered":"<p>\u672c\u8a18\u4e8b\u3067\u306f\u3001Transformer\u306e\u57fa\u790e\u3068\u3057\u3066\u3001Multi-Head Attention\u306e\u4ed5\u7d44\u307f\u3092\u5206\u304b\u308a\u3084\u3059\u304f\u89e3\u8aac\u3057\u307e\u3059\u3002 \u672c\u8a18\u4e8b\u306e\u69cb\u6210\u306f\u3001\u306f\u3058\u3081\u306bTransformer\u304a\u3088\u3073Transformer  &#8230; <\/p>\n","protected":false},"author":1,"featured_media":4714,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_vk_print_noindex":"","sitemap_hide":"","_veu_custom_css":"","veu_display_promotion_alert":"common","vkexunit_cta_each_option":"","_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[170,171],"tags":[],"class_list":["post-3975","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog","category-learn"],"veu_head_title_object":{"title":"","add_site_title":""},"jetpack_featured_media_url":"https:\/\/developers.agirobots.com\/jp\/wp-content\/uploads\/2022\/04\/0BEBE537-0128-44BE-B0BB-932B12A5680B.jpeg","jetpack-related-posts":[],"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/developers.agirobots.com\/jp\/wp-json\/wp\/v2\/posts\/3975","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/developers.agirobots.com\/jp\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/developers.agirobots.com\/jp\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/developers.agirobots.com\/jp\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/developers.agirobots.com\/jp\/wp-json\/wp\/v2\/comments?post=3975"}],"version-history":[{"count":127,"href":"https:\/\/developers.agirobots.com\/jp\/wp-json\/wp\/v2\/posts\/3975\/revisions"}],"predecessor-version":[{"id":10560,"href":"https:\/\/developers.agirobots.com\/jp\/wp-json\/wp\/v2\/posts\/3975\/revisions\/10560"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/developers.agirobots.com\/jp\/wp-json\/wp\/v2\/media\/4714"}],"wp:attachment":[{"href":"https:\/\/developers.agirobots.com\/jp\/wp-json\/wp\/v2\/media?parent=3975"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/developers.agirobots.com\/jp\/wp-json\/wp\/v2\/categories?post=3975"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/developers.agirobots.com\/jp\/wp-json\/wp\/v2\/tags?post=3975"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}