René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"电影推荐项目实战（双塔模型）","articleBody":"## 一、推荐的技术方法\r\n\r\n推荐系统简单来说就是， 高效地达成用户与意向对象的匹配。具体可见之前文章：[【一窥推荐系统的原理】](https://mp.weixin.qq.com/s?__biz=MzI4MDE1NjExMQ==\u0026mid=2247487854\u0026idx=1\u0026sn=b3b16d7c10f3e36179b4c5c6f7547c7f\u0026chksm=ebbd93cedcca1ad8682c47d3100d07b31ea517e39d041e441c40f3fa74592034ba9f0cdb3f57\u0026token=229080214\u0026lang=zh_CN#rd)。而技术上实现两者匹配，简单来说有两类方法：\r\n\r\n### 1.1 基于分类方法\r\n\r\n分类的方法很好理解，可以训练一个意向物品的多分类模型，预测用户偏好那一类物品。或者将用户+物品等全方面特征作为拼接训练二分类模型，预测为是否偏好（如下经典的CTR模型，以用户物品特征及对应的标签 0或 1 构建分类模型，预测该用户是否会点击这物品，）。\r\n![](https://upload-images.jianshu.io/upload_images/11682271-4584c6850848b2f0.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240)\r\n\r\n基于分类的方法，精度较高，常用于推荐的排序阶段（如粗排、精排）。\r\n\r\n### 1.2 基于相似度方法\r\n利用计算物与物或人与人、人与物的距离，将物品推荐给喜好相似的人。\r\n如关联规则推荐，可以将物与物共现度看做为某种的相似度；协同过滤算法可以基于物品或者基于用户计算相似用户或物品；以及本文谈到的**双塔模型**，通过计算物品与用户之间的相似度距离并做推荐。\r\n![](https://upload-images.jianshu.io/upload_images/11682271-263721bee326d563.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240)\r\n\r\n利用相似度的方法效率快、准确度差一些常用于推荐中的粗排、召回阶段。\r\n\r\n\r\n\r\n\r\n## 2. DSSM双塔模型\r\n\r\n### 2.1 DSSM模型的原理\r\nDSSM(Deep Structured Semantic Models)也叫深度语义匹配模型，最早是微软发表的一篇应用于NLP领域中计算语义相似度任务的文章。\r\n\r\nDSSM深度语义匹配模型原理很简单：获取搜索引擎中的用户搜索query和doc的海量曝光和点击日志数据，训练阶段分别用复杂的深度学习网络构建query侧特征的query embedding和doc侧特征的doc embedding，线上infer时通过计算两个语义向量的cos距离来表示语义相似度，最终获得语义相似模型。这个模型既可以获得语句的低维语义向量表达sentence embedding，还可以预测两句话的语义相似度。\r\n\r\n### 2.2 DSSM模型结构\r\nDSSM模型总的来说可以分成三层结构，分别是输入层、表示层和匹配层。\r\n- 输入层 将用户、物品的信息转化为数值特征输入；\r\n- 表示层 进一步用神经网络模型学习特征表示；\r\n- 匹配层 计算用户特征向量与物品特征向量的相似度；\r\n\r\n结构如下图所示：\r\n![](https://upload-images.jianshu.io/upload_images/11682271-c61a9338e87df2ca.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240)\r\n\r\n\r\n\r\n## 3.双塔模型代码实践\r\n- 读取电影数据集（用户信息、电影信息、评分行为信息），数据格式处理、特征序列编码、数据拼接,并做评分的归一化处理作为模型学习的相似度目标（注：这里也可以另一个思路对评分做阈值划分，按照一个分类任务来解决）\r\n```\r\nimport pandas as pd\r\nimport numpy as np\r\nimport tensorflow as tf\r\nfrom tensorflow import keras\r\nfrom tensorflow.keras import layers\r\nimport matplotlib.pyplot as plt\r\n\r\n# ### 1. 读取电影数据集（用户信息、电影信息、评分行为信息）\r\n\r\ndf_user = pd.read_csv(\"./ml-1m/users.dat\",\r\n                     sep=\"::\", header=None, engine=\"python\",encoding='iso-8859-1',\r\n                     names = \"UserID::Gender::Age::Occupation::Zip-code\".split(\"::\"))\r\n\r\ndf_movie = pd.read_csv(\"./ml-1m/movies.dat\",\r\n                     sep=\"::\", header=None, engine=\"python\",encoding='iso-8859-1',\r\n                     names = \"MovieID::Title::Genres\".split(\"::\"))\r\n\r\ndf_rating = pd.read_csv(\"./ml-1m/ratings.dat\",\r\n                     sep=\"::\", header=None, engine=\"python\",encoding='iso-8859-1',\r\n                     names = \"UserID::MovieID::Rating::Timestamp\".split(\"::\"))\r\n\r\n\r\n\r\nimport collections\r\n\r\n# 计算电影中每个题材的次数\r\ngenre_count = collections.defaultdict(int)\r\nfor genres in df_movie[\"Genres\"].str.split(\"|\"):\r\n    for genre in genres:\r\n        genre_count[genre] += 1\r\ngenre_count\r\n\r\n\r\n# # 每个电影只保留频率最高（代表性）的电影题材标签\r\ndef get_highrate_genre(x):\r\n    sub_values = {}\r\n    for genre in x.split(\"|\"):\r\n        sub_values[genre] = genre_count[genre]\r\n    return sorted(sub_values.items(), key=lambda x:x[1], reverse=True)[0][0]\r\n\r\ndf_movie[\"Genres\"] = df_movie[\"Genres\"].map(get_highrate_genre)\r\ndf_movie.head()\r\n\r\n\r\n# #### 给特征列做序列编码\r\ndef add_index_column(param_df, column_name):\r\n    values = list(param_df[column_name].unique())\r\n    value_index_dict = {value:idx for idx,value in enumerate(values)}\r\n    param_df[f\"{column_name}_idx\"] = param_df[column_name].map(value_index_dict)\r\n\r\n\r\nadd_index_column(df_user, \"UserID\")\r\nadd_index_column(df_user, \"Gender\")\r\nadd_index_column(df_user, \"Age\")\r\nadd_index_column(df_user, \"Occupation\")\r\nadd_index_column(df_movie, \"MovieID\")\r\nadd_index_column(df_movie, \"Genres\")\r\n\r\n# 合并成一个df\r\ndf = pd.merge(pd.merge(df_rating, df_user), df_movie)\r\ndf.drop(columns=[\"Timestamp\", \"Zip-code\", \"Title\"], inplace=True)\r\n\r\nnum_users = df[\"UserID_idx\"].max() + 1\r\nnum_movies = df[\"MovieID_idx\"].max() + 1\r\nnum_genders = df[\"Gender_idx\"].max() + 1\r\nnum_ages = df[\"Age_idx\"].max() + 1\r\nnum_occupations = df[\"Occupation_idx\"].max() + 1\r\nnum_genres = df[\"Genres_idx\"].max() + 1\r\n\r\nnum_users, num_movies, num_genders, num_ages, num_occupations, num_genres\r\n\r\n\r\n# #### 评分的归一化\r\n\r\nmin_rating = df[\"Rating\"].min()\r\nmax_rating = df[\"Rating\"].max()\r\n\r\ndf[\"Rating\"] = df[\"Rating\"].map(lambda x : (x-min_rating)/(max_rating-min_rating)) # 评分作为两者的相似度\r\n# df[\"is_rating_high\"] = (df[\"Rating\"]\u003e=4).astype(int)  # 可生成是否高评分作为分类模型的类别标签\r\ndf.sample(frac=1).head(3)\r\n# 构建训练集特征及标签\r\ndf_sample = df.sample(frac=0.1)  # 训练集抽样\r\nX = df_sample[[\"UserID_idx\",\"Gender_idx\",\"Age_idx\",\"Occupation_idx\",\"MovieID_idx\",\"Genres_idx\"]]\r\ny = df_sample[\"Rating\"]\r\n```\r\n\r\n![](https://upload-images.jianshu.io/upload_images/11682271-cc9ddcb65f3d3e08.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240)\r\n\r\n- 构建双塔模型，训练预测用户/产品间的相似度。进一步可以提取用户、产品的特征表示方便后续直接结算相似度。\r\n![](https://upload-images.jianshu.io/upload_images/11682271-b73976af932ef253.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240)\r\n\r\n```\r\ndef get_model():\r\n    \"\"\"搭建双塔DNN模型\"\"\"\r\n    \r\n    # 输入\r\n    user_id = keras.layers.Input(shape=(1,), name=\"user_id\")\r\n    gender = keras.layers.Input(shape=(1,), name=\"gender\")\r\n    age = keras.layers.Input(shape=(1,), name=\"age\")\r\n    occupation = keras.layers.Input(shape=(1,), name=\"occupation\")\r\n    movie_id = keras.layers.Input(shape=(1,), name=\"movie_id\")\r\n    genre = keras.layers.Input(shape=(1,), name=\"genre\")\r\n    \r\n    # user 塔\r\n    user_vector = tf.keras.layers.concatenate([\r\n            layers.Embedding(num_users, 100)(user_id), \r\n            layers.Embedding(num_genders, 2)(gender), \r\n            layers.Embedding(num_ages, 2)(age), \r\n            layers.Embedding(num_occupations, 2)(occupation)\r\n    ])\r\n    user_vector = layers.Dense(32, activation='relu')(user_vector)\r\n    user_vector = layers.Dense(8, activation='relu', \r\n                               name=\"user_embedding\", kernel_regularizer='l2')(user_vector)\r\n\r\n    # item 塔\r\n    movie_vector = tf.keras.layers.concatenate([\r\n        layers.Embedding(num_movies, 100)(movie_id),\r\n        layers.Embedding(num_genres, 2)(genre)\r\n    ])\r\n    movie_vector = layers.Dense(32, activation='relu')(movie_vector)\r\n    movie_vector = layers.Dense(8, activation='relu', \r\n                                name=\"movie_embedding\", kernel_regularizer='l2')(movie_vector)\r\n\r\n    # 每个用户的embedding和item的embedding作点积\r\n    dot_user_movie = tf.reduce_sum(user_vector*movie_vector, axis = 1)\r\n    dot_user_movie = tf.expand_dims(dot_user_movie, 1)\r\n\r\n    output = layers.Dense(1, activation='sigmoid')(dot_user_movie)\r\n    \r\n    return keras.models.Model(inputs=[user_id, gender, age, occupation, movie_id, genre], outputs=[output]) \r\n\r\nmodel = get_model()\r\nmodel.compile(loss=tf.keras.losses.MeanSquaredError(), \r\n              optimizer=keras.optimizers.RMSprop())\r\nfit_x_train = [\r\n        X[\"UserID_idx\"], \r\n        X[\"Gender_idx\"],\r\n        X[\"Age_idx\"],\r\n        X[\"Occupation_idx\"],\r\n        X[\"MovieID_idx\"],\r\n        X[\"Genres_idx\"]\r\n    ]\r\n\r\n\r\nhistory = model.fit(\r\n    x=fit_x_train,\r\n    y=y,\r\n    batch_size=32,\r\n    epochs=5,\r\n    verbose=1\r\n)\r\n\r\n\r\n# ### 3. 模型的预估-predict\r\n# 输入前5个样本并做预测\r\n\r\ninputs = df[[\"UserID_idx\",\"Gender_idx\",\"Age_idx\",\"Occupation_idx\",\"MovieID_idx\", \"Genres_idx\"]].head(5)\r\ndisplay(df.head(5))\r\n\r\n# 对于（用户ID，召回的电影ID列表），计算相似度分数\r\nmodel.predict([\r\n        inputs[\"UserID_idx\"], \r\n        inputs[\"Gender_idx\"],\r\n        inputs[\"Age_idx\"],\r\n        inputs[\"Occupation_idx\"],\r\n        inputs[\"MovieID_idx\"],\r\n        inputs[\"Genres_idx\"]\r\n    ])\r\n\r\n# 可以提取模型中的user或movie item 的embedding\r\nuser_layer_model = keras.models.Model(\r\n    inputs=[model.input[0], model.input[1], model.input[2], model.input[3]],\r\n    outputs=model.get_layer(\"user_embedding\").output\r\n)\r\n\r\nuser_embeddings = []\r\nfor index, row in df_user.iterrows():\r\n    user_id = row[\"UserID\"]\r\n    user_input = [\r\n        np.reshape(row[\"UserID_idx\"], [1,1]),\r\n        np.reshape(row[\"Gender_idx\"], [1,1]),\r\n        np.reshape(row[\"Age_idx\"], [1,1]),\r\n        np.reshape(row[\"Occupation_idx\"], [1,1])\r\n    ]\r\n    user_embedding = user_layer_model(user_input)\r\n    \r\n    embedding_str = \",\".join([str(x) for x in user_embedding.numpy().flatten()])\r\n    user_embeddings.append([user_id, embedding_str])\r\ndf_user_embedding = pd.DataFrame(user_embeddings, columns = [\"user_id\", \"user_embedding\"])\r\ndf_user_embedding.head()\r\n```\r\n- 输入前5个样本并做预测，计算用户与电影之间的相似度匹配的分数, 进一步就可以推荐给用户匹配度高的电影。\r\n\r\n![](https://upload-images.jianshu.io/upload_images/11682271-11c0e21b7d864776.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240)\r\n\r\n\r\n\r\n\r\n\r\n(END)\r\n--- \r\n文章首发公众号“算法进阶”，公众号阅读原文可访问[文章相关数据代码及资料](https://github.com/aialgorithm/Blog)","author":{"url":"https://github.com/aialgorithm","@type":"Person","name":"aialgorithm"},"datePublished":"2022-03-19T04:29:49.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":2},"url":"https://github.com/45/Blog/issues/45"}