|
所在平台: Udemy |
课程主页: https://www.udemy.com/course/semantic-image-search-using-clip/
课程评论：没有评论
课程名称：基于 CLIP 的多模态语义搜索 概述：传统的图像搜索引擎采用词汇方法，对于长查询的表现不佳。这是因为搜索引擎主要关注单个词语的频率以及在文档中的稀有性，同时将搜索关键词与图像的文本元数据（如标签和标题）进行匹配。而对于较长的查询，我们需要一个能够理解文本查询整体意义的搜索引擎，该引擎能够在同一空间中表达文本和图像的嵌入，从而能够通过最近邻方法进行搜索。 OpenAI 的对比语言图像预训练模型（CLIP）提供了跨多模态输入（如文本和图像）的相似性能力，并可以在同一空间中表达它们的嵌入。该模型利用来自 ImageNet 和互联网的现有数据，并采用对比训练方法。 CLIP 是一个零样本模型，这意味着我们不需要在自己的语料库上重新训练，只需使用它生成文本和视觉嵌入，然后根据向量之间的距离排序结果。 在本课程中，我们将使用 OpenAI 的 CLIP 模型，以及 PyTorch 来预处理图像和文本，通过简单的最近邻方法，展示长查询对 Unsplash 图像搜索引擎的影响（使用和不使用 CLIP 模型的对比）。本课程将为学员提供深度学习和嵌入的基础理解。
Conventional Image Search engines use a lexical approach and do not perform well for long queries. The reason behind this is that the search engines look for individual tokens, their frequency in the same document, and the rarity of the tokens among all documents. Also, the search keywords are matched with image textual metadata like tags and titles. For longer queries, we need a search engine that can understand the wholistic meaning of text query and expresses text and image embedding in the same space so that it can perform a search by using the nearest neighbor approach.Contrastive Laniage Image Pre-training model from OpenAI provides similarity capability across multi-modal input such as text and image and express their embeddings in the same space. It leverages existing data from imagenet, internet and uses a contrastive approach to training. It's zero shot modal which means we don't have to retrain on our own corpus. We can just use it to generate text and visual embeddings and just sort the result based on the distance between vectors.In this course, we'll use the CLIP model from openAI, PyTorch for preprocessing images and text, and simple nearest neighbor to illustrate the effect of longer queries on the Unsplash image search engine with and without the CLIP model.This course provides basic understanding of deep learning and embeddings.
