多模态知识图谱综述笔记

Posted on 2022-12-09 Edited on 2024-03-14 In Notes Views:

简介

Tittle: Multi-Modal Knowledge Graph Construction and Application: A Survey
Author: Xiangru Zhu, Zhixu Li Member, IEEE, Xiaodan Wang, Xueyao Jiang, Penglei Sun, Xuwu Wang, Yanghua Xiao Member, IEEE, Nicholas Jing Yuan Member, IEEE
Time: 11 Feb 2022
Journal: IEEE Transactions on Knowledge and Data Engineering

22年多模态知识图谱综述。

主旨

背景

RDF

RDF(Resource Description Framework)，即资源描述框架，其本质是一个数据模型（Data Model）。它提供了一个统一的标准，用于描述实体/资源。可以类比知识图谱的三元组。

Traditional KG

Traditional KG is defined as a directed graph \(\mathcal{G}=\{\mathcal{E,R,A,V,T_R,T_A}\}\).

\(\mathcal{E}\): entities
\(\mathcal{R}\): relations
\(\mathcal{A}\): attributes
\(\mathcal{V}\): literal attribute values
\(\mathcal{T_R=E\times R\times E}\), \(\mathcal{T_A=E\times A\times V}\) are sets of relation triples and attribute triples respectively.

Two Ways for representing MMKG

MMKG (Multi-Modal Knowledge Graph，多模态知识图谱)

A-MMKG: Taking multi-model data as particular attribute values of entities or concepts, as shown in Fig.1(a).
Denote as \(\mathcal{G}=\{\mathcal{E,R,A,V,T_R,T_A}\}\) where \(\mathcal{T_A=E\times A\times (V_{KG}\cup V_{MM})}\) is the set of attribute triples,
\(\mathcal{V_{KG}}\) is the set of the KG’s attribute values and \(\mathcal{V_{MM}}\) is the set of multi-modal data.

就是将原来的 \(\mathcal{T_A}\) 中的 \(V\) 并上多模态的数据。
N-MMKG: Taking multi-model data as entities in KGs as the example shown in Fig.1(b).
Denote as \(\mathcal{G}=\{\mathcal{E,R,A,V,T_R,T_A}\}\) where
\(\mathcal{T_R=(E_{KG}\cup E_{MM})\times R\times (E_{KG}\cup E_{MM})}\) is the set of relation triples,
\(\mathcal{E_{KG}}\) is the set of KG entities and \(\mathcal{E_{MM}}\) is the set of multi-modal data.

将多模态数据拓展到实体集，当作实体考虑。

Two images can also be associated in one of the following relations:
- contain: One image entity visually contains another image entity by the relative position of images.
- nearBy: One image entity is visually nearby another image entity in an image.
- sameAs: Two different image entities refer to the same entity.
- similar: Two image entities are visually similar to each other.
In addition, in N-MMKGs an image is usually abstracted into a number of image descriptors, which are usually summarized into feature vectors of the image entity at pixel level, such as GHD (灰度直方图描述符, Gray Histogram Descriptor), HOG (定向梯度直方图, Histogram of Oriented Gradients Descriptor), CLD (颜色布局描述符, Color Layout Descriptor) and so on.

简单来说，N-MMKG中的图像是经过某些方法处理过后的特征向量。

Examples of A-MMKG and N-MMKG illustrated in Table.1.

Image Captioning. 图像描述. Image captioning aims at generating the descriptive caption for a given image.
Visual Grounding. 视觉定位. Visual grounding aims at locating an object with designated description in a given image.
Visual Question Answering (VQA). 视觉问答. VQA aims at generating a textual answer for a textual question with the help of a relevant image.
Cross-Modal Retrieval. 跨模态检索. There are two classic cross-modal retrieval tasks including searching for images through a text, and searching for texts through an image.

Multi-modal Representation. The multi-modal representation uses the potential complementary of multi-modality to learn feature representation.
主要是两种方法，拼接和融合。拼接是将每种模态投射到各自的向量空间，如linear correlation，再通过一些别的方法来将各个模态进行交融。融合是将所有模态全部投射到一个统一的空间，如VGG、ResNet。

potential complementary: 指模态的潜在互补，每种模态都有一些别的模态没有的信息，如文字“这有一个男孩”，在图片中可以看出男孩的肤色是黑色，那么实际上我们知道这是一个黑人男孩。
Multi-Model Translation. Multi-modal translation learns to translate from a source instance in one modality to a target instance in another.
The example-based translation models build bridges between different modality through dictionary,
while the generative translation models build a more flexible model which can transform one modal to another.
Multi-Model Alignment. Multi-modal alignment aims to find the correspondences between different modalities.
It can either be directly applied in some multi-modal tasks such as visual grounding, or be taken as a pre-training task in multi-modal pretrained language models.
Multi-modal Fusion. Multi-modal fusion refers to the process of joining information from different modalities to perform a prediction,
where various attention mechanism such as gated cross-modality attention, bottom up attention etc. are applied to model the interaction between different kinds of features in the cross-modal module.
Multi-modal Co-Learning. Multi-modal co-learning aims to alleviate the low-resource problems in a certain modality by leveraging the resources of other modalities through the alignment between them.
多模态联合学习通过多模态对齐利用其他类型的模态来缓解某一low-resource的模块。

Based on a large-scale unsupervised multi-modal data set with text-image pairs, 许多工作都在学习多模态预训练语言模型 with designed 预训练自监督任务, including masked language model, sentence image alignment, masked region label classification, masked region feature regression, masked object prediction, etc. 多模态预训练语言模型在下游任务中确实有效。

Can be divided into 1. single-stream models and 2. two-stream models, in terms of the Transformer-based fusion process of different modality.

单流模型 Single-stream models, such as VL-BERT, ViLT, input all modal information into a single Transformer encoder for fusion by self-attention modules.
双流模型 Two-stream models, such as LXMERT, input different modal information into their own encoders and fuse these representation from different modal encoder by an additional cross-attention module.
The final output representation not only contains the cross-modal interaction, but also preserves the interaction within each modality.
不仅包含了每个模块之间的交互，还保留了每个模块内部的交互。

MMKG benefit downstream tasks

MMKG为实体与概念的表达补充足够丰富的background knowledge，特别是那些long-tail ones 长尾问题。
long-tail: a few classes(a.k.a. head class) occupy most of the data, while most classes(a.k.a. tail class) have rarely few samples. 少数类(头类)占用大部分数据，而大多数类(尾类)只有少量的数据。
MMKG可以提升模型对图像中隐藏的物体的理解能力。Unseen object pose great challenge for statistic based models. 这主要是利用符号知识提供的在视觉上不可见物体的符号信息，或在可见物体和不可见物体之间建立语义关系。
MMKG支持多模态推理。Multi-modal reasoning.
MMKG在一些NLP任务中，可以提供多模态数据作为附加特征来帮助弥合信息鸿沟。Take entity recognition for example, an image could provide sufficient infomation to identify whether "Rocky" is the name of a dog or a person.

MMKG Construction. MMKG的构建

MMKG构建的实质是将KG中的符号知识（实体、概念、关系等）与他们对应的图像相关联。有两种相反的方式来完成：1. From images to symbols: Labeling images; 2. From symbols to images: Symbol grounding.

From Images to Symbols: Labeling Images

The CV community has developed many image labeling solutions, which could be leveraged in labeling images with knowledge symbols in KG. Most image labeling solutions learn the mapping from image content to a wide varity of label sets (including objects, sences, entities, attributes, relations, events and other symbols).
The learning procedure is supervised by a human annotated data set, seen Fig.2.

Table.2(a) lists some well-known image-based visual knowledge extraction systems constructed. 这些系统可以通过图像标注来构建MMKG。 According to the category of symbols to be linked, the process of linking images to symbols could be divided into several fractionized tasks:

visual entity/concept extraction: Aim to detect and locate target visual objects in images, and then label these objects with entity/concept symbols in KGs. 从图像中识别实体并用KG中的entity标记;
visual relation extracion: Aim at identifying semantic relations among detected visual entities/concepts in images, and then labeling them with the relations in KGs. 识别已识别的实体之间的语义关系并用KG中的relation标记;
visual event extraction: Aim at 1) predicting the visual event types and 2) locating and exacting objects in source images or videos as visual arguments. 预测视觉事件类型，定位并提取对象作为可视参数。
- EVENT: dynamic interaction among arguments, including a trigger and several arguments with their corresponding argument roles.
- Trigger: a verb or a noun indicating the occurrence of an event.
- Argument role: the semantic relation between an event and an argument such as Time, Person, Place.
如“我在十点半睡觉”是一个event，睡觉是trigger，argument是十点半，argument role是Time，另外我也可以是argument，这时argument role就是Person。

Visual Entity/Concept Extraction 视觉实体/概念提取

Challanges

The main challenge with this task lies on how to learn an effective fine-grained extraction model without a large-scale, fine-grained, well-annotated concept and entity image dataset.
CV中虽然有丰富的标注良好的图像数据，但这些数据集几乎都是粗粒度coarse-grained的概念图像，无法满足MMKG构建对细粒度概念和实体图像标注数据的要求。

CV图像分类中的细粒度/粗粒度(fine-grained/coarse-grained): 粗粒度指类别之间差异大，如人、汽车、树；细粒度指类别之间差异小，如200种鸟的分类、100种花的分类。
由于细粒度类别属于同一个大类，所以各类别之间的差距很小，这些细微的差距容易被光照、颜色、背景、形状和位置等变化因素覆盖，导致细粒度图像分类相对困难。

Progresses

Existing efforts could be roughly divided into two categories:

object recognition methods, which label a visual entity/concept by classifying the region of a detected object; and
visual grounding methods, which label a visual entity/concept by mapping a word or phrase in a caption to the most relevant region.

对象识别方法对检测到对象的区域进行分类，来标记视觉实体/概念；视觉定位是将标题中的单词或短语映射到图像中最相关的区域。这两个的侧重点分别是从图像区域到实体，一个是从实体到图像区域。

Object Recognition methods

早期工作中，提供的图片基本都只有一个物体，但现实生活中的图像往往十分复杂，以至于无法用仅仅一个label来表示。为了识别多个视觉实体，我们需要预训练的检测器和分类器来标注视觉实体（以及属性及场景）及其在图像中的位置。
检测器的训练数据都是有监督数据，主要来自public image-text datasets(such as MSCOCO, Flickr30k, Flick30k Entities and OpenImages)或预标注的种子图像 pre-labeled seed images.

在检测过程中，检测器捕获一组可能对象的候选区域region proposals，并挑出实际包含对象的候选。在检测到的区域中，预先训练的分类器识别具有entity-level(例如BMW 320)或concept-level(例如Car)标签的候选视觉对象。

被识别出来的实体一般不会直接被认为是visual entities，这是因为一般会有大量重复的实例 at different view points, positions, poses and appearances. 因此我们一般会选择最具有代表性的 visual object 作为 visual entitiy。
最常用的方法是对选出来的图像区域进行聚类，每个聚类的中心就被认为是一个新的 visual entity。

缺点：当该方案需要生成大量labels时，需要很多的预处理工作，比如预训练的规则、预确定的实体列表、预训练的细粒度检测器和分类器等，这会导致该方案的鲁棒性scalability下降。

Visual Grounding Method

视觉实体提取问题简化为开域视觉定位问题，即定位标题中的每个短语对应的图像区域，从而获得图像中的视觉对象及其标签。

一般在visual entity extraction中，训练用的 labeled data 需要 with bounding boxes and pre-defined schema with a fixed set of concept, which is difficult to be used for large-scale visual knowledge acquisition.
幸运的是，Web上有很多image-captions pairs，来为visual knowledge extraction提供弱监督weakly supervise without relying on the labeled bounding boxes.

As illustrated in Fig.3 below, weakly supervised learning mainly includs three classical categories:

不完全监督 Incomplete supervision: 训练数据中只有一部分数据被给了标签，有一些数据是没有标签的。
不确切监督 Inexact supervision: 训练数据只给出了粗粒度标签。
不精确监督 Inaccurate supervision: 给出的标签不总是正确的。
在实际操作中，这三类经常同时发生。

关于弱监督学习可以参考周志华《A Brief Introduction to Weakly Supervised Learning》（2018.1）

当从弱监督的图像标题对重提取信息时，通常根据空间热力图(shown as Fig.4)直接选择给定单词的活动像素作为视觉对象的区域。

在同一语义空间中的文本、图像表示，每个短语的热图可以通过基于注意力的方法和基于显著性的方法作为跨模态权重来学习，shown as Fig.5。

在训练时，基于显著性的方法通过梯度计算，直接将像素对给定短语的敏感度视为热力图的值。
基于注意力的方法将跨模态相关性视为热力图的值，与基于显著性的方法相比，其更受欢迎。

有的热图是根据标题中提到的每个实体的图像区域之间的相似性生成的，而有的热图是根据图像区域与可能的事件参数角色类型之间的相似性生成的。

在测试时，对热图进行阈值设置，以获得可视对象的合适包围框。如果kg中已有视觉实体/概念的边界框与新的边界框没有重叠，则将该边界框创建为新的视觉实体或概念。

尽管视觉定位方法不依赖于带有边界框的标记数据，但实际上仍需要人工验证。一些工作试图在训练阶段增加对常识、关系和事件参数的约束，以增加监督信息。在与MMKG的构建相关的工作中，视觉定位的精确度低于 70%。

通过视觉定位的视觉对象可以是实体（例如 Barack Hussein Obama）、概念（例如地点、汽车、石头）、属性（例如红色、短）。然而，图像和文本的语义尺度不一致可能导致不正确的匹配。例如，troops可能会映射到several individuals wearing military uniforms，而Ukraine (country)可能会映射到Ukrainian flag，这两者都是相关的但并不等同。

Opportunities

随着多模态预训练语言模型的出现，多模态预训练语言模型强大的表示能力增强了提取实体和概念的能力。图像块和单词的映射可以直接在模型的自注意力图中可视化，而无需额外的训练。 ViLT 的预测示例如图 6 所示。多模态预训练语言模型如 CLIP，在数亿网络规模的图像-文本数据上进行了预训练，在著名人物和地标建筑上具有很高的准确性，这将在构建一个MMKG人物或建筑时减少大量的数据收集和模型训练工作量。一些预训练的视觉转换器模型已经具有很强的视觉对象分割能力，即使在高度模糊的情况下也能聚焦于前景对象，例如 DINO ，这将提高定位视觉对象和对齐跨模态知识的性能。

Visual Relation Extraction 视觉关系提取