scene graph generation

Scene graph generation (SGG) is the task of summarizing an image as a graph of objects and the relationships between them. Each detected object becomes a node (with a category and a box, e.g. 'man', 'horse', 'hat'), and each meaningful relationship becomes a directed edge labeled with a predicate, written as a triplet subject-predicate-object: 'man riding horse', 'man wearing hat', 'horse on grass'. The result is a compact, structured, machine-readable description of who is doing what to whom — going beyond 'what objects are present' to 'how they relate'.

SGG is built on top of object detection plus relationship prediction. After detecting objects, the model considers pairs of objects and predicts whether and how they are related — classifying the predicate (spatial like 'on'/'under', possessive like 'has'/'wearing', or action like 'riding'/'eating') using the two boxes, their relative geometry, and their visual and semantic features. Because relationships are contextual, strong models pass messages between objects (graph neural networks, or the MOTIFS model's observation that object labels alone strongly predict likely predicates), so the graph is refined jointly rather than each edge in isolation.

The standard dataset is Visual Genome, and a defining challenge is the long-tailed, biased predicate distribution: trivial predicates like 'on' and 'has' dominate, so a model can score well on recall while ignoring informative relations like 'riding' or 'standing on'. This motivated unbiased SGG and the mean-recall metric (averaging recall across predicate classes to reward rare ones). Scene graphs matter because they provide explicit structure for downstream reasoning — they underpin compositional VQA (GQA is built from scene graphs), grounded image generation and retrieval, and image captioning that respects relationships rather than just listing objects.