A labeling system uses a two-dimensional conditional random fields technique to label the object elements. The labeling system represents transition features and state features that depend on object elements that are adjacent in two dimensions. The labeling system represents the grid as a graph of vertices and edges with a vertex representing an object element and an edge representing a relationship between the object elements. The labeling system represents each diagonal of the graph as a sequence of states. The labeling system selects a labeling for the vertices of the diagonals that has the highest probability based on transition probabilities between vertices of adjacent diagonals and on the state probabilities of a position within a diagonal.