A system determines correspondence between locations on a display surface and pixels in an output image of a projector. The display surface can have an arbitrary shape and pose. Locations of known coordinates are identified on the display surface. Each location is optically coupled to a photo sensor by an optical fiber installed in a throughhole in the surface. Known calibration patterns are projected, while sensing directly an intensity of light at each location for each calibration pattern. The intensities are used to determine correspondences between the locations and pixels in an output image of the projector so that projected images can be warped to conform to the display surface.