Miniature optically transparent windows are disclosed that extend vertically from a plane, which may be used to transmit light traveling in a direction substantially parallel with the plane. In one illustrative embodiment, a method for forming such miniature optically transparent windows includes: providing a substrate having a first surface and an opposing second surface, the substrate having a first layer and an adjacent second layer; forming a recess in the first layer of the substrate, the recess extending to the second layer; providing an optically transparent material in the recess to form an optically transparent feature; and removing at least a portion of the first layer that extends adjacent the optically transparent feature so that light can pass through the optically transparent feature in a direction that is substantially parallel to the first surface of the substrate.