Besides high-resolution images, Google Street View also provides a depth map for each panorama, containing the distance from the camera to the nearest surface at each pixel. As far as I know, there is no official documentation of the format, but there are some open source projects which contain code showing how to interpret the data. The depth information is stored in a slightly complicated way, probably to save bandwidth. Each pixel in a grid of 512×256 pixels references one of several planes. A plane is given by its normal vector and its distance to the camera. Therefore, in order to calculate the depth at a pixel, one has to determine the intersection point of a ray starting at the center of the camera and the plane corresponding to the pixel.
The geometry of the planes and the map containing the index of the plane at each pixel can be retrieved as Base64-encoded and zlib-compressed data by requesting the following URL:
As usual, you can find the code on my github page.