After working with these things for the better part of the year, I'm fairly used to how they operate. However even I am a little unsure about the specifics of the technology, and I sometimes forget that they are reasonably unfamiliar to a lot of people who encounter my work. Below are some notes on what LiDAR, photogrammetry and point clouds are, along with some brief notes here and there about how I use them in my studio work (this will be elaborated on in my research summary).
What is LiDAR?
LiDAR stands for Light Detection and Ranging and is a remote-sensing technology that uses a laser beam to gather information about objects. Whilst initially a fairly expensive type of technology limited to industrial and scientific uses, it is now found in various consumer devices including self-driving cars, AR and VR headsets and gadgets, and smartphones.
In my practice, I alternate between using a smartphone to capture images for photogrammetry, and using an iPhone 12 with LiDAR capability to make direct captures - the latter often due to its built-in model generating capabilities.
There are three main components of any LIDAR device: a laser, a scanner, and a GPS receiver.
The basic process:
A laser emits a signal
The signal reaches an obstacle
The signal reflects off the obstacle
The signal returns to a receiver
The laser pulse is registered
Each registered pulse is assigned a location and representation in a LiDAR point cloud
(right: LiDAR scanner on an Apple iPhone 13)
Systems that use LiDAR send out pulses of light that in invisible to the human eye and measure how long it takes to return. Whenever the pulse reaches something, data points are collected regarding the object's direction and distance. As a result, LIDAR can use geo-referencing in regard to each point in a point cloud to create detailed maps of real world environments.
Using LiDAR, one can determine the shape and orientation of any object based on the time of flight, a.k.a. the time difference between the emission of a laser signal and its return to a sensor after bouncing off an object or surface. Through repetition of this process, a device can figure out how objects are positioned in space, including if and how they move and the speed at which they do so. If you've ever seen a land surveying team at work, this is a good real-world viewable example of the technology in use! (When I use my iPhone to create LiDAR scans, I'm basically doing a scaled-down version of this.)
Some devices, including the iPhone 12, feature in-built software that automatically generate 3D models from LiDAR scans. (However, the raw unprocessed images and point cloud data from scans can be exported as they are - in my case, downloaded from the iPhone to my computer for viewing or assembly in other programs.)
LiDAR has been identified by NASA as a promising technology that could be used in future for the autonomous piloting and landing of robotic, lunar vehicles. Other industries that are making use of LiDAR's detail capturing abilities include forensics, the automobile industry, architecture, robotics, manufacturing, geology, the oil and gas industry, environmental conservation, forestry, and of course, art, among many others.
What is photogrammetry?
Photogrammetry is the science and technology of obtaining information about physical objects and the environment through recording, measuring and interpreting photographic images.
(above: an image created by putting hundreds of images of a bumblebee on a corn plant in my garden into Meshroom, a free photogrammetry program.)
While LiDAR is a technology for making point clouds, not all point clouds are created using LiDAR. For example, point clouds can be made from images obtained from digital cameras, a technique known as photogrammetry. As opposed to 3D scanning, photogrammetry uses photographs rather than light to gather data.
When taking photos using digital cameras, the exterior orientation of the device (which defines its location in space and view direction) and the inner orientation (which defines geometric parameters such as lens focal length, lens distortions, etc.) are documented. This information can be used along with the images captured to generate 3D models and point clouds. If this locational data isn't available, the overlapping and meshing together of images can generate a model alone. Through both processes, measurements of scale, distance, etc. can be calculated in photogrammetry programs.
As with LiDAR, photos have to be taken from multiple angles to capture an object's full geometry. While photogrammetry is not as accurate at capturing 3D information as LiDAR, the main advantage of using photogrammetry for 3D modelling is its ability to reproduce an object in full colour and texture. While some 3D scanners can do this, photogrammetry lends itself to this purpose better as photographs can more easily capture realism present in environments.
Photogrammetry is also more easily accessible to many people as equipment and software are not as expensive. In my own practice, I use a combination of iPhone technology (which blends LiDAR modellings + photogrammetry texturing) and a more "analogue" combination of smartphone photos + free software (Meshroom and RealityCapture) to create files.
As mentioned above, the iPhone I use in my practice to capture some of my scans and images has both LiDAR and photogrammetry capabilities - the latter is used primarily to create texture maps for scanned objects and environments:
"The [iPhone's] 3D model is generated by combining lidar and photogrammetry (pictures taken from different views of the object to create a 3D reconstruction),... the lidar provides a known scale and the photos provide texture" *
What are point clouds?
In very simple terms, a point cloud is a bunch of 3D points on the surface of objects that are present in the sight of a sensor. Point clouds are essentially datasets made up of ‘points.’ Point clouds may represent 3D shapes or objects, and these points are always located on the surfaces of objects. Each point has three coordinates - known as Cartesian coordinates (X, Y, Z) - assigned to it to position it in space; and colour, light intensity, contrast and other information is also often captured along with this geographical data.
(above: point cloud generated from a combined iPhone LiDAR and photogrammetry scan)
In most cases, a point cloud works as a "middleman" between the raw data collected by a LiDAR device and 3D models. Point clouds can be used to store any spatial information. Because every point has data about where it is in space, point clouds can be used to take incredibly accurate measurements of a space, volume, object, etc.
Each point represents something (or a part of something) and the point contains information on where that thing is in space. Together, these points form an image of what has been scanned. Viewed from a distance in point cloud software, they look a bit like abstract, impressionist paintings; and up close, you can see all the individual points that make up the scan.
Point clouds are produced by 3D scanners and photogrammetry software by measuring many points on the external surfaces of objects. One needs specific software to view, manipulate or extract value from point cloud data. I use a combination of Meshroom, Blender and RealityCapture to do this in my practice - either used to view point clouds generated via iPhone-based LiDAR scans, or to assemble photogrammetry files that contain exportable point clouds. Before point cloud data can be viewed it must be ‘registered’ so that the data lines up and looks like the object or space that has been scanned.
Multiple scans are often required to create an accurate representation of a space or object, taken from multiple angles. These then may be stitched together. As the lasers from 3D scanners can’t penetrate solid objects or structures, point clouds only ever represent the surface of an object (however, as with other similar illusions, can generate a sense of space, physicality, etc.):
On the one hand, a point cloud is the most complete set of raw measurements of real-world objects possible. On the other hand, it is just a collection of "dumb" points without interpretation nor physical meaning of what is represented... Point clouds only contain information on the outside of objects. This is in contrast with data from ultrasound, CT or MRI scanners, which contain data of the full inside of the objects as well. *
(illustration of a LiDAR device emitting laser pulses to construct point clouds)
What is texture mapping?
Texture mapping, also known as UV texturing or UV mapping, is a graphic design process in which a 2D image is "wrapped" around a 3D object - imagine covering a globe with a paper print-out of a map. The 2D surface is called a texture map. It is the digital equivalent of applying wallpaper, paint, or veneer to a real world object.
The UV mapping process at its simplest requires three steps: unwrapping the mesh, creating the texture, and applying the texture to a respective face of polygon.
The letters "U" and "V" stand for the axes of the 2D texture. This is because the letters "X", "Y", and "Z" are already used to denote the axes of the 3D object in space.
A UV map can either be generated automatically by the software application, made manually, or through a combination of both. Often a UV map will be generated and then adjusted and optimized to minimize seams and overlaps.
When a model is created as a polygon mesh in a 3D modelling program, UV coordinates can be generated for each vertex in the mesh. One way to do this is for the 3D program to unfold the triangle mesh at the seams, automatically laying out the triangles on a flat page. Once the model is "unwrapped", the artist can paint a texture on each triangle individually, using the unwrapped mesh as a template. This process is known as "UV unwrapping".
During the process of LiDAR scanning with an iPhone (as in my work), an image texture/UV texture is automatically created by the phone's built in camera, which captures the texture (colour detail, etc.) of the object as the scanning takes place. This is then wrapped onto the generated 3D object.
Below is an example of a UV/image texture generated via an iPhone LiDAR scan of mine, and the resulting textured object with the image wrapped onto it. If you zoom in on the texture image, you will see some blurred boundaries between areas in focus - this is where the program has wrapped the texture around the object and either filled in missing visual information or assembled the image so that necessary parts lie next to one another in a way that makes sense. Below this a separate example - a preview of a texture mapped onto an object in the scanning process, in which the polygons of the object are visible as they are being calculated .
Comments