“State-of-the-art active optical techniques for three-dimensional surface metrology: a review,” (2020)

A. G. Marrugo, F. Gao, and S. Zhang, “State-of-the-art active optical techniques for three-dimensional surface metrology: a review,” Journal of Optical Society of America A 37(9) B60-B77 (2020) [Open Access]

Abstract

This paper reviews recent developments of non-contact 3D three-dimensional (3D) surface metrology using an active structured optical probe. We primarily focus on those active non-contact 3D surface mea- surement techniques that could be applicable to the manufacturing industry. We discuss principles of each technology, and its advantageous characteristics as well as limitations. Towards the end, we will discuss our perspectives on the current technological challenges in designing and implementing these methods in practical applications. 

"Status, challenges, and future perspectives of fringe projection profilometry," (2020)

J. Xu and S. Zhang, “Status, challenges, and future perspectives of fringe projection profilometry.” Optics and Lasers in Engineering, 135, 106193 (2020); doi: j.optlaseng.2020.106193

Abstract

As one of the most popular techniques for non-contact three-dimensional (3D) sensing/imaging, fringe projection profilometry (FPP) has been growing rapidly over the past decades partially because of the improved speed of computing devices and reduced cost of hardware.  3D optical sensing has started being an integral part of  our daily lives such as Face ID enabled by 3D sensors on smart phones. The impact of such techniques can be even greater with the ever-growing artificial intelligent (AI), machine learning, smart manufacturing, robotics as well as other fields.  However,  there are still fundamental challenges to be tackled to make such advanced optical sensing techniques ubiquitous.  This paper presents the current status of FPP techniques, the major challenges still facing in the field, and our perspectives on the future of FPP techniques.

"Autofocusing method for digital fringe projection system with dual projectors," (2020)

M. Zhong, X. Hu, F. Chen, C. Xiao, D. Peng, and S. Zhang, "Autofocusing method for digital fringe projection system with dual projectors," Optics Express 28(9), 12609-12620 (2020)

Abstract

This paper presents a novel technique to achieve autofocusing for a three-dimensional (3D) profilometry system with dual projectors. The proposed system uses a camera that is attached with an electronically focus-tunable lens (ETL) that allows dynamic change of camera's focal plane such that the camera can focus on the object; the camera captures fringe patterns projected by each projector to establish corresponding points between two projectors; two pre-calibrated projectors form triangulation for 3D reconstruction. We pre-calibrate the relationship between the depth and the current being used for each focal plane, perform a 3D shape measurement with an unknown focus level, and calculate the desired current value based on the initial 3D result. We developed a prototype system that can automatically focus on an object positioned between 450 mm to 850 mm.

"Influence of projector pixel shape on ultrahigh-resolution 3D shape measurement", (2020)

J.-S. Hyun and S. Zhang, “Influence of projector pixel shape on ultrahigh-resolution 3D shape measurement”, Optics Express 28(7), 9510-9520 (2020); do:10.1364/OE.389331

Abstract

The state-of-art three-dimensional (3D) shape measurement with digital fringe projection (DFP) techniques assume that the influence of projector pixel shape is negligible. However, our research reveals that when the camera pixel size is much smaller than the projector pixel size in object space (e.g., 1/5), the shape of projector pixel can play a critical role on ultimate measurement quality. This paper evaluates the performance of two shapes of projector pixels: rectangular and diamond shaped. Both simulation and experimental results demonstrated that when, the camera pixel size is significantly smaller than the projector pixel size, it is advantageous for ultrahigh resolution 3D shape measurement system to use a projector with rectangular shaped pixels than the projector with diamond-shaped pixels.

"A convenient 3D reconstruction model based on coaxial structure-light system," (2020)

J. Zhang, B. Luo, X. Su, L. Li, B. Li, S. Zhang, and Y. Wang,  "A convenient 3D reconstruction model based on coaxial structure-light system," Optics and Lasers in Engineering, 138, 106366 (2021)

Abstract

Conventional high-accuracy three-dimensional (3D) reconstruction methods usually involve complicated and time- consuming system calibration. This Letter presents a novel and simple 3D reconstruction model based on a coaxial structure-light system (CSLS). Compared to traditional structured light system, the proposed system contains a perspective projection unit and a telecentric imaging unit. Instead of obtaining the complicated structure-light system parameters, the proposed model only requires the retrieval of a product matrix which could be easily calibrated by using an ordinary white plane. For an arbitrary point A on the object surface, it is mathematically proved that the product matrix of its phase map and height distribution is proportional to the distance between A and projector optical center in XY plane, and independent of the depth in Z direction. Experiments were carried out to measure a step- height object and a cylindrical surface, and the results verify that the proposed method can achieve high-accuracy 3D shape measurement.

"Hybrid calibration procedure for fringe projection profilometry based on stereo-vision and polynomial fitting," (2020)

R. Vargas, A. G. Marrugo, S. Zhang, L. A. Romero "Hybrid calibration procedure for fringe projection profilometry based on stereo-vision and polynomial fitting," Applied Optics, 59(13), D163-D169 (2020); doi:10.1364/AO.383602

Abstract

The key to accurate 3D shape measurement in Fringe Projection Profilometry (FPP) is the proper calibration of the measurement system. Current calibration techniques rely on phase-coordinate mapping (PCM) or back-projection stereo-vision (SV) methods. PCM methods are cumbersome to implement as they require precise positioning of the calibration target relative to the FPP system but produce highly accurate measurements within the calibration volume. SV methods generally do not achieve the same accuracy level. However, the calibration is more flexible in that the calibration target can be arbitrarily positioned. In this work, we propose a hybrid calibration method that combines SV and PCM methods. The method has the flexibility of SV methods, is robust to lens distortions, and has a simple relation between the recovered phase and coordinates. Experimental results show that the proposed Hybrid method outperforms the SV method in terms of accuracy, acquisition time (as it requires fewer acquired images), and reconstruction time due to its low computational complexity.

"High-speed three-dimensional absolute shape measurement with three binary patterns," (2020)

J.-S. Hyun and S. Zhang, "High-speed three-dimensional absolute shape measurement with three binary patterns," Optical Engineering, 59(2), 024104 (2020); doi: 10.1117/1.OE.59.2.024104

Abstract

Reducing the number of structured patterns for three-dimensional (3D) reconstruction is of great importance for high-speed 3D shape measurement. This paper presents a method that reconstructs absolute 3D shape using three binary patterns: one DC, one low-frequency and one high-frequency fringe pattern. The procedures are: 1) take the difference between the sinusoidal fringe patterns and the DC pattern;  2) apply Hilbert transform to the difference images to generate two phase maps; 3) employ the geometric constraint based phase unwrapping method to unwrap the low-frequency phase map; 4) unwrap the high-frequency phase map using the unwrapped low-frequency phase map; and 5) reconstruct 3D shape.  We developed a prototype system that can capture 2D images at 6,000 Hz, achieving 2,000 Hz 3D shape measurement speed.

"Multilevel symmetric pattern design and optimization for high-speed and high- accuracy 3D shape measurement," (2020)

Y. Wang, J. S. Hyun, S. Zhang, B. Luo, Z. Liu, C. Jiang, B. Tao, "Multilevel symmetric pattern design and optimization for high-speed and high- accuracy 3D shape measurement," Optics and Laser Technology, 126, 106103 (2020); doi:10.1016/j.optlastec.2020.106103

Abstract

The binary defocusing technique has enabled speed breakthroughs for 3D shape measurement, yet simultaneously achieving high accuracy and high speed remains difficult. To overcome this limitation, we propose to utilize multilevel symmetric pattern for high-speed and high-accuracy 3D measurement. Compared to conventional binary patterns, multilevel pattern could bring more flexibility for eliminating undesired high-frequency harmonics, thus has the potential to greatly enhance the phase quality and measurement accuracy. In this paper, the symmetric pattern design principle and related optimization procedure were presented to find the best multilevel fringe patterns. Both simulation and experiments verify that comparing with conventional methods, the proposed method could consistently generate better fringe patterns for a wide range of fringe periods. Furthermore, we developed an absolute 3D shape measurement system with the speed of 667 Hz, verifying that the proposed method is applicable for high-speed, high-accuracy applications.

"Uniaxial 3D phase-shifting profilometry using a dual-telecentric structured light system in micro-scale devices," (2020)

M. Zhong, J. Cui, J.-S., Hyun, L. Pan, P. Duan, and S. Zhang, "Uniaxial 3D phase-shifting profilometry using a dual-telecentric structured light system in micro-scale devices," Measurement Science and Technology, 31(8), 085003 (2020); doi: 10.1088/1361-6501/ab63b2

Abstract

This paper presents a novel uniaxial microscopic 3D profilometry method using a structured light system with dual-telecentric lenses in micro-scale devices. Specifically, a telecentric lens can produce an orthographic view of an object and provide the exact size of objects in \textit and \textit-direction. An electronically focus-tunable lens attached to the projector rapidly and precisely changes the focal plane of the projected structured patterns, a camera captures the structured patterns from the same perspective, and a computational framework analyzes the relationship between captured structure images and the drive current of the electronically focus-tunable lens to obtain depth information for each pixel. By replacing the normal lens with telecentric lenses, the proposed method dodges perspective error and is easier for precision measurement. We developed a shadow-free coaxial 3D shape measurement system with dual telecentric lenses that achieved approximately 5.86 $\mu$m spatial resolution and 4.18 $\mu$m root-mean-square (RMS) error with a depth range of 1200 $\mu$m.

"Rapid and automatic optimal exposure control for digital fringe projection technique," (2020)

S. Zhang, "Rapid and automatic optimal exposure control for digital fringe projection technique," Optics and Lasers in Engineering 128, 106029 (2020); doi:10.1016/j.optlaseng.2020.106029

Abstract

This paper presents a method that can determine the optimal exposure time for high-quality three-dimensional (3D) shape measurement with digital fringe projection technique. The proposed method only requires capture fringe image(s) with one exposure time to automatically determine the global optimal exposure time for high-quality 3D shape measurement; and we further propose to automatically achieve high-dynamic range (HDR) 3D shape measurement by capture  image(s) with the optimized global single exposure.  Experimental results demonstrated the proposed method can successful work on a static complex scene and can also be applicable for real-time applications.

"Autofocusing method for high-resolution three-dimensional profilometry," (2020)

[121] X. Hu, G. Wang, J.-S. Hyun, Y. Zhang, H. Yang, S. Zhang, "Autofocusing method for high-resolution three-dimensional profilometry," Opt. Lett. 45(2), 375-378 (2020)

Abstract

State-of-the-art high-accuracy three-dimensional (3D) profilometry systems typically use lens with a fixed focal length, making it difficult for them to measure scenes with large depth variations especially dynamically changing ones. To address this need, this Letter proposes a novel autofocusing method for high-resolution 3D profilometry with a digital fringe projection (DFP) technique by 1) developing a novel continuous geometric parameter model for systems using electronically tunable lenses, and 2) employing a focal plane detection algorithm. The validity of the proposed method is confirmed by experiments.

"Calibration method for panoramic 3D shape measurement with plane mirrors," Opt. Express (2019)

[120] W. Yin, S. Feng, T. Tao, L. Huang, S. Zhang, Q. Chen, C. Zuo, “Calibration method for panoramic 3D shape measurement with plane mirrors," Opt. Express, 27(25), 36538-36550 (2019)

Abstract

High-speed panoramic three dimensional (3D) shape measurement can be achieved by introducing plane mirrors into the traditional fringe projection profilometry (FPP) system because such a system simultaneously captures fringe patterns from three different perspectives ( i.e., by a real camera and two virtual cameras in the plane mirrors). However, calibrating such a system is nontrivial due to the complicated setup. This work introduces a flexible new technique to calibrate such a system. We first present the mathematical representation of the plane mirror, and then mathematically prove that it only requires the camera to observe a set of feature point pairs (including real points and virtual points) to generate a solution to the reflection matrix of a plane mirror. By calibrating the virtual and real camera in the same world coordinate system, 3D point cloud data obtained from real and virtual perspectives can be automatically aligned to generate a panoramic 3D model of the object. Finally, we developed a system to verify the performance of the proposed calibration technique for panoramic 3D shape measurement.

"Large depth-of-field 3D shape measurement using an electrically tunable lens," Opt. Express (2019)

X. Hu, G. Wang, Y. Zhang, H. Yang, and S. Zhang, “Large depth-of field 3d shape measurement using an electrically tunable lens," Opt. Express 27(21), 29697-29709 (2019).

Abstract

The state-of-the-art 3D shape measurement system has rather shallow working volume due to the limited depth-of-field (DOF) of conventional lens. In this paper, we propose to use the electrically tunable lens to substantially enlarge the DOF. Specifically, we capture always in-focus phase-shifted fringe patterns by precisely synchronizing the tunable lens attached to the camera with the image acquisition and the pattern projection; we develop a phase unwrapping framework that fully utilizes the geometric constraint from the camera focal length setting; and we pre-calibrate the system under different focal distance to reconstruct 3D shape from unwrapped phase map. To validate the proposed idea, we developed a prototype system that can perform high-quality measurement for the depth range of approximately 1,000 mm (400 mm – 1400 mm) with the measurement error of 0.05%. Furthermore, we demonstrated that such a technique can be used for real-time 3D shape measurement by experimentally measuring moving objects.

"Pixel-by-pixel absolute phase retrieval assisted by an additional three-dimensional scanner," Appl. Opt., (2019)

Y. An and S. Zhang, “Pixel-by-pixel absolute phase retrieval assisted by an additional three-dimensional scanner”, Appl. Opt., 58(8), 2033-2041, 2019, doi:10.1364/AO.58.002033

Abstract

This paper presents a novel absolute phase unwrapping method assisted by a low-cost three-dimensional (3D) scanner. The proposed absolute phase unwrapping method leverages a low-cost 3D scanner to capture rough 3D data of the scene, and transforms the rough 3D data to the world coordinate system to generate an artificial reference phase map Φ_. By referring to Φ_, we can do absolute phase unwrapping directly without projecting any additional patterns, such that the digital fringe projection (DFP) system can achieve higher measurement speed. We develop a multi-resolution system consisting of a DFP system and Kinect V2 to validate our method. Experiments demonstrate that our method works for a large depth range, and the speed of the low-cost 3D scanner is not necessarily the maximum speed of our proposed method. Assisted by Kinect V2 whose maximum speed is only 30Hz, our DFP system achieves 53Hz with a resolution 1600x1000 pixels when we measure dynamic objects that are moving in a large depth range of 400mm.

"Motion induced phase error reduction using a Hilbert transform," Opt. Express (2018)

 [117] Y. Wang, Z. Liu, and C. Jiang, and S. Zhang, "Motion induced phase error reduction using a Hilbert transform," Opt. Express 26(26), 34224-34235 (2018); doi:10.1364/OE.26.034224

Abstract

The motion of object could introduce phase error and thus measurement error for phase-shifting profilometry. This paper proposes a generic motion error compensation method based on our finding that the dominant motion introduced phase error doubles the frequency of the projected fringe frequency, and Hilbert transform shifts the phase of a fringe pattern by $\pi/2$. We apply Hilbert transform to phase-shifted fringe patterns to generate another set of fringe patterns, calculate one phase map using the original fringe patterns and another phase map using Hilbert transformed fringe patterns, and then use the average of these two phase maps for 3D reconstruction. Both simulation and experiments demonstrated the proposed method can substantially reduce motion-introduced measurement error.

"Depth-driven variable-frequency sinusoidal fringe pattern for accuracy improvement in fringe projection profilometry," Opt. Express (2018)

 [116] G. Rao, L. Song, and S. Zhang, X. Yang, K. Chen, and J. Xu, "Depth-driven variable-frequency sinusoidal fringe pattern for accuracy improvement in fringe projection profilometry," Opt. Express 26(16), 19986-20008 (2018); doi:10.1364/OE.26.019986

Abstract

Sinusoidal fringe pattern is widely used in optical profilometry; however, the traditional constant-frequency sinusoidal fringe pattern reduces 3D measurement accuracy in the defocus region. To this end, this paper presents a variable-frequency sinusoidal fringe pattern method that is optimized by the measurement depth. The proposed method  improves the pixel matching accuracy and thus increases measurement accuracy. This paper  1) theoretically determines the optimal frequency by analyzing the pixel matching error caused by intense noise in a captured image; 2) presents the online frequency  optimization along abscissa and ordinate axes in the sinusoidal fringe patterns; 3) details the encoding and decoding to use variable-frequency  fringe patterns for 3D profilometry. Simulations and experiments demonstrate that that our proposed method can improve the 3D measurement accuracy and increase measurement robustness.

"Guest Editorial Focused Section on sensing and perception systems for intelligent manufacturing (SPIM)," IEEE/ASME Trans. Mechatronics (2018)

[115] X. Chen, S. Zhang, and J. M.P. Geraedts, "Guest Editorial Focused Section on sensing and perception systems for intelligent manufacturing (SPIM)," IEEE/ASME Trans. Mechatronics 23(3), 983-984 (2018); doi:10.1109/TMECH.2018.2837008

Abstract

This Focused Section provides a state-of-the-art update of research fronts in Field Sensing and Perception as  well as the applications to intelligent manufacturing problems. The topics covered in the collected papers include: optimization of field sensing network and systems, intelligent sensing and perception for robotic manufacturing processes, SLAM of indoor environments, and model guided measurement of various field effects. The problems related to field sensing and perception are challenging and spread out in various areas in practices. To our best knowledge, this Focused Section is the first effort to try to collectively present the research results related sensing and perception of ‘field effects’ which exist in various manufacturing  processes. It is our intention that this Focused Section would serve as an efficient highlight of such challenging and important problems to attract intensive research interests on these problems by arguably claiming an effective research area for field sensing and perception. 

"Motion-induced error compensation for phase shifting profilometry," Opt. Express (2018)

[114] Z. Liu, P. Zibley and S. Zhang, "Motion-induced error compensation for phase shifting profilometry," Opt. Express, 26(10), 12632-12637 (2018); doi:10.1364/OE.26.012632

Abstract

This paper proposes a novel method to substantially reduce motion-introduced phase error in phase-shifting profilometry. We first estimate the motion of an object from the difference between two subsequent 3D frames. After that, by leveraging the projector’s pinhole model, we can determine the motion-induced phase shift error from the estimated motion. A generic phase-shifting algorithm considering phase shift error is then utilized to compute the phase. Experiments demonstrated that proposed algorithm effectively improved the measurement quality by compensating for the phase shift error introduced by rigid and nonrigid motion for a standard single-projector and single-camera digital fringe projection system.

"Three-dimensional shape measurement using a structured light system with dual projectors", Appl. Opt., (2018)

[113] C. Jiang, B. Lim and S. Zhang, "Three-dimensional shape measurement using a structured light system with dual projectors," Appl. Opt.,  57(14), 3983-3990(2018); doi:10.1364/AO.57.003983

This paper introduces a structured light system with two projectors and one camera for three-dimensional (3D) shape measurement to alleviate problems created by a single projector such as the shadow problem. In particular, we developed (1) a system calibration framework that can accurately calibrate each such camera-projector system; (2) a residual error correction method based on the system error function; and (3) a data fusion method utilizing the angle between the projection direction and surface normal. Experimental results demonstrate that the proposed dual-projector structured light system improves the measurement accuracy besides extending the measurement range of a single projector system.

"High-speed 3D shape measurement with structured light methods: a review," Opt. Laser Eng. (2018)

[111] S. Zhang, "High-speed 3D shape measurement with structured light methods: a review," Opt. Laser Eng. 106, 119-131 (2018)

Abstract

High-speed 3D shape measurement (or imaging) has seen tremendous growths over the past decades, especially the past few years due to the improved speed of computing devices and reduced costs of hardware components. 3D shape measurement technologies have started penetrating more into our daily lives than ever before with the recent release of iPhone X that has an built-in 3D sensor for Face ID, along with prior commercial success of inexpensive commercial sensors (e.g., Microsoft Kinect).  This paper overviews the primary state-of-the-art  3D shape measurement techniques based on structured light methods, especially those that could achieve high measurement speed and accuracy. The fundamental principles behind those technologies will be elucidated,  experimental results will be presented to demonstrate capabilities and/or limitations for those popular techniques, and finally present our perspectives on those remaining challenges to be conquered to make advanced 3D shape measurement techniques ubiquitous.