Reconstruction of High Resolution 3D Objects from Incomplete Images and 3D Information

— To this day, digital object reconstruction is a quite complex area that requires many techniques and novel approaches, in which high-resolution 3D objects present one of the biggest challenges. There are mainly two different methods that can be used to reconstruct high resolution objects and images: passive methods and active methods. This methods depend on the type of information available as input for modeling 3D objects. The passive methods use information contained in the images and the active methods make use of controlled light sources, such as lasers. The reconstruction of 3D objects is quite complex and there is no unique solution-The use of specific methodologies for the reconstruction of certain objects it’s also very common, such as human faces, molecular structures, etc. This paper proposes a novel hybrid methodology, composed by 10 phases that combine active and passive methods, using images and a laser in order to supplement the missing information and obtain better results in the 3D object reconstruction. Finally, the proposed methodology proved its efficiency in two complex topological complex objects


I. INTRODUCTION
HE object and image reconstruction is a very useful technique that is commonly applied in many areas, such as human faces recognition [1], quality production control [2], aerial images and maps [3], etc. The high-resolution 3D objects are one of the biggest challenges in the field of objects and image reconstruction methods. Overall, the reconstruction of high resolution 3D objects is an expensive and tedious process, since a user usually has to spend so much time on manual work [4]. The objects reconstruction technology has a very important role in archeology and ancient history; these disciplines often require simple, robust and cheap methods for scanning objects, so these can be studied. The poor state of some discovered antiques implies having complex topologies [5].
The 3-D object reconstruction is a procedure for recovering 3-D geometry and color information from an object [6].
Majorly, there are two kinds of methods to make digital reconstructions of 3D objects: the passive methods [7] and the active methods [8]. The passive methods make use of a set of digital images of the object and the active methods make use of data obtained by laser, which is used to scan the object. Each method has its own advantages and disadvantages, each object reconstruction poses its own problems and appropriate solution must be sought, depending on the requirements. The active methods are quite expensive in comparison to the passive methods, as they make use of controlled light sources, such as a laser scanner. The object data obtained with the active methods is usually very accurate and the reconstruction algorithms are generally computationally faster than those used in passive methods. Generally, these computational processes are responsible for surface constructions and data registration. In some cases, the configuration of these methods is not easy because the laser scanners require proper calibrations to obtain high quality data. Another aspect of these methods is that they are often not able to capture the actual texture of the physical object; it would need to be combined with the use of cameras. The brightness of objects can also be a problem when trying to scan the object with the laser, the light reflected from the object does not allow laser triangulation and therefore data occlusion is generated.
Passive methods can be used with less technical resources and different types of digital cameras. Recovering the 3D object's geometry by using passive methods involves solving problems related to the camera's calibration, occlusion, correspondence and data fusion. Nowadays, passive methods are widely used in reconstruction and recognition systems, and in many cases they can reconstruct objects with a high level of quality. Generally, these systems involve high computational costs [9] , as they obtain many data sets from digital images and these must be processed through several algorithms. As in active methods, passive methods can also have problems obtaining data shape of bright objects because the bright parts of the object generate texture degradation. Passive methods are very appropriate when the goal is to obtain a real object's reconstruction and is necessary to obtain its texture.
There are several methodologies based on passive or active methods; these methodologies often pursue different objectives, such as: improving the quality of the object reconstructions, improving the processing speed, reducing the computational cost of the process, reconstructions of objects in real time, etc. As noted above, both passive and active methods have advantages and disadvantages. This research aims to achieve a specific methodology in order to get high resolution reconstructions of topologically complex objects with great detail and with a relatively low computational cost, or at least lower than the cost involved in passive methods. The proposed method must be able to obtain the object texture in order to achieve a more realistic reconstruction. For the rebuilding of an object of great quality we will combine passive and active methods, the objective is to obtain the best data for the reconstruction of the object. The combination of two methods presents challenges, especially if we take into consideration the objectives of the proposed methodology. The need of applying two different methodologies increases the computational costs, which is a problem to be solved. The combination of active and passive methods in a unique methodology raises the number of challenges, is quite possible that the computational cost due to the application of the two methods is very high. This is a problem that the proposed methodology has to solve, combining the appropriate tasks and algorithms for data processing and reusing the object information and tool calibration.
The proposed method is composed by 10 phases that combine the use of high resolution images and laser scanner in order to supplement the missing information, the object's information obtained is analyzed with image reconstruction algorithms to obtain the best result for the 3D high resolution reconstruction of the object. In order to evaluate the effectiveness and quality of the proposed methodology we have applied it to the reconstruction of topologically complex, detailed objects. We intend to take advantage of the two types of reconstruction methods in order to achieve successful results. Although there is a proposed methodology which is very similar to the one being developed [10], the proposed method uses the NextEngine laser scanner and digital cameras with 13 Mpixels. This paper is divided into the following sections: Section 2 describes related work. Section 3 describes the proposed hybrid methodology for the reconstruction of 3D objects. In Section 4 we use the proposed methodology to reconstruct two topologically complex 3D objects. Then, in Section 5 we present the results of the application of the proposed methodology. Finally, Section 6 presents conclusions and future work.

II. RELATED WORK
Within the area of reconstruction of high resolution objects from data obtained from images (passive methods [7]) a wide range of solutions have been proposed, depending on the physical property we want to retrieve, such as Space Sculpting Algorithm, Set Level Algorithm, Graph Cut Algorithm, Visual Hull Algorithm [11], Stereo Fitting Algorithm [12,13,14,15]. Several proposals have been combining some of these classic algorithms to improve some aspects of the image reconstruction systems, the main goal of this has been to improve the quality of the reconstruction and the computational costs [8]. Although they are widely used, most of these algorithms do not obtain perfect reconstructions of 3D objects [16,17]. However, in some cases several solutions achieved high quality reconstructions of particular types of 3D objects, such as human faces [1], industrial parts, simple topological objects, etc. Either way, there is no general solution based on passive methods that is capable of making perfect reconstructions of any type of 3D object, especially in the case of topologically complex objects. Another very common drawback of these solutions is that, in order to slightly increase the quality of the object reconstruction, a great increase of the computation costs becomes inevitable [15]. Additionally, these methods which use digital cameras generate the need to control parameters such as light, color or movement.
There are several types of algorithms that are used commonly in passive methods for the reconstruction and recognition of objects. Some of the most relevant algorithms are: space sculpting, set level, graph cut, Visual hull and stereo fitting.
The space sculpting algorithm consists in initially creating a volume surrounding the surface of interest in order to be able to make iterations on it and reconstruct the shape of the object. In most cases, the refinement is performed under the photoconsistency conditions and starting from an initial volume represented by voxels. Therefore it's an algorithm that commonly uses the color information method and takes volumetric orientation. An example of such solution can be found in [1,18] The set level algorithm represents the surface as an initial set, which varies in time according to the volumetric density function defined. These algorithms are numerically stable and handle topology changes automatically. Several authors have worked on this type of solution, the most representative being Jean Pons, R.Keriven, and O. Faugeras [19], M. Habbecke and L.Kobbelt in [13].
The graph cut algorithm consists in making a representation of the object's volume in a graph structure with weight on the edges. The problem is finding the most efficient way to cut in a graph with the lowest cost. The weight indicates the discrepancies between images, in other words, it integrates the function of photo-consistency. Several authors have proposed solutions on this methodology, Kolmogorov and R. Zabih in their work of scene reconstruction via graph cuts [20], G. Vogiatzis P. Torr and R. Cipolla in their work of multi-view stereo via volumetric graph-cut [11], A. Hornung and L. Kobbelt in their work of hierarchical volumetric multi-view stereo reconstruction of manifold surfaces based on dual graph embedding [21].
Visual hull algorithm is defined as the intersection of all the possible cones containing the object. It was proposed by Laurentini [22] and several other authors which made use of this algorithm as an initial step to make refinements on it. [23,24] Visual hull is the maximal object shape that is consistent with the silhouettes of the object. Supposedly, it can be obtained by the intersection of back-projected visual cones of all the silhouettes [24].
The stereo fitting algorithm is designed to find the correspondence of objects between two images. With the use of this information we can find the fundamental matrix that relates the image space with the object space, and then reconstruct the object through triangulation. The following authors have worked with this type of solution: Y. Furukawa and J. Ponce [13]; and Derek Bradley, T. Boubekeur, and W. Heidrich in [25]. Depending on the physical property we want to analyze and characterize in the object, the restrictions and properties described in a method may be applied. There are three types of methods shown in figure 1: the pattern through shading, pattern through shape and the use of the color information.
Besides, the type of algorithm can be applied to other important factors that determine the quality of the reconstruction of the object. Pattern through the shading requires light calibration in order to assume the surface is Lambertian and make use of the properties this assumption implies. Pattern through the shape requires space calibration to allow the use of segmentation and use geometry properties to obtain a representation of the object's pattern. The color information allows photo-consistency verifications between images. The orientation of the algorithm depends on how it is generating the structure of the object. It may be by volumetry (voxel), or through representation by means of the construction of surfaces.
Within the area of reconstruction of high resolution objects from data obtained by scanners (active methods [8]), reconstruction algorithms are based on parametric surface orientation or implicit functions. [26] is an example of algorithms based on parametric surfaces (Delaunay triangulation) and [27] shows an example of algorithms based on implicit functions (signed functions followed by iso-surface extraction [28] ).
Active methods have proven their effectiveness and quality in many solutions, but due to the technical cost involved by its use they are not suitable for all scenarios, not always have the resources or can be used to that extent. There are several solutions using hybrid methods, based on a combination of passive and active methods. [29] Combines the active and passive vision and determines viewpoints in a similar way to human intuition, but in the words of the authors the recovered surface is not very accurate, so the solution is not very effective for reconstructing topologically complex objects.
The proposed solution aims to define a novel methodology to reconstruct objects with high quality, including texture mapping. To obtain the best possible quality data, this methodology merges data from active and passive methods. This methodology is based on the solutions that use volumetric techniques to merge laser scan data and digital images for the reconstruction of the surface. This approach will reduce the characteristic reconstruction errors of active and passive methods by merging data, which can be useful when reconstructing topologically complex objects with better quality. To achieve this, the proposed methodology has to solve the problem of computational cost derived from the combination of two methods, this can be done by combining the appropriate tasks and algorithms for data processing and reusing the object's information. Also, if the information is not reused properly, the calibrations in the cameras and the scanner can also mean a great loss of efficiency. For this purposes, the proposal introduces a new approach to reduce the complexity presented by the scanner's laser calibrations, it uses the box enclosing the 3D object convex hull to register the models. Finally, the geometric 3D object is calibrated through the high resolution shapes, obtained from photography, to make texture mapping as realistic as possible.

III. HYBRID METHODOLOGY FOR THE RECONSTRUCTION OF 3D OBJECTS
As mentioned above, the proposed methodology uses both the data obtained from digital images and the data acquired via laser scanner to retrieve the geometric information of the object. This methodology is divided in two stages. The first one would be to use the digital image information alone to generate a reconstruction of the initial model, and the second one is to register both models: the one obtained in the first stage and the object reconstructed by the laser scanner, in order to make a refinement and find the final volume that best represents the 3D object we want to rebuild. The first phase, reconstruction from images, uses space sculpting algorithm, pattern methods through shape and volumetric orientation. Figure 1 shows in detail the methodology and each of its steps. The first stage consists of phases 1 to 7, and the second stage comprises the phases of 8 to 10.
We will describe the tasks involved in each phase of the proposed methodology below.

A. STAGE 1 1) Phase 1: Data Acquisition
The objective of this phase is to collect information concerning the object, mainly its shape, but also the calibration pattern for each object view. There are some physical factors that can affect this data collection, one of them is the brightness of the object. Bright objects, like shiny metal objects are difficult to reconstruct with either the active or the passive method. In the active method, the light reflected from the object does not allow laser triangulation and, therefore, data occlusion is generated; in the passive method, the bright parts of the object generate texture degradation.
In this first stage we must minimize the effect of light in order to obtain acceptable images to be processed in the following phases. We have developed a photo-montage with the following elements: Light box (own production), professional lights, a semi-professional digital camera, a revolving board and a computer. Figure 2 shows the montage developed for this phase of data acquisition. The revolving board allows us to obtain up to 36 views of the object. The lights are calibrated with the light box and the digital camera to minimize the reflection of light on the object as much as possible. The pictures have been taken from the computer in order to obtain more stability and consistency between images.
The background has been arranged as uniformly as possible (white color). However, shadows and dark lines that may affect the process of thresholding of the images can still appear As noted, this phase has the object and the calibration pattern as inputs. The object is photographed in different views and afterwards a thresholding pre-processing is performed to obtain the shape of the object. In practice, many image-editing software can be used to obtain an accurate shape of the object. The calibration pattern corresponds to a table with black and white squares, the result of the pictures taken in each view of the object.
As shown, this first phase has two types of data sets as output. The first data set corresponds to images containing shapes of the object and the second data set corresponds to images with the calibration pattern for the object in each view.

2) Phase 2: Data Calibration
In order to calibrate data we can use a specific open source computer vision software or commercial software. In this case we used Matlab Toolbox [30] developed by Jean-Yves Bouquet. Matlab toolbox allows us to find both the intrinsic and extrinsic parameters of the camera quickly and accurately. In addition, intermediate results can be seen graphically and projections can be created on the calibration to minimize the errors. The output of this phase is the projection matrix M3x4 for each view of the object. The projection matrix is: Where K is the matrix, containing the intrinsic parameters of the camera, R is the rotation matrix and T is the translation vector of the object in relation to the camera.

3) Phase 3: Minimum Box Surround
In order to speed up the computational processing of data we must know the minimum work space.
From [18], "the 2D projections of N boxes define a 3D convex hull formed by 4N planes". The minimum 3D box that encloses the 3D convex hull can be calculated if posed as an optimization problem where the objective function becomes one of the 6 variables that define the 3D box we want to recover under the constraints of each 2D plane of the box. The restrictions imposed by each 2D plane of the box are: We can find the six variables that define the 3D box by using the simplex method. In practice, it is necessary to calculate the 2D convex hull of the object in each of the images to find the box enclosing the 2D object, and then project the sides of the box (lines) to find the equation of the 3D plane.

4) Phase 4: Visual Hull
There are several ways to construct the visual hull of the 3D object depending on several factors: the algorithm, the method and approach used, etc. In this method we will use volume-oriented algorithms. In order to do so we have decided to represent the object with octree structure.
The structure of the octree is described in [31]. The advantage of using this structure is that it uses little memory as it is described by a string of '1's, '0's, '('s that represent the position of each cube (inside, outside and above respectively). Moreover, the order of the chain is due to the pre-order path in the tree. In order to generate the octree we first take a cube, we project it in all images and we check the iso-level function's value to determine the position of the cube. If the position is 'inside', the cube is drawn, if it is 'outside' it is not drawn, and if it is 'above' it must be subdivided in eight cubes. This process continues recursively projecting and determining positions to smaller and smaller cubes until we reach the degree of resolution that is wanted. In this case, the digital images have a resolution of 2008x3040 pixels; the recommended depth level of the tree will be 8 or 9.
To define the iso-level function the images that match the shapes must be encoded so the background has a value (e.g. -0.5) and the figure has a value with opposite sign (e.g. 0.5). This way, the boundary of the given object will have an intermediate value, 0. Using these values, the iso-level function would be: is the value taken by the projection of the 3D point in the 2D image, depending on whether it is inside or outside the silhouette or not.
In order to determine the status of the cube we must compare the iso-level values from the eight points of the cube. If all of them were negative, it would mean that the cube is outside the valid volume range for the model. If all of them were positive, the cube would be within the valid range, but if we'd find both positive and negative values, we wouldn't be able to validate it with certainty, and we'd must therefore continue subdividing.
In practice, we should project points that are not only at the vertex but on the edges of the cube, in order to avoid sampling problems and to be able to determine the position of the cube correctly.

5) Phase 5: Marching cubes.
Once we have obtained the volumetric representation of the object, phases 5 Marching cubes, 6 Iso-surface and 7 Simplification are applied to obtain the mesh that best represents the surface of the object. The marching cubes algorithm takes the iso-level values containing the vertexes of the cube and determines the points cutting the isosurface to generate the corresponding mesh configuration. As they are cubes, there are 256 meshing possibilities that can be reduced to 15. Once the mesh in each cube has been created, they are merged in order to create the surface that represents the object.

6) Phase 6: Iso-surface & Phase 7: Simplification
In the Phase 6, the iso-surface for the proposed algorithm corresponds with values equal to zero. Then we proceed to apply the phase 7: Simplification. Once the mesh has been obtained, the next step is to go through an optimization process by applying mesh simplification algorithms. These are algorithms of decimation and our intention is to maintain the shape of the surface for a smaller quantity of sampled data.

B. STAGE 2
After the first stage, based on a passive method, the second stage began. At this stage the methodology focuses on data obtained using laser scanner. In order to reduce the computational cost during this stage we use data obtained from the preceding stage.

1) Phase 7: Minimum bounding box
This phase consists in the calculation of the scanned object's workspace. As the object is already a 3D object and is also aligned with the coordinate system so it doesn't present inclination along any axis, the minimum containing box is determined by finding the six variables defining the cube through simple minimum and maximum functions.

2) Phase 8: Model Registration
At this stage we aim to register the two geometric models obtained through the two methods we have used so far: passive and active. As the object that has been reconstructed by passive methods is being calibrated, we want to transport the model obtained by the laser scanner to the coordinate system of the model obtained from the digital image data.
The common way to register models is to calibrate the laser, using methods like the one described in [32], and then apply ICP algorithms (iterative closest point) and find the rotation and translation matrix that relates both models. However, due to the way the acquisition of data has been performed; this paper suggests that both models are brought to register through the minimum box enclosing the object.
The registration process from the bounding boxes can be a manual process in practice, but significantly reduces the complexity of the problem, bearing in mind that the models have been reconstructed by different means and have considerable variations.

3) Phase 9: Reconstruction of the final model
Once we have brought the two objects to register we proceed to generate the final surface of the object detailed and free of gaps.
The idea of this algorithm is to take the visual hull represented by the octree and redefine the states of the cube depending on the intersection of the line defined by the laser spot and the source point of the camera. This algorithm makes sculpting in 3D space on the volume defined by the octree guided by 3D points obtained using the laser scanner. In order to generate the model correct, several cases must be considered, such as the one described in [10].
Each octree cube which is 'inside' or 'on' changes to 'off' if the ray defined by the 3D point and the origin of the camera intersects the surface. Finally, the marching cubes algorithm is applied in order to reconstruct the final, gap-free volume of the object.

4) Phase 10: Texture mapping
To offer realistic 3D digital objects, these should include textures. The textures we used were collected from digital images. As the data is calibrated, texture mapping can be done with any algorithm that may be developed for the purpose. In this instance we are proposing a mapping based on particles such as in [33,34]. After phase 10 the reconstruction process of the 3D object was successfully completed. As an output from the application of the methodology, a high definition 3D reconstruction of the object is obtained. This reconstruction has a high degree of quality and texture mapping.

IV. APPLICATION OF THE PROPOSED METODOLOGY IN A REAL SCENARIO
To validate the objectives set at the beginning of this research and illustrate the use of the proposed methodology we used it to reconstruct topologically complex 3D objects. The objects involved in the reconstruction process are two replicas of archaeological objects from the Gold Museum in Bogota, Colombia. We choose these objects because of its morphological properties, great detail and brightness management (Phase 1 & 2). Figures 3a and 3b shows the images of the objects we want to rebuild.    For the visual hull's phase we define the structure of the octree given in [31]. The right part of Figure 5 shows the simulation results for the octree represented on the left side of the figure. The results of the octree for the replica from the gold museum are shown in Figure 6. Figure 7 shows the results of phases: 5 Machine Cubes, 6 Iso-surface and 7 Mesh Simplification.  For the second stage we used the NextEngine 3D desktop scanner. The data obtained by the laser scanner is shown in Figure 8. As shown, the objects have some deficiencies in the reconstruction. Then we proceed to implement Phase 8 of the methodology: Model Registration, bringing the two models into register, with the help of the minimum encasing boxes that contain the object, as shown in figure 9. In practice, the boxes do not have the same dimensions and the objects must be brought into register manually. However, they are close enough to apply registration algorithms such as ICP (iterative closest point) so that the object is eventually registered.  Finally, in the Phase 10: Texture Mapping we must obtain the merged geometric model and make a photorealistic texture mapping. An approximation of the final geometry of the 3D objects is shown in Figure 10. After applying the proposed methodology we have successfully managed to build two morphologically complex 3D objects, obtaining a high quality reconstruction of the object, with a detail level significantly higher than those obtained by using passive methods.

V. PROPOSAL EVALUATION
The proposed methodology has been previously used in the reconstruction of two morphologically complex objects. This evaluation examines the relevant aspects related to the objectives set at the beginning of this research.
One of the objectives of the methodology was to obtain quality reconstructions from objects that are difficult to reconstruct, like topologically complex objects with shiny surfaces. To verify that the quality of the reconstructions has increased using the proposed methodology we will compare it with a passive method. A parameter that is often used to validate the quality of an object's reconstruction is the number of errors in the topology generated during the reconstruction. As an object is topologically more complex needs a larger number of voxels to be reconstructed, when more voxels are used the potential for reconstruction errors increases. The error relationship between the two methods is shown in Figure 11, observing results it is possible to detect a function with an associated trend to the occurrence of errors depending on to the number of voxels of the object to be reconstructed.
The combination of two methods would increase the computational cost. The other major objective of this research was to achieve an adequate computational cost, which did not exceed the computational cost of a passive methodology. To achieve this requirement the proposed methodology reuses the object data captured and the calibrations calculated at different phases of the process. The trend of error occurrence in the calibration as a function of the number of voxels is potential with a coefficient of determination of 0.94. Second, calculate the relationship between the time spent on the reconstruction process of the 3D object and the object's voxels. In the graph of Figure 11b the comparison of the proposed methodology with a passive one is shown  The tendency of the processing time as a function of the number of voxels is polynomial and has a coefficient of determination of 0.99. As a result of this analysis it can be concluded that the proposed methodology improves in several aspects respect passive methods for reconstructing 3D objects, being able to reduce significantly the number of errors and also reducing the processing time used. The improvement of these features is complemented by a more optimal use of the technical resources that enable active methodologies, such as laser scanners.

VI. CONCLUSIONS AND FUTURE WORK
In this paper we propose a novel hybrid methodology for reconstruction of high resolution 3D objects. These objects are also topologically complex and can have bright parts. The methodology is composed by 10 phases that combine active and passive methods, using data obtained from digital images and a laser scanner in order to supplement the missing information and obtain better results in the 3D object reconstruction. The proposed methodology takes advantage of the passive and active methods of object reconstruction, using the level of detail obtained with the laser scanner and the topology recovered by the digital images method. As it has been demonstrated in the Section 5 with the combination method, the proposed methodology has improved the quality of the reconstruction by significantly reducing the number of errors depending on the object's voxels. The proposed methodology solved the problems derived from the need to handle large volumes of object data from multiple methods and the need for different tool calibrations. In the different phases of the proposed methodology, several specific techniques have been applied to reuse object data, calibrations and optimize the computation cost.
Also, we observed that small scanner laser errors generate large errors in the reconstruction of the object. For this purposes the methodology introduces a new approach to reduce the complexity presented by the scanner's laser calibrations, it uses the box enclosing the 3D object convex hull to register the models. The use of digital images in this methodology allows us to obtain the texture-mapping of the object, making it well suited to achieve realistic object reconstructions. The geometric 3D object is calibrated with the high resolution shapes, obtained from photography, to make texture mapping as realistic as possible.
There are many directions for further research. The methodology can be modified at some phase to guide their tasks and algorithms towards achieving better quality and efficiency in the reconstruction of a specific type of topological complex objects, such as industrial material. A modification of the methodology it's being evaluated in order to optimize it for use it in real-time objects recognition under unfavorable circumstances for active recognition.