If we look at any simple scene surrounding us, such as the photography in Figure 1.3, we can find a plethora of optical interactions: indirect lighting (color bleeding), internal scattering in translucent objects, caustics, anisotropic and glossy reflections, etc. Far from the traditional assumptions in intrinsic imaging of diffuse (lambertian) shading, direct lighting, and diffuse albedo materials.

In the following, we provide an overview of the theoretical background behind the image formation model, its derivation for non-diffuse materials, and the link with physically-based and inverse rendering. For a deeper dive into any of the concepts quite briefly described below, we recommend reading the book on physically based rendering by Pharr et al. [PJH20] and the survey by Guarnera et al. [Gua+19].

The color and luminosity at any point of an image, the incoming irradiance, is proportional to the sum of the outgoing radiance from all the visible points of the scene towards the camera sensor at the corresponding pixel, I, resulting from multiple interactions between light and matter in the scene. Naturally, this is a simplification: even if we consider the camera lenses and color filters as part of the scene, the interaction of irradiance and the sensor point affects the result, and both electronic and film cameras have specific additional image formation steps which can be simulated. In physically-based rendering, the general approach to compute this value is to use Monte Carlo estimators of the pixel and shading integral [Kaj86], which has the general form of:

(1.1) I=∫χf(x,Θ)dx

where f is a function that defines the radiance towards pixel I and is defined on a domain χ, generally a unit sphere, or the set of all the surfaces (A) in the scene, and depends on the scene parameters Θ, which might include the definition of the geometry (normals, z-depth, vertices), material (albedo, BRDF), or illumination sources ś far-field environment lighting, or 3D light emitters (point, area, objects).

The value of I is defined for a given λ, which is the spectral band of the camera sensor. We could define it for as many bands as desired (including non-visible ones) but in practice, the majority of camera sensors mimic the human visual system and are commonly three-band: λ∈{R,G,B}. In subsequent rendering equations, we will simplify the notation by assuming a single-band, and omitting the term λ.

Equation 1.1 is an integral of integrals (see Figure 1.4): to account for all the light arriving at a surface point p₁ to a sensor pixel at p₀, we have to estimate all the contributions of light from all the surfaces of the scene, recursively tracing paths bouncing in surfaces (p2,p3…pn) until we reach the light emitted by a source Le(pn→pn−1). This is referred as the Light Transport Equation (LTE) in rendering and it is another way of seeing the equation 1.1. To compute the radiance reaching the pixel I, that is from p₁ to p₀ in Figure 1.4, we would need to solve:

(1.2) L(p1→p0)=∑k=1∞P(p¯k)

with p¯k being the radiance scattered over a path p¯k with n + 1 vertices (p0,p1,p2,p3…pn) and computed as:

(1.3) P(p¯k)=∫A∫A…∫A⏟n−1Le(pn→pn−1)T(p¯k)dA(p2)…dA(pn)

We can integrate over solid angles in the unit sphere, or surfaces (A) of the scene, being dA(pi) the differential area at point p_i. Note the term T(p¯k), named throughput of the path: the fraction of radiance from the light source that arrives at the camera after all of the scattering at vertices between them. The total transmitted energy will be reduced at each interaction event, as some wavelengths (colors) are absorbed or scattered away from the observer. This is a common trade-off decision in many Monte Carlo rendering engines; longer paths per sample are costly to compute, while contributing less and less energy with each additional vertex, but in some scenes, they might be very relevant to reduce variance (noise) and converge with fewer samples to an accurate image.

If there are interactions with non-opaque materials (human skin, cloth, marble) or participating media, such as liquids or smoke, we have to integrate volumetric scattering interactions of photons along the path between the light sources and the camera sensor pixel, requiring a more complex mathematical model such as the Radiative Transport Equation (RTE).

If we take a look at Figure 1.4, we can observe that materials are distributed on discrete 3D objects: the table and the cup, and even the light source if we consider it as an emissive material. We can assume that one of the most important Θ parameters is geometry, usually in the form of 3D vertices and edges, normals (N), or depth maps (D). Please note that, in contrast to a a full 3D mesh, a single camera-view depth image (a.k.a. Z-buffer) is an incomplete definition because the non-visible surfaces are undefined, and the reflected light paths cannot be traced behind the visible objects. For example, Nimier et al.[Nim+19] require multiple views of a smoke volume in order to reconstruct its 3D density distribution by inverse rendering.

There are additional ways of defining this geometry, such as implicit surfaces, but the material distribution is more complex when there are no clear surface boundaries. That is the case of heterogeneous participating materials: human skin, airlight, mixed liquids, smoke, etc., where light transport between two points has to be in turn evaluated along the path to account for all the possible scattering effects. For instance, imagine a small cloud of vapor between points p₁ and p₂ in Figure 1.4, at each infinitesimal step along the path between those points, there will be a possibility of absorption (collision with a water particle) that will reduce the energy, but there is also a possibility of receiving incoming energy (not emitted by p₂), as the cloud itself is receiving direct illumination from the light source and multiple scattering events distribute the light across its volume.

In order to compute the rendering equation in volumetric media, a distribution of parameters is required at any point of the space, not only on the 3D surfaces. The usual representations include solid 3D implicit functions (for instance 3D Perlin noise, used in cloud procedural generation), meshes and distance fields, or discrete volumetric grids which store the scattering probability function at any point of the space (also known as phase function) by means of voxels.

Each time the light interacts with a material, there is a loss of energy and a transformation of the original wavelength reflected towards the observed direction. In rendering, the result depends on the intrinsic material response for those two angles: incident light and viewing direction (e.g.: the camera, or another element of the scene). For surfaces, this response is modeled with a Bidirectional Reflectance Distribution Function (BRDF) fr(x,ωi,ωo) which yields at each particular 3D surface point x, and for each incident direction ωi, the fraction of reflected radiance observed from a direction ωo.

The total reflected radiance L at any point x can be obtained by integrating with Equation 1.4 over the positive hemisphere Ω⁺, to sample the whole incident light attenuated by the cosine term (dot product between the incident light direction L_i and the normal of the surface) [Kaj86]:

(1.4) L(x,ωo)=∫Ω+fr(x,ωi,ωo)Li(x,ωi)(ωi⋅N)dωi

Note that by integrating the computed radiance L(x,ωo) of the points sampled from p₀ at the camera sensor (x=p1,ωo=p1−p0, see Figure 1.4), we are obtaining the irradiance at the sensor and the corresponding image pixel values (I in equation 1.1). Naturally, even the pixels themselves can be sampled several times and integrated over the camera sensor with another Monte Carlo estimator to minimize aliasing effects.

The BRDF can be extended with a Bidirectional Transmittance Distribution Function (BTDF) to conform a full Bidirectional Scattering Function (BSDF), defined in the full sphere Ω. The model can be further extended to account for Surface Scattering phenomena (BSSDF).

These functions have a minimum of four dimensions (input-output pair directions in polar coordinates) and usually three RGB values as output. It is thus technically possible to choose a discrete set of (ωi,ωo) orientations and create a lookup table to interpolate the response of the material, which is captured with multiple light and camera positions (e.g.with a gonioreflectometer). Storage becomes a major drawback for tabulated data, which can only be reduced through a significant reduction of quality. Moreover, we are considering only homogeneous surface materials, which is not often the case in actual scenes (e.g., a printed paper). Those spatially-varying values (SVBRDF) can be stored in a stack of textures; or a Bidirectional Distribution Texture Function (BTF), increasing the dimensions and size of the table.

Beyond direct compression techniques, the most successful approach in graphics has been the use of analytic N-dimensional functions to approximate the reflectance and scattering distributions. The simplest of them, Lambertian diffuse, and Phong specular shading are also well known in the computer vision community. These functions leverage symmetry (isotropy) to model the material with a few parameters. For instance, a Lambertian material only requires knowing the intrinsic albedo, a sort of base color, while the Phong model requires three additional parameters for the specular component (e.g., shininess).

In the following subsections, we review the most common and simple material assumptions used in recent papers, finalizing with the most sophisticated inverse material models which are starting to be studied in computer graphics.

The illumination is a significant contributor to the shading term (S) in most decompositions. From a rendering perspective, as shown in Equation 1.15, the computation of the pixel radiance, L, requires considering both the emitters L_e and the irradiance: the integral of all the incoming lighting at the observed point. This incoming illumination is often neglected, considering only point light or directional analytic emitters, which simplify the shading computation by removing L_i from the integral. If f_r is also assumed to be Lambertian, only the form factor given by the cosine of the surface normal, and the light direction remains.

(1.15) L(x,ωo)=Le(x,ωo)+∫Ω+fr(x,ωi,ωo)Li(x,ωi)max(ωi⋅N,0)dωi

In actual scenes, the incoming lighting is a combination of emitted or reflected illumination from distant objects (far field) and local surfaces close to the observed area (near field). The former is usually approximated in computer graphics with environment lighting, often based in High-Dynamic-Range (HDR) images mapped into an infinite sphere or cube surrounding the scene, while the latter can be derived from the far field illumination, by simulating the local secondary light bounces. If the geometry does not change, an environment map can be stored at multiple scene locations and distances, to include near field effects more accurately (Spatially Varying Environment Maps), although at a great memory cost, and only for static scenes.

To reduce the sampling and size of environment maps and simplify the computation, Ramamoorthi and Hanrahan [RH01a] proposed their compression with Spherical Harmonics (SH), a set of orthonormal basis functions defined on the spherical domain (elevation θ and azimuth ϕ angles). Thus the equation 1.16 describes the irradiance E as the sum of bases weighted by the cosine decay term A_θ and the illumination coefficient L_θ,. By changing the representation of the bases Y^ to polynomial coordinates of a unit normal n=(x,y,z)T, this becomes an efficient vector dot product operation (Equation 1.17). With the required modifications, this strategy is feasible with other orthogonal basis functions on the sphere.

(1.16) E(θ,ϕ)=∑l,mA^lLl,mYl,m(θ,ϕ)

(1.17) E=Y^TL

(1.18) E=TTL

If we want to account for near field occlusion and interreflection, it is possible to precompute those local interactions, because they depend on the object geometry and materials, and not on far field illumination. This family of techniques is known as precomputed radiance transfer (PRT) [SKS02]: they precompute multiple events of light transport (see Figure 1.4) into the T term in Equation 1.18 with Monte Carlo pathtracing. In this fashion, each pixel will have secondary light bounces stored in a light transport map. If only the visibility term V (w_i) is considered for, the method will be storing the ambient occlusion shadows, but not colored interreflections.

In Figure 1.6, we can see a pyramid of spherical harmonics bases Y_θ, with different coefficients. It is important to know that, although usually nine bases are considered enough to account for 99% of the far-field irradiance at diffuse surfaces, this percentage is significantly smaller in glossy surfaces (requiring many more coefficients). Moreover, a small number of coefficients will never account for high frequency effects, such as cast shadows from high-frequency light sources (E.g: a point light representing the sun), even producing ringing artifacts if we try to increase the accuracy by adding more bases. There are other popular basis in rendering such as Haar Wavelets or Spherical Gaussians (SG) [Wan+09], which also have very interesting properties.

Accurate and high resolution material reflectance digitization is a crucial problem for generating compelling and realistic virtual environments. As we have mentioned in Section 1.1.3, this digitization process is typically done using a gonioreflectometer, a highly complex machine designed to measure the photometric response of a material using light sources and cameras placed at different positions around the hemisphere. Existing devices for measuring material appearance in spatially-varying samples are limited to a single scale, either micro or mesoscopic. This is a practical limitation when the material has a complex multi-scale structure, like is common on many real-world materials. In many materials, accurately measuring their spatially-varying reflectance requires high resolution microscopic images. For example, the optical behavior of textiles is very dependent on small fibers or their yarn twist [CLA19]. Moreover, many materials also show large-scale mesoscopic variations, like prints, plaid, or tartan patterns, which cannot be captured using microscopic photographs. Further, many of such devices cannot capture important optical properties, like the material transmittance, which is key for the realistic rendering of objects like foliage or fabrics. There are no available capture devices or algorithms for accurate and high resolution SVBSDF digitization of heterogeneous materials. In order to introduce such materials, which are very common in many industries like fashion, textile, or leather manufacturing, into rendering pipelines, these machines need to be created and accurate algorithms need to be developed to handle the data they capture. Finally, such a device may serve as a ground truth data generation source of training models that operate on more limited data scenarios, like in single image digitization

Tileable SVBRDFs Even if reflectance is accurately measured for a large portion of a material, in rendering settings, it is also important that the digital material is tileable. A tileable texture, also known as a seamless texture, is a texture image that can be repeated infinitely in all directions without any visible seams or discontinuities. In other words, a tileable texture is designed in a way that it seamlessly connects with identical copies of itself, which can be arranged side-by-side or stacked on top of one another to create a larger, continuous pattern without any visible repetition. This is often used in computer graphics, especially for creating textures on 3D models, or even for tiling backgrounds on websites or applications. We illustrate the importance of tileable SVBRDFs in Figure 1.8. Recent progress has been made in generating tileable textures from a single input example [Mor+17; Rod+19; Mor+20], however, these works present several shortcomings for tileable SVBRDF generation. First, they either assume a particular level of regularity, or the generated textures lose a significant amount of visual fidelity with respect to the input exemplars. Further, these methods are limited to generating a single texture map (i.e. the albedo of the material). This is problematic for tileable SVBRDF synthesis, which requires transforming every texture map in the SVBRDF into a tileable map, all while preserving pixel-wise coherence between maps. All in all, generating high-quality tileable SVBRDFs from a single input example is not currently possible with the methods proposed in previous work.

Material Attribute Transfer Effective material representations require understanding the properties that uniquely define them, which we refer to as visual material attributes. These attributes are spatially-varying parameters that maintain spatial coherency with respect to the material structure, while remaining invariant to changes in the scene illumination or the geometry of the underlying object. For example, they may represent optical properties of a SVBRDF, artistic stylizations or higher-level properties, as in semantic segmentation masks. Obtaining these attributes for large samples of the material is very challenging. One way to address this problem is to obtain them at a small exemplar of the material, and transferring them into larger portions of it, using a large input image which serves as guidance. Different visual attribute transfer methods have been proposed in the past, however, they are limited in their efficiency, robustness, generalization or overall performance. Ideally, such a transfer method should be robust to new material illumination conditions, camera degradations or input distortions; as well as being efficient (providing interactive transfer times even in low computational budgets), controllable and predictable, and capable of generating high resolution outputs, which are required for realistic SVBRDFs. Developing a method with these characteristics may prove useful for many downstream tasks, like efficient data generation or material digitization pipelines.

Material Reflectance Estimation As mentioned, accurate material digitization typically requires expensive devices which take significant amounts of time and manual input to operate. One such case is the capture machine illustrated in Figure 1.7, which is presented in [Gar+23]. These provide highly realistic digital representations of material reflectance at the cost of scalability, hindering their applicability in many real world settings. For instance, many manufacturing processes, like those in the textile industry, generate a massive variety of different materials, which cannot be easily nor cheaply digitized with the required cadence that characterizes these industries. Further, the limited availability of these devices introduces additional inefficiencies and economic and ecological costs, as physical samples of the materials must be shipped from the client to the institution holding the capture device.

Data-driven material reflectance estimation from low-cost devices is a long-standing problem in the literature [AWL13; AWL15; Des+18; Hen+21; Guo+20b; VPS21]. These methods promise to provide scalable, efficient, inexpensive on-site material digitization. They typically rely on one or more flash-lit images of a material taken with a smartphone camera, and a deep learning model trained on synthetic materials, which outputs plausible estimations. However, these approaches present several drawbacks that make them unsuitable for practical digitization workflows: First, many of such methods rely on generative models which tend to produce artifacts, or use large neural networks which puts limits on the output resolution, hindering the quality of the estimated materials. Besides, smartphone flash-lit images introduce significant calibration challenges; while most existing datasets used to train these models are purely synthetic, further limiting the quality of these estimations. Moreover, many of these methods use perceptual losses for training or in test-time optimization, which, while they help digitize stochastic materials, present additional challenges for guaranteeing the repeatability and consistency required for building a digital inventory. On top of these limitations, few-shot material reflectance estimation is still an ill-posed problem, as such, many different outputs may be plausible given the same input. So far, there is no way to quantify uncertainty for this problem, which hinders the applicability of these systems in real-world scenarios, and limits their trustworthiness, efficiency, and reliability.

Fabric Mechanical Behaviour Besides material reflectance, another important component of scenes is object geometry. While many real-world objects have static shapes, which can be scanned with relatively accessible devices like the LiDAR sensor in iPhones, there are materials that change shape when external forces are applied to them, like gravity. In this sense, deformable materials require particular computational models and capture pipelines. One of the most common of such materials is fabrics, which are ubiquitous in the real world, and used in our clothing, furniture, or vehicles. For them, accurately capturing their static shape is not enough: We need to be able to simulate their behavior on new scenarios, like new garments or dynamic forces, like wind [Ber+23] or human movement [SOC22]. Furthermore, textiles are incredibly varied in their fabrication patterns (eg different weave patterns or knitted structures), compositions, and finishing, which not only determine their optical appearance [CLA19] but also their mechanical behavior, as shown in Figure 1.9. The problem of capturing fabric mechanical properties so they can be re-simulated in virtual settings has been amply studied in the literature, however, current solutions require tedious human intervention or expensive capture devices, hindering their scalability. Further, there is a lack of understanding of the perceptual similarity of fabric mechanical behavior, which creates additional challenges for designing accurate fabric mechanical parameter estimation pipelines.

Reflectance In Section 1.1.3, we have mentioned several limitations of current representations of material reflectance. In particular, Bidirectional Texture Functions (BTFs) provide dense and accurate material reflectance measurements, however, they are typically prohibitive in terms of computational and memory cost. Recent work [Rai+19; Rai+20; Kuz+21] introduce different neural compression algorithms for BTFs, achieving remarkable realism and efficiency. Neural reflectance encoding methods trained on either synthetic or measured BTFs have thus shown promising results for increased realism of rendered materials. However, existing neural material encodings are immutable, meaning that their output for a certain query of UVs, camera, and light vector is fixed once they are trained. This can become very limiting when the fragment of the material used for training is too small or not tileable, which frequently happens when the material has been scanned with a capture device. Therefore, current neural reflectance encodings, while accurate and efficient, have severe limitations in terms of editing capabilities, which hinder their applicability and usefulness in real world rendering settings.

Illumination Finally, as we mention in Section 1.1.4, there are many available approximations for representing real-world illumination in virtual scenes, including Spherical Harmonics, Spherical Gaussians, or Haar Wavelets. These, however, struggle to efficiently and accurately represent non-diffuse illumination conditions, like high-frequency light sources, as the sun or light bulbs. Recent work on neural representations of natural illumination [GES22] provides better approximations for such cases, but struggle with indoor lighting or night scenes with multiple light sources. Furthermore, these neural approximations do not provide sampling and PDF evaluation capabilities, which are essential for Monte Carlo rendering in path tracing. To the best of our knowledge, there are no lighting approximation methods that provide these sampling capabilities, while working accurately on any type of input environment map. Recent work on invertible generative models and implicit neural representations may provide a promising pathway to achieve these objectives.

The work developed in this thesis has led to the following contributions to achieve the goals presented above:

Each of these contributions has been developed in collaboration with other researchers and engineers, many of whom are listed as authors of the publications listed below. However, most of this thesis would not have been possible without the outstanding work of the whole team at SEDDI, which has played roles in important tasks, including administrative work, dataset creation, financial support, help with software and infrastructure, as well as valuable scientific discussions. My individual contributions to each project can be inferred from my relative position in the author list.

We include here a list of awards and other merits received throughout the development of this thesis.

During the development of this thesis, I have co-supervised the bachelor’s final thesis of four computer science students:

During the first half of 2023, I worked as an adjunct lecturer at Universidad Carlos III in Madrid, where I taught and led a mandatory graduate course on Intelligent Data Analysis, at the MSc in Computer Engineering.

Throughout my Ph.D., I had the opportunity to disseminate my work across different formal and informal events. I presented some of the work in this Ph.D. thesis on different venues, including:

During the development of this thesis, I have served as a reviewer at multiple conferences, workshops, and journals on computer vision and machine learning:

This thesis has been developed within an industrial Ph.D. in SEDDI, a startup based in Madrid, Spain, which has the goal of providing deep tech solutions to the textile and fashion industries. A significant part of the work presented in this thesis is part of commercially-available products and industry research projects. Notably, some of the algorithms (or variations thereof) presented in Chapters 2, 2.C and 5 are part of Textura.ai, an AI-powered commercially available material digitization product. Further, the drape similarity metric presented in Chapter 6 has also been a part of a project developed for a textile manufacturer.

Visual attribute transfer refers to the problem of transferring some visual attributes (e.g. color, style, texture, or geometry) of one or many exemplars to another exemplar while preserving its content.

This problem can be formulated within the context of Image Analogies [Her+01], in which the goal is to stylize a target un-stylized image B, given a pair of images A (un-stylized) and A’ (stylized). The most common approach to tackle this problem has been via patch-based texture synthesis [Bén+13; Bar+15; Jam+19]. More recent approaches have leveraged the capabilities of deep latent spaces within convolutional neural networks to disentangle style from content [Ree+15]. A seminal work by Gatys et al.[GEB15b] utilizes a VGG-19 convolu-tional neural network [SZ15] pre-trained on ImageNet [Den+09] as a feature descriptor for images, in which style and content are associated with different layers of the network, and transferred by gradient descent optimization. Their work on style transfer has been extended for single images [HB17; Li+17b; JAF16; Che+17b] and video [Che+17a], as well as for developing image-space distance metrics which resemble human perception [Zha+18]. A shared limitation of many of these methods is their narrow capabilities to provide predictable edits, and considerable focus nowadays is put towards this end [Gat+17; Gu+18]. A comprehensive review on the topic of neural style transfer is provided by Jing et al. [Jin+19]. Our work differs from traditional style transfer approaches in the sense that we deal with a more constrained problem that requires predictable outcomes.

Exploiting the power of deep neural networks in the image analogies problem was tackled by Liao et al. [Lia+17] who, by assuming a semantic prior over an exemplar input image and a target one, propose a method capable of finding a bijective mapping between both inputs, enabling two-way stylizations. Single-image generative models [SDM19] were extended to the image analogies problem in [Ben+21], by using convolutional neural networks to generate a new image with the style of an input style and the structure of another image. These methods, however, rely heavily on content or semantic features, making them vulnerable to lighting or geometric differences between the input images; and are computationally expensive, rendering them impractical for interactive applications. Our method is robust to both geometric distortions and illumination variations, and works at interactive rates. Similar in spirit to our method, as it explicitly considers texture variations due to illumination, is the work of Fišer et al. [Fiš+16], which applies patch-match to provide illumination-dependent exemplar-based stylizations to cartoon pictures. In contrast, our work is meant to be illumination-invariant. Also concerned with stylization problems, Texler et al. [Tex+20b] present a method highly related to ours. They apply patch-based training of an encoder-decoder deep neural network using key frames stylized by a user. Resembling our approach, their algorithm also follows a few-shot learning strategy using as training data a few exemplar patches. However, as opposed to our method, they do not account for the variability of the appearance of materials under different lighting and viewpoint so, as we show in Section 2.7, their method does not generalize to unseen illuminations or geometric variations.

Image colorization is concerned with colorizing a gray-scale image given a few colorized exemplars. In this problem, it is critical to infer semantic relationships between the images so that the new scene is perceptually coherent and plausible [He+18; Zha+19a; He+19]. Similarly, edit propagation methods [AP08; End+16] work by propagating strokes provided by a user to the rest of the image, removing the need for a semantic understanding of the input image. Our work is related to the latter techniques, as we leverage the feature spaces of the CNNs and do not require a large labeled dataset to effectively solve our problem, and it can also be used to propagate segmentation masks [LAA08].

Many real-world materials show spatial regularities, commonly referred to as textures. The patterns present in textures can be parameterized, which allows for low-cost material capture or synthesis models. A way to model textured materials is through BTFs (Bidirectional Texture Functions) [Dan+99; LM01], a technique that uses multiple camera views and lighting angles to capture a dense sampling of the appearance of a material. Inspired by such methods, we leverage several images of the material under different illumination conditions, however, we require fewer data than typical BTFs capture setups [Rai+19; Rai+20]. Similarly, the problem of extrapolating BTFs captures to larger material samples was addressed by Steinhausen et al. [Ste+15a; Ste+15b], who propagate measured BTFs using texture synthesis. Our method is not meant to propagate full BTFs measurements but could potentially be applied to such datasets.

The goal of texture synthesis is to reconstruct a larger image given a small sample leveraging structural content. This is a long-standing problem in the computer graphics field and different strategies have been proposed, for instance, using PatchMatch [Dia+15], texture transport [AWL15], point processes [Gue+20; LH06], or neural networks [EM17; Zho+18; FAW19; Rod+19]. Also related to our work, Li et al. [LPG19] capture the appearance of materials by first estimating their BRDF and, then, synthesizing the high resolution microstructure from a dataset of measured SVBRDFs. Our problem is unlike texture synthesis, as we do not aim to create novel content but to predictably transfer visual material attributes.

The problem of estimating a SVBRDF model from one or several images using lightweight capture setups is becoming increasingly popular in the literature. Early work [HS05; HS03] leveraged Photometric Stereo [Ike81] and SVBRDF manifold bootstrapping [Don+10] for surface geometry reconstruction, while newer methods exploit the power of deep neural networks. Recent surveys by Guarnera [Gua+16] and Dong [Don19] contain relevant approaches. While our method is not meant to estimate the SVBRDF properties of a material, it can be used in combination with those techniques to create larger material assets.

There are a few methods that follow a similar paradigm to ours, transferring pre-estimated SVBRDF maps to a larger material sample. Using PatchMatch texture synthesis, Melendez et al. [Mel+12] transfer displacement and albedo maps from small samples of the materials. Their method is limited to daylight illumination and materials present in façades. By means of look-up-tables, and using surface normals and specular as guidance, Riviere et al. [RPG16] transfer surface reflectance captured with controlled LCD lighting to a material sample observed under natural lighting. Recently, Deschaintre et al. [Des+19] fine-tune a network trained to estimate SVBRDFs [Des+18], to work on larger material samples taking a guidance image as input. This approach is limited to transfer a pre-defined set of property maps while our method can transfer any kind. The strategy of using multiple images of the material under different illuminations as input data is not new. Li et al. [Li+17a] and Ye et al. [Ye+18] utilize a self-augmentation strategy to make the estimation of the SVBRDF more robust to unknown environment illumination. We are inspired by these approaches to increase robustness in the model predictions.

Each training step takes as input a pair of corresponding image patches taken from the photometric input IL, and the visual attribute w. Using this data alone already provides a good starting point for generalizing to unseen illumination setups. However, it is not sufficient in scenarios in which the guidance image contains variations due to image noise, a different scale, or any other affine distortion. In order to make the transfer invariant to these transformations, the network needs to be trained with the appropriate data. Data augmentation strategies are essential for reducing the amount of necessary data for training [SK19; Kar+20a], however, random strategies not taking into account the particular domain might degrade the quality of the prediction. We therefore follow a pre-defined data augmentation policy T (illustrated in Figure 2.3), where random operations are performed sequentially.

Color Augmentation Even if the microstructure of the material is homogeneous and can be measured using only a small patch of it, it may not be possible for the network to estimate its visual attributes for parts of the material which contain previously unseen colors. We correct this by randomly permuting the color channels of the photometric input (see Figure 2.4). As we show in Section 2.7, this data augmentation policy also helps make the model generalize the transfer to similar materials, by learning features that are more related to the structure of the material than to its color. Similar operations have been recently proposed for finding robust visual representations on self-supervised settings [Che+20b]. Only the photometric input is subject to this transformation.

Affine Transforms In order to allow for material editing applications in real images, the model should generalize to images taken under camera perspectives and geometry distortions different than those present in the fronto-planar images used for training. Many of such texture irregularities can be defined as an affine transform: translation, rotation, shear, or scaling. As CNNs are shift-invariant by design [Kau17], we propose to augment our datasets with random transformations for scalings, rotations, and shears. Those transforms can be efficiently performed on images through matrix multiplications. However, some visual property maps need to be treated specially, as the spatial transform might have a different behavior in 3D vector space, e.g. normals or tangents. In those maps, each pixel is a representation of a 3D vector. As such, we also perform the rotations and shear operations to these maps in 3D space, by multiplying each normal vector by the same affine transformation matrix applied to the 2D image. We perform the rotation around the Z axis, thus, assuming the camera sensor is parallel to the object plane.

Cropping Inspired by recent work on patch-based learning [Tex+20b; Par+20], during training, the network receives small patches of each input pair. Those patches are randomly cropped from the randomly augmented images, so the network receives a considerable amount of variations of the same material, thus making generalization possible.

We follow a uni-modal image-to-image translation learning strategy, assuming there is only one correct mapping from input to output image. This approach is reasonable for the kind of transfers we test in this work. However, it might fail for ambiguous cases where there are multiple suitable outputs for the same input, for which multi-modal approaches [Zhu+17b] or GANs are more advisable, albeit harder to train. Specifically, our model M is a shallow U-Net network [RFB15] with 4 blocks of layers, containing a small number of trainable parameters, inspired on few-shot learning strategies [Wan+20d; Wan+19; Liu+19b]. This type of fully convolutional architecture is of common usage in image-space regression problems, due to its capability of efficiently learning patterns at different levels of abstraction thanks to its multi-scale design. Skip connections are added to enhance local details [RFB15; Iso+17]. The small number of parameters allows for faster training and inference, as well as reduced memory usage.

Note that, as opposed to previous work [DDB20] on material transfer, which is initialized from a pre-trained network, but inspired by single-sample image synthesis methods [Zho+18; SDM19], we train one network per material and visual property. This strategy, although increasing training time, is key for the following reasons: First, it allows us to better understand the generalization capabilities obtained through our data augmentation policies. Second, it guarantees predictability of the trained model as every feature learned by the network is specific for each material and visual property pairs. Finally, it removes potential problems of a biased dataset as there is no cross-material or cross-domain learning. If the input dataset does not contain enough variations of the material to represent the whole material, the model will fail in areas with unseen patterns. In those cases, having a pre-trained network may help as a material prior, as in [DDB20]. However, our method obtains comparable results to pre-trained methods, with a smaller computational footprint and with the additional flexibility of not needing an expensive dataset and large models. In practice, training a single network per material and attribute is not problematic as this process takes less than a minute.

Loss Function Choosing the appropriate loss function for a learning framework highly depends on the problem. For example, some methods [Iso+17] combine a per-pixel L1 metric with adversarial [Goo+14a] losses, as the latter allows for better semantic mappings in multi-modal learning scenarios, whilst the former allows for improved predictability. Texler et al. [Tex+20b] further includes a perceptual loss [Tex+20a], while using a render loss is common in methods that estimate material parameters from photos [Des+18; DDB20]. In our method, we show that using a L1 loss for training is enough for learning accurate and predictable mappings in our regression task, and binary cross-entropy loss for semantic segmentation, following standard practice in image segmentation [RFB15]. We found that the L2 loss function yields overly-smooth outputs, and perceptual loss functions like LPIPS [Zha+18] are more prone to artifacts than pixel-wise ℓnorm metrics. We include an ablation study of the impact of the loss function in the supplementary material of this chapter.

We use PyTorch [Pas+19] as the learning framework, Adam [KB15] for optimization, a learning rate of 0.002, and a batch size of 16. The training images are randomly augmented using uniform distributions by the following operations, in order: First, the photometric input is subject to the color augmentation policy. Then, both inputs and targets are randomly rotated by an angle in the [−90, 90] range, randomly sheared by an angle in the [−45, 45] range, and randomly rescaled in the [0.5, 2] range of scale factors. Then, patches of 128 × 128 pixels are randomly cropped during training to generate a large dataset of images.

All inputs are always standardized using their own mean and variance. Each M is trained for 1000 iterations, which takes around 1 minute on a single Nvidia 1080Ti GPU. Due to the fully convolutional nature of our models and their reduced number of trainable parameters, the guidance images X used for evaluation can be of arbitrary dimensions. We used images of up to 5000 × 5000 pixels for which evaluation takes around 150ms. We refer the reader to the supplementary material in this chapter for a comprehensive description and a diagram of this model, as well as further implementation details.

Our method is agnostic to the capture setup [Nam+16; MWK17; Alc+19], and it may work for any kind of input data (e.g. BTFs [LM01; Rai+19]) as long as the photometric images are pixel-wise aligned with each other and to the visual attribute. This dataset could also be created synthetically by rendering the outputs of any SVBRDF estimation method [DDB20]. In Section 2.7, we show results of our method using these acquisition pipelines.

However, for the purpose of this evaluation, we choose to work with real data. The main reason is that the data obtained with material capture devices poses extra challenges that are difficult to reproduce with render engines. For example, material irregularities, complex optical behavior, or distortions and color shifts introduced by the optical system that might cause the models to produce inaccurate estimations.

We create a dataset containing images of the same material taken with two imaging camera systems: a high-resolution camera that allows us to take pictures of 0.7 × 0.9 cm, with a resolution of 367 × 490 pixels and a macroscopic camera which provides images of 11 × 11 cm, with a resolution of 4800 × 4800 pixels. In terms of illumination, our setup has 27 different collimated light sources uniformly distributed across the hemisphere as well as diffuse illumination. We build a dataset of four different textile materials, whose complex optical behavior due to anisotropy, transmittance, directionality and microstructure [CLA19] turns them particularly challenging for synthesis and editing operations [Kam+16; Rai+19]. For training, we capture one image for each light source, making a total of 28 different images (Figure 2.5 (a)). For the evaluation set, we take one guidance image with diffuse illumination and another guidance image with a directional light source (Figure 2.5 (c)). The visual attributes (Figure 2.5 (b)) are generated automatically using photometric stereo [Ike81] in the case of normal maps and manually by artists in the cases of colorizations and segmentation masks.

For evaluating our models, we choose domain-specific distance metrics different from those they were trained with to better understand their generalization capabilities [TOB15]. For normal maps, we compute the cosine distance between ground truth and estimated maps, as it accounts for the geometric space in which normals lie. In the case of image segmentation, we evaluate the results using the Jaccard similarity coefficient (IoU) [Zho+19], which is well-suited for sparse segmentation tasks. Finally, we use the state-of-the-art metric Learned Perceptual Image Patch Similarity (LPIPS) to evaluate the quality of the colorizations, as it has been shown to outperform L2 norm for visual perception tasks [Zha+18].

In this set of experiments, we aim to evaluate if the model produces the same output after changing the training data and input guidance images. To this end, we measure both the impact of different illuminations and sizes of the photometric dataset, as well as variations of the input guidance images.

Our goal in this experiment is to analyze the equivariance of our model to affine distortions of the guidance images. A function f is said to be equivariant with respect to a transformation T if f(T(x))=T(f(x)). Inspired by [Ben+20], we measure the equivariance of our model M with respect to different affine transforms T performed to an input guidance image X, by computing their difference using the corresponding metric defined in Section 2.5.2 (d(M(T(X)),T(M(X))). To achieve this, we extend the augmentation policy in the previous experiments by adding random shears and rotations to the training process. We train the photometricNet with and without random shears and rotations for every visual attribute in our dataset and on the denim material. In the case of the normals, as described in Section 2.4.1, we also perform the shears and rotations in the geometric space in which normals lie.

We measure their robustness with respect to three different transformations T : rescalings, rotations and shears. As before, we use a guidance image taken under diffuse lighting as input for these experiments and attribute-specific distance metrics. Figure 2.6 (b) shows that not adding those transforms generates visual artifacts for rotations and shears. Interestingly, without affine augmentations, the models hallucinates vertical yarns, as it is the only type of data that it has seen as input. Figure 2.9 shows quantitative metrics for the range of transformations T in which we evaluated the models. Augmenting the training dataset with both shears and rotations generally provides the best results. Furthermore, this enhanced data augmentation policy improves the robustness of the models with respect to the scale of their inputs. Notably, applying random rotations during training appears to have a larger impact than random shears on the robustness of the models. This finding suggests that applying blind policies of data augmentation [ASE17; SK19; San+19; Che+20b] may not be an optimal strategy for some applications like ours, as the networks may not learn relevant features, or overfit to noise present in the training dataset. Finally, it is worth noting that every network has the same number of parameters, and is trained for the same number of iterations, which shows that the generalization capabilities of the models can be increased at no extra parameter cost.

In the first category, the method of Texler et al. [Tex+20a] allows artists to interactively edit a few keyframes of a video and propagate that edition to the rest of the video. As ours, they formulate this transfer problem by training an encoder-decoder network using patch-based learning. But, in contrast to our work, they do not perform any data augmentation policy outside of random cropping. For a fair comparison with such method, we compare two setups: 1) using a single image as training input, and 2) using the photometric dataset. Results are shown in Figures 2.2 and 2.10. As Texler’s method is not scale invariant, in both cases the training data provided for their model has the same scale as the images used for testing. Our models have been trained with the full policy of data augmentation. In the first setup (Figure 2.10), we use the diffuse illumination and hence compare their output with our diffuseNet output. As shown, none of the methods provide high quality results but our model manages to provide closer estimations. In the second setup (Figure 2.2), we train their model with the photometric input. The best results are obtained with our method. Even though extending the input data using photometric cues has an impact on the quality of Texler’s results, the lack of a data augmentation policy makes the transfer fuzzier and noisier. Further, their combination of style, adversarial and pixel-wise losses fails to yield predictable mappings. These results confirm the importance of a comprehensive data-augmentation policy, such as the one we propose, when using neural networks for image processing tasks of this kind. In addition, our model is trained in less time with a smaller computational footprint (1 minute vs 5 minutes).

Another way of formulating this visual attribute transfer problem is through image analogies. Using our single diffuse image for input, we compare our approach with two methods as shown in Figure 2.11. First, the work of Liao et al. [Lia+17], which uses deep latent spaces as image descriptors; and the method of Benaim et al. [Ben+21] that trains single-image generative models to find bijective mappings between the structure of one image and the style of another. Our method qualitatively outperforms these methods with a fraction of the computational cost: 1 minute in our case, 10 minutes in [Her+01], 40 minutes in [Lia+17] and 10 hours in [Ben+21]. Once trained, our models can be used to evaluate any guidance image in real time for materials with similar microstructure. In contrast, image analogies methods require expensive optimizations for each guidance image. We refer the reader to the supplementary material for more comparisons with these methods.

Figure 2.12 shows additional results of material stylizations. In these examples, we used the method of Gatys et al. [GEB15b] to stylize a small patch of the material. Then, we trained a model using photometricNet and diffuseNet. As the guidance image we used a bigger image with diffuse illumination. Compared with naïve style transfer applied to the whole image, our approach provides detailed stylizations where the microstructure of the material is preserved. We further see that diffuseNet provides noisier results than photometricNet, probably due to the fact that the photometric cues help to preserve the local shading variations.

Our method can be used to propagate SVBRDFs estimated locally in a small area of the material to larger samples. Figure 2.13 illustrates for a diverse set of materials that we can propagate albedo and normals estimated at high resolution in a small area of 0.7 x 0.9 cm to guidance images of 13 x 13 cm taken with a smartphone. Even though we train the albedo and normals models separately, which does not guarantee pixel-wise coherence between the estimated maps, the rendered images show realistic-looking materials, even with a diffuse material model. As opposed to the method of Deschaintre et al. [DDB20], which is trained to output directly Cook-Torrance [CT82] material layers, our method is agnostic to the parameters of the SVBRDF.

We assess the capabilities of our method on this setting using the same SVBRDF propagation scenario as proposed in [DDB20]. Using a small crop of a synthetic SVBRDF as input, we render 27 images using the same directional light position as we used in our real dataset, and train a photometricNet to estimate the surface normals from each of those renders. We then evaluate this model using a larger area of the material, illuminated under an unknown lighting position. In Table 2.1, we show a quantitative comparison with [DDB20], under different image quality metrics. As shown, our method achieves better scores on pixel-wise metrics, whilst [DDB20] achieves better deep perceptual scores, as in the LPIPS metric [Zha+18]. This might be related to the design of our loss function: we directly minimize pixel-wise differences, while [DDB20] is optimized using a render-aware loss. Qualitatively, as shown in Figure 2.14, our method obtains comparable quality mappings, with fewer artifacts. Despite capturing SVBRDF or arbitrary materials is not the goal of our method, we include in the supplementary comparisons with a direct SVBRDF acquisition method [Gao+19] using our own real dataset.

A similar setting can be used to propagate attributes leveraging BTF measurements as training data. Figure 2.15 shows an example using a BTF from [WGK14]. Using Photometric Stereo [Ike81], we compute their surface normals and train a photometricNet on a central crop of the BTF, using the captures at a camera position of (ϕ = 0°, θ = 0°). We then evaluate this model using the full material surface, and a novel camera position, of (ϕ = 15°, θ = 11°). As shown, our model is capable of working with captured BTF data.

In previous experiments, we have shown the performance of our models when the guidance image corresponds to the material used for training. In this experiment, we show that our models, despite being trained only on a set of captures of a single material, generalize to materials of the same category. Figure 2.16 shows some examples for models trained using our dataset (Figure 2.5), taking input guidance images of different albedos and scales. The transfer works thanks to the network design and data augmentation strategy that is designed to use microstructure details as guidance. Our method could thus be used to transfer visual attributes for a diverse set of materials by simply training one model using a single but representative material of each category.

Our models are not guaranteed to provide high-quality results outside the range of input data and data augmentation policies we train them on. This limitation is common to all learning-based approaches. It is unlikely that the framework is capable of generalizing to resolutions higher to those of the training data; or down-sampled images in which the texture details are not recognizable. Furthermore, the type of transformations we apply to the training data may not represent all the possible geometric variations, or non-linear warpings that materials are subject to in the real world. For materials which exhibit a strong variation in their microstructure, which cannot be fully captured using a single photometric dataset, our patch-based approach will likely fail to generalize to the full heterogeneity. Figure 2.17 shows two examples of failure cases for which the test data is not included in the training data and where the affine and illumination transformations are outside the suitable range.

Our model, summarized in Figure 2.18, is a standard U-Net [RFB15] encoder-decoder network with 4 blocks of layers and skip connections. U-Nets are widely used in pixel-wise regression and image-to-image translation problems, due to their efficient design, capable of exploiting texture and semantic patterns at different levels of abstraction, due to their multi-scale design. Skip connections are known to significantly enhance the quality of deep image-to-image translation models [Iso+17]. Each block is comprised of a tuple of convolutional layers with kernel sizes of 3 × 3, followed by a batch normalization operation [IS15] and a ReLU [NH10] non-linearity. The number of convolutional filters in each layer is shown in Figure 2.18. The last layer of the network uses 1 × 1 convolutional filters to transform the feature vectors into the desired number of classes (e.g 3 output channels for RGB regression tasks, as in normal map estimation or recoloring; or 1 output channel for the semantic segmentation problem). The total number of trainable parameters depends on the number of property maps that the network should jointly learn, but are around 483000 for all the problems we showcased in this work. Trainable weights are initialized randomly, by sampling a N(μ=0,σ=0.02). The output of each block in the encoder part of the network is max-pooled to halve its spatial resolution. Upsampling in the decoder side of the network is done through up-convolutions. For image segmentation problems, we add a Sigmoid non-linearity to the output of the network. Due to the reduced number of trainable parameters, models take only 1.94Mbs, allowing for fast training and evaluation. On Table 2.2, we show a quantitative study on the influence of the depth of the model in its accuracy, using the 1684 material from [Des+18] as the training dataset. A visualization of these results is included in Figure 2.19, showing that deeper models yield overly-smooth results for this few-shot learning problem.

We use PyTorch [Pas+19] as the learning framework, Adam [KB15] for optimization, a learning rate of α=0.002,β1=0.9,β2=0.999,ϵ=10e−8, and a batch size of 16. The training images are randomly augmented using a uniform distribution by the following operations, in order: randomly rotated by an angle in the [−90, 90] range, randomly sheared by an angle in the [−45, 45] range, and randomly rescaled in the [0.5, 2] range of scale factors. Their color channels are randomly reordered and patches of 128 × 128 pixels are randomly cropped during training, to generate a large dataset of images. The rescaling factors and rotation angles are randomly chosen for each image in each batch using a uniform distribution on the mentioned ranges. All these data augmentation operations are also applied to the target visual attributes, except for the color channel reordering, which is only applied to the input images. Images and visual attributes are always standardized using their own mean and variance. All the data augmentation operations are performed natively in GPU, which allows for faster training. To further reduce the computational cost and training times of our models, we train our models using automatic mixed precision training [Mic+18], which reduces the memory footprint of the models in GPU, while allowing for faster computation of certain operations, such as convolutions.

Each Mω is trained independently for each material and application (no cross-domain training or fine-tuning) for 1000 iterations, which takes around 1 minute on a single 1080Ti GPU. Due to the fully convolutional nature of our models, and their reduced number of trainable parameters, the guidance images X used for evaluation can be of arbitrary resolutions. We evaluate the models using half-precision, which allows us to utilize guidance images of up to 5000 × 5000 pixels, for which evaluation takes around 150ms in GPU. Thanks to the reduced number of parameters, an efficient use of GPU-native data augmentation techniques and leveraging half precision training and evaluation, we are able to train and evaluate our photometricNets in less than a minute of total computational time.

Finally, in Figure 2.20, we show a comparison of the results of our models when trained on different loss functions. As shown, even though the differences are sometimes subtle, the L1 consistently yields sharper estimations than the L2, whilst the perceptual loss function is more artifact-prone and thus less predictable. L1 is a common loss function in many image regression tasks, as in image-to-image translation [Iso+17] or in image synthesis [Zho+18].

We show the positions of the lights in the photometric dataset in Figure 2.21.

In Figure 2.22, we compare our method to different image analogies approaches. We show a comparison between [Her+01], [Lia+17], [Ben+21], and our method. The seminal work in image analogies [Her+01] fails in this visual attribute transfer problem, most likely due to the difference in scale between input and target images. For the comparisons with [Lia+17] and [Ben+21], we reduced the scale of the target images D to 256 × 256 pixels to make their methods computationally tractable. The methods in [Lia+17; Ben+21] never see the diffuse image A, as they generate an intermediate representation by themselves. While showing better results than [Her+01], methods which use deep learning for this task still fail to yield compelling or predictable results. The method in [Ben+21], which leverages structural information in the images to train a single-image generative model [SDM19], shows more plausible results, but with an expensive computational footprint (10 hours per network), compared to the 45 minutes in [Lia+17], 10 minutes in [Her+01] and less than 1 minute in our case. Because we assume pixel-wise semantic correspondence between input image and visual attributes, we can find mappings in interactive times, and which are significantly more predictable and of higher quality than these image analogies methods.

In Figure 2.23, we show a comparison with different configurations of [Tex+20b]. Specifically, we train their method using a dataset with a single image using diffuse illumination (left) and the full photometric dataset (middle column). Their method is trained individually for each dataset and visual attribute, for around 5 minutes to guarantee convergence, as suggested in their paper. As [Tex+20b] assumes that the scale (pixels/cm) of the dataset and evaluation images are the same, we rescale our input dataset to match the scale of the evaluation images. We evaluate each method using three different images X, one with diffuse-light illumination (top row), another with diffuse-light illumination but with a geometric transformation (middle row), and an image captured with directional lighting (bottom row). As shown, by enhancing the method in [Tex+20b] with a photometric dataset, their model is able to generalize better to every image X in the evaluation dataset than its counterpart trained only using a single image. However, our method, specifically designed for this illumination and geometry invariant visual attribute transfer outperforms their work in every type of input and visual attribute, whilst maintaining a lower computational overhead (1 minute of training in our case) and memory footprint (their trained network uses up 12 Mbs, while ours is 1.9 Mbs). Our model trained using a single illumination (diffuseNet) shows better results than their method trained using the photometric dataset, suggesting that our transfer method has more adequate inductive biases for the visual attribute transfer problem.

In Figure 2.24, we show a comparison with a recent deep SVBRDF acquisition method [Gao+19], which relies on a pre-trained autoencoder and a differentiable render engine for optimizing the initial estimation of the autoencoder. As shown, our method generates sharper and more coherent normal maps. PhotometricNet relies on a small sample of the same material for training, and is thus less sensitive to potential biases of the training dataset, with the cost of having reduced generalization capabilities compared to multi-material methods.

To propagate material properties, a common solution is to utilize the estimation of material properties in a localized area and extended it to larger samples [ND06; RSK13; Ste+14; DDB20]. We follow this approach to propagate SVBSDF measurements, and build upon the work presented in Section 2 and [RG21], which leverage images of the material taken under different illumination conditions to train a neural network that propagates individual maps.

As input to this propagation step, we use a photometric dataset obtained with a microscopic setup, spatially-varying parameters of a micro SVBSDF (Illustrated in Figure 2.28), and a guidance image that will serve as a reference for the mesoscale propagation. We use as guidance the image taken with a mid-range camera and diffuse lighting. Unlike the previous approach, aimed at transferring individual maps, we introduce some modifications to jointly propagate the full stack of parameters. We further adapt the method to be able to take as input more than one micro SVBSDF, necessary if the variability of material is not visible in a single micro capture (see Figure 2.27 (a)). We propose the following modifications:

In Figure 2.30, we show an ablation study on the improvements of the maps propagation method proposed in [RG21]. We build upon their proposed implementation, removing their random shifts and rotations, and make progressive changes, in order, to the dataset size, model architecture, loss function, and data augmentation. We observe high-quality propagations, but the baseline sometimes produces overly smooth outputs (Fleece, Leather-Brown, Rib-Silver, or Leather-Lizard). Our proposed increased dataset and model architecture modifications enhance the maps’ sharpness, and our perceptual loss function allows for the propagation of additional details. Our final model removes the color augmentation proposed in [RG21] whenever multiple microscale captures are available, and introduces random blurs during training, allowing for achieving the highest quality results and for accurate base color propagations in challenging cases, as in Tartan.

Traditionally, non-parametric texture synthesis algorithms worked by ensuring that every patch in the output textures approximates a patch in the input texture. Earlier methods included image quilting [EF01; EL99], GraphCuts [Kwa+03], genetic algorithms [DZP07], and optimization [Kwa+05; PS00]. More recent approaches use variations of PatchMatch [Bar+09; Bar+15] as a way of finding correspondences between generated and input images [Kas+15; Dar+12; Zho+17].

Despite those methods showing high-quality results for textures of different characteristics, recent work on deep parametric texture synthesis shows better generality and scalability, requiring less manual input. Our approach belongs to the category of Parametric texture synthesis. These methods learn statistics from the example textures, which can then be used for the generation of new images that match those statistics. While traditional methods used hand-crafted features [De 97; HB95], recent parametric methods rely on deep neural networks as their parameterization. Activations within latent spaces in pre-trained CNNs have been shown to capture relevant statistics of the style and texture of images [GEB16a; Gat+17; JAF16; Ren+17; RB19]. Textures can be synthesized through this approach by gradient-descent optimization [Sne17; GEB15a] or by training a neural network that learns those features [DB16; Nik+21]. Finding generic patterns that precisely describe the example textures is one of the main challenges in parametric texture synthesis. Features that describe textured images in a generic way are hard to find and they typically require hand-tuning. Generative Adversarial Networks (GANs), which have shown remarkable capabilities in image generation in multiple domains [Kar+18; KLA19; Kar+20b], can learn those features from data. Specifically, in texture synthesis, they have proven successful at generating new samples of textures from a single input image [Zho+18] or from a dataset of images [BJV17; FAW19; JBV16; Liu+20; Mar+20]. We build upon the method of Zhou et al. [Zho+18] which shows good performance on the synthesis of non-stationary single image textures, and extend it to synthesize texture stacks, as well as generate tileable outputs.

Synthesizing tileable textures received surprisingly little attention in the literature until recent years. Moritz et al. [Mor+17] propose a non-parametric approach that is able to synthesize textures from a single example while preserving its stationarity, which measures how tileable the texture is. Li et al. [Li+20] propose a GraphCuts-based algorithm. They first find a patch that optimally represents the texture, then use graph cuts to transform its borders to improve its tileability. This method allows for synthesizing multiple maps at the same time. Relatedly, Deliot and Heitz [DH19] propose a histogram-preserving blending operation for patch-based synthesis of tileable textures, particularly suited for stochastic textures. The power of deep neural networks for tileable texture synthesis has also been leveraged in the past years. First, Rodriguez-Pardo et al. [Rod+19] exploit latent spaces in a pre-trained neural network to find the size of the repeating pattern in the input texture. Then, they use perceptual losses for finding the optimal crop of the image such that, when tiled, looks the most similar to the original image. Also leveraging deep perceptual losses and using Neural Cellular Automata [Mor+20] as an image parameterization, Niklasson et al. [Nik+21] generate self-organizing textures that are seamlessly tileable by design, but are limited in resolution and by the quality of the gram matrix perceptual metric used as a loss function [Hei+21]. Bergmann et al. [BJV17] achieve tileability in their output textures by training a multi-image GAN in a periodic spatial manifold. Our proposed method does not follow any of these approaches. Instead, we build upon a state-of-the-art single-image texture synthesis method, which is able to generate high-quality and high-resolution images, and extend it to generate textures which are tileable. To this end, we propose a sampling algorithm that finds the input of the generative process that maximizes a novel tileability metric. Our goal is to preserve the input original texture as intact as possible while imposing artifact-free continuity at the intersection of the seams when the texture is tiled. Furthermore, our method has a reduced computational footprint compared to other deep generative texture synthesis methods, as we show in the results section.

Learning patterns from a single image has been studied in recent years, in contexts different to those of texture synthesis. Ulyanov et al. [UVL18] show that single images can be represented by randomly initialized CNNs, and show applicability on denoising or image inpainting problems. A similar method is proposed by Shocher et al. [SCI18] for single-image super-resolution. Additionally, single images have shown to be enough for learning low-level features that generalize to multiple problems [ARV19]. Hypernetworks [HDL16] in Implicit Neural Representations [Sit+20; Tan+20] also allow for shift-invariant priors over images, which can then used in generative models [DTD21]. Single-image generative models have been explored for domains other than textures. GANs trained on a single image have been used for image retargeting [Sho+19], deep image analogies [Ben+21], or for learning a single-sample image generative model [SDM19; Hin+21; Sho+19; SGK21; VZH20]. These methods, while powerful for natural images, are not well-behaved for textures, as shown in [Liu+20] and [Mar+20]. By introducing inductive biases specially designed for textured images, characterized by their repeating patterns, deep texture synthesis methods achieve better performance in textured images than generic single-image synthesis approaches, which need to account for more globally coherent semantic patterns.

SeamlessGAN is comprised of an encoder-decoder convolutional generator G with residual connections, and a convolutional discriminator D. So as to be able to synthesize textures of multiple different sizes, the networks are designed to be fully-convolutional. We follow the residual architectural design in [Zho+18], with two extensions: First, building on recent advances in style transfer algorithms, we use Instance Normalization [UVL16] before each ReLU non-linearity in the generator and each Leaky-ReLU [MHN13] operation in the discriminator. This allows to use normalization for training the networks without the typical artifacts caused by Batch Normalization [Cho+18; Des+18; NK18; Kur+19]. Second, in order to allow for the synthesis of multiple texture maps at the same time, we propose a variation in the generator architecture. In the following, we describe the details of each component, with a particular focus on the elements that are different from [Zho+18].

After training, the generator G is able to synthesize novel samples of the texture given a small exemplar of it. Although these novel samples duplicate the size of the input, they are not tileable by default. Previous work [FAW19] showed that by spatially concatenating different latent spaces in a ProGAN [Kar+18] generator, it is possible to generate textures that contain the visual information of those tiles while seamlessly transitioning between the generated tiles. Inspired by this idea, we spatially repeat (horizontally and vertically) the first latent space x₀ within the generative model, obtaining a latent field F₀. This field is passed through the residual layers Rl and the decoders G to get a texture stack, T^, that contains four copies of the same texture with seamless transitions between them (see Figure 3.3). At first, the latent field F₀ shows strong discontinuities at the borders between the four copies. Later, as the texture is passed through the network, these artifacts are progressively transformed into seamless borders by the rest of the residual blocks and the decoder of the model. The intuition behind this idea is that, after training the network with the target texture, each of these latent spaces encode low resolution versions of the input in the spatial domain and semantically-rich information in the deeper layers.

A seamlessly tileable texture stack T^c is obtained by cropping the central region (with an area of 50% of T^) . The predicted texture T^ has 4 × 4 the resolution of the input crop t. Thus, after the cropping operation, T^c has twice the resolution of the input t.

A key parameter to select is the level l at which to split the generative process. As shown in Figure 3.5, generating the latent field F by tiling earlier levels of the latent space forces the network to transform F more times, thus resulting in more seamlessly tileable textures. We confirm the results found in [FAW19] and noticed that generating this latent field at earlier levels of the latent space (l∈{0,1}) yields the best visual results, by allowing for smoother and more semantically coherent transitions between tiles. We thus tile the output of the encoder x0=E(t), before transforming it by the residual blocks Ri.

Our strategy to tile the latent space guarantees that the generated texture is a continuous function with smooth transitions between the boundaries of the tiles. However, in contrast to the algorithm in [FAW19], where the latent spaces are drawn from random vectors, ours are encoded representations of input textures. Thus, the selection of the input that the network receives plays an important role on the quality of the output textures. As shown in Figure 3.6, not all the generated textures are equally valid. This selection can be posed as an optimization problem: t∗=argmaxtQ(G(t)), where the goal is to find the crop t^* that maximizes the quality Q of the generated texture G(t)=T^c. To solve this optimization problem, one option could be to pose it as an optimization in the generative latent space. However, we found that a simpler solution based on sampling already provides satisfactory results. The remainder of this section explains the sampling process and the quality metric.

Sampling By using a fully-convolutional GAN, our model can generate textures of any size, hardware being the only limiting factor. This is key for tileable texture synthesis as, even if a given texture is seamlessly tileable, larger textures require fewer repetitions to cover the same spatial area, which ultimately results in fewer repeating artifacts, as illustrated in Figure 3.7 (a). There are two main challenges when finding tileable textures: the input texture needs to contain the distribution of the existing repeating patterns, and the input tile itself must not create strong artifacts when tiling the latent spaces of the generator.

Our goal is thus to find the largest possible tileable texture stack. To do so, we sample multiple candidate crops for a given crop size c∈{cmin,cmax} is the resolution of the input T. We sample crop sizes starting at the largest possible size c=cmax, and stop when we find a suitable candidate according to the tileability metric Q. As shown in Figure 3.7 (b), this sampling mechanism also allows us to generate multiple tileable candidates for a single exemplar if we choose different parts of T as input.

Discriminator-guided Quality Function, Q The second component of our sampling strategy is the quality function used to determine whether a stack is tileable or not. We observed that the artifacts appear on vertical and horizontal frames around the center of the textures (Figure 3.6). This is likely caused by strong discontinuities or gradients on the same areas of the tiled latent spaces, which the rest of the generative network fails to transform into realistic textures. Following recent work on generative models [SSK20], we use the discriminator D as a semantic-aware error metric that can be exploited for detecting local artifacts in the generated textures. This can be done in our case because the global loss function contains pixel-wise, style, and adversarial losses. The adversarial loss learns the semantics of the texture, whereas the L1 and style losses model color distances or repeated patterns [Rod+19]. This combination of loss functions allows us to balance textural, semantic, and perceptual color properties.

We thus design a quality evaluation function Q that estimates if the generated texture stack T^c is tileable, looking for artifacts on a central area, S∈D(T^c) of the discriminator. This area is composed of two regions, S=sv∪sh, where Sv is a vertical area, and Sh is an horizontal one, both centered on the output of the discriminator (Figure 3.3). The function Q leverages the fact that D outputs 0 when it believes a patch to be synthetic. However, as the values are sample-dependent, we establish a threshold τ∈ℝ using the values of the rest of the image S as a reference τ=γ⋅min(Sr), where Sr=D(T^c)∩S¯ is the remaining part of the image, and γ∈ℝ is a threshold that allows to control the sensitiveness of Q. Consequently, Q(T^c) is 1 if min(sv) and min(sh) are greater or equal than τ, considering the texture as tileable, and 0 otherwise. The goal of Q is to estimate whether or not the central areas, where discontinuities may be present, are as realistic as the rest of the texture. It works as a classification function which returns a high value if the texture is identified as real by the discriminator in such areas. This allows us to create a sampling strategy that distinguishes between images with local artifacts from seamless textures. By using minimum values instead of the mean estimation of the discriminator, our quality metric focuses on detecting artifacts which may arise during the latent space tiling operation.

One of the main challenges when synthesizing texture stacks using a single generator is to preserve the low-level details of each of the texture maps whilst maintaining the local spatial coherence between them; if this coherence is lost, renders that use the synthesized maps will show artifacts or look unrealistic. We propose two different variations of the single-map generative architecture G presented in [Zho+18], each of which makes different assumptions on how the synthesis should be learned taking into account the particular semantics and purpose of each map. A diagram of each proposed architecture is shown in Figure 3.10. For a fair evaluation, we follow the criteria that both networks must have approximately the same number of trainable parameters.

Our baseline, G1, treats the texture stack as a multiple-channel input image, and entangles every texture map in the same layers. It assumes that the maps in the stack share most of the structural information and, as such, there is no need to generate them separately. Thus, the last layer in the decoder outputs every texture map. Our proposed alternative architecture, G2 finds a shared representation of each texture map, but has a separate decoder for each of them. As such, the residual blocks are shared for all the texture stack, but each decoder can be optimized for the semantics and statistics of each particular map.

To quantitatively evaluate which architecture produces the highest quality output we compare the original texture with the generated one using standard metrics: SSIM, Si-FID, and LPIPS. The Structural Similarity Index Measure (SSIM) [Wan+04] is a perceptual-aware metric, working on the pixel space, that measures the similarity in structural information, and may be appropriate to evaluate synthesized textures. The Si-FID [SDM19] is a single image extension of the Fréchet Inception Score (FID) [Heu+17], which measures the difference in deep latent statistics between natural and artificially generated images. Finally, we use the Learned Perceptual Image Patch Similarity (LPIPS) [Zha+18] as a perceptual distance metric in deep image spaces. This metric is widely used for evaluating generative models [Kar+20b; Hua+18b; Cha+19; Alm+18; Mar+20].

The baseline G1 shows more artifacts than G2, most likely due to the fact that the generation parts of the network are not fully separated. A quantitative evaluation is shown in Table 3.1, showing that G2 outperforms G1 from perceptual and statistical standpoints. Those metrics require the input and target images to have the same spatial dimensions. To obtain this, we crop the 50% center area of each generated stack, which doubles the dimensions of the inputs due to the adversarial expansion approach we follow, and compare them with the input textures. In summary, G2 allows to generate textures that better preserve the properties of the input images without any additional computational cost and without requiring to modify the loss function, or the discriminator design.

Whilst our generators G can generate high-quality tileable pairs of albedo and normal maps, there is no guarantee that those maps are pixel-wise coherent, as no part in the loss function explicitly accounts for this relationship. The Lstyle loss is computed separately for each map in the texture stack, which may generate non-coherent gradients. Furthermore, our architecture of choice G2 separates the decoding of each map in different parts of its architecture, which may hinder the generation of spatially-coherent maps. Nevertheless, we show that this is not a problem in practice. Figure 3.8 shows crops of our synthesized texture maps. It can be seen that pixel-wise coherency between maps is preserved even for challenging geometric structures. We argue that the role of the discriminator is key to detect inter-layer inconsistencies by yielding lower probabilities for non-coherent maps. Even if separating the decoders may increase the risk of incoherent texture maps, this coherence is forced by the discriminator during training. To test this empirically, for the textures in Figure 3.8, we fed the discriminator with a crop of the original stack, a crop with the normals translated 5 px, and translated 100 px, for which we obtain average values of 0.99, 0.35, and 0.32, respectively. This suggests that, since during training, the discriminator receives the entire texture stack at once, it learns to identify spatial inconsistencies between maps.

As described in Section 3.4, the generator G is trained to minimize a loss function, which is comprised of adversarial, perceptual and pixel-wise components. Each of these have different impacts on the output synthesis. We hereby study the impact of each of these components.

To better isolate the impact of each metric, we perform this study using a limited generator that only outputs albedo maps. As we show in Figure 3.9, the adversarial loss Ladv provides high-level semantic consistency, the L1 norm acts as a regularization method which removes artifacts, while the perceptual loss Lstyle adds additional details to fully represent the texture. Similar findings are reported in [Zho+18]. None of these loss functions yield compelling or artifact-free results on isolation, being the adversarial loss the most important factor of the global function.

Our neural network is composed of three modules: an autoencoder A, a neural texture T∈ℝH×W×D, and a renderer R. The renderer R(T(u,v),ωo,ωi)=RGB takes as input the feature vector at the (u, v) coordinates of the neural texture T , the view ωO and light ωi positions, and returns an RGB value. R acts as a conventional BTF and can be used as such in any render engine. An input guidance image, G∈ℝH×W×3, is used during inference to condition the generation of the neural texture T. This conditioning allows us to propagate the learned reflectance to novel guidance images that can be: a larger sample of the same material, a different material, or a structural image. In the simpler case, the guidance image comes from the BTF used for training, and our process is equivalent to previous work [Rai+19; Kuz+21].

The autoencoder A takes the guidance image G∈ℝH×W×3 as input and outputs a neural texture T∈ℝH×W×D with the same size H×W as the input guidance image but with more latent dimensions D. As a result, each pixel in the guidance image has a higher-dimensional neural representation in T. Because it is trained without explicit supervision, this latent representation can capture the reflectance and structural patterns automatically. Kuznetsov et al. [Kuz+21] also used a latent neural representation of the material, however, lacking the initial autoencoder their approach cannot synthesize novel BTFs without retraining, while our conditioning module allow us to generate novel BTFs during test time. The autoencoder and the renderer are neural networks trained jointly, using an end-to-end image-to-image approach we describe below.

Figure 4.2 (bottom) illustrates our training process. It has two objectives: First, equivalent to regular BTF encoding, we aim to find the mapping between camera direction ωi, light direction w_i, and output slices of the BTF: (ωo,ωi)→Vωo,ωi∈BTF. Second, we aim to condition the synthesis process with an input guidance image, G.

To this end, for training, we feed the model with images, Vω~0,ω~i, which are randomly sampled from the BTF, and are subject to additional data augmentation processes.

This extensive augmentation process guarantees invariance to different input variations during test time, like camera or illumination conditions, while keeps consistency of the outputs.

Loss Function Design Our loss function, that compares ground truth slices Vω0,ωi with generated ones R(A(f(Vω~0,ω~i)),ωo,ωi), is a weighted sum of three terms: a pixel-wise loss, a style loss, and a frequency loss,

(4.1) L=λL1L1+λstyleLstyle+λfreqLfreq

The main driver of our loss is the pixel-wise norm L1⋅L1 produces sharper results than higher-order alternatives, such as L2 [Iso+17; RG21]. Following [Kuz+21], we apply a θo(x + 1) compression to improve the model results on high dynamic range. This compression is only done to the pixel-wise component of the loss function. Inspired by recent work on texture synthesis, capture and transfer [Mar+20; AWL13; RG22; Rod+23a; Zho+22], we introduce a Lstyle loss to help the model generate higher quality and sharper results. Further, to mitigate the spectral bias of convolutional neural networks and help ameliorate the results further, we also introduce a focal frequency loss into our learning framework [Jia+21]. This combination of loss functions proves effective for our problem, without the need for complex adversarial losses which could reduce efficiency or destabilize training.

Data Augmentation We train our models using a comprehensive data augmentation policy aimed at achieving high quality reflectance propagation, increasing performance and generalization, and allowing for generation of multiple resolution materials at test time. We build upon recent work on material transfer [RG21] and use images of the material taken under different illumination and viewing conditions as inputs to our autoencoder. This helps it generalize to novel capture setups, which allows for multiple applications we describe on Section 4.6. In particular, we use every image available on the input BTF, selected uniformly at random for each element in each batch during training. As in [Tex+20b; RG21; Rod+23a],

we also use random rescaling, which helps the model generalize to new scales, and, and build neural materials of multiple resolutions at test time, as we describe on Section 4.6.3 inspired by recent work on image synthesis [RG21; Tex+20b; RG22], we use random cropping, which helps generalization by effectively increasing the dataset size. Finally, we extend the color augmentation policy in [RG21] with random hue changes across the entire color wheel, and introduce random Gaussian noise and blurs to the input images, to help it generalize further, as proposed in [Rod+23a] (Chapter 5).

We evaluate NeuBTF on materials from different sources including acquired BTFs from [WGK14; Hai12], and rendered BTFs from procedurally generated and scanned SVBRDFs. In Figure 4.3, we show examples of the results of NeuBTF for a variety of materials with highly complex structures and reflectance properties, like colored specular (first column) or anisotropy (last). We show some additional results in Figure 4.4 for materials of different datasets. As shown, our model achieves high quality reconstructions regardless on the type of data source.

In Figure 4.5, we show a colored visualization for a few channels of the latent neural texture T found for a variety of materials. Because the values for the neural texture are unbounded, to each channel c∈T, we standardize them to 0 mean and unit variance, and apply a sigmoid(c)=1e1−c+1 non-linearity to make the maps comparable. Without any explicit training, the models learn to separate distinct parts of the material. For example, the model finds distinct latent spaces for warp and weft yarns on woven fabrics, or separation between color and geometric patterns. This disentanglement provides clues on why the material propagation is possible, and suggests potential future research directions for fine-grained neural material edition.

Finally, we also evaluate whether the encoded materials preserve the geometric properties in their training datasets. We leverage Photometric Stereo [Ike81] to compute surface normals of encoded and ground truth materials. We provide results in Figure 4.6, where we show that the reconstructed surface geometry is accurate albeit noisier, and preserves the structure of the material.

In Table 4.1, we show the number of trainable parameters on the decoders of different neural BTF compression algorithms. As shown, our model is competitive with previous work in terms of trainable parameters. This is achieved as we use more complex loss functions than previous work, which help regularize the models, and because our sinusoidal MLP achieves higher quality reconstructions for natural signals than ReLU MLPs, as shown in [Sit+20]. NeuMIP [Kuz+21] uses smaller MLPs, however, they require an additional decoder for their neural offset module, which helps them encode parallax effects (See Figure 4.7), for which our model struggles. Our decoder has one order of magnitude fewer parameters than [Rai+19; Rai+20], however, the method in [Rai+20] provides the benefit of fast encoding of new materials, while ours requires a different model for each new material.

As we show in Figure 4.7, our model struggles with materials with strong displacement. While our method provides accurate encodings on viewing angles close to the material surface, it cannot accurately encode grazing angles for such extreme cases. Displacement maps translate the geometric position of the points over the surface, breaking the underlying assumptions behind our neural texture. NeuMIP [Kuz+21] solves this issue by explicitly modelling parallax effects with a neural offset module. While we did not observe that such extension was needed for acquired BTF data, like the UBO2014 [Rei+14] dataset, introducing a similar module into our editable neural material framework is an interesting future research direction to increase its generality.

Many material reflectance acquisition devices are limited in the surface dimensions they can digitize. This hinders their applicability to many real-world materials, which exhibit variations that cannot be captured at such small scales. Further, in many applications like SVBRDF acquisition, obtaining larger samples of the material improves realism and helps tileable texture synthesis. In this context, previous work on BTF reflectance compression inherits the surface area limitations of the capture devices used to generate their training data. Our method can easily be applied for reflectance propagation. We build upon the work of Rodriguez-Pardo and Garces [RG21] and leverage our encoder to propagate the neural texture optimized using a small portion of the material (e.g. a 1 × 1 cm capture) to a larger portion of the same material, represented with a guidance image captured using a commodity device like a flatbed scanner. Because our model is trained using a large amount of lighting conditions, as in [RG21], the propagation is invariant to how the images are illuminated. We show results of such pipeline in Figure 4.3. For instance, on the last row, we show an anisotropic and specular Silver Jacquard fabric, for which we generated a BTF by rendering a 1 × 1 cm SVBRDF. This small crop cannot represent the complex pattern in the fabric, which we show on the guidance image, which covers a 10 × 10 cm area. Using our encoder, we propagate the neural material to this guidance image, generating a new, high-resolution, latent space which we can render, enabling realistic material representations with a reduced digitization cost. This propagated neural material has a 2000 × 2000 texels resolution, and it requires no retraining during test time.

Relatedly, our propagation framework can also be easily leveraged for generating tileable BTFs. Given any guidance image of the material, we can generate a tileable version of it, either using manual editions by artists or automatic algorithms [Mor+17; Li+20; Rod+19; RG22]. With this tileable input guidance, we can use our autoencoder A to propagate the neural texture T , effectively generating tileable BTFs, as we show in Figure 4.3. This propagation algorithm can leverage state-of-the-art algorithms for tileable texture synthesis without any modification of our material model or training framework. Tileable BTFs were not achievable with previous approximations and this simple pipeline has the potential of enabling novel applications of this type of material representation in rendering scenarios.

Besides propagating BTF measurements to larger portions of the material, NeuBTF allows for generating novel materials using structural editions. Given a trained NeuBTF and a guidance image representing some particular target structure, we can propagate the neural texture to this guidance image, generating high-quality neural materials which preserve the structure of the guidance image and the reflectance properties of the trained neural material. This pipeline allows for easily generating multiple different neural materials without the need for retraining. As we show in Figure 4.8, this propagation method works for many types of input guidances, including vector black and white images, procedurally generated textures, or real photographs of materials and textures. We show results on acquired BTFs and from synthetic BTFs, rendered from scanned and manually generated SVBRDFs. As shown, our propagation frameworks provides high-quality material editions, even for very challenging cases, like the circles pattern.

Another useful application enabled by our method is the generation of materials at different resolutions. Unlike previous work [Kuz+21], which explicitly optimizes a pyramid of levels of detail during training, we can generate materials at any resolution at test time without introducing any additional complexities to our material representation. Because we train our models using random rescales as a data augmentation policy, they are equivariant to rescales of its input guidance images G↓:R(A(G))↓=R(A(G↓)). As such, we can generate any continuous resolution for a particular BTF by downsampling the guidance image to the target resolution and propagating its neural texture, as we illustrate in Figure 4.9. Note that this algorithm only guarantees accurate results for the rescaling ranges that we use during data augmentation.

Single Image SVBRDF Capture Capturing accurate SVBRDFs from a single image is a challenging problem that requires predicting the photometric response of a material given only a sample of it. The most common approximation is to use a flash-lit front planar image captured with a smartphone. Extending neural style transfer [GEB15b] to material capture, Aittala et al. [AAL16] leverage pre-trained CNNs and texture priors for smartphone material acquisition. Relatedly, Henzler et al. [Hen+21] use style losses for training generative models for BRDF synthesis. These approaches are optimized for synthesis, which allow for seamlessly tileable outputs, and do not require supervised training. However, they work best for stochastic materials, limiting their scope.

Leveraging datasets of labeled materials, different methods have trained autoencoders for SVBRDF capture. Li et al. [Li+17a] reconstruct spatially-varying albedo and normals using a U-Net [RFB15], and homogeneous specular albedo and roughness using a CNN regressor. This work was extended through Self-Augmented CNNs in [Ye+18]. By leveraging Conditional Random Fields (CRFs), a material classifier and one decoder per map, Li et al. [LSC18] reconstruct spatially-varying albedo, roughness, and normals. Deschaintre et al. [Des+18] propose a modified U-Net, synthetic datasets, and a render loss. Cascaded models [Li+18; SC20]; and deep latent spaces optimized using inverse rendering [Gao+19] have also shown success for this problem. Recently, Generative Adversarial Networks (GANs) have shown improved capabilities compared to more naive losses. These require a discriminator, which can be trained on renders [Wen+22], SVBRDF maps [Guo+21; VPS21], or both [ZK21].

Our approach differs from previous work in several factors. Importantly, we use flatbed scanners instead of smartphones for material capture. While they limit the materials which can be captured, they provide adequate illumination for easier digitizations, and a higher level of resolution and detail. We hypothesize that material specularity can be estimated accurately by leveraging its microgeometry. From this assumption, we build a GAN which, in contrast with previous work, leverages state-of-the-art attention mechanisms and discriminator design for obtaining a more holistic understanding of its inputs. Further, we train exclusively on real data, and propose a more comprehensive evaluation.

Multiple Image SVBRDF Capture A different corpus of work relies on multiple images of the material. By capturing more evidence of the material photometric response, they provide more accurate SVBRDFs. This hinders scalability, as they require a larger capture and calibration effort. The simplest approach is to require two samples, a diffuse and a flash-lit image [AWL15; Bos+20]. More flexible alternatives allow for more samples, combined with a learned prior [Guo+20b; Des+19]; or Monte Carlo rendering [Lua+21]. A different approach is to capture the material at a high resolution, and transfer those details to larger samples of the same material [DDB20; RG21]. Finally, videos for material acquisition have also shown to be promising capture systems [Ye+21].

Procedural Graphs Procedural graphs have also been used for material generation. Instead of relying on material priors, these approaches work by optimizing a material graph through a differentiable pipeline. These provide interesting capabilities, such as easy edition or tiling, but are limited by the expressiveness of the procedural model. These have been explored for general SVBRDF estimation [Shi+20; Hu+22c; Guo+20a], or for high-quality woven fabric digitizations [Jin+22].

Measuring the confidence of deep learning models is an active research area [KG17] with multiple applications, including safety-critical problems, like self-driving [Hua+19] or medicine [Kur+22]; and active dataset creation [Lei+21; Sol+21]. In computer vision, uncertainty quantification

has focused on image classification [SKK18; Lei+21], segmentation [Kry+21], and depth regression [Hu+22a; Ami+20]. To overcome the computational intractability of Bayesian Neural Networks, different approximations have been proposed, including MC Dropout [GG16], Deep Ensembles [LPB17] or Variational Inference [MNG17]. Orthogonal alternatives exist, including Evidential Deep Learning [Ami+20; SKK18] or frequentist approaches like Constrained Ordinal Regression [Hu+22a]. We refer the reader to recent surveys [Jos+22; Gaw+21] for more comprehensive reviews.

Quantifying uncertainties allows communicating the end user that the model predictions may be inaccurate, suggesting alternative pathways; as well as cheaper dataset creation through active learning. Bayesian material parameter estimation has been proposed for procedural frameworks [Guo+20a], but, to the best of our knowledge, it has not been explored for SVBRDF estimation. We aim to propose an efficient uncertainty quantification framework for deep SVBRDF capture methods which accounts for material perception and is agnostic to the material model.

We use a U-Net [RFB15] with residual connections [He+16; Dia+20; DLG21] in all of our convolutional blocks, an individual decoder per map [Gar+22; RG22; DLG21; ZK21], and group normalization [WH18]. To each decoder, we add a pixel-wise dropout-regularized MLP [Sri+14], aimed at increasing the accuracy of the predictions and allowing us to measure uncertainty at test-time.

While this multi-decoder residual U-Net is relatively accurate, it is limited by its receptive field, as is common on fully-convolutional architectures. Previous work [Des+18] proposed the use of a global track for fusing spatially distant information. We instead draw inspiration from recent advances in attention and diffusion models [Sah+22; HJA20; Rom+22], and add a self-attention module with linear complexity [Wan+20b] to the output of every convolutional block in the encoder. Finally, we add a lightweight MobileViT [MR21] to the bottleneck to provide the model with a global understanding of its input. By performing the most complex computations at the encoder, we provide the specialized decoders with dense inputs which are computed only once.

(5.1) LG=∑iλiLpixeli+λadvLadv+λstyleLstyle+λfreqLfreq

Lpixel is the L1 norm weighted per map, λ_i. L1 produces sharper results than higher-order alternatives, such as L2. We introduce an adversarial loss to handle the intrinsic ambiguity of ill-posed problems [Tex+20b; Iso+17; Gar+22; VPS21]. In our case, the choice of the discriminator is a critical design decision.

Recent work [SSK20] proposed U-Net architectures for discriminators, which allows for better learn both low and high-level features, and introduces further regularization. These result in more conservative albeit less diverse generations [Han+22a]. We use a U-Net discriminator [SSK20] with attention [Woo+18], which outputs two estimations: a scalar output Denc, provided by its encoder, and a 2D estimation Ddec, provided by its decoder. Denc provides a global estimate of the quality of the stack M, while Ddec gives pi estimations. As in [SSK20], we add a regularization term LDdeccons, and leverage cut-mix as for discriminator data-augmentation. Our discriminator and adversarial losses are:

(5.2) LD=LDenc+LDdec+λconsLDdeccons

(5.3) Ladv=log⁡(Denc(G(X))+log⁡(Ddec(G(X))

where the implementation of LDenc and LDdec follows [SSK20].

We further add two losses to improve the accuracy and sharpness of the results: a frequency loss Lfreq and a style loss Lstyle. Lfreq is estimated by averaging the focal frequency loss [Jia+21] computed over each individual channel of M. This is designed to help GANs preserve high-frequency details. Further, as shown in prior work, working in the frequency domain is beneficial when handling textures [AWL13; Mar+20]. Our style loss Lstyle is inspired by the success of neural losses when dealing with textures [GEB15b; GEB15a]. However, off-the-shelf metrics which are designed for 3-channel images, are not immediately usable in SVBRDF. While it is possible to compute them for each map separetely [RG22], this does not necessarily preserve inter-map consistency. We follow recent work [CHB21] and use LPIPS [Zha+18] taking as input a 3-channel image created by randomly sampling three channels from the set of five available channels of M.

We follow two strategies for data augmentation. First, we perform patch-based training and affine transforms with randomly cropped patches [RG21; Tex+20b; VPS21]. We also apply random rescales for generalization at lower resolutions, and rotations to account for possible misalignments that may arise when capturing the samples. Second, we apply several image transformations to increase model robustness: random intensity changes in HSV space to the inputs, gaussian noise and blurs, and random erasing [Zho+20] for regularization.

We gathered a novel dataset for training and testing our method. It comprises 2000 textile materials of a variety of families and microstructures that we divided into 14 families: crepe, jacquard, pile, plain, satin, and twill for wovens; fleece, french terry, interlock, jersey, milano, pique, and rib for knits; and, finally, leathers. Each family differs in its construction pattern

(i.e., its microstructure), which directly impacts its optical appearance. In the supplementary material, we show a more detailed analysis of the dataset. For each material, we have an image scanned with the flatbed scanner EPSON V850 whose lighting configuration is close to diffuse (as we show in Figure 5.3), and its corresponding ground truth specular maps of the SVBRDF (normals, specular, and roughness). To obtain these maps, first, we digitized the material with an optical gonioreflectometer and then, propagated the maps to the scan using map propagation techniques [RG21]. All our images and maps have a resolution of 1000 PPIs, allowing us to leverage the full semantics of the microstructure for inference. We split our dataset into 90-10 for train and test, making sure that every family is equally represented in both splits.

We quantify individual per-map accuracy, rendered perceptual accuracy, and artifacts. Per-map accuracy is computed differently depending on the semantics of the map: the roughness and specular maps are evaluated using Mean Absolute Error (L1), and the normals are evaluated using the angular distance in vector space (L<)). Further, to account for the possibility that the model always returns an accurate average value, resulting in relatively low L1, we also measure Pearson correlation ρ.

We evaluate rendered perceptual accuracy LBRDF between the ground truth stack MGT and the estimation M^ following existing metrics [Lav+21],

(5.5) LBRDF=1|xy|∑xy1|S|∑(l,v)∈Scos2⁡(θl)(fl,v(MGT,K)−fl,v(M^,K))23 where the terms are the same as in Equation 5.4.

Finally, we found that some architectural improvements in the neural network introduce artifacts in the specular and roughness maps that were not present in the input image, as illustrated in Figure 5.6. Thus, we provide an artifacts detection metric to quantify them. We start by defining a metric of homogeneity for an input image H(I),

(5.6) H(I)=1|xy|∑xy1|d|∑d={↑,↓,←,→}∥FBox(I)−FBox(Id)∥1 where I^d is the image shifted up, down, left, and right by a number of pixels equal to the kernel size of the box filter. Then, we define three metrics that we compute per map: e1(M)=H(M), e2(M)=H(M)H(X),ande3(M)=MI(X,M)−1. X is the original input image, and MI is the Mutual Information [Rus+04], which helps discern artifacts that appear in a single map from semantic patterns (e.g., plaids, prints). Each map is labeled as having artifacts if the majority of metrics exceed their corresponding thresholds, tm(M). If any of the maps contain artifacts, the entire stack M is classified as having artifacts.

In Figure 5.6 and Table 5.1, we present an ablation study to validate our model design. From a baseline U-Net [RFB15] trained with a pixel-wise loss and no data augmentation other than rescales, we add different components to the model to improve its generalization. First, we observe that using a PatchGAN [Iso+17; VPS21] discriminator provides a significant increase in accuracy. However, using a similarly-sized U-Net discriminator [SSK20], we achieve better results, particularly when using discriminator regularization. Further, Lstyle and Lfreq yield significant improvements, most notably in the normal map. To the baseline U-Net trained with the full loss, adding residual connections and one decoder per map increases accuracy. However, this setup tends to produce artifacts as it struggles to integrate global information. By adding self-attention to the encoder, we remove the artifacts; and with the MobileViT [MR21], we achieve higher quality results. Our final model contains full data augmentation, which provides small gains on generalization.