2023 - Paper Copilot

DANI-Net: Uncalibrated Photometric Stereo by Differentiable ShadowHandling, Anisotropic Reflectance Modeling, and Neural Inverse Rendering

Zongrui Li, Qian Zheng, Boxin Shi, Gang Pan, Xudong Jiang

Nanyang Technological University; Zhejiang University; Peking University

Portals

Abstract

Uncalibrated photometric stereo (UPS) is challenging due to the inherent ambiguity brought by the unknown light. Although the ambiguity is alleviated on non-Lambertian objects, the problem is still difficult to solve for more general objects with complex shapes introducing irregular shadows and general materials with complex reflectance like anisotropic reflectance. To exploit cues from shadow and reflectance to solve UPS and improve performance on general materials, we propose DANI-Net, an inverse rendering framework with differentiable shadow handling and anisotropic reflectance modeling. Unlike most previous methods that use non-differentiable shadow maps and assume isotropic material, our network benefits from cues of shadow and anisotropic reflectance through two differentiable paths. Experiments on multiple real-world datasets demonstrate our superior and robust performance.

Related Works

Unsupervised calibrated photometric stereo; Uncalibrated photometric stereo; Shadow handling in photometric stereo; Neural reflectance representation in 3D vision

Comparisons

LL22, SCPS-NIR

Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold

Xingang Pan, Ayush Tewari, Thomas Leimkühler, Lingjie Liu, Abhimitra Meka, Christian Theobalt

Portals

Abstract

Synthesizing visual content that meets users' needs often requires flexible and precise controllability of the pose, shape, expression, and layout of the generated objects. Existing approaches gain controllability of generative adversarial networks (GANs) via manually annotated training data or a prior 3D model, which often lack flexibility, precision, and generality. In this work, we study a powerful yet much less explored way of controlling GANs, that is, to "drag" any points of the image to precisely reach target points in a user-interactive manner, as shown in Fig.1. To achieve this, we propose DragGAN, which consists of two main components: 1) a feature-based motion supervision that drives the handle point to move towards the target position, and 2) a new point tracking approach that leverages the discriminative generator features to keep localizing the position of the handle points. Through DragGAN, anyone can deform an image with precise control over where pixels go, thus manipulating the pose, shape, expression, and layout of diverse categories such as animals, cars, humans, landscapes, etc. As these manipulations are performed on the learned generative image manifold of a GAN, they tend to produce realistic outputs even for challenging scenarios such as hallucinating occluded content and deforming shapes that consistently follow the object's rigidity. Both qualitative and quantitative comparisons demonstrate the advantage of DragGAN over prior approaches in the tasks of image manipulation and point tracking. We also showcase the manipulation of real images through GAN inversion.

Related Works

Generative Models for Interactive Content Creation; Unconditional GANs; Conditional GANs; Controllability using Unconditional GANs; 3D-aware GANs; Diffusion Models; Point Tracking

BakedSDF: Meshing Neural SDFs for Real-time View Synthesis

Error: Cannot create object

Tag: 2023

Portals

Abstract

Related Works

Comparisons

Portals

Abstract

Related Works