Learning function operators has gained significant attention in recent years, particularly in the context of solving parametric partial differential equations (PDEs). Prominent architectures include DeepONets [Lu21L] and Fourier neural operators (FNO) [Kov23N] with different properties, advantages or trade-offs.
An ideal operator learning framework should not require a particular choice of discretization for the input functions, allowing for the input and output functions to be on different domains, and being able to have different grids between samples.
A recent paper [Zha23B] outlines these challenges in operator learning nicely and introduces the following definitions of discretization-invariant, prediction-free, and domain-independent neural operators.
Fig. 5 [Zha23B] Illustration of domain-independence.Properties
Discretization-invariant: Locations of sensors in the input function domain are not fixed. This is important for unstructured input data, e.g., meshes.
Prediction-free: Evaluation point can be any point in the output function domain. This enables, e.g., physics-informed training or super-resolution.
Domain-independent: Output function domain is independent of input function domain. Provides much more flexibility, e.g., map boundary condition to solution.
Interestingly, the paper compares the properties of DeepONet and FNO, see Table 1, and shows that neither DeepONets nor FNOs satisfy all three properties.
| Discretization-invariant | Prediction-free | Domain-independent | |
|---|---|---|---|
| DeepONet | 🚫 | ✅ | ✅ |
| FNO | ✅ | 🚫 | 🚫 |
| BelNet | ✅ | ✅ | ✅ |
BelNet
The basis enhanced learning network (BelNet) [Zha23B], a new neural operator architecture proposed in the paper, is designed to meet these requirements. The derivation starts from the general neural operator framework [Kov23N], assumes a specific kernel structure, and uses quadrature rules to approximate the integral. In the end, the architecture introduces projection nets that project the input function into a latent space of learned basis functions.
Fig. 7 [Zha23B] Plots of two solutions to the viscous
Burgers’ equation with our initialization procedure. Note that each example’s
sampling points (black dots) for the initial condition differ. The yellow curves
are used to generate the initial conditions for the model problem (viscous
Burgers’ equation). The initial conditions for the viscous Burgers’ equation are
displayed in blue.
Fig. 6 [Zha23B] Illustration of the network structure: Projection nets are $K$ independent fully connected neural network with weights and bias $W^{2,k}_y \in \mathbb{R}^{N \times N_1}$, $W^{1,k}_y \in \mathbb{R}^{N_1 \times N}$ and $b^k_y \in \mathbb{R}^{N_1}$. Construction net is a fully connected neural network with weights and bias bias $Q \in \mathbb{R}^{K \times d}$ and $b_x \in \mathbb{R}^d.$ Here, $Q = [q^1, q^2, \dots, q^K]$, where $q^i$ are defined in equation (3.3). In addition, $a_x, a_y, a_u$ are activation functions.
BelNet is both a generalization of the networks proposed in [Che95U] (aka DeepONets [Lu21L]) and Fourier neural operators [Kov23N] by introducing a learnable set of basis functions for projection.
Through several challenging high-contrast and multi-scale problems, the paper shows that the approach outperforms other operator learning methods for these tasks and allows for more freedom in the sampling and discretization process.
Note that the universal approximation theorem for BelNet should follow from a generalization of [Che95U], but one needs to show that the approximation does not require fixed or similar grids.
Conclusion
The paper is a great outline of independence of discretization in operator learning, but particularly the idea of learning a projection of the input function into a latent space is an interesting one, and it will be exciting to see how this approach develops in the future.
If you are interested in trying out BelNet in a generalized framework, check out continuiti, our Python package for learning function operators with neural networks.
