Simulations serve as an indispensable tool for designers, scientists, and
engineers, enabling them to **predict behavior of complex systems**. Whether
it’s optimizing machinery, safeguarding engineering designs, or forecasting
dynamic environmental conditions, simulations provide deep insights needed to
make informed decisions.

Currently, the landscape of modeling and simulation is undergoing a profound
transformation, driven by the advances in machine learning. The emerging
discipline of **scientific machine learning** (SciML) is changing the way we
think about modeling physical systems by combining traditional simulation
methods with ML algorithms, particularly deep learning. Combining data with
physical knowledge opens up previously uncharted opportunities, paving the way
for innovations and insights within complex systems with unprecedented depth,
precision and efficiency.

# Physics-based & Data-driven

In exploring the landscape of scientific machine learning, we traverse a
**spectrum** that ranges from purely physics-based approaches to entirely
data-driven methodologies [Kar21P].

### Physics-based modeling

Traditionally, the cornerstone of simulating complex systems has been
physics-based modeling. Rooted in centuries of scientific discovery, these
models are crafted around **well-understood physical laws**, often articulated
through differential equations. They offer a deterministic perspective, where
given a set of initial conditions and parameters, the outcome is predictable and
consistent. Their strength lies in their accuracy under well-defined conditions
and the interpretability they bring, anchored in established physical
principles. However, their limitation becomes evident in scenarios with unknown
parameters, missing boundary conditions, or when simulating over large scales
becomes computationally prohibitive.

### Data-driven modeling

Parallel to this, the digital age brought forth an unprecedented deluge of data,
giving rise to data-driven modeling. Instead of relying exclusively on
pre-established physical laws, these models **learn patterns from data**.
Machine learning, especially deep learning, has been at the forefront of this
shift, enabling models to uncover intricate patterns in massive datasets. While
data-driven models excel in situations with abundant data, offering the ability
to capture nonlinearities and complex interdependencies, they often lack the
interpretability of their physics-based counterparts. Moreover, they can be
sensitive to the quality and quantity of the data they are trained on.

### Physics-informed machine learning

Recognizing the strengths and weaknesses of both aforementioned paradigms, a new
approach has been gaining momentum: physics-informed machine learning. This
framework aims to bridge the gap by **integrating physical knowledge into
machine learning** architectures. It’s more than just a combination, but a deep
fusion where models are informed and constrained by physical laws, ensuring
consistency with known principles. The result is a synergistic approach where
models benefit from the flexibility and adaptability of machine learning while
maintaining a grounding in physical reality. This not only ensures better
generalizability but also opens avenues for innovations in scenarios where data
is sparse yet some physical understanding exists. Physics-informed ML approaches
offer distinct foundational advantages, including:

**Speed & Efficiency**: ML-enhanced methods can outpace traditional simulations by orders of magnitude.**Enhanced Accuracy**: Some methods have shown superior predictive capabilities by leveraging data.**Data-Driven Insight**: With a foundation in data, ML-based methods provide insights beyond known physics and reveal unseen patterns.

# Advancements & Methods

The field of scientific machine learning, especially in its endeavor to model physical systems, has seen several groundbreaking techniques emerge. These advancements, which incorporate a blend of physics and deep learning, are reshaping the way we approach complex simulations and predictions.

## Physics-Informed Neural Networks

Physics-informed neural networks (PINNs) have emerged as a groundbreaking approach where a neural network is trained with respect to a loss function that incorporates physical laws, often represented as differential equations [Rai19P]. They seamlessly integrate the underlying physics of a problem with the representational power of neural networks and are thereby striking a balance between data-informed insights and physically grounded knowledge. They can be used for a variety of tasks, e.g., solving PDEs, incorporating measurement data or parameter estimation. Our blog post on Solving PDEs With Neural Networks presents the approach in more detail, and a recent review of PINNs can be found in [Cuo22S].

## Data-Driven Discovery of Physical Laws

Many methods in scientific machine learning can be utilized to unearth
underlying physical laws purely from data.
These approaches can highlight previously unknown
relationships and principles, especially in systems with limited prior
knowledge. Notable methods include **PINNs for inverse problems**
[Rai19P], sparse
identification of nonlinear dynamical systems
[Bru22D], symbolic regression
[Cra20D], or Equation Learner
[Sah18L].

## Graph Neural Networks

Specialized neural network architectures, particularly graph neural networks
(GNNs), are designed to process structured data and can
been applied to a variety of problems.
In the context of physical modeling, they
have been effective in **particle-based simulations** [San20L] and purely **data-driven weather forecasting**
[Lam23G].

## Operator Learning

Emerging in the field of scientific machine learning, operator learning
represents a cutting-edge approach that leverages machine learning algorithms
to **approximate complex mathematical operators**, opening new avenues for
efficiently and accurately solving differential equations and related
mathematical challenges.
For instance, **DeepONets** are a class of neural networks that are able to
approximate operators of functions [Lu21L]. This makes
them particularly powerful for problems like capturing the solution operators
of parameterized differential equations, where inputs can be entire functions,
allowing to generate outcomes more efficiently. See our paper pills
Learning non-linear operators and
physics-informed DeepONets for more details.
Moreover, built to accelerate simulations, **Fourier Neural Operators**
were proposed to marry Fourier transformations with neural networks by
translating data into the frequency domain. In geometrically rather simple
settings, they have demonstrated the potential to speed up fluid dynamical
simulations by orders of magnitude [Kov23N].
Another example is **BelNet**, a neural network architecture that is
discretization-invariant as it learns basis for interpolation of the input
functions [Zha23B].

# Tools & Frameworks

With the growth of ML in simulations, an array of tools and frameworks have
emerged. Many projects are based on classical deep learning frameworks like
PyTorch, Tensorflow or JAX. A non-exhaustive list of **notable frameworks** for
scientific ML includes
DeepXDE,
Modulus,
torchphysics,
SciML,
Neural Operator, or
SciANN.

We are also developing our own Python library, **continuiti**,
for learning function operators with neural networks. It is a high-level library
for (physics-informed) operator learning based on PyTorch, and it includes
implementations of common neural operator architectures (like DeepONets or FNOs),
tools for physics-informed training and several benchmarks.

# Challenges & Limitations

As promising as the merger of machine learning with physics-based simulations might be, it comes with its own set of challenges and limitations:

**Theoretical Foundations:**The integration of ML models, especially deep learning, into physical simulations often lacks a robust theoretical foundation. This can lead to models that work empirically but are hard to rationalize.**Model Complexity:**Deep learning architectures, by nature, can become overly complex. Training and fine-tuning them, especially in tandem with physical simulations, can be resource-intensive and time-consuming.**Data Scarcity:**Even though some models are data-driven, they still require substantial amounts of high-quality training data. In many real-world scenarios, obtaining such data is challenging.**Generalizability:**While models like PINNs offer a blend of data-driven insights and physical laws, ensuring that they generalize well across diverse scenarios remains a hurdle.**Architecture Dependency:**The right architecture is often problem-dependent. Finding the optimal structure for a specific problem can be difficult.**Interpretability:**Deep learning models, when integrated with physics-based simulations, can become “black boxes”. This makes understanding the internal workings of the model, especially in critical applications, difficult.

# Future & Perspectives

The fusion of machine learning with traditional modeling methods has initiated a paradigm shift in how we approach simulations and predictions of physical systems. This integration offers a powerful toolkit for engineers, scientists, and practitioners, enabling them to harness both the rigor of physical laws and the adaptability of data-driven techniques. For those at the intersection of engineering, physics, and computational sciences, this evolving landscape offers unprecedented opportunities to innovate, optimize, and redefine the boundaries of what’s possible. The future, illuminated by this synergy, holds immense promise and potential, and we stand at the cusp of a new era in modeling and simulation.