Simulations serve as an indispensable tool for designers, scientists, and engineers, enabling them to predict behavior of complex systems. Whether it’s optimizing machinery, safeguarding engineering designs, or forecasting dynamic environmental conditions, simulations provide deep insights needed to make informed decisions.

Currently, the landscape of modeling and simulation is undergoing a profound transformation, driven by the advances in machine learning. The emerging discipline of scientific machine learning (SciML) is changing the way we think about modeling physical systems by combining traditional simulation methods with ML algorithms, particularly deep learning. Combining data with physical knowledge opens up previously uncharted opportunities, paving the way for innovations and insights within complex systems with unprecedented depth, precision and efficiency.

Physics-based & Data-driven

In exploring the landscape of scientific machine learning, we traverse a spectrum that ranges from purely physics-based approaches to entirely data-driven methodologies [Kar21P].

Physics-based modeling

Traditionally, the cornerstone of simulating complex systems has been physics-based modeling. Rooted in centuries of scientific discovery, these models are crafted around well-understood physical laws, often articulated through differential equations. They offer a deterministic perspective, where given a set of initial conditions and parameters, the outcome is predictable and consistent. Their strength lies in their accuracy under well-defined conditions and the interpretability they bring, anchored in established physical principles. However, their limitation becomes evident in scenarios with unknown parameters, missing boundary conditions, or when simulating over large scales becomes computationally prohibitive.

Data-driven modeling

Parallel to this, the digital age brought forth an unprecedented deluge of data, giving rise to data-driven modeling. Instead of relying exclusively on pre-established physical laws, these models learn patterns from data. Machine learning, especially deep learning, has been at the forefront of this shift, enabling models to uncover intricate patterns in massive datasets. While data-driven models excel in situations with abundant data, offering the ability to capture nonlinearities and complex interdependencies, they often lack the interpretability of their physics-based counterparts. Moreover, they can be sensitive to the quality and quantity of the data they are trained on.

Physics-informed machine learning

Recognizing the strengths and weaknesses of both aforementioned paradigms, a new approach has been gaining momentum: physics-informed machine learning. This framework aims to bridge the gap by integrating physical knowledge into machine learning architectures. It’s more than just a combination, but a deep fusion where models are informed and constrained by physical laws, ensuring consistency with known principles. The result is a synergistic approach where models benefit from the flexibility and adaptability of machine learning while maintaining a grounding in physical reality. This not only ensures better generalizability but also opens avenues for innovations in scenarios where data is sparse yet some physical understanding exists. Physics-informed ML approaches offer distinct foundational advantages, including:

• Speed & Efficiency: ML-enhanced methods can outpace traditional simulations by orders of magnitude.
• Enhanced Accuracy: Some methods have shown superior predictive capabilities by leveraging data.
• Data-Driven Insight: With a foundation in data, ML-based methods provide insights beyond known physics and reveal unseen patterns.

Advancements & Methods

The field of scientific machine learning, especially in its endeavor to model physical systems, has seen several groundbreaking techniques emerge. These advancements, which incorporate a blend of physics and deep learning, are reshaping the way we approach complex simulations and predictions.

Physics-Informed Neural Networks

Physics-informed neural networks (PINNs) have emerged as a groundbreaking approach where a neural network is trained with respect to a loss function that incorporates physical laws, often represented as differential equations [Rai19P]. They seamlessly integrate the underlying physics of a problem with the representational power of neural networks and are thereby striking a balance between data-informed insights and physically grounded knowledge. They can be used for a variety of tasks, e.g., solving PDEs, incorporating measurement data or parameter estimation. Our blog post on Solving PDEs With Neural Networks presents the approach in more detail, and a recent review of PINNs can be found in [Cuo22S].

Data-Driven Discovery of Physical Laws

Many methods in scientific machine learning can be utilized to unearth underlying physical laws purely from data. These approaches can highlight previously unknown relationships and principles, especially in systems with limited prior knowledge. Notable methods include PINNs for inverse problems [Rai19P], sparse identification of nonlinear dynamical systems [Bru22D], symbolic regression [Cra20D], or Equation Learner [Sah18L].

Graph Neural Networks

Specialized neural network architectures, particularly graph neural networks (GNNs), are designed to process structured data and can been applied to a variety of problems. In the context of physical modeling, they have been effective in particle-based simulations [San20L] and purely data-driven weather forecasting [Lam23G].

Operator Learning

Emerging in the field of scientific machine learning, operator learning represents a cutting-edge approach that leverages machine learning algorithms to approximate complex mathematical operators, opening new avenues for efficiently and accurately solving differential equations and related mathematical challenges. For instance, DeepONets are a class of neural networks that are able to approximate operators of functions [Lu21L]. This makes them particularly powerful for problems like capturing the solution operators of parameterized differential equations, where inputs can be entire functions, allowing to generate outcomes more efficiently. See our paper pills Learning non-linear operators and physics-informed DeepONets for more details. Moreover, built to accelerate simulations, Fourier Neural Operators were proposed to marry Fourier transformations with neural networks by translating data into the frequency domain. In geometrically rather simple settings, they have demonstrated the potential to speed up fluid dynamical simulations by orders of magnitude [Kov23N]. Another example is BelNet, a neural network architecture that is discretization-invariant as it learns basis for interpolation of the input functions [Zha23B]. Regarding real-world applications where measurements of a system are often sparsely and irregularly distributed due to the geometries of the domain, environmental conditions, or unstructured meshes, mesh-independent operator learning can be considered, e.g., the mesh-independent neural operator (MINO) [Lee22aM].

Tools & Frameworks

With the growth of ML in simulations, an array of tools and frameworks have emerged. Many projects are based on classical deep learning frameworks like PyTorch, Tensorflow or JAX. A non-exhaustive list of notable frameworks for scientific ML includes DeepXDE, Modulus, torchphysics, SciML, Neural Operator, or SciANN.

We are also developing our own Python library, continuiti, for learning function operators with neural networks. It is a high-level library for (physics-informed) operator learning based on PyTorch, and it includes implementations of common neural operator architectures (like DeepONets or FNOs), tools for physics-informed training and several benchmarks.

Challenges & Limitations

As promising as the merger of machine learning with physics-based simulations might be, it comes with its own set of challenges and limitations:

• Theoretical Foundations: The integration of ML models, especially deep learning, into physical simulations often lacks a robust theoretical foundation. This can lead to models that work empirically but are hard to rationalize.
• Model Complexity: Deep learning architectures, by nature, can become overly complex. Training and fine-tuning them, especially in tandem with physical simulations, can be resource-intensive and time-consuming.
• Data Scarcity: Even though some models are data-driven, they still require substantial amounts of high-quality training data. In many real-world scenarios, obtaining such data is challenging.
• Generalizability: While models like PINNs offer a blend of data-driven insights and physical laws, ensuring that they generalize well across diverse scenarios remains a hurdle.
• Architecture Dependency: The right architecture is often problem-dependent. Finding the optimal structure for a specific problem can be difficult.
• Interpretability: Deep learning models, when integrated with physics-based simulations, can become “black boxes”. This makes understanding the internal workings of the model, especially in critical applications, difficult.

Future & Perspectives

The fusion of machine learning with traditional modeling methods has initiated a paradigm shift in how we approach simulations and predictions of physical systems. This integration offers a powerful toolkit for engineers, scientists, and practitioners, enabling them to harness both the rigor of physical laws and the adaptability of data-driven techniques. For those at the intersection of engineering, physics, and computational sciences, this evolving landscape offers unprecedented opportunities to innovate, optimize, and redefine the boundaries of what’s possible. The future, illuminated by this synergy, holds immense promise and potential, and we stand at the cusp of a new era in modeling and simulation.