Physics-Informed Neural Networks
Why Embedding Physical Laws into Machine Learning Changes Everything

How We Got Here: A Decade of PINN Progress

The 2019 Journal of Computational Physics paper, the peer-reviewed consolidation of the 2017 arXiv preprints, had accumulated 28,581 citations on Google Scholar as of August 2026, making it one of the most cited papers in computational science this decade. The 2021 Nature Reviews Physics survey by Karniadakis et al. had reached 8,009 citations and 149,000 page accesses. These numbers are not given to impress; they are given to contextualize. A field that grows this fast also generates failure modes, corrections, and methodological revisions at the same rate.

The most useful lens for reading this history is not chronological but causal: almost every major methodological advance was a direct response to a documented failure mode. The timeline below traces the failure → diagnosis → fix pattern that repeats across the decade.

• Nov 2017, Foundational preprints

  Raissi–Perdikaris–Karniadakis submit Parts I and II to arXiv (arXiv:1711.10561; arXiv:1711.10566). Part I covers forward PDE problems using physics as the training signal. Part II covers data-driven discovery of governing equations. The residual-loss formulation is introduced in essentially its current form.

• 2018–2019, Consolidation and peer review

  The Journal of Computational Physics article (DOI: 10.1016/j.jcp.2018.10.045) establishes the widely-cited peer-reviewed formulation for forward and inverse problems with nonlinear PDEs. Sirignano & Spiliopoulos publish the Deep Galerkin Method (DGM) as a complementary mesh-free solver. The basic toolkit is in place.

• 2019–2021, Reproducible software ecosystems

  Lu, Meng, Mao, and Karniadakis release DeepXDE (SIAM Review 2021), making production-quality PINN workflows accessible in a dozen lines of Python. NeuroDiffEq provides a PyTorch alternative. NVIDIA's Modulus (later PhysicsNeMo) scales the approach to industrial geometries and is subsequently open-sourced.

• 2020–2021, First wave of variants, responding to scaling failures

  Domain decomposition variants emerge: conservative PINNs (cPINN), Extended PINNs (XPINN), and variational PINNs (VPINN / hp-VPINN). Bayesian PINNs (B-PINN) add uncertainty quantification. Self-adaptive loss weighting methods appear. Each variant targets a specific failure of vanilla PINNs on stiff or multi-domain problems.

• 2021, Formal diagnosis of training failure

  Wang, Teng, and Perdikaris (SIAM J. Sci. Comput.) prove that gradient flow pathologies cause one loss term to dominate training, leaving the others ignored. Wang, Yu, and Perdikaris apply the Neural Tangent Kernel (NTK) to explain when and why PINNs fail to train. Krishnapriyan et al. (NeurIPS) demonstrate that vanilla PINNs can fail on simple convection and reaction problems due to ill-conditioned loss landscapes, a significant finding that temperes the field's early optimism.

• 2022–2023, Adaptive sampling and causal training

  Residual-Based Adaptive Refinement (RAR) and comprehensive non-adaptive/adaptive sampling studies (Wu et al., CMAME 2023) address the collocation coverage problem. Causal training (Wang, Sankaran, Perdikaris) resolves the non-causal propagation issue in time-dependent PDEs, where the network learns later times before earlier ones are well-resolved. Wang et al. publish the "Expert's Guide to Training PINNs" (arXiv:2308.08468), consolidating practitioner knowledge.

• 2023–2025, New architectures and theoretical maturity

  PirateNets (Wang et al., 2024) address training instability via adaptive residual connections. Physics-informed Kolmogorov–Arnold Networks (PIKANs, Liu et al. 2024/2025) explore alternative function approximators with interpretability advantages. De Ryck and Mishra publish a comprehensive numerical analysis of PINNs in Acta Numerica 33 (2024). PINN–neural operator hybrids bridge single-instance solving and function-space learning.

• 2024–2026, The operator learning convergence

  Physics-informed DeepONet and physics-informed Fourier Neural Operators merge the PINN residual constraint with operator-learning architectures that generalize across problem families rather than solving a single instance. The field splits productively: vanilla PINNs remain valuable for inverse problems and sparse-data fusion; operator methods dominate repeated-query forward problem settings.

What PINNs can do that classical solvers cannot

Mesh-free computation

Creating a quality mesh for a complex 3D engineering geometry, a pharmaceutical tablet, a turbine blade, a reactor with internal baffles, can take days of preprocessing effort and requires specialized domain expertise. PINNs learn a continuous function over the domain without discretization, reducing geometry-to-simulation time dramatically.

Inverse problems without an outer optimization loop

In classical PDE solvers, estimating an unknown parameter (a reaction rate constant, a diffusion coefficient, a thermal conductivity) requires solving a forward problem thousands of times inside an outer optimization loop, computationally expensive and prone to getting stuck in local minima. In a PINN, the unknown parameter becomes an additional trainable variable in the same loss function and the same training loop. The forward solve and the parameter estimation happen simultaneously.

Data assimilation from sparse measurements

Adding 10 noisy experimental observations to the loss function, as a data term alongside the physics term, allows the PINN to incorporate real measurements while the physics automatically regularizes the solution between measurement points. The result is a physically consistent reconstruction that a pure data-driven model could not achieve with only 10 points.

Continuous and differentiable everywhere

A classical finite difference solution is a lookup table, values at fixed grid points. A PINN is a differentiable function defined for any $x$ in the domain. It can be evaluated at arbitrary resolution without retraining. Derivatives, integrals, and sensitivity analyses over the domain are all available analytically from the network itself.

Why Traditional ML Struggles in Engineering

Consider a pharmaceutical manufacturing reactor. Every critical variable, temperature, pH, dissolved oxygen, product concentration, is governed by conservation equations derived over a century of chemistry and engineering. Yet when you try to model it with a standard neural network, the model sees only patterns in numbers. It has no knowledge that energy must be conserved, that concentrations cannot go negative, or that the dynamics follow the Arrhenius equation. Train on one batch and it extrapolates to another with physically impossible predictions.

This is not a data quality problem or a model capacity problem. It is a structural mismatch: purely data-driven models cannot extrapolate through the laws of physics they were never shown. Classical numerical solvers know the physics but require complete specification of boundary conditions, parameters, and geometry, and collapse when any of those are uncertain or unavailable. Physics-Informed Neural Networks were built to occupy the middle ground.

The solution: embed physics directly into the training objective

The fix is not more data. It is making the differential equation part of the model itself. A standard neural network minimises a single data-misfit loss. A Physics-Informed Neural Network (PINN) minimises three terms simultaneously:

A standard neural network learns a mapping $f_\theta: x \rightarrow y$ by minimizing the prediction error on labeled outputs:

A Physics-Informed Neural Network learns the same mapping, but adds the requirement that whatever the network predicts must also satisfy the governing differential equation. The training objective becomes:

In many engineering applications, and in the first example below, there are no labeled observations at all. The data term vanishes entirely, and the differential equation is the training signal.

A concrete example: the simplest first-order ODE

Consider:

Analytical solution: $y(x) = e^{-x}$. In the PINN approach, the network $\hat{y}(x)$ is evaluated at a set of collocation points $\{x_i\}$ scattered across the domain. At each point, the ODE residual is computed:

The physics loss forces this residual to zero across the entire domain:

The boundary condition loss pins the initial value:

No labeled $(x, y)$ pairs are needed anywhere in this training procedure. The differential equation is the training signal.

Why automatic differentiation makes this possible

Enforcing $\frac{d\hat{y}}{dx} + \hat{y} = 0$ requires computing the derivative of the network's output with respect to its input. Three methods exist for this, with dramatically different tradeoffs:

When you compute $\hat{y} = f_\theta(x)$ in PyTorch with requires_grad=True on the input $x$, PyTorch constructs a computational graph recording every operation. Calling torch.autograd.grad() traverses this graph in reverse, applying the chain rule exactly, to machine floating-point precision. No mesh, no approximation, no truncation error.

The key parameter is create_graph=True on the first derivative call. Without it, PyTorch frees the intermediate graph and second derivatives become unavailable. With it, the derivative itself is a differentiable expression, enabling nested calls for $\frac{d^2\hat{y}}{dx^2}$, Laplacians, and beyond.

# Demonstrate exact automatic differentiation
# f(x) = x³ + 2x² − 5x + 1  →  f'(x) = 3x² + 4x − 5  →  f''(x) = 6x + 4

x = torch.tensor([2.0], requires_grad=True)

f = x**3 + 2*x**2 - 5*x + 1

# First derivative, create_graph=True keeps the graph alive for second pass
dfdx = torch.autograd.grad(f, x, create_graph=True)[0]

# Second derivative
d2fdx2 = torch.autograd.grad(dfdx, x)[0]

# Results at x=2: f=7, f'=15, f''=16  (all exact)
print(f.item(), dfdx.item(), d2fdx2.item())
# Output: 7.0  15.0  16.0

When the simple polynomial above is replaced by the full neural network $f_\theta(x)$, autograd can still propagate the chain rule through all layers, activation functions, and weight matrices, automatically and exactly. This is the mathematical foundation that makes PINNs possible.

Network architecture

For the first-order ODE problem, a compact architecture is sufficient: one input node (the scalar $x$), three hidden layers of 32 neurons each with Tanh activations, and a single output (the predicted $\hat{y}$). This yields 2,209 trainable parameters.

Why Tanh? Tanh is infinitely differentiable everywhere on $\mathbb{R}$. ReLU and its variants have zero second derivative almost everywhere, which breaks the higher-order autograd passes required for second-order ODEs and PDEs. Xavier normal initialization is used to ensure the gradients neither vanish nor explode at the start of training.

class PINN(nn.Module):
    def __init__(self, layers):
        super().__init__()
        network = []
        for i in range(len(layers) - 1):
            network.append(nn.Linear(layers[i], layers[i+1]))
            if i < len(layers) - 2:
                network.append(nn.Tanh())
        self.net = nn.Sequential(*network)
        self._init_weights()

    def _init_weights(self):
        for m in self.net:
            if isinstance(m, nn.Linear):
                nn.init.xavier_normal_(m.weight)
                nn.init.zeros_(m.bias)

    def forward(self, x):
        return self.net(x)

# Architecture: 1 → 32 → 32 → 32 → 1  (2,209 parameters)
model = PINN([1, 32, 32, 32, 1]).to(device)

Collocation points

Collocation points are the locations in the domain where the ODE residual is evaluated and penalized. They require no labels, only the $x$ coordinates. The key requirement is that requires_grad_(True) is set so that PyTorch can differentiate the network output with respect to these inputs.

The training loop and actual convergence

def ode_residual(model, x):
    y_hat = model(x)
    dy_dx = torch.autograd.grad(
        y_hat, x,
        grad_outputs=torch.ones_like(y_hat),
        create_graph=True     # required for loss to be differentiable w.r.t. weights
    )[0]
    return dy_dx + y_hat     # r(x) = dy/dx + y  →  should be zero

def total_loss(model, x_col, x_bc, y_bc):
    r = ode_residual(model, x_col)
    loss_physics = torch.mean(r**2)
    y_bc_pred = model(x_bc)
    loss_bc = torch.mean((y_bc_pred - y_bc)**2)
    return loss_physics + loss_bc

optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
for epoch in range(5000):
    optimizer.zero_grad()
    loss = total_loss(model, x_col, x_bc, y_bc)
    loss.backward()
    optimizer.step()

Training output from the notebook, showing convergence over 5,000 epochs:

Notice the characteristic pattern: the BC loss drops rapidly to near-zero within the first few hundred epochs, the network quickly learns to satisfy the single initial condition $y(0)=1$. The physics loss then dominates and drives the slow, steady improvement as the network learns to satisfy the ODE residual across the entire domain.

Practical examples: the equations PINNs are built to solve

Now that you have seen a PINN train from scratch, here are the governing equations it was designed to handle in practice. Each one is a case where the physics is trusted but something, a parameter, a boundary condition, a full state trajectory, is unknown and must be recovered from data.

Energy balance in a CSTR:

Diffusion-reaction in a catalyst pellet:

Navier-Stokes for incompressible flow:

In every case, a purely data-driven model trained on historical runs will violate these relationships the moment it extrapolates. A PINN enforces them explicitly, through residual terms in the training loss, so the predictions remain physically consistent even outside the training distribution.

The Research Landscape: How Major Reviews Compare

If you want to go deeper into the literature, knowing which review covers what is essential. Eight major PINN/PIML reviews have shaped the field since 2021. The table below maps each against eight evaluation criteria, not to rank them, but to help you navigate to the right source for your question.

The coverage audit was conducted by reading each paper's abstract, table of contents, section headings, and representative body text. Ratings: ● deep (major organising component); ◐ partial (meaningful but not central); ○ absent or minimal.

The Two Problems PINNs Were Built to Solve

Raissi, Perdikaris, and Karniadakis did not set out to build a general-purpose AI framework. They submitted two companion papers on the same day in November 2017, targeting two specific failure modes in the existing toolkit. Part I: Data-driven Solutions and Part II: Data-driven Discovery, published together on arXiv, established a vocabulary that still organizes the field today.

The core distinction is not about which equation you are solving. It is about what is known and what is unknown when you start. This changes everything downstream: the loss function structure, the validation strategy, the identifiability diagnostics you need, and the honest expectations you should set.

Beyond the two archetypes

Reviews and primary studies document PINNs applied to state reconstruction from sparse sensor data, surrogate modeling where training cost is amortized over many evaluations, partially-known physics where some equation terms are uncertain, and multi-fidelity learning from both high-cost simulations and cheap experimental data. The value proposition is genuinely different in each regime. The honest verdict, developed in the sections on field evolution and limitations below, is that PINNs can reduce data requirements and improve physical plausibility when the constraints are informative, correctly specified, and trainable. They do not automatically deliver robust identification or solver-grade reliability across every problem class.

The important idea is not that neural networks replace physics. The important idea is that physics becomes part of the neural network. For decades, engineers described systems through differential equations. For the last decade, AI described systems through data. PINNs represent the convergence of these two traditions.

In this article, we built a network that solved $dy/dx = -y$ to better than $10^{-3}$ absolute error using zero labeled data, with a 2,209-parameter network, 5,000 training epochs, and the ODE itself as the sole training signal. The BC loss vanished to $10^{-11}$ while the physics loss drove systematic improvement across the entire domain.

Where PINNs Currently Struggle

Most introductory articles stop after the success stories. That would be a disservice to anyone planning to deploy PINNs in production. Here are the well-documented failure modes, with concrete mitigations where they exist.

The decade of research reviewed above has converged on a clear picture: PINNs are not general-purpose solvers. They are a specialized tool that excels in specific regimes and fails predictably in others. The following are not edge cases, they are the typical failure modes documented in peer-reviewed studies across multiple research groups.

Training instability and gradient pathologies

The PINN loss landscape is non-convex and poorly conditioned. Gradients from the physics loss and the boundary condition loss can point in conflicting directions, causing oscillations or premature plateauing. Wang, Teng, and Perdikaris (2021) proved this formally: when PDE-residual and boundary/data losses are naïvely combined, one term dominates the gradient flow and the other is effectively ignored during training. The follow-up NTK analysis (Wang, Yu, Perdikaris 2022) explains when and why PINNs fail using the eigenspectrum of the neural tangent kernel, and motivates the adaptive weighting methods that are now standard. Mitigation: adaptive loss weighting (self-adaptive PINNs, NTK-informed weighting), learning rate scheduling, L-BFGS warm-up after Adam pre-training.

Spectral bias

Standard multi-layer perceptrons preferentially learn low-frequency components of the target function. This creates systematic blind spots for solutions with sharp gradients, boundary layers, or oscillatory dynamics, exactly the features that make PDE solutions interesting and that classical methods handle with mesh refinement. Mitigation: Fourier feature embeddings that lift inputs into a high-frequency feature space before the network layers; or architectural choices such as PirateNets that use adaptive residual connections to escape the low-frequency attractor.

Stiff equations

A stiff ODE has solution components that evolve on vastly different timescales. For example, $\frac{dC}{dt} = -10^6 C$ decays to near-zero within microseconds. Standard PINN training with uniform collocation points entirely misses the fast dynamics, and the physics loss at late times provides negligible gradient information about the early transient. Mitigation: adaptive collocation density, curriculum training (gradually expanding the time horizon), domain decomposition.

Slow convergence compared to classical solvers

PINNs typically require tens of thousands of gradient descent steps to converge, orders of magnitude more computational work than a classical solver for equivalent accuracy on a simple well-posed problem:

PINNs are not faster solvers for classical forward problems. Their value is in other capabilities: inverse estimation, sparse data fusion, mesh-free handling of complex geometries.

Scaling to high dimensions and long time horizons

3D Navier-Stokes at high Reynolds numbers with complex boundary conditions remains largely unsolved by vanilla PINNs. Long time horizons compound the stiffness and spectral bias problems. High-dimensional PDEs suffer from the curse of dimensionality even in the collocation regime.

Multi-physics coupling

When heat, mass, momentum, and reaction terms are coupled, the relative magnitudes of the residuals differ by orders of magnitude. A naively assembled loss function will be dominated by the largest-magnitude residual, with the others ignored. Non-dimensionalization and learned or manually tuned loss weights are essential.

Beyond vanilla PINNs

Eight Articles on PINNs

The Jupyter notebooks for this series are available on GitHub: github.com/alishahmohammadi22/physics-ai-pinns. The notebook for this article is 01_Beginner/01_intro_to_pinn.ipynb.

Further Reading