~~写到这里，我才意识到，老哥是不是想拿我们当免费劳动力啊~~~

• Abstract
• PINNs are neural networks that encode model equations. a NN must fit observed data while reducing a PDE residual.
1. Introduction

• The “curse of dimensionality” was first described by Bellman in the context of optimal control problems. (Bellman R.: Dynamic Programming. Sci. 153(3731), 34-37 (1966))

• Early work: MLP (multilayer perceptron) with few hidden layers to solve PDEs. (https://doi.org/10.1109/72.712178)

• 感觉可能更全面的一篇综述：https://doi.org/10.1007/s12206-021-0342-5。该文关注 what deep NN is used, how physical knowledge is represented, how physical information is integrated，本文只关于 PINN, a 2017 framework。

1. What the PINNs are
• PINNs solve problems involving PDEs:
• approximates PDE solutions by training a NN to minimize a loss function
• includes terms reflecting the initial and boundary conditions
• and PDE residual at selected points in the domain (called collocation points)
• given an input point in the integration domain, returns an estimated solution at that point.
• incorporates a residual network that encodes the governing physical equations
• can be thought of as an unsupervised strategy when they are trained solely with physical equations in forward problems, but supervised learning when some properties are derived from data
• Advantages:
• mesh-free? 但是我们给模型喂训练数据的时候往往已经暗含了 mesh 了吧
• on-demand computation after training
• forward and inverse problem using the same optimization, with minimal modification
2. What this Review is About
• 提到了一个做综述找文章的方法：本文涉及的文章可以在 Scopus 上进行高级搜索：((physic* OR physical)) W/2 (informed OR constrained) W/2 “neural network”)
2. The Building Blocks of a PINN

• question:
$F(u(z);\gamma)=f(z),\quad z\ \in\ \Omega \\ B(u(z))=g(z), \quad z\ \in\ \partial \Omega$
• solution:
$\hat u_{\theta}(z)\approx u(z)\\ \theta^* = \arg\min_{\theta}\left(\omega_F L_F(\theta)+\omega_BL_B(\theta)+\omega_{data}L_{data}(\theta)\right)$
1. Neural Network Architecture
• DNN (deep neural network) is an artificial neural network that is deeper than 2 layers.

1. Feed-Forward Neural Network:
• $u_{\theta}(x) = C_{K} \circ C_{k-1} ...\alpha \circ C_1(x),\quad C_k(x) = W_k x_k + b_k$

• Just change CNN from convolution to fully connected.

• Also known as multi-layer perceptrons (MLP)

1. FFNN architectures
• Tartakovsky et al used 3 hidden layers, 50 units per layer, and a hyperbolic tangent activation function. Other people use different numbers but of the same order of magnitude.
• A comparison paper: Blechschmidt, J., Ernst, O.G.: Three ways to solve partial differential equations with neural networks –A review. GAMM-Mitteilungen 44(2), e202100,006 (2021).
2. multiple FFNN: 2 phase Stephan problem.
3. shallow networks: for training costs
4. activation function: the swish function in the paper has a learnable parameter, so — how to add a learnable parameter in PyTorch
2. Convolutional Neural Networks:
• I am most familiar with this one.

• $f_i(x_i;W_i)=\Phi_i(\alpha_i(C_i(W_i,x_i)))$

• performs well with multidimensional data such as images and speeches

1. CNN architectures:
• PhyGeoNet: a physics-informed geometry-adaptive convolutional neural network. It uses a coordinate transformation to convert solution fields from irregular physical domains to rectangular reference domains.
• According to Fang (https://doi.org/10.1109/TNNLS.2021.3070878), a Laplacian operator can be discretized using the finite volume approach, and the procedures are equivalent to convolution. Padding data can serve as boundary conditions.
2. convolutional encoder-decoder network
3. Recurrent Neural Network
• $f_i(h_{i-1})=\alpha\left(W\cdot h_{i-1}+U\cdot x_i+b\right)$, where f is the layer-wise function, x is the input, h is the hidden vector state, W is a hidden-to-hidden weight matrix, U is an input-to-hidden matrix and b is a bias vector. 我认为等号左边的 $h_{i-1}$ 应当作为下标

• 感觉有点像 hidden Markov model，只不过 Markov 中间的 hidden layers 好像与序号无关（记不清了），RNN 看起来各个 W 和 H 似乎不同RNN cell is actually the exact same one and reused throughout. (from https://blog.floydhub.com/a-beginners-guide-on-recurrent-neural-networks-with-pytorch/). Cartoon from Wikipedia:

![Untitled]({{ site.baseurl }}/assets/photos/2023-03-20-rnn-unit.png)

• ![Untitled]({{ site.baseurl }}/assets/photos/2023-03-20-rnn-types.png)

1. RNN architectures
• can be used to perform numerical Euler integration
• 基本上输出的第 i 项只与输入的第 i 和 i-1 项相关。
2. LSTM architectures
• 比 RNN 多更多中间隐变量，至于怎么做到整合长期记忆的，技术细节现在可以先略过
4. other architectures for PINN
1. Bayesian neural network: weights are distributions rather than deterministic values, and these distributions are learned using Bayesian inference. 只介绍了一篇文章
2. GAN architectures:
• two neural networks compete in a zero-sum game to deceive each other
• physics-informed GAN uses automatic differentiation to embed the governing physical laws in stochastic differential equations. The discriminator in PI–GAN is represented by a basic FFNN, while the generators are a combination of FFNNs and a NN induced by the SDE
3. multiple PINNs
2. Injection of Physical Laws
• 既然是要解常/偏微分方程，那么微分计算必不可少。四种方法：hand-coded, symbolic, numerical, auto-differentiation，最后一种显著胜出。所谓 auto-differentiation, 就是利用现成框架，框架自动给出原函数的导数的算法。
• Differential equation residual:
• $r_F[\hat u_\theta](z)=r_\theta(z):=F(\hat u_\theta(z);\gamma)-f$
• $r_F[\hat u_\theta](z)=r_\theta(x,t)=\frac{\partial}{\partial t}\hat u_\theta(x,t)+F_x(\hat u_\theta(x,t))$: 原文给出了来源，但是从字面上看不出来与前式的等价性
• Boundary condition residual: $r_B[\hat u_\theta](z):=B(\hat u_\theta(z))-g(z)$
3. Model Estimation by Learning Approaches
1. Observations about the Loss
• $\omega_F$ accounts for the fidelity of the PDE model. Setting it to 0 trains the network without knowledge of underlying physics.
• In general, the number of $\theta$ is more than the measurements, so regularization is needed.
• The number and position of residual points matter a lot.
2. Soft and Hard Constraints
• Soft: penalty terms. Bad:
• satisfying BC is not guaranteed
• assignment of the weight of BC affects learning efficiency, no theory for this.
• Hard: encoded into the network design. Zhu et. al
3. Optimization methods
• minibatch sampling using the Adam algorithm
• increased sample size with L-BFGS (limited-memory Broyden-Fletcher-Goldfarb-Shanno)
4. Learning theory of PINN: roughly in DE, consistency + stability → convergence
1. convergence aspects: related to the number of parameters in NN
2. statistical learning error analysis: use risk to define error
• Empirical risk: $\hat R[u_\theta]:=\frac{1}{N}\sum_{i=1}^N \left\|\hat u_{\theta}(z_i)-h_i\right\|^2$
• Risk of using approximator: $R[\hat u_{\theta}]:=\int_{\bar \Omega}(\hat u_{\theta}(z)-u(z))^2dz$
• Optimization error: the difference between the local and global minimum, is still an open question for PINN. $E_O:=\hat R[\hat u_{\theta}^*]-\inf_{\theta \in \Theta}\hat R[u_\theta]$
• Generalization error: error when applied to unseen data. $E_G:=\sup_{\theta \in \Theta}\left\|R[u_\theta]-\hat R[u_\theta]\right\|$
• Approximation error: $E_A:=\inf_{\theta \in \Theta}R[u_\theta]$
• Global error between trained deep NN $u^*_\theta$ and the correct solution is bounded: $R[u^*_\theta]\le E_O+2E_G+E_A$
• 有点乱，本来说 error 是误差，结果最后还是用 risk 作为误差
3. error analysis results for PINN
3. Differential Problems Dealt with PINNs：读来感觉这一部分意义不大，将来遇到需要解决的问题时，回来看看之前有没有人做过就行了——另一方面看，一类方程就需要一类特殊构造的神经网络来解，那么说明神经网络解方程的通用性并不好~

1. Ordinary differential equations:
• Neural ODE as learners, a continuous representation of ResNet. [Lai et al], into 2 parts: a physics-informed term and an unknown discrepancy
• LSTM [Zhang et al]
• Directed graph models to implement ODE, and Euler RNN for numerical integration
• Symplectic Taylor neural networks in Tong et al use symplectic integrators
2. Partial differential equations: steady/unsteady的区别就是是否含时
1. steady-state PDEs
2. unsteady PDEs
1. Advection-diffusion-reaction problems
1. diffusion problems
2. advection problems
2. Flow problems
1. Navier-Stokes equations
2. hyperbolic equations
3. quantum problems
3. Other problems
1. Differential equations of fractional order
2. Uncertainty Estimation: Bayesian
4. Solving a Differential Problem with PINN
• 1d non-linear Schrödinger equation
• dataset by simulation with MATLAB-based Chebfun open-source(?) software
4. PINNs: Data, Applications, and Software

1. Data
2. Applications
1. Hemodynamics
2. Flows Problems
3. Optics and Electromagnetic Applications
4. Molecular Dynamics and Materials-Related Applications
5. Geoscience and Elastiostatic Problems
6. Industrial Application
3. Software
1. DeepXDE: initial library by one of the vanilla PINN authors
2. NeuroDiffEq: PyTorch based used at Harvard IACS
3. Modulus: previously known as Nvidia SimNet
4. SciANN: implementation of PINN as Keras wrapper
5. PyDENs: heat and wave equations
6. NeuralPDE.jl: part of SciML
7. ADCME: extending TensorFlow
8. Nangs: stopped updates, but faster than PyDENs
9. TensorDiffEq: TensorFlow for multi-worker distributed computing
10. IDRLnet: a python toolbox inspired by Nvidia SimNet
11. Elvet: coupled ODEs or PDEs, and variational problems about the minimization of a functional
12. Other Packages
5. PINN Future Challenges and Directions

1. Overcoming Theoretical Difficulties in PINN
2. Improving Implementation Aspects in PINN
3. PINN in the SciML Framework
4. PINN in the AI Framework
6. Conclusion