About DML-CZ | FAQ | Conditions of Use | Math Archives | Contact Us
  Previous |  Up |  Next

Article

Opschoor, Joost A. A. ; Schwab, Christoph
Exponential expressivity of ${\rm ReLU}^k$ neural networks on Gevrey classes with point singularities. (English). Applications of Mathematics, vol. 69 (2024), issue 5, pp. 695-724
MSC: 41A25, 65N30 | DOI: 10.21136/AM.2024.0052-24
Full entry | Fulltext not available (moving wall 24 months)      Feedback
Keywords:
neural network; $hp$-finite element method; singularities; Gevrey regularity; exponential convergence
Summary:
We analyze deep Neural Network emulation rates of smooth functions with point singularities in bounded, polytopal domains ${\rm D} \subset \mathbb R^d$, $d=2,3$. We prove exponential emulation rates in Sobolev spaces in terms of the number of neurons and in terms of the number of nonzero coefficients for Gevrey-regular solution classes defined in terms of weighted Sobolev scales in ${\rm D}$, comprising the countably-normed spaces of I. M. Babuška and B. Q. Guo. \endgraf As intermediate result, we prove that continuous, piecewise polynomial high order (``$p$-version'') finite elements with elementwise polynomial degree $p\in \mathbb{N} $ on arbitrary, regular, simplicial partitions of polyhedral domains ${\rm D} \subset \mathbb R^d$, $d\geq 2$, can be \emph {exactly emulated} by neural networks combining ReLU and ReLU$^2$ activations. \endgraf On shape-regular, simplicial partitions of polytopal domains ${\rm D}$, both the number of neurons and the number of nonzero parameters are proportional to the number of degrees of freedom of the $hp$ finite element space of I. M. Babuška and B. Q. Guo.
References:
[1] Arora, R., Basu, A., Mianjy, P., Mukherjee, A.: Understanding deep neural networks with rectified linear units. Available at https://arxiv.org/abs/1611.01491 (2018), 17 pages. DOI 10.48550/arXiv.1611.01491
[2] Babuška, I., Guo, B. Q.: Regularity of the solution of elliptic problems with piecewise analytic data. I. Boundary value problems for linear elliptic equations of second order. SIAM J. Math. Anal. 19 (1988), 172-203. DOI 10.1137/0519014 | MR 0924554 | Zbl 0647.35021
[3] Babuška, I., Guo, B. Q.: The $h$-$p$ version of the finite element method for domains with curved boundaries. SIAM J. Numer. Anal. 25 (1988), 837-861. DOI 10.1137/0725048 | MR 0954788 | Zbl 0655.65124
[4] Babuška, I., Guo, B. Q.: Regularity of the solution of elliptic problems with piecewise analytic data. II. The trace spaces and application to the boundary value problems with nonhomogeneous boundary conditions. SIAM J. Math. Anal. 20 (1989), 763-781. DOI 10.1137/0520054 | MR 1000721 | Zbl 0706.35028
[5] Babuška, I., Guo, B. Q.: Approximation properties of the $h$-$p$ version of the finite element method. Comput. Methods Appl. Mech. Eng. 133 (1996), 319-346. DOI 10.1016/0045-7825(95)00946-9 | MR 1399640 | Zbl 0882.65096
[6] Banjai, L., Melenk, J. M., Schwab, C.: Exponential convergence of $hp$ FEM for spectral fractional diffusion in polygons. Numer. Math. 153 (2023), 1-47. DOI 10.1007/s00211-022-01329-5 | MR 4530200 | Zbl 1511.65117
[7] Banjai, L., Melenk, J. M., Schwab, C.: $hp$-FEM for reaction-diffusion equations. II. Robust exponential convergence for multiple length scales in corner domains. IMA J. Numer. Anal. 43 (2023), 3282-3325. DOI 10.1093/imanum/drac070 | MR 4673336 | Zbl 07800835
[8] Bao, G., Hu, G., Liu, D.: An $h$-adaptive finite element solver for the calculations of the electronic structures. J. Comput. Phys. 231 (2012), 4967-4979. DOI 10.1016/j.jcp.2012.04.002 | Zbl 1245.65125
[9] Bolley, P., Dauge, M., Camus, J.: Régularité Gevrey pour le problème de Dirichlet dans des domaines à singularités coniques. Commun. Partial Differ. Equations 10 (1985), 391-431 French. DOI 10.1080/03605308508820383 | MR 0784683 | Zbl 0573.35024
[10] Cancès, E., Chakir, R., Maday, Y.: Numerical analysis of nonlinear eigenvalue problems. J. Sci. Comput. 45 (2010), 90-117. DOI 10.1007/s10915-010-9358-1 | MR 2679792 | Zbl 1203.65237
[11] Chemin, J.-Y.: Perfect Incompressible Fluids. Oxford Lecture Series in Mathematics and Its Applications 14. Clarendon Press, New York (1998). DOI 10.1093/oso/9780198503972.001.0001 | MR 1688875 | Zbl 0927.76002
[12] Chen, Q., Babuška, I.: The optimal symmetrical points for polynomial interpolation of real functions in the tetrahedron. Comput. Methods Appl. Mech. Eng. 137 (1996), 89-94. DOI 10.1016/0045-7825(96)01051-1 | MR 1420062 | Zbl 0877.65004
[13] Costabel, M., Dauge, M., Nicaise, S.: Singularities of Maxwell interface problems. M2AN, Math. Model. Numer. Anal. 33 (1999), 627-649. DOI 10.1051/m2an:1999155 | MR 1713241 | Zbl 0937.78003
[14] Costabel, M., Dauge, M., Nicaise, S.: Analytic regularity for linear elliptic systems in polygons and polyhedra. Math. Models Methods Appl. Sci. 22 (2012), Article ID 1250015, 63 pages. DOI 10.1142/S0218202512500157 | MR 2928103 | Zbl 1257.35056
[15] Dubiner, M.: Spectral methods on triangles and other domains. J. Sci. Comput. 6 (1991), 345-390. DOI 10.1007/BF01060030 | MR 1154903 | Zbl 0742.76059
[16] Elbrächter, D., Grohs, P., Jentzen, A., Schwab, C.: DNN expression rate analysis of high-dimensional PDEs: Application to option pricing. Constr. Approx. 55 (2022), 3-71. DOI 10.1007/s00365-021-09541-6 | MR 4376559 | Zbl 1500.35009
[17] Ern, A., Guermond, J.-L.: Finite Elements. I. Approximation and Interpolation. Texts in Applied Mathematics 72. Springer, Cham (2021). DOI 10.1007/978-3-030-56341-7 | MR 4242224 | Zbl 1476.65003
[18] Faustmann, M., Marcati, C., Melenk, J. M., Schwab, C.: Exponential convergence of $hp$-FEM for the integral fractional Laplacian in polygons. SIAM J. Numer. Anal. 61 (2023), 2601-2622. DOI 10.1137/22M152493X | MR 4667264 | Zbl 1533.65224
[19] Feischl, M., Schwab, C.: Exponential convergence in $H^1$ of $hp$-FEM for Gevrey regularity with isotropic singularities. Numer. Math. 144 (2020), 323-346. DOI 10.1007/s00211-019-01085-z | MR 4057425 | Zbl 1433.65290
[20] Fournais, S., Hoffmann-Ostenhof, M., Hoffmann-Ostenhof, T., Sørensen, T. Ø.: Analytic structure of solutions to multiconfiguration equations. J. Phys. A. Math. Theor. 42 (2009), Article ID 315208, 11 pages. DOI 10.1088/1751-8113/42/31/315208 | MR 2521310 | Zbl 1181.35282
[21] Guo, B.: The $h$-$p$ Version of Finite Element Method in Two Dimensions. Mathematical Theory and Computational Experience: Ph.D. Thesis. University of Maryland, College Park (1985). MR 2634330
[22] Guo, B. Q., Babuška, I.: On the regularity of elasticity problems with piecewise analytic data. Adv. Appl. Math. 14 (1993), 307-347. DOI 10.1006/aama.1993.1016 | MR 1228743 | Zbl 0790.35028
[23] Guo, B., Babuška, I.: Regularity of the solutions for elliptic problems on nonsmooth domains in $\Bbb R^3$. I. Countably normed spaces on polyhedral domains. Proc. R. Soc. Edinb., Sect. A 127 (1997), 77-126. DOI 10.1017/S0308210500023520 | MR 1433086 | Zbl 0874.35019
[24] Guo, B., Babuška, I.: Regularity of the solutions for elliptic problems on nonsmooth domains in $\Bbb R^3$. II. Regularity in neighbourhoods of edges. Proc. R. Soc. Edinb., Sect. A 127 (1997), 517-545. DOI 10.1017/S0308210500029899 | MR 1453280 | Zbl 0884.35022
[25] Guo, B., Schwab, C.: Analytic regularity of Stokes flow on polygonal domains in countably weighted Sobolev spaces. J. Comput. Appl. Math. 190 (2006), 487-519. DOI 10.1016/j.cam.2005.02.018 | MR 2209521 | Zbl 1121.35098
[26] He, J., Li, L., Xu, J., Zheng, C.: ReLU deep neural networks and linear finite elements. J. Comput. Math. 38 (2020), 502-527. DOI 10.4208/jcm.1901-m2018-0160 | MR 4087799 | Zbl 1463.68072
[27] He, J., Xu, J.: Deep neural networks and finite elements of any order on arbitrary dimensions. Available at https://arxiv.org/abs/2312.14276 (2024), 29 pages. DOI 10.48550/arXiv.2312.14276
[28] He, Y., Marcati, C., Schwab, C.: Analytic regularity of solutions to the Navier-Stokes equations with mixed boundary conditions in polygons. SIAM J. Math. Anal. 56 (2024), 2488-2520. DOI 10.1137/22M1527428 | MR 4719953 | Zbl 07849793
[29] Herrmann, L., Opschoor, J. A. A., Schwab, C.: Constructive deep ReLU neural network approximation. J. Sci. Comput. 90 (2022), Article ID 75, 37 pages. DOI 10.1007/s10915-021-01718-2 | MR 4362469 | Zbl 1500.41002
[30] Hesthaven, J. S.: From electrostatics to almost optimal nodal sets for polynomial interpolation in a simplex. SIAM J. Numer. Anal. 35 (1998), 655-676. DOI 10.1137/S003614299630587X | MR 1618874 | Zbl 0933.41004
[31] Holm, B., Wihler, T. P.: Continuous and discontinuous Galerkin time stepping methods for nonlinear initial value problems with application to finite time blow-up. Numer. Math. 138 (2018), 767-799. DOI 10.1007/s00211-017-0918-2 | MR 3767700 | Zbl 1453.65118
[32] Li, B., Tang, S., Yu, H.: Better approximations of high dimensional smooth functions by deep neural networks with rectified power units. Commun. Comput. Phys. 27 (2020), 379-411. DOI 10.4208/cicp.OA-2019-0168 | MR 4040947 | Zbl 1474.65031
[33] Longo, M., Opschoor, J. A. A., Disch, N., Schwab, C., Zech, J.: De Rham compatible deep neural network FEM. Neural Netw. 165 (2023), 721-739. DOI 10.1016/j.neunet.2023.06.008 | Zbl 1532.65083
[34] Maday, Y., Marcati, C.: Regularity and $hp$ discontinuous Galerkin finite element approximation of linear elliptic eigenvalue problems with singular potentials. Math. Models Methods Appl. Sci. 29 (2019), 1585-1617. DOI 10.1142/S0218202519500295 | MR 3986800 | Zbl 1431.65209
[35] Marcati, C., Opschoor, J. A. A., Petersen, P. C., Schwab, C.: Exponential ReLU neural network approximation rates for point and edge singularities. Found. Comput. Math. 23 (2023), 1043-1127. DOI 10.1007/s10208-022-09565-9 | MR 4603228 | Zbl 1519.35253
[36] Marcati, C., Schwab, C.: Analytic regularity for the incompressible Navier-Stokes equations in polygons. SIAM J. Math. Anal. 52 (2020), 2945-2968. DOI 10.1137/19M1247334 | MR 4113068 | Zbl 1447.35252
[37] Melenk, J. M.: $hp$-Finite Element Methods for Singular Perturbations. Lecture Notes in Mathematics 1796. Springer, Berlin (2002). DOI 10.1007/b84212 | MR 1939620 | Zbl 1021.65055
[38] Melenk, J. M., Schwab, C.: $hp$ FEM for reaction-diffusion equations. I. Robust exponential convergence. SIAM J. Numer. Anal. 35 (1998), 1520-1557. DOI 10.1137/S0036142997317602 | MR 1626030 | Zbl 0972.65093
[39] Opschoor, J. A. A.: Constructive Deep Neural Network Approximations of Weighted Analytic Solutions to Partial Differential Equations in Polygons: Ph.D. Thesis. ETH, Zürich (2023). DOI 10.3929/ethz-b-000614671
[40] Opschoor, J. A. A., Petersen, P. C., Schwab, C.: Deep ReLU networks and high-order finite element methods. Anal. Appl., Singap. 18 (2020), 715-770. DOI 10.1142/S0219530519410136 | MR 4131037 | Zbl 1452.65354
[41] Opschoor, J. A. A., Schwab, C.: Deep ReLU networks and high-order finite element methods. II. Chebyšev emulation. Comput. Math. Appl. 169 (2024), 142-162. DOI 10.1016/j.camwa.2024.06.008 | MR 4768195 | Zbl 7894795
[42] Opschoor, J. A. A., Schwab, C., Xenophontos, C.: Neural networks for singular perturbations. Available at https://arxiv.org/abs/2401.06656 (2024), 39 pages. DOI 10.48550/arXiv.2401.06656
[43] Opschoor, J. A. A., Schwab, C., Zech, J.: Exponential ReLU DNN expression of holomorphic maps in high dimension. Constr. Approx. 55 (2022), 537-582. DOI 10.1007/s00365-021-09542-5 | MR 4376568 | Zbl 1500.41008
[44] Petersen, P., Voigtlaender, F.: Optimal approximation of piecewise smooth functions using deep ReLU neural networks. Neural Netw. 108 (2018), 296-330. DOI 10.1016/j.neunet.2018.08.019 | Zbl 1434.68516
[45] Samarskii, A. A., Galaktionov, V. A., Kurdyumov, S. P., Mikhailov, A. P.: Blow-Up in Quasilinear Parabolic Equations. de Gruyter Expositions in Mathematics 19. Walter De Gruyter, Berlin (1995). DOI 10.1515/9783110889864.535 | MR 1330922 | Zbl 1020.35001
[46] Schwab, C.: $p$- and $hp$-Finite Element Methods: Theory and Applications in Solid and Fluid Mechanics. Numerical Mathematics and Scientific Computation. Clarendon Press, Oxford (1998). MR 1695813 | Zbl 0910.73003
[47] Szabó, B., Babuška, I.: Finite Element Analysis. John Wiley & Sons, New York (1991). MR 1164869 | Zbl 0792.73003
[48] Yang, Y., Wu, Y., Yang, H., Xiang, Y.: Nearly optimal approximation rates for deep super ReLU networks on Sobolev spaces. Available at https://arxiv.org/abs/2310.10766 (2023), 32 pages. DOI 10.48550/arXiv.2310.10766
[49] Yarotsky, D.: Error bounds for approximations with deep ReLU networks. Neural Netw. 94 (2017), 103-114. DOI 10.1016/j.neunet.2017.07.002 | Zbl 1429.68260
Partner of
EuDML logo