"Written by three experts in the field, Deep Learning is the only comprehensive book on the subject." -- Elon Musk, co-chair of OpenAI; co-founder and CEO of Tesla and SpaceX
Deep learning is a form of machine learning that enables computers to learn from experience and understand the world in terms of a hierarchy of concepts. Because the computer gathers knowledge from experience, there is no need for a human computer operator to formally specify all the knowledge that the computer needs. The hierarchy of concepts allows the computer to learn complicated concepts by building them out of simpler ones; a graph of these hierarchies would be many layers deep. This book introduces a broad range of topics in deep learning.
The text offers mathematical and conceptual background, covering relevant concepts in linear algebra, probability theory and information theory, numerical computation, and machine learning. It describes deep learning techniques used by practitioners in industry, including deep feedforward networks, regularization, optimization algorithms, convolutional networks, sequence modeling, and practical methodology; and it surveys such applications as natural language processing, speech recognition, computer vision, online recommendation systems, bioinformatics, and videogames. Finally, the book offers research perspectives, covering such theoretical topics as linear factor models, autoencoders, representation learning, structured probabilistic models, Monte Carlo methods, the partition function, approximate inference, and deep generative models.
Deep Learning can be used by undergraduate or graduate students planning careers in either industry or research, and by software engineers who want to begin using deep learning in their products or platforms. A website offers supplementary material for both readers and instructors.
Acknowledgments xv
Notation xix
1 Introduction 1
1.1 Who Should Read This Book? 8
1.2 Historical Trend sin Deep Learning 12
I Applied Math and Machine Learning Basics 27
2 Linear Algebra 29
2.1 Scalars, Vectors, Matrices and Tensors 29
2.2 Multiplying Matricesand Vectors 32
2.3 Identity and Inverse Matrices 34
2.4 Linear Dependence and Span 35
2.5 Norms 36
2.6 Special Kinds of Matrices and Vectors 38
2.7 Eigendecomposition 39
2.8 Singular Value Decomposition 42
2.9 The Moore-Penrose Pseudoinverse 43
2.10 The Trace Operator 44
2.11 The Determinant 45
2.12 Example: Principal Components Analysis 45
3 Probability and Information Theory 51
3.1 Why Probability? 52
3.2 Random Variables 54
3.3 Probability Distributions 54
3.4 Marginal Probability 56
3.5 ConditionalProbability 57
3.6 The Chain Rule of Conditional Probabilities 57
3.7 Independence and Conditional Independence 58
3.8 Expectation, Varianceand Covariance 58
3.9 Common Probability Distributions 60
3.10 UsefulPropertiesofCommonFunctions 65
3.11 Bayes’Rule 68
3.12 Technical Details of Continuous Variables 68
3.13 Information Theory 70
3.14 Structured Probabilistic Models 74
4 Numerical Computation 77
4.1 Overflow and Underflow 77
4.2 Poor Conditioning 79
4.3 Gradient-Based Optimization 79
4.4 Constrained Optimization 89
4.5 Example: Linear Least Squares 92
5 Machine Learning Basics 95
5.1 Learning Algorithms 96
5.2 Capacity, Overfitting and Underfitting 107
5.3 Hyperparameters and Validation Sets 117
5.4 Estimators, Bias and Variance 119
5.5 Maximum Likelihood Estimation 128
5.6 BayesianStatistics132
5.7 Supervised Learning Algorithms 136
5.8 Unsupervised Learning Algorithms142
5.9 StochasticGradientDescent 147
5.10 Building a Machine Learning Algorithm 149
5.11 Challenges Motivating Deep Learning 151
II Deep Networks: Modern Practices 161
6 Deep Feedforward Networks 163
6.1 Example:Learning XOR 166
6.2 Gradient-Based Learning 171
6.3 Hidden Units 185
6.4 Architecture Design 191
6.5 Back-Propagation and Other Dierentiation Algorithms 197
6.6 Historical Notes 217
7 Regularization for Deep Learning 221
7.1 Parameter Norm Penalties 223
7.2 Norm Penalties as Constrained Optimization 230
7.3 Regularization and Under-Constrained Problems 232
7.4 Dataset Augmentation 233
7.5 Noise Robustness 235
7.6 Semi-Supervised Learning236
7.7 Multitask Learning 237
7.8 Early Stopping 239
7.9 Parameter Tying and Parameter Sharing 246
7.10 Sparse Representations 247
7.11 Bagging and Other Ensemble Methods 249
7.12 Dropout 251
7.13 Adversarial Training261
7.14 Tangent Distance, Tangent Prop and Manifold Tangent Classiffer 263
8 Optimization for Training DeepModels 267
8.1 How Learning Differs from Pure Optimization 268
8.2 Challenges in Neural Network Optimization 275
8.3 Basic Algorithms 286
8.4 Parameter Initialization Strategies 292
8.5 Algorithms with Adaptive Learning Rates 298
8.6 Approximate Second-Order Methods 302
8.7 Optimization Strategies and Meta-Algorithms 309
9 Convolutional Networks 321
9.1 The Convolution Operation 322
9.2 Motivation 324
9.3 Pooling 330
9.4 Convolution and Pooling as an Infinitely Strong Prior 334
9.5 Variants of the Basic Convolution Function 337
9.6 Structured Outputs 347
9.7 Data Types 348
9.8 Efficient Convolution Algorithms 350
9.9 Random or Unsupervised Features 351
9.10 The Neuroscientific Basis for Convolutional Networks 353
9.11 Convolutional Networks and the History of Deep Learning 359
10 Sequence Modeling: Recurrent and Recursive Nets 363
10.1 Unfolding Computational Graphs 365
10.2 Recurrent Neural Networks 368
10.3 Bidirectional RNNs 383
10.4 Encoder-Decoder Sequence-to-Sequence Architectures 385
10.5 Deep Recurrent Networks 387
10.6 Recursive Neural Networks 388
10.7 The Challenge of Long-Term Dependencies 390
10.8 Echo State Networks 392
10.9 Leaky Units and Other Strategies for Multiple Time Scales 395
10.10 The Long Short-Term Memory and Other Gated RNNs 397
10.11 Optimization for Long-Term Dependencies 401
10.12 Explicit Memory 405
11 Practical Methodology 409
11.1 Performance Metrics 410
11.2 DefaultBaselineModels 413
11.3 Determining Whether to Gather More Data 414
11.4 Selecting Hyperparameters 415
11.5 Debugging Strategies 424
11.6 Example: Multi-Digit Number Recognition 428
12 Applications 431
12.1 Large-Scale Deep Learning 431
12.2 Computer Vision.440
12.3 Speech Recognition 446
12.4 Natural Language Processing 448
12.5 Other Applications 465
III Deep Learning Research 475
13 Linear Factor Models 479
13.1 Probabilistic PCA and Factor Analysis 480
13.2 Independent Component Analysis (ICA) 481
13.3 Slow Feature Analysis.484
13.4 Sparse Coding 486
13.5 Manifold Interpretation of PCA 489
14 Autoencoders 493
14.1 Undercomplete Autoencoders 494
14.2 Regularized Autoencoders 495
14.3 Representational Power, Layer Size and Depth 499
14.4 Stochastic Encodersand Decoders 500
14.5 Denoising Autoencoders501
14.6 Learning Manifolds with Autoencoders 506
14.7 Contractive Autoencoders 510
14.8 Predictive Sparse Decomposition 514
14.9 Applications of Autoencoders515
15 Representation Learning 517
15.1 Greedy Layer-Wise Unsupervised Pretraining 519
15.2 Transfer Learning and Domain Adaptation 526
15.3 Semi-Supervised Disentangling of Causal Factors 532
15.4 Distributed Representation 536
15.5 Exponential Gains from Depth 543
15.6 Providing Clues to Discover Underlying Causes 544
16 Structured Probabilistic Models for Deep Learning 549
16.1 The Challenge of Unstructured Modeling 550
16.2 Using Graphs to Describe Model Structure 554
16.3 Sampling from Graphical Models 570
16.4 Advantages of Structured Modeling 572
16.5 Learning about Dependencies 572
16.6 Inferenceand Approximate Inference 573
16.7 The Deep Learning Approach to Structured Probabilistic Models 575
17 Monte Carlo Methods 581
17.1 Sampling and Monte Carlo Methods 581
17.2 Importance Sampling 583
17.3 Markov Chain Monte Carlo Methods 586
17.4 Gibbs Sampling 590
17.5 The Challenge of Mixing between Separated Modes 591
18 Confronting the Partition Function 597
18.1 The Log-Likelihood Gradient 598
18.2 Stochastic Maximum Likelihood and Contrastive Divergence 599
18.3 Pseudolikelihood 607
18.4 Score Matching and Ratio Matching 609
18.5 DenoisingScore Matching 611
18.6 Noise-Contrastive Estimation 612
18.7 Estimatingthe Partition Function 614
19 Approximate Inference 623
19.1 Inferenceas Optimization 624
19.2 Expectation Maximization 626
19.3 MAP Inferenceand Sparse Coding 627
19.4 Variational Inferenceand Learning 629
19.5 Learned Approximate Inference 642
20 Deep Generative Models 645
20.1 Boltzmann Machines 645
20.2 Restricted Boltzmann Machines 647
20.3 Deep Belief Networks 651
20.4 Deep Boltzmann Machines 654
20.5 Boltzmann Machines for Real-Valued Data 667
20.6 Convolutional Boltzmann Machines 673
20.7 Boltzmann Machines for Structured or Sequential Outputs 675
20.8 Other Boltzmann Machines.677
20.9 Back-Propagation through Random Operations 678
20.10 Directed Generative Nets 682
20.11 Drawing Samples from Autoencoders 701
20.12 Generative Stochastic Networks 704
20.13 Other Generation Schemes 706
20.14 Evaluating Generative Models 707
20.15 Conclusion 710
Bibliography 711
Index 767
内容简介Spring3.0是Spring在积蓄了3年之久后,隆重推出的一个重大升级版本,进一步加强了Spring作为Java领域第一开源平台的翘楚
要想在网络经济中取得竞争优势,仅有网络是远远不够的,本书是为那些面临规划网络战略重任的经理们所写的,这些战略将为他们公司
《雷达手册(第3版)》内容简介:雷达是电子工程系统的一个重要例子。在大学工科课程中,人们通常把重点放在诸如电路设计、信号、固
电机与拖动MATLAB仿真与学习指导 内容简介 书是与孙建忠、刘凤春主编的《电机与拖动》相配套的教学辅助教材,是一本将电机与拖动理论和MATLAB仿真技术有机结...
《社会共识的形成》内容简介:社会共识是社会凝聚和稳定发展的心理基础,是实现和促进社会共识凝聚的一个重要的前提,因而深入理解
《财务共享的智能化升级:业财税一体化的深度融合》内容简介:财务共享是数字技术和管理变革的共同产物,是解决企业规模不断扩大的
哈特穆特•艾斯林格HartmutEsslinger,艾斯林格是全球工业设计教父,世界顶级创意咨询公司青蛙公司创始人,公认的高科技设计领域首
《隋唐五代文学思想史》内容简介:本书采取以时间段落为纲的体制,根据隋唐五代文学思想发展过程中自然形成的时间段落,把起自公元
机器人学.机器视觉与控制-MATLAB算法基础 本书特色 本书是关于机器人学和机器视觉的实用参考书, **部分“基础知识”(第2章和第3...
《突破现实的困境》内容简介:本书讲述了如何采取重大举措提升企业战略的成功概率。本书以几千家企业的“硬数据”为基础,指出哪些
Thisvolumeoffersacritical,philosophicalandepistemologicalframeworktounderstandbe...
JavaScript是一种脚本语言,已广泛用于Web应用开发。本书就是一本引导读者深入学习JavaScript,并能成为JavaScript专家的书。全书
本书系统、深入地介绍了通信系统及通信网的基本原理及基本分析方法,是通信及信息专业的专业基础课教材。全书共二十一章,内容包
《新农人看农村》内容简介:随着“大众创业、万众创新”时代的到来,越来越多的大学生村官凭借多年所学和灵活的创新意识、丰富的互
电脑组装一点通 本书特色 “让读者学以致用,学知识事半功倍,解决问题不求人”,用*轻松的语言、*简捷的方式,讲解*有用的东西,让您不知不觉置身于电脑的殿堂中。电...
Todaymanycompaniesareemployingauser-centereddesign(UCD)process,butformostcompani...
五笔字型即时查(第2版) 内容简介 本字典以*新版《现代汉语词典》为准,列出了200多个汉字常用的86版五笔字型输入法的字根及编码;同时列出了98版五笔字型编码...
TheIntel®64andIA-32ArchitecturesSoftwareDeveloper’sManual,Volume1:BasicArchitect...
Photoshop CS4数码照片精修专家技法精粹(附光盘) 本书特色 精彩光盘超值附赠73个实例素材及*终效果文件,200分钟多媒体演示视频,70个精美相框和...
《轻松攒够养老金》内容简介:养老要趁早,养老金怎么攒,多少够? 《轻松攒够养老金》对年轻人关心的养老金问题进行了解答: ·需