「DeepLearning」深度学习方法

「DeepLearning」深度学习方法

模型


[1] Dropout ★★★

Hinton, Geoffrey E., et al. “Improving neural networks by preventing co-adaptation of feature detectors.” arXiv preprint arXiv:1207.0580 (2012).

https://arxiv.org/pdf/1207.0580.pdf

[2] 过拟合 ★★★

Srivastava, Nitish, et al. “Dropout: a simple way to prevent neural networks from overfitting.” Journal of Machine Learning Research 15.1 (2014): 1929-1958.

http://www.jmlr.org/papers/volume15/srivastava14a.old/source/srivastava14a.pdf

[3] Batch归一化——2015年杰出成果 ★★★★

Ioffe, Sergey, and Christian Szegedy. “Batch normalization: Accelerating deep network training by reducing internal covariate shift.” arXiv preprint arXiv:1502.03167 (2015).

http://arxiv.org/pdf/1502.03167

[4] Batch归一化的升级 ★★★★

Ba, Jimmy Lei, Jamie Ryan Kiros, and Geoffrey E. Hinton. “Layer normalization.” arXiv preprint arXiv:1607.06450 (2016).

https://arxiv.org/pdf/1607.06450.pdf

[5] 快速训练新模型 ★★★

Courbariaux, Matthieu, et al. “Binarized Neural Networks: Training Neural Networks with Weights and Activations Constrained to+ 1 or−1.”

https://pdfs.semanticscholar.org/f832/b16cb367802609d91d400085eb87d630212a.pdf

[6] 训练方法创新 ★★★★★

Jaderberg, Max, et al. “Decoupled neural interfaces using synthetic gradients.” arXiv preprint arXiv:1608.05343 (2016).

https://arxiv.org/pdf/1608.05343

[7] 修改预训练网络以降低训练耗时 ★★★

Chen, Tianqi, Ian Goodfellow, and Jonathon Shlens. “Net2net: Accelerating learning via knowledge transfer.” arXiv preprint arXiv:1511.05641 (2015).

https://arxiv.org/abs/1511.05641

[8] 修改预训练网络以降低训练耗时★★★

Wei, Tao, et al. “Network Morphism.” arXiv preprint arXiv:1603.01670 (2016).

https://arxiv.org/abs/1603.01670

优化


[1] 动量优化器 ★★

Sutskever, Ilya, et al. “On the importance of initialization and momentum in deep learning.” ICML (3) 28 (2013): 1139-1147.

http://www.jmlr.org/proceedings/papers/v28/sutskever13.pdf

[2] 可能是当前使用最多的随机优化 ★★★

Kingma, Diederik, and Jimmy Ba. “Adam: A method for stochastic optimization.” arXiv preprint arXiv:1412.6980 (2014).

http://arxiv.org/pdf/1412.6980

[3] 神经优化器 ★★★★★

Andrychowicz, Marcin, et al. “Learning to learn by gradient descent by gradient descent.” arXiv preprint arXiv:1606.04474 (2016).

https://arxiv.org/pdf/1606.04474

[4] ICLR最佳论文,让神经网络运行更快的新方向★★★★★

Han, Song, Huizi Mao, and William J. Dally. “Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding.” CoRR, abs/1510.00149 2 (2015).

https://pdfs.semanticscholar.org/5b6c/9dda1d88095fa4aac1507348e498a1f2e863.pdf

[5] 优化神经网络的另一个新方向 ★★★★

Iandola, Forrest N., et al. “SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and< 1MB model size.” arXiv preprint arXiv:1602.07360 (2016).

http://arxiv.org/pdf/1602.07360

无监督学习 / 深度生成式模型


[1] Google Brain找猫的里程碑论文,吴恩达 ★★★★

Le, Quoc V. “Building high-level features using large scale unsupervised learning.” 2013 IEEE international conference on acoustics, speech and signal processing. IEEE, 2013.

http://arxiv.org/pdf/1112.6209.pdf

[2] 变分自编码机 (VAE) ★★★★

Kingma, Diederik P., and Max Welling. “Auto-encoding variational bayes.” arXiv preprint arXiv:1312.6114 (2013).

http://arxiv.org/pdf/1312.6114

[3] 生成式对抗网络 (GAN) ★★★★★

Goodfellow, Ian, et al. “Generative adversarial nets.” Advances in Neural Information Processing Systems. 2014.

http://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf

[4] 解卷积生成式对抗网络 (DCGAN) ★★★★

Radford, Alec, Luke Metz, and Soumith Chintala. “Unsupervised representation learning with deep convolutional generative adversarial networks.” arXiv preprint arXiv:1511.06434 (2015).

http://arxiv.org/pdf/1511.06434

[5] Attention机制的变分自编码机 ★★★★★

Gregor, Karol, et al. “DRAW: A recurrent neural network for image generation.” arXiv preprint arXiv:1502.04623 (2015).

http://jmlr.org/proceedings/papers/v37/gregor15.pdf

[6] PixelRNN ★★★★

Oord, Aaron van den, Nal Kalchbrenner, and Koray Kavukcuoglu. “Pixel recurrent neural networks.” arXiv preprint arXiv:1601.06759 (2016).

http://arxiv.org/pdf/1601.06759

[7] PixelCNN ★★★★

Oord, Aaron van den, et al. “Conditional image generation with PixelCNN decoders.” arXiv preprint arXiv:1606.05328 (2016).

https://arxiv.org/pdf/1606.05328

 RNN / 序列到序列模型


[1] RNN的生成式序列,LSTM ★★★★

Graves, Alex. “Generating sequences with recurrent neural networks.” arXiv preprint arXiv:1308.0850 (2013).

http://arxiv.org/pdf/1308.0850

[2] 第一份序列到序列论文 ★★★★

Cho, Kyunghyun, et al. “Learning phrase representations using RNN encoder-decoder for statistical machine translation.” arXiv preprint arXiv:1406.1078 (2014).

http://arxiv.org/pdf/1406.1078

[3] 神经网络的序列到序列学习 ★★★★★

Sutskever, Ilya, Oriol Vinyals, and Quoc V. Le. “Sequence to sequence learning with neural networks.” Advances in neural information processing systems. 2014.

http://papers.nips.cc/paper/5346-information-based-learning-by-agents-in-unbounded-state-spaces.pdf

[4] 神经机器翻译 ★★★★

Bahdanau, Dzmitry, KyungHyun Cho, and Yoshua Bengio. “Neural Machine Translation by Jointly Learning to Align and Translate.” arXiv preprint arXiv:1409.0473 (2014).

https://arxiv.org/pdf/1409.0473v7.pdf

[5] 序列到序列Chatbot ★★★

Vinyals, Oriol, and Quoc Le. “A neural conversational model.” arXiv preprint arXiv:1506.05869 (2015).

http://arxiv.org/pdf/1506.05869.pdf%20(http://arxiv.org/pdf/1506.05869.pdf

神经网络图灵机


[1] 未来计算机的基本原型 ★★★★★

Graves, Alex, Greg Wayne, and Ivo Danihelka. “Neural turing machines.” arXiv preprint arXiv:1410.5401 (2014).

http://arxiv.org/pdf/1410.5401.pdf

[2] 强化学习神经图灵机★★★

Zaremba, Wojciech, and Ilya Sutskever. “Reinforcement learning neural Turing machines.” arXiv preprint arXiv:1505.00521 362 (2015).

https://pdfs.semanticscholar.org/f10e/071292d593fef939e6ef4a59baf0bb3a6c2b.pdf

[3] 记忆网络 ★★★

Weston, Jason, Sumit Chopra, and Antoine Bordes. “Memory networks.” arXiv preprint arXiv:1410.3916 (2014).

http://arxiv.org/pdf/1410.3916

[4] 端对端记忆网络 ★★★★

Sukhbaatar, Sainbayar, Jason Weston, and Rob Fergus. “End-to-end memory networks.” Advances in neural information processing systems. 2015.

http://papers.nips.cc/paper/5846-end-to-end-memory-networks.pdf

[5] 指针网络 ★★★★

Vinyals, Oriol, Meire Fortunato, and Navdeep Jaitly. “Pointer networks.” Advances in Neural Information Processing Systems. 2015.

http://papers.nips.cc/paper/5866-pointer-networks.pdf

[6] 整合神经网络图灵机概念的里程碑论文 ★★★★★

Graves, Alex, et al. “Hybrid computing using a neural network with dynamic external memory.” Nature (2016).

https://www.dropbox.com/s/0a40xi702grx3dq/2016-graves.pdf

深度强化学习


[1] 第一篇以深度强化学习为名的论文 ★★★★

Mnih, Volodymyr, et al. “Playing atari with deep reinforcement learning.” arXiv preprint arXiv:1312.5602 (2013).

http://arxiv.org/pdf/1312.5602.pdf

[2] 里程碑 ★★★★★

Mnih, Volodymyr, et al. “Human-level control through deep reinforcement learning.” Nature 518.7540 (2015): 529-533.

https://storage.googleapis.com/deepmind-data/assets/papers/DeepMindNature14236Paper.pdf

[3] ICLR最佳论文 ★★★★

Wang, Ziyu, Nando de Freitas, and Marc Lanctot. “Dueling network architectures for deep reinforcement learning.” arXiv preprint arXiv:1511.06581 (2015).

http://arxiv.org/pdf/1511.06581

[4] 当前最先进的深度强化学习方法 ★★★★★

Mnih, Volodymyr, et al. “Asynchronous methods for deep reinforcement learning.” arXiv preprint arXiv:1602.01783 (2016).

http://arxiv.org/pdf/1602.01783

[5] DDPG ★★★★

Lillicrap, Timothy P., et al. “Continuous control with deep reinforcement learning.” arXiv preprint arXiv:1509.02971 (2015).

http://arxiv.org/pdf/1509.02971

[6] NAF ★★★★

Gu, Shixiang, et al. “Continuous Deep Q-Learning with Model-based Acceleration.” arXiv preprint arXiv:1603.00748 (2016).

http://arxiv.org/pdf/1603.00748

[7] TRPO ★★★★

Schulman, John, et al. “Trust region policy optimization.” CoRR, abs/1502.05477 (2015).

http://www.jmlr.org/proceedings/papers/v37/schulman15.pdf

[8] AlphaGo ★★★★★

Silver, David, et al. “Mastering the game of Go with deep neural networks and tree search.” Nature 529.7587 (2016): 484-489.

http://willamette.edu/%7Elevenick/cs448/goNature.pdf

深度迁移学习 / 终生学习 / 强化学习


[1] Bengio教程 ★★★

Bengio, Yoshua. “Deep Learning of Representations for Unsupervised and Transfer Learning.” ICML Unsupervised and Transfer Learning 27 (2012): 17-36.

http://www.jmlr.org/proceedings/papers/v27/bengio12a/bengio12a.pdf

[2] 终生学习的简单讨论 ★★★

Silver, Daniel L., Qiang Yang, and Lianghao Li. “Lifelong Machine Learning Systems: Beyond Learning Algorithms.” AAAI Spring Symposium: Lifelong Machine Learning. 2013.

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.696.7800&rep=rep1&type=pdf

[3] Hinton、Jeff Dean大神研究 ★★★★

Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. “Distilling the knowledge in a neural network.” arXiv preprint arXiv:1503.02531 (2015).

http://arxiv.org/pdf/1503.02531

[4] 强化学习策略 ★★★

Rusu, Andrei A., et al. “Policy distillation.” arXiv preprint arXiv:1511.06295 (2015).

http://arxiv.org/pdf/1511.06295

[5] 多任务深度迁移强化学习 ★★★

Parisotto, Emilio, Jimmy Lei Ba, and Ruslan Salakhutdinov. “Actor-mimic: Deep multitask and transfer reinforcement learning.” arXiv preprint arXiv:1511.06342 (2015).

http://arxiv.org/pdf/1511.06342

[6] 累进式神经网络 ★★★★★

Rusu, Andrei A., et al. “Progressive neural networks.” arXiv preprint arXiv:1606.04671 (2016).

https://arxiv.org/pdf/1606.04671

一次性深度学习


[1] 不涉及深度学习,但值得一读 ★★★★★

Lake, Brenden M., Ruslan Salakhutdinov, and Joshua B. Tenenbaum. “Human-level concept learning through probabilistic program induction.” Science 350.6266 (2015): 1332-1338.

http://clm.utexas.edu/compjclub/wp-content/uploads/2016/02/lake2015.pdf

[2] 一次性图像识别 ★★★

Koch, Gregory, Richard Zemel, and Ruslan Salakhutdinov. “Siamese Neural Networks for One-shot Image Recognition.”(2015)

http://www.cs.utoronto.ca/%7Egkoch/files/msc-thesis.pdf

[3] 一次性学习基础 ★★★★

Santoro, Adam, et al. “One-shot Learning with Memory-Augmented Neural Networks.” arXiv preprint arXiv:1605.06065 (2016).

http://arxiv.org/pdf/1605.06065

[4] 一次性学习网络 ★★★

Vinyals, Oriol, et al. “Matching Networks for One Shot Learning.” arXiv preprint arXiv:1606.04080 (2016).

https://arxiv.org/pdf/1606.04080

[5] 大型数据 ★★★★

Hariharan, Bharath, and Ross Girshick. “Low-shot visual object recognition.” arXiv preprint arXiv:1606.02819 (2016).

http://arxiv.org/pdf/1606.02819

发表评论