@

0. Abstract

• We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0% which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of ?ve convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a ?nal 1000-way softmax. To make training faster, we used non-saturating neurons and a very ef?cient GPU implementation of the convolution operation. To reduce over?tting in the fully-connected layers we employed a recently-developed regularization method called “dropout” that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012competitionandachievedawinningtop-5testerrorrateof15.3%, compared to 26.2% achieved by the second-best entry.

  我们训练了一个大型的、深卷积的神经网络，将ImageNet LSVRC-2010比赛中的120万张高分辨率图像分为1000个不同的类别。在测试数据上，我们获得了37.5%和17.0%的前1和前5错误率，这比以前的最新技术要好得多。该神经网络拥有6000万个参数和65万个神经元，由五个卷积层组成，其中一些卷积层之后是最大池层，三个完全连接层具有最终的1000路softmax。为了加快训练速度，我们使用了非饱和神经元和非常有效的卷积运算的GPU实现。为了减少全连通层的过切，我们采用了一种最近发展起来的称为“脱落”的正则化方法，该方法被证明是非常有效的。我们还在ILSVRC-2012竞争中输入了该模型的一个变体，并在Top-5TesterRorrate15.3%的范围内实现了一个新的目标，而第二个最佳的目标达到了26.2%。

1. Introduction

• Current approaches to object recognition make essential use of machine learning methods. To improve their performance, we can collect larger datasets, learn more powerful models, and use better techniques for preventing over?tting. Until recently, datasets of labeled images were relatively small — on the order of tens of thousands of images (e.g., NORB [16], Caltech-101/256 [8, 9], and CIFAR-10/100 [12]). Simple recognition tasks can be solved quite well with datasets of this size, especially if they are augmented with label-preserving transformations. For example, the currentbest error rate on the MNIST digit-recognition task (<0.3%) approaches human performance [4]. But objects in realistic settings exhibit considerable variability, so to learn to recognize them it is necessary to use much larger training sets. And indeed, the shortcomings of small image datasets havebeenwidelyrecognized(e.g.,Pintoetal [21]),butithasonlyrecentlybecomepossibletocollect labeled datasets with millions of images. The new larger datasets include LabelMe [23], which consists of hundreds of thousands of fully-segmented images, and ImageNet [6], which consists of over 15 million labeled high-resolution images in over 22,000 categories.

  目前的目标识别方法主要采用机器学习方法。为了提高它们的性能，我们可以收集更大的数据集，学习更强大的模型，并使用更好的技术来防止过度设置。直到最近，标记图像的数据集还相对较小，大约是上万幅图像（例如，NORB[16]、Caltech-101/256[8,9]和CIFAR-10/100[12]）。使用这种大小的数据集可以很好地解决简单的识别任务，特别是当它们通过保留标签的转换进行扩充时。例如，MNIST数字识别任务的当前最佳错误率（<0.3%）接近人类性能[4]。但现实环境中的物体表现出相当大的可变性，因此要学会识别它们，就必须使用更大的训练集。事实上，小图像数据集的缺点已经被认识到了（例如Pintoetal[21]），但是它们有可能收集到有数百万图像的标记数据集。新的更大的数据集包括LabelMe[23]和ImageNet[6]，前者由几十万个完全分割的图像组成，后者由22000多个类别的1500多万个标记的高分辨率图像组成。

• To learn about thousands of objects from millions of images, we need a model with a large learning capacity. However, the immense complexity of the object recognition task means that this problem cannot be speci?ed even by a dataset as large as ImageNet, so our model should also have lots of prior knowledge to compensate for all the data we don’t have. Convolutional neural networks (CNNs) constitute one such class of models [16, 11, 13, 18, 15, 22, 26]. Their capacity can be controlled by varying their depth and breadth,and they also make strong and mostly correct assumptions about the nature of images (namely, stationarity of statistics and locality of pixel dependencies). Thus, compared to standard feedforward neural networks with similarly-sized layers, CNNs have much fewer connections and parameters and so they are easier to train,while their theoretically-best performance is likely to be only slightly worse.

  为了从数以百万计的图像中了解成千上万的对象，我们需要一个具有很大学习能力的模型。然而，对象识别任务的巨大复杂性意味着，即使是像ImageNet这样大的数据集也无法描述这个问题，因此我们的模型也应该有大量的先验知识来补偿我们没有的所有数据。卷积神经网络（CNNs）就是这样一类模型。它们的容量可以通过改变它们的最小宽度和对图像性质（即统计数据的平稳性和像素相关性的局部性）的最有力和最正确的假设来控制。因此，与具有相似大小层的标准前馈神经网络相比，cnn具有更少的连接和参数，因此它们更易于训练，而理论上它们的最佳性能可能只差一点点。

• Despite the attractive qualities of CNNs,and despite the relative ef?ciency of their local architecture, they have still been prohibitively expensive to apply in large scale to high-resolution images. Luckily, current GPUs, paired with a highly-optimized implementation of 2D convolution, are powerful enough to facilitate the training of interestingly-large CNNs, and recent datasets such as ImageNet contain enough labeled examples to train such models without severe over?tting.

  尽管cnn具有吸引人的特性，尽管其本地体系结构相对有效，但它们在大规模应用于高分辨率图像时仍然昂贵得令人望而却步。幸运的是，当前的GPU，加上高度优化的二维卷积实现，功能强大，足以促进有趣的大型CNN的训练，而最近的数据集，如ImageNet，包含了足够多的标记示例来训练此类模型，而不会出现严重的过度设置。

• The speci?c contributions of this paper are as follows: we trained one of the largest convolutional neural networks to date on the subsets of ImageNet used in the ILSVRC-2010 and ILSVRC-2012 competitions [2] and achieved by far the best results ever reported on these datasets. We wrote a highly-optimized GPU implementation of 2D convolution and all the other operations inherent in training convolutional neural networks, which we make available publicly1. Our network contains a number of new and unusual features which improve its performance and reduce its training time, whicharedetailedinSection3. The size of our network made over?tting as igni?cant problem,even with 1.2 million labeled training examples, so we used several effective techniques for preventing over?tting, which are described in Section 4. Our ?nal network contains ?ve convolutional and three fully-connected layers, and this depth seems to be important: we found that removing any convolutionallayer(eachofwhichcontainsnomorethan1%ofthemodel’sparameters)resultedin inferior performance.

  本文的具体贡献如下：我们在ILSVRC-2010和ILSVRC-2012比赛中使用的ImageNet子集上训练了迄今为止最大的卷积神经网络之一[2]，并在这些数据集上取得了迄今为止最好的结果。我们编写了一个高度优化的GPU实现2D卷积和训练卷积神经网络所固有的所有其他操作，我们将公开提供1。我们的网络包含了许多新的和不寻常的特性，这些特性提高了它的性能并减少了它的训练时间，下面是第3节。我们的网络规模使过度培训成为一个重大问题，即使有120万个标记培训示例，因此我们使用了几种有效的预防过度培训的技术，如第4节所述。我们的最终网络包含五个卷积层和三个完全连接的层，这个深度似乎很重要：我们发现移除任何卷积层（每个卷积层包含的模型参数不超过1%）会导致性能低下。

• Intheend,thenetwork’ssizeislimitedmainlybytheamountofmemoryavailableoncurrentGPUs and by the amount of training time that we are willing to tolerate. Our network takes between ?ve and six days to train on two GTX 580 3GB GPUs. All of our experiments suggest that our results can be improved simply by waiting for faster GPUs and bigger datasets to become available.

  最后，网络的发展主要取决于当前GPU上可用的内存量和我们愿意忍受的训练时间。我们的网络需要五到六天的时间来训练两个GTX5803GB的GPU。我们所有的实验都表明，只要等待更快的gpu和更大的数据集可用，我们的结果就可以得到改善。

2. The Dataset

• ImageNetisadatasetofover15millionlabeledhigh-resolutionimagesbelongingtoroughly22,000 categories. The images were collected from the web and labeled by human labelers using Amazon’s Mechanical Turk crowd-sourcing tool. Starting in 2010, as part of the Pascal Visual Object Challenge, an annual competition called the ImageNet Large-Scale Visual Recognition Challenge (ILSVRC) has been held. ILSVRC uses a subset of ImageNet with roughly 1000 images in each of 1000 categories. In all, there are roughly 1.2 million training images, 50,000 validation images, and 150,000 testing images.

  超过1500万张高分辨率图像的图像网络数据集，大约有22000个类别。这些图片是从网上收集的，并由人用亚马逊的机械土耳其人群体采购工具贴上标签。从2010年开始，作为帕斯卡视觉物体挑战赛的一部分，每年举办一次名为ImageNet大型视觉识别挑战赛（ILSVRC）的比赛。ILSVRC使用ImageNet的一个子集，在1000个类别中的每个类别中大约有1000个图像。总共大约有120万张训练图像、5万张验证图像和15万张测试图像。

• ILSVRC-2010 is the only version of ILSVRC for which the test set labels are available, so this is the version on which we performed most of our experiments. Since we also entered our model in the ILSVRC-2012 competition, in Section 6 we report our results on this version of the dataset as well,for which test set labels are unavailable. On ImageNet,it is customary to report two errorrates: top-1 and top-5, where the top-5 error rate is the fraction of test images for which the correct label is not among the ?ve labels considered most probable by the model.

  ILSVRC-2010是ILSVRC的唯一一个可用测试集标签的版本，因此这是我们执行大多数实验的版本。由于我们也在ILSVRC-2012竞争中输入了我们的模型，因此在第6节中，我们也报告了这个版本的数据集的结果，对于这个版本的数据集，测试集标签是不可用的。在ImageNet上，通常报告两个错误率：top-1和top-5，其中top-5错误率是测试图像的一部分，正确的标签不在模型认为最有可能的五个标签中。

• ImageNet consists of variable-resolution images, while our system requires a constant input dimensionality. Therefore, we down-sampled the images to a ?xed resolution of 256 × 256. Given a rectangular image, we ?rst rescaled the image such that the shorter side was of length 256, and then croppedoutthecentral256×256patchfromtheresultingimage. Wedidnotpre-processtheimages in any other way, except for subtracting the mean activity over the training set from each pixel. So we trained our network on the (centered) raw RGB values of the pixels.

  ImageNet由可变分辨率的图像组成，而我们的系统需要恒定的输入维数。因此，我们将图像采样到256×256的固定分辨率。给定一个矩形图像，我们首先重新缩放图像，使其较短的边长度为256，然后从结果图像中剪切出中心256×256个补丁。我们不会以任何其他方式预处理图像，除了从每个像素减去训练集上的平均活动。因此，我们训练我们的网络的（中心）原始RGB值的像素。

3. The Architecture

• The architecture of our network is summarized in Figure 2. It contains eight learned layers — ?ve convolutional and three fully-connected. Below, we describe some of the novel or unusual features of our network’s architecture. Sections 3.1-3.4 are sorted according to our estimation of their importance, with the most important ?rst.

  我们的网络架构如图2所示。它包含八个学习层-五个卷积层和三个完全连接层。下面，我们将描述我们的网络架构的一些新颖或不寻常的特性。第3.1-3.4节根据我们对其重要性的估计进行分类，首先是最重要的部分。

3.1 ReLU Nonlinearity

• The standard way to model a neuron’s output f as a function of its input x is with f(x) = tanh(x) or f(x) = (1 + e?x)?1. In terms of training time with gradient descent, these saturating nonlinearities are much slower than the non-saturating nonlinearity f(x) = max(0,x). Following Nair and Hinton [20], we refer to neurons with this nonlinearity as Recti?ed Linear Units (ReLUs). Deep convolutional neural networks with ReLUs train several times faster than their equivalents with tanh units. This is demonstrated in Figure 1, which shows the number of iterations required to reach 25% training error on the CIFAR-10 dataset for a particular four-layer convolutional network. This plot shows that we would not have been abletoexperimentwithsuchlargeneuralnetworksfor this work if we had used traditional saturating neuron models.

  将神经元的输出f建模为其输入x函数的标准方法是f（x）=tanh（x）或f（x）=（1+e-x）-1。对于梯度下降的训练时间，这些饱和非线性比非饱和非线性f（x）=max（0，x）慢得多。继Nair和Hinton[20]之后，我们将具有这种非线性的神经元称为直线单位（ReLUs）。具有ReLUs的深卷积神经网络比具有tanh单元的深卷积神经网络训练速度快几倍。如图1所示，它显示了在特定的四层卷积网络的CIFAR-10数据集上达到25%训练误差所需的迭代次数。这张图显示，如果我们使用传统的饱和神经元模型，我们将无法在这项工作中使用如此庞大的网络进行实验。

• We are not the ?rst to consider alternatives to traditional neuron models in CNNs. For example, Jarrett etal.[11]claimthatthenonlinearityf(x) = |tanh(x)| works particularlywell withtheir type of contrast normalization followed by local average pooling on the Caltech-101 dataset. However, on this dataset the primary concern is preventing over?tting, so the effect they are observing is different from the accelerated ability to ?t the training set which we report when using ReLUs. Fasterlearninghasagreatin?uenceonthe performance of large models trained on large datasets.

  我们并不是第一个考虑在CNNs中替代传统神经元模型的人。例如，Jarrett等人[11]声称，非线性系数f（x）=tanh（x）|在Caltech-101数据集上特别适用于其对比度标准化类型，然后是局部平均值池。然而，在这个数据集上，主要关注的是防止过度设置，因此他们观察到的效果不同于我们在使用ReLUs时报告的加速设置训练集的能力。快速学习和提高大型数据集上训练的大型模型的性能。

• Figure 1: A four-layer convolutional neural networkwithReLUs(solidline)reachesa25% trainingerrorrateonCIFAR-10sixtimesfaster than an equivalent network with tanh neurons (dashed line). The learning rates for each networkwerechosenindependentlytomaketraining as fast as possible. No regularization of any kind was employed. The magnitude of the effect demonstrated here varies with network architecture, but networks with ReLUs consistentlylearnseveraltimesfasterthanequivalents with saturating neurons.

  图1:RELUS（实线）四层卷积神经网络的训练错误率比tanh神经元（虚线）的等效网络快25%。每个网络的学习率都是独立的，以尽可能快的速度进行训练。没有采用任何形式的正规化。这种效应的大小随网络结构的不同而不同，但具有ReLUs的网络与具有饱和神经元的等效网络相比，具有更高的学习效率。
  

  3.2 Training on Multiple GPUs

• A single GTX 580 GPU has only 3GB of memory, which limits the maximum size of the networks that can be trained on it. It turns out that 1.2 million training examples are enough to train networks which are too big to ?t on one GPU. Therefore we spread the net across two GPUs. Current GPUs are particularly well-suited to cross-GPU parallelization, as they are able to read from and write to one another’s memory directly, without going through host machine memory. The parallelization scheme that we employ essentially puts half of the kernels (or neurons) on each GPU, with one additional trick: the GPUs communicate only in certain layers. This means that, for example, the kernels of layer 3 take input from all kernel maps in layer 2. However, kernels in layer 4 take input only from those kernel maps in layer 3 which reside on the same GPU. Choosing the pattern of connectivity is a problem for cross-validation, but this allows us to precisely tune the amount of communication until it is an acceptable fraction of the amount of computation.

  一个GTX580GPU只有3GB的内存，这限制了可以在它上训练的网络的最大大小。事实证明，120万个培训实例足以培训那些大到不能在一个GPU上运行的网络。因此，我们将网络分布在两个gpu上。当前的GPU特别适合于跨GPU并行化，因为它们能够直接读写彼此的内存，而不必经过主机内存。我们采用的并行化方案实际上是将一半的内核（或神经元）放在每个GPU上，还有一个额外的技巧：GPU只在特定的层上通信。这意味着，例如，第3层的内核从第2层的所有内核映射中获取输入。然而，第4层的内核只接受来自第3层的内核映射的输入，这些映射位于同一个GPU上。选择连接模式是交叉验证的一个问题，但这允许我们精确地调整通信量，直到它是计算量的可接受部分。

• The resultant architecture is somewhat similar to that of the “columnar” CNN employed by Cires?an etal.[5],exceptthatourcolumnsarenotindependent(seeFigure2). Thisschemereducesourtop-1 and top-5 error rates by 1.7% and 1.2%, respectively, as compared with a net with half as many kernels in each convolutional layer trained on one GPU. The two-GPU net takes slightly less time to train than the one-GPU net2.

  由此产生的结构与Cires?an etal[5]所采用的“柱状”CNN有点相似，除了柱状结构是不依赖的（见图2）。与在一个GPU上训练的每个卷积层有一半核的网络相比，该方案分别将第1和第5错误率降低了1.7%和1.2%。两个GPU网的训练时间比一个GPU网稍微短一些。

• The one-GPU net actually has the same number of kernels as the two-GPU net in the ?nal convolutional layer. This is because most of the net’s parameters are in the ?rst fully-connected layer, which takes the last convolutional layer as input. So to make the two nets have approximately the same number of parameters, we did not halve the size of the ?nal convolutional layer (nor the fully-conneced layers which follow). Therefore this comparison is biased in favor of the one-GPU net, since it is bigger than “half the size” of the two-GPU net.

  在最终卷积层中，一个GPU网的核数实际上与两个GPU网的核数相同。这是因为网络的大多数参数都在第一个完全连接的层中，它以最后一个卷积层作为输入。因此，为了使两个网络的参数数目大致相同，我们没有将最终卷积层的大小减半（也没有随后的完全连接层）。因此，这种比较偏向于一个GPU网，因为它比两个GPU网的“一半大”。

3.3 Local Response Normalization

• ReLUs have the desirable property that they do not require input normalization to prevent them fromsaturating. IfatleastsometrainingexamplesproduceapositiveinputtoaReLU,learningwill happen in that neuron. However, we still ?nd that the following local normalization scheme aids generalization. Denoting by aix,y the activity of a neuron computed by applying kernel i at position (x,y) and then applying the ReLU nonlinearity, the response-normalized activity bix,y is given by the expression $b_x,y^i=a{x,y}^i/(k+\alpha\sum_{j=max(0,i-n/2)}^{min(N-1,i+n/2)}(a_{x,y}^j)^{2})^{\beta}$

ReLUs的理想特性是不需要输入规范化来防止它们饱和。如果至少有一些训练例子产生了对ReLU的正向输入，学习将发生在该神经元中。然而，我们仍然发现下面的局部规范化方案有助于推广。用a i x，y表示在（x，y）位置应用核i，然后应用ReLU非线性计算出的神经元活动，响应规范化活动bix，y由表达式给出$b_x,y^i=a{x,y}^i/(k+\alpha\sum_{j=max(0,i-n/2)}^{min(N-1,i+n/2)}(a_{x,y}^j)^{2})^{\beta}$

• where the sum runs over n “adjacent” kernel maps at the same spatial position, and N is the total numberofkernelsinthelayer. The ordering of the kernel maps is of course arbitrary and determined before training begins. This sort of response normalization implements a form of lateral inhibition inspired by the type found in real neurons, creating competition for big activities amongst neuron outputs computed using different kernels. The constants k,n,α, and β are hyper-parameters whose values are determined using a validation set; we used k = 2, n = 5, α = 10?4, and β = 0.75. We applied this normalization after applying the ReLU nonlinearity in certain layers (see Section 3.5).

  其中，和在同一空间位置的n个“相邻”核映射上运行，n是层中的核总数。核映射的顺序当然是任意的，在训练开始之前就已经确定了。这种反应规范化实现了一种侧抑制的形式，这种侧抑制的灵感来源于真实神经元中发现的类型，在使用不同核计算的神经元输出之间产生了对大活动的竞争。常数k、n、α和β是超参数，其值由验证集确定；我们使用k=2、n=5、α=10-4和β=0.75。在某些层中应用ReLU非线性后，我们应用了该标准化（见第3.5节）。

• This scheme bears some resemblance to the local contrast normalization scheme of Jarrettetal.[11], but ours would be more correctly termed “brightness normalization”, since we do not subtract the mean activity. Response normalization reduces our top-1 and top-5 error rates by 1.4% and 1.2%, respectively. We also veri?ed the effectiveness of this scheme on the CIFAR-10dataset: afour-layer CNN achieved a 13% test error rate without normalization and 11% with normalization3.

  这个方案与Jarrettetal[11]的局部对比度标准化方案有一些相似之处，但是我们的方案更准确地称为“亮度标准化”，因为我们没有减去平均活动。响应规范化将前1名和前5名的错误率分别降低了1.4%和1.2%。我们还在CIFAR-10数据集上验证了该方案的有效性：在没有标准化的情况下，四层CNN的测试错误率达到了13%，在标准化的情况下达到了11%。

3.4 OverlappingPooling

• Pooling layers in CNNs summarize the outputs of neighboring groups of neurons in the same kernel map. Traditionally, the neighborhoods summarized by adjacent pooling units do not overlap (e.g., [17, 11, 4]). To be more precise, a pooling layer can be thought of as consisting of a grid of pooling units spaced s pixels apart, each summarizing a neighborhood of size z×z centered at the location of the pooling unit. If we set s = z, we obtain traditional local pooling as commonly employed in CNNs. If we set s < z, we obtain overlapping pooling. This is what we use throughout our network, with s = 2 and z = 3. This scheme reduces the top-1 and top-5 error rates by 0.4% and 0.3%, respectively, as compared with the non-overlapping scheme s = 2,z = 2, which produces outputofequivalentdimensions. Wegenerallyobserveduringtrainingthatmodelswithoverlapping pooling ?nd it slightly more dif?cult to over?t

  cnn中的池层总结了同一核映射中相邻神经元群的输出。传统上，由相邻池单元汇总的邻域不会重叠（例如[17、11、4]）。更准确地说，池层可以被认为是由间隔s个像素的池单元组成的网格，每个单元汇总以池单元的位置为中心的z×z大小的邻域。如果我们设置s=z，我们将获得CNNs中常用的传统本地池。如果我们设置s<z，我们得到重叠池。这是我们在整个网络中使用的，s=2和z=3。与产生输出等效维数的非重叠方案s=2，z=2相比，该方案的前1和前5错误率分别降低了0.4%和0.3%。一般来说，我们都会观察到，在超过池的情况下，对模型的训练会稍微增加对池的难度

3.5 OverallArchitecture

• Now we are ready to describe the overall architecture of our CNN. As depicted in Figure 2, the net contains eight layers with weights; the ?rst ?ve are convolutional and the remaining three are fullyconnected. Theoutputofthelastfully-connectedlayerisfedtoa1000-waysoftmaxwhichproduces adistributionoverthe1000classlabels. Ournetworkmaximizesthemultinomiallogisticregression objective,whichisequivalenttomaximizingtheaverageacrosstrainingcasesofthelog-probability of the correct label under the prediction distribution.

  现在我们准备描述我们的CNN的整体架构。如图2所示，我们的网络包含8个带权重的层；前5层是卷积层，剩下的3层是全连接层。最后一层全连接层的输出是1000维softmax的输入，softmax会产生1000类标签的分布。我们的网络最大化多项逻辑回归的目标，这等价于最大化预测分布下训练样本正确标签的对数概率的均值

• The kernels of the second, fourth, and ?fth convolutional layers are connected only to those kernel maps in the previous layer which reside on the same GPU (see Figure 2). The kernels of the third convolutional layer are connected to all kernel maps in the second layer. The neurons in the fullyconnected layers are connected to all neurons in the previous layer. Response-normalization layers followthe?rstandsecondconvolutionallayers. Max-poolinglayers,of the kind described in Section 3.4, follow both response-normalization layers as well as the ?fth convolutional layer. The ReLU non-linearity is applied to the output of every convolutional and fully-connected layer.

  第2，4，5卷积层的核只与位于同一GPU上的前一层的核映射相连接（看图2）。第3卷积层的核与第2层的所有核映射相连。全连接层的神经元与前一层的所有神经元相连。第1，2卷积层之后是响应归一化层。3.4节描述的这种最大池化层在响应归一化层和第5卷积层之后。ReLU非线性应用在每个卷积层和全连接层的输出上。

• The ?rst convolutional layer ?lters the 224×224×3 inputimagewith96kernelsofsize 11×11×3 with a stride of 4 pixels (this is the distance between the receptive ?eld centers of neighboring neurons in a kernel map). The second convolutional layer takes as input the (response-normalized and pooled) output of the ?rst convolutional layer and ?lters it with 256 kernels of size 5×5×48. Thethird,fourth,and?fthconvolutionallayersareconnectedtooneanotherwithoutanyintervening pooling or normalization layers. The third convolutional layer has 384 kernels of size 3 × 3 × 256 connected to the (normalized, pooled) outputs of the second convolutional layer. The fourth convolutional layer has 384 kernels of size 3×3×192 , and the ?fth convolutional layer has 256 kernels of size 3×3×192. The fully-connected layers have 4096 neurons each.

  第1卷积层使用96个核对224 × 224 × 3的输入图像进行滤波，核大小为11 × 11 × 3，步长是4个像素（核映射中相邻神经元感受野中心之间的距离）。第2卷积层使用用第1卷积层的输出（响应归一化和池化）作为输入，并使用256个核进行滤波，核大小为5 × 5 × 48。第3，4，5卷积层互相连接，中间没有接入池化层或归一化层。第3卷积层有384个核，核大小为3 × 3 × 256，与第2卷积层的输出（归一化的，池化的）相连。第4卷积层有384个核，核大小为3 × 3 × 192，第5卷积层有256个核，核大小为3 × 3 × 192。每个全连接层有4096个神经元。

• Figure 2: An illustration of the architecture of our CNN, explicitly showing the delineation of responsibilities betweenthetwoGPUs. OneGPUrunsthelayer-partsatthetopofthe?gurewhiletheotherrunsthelayer-parts atthebottom. TheGPUscommunicateonlyatcertainlayers. Thenetwork’sinputis150,528-dimensional,and thenumberofneuronsinthenetwork’sremaininglayersisgivenby253,440–186,624–64,896–64,896–43,264– 4096–4096–1000.

  图2：我们的CNN架构的一个图解，明确显示了网络之间的职责划分。一个位于图的顶层部分，另一个在底部运行层部分。GPU只与证书层通信。网络输入为150528维，网络维护层编号为253440–186624–64896–64896–43264–4096–4096–1000。
  

  4. ReducingOver?tting

• Our neural network architecture has 60 million parameters. Although the 1000 classes of ILSVRC make each training example impose 10 bits of constraint on the mapping from image to label, this turns out to be insuf?cient to learn so many parameters without considerable over?tting. Below, we describe the two primary ways in which we combat over?tting.

  我们的神经网络架构有6000万参数。尽管ILSVRC的1000类使每个训练样本从图像到标签的映射上强加了10比特的约束，但这不足以学习这么多的参数而没有相当大的过拟合。下面，我们会描述我们用来克服过拟合的两种主要方式。

4.1 DataAugmentation

• The easiest and most common method to reduce over?tting on image data is to arti?cially enlarge the dataset using label-preserving transformations (e.g., [25, 4, 5]). We employ two distinct forms of data augmentation, both of which allow transformed images to be produced from the original images with very little computation, so the transformed images do not need to be stored on disk. In our implementation, the transformed images are generated in Python code on the CPU while the GPUistrainingonthepreviousbatchofimages. Sothesedataaugmentationschemesare, ineffect, computationally free.

  图像数据上最简单常用的用来减少过拟合的方法是使用标签保留变换（例如[25, 4, 5]）来人工增大数据集。我们使用了两种独特的数据增强方式，这两种方式都可以从原始图像通过非常少的计算量产生变换的图像，因此变换图像不需要存储在硬盘上。在我们的实现中，变换图像通过CPU的Python代码生成，而此时GPU正在训练前一批图像。因此，实际上这些数据增强方案是计算免费的。

• The ?rst form of data augmentation consists of generating image translations and horizontal re?ections. Wedothisbyextractingrandom 224×224 patches(andtheirhorizontalre?ections)fromthe 256×256images and training our network on these extracted patches4. Thisincreasesthesizeofour training set by a factor of 2048, though the resulting training examples are, of course, highly interdependent. Withoutthisscheme,ournetworksuffersfromsubstantialover?tting,whichwouldhave forced us to use much smaller networks. At test time, the network makes a prediction by extracting ?ve 224 × 224 patches (the four corner patches and the center patch) as well as their horizontal re?ections (hence ten patches in all), and averaging the predictions made by the network’s softmax layer on the ten patches.

  数据增强的第一种形式包括生成图像平移和水平反射。我们将从256×256幅图像中提取224×224个补丁（以及它们的矩形），并在这些提取的补丁上训练我们的网络4。这将增加2048倍的四个训练集的大小，尽管最终得到的训练示例当然是高度相互依赖的。如果没有这个计划，我们的网络会受到实质性的影响，这将迫使我们使用更小的网络。在测试时，网络通过提取5个224×224补丁（四个角补丁和中心补丁）及其水平反射（因此总共10个补丁）进行预测，并平均网络的softmax层对10个补丁的预测。

• The second form of data augmentation consists of altering the intensities of the RGB channels in training images. Speci?cally, we perform PCA on the set of RGB pixel values throughout the ImageNet training set. Toeachtrainingimage,weaddmultiplesofthefoundprincipalcomponents, withmagnitudesproportionaltothecorrespondingeigenvaluestimesarandomvariabledrawnfrom a Gaussian with mean zero and standard deviation 0.1. Therefore to each RGB image pixel Ixy = [IR xy,IG xy,IB xy]T we add the following quantity:[p1,p2,p3][α1λ1,α2λ2,α3λ3]T

  第二种形式的数据增强包括改变训练图像中RGB通道的强度。具体来说，我们在整个ImageNet训练集中对RGB像素值集执行PCA。在训练图像中，弱多个基本主成分，其大小占对应特征值估计的比例随机变化，从均值为零且标准偏差为0.1的高斯图中提取。因此，对于每个RGB图像像素Ixy=[IR xy，IG xy，IB xy]T，我们添加以下数量：[p1，p2，p3][α1λ1，α2λ2，α3λ3]T

• where pi and λi are ith eigenvector and eigenvalue of the 3 × 3 covariance matrix of RGB pixel values, respectively, and αi is the aforementioned random variable. Each αi is drawn only once for all the pixels of a particular training image until that image is used for training again, at which point it is re-drawn. This scheme approximately captures an important property of natural images, namely,thatobjectidentityisinvarianttochangesintheintensityandcoloroftheillumination. This scheme reduces the top-1 error rate by over 1%.

  其中pi和λi分别是RGB像素值的3×3协方差矩阵的特征向量和特征值，αi是上述随机变量。对于特定训练图像的所有像素，每个αi仅绘制一次，直到该图像再次用于训练，此时将重新绘制。该方案近似地捕捉了自然图像的一个重要性质，即物体的同一性是变化的。该方案将前1位错误率降低了1%以上。

4.2 Dropout

• Combining the predictions of many different models is a very successful way to reduce test errors [1, 3], but it appears to be too expensive for big neural networks that already take several days to train. There is, however, a very ef?cient version of model combination that only costs about a factor of two during training. The recently-introduced technique, called “dropout” [10], consists of setting to zero the output of each hidden neuron with probability 0.5. The neurons which are “dropped out” in this way do not contribute to the forward pass and do not participate in backpropagation. So everytime an input is presented,the neural network samples a different architecture, but all the search it ectures share weights. Thistechniquereducescomplexco-adaptationsofneurons, since a neuron cannot rely on the presence of particular other neurons. It is, therefore, forced to learn more robust features that are useful in conjunction with many different random subsets of the other neurons. At test time, we use all the neurons but multiply their outputs by 0.5, which is a reasonable approximation to taking the geometric mean of the predictive distributions produced by the exponentially-many dropout networks.

  结合许多不同模型的预测是减少测试误差的一个非常成功的方法[1，3]，但是对于已经需要几天训练的大型神经网络来说，这似乎太昂贵了。然而，有一个非常有效的模型组合版本，在培训期间只需花费大约两倍的成本。最近引入的技术，称为“退出”[10]，包括将每个隐藏神经元的输出设置为零，概率为0.5。以这种方式“退出”的神经元不参与正向传递，也不参与反向传播。因此，每次输入出现时，神经网络都会对不同的结构进行采样，但所有的搜索结构都共享权重。这项技术使神经元产生复杂的适应性，因为神经元不能依赖于特定的其他神经元的存在。因此，它被迫学习更健壮的特征，这些特征与其他神经元的许多不同随机子集结合使用是有用的。在测试时，我们使用所有的神经元，但是将它们的输出乘以0.5，这是一个合理的近似值，可以得到指数级多个失活网络产生的预测分布的几何平均值。

• We use dropout in the ?rst two fully-connected layers of Figure2. Without dropout,our network exhibits substantial over?tting. Dropout roughly doubles the number of iterations required to converge

  我们在图2的前两个完全连接的层中使用dropout。在没有辍学的情况下，我们的网络表现出明显的过度配置。丢失大约是收敛所需迭代次数的两倍

5. Detailsoflearning

• We trained our models using stochastic gradient descent with a batch size of 128 examples, momentum of 0.9, and weight decay of 0.0005. We found that this small amount of weight decay was important for the model to learn. In other words,weight decay hereis not merely a regularizer: it reduces the model’s training error. The update rule for weight w was
  

  我们使用随机梯度下降训练我们的模型，批量大小为128例，动量为0.9，权重衰减为0.0005。我们发现，这种少量的重量衰减对模型的学习很重要。换句话说，这里的权重衰减不仅仅是一个正则化器：它减少了模型的训练误差。权重w的更新规则是

• We initialized the weights in each layer from a zero-mean Gaussian distribution with standard deviation 0.01. We initialized the neuron biases in the second, fourth, and ?fth convolutional layers, as well as in the fully-connected hidden layers, with the constant 1. This initialization accelerates the early stages of learning by providing the ReLUs with positive inputs. We initialized the neuron biases in the remaining layers with the constant 0.

  我们用标准差为0.01的零均值高斯分布初始化每一层的权重。我们用常数1初始化第二、第四和第五卷积层以及完全连接的隐藏层中的神经元偏差。这种初始化通过向ReLUs提供积极的输入来加速学习的早期阶段。我们用常数0初始化剩余层的神经元偏差。

• We used an equal learning rate for all layers, which we adjusted manually throughout training. The heuristic which we followed was to divide the learning rate by 10 when the validation error rate stopped improving with the current learning rate. The learning rate was initialized at 0.01 and
  reduced three times prior to termination. We trained the network for roughly 90 cycles through the training set of 1.2millionimages,which took ?ve to six days on two NVIDIAGTX5803GBGPUs.

  我们对所有层使用相同的学习率，在整个训练过程中我们手动调整。我们采用的启发式方法是，当验证错误率不再随当前学习率提高而提高时，将学习率除以10。学习率初始化为0.01
  在终止前减少三倍。我们通过120万张图像的训练集对网络进行了大约90个周期的训练，在两台NVIDIAGTX5803GBGPUs上花了5到6天的时间。

• Figure 3: 96 convolutional kernels of size 11×11×3 learned by the ?rst convolutional layer on the 224×224×3 input images. The top 48 kernels were learned on GPU1 while the bottom 48 kernels were learned on GPU 2. See Section 6.1 for details.

  图3:224×224×3输入图像上第一卷积层学习到的96个11×11×3大小的卷积核。前48个核在GPU1上学习，后48个核在GPU 2上学习。详见6.1节。
  

6. Results

• Our results on ILSVRC-2010 are summarized in Table 1. Our network achieves top-1 and top-5 test set error rates of 37.5% and 17.0%5. The best performance achieved during the ILSVRC2010 competition was 47.1% and 28.2% with an approach that averages the predictions produced from six sparse-coding models trained on different features [2], and since then the best published results are 45.7% and 25.7% with an approach that averages the predictions of two classi?ers trained on Fisher Vectors (FVs) computed from two types of densely-sampled features [24].

  我们关于ILSVRC-2010的结果总结在表1中。我们的网络达到了37.5%和17.0%5的前1和前5测试集错误率。在ILSVRC2010竞赛中，获得的最佳性能分别为47.1%和28.2%，其方法是对六个基于不同特征训练的稀疏编码模型产生的预测进行平均[2]，从那时起，发表的最佳结果是45.7%和25.7%，方法是平均两个接受Fisher向量（fv）训练的分类人员的预测，这两个分类人员是从两种密集采样特征计算出来的[24]。

• We also entered our model in the ILSVRC-2012 competition and report our results in Table 2. Since the ILSVRC-2012 test set labels are not publicly available, we cannot report test error rates for all the models that we tried. In the remainder of this paragraph, we use validation and test error rates interchangeably because in our experience they do not differ by more than 0.1% (seeTable2). The CNN described in this paper achieves a top-5 error rate of 18.2%. Averaging the predictions of ?ve similar CNNs gives an error rate of 16.4%. Training one CNN, with an extra sixth convolutional layer over the last pooling layer, to classify the entire ImageNet Fall 2011 release (15M images, 22K categories), and then “?ne-tuning” it on ILSVRC-2012 gives an error rate of 16.6%. Averaging the predictions of two CNNs that were pre-trained on the entire Fall 2011 release with the aforementioned ?ve CNNs gives an error rate of 15.3%. The second-best contest entry achieved an error rate of 26.2% with an approach that averages the predictions of several classi?ers trained on FVs computed from different types of densely-sampled features [7]

  我们还在ILSVRC-2012比赛中输入了我们的模型，并在表2中报告了我们的结果。由于ILSVRC-2012测试集标签不公开，因此我们无法报告我们尝试的所有模型的测试错误率。在本段的其余部分中，我们交替使用验证和测试错误率，因为根据我们的经验，它们之间的差异不超过0.1%（见表2）。本文所描述的CNN的前5位错误率为18.2%。对五个相似CNN的预测进行平均，误差率为16.4%。训练一个CNN，在最后一个池层上有额外的第六个卷积层，对2011年秋季发布的整个ImageNet进行分类（1500万张图像，22K个类别），然后在ILSVRC-2012上对其进行“微调”，错误率为16.6%。用上述五个CNN对2011年秋季发布的两个CNN的预测进行平均，误差率为15.3%。第二名参赛者的错误率为26.2%，其方法是将几个接受过FVs训练的分类者的预测从不同类型的密集采样特征中计算出来平均[7]

  

• Finally, we also report our error rates on the Fall 2009 version of ImageNet with 10,184 categories and 8.9 million images. On this dataset we follow the convention in the literature of using half of the images for training and half for testing. Since there is no established test set, our split necessarily differs from the splits used by previous authors,but this does not affect the results appreciably. Our top-1 and top-5 error rates on this dataset are 67.4% and 40.9%,attained by the net described above but with an additional,sixth convolutional layer over the last pooling layer. The best published results on this dataset are 78.1% and 60.9% [19].

  最后，我们还报告了2009年秋季版ImageNet的错误率，共有10184个类别和890万张图像。在这个数据集上，我们遵循文献中使用一半图像进行训练和一半图像进行测试的惯例。由于没有已建立的测试集，我们的分割必然不同于先前作者使用的分割，但这不会对结果产生明显的影响。在这个数据集上，我们的前1和前5错误率分别为67.4%和40.9%，通过上述网络实现，但是在最后一个池层上增加了第6个卷积层。在这个数据集上发表的最佳结果是78.1%和60.9%[19]。

  

  6.1 QualitativeEvaluations

• Figure 3 shows the convolutional kernels learned by the network’s two data-connected layers. The network has learned avariety of frequency-andorientation-selective kernels,as well as various colored blobs. Notice the specialization exhibited by the two GPUs, a result of the restricted connectivity described in Section 3.5. The kernels on GPU 1 are largely color-agnostic, while the kernels on on GPU 2 are largely color-speci?c. This kind of specialization occurs during every run and is independent of any particular random weight initialization (modulo a renumbering of the GPUs).

  图3显示了由网络的两个数据连接层学习的卷积核。该网络学习到了频率和方向选择核的丰富性，以及各种颜色的斑点。注意，两个gpu所显示的专门化，这是第3.5节中描述的受限连接的结果。GPU 1上的内核主要是颜色不可知的，而GPU 2上的内核主要是颜色特定的。这种特殊化在每次运行期间发生，并且独立于任何特定的随机权重初始化（GPU的模重新编号）。

• Figure 4: (Left) Eight ILSVRC-2010 test images and the ?ve labels considered most probable by our model. The correct label is written under each image, and the probability assigned to the correct label is also shown with a red bar(if it happens to be in the top5). (Right)Five ILSVRC-2010 test images in the ?rst column. The remaining columns show the six training images that produce feature vectors in the last hidden layer with the smallest Euclidean distance from the feature vector for the test image.

  图4：（左）八张ILSVRC-2010测试图像和五个标签被我们的模型认为是最有可能的。正确的标签写在每个图像下，并且分配给正确标签的概率也用红色条显示（如果它恰好在前5位）。（右）第一列中的五张ILSVRC-2010测试图像。其余的列显示了在最后一个隐藏层中生成特征向量的六个训练图像，与测试图像的特征向量的欧氏距离最小。
  

• In the left panel of Figure 4 we qualitatively assess what the network has learned by computing its top-5 predictions on eight test images. Notice that even off-center objects, such as the mite in the top-left, can be recognized by the net. Most of the top-5 labels appear reasonable. For example, only other types of cat are considered plausible labels for the leopard. In some cases (grille, cherry) there is genuine ambiguity about the intended focus of the photograph.

  在图4的左面板中，我们通过计算8张测试图像上的前5个预测，定性地评估网络所学到的知识。请注意，即使是偏离中心的对象（例如左上角的斜接）也可以被网络识别。排名前五的大多数标签看起来都很合理。例如，只有其他类型的猫才被认为是豹的合理标签。在某些情况下（格栅，樱桃色），照片的焦点确实模棱两可。

• Another way to probe the network’s visual knowledge is to consider the feature activations induced by an image at the last, 4096-dimensional hidden layer. If two images produce feature activation vectors with a small Euclidean separation, we can say that the higher levels of the neural network consider them to be similar. Figure 4 shows ?ve images from the test set and the six images from the training set that are most similar to each of them according to this measure. Notice that at the pixellevel,there trieved training images are generally not close in L2 to the query images in the ?rst column. For example, the retrieved dogs and elephants appear in a variety of poses. We present the results for many more test images in the supplementary material.

  另一种探索网络视觉知识的方法是考虑图像在最后4096维隐藏层的特征激活。如果两幅图像用一个小的欧几里德分离产生特征激活向量，我们可以说神经网络的高层认为它们是相似的。图4显示了来自测试集的五幅图像和来自训练集的六幅图像，根据这一度量，这六幅图像与它们中的每一幅最为相似。请注意，在像素级，检索到的训练图像通常不在第1列的查询图像的第2级。例如，找回的狗和大象以各种姿势出现。我们在补充材料中提供了更多测试图像的结果。

• Computing similarity by using Euclidean distance between two 4096-dimensional, real-valued vectors is inef?cient,but it could be made ef?cient by training anauto-encoder to compress these vectors to short binary codes. This should produce a much better image retrieval method than applying autoencoders to the raw pixels [14], which does not make use of image labels and hence has a tendency to retrieve images with similar patterns of edges, whether or not they are semantically similar.

  利用两个4096维实值向量之间的欧几里德距离计算相似度是不合适的，但可以通过训练自动编码器将这些向量压缩为短二进制码来提高相似度。这将产生比对原始像素应用自动编码器更好的图像检索方法[14]，后者不使用图像标签，因此具有检索具有相似边缘模式的图像的趋势，无论它们在语义上是否相似。

7. Discussion

• Our results show that a large, deep convolutional neural network is capable of achieving recordbreaking results on a highly challenging dataset using purely supervised learning. It is notable that our network’s performance degrades if a single convolutional layer is removed. For example, removing any of the middle layers results in a loss of about 2% for the top-1 performance of the network. So the depth really is important for achieving our results.

  我们的结果表明，一个大型的、深卷积的神经网络能够使用纯监督学习在一个高挑战性的数据集上获得破记录的结果。值得注意的是，如果去掉一个卷积层，我们的网络性能就会下降。例如，删除任何中间层都会导致网络的最高性能损失约2%。所以深度对我们的成果非常重要。

• To simplify our experiments, we did not use any unsupervised pre-training even though we expect that it will help, especially if we obtain enough computational power to signi?cantly increase the size of the network without obtaining a corresponding increase in the amount of labeled data. Thus far, our results have improved as we have made our network larger and trained it longer but we still have many orders of magnitude to go in order to match the infero-temporal pathway of the human visual system. Ultimately we would like to use very large and deep convolutional nets on video sequences where the temporal structure provides very helpful information that is missing or far less obvious in static images.

  为了简化我们的实验，我们没有使用任何无监督的预训练，即使我们期望它会有帮助，特别是如果我们获得足够的计算能力，显著增加网络的大小，而不获得相应的增加标记数据的数量。到目前为止，我们的结果已经有所改善，因为我们已经使我们的网络更大，训练时间更长，但我们仍然有许多数量级去匹配人类视觉系统的下时间路径。最后，我们希望在视频序列上使用非常大和深度的卷积网络，其中时间结构提供非常有用的信息，这些信息在静态图像中丢失或不太明显。