The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

May combine two claims.

For the first pruning rounds, networks learn faster and reach higher test accuracy the more they are pruned.

“For the first pruning rounds, networks learn faster and reach higher test accuracy the more they are pruned”

May combine two claims.

Unlike winning tickets, the reinitialized networks learn increasingly slower than the original network and lose test accuracy after little pruning.

“Unlike winning tickets, the reinitialized networks learn increasingly slower than the original network and lose test accuracy after little pruning.”

May combine two claims.

The winning tickets learn faster as P_m decreases from 100% to 21%, at which point early-stopping occurs 38% earlier than for the original network.

“The winning tickets learn faster as P m decreases from 100% to 21%, at which point early-stopping occurs 38% earlier than for the original network.”

May combine two claims.

Test accuracy increases with pruning, improving by more than 0.3 percentage points when P_m = 13.5%; after this point, accuracy decreases, returning to the level of the original network when P_m = 3.6%.

“Test accuracy increases with pruning, improving by more than 0.3 percentage points when P m = 13 . 5% ; after this point, accuracy decreases, returning to the level of the original network when P m = 3 . 6% .”

May combine two claims.

Above this size, the winning tickets found learn faster than the original network and reach higher test accuracy.

“Above this size, the winning tickets that we find learn faster than the original network and reach higher test accuracy.”

May combine two claims.

Further pruning causes learning to slow, returning to the early-stopping performance of the original network when P_m = 3.6%.

“Further pruning causes learning to slow, returning to the early-stopping performance of the original network when P m = 3 . 6% .”

May combine two claims.

The winning tickets found are 10-20% (or less) of the size of the original network; down to that size, they meet or exceed the original network's test accuracy in at most the same number of iterations.

“The winning tickets we find are 10-20% (or less) of the size of the original network ( smaller size ). Down to that size, they meet or exceed the original network's test accuracy ( commensurate accuracy ) in at most the same number of iterations ( commensurate training time ).”

May combine two claims.

Randomly sampled sparse subnetworks learn more slowly and reach lower eventual test accuracy as sparsity increases.

“Across various levels of sparsity, dashed lines trace the iteration of minimum validation loss 2 and the test accuracy at that iteration. The sparser the network, the slower the learning and the lower the eventual test accuracy.”

May combine two claims.

Iteratively-pruned winning tickets learn faster and reach higher test accuracy at smaller network sizes than one-shot pruning.

“However, iterativelypruned winning tickets learn faster and reach higher test accuracy at smaller network sizes.”

May combine two claims.

Test accuracy improves at best 3.4 percentage points for Conv-2 (P_m = 4.6%), 3.5 for Conv-4 (P_m = 11.1%), and 3.3 for Conv-6 (P_m = 26.4%).

“Test accuracy improves at best 3.4 percentage points for Conv-2 ( P m = 4 . 6% ), 3.5 for Conv-4 ( P m = 11 . 1% ), and 3.3 for Conv-6 ( P m = 26 . 4% ).”

May combine two claims.

Winning tickets reach minimum validation loss at best 3.5x faster for Conv-2 (P_m = 8.8%), 3.5x for Conv-4 (P_m = 9.2%), and 2.5x for Conv-6 (P_m = 15.1%).

“Winning tickets reach minimum validation loss at best 3.5x faster for Conv-2 ( P m = 8 . 8% ), 3.5x for Conv-4 ( P m = 9 . 2% ), and 2.5x for Conv-6 ( P m = 15 . 1% ).”

May combine two claims.

Winning tickets both arrive at early-stopping sooner and reach higher test accuracy at this point.

“In the main body of the paper, we find that winning tickets both arrive at early-stopping sooner and reach higher test accuracy at this point.”

May combine two claims.

The paper reliably finds winning tickets that generalize better, exceeding the test accuracy of the original network while matching its training accuracy.

“We reliably find winning tickets that generalize better, exceeding the test accuracy of the original network while matching its training accuracy.”

May combine two claims.

Warmup makes it possible to find winning tickets, exceeding this initial accuracy when P_m ≥ 1.5%.

“Warmup makes it possible to find winning tickets, exceeding this initial accuracy when P m ≥ 1 . 5% .”

May combine two claims.

At the lower learning rate, the usual pattern reemerges, with subnetworks that remain within 1 percentage point of the original accuracy while P_m ≥ 3.5%.

“However, at the lower learning rate, the usual pattern reemerges, with subnetworks that remain within 1 percentage point of the original accuracy while P m ≥ 3 . 5% .”

May combine two claims.

Strategy 1 maintains higher validation accuracy and faster early-stopping times to smaller network sizes than Strategy 2.

“In all cases, the Strategy 1 maintains higher validation accuracy and faster early-stopping times to smaller network sizes.”

May combine two claims.

Validation loss initially drops, then forms a clear bottom, and then begins increasing again.

“In all cases, validation loss initially drops, after which it forms a clear bottom and then begins increasing again.”

May combine two claims.

After further pruning, initialization matters: the paper finds winning tickets when VGG-19 is pruned by up to 98.5%; when reinitialized, these tickets reach much lower accuracy.

“However, after further pruning, initialization matters: we find winning tickets when VGG-19 is pruned by up to 98.5%; when reinitialized, these tickets reach much lower accuracy.”

May combine two claims.

When P_m = 21%, the winning ticket reaches minimum validation loss 2.51x faster than when reinitialized and is half a percentage point more accurate.

“When P m = 21% , the winning ticket reaches minimum validation loss 2.51x faster than when reinitialized and is half a percentage point more accurate.”

May combine two claims.

Unlike Lenet, test accuracy at early-stopping time initially remains steady and even improves for Conv-2 and Conv-4, indicating that at moderate levels of pruning the structure of the winning tickets alone may lead to better accuracy.

“However, unlike Lenet, test accuracy at early-stopping time initially remains steady and even improves for Conv-2 and Conv-4, indicating that-at moderate levels of pruning-the structure of the winning tickets alone may lead to better accuracy.”

May combine two claims.

Learning becomes faster with iterative pruning as before, but less dramatically in the case of Conv-2.

“Learning becomes faster with iterative pruning as before, but less dramatically in the case of Conv-2.”

May combine two claims.

Winning tickets trained with warmup close the accuracy gap with the unpruned network at the higher learning rate, reaching 90.5% test accuracy with learning rate 0.03 (warmup, k = 20000) at P_m = 27.1%.

“Winning tickets trained with warmup close the accuracy gap with the unpruned network at the higher learning rate, reaching 90.5% test accuracy with learning rate 0.03 (warmup, k = 20000 ) at P m = 27 . 1% .”

May combine two claims.

Winning tickets can reach accuracy equivalent to that of the original, unpruned network, but with significantly fewer parameters.

“Winning tickets can reach accuracy equivalent to that of the original, unpruned network, but with significantly fewer parameters.”

May combine two claims.

At the higher learning rate, iterative pruning does not find winning tickets, and performance is no better than when the pruned networks are randomly reinitialized.

“At the higher learning rate, iterative pruning does not find winning tickets, and performance is no better than when the pruned networks are randomly reinitialized.”

May combine two claims.

At iteration 20,000 for Conv-2, 25,000 for Conv-4, and 30,000 for Conv-6, training accuracy reaches 100% for all networks when P_m ≥ 2% and winning tickets still maintain higher test accuracy.

“at iteration 20,000 for Conv-2, 25,000 for Conv-4, and 30,000 for Conv-6 (the iterations corresponding to the final training iteration for the original network), training accuracy reaches 100% for all networks when P m ≥ 2% (Appendix D, Figure 13) and winning tickets still maintain higher test accuracy”

May combine two claims.

The paper continues to find winning tickets for VGG-19 and Resnet-18 architectures; however, iterative pruning is sensitive to the particular learning rate used.

“We continue to find winning tickets for all of these architectures; however, our method for finding them, iterative pruning, is sensitive to the particular learning rate used.”

May combine two claims.

Although these subnetworks learn faster than the unpruned network early in training, this accuracy advantage erodes later in training due to the lower initial learning rate.

“Although these subnetworks learn faster than the unpruned network early in training (Figure 7 left), this accuracy advantage erodes later in training due to the lower initial learning rate.”

May combine two claims.

Dropout increases initial test accuracy (2.1, 3.0, and 2.4 percentage points on average for Conv-2, Conv-4, and Conv-6, respectively), and iterative pruning increases it further (up to an additional 2.3, 4.6, and 4.7 percentage points, respectively, on average).

“Dropout increases initial test accuracy (2.1, 3.0, and 2.4 percentage points on average for Conv-2, Conv-4, and Conv-6, respectively), and iterative pruning increases it further (up to an additional 2.3, 4.6, and 4.7 percentage points, respectively, on average).”

May combine two claims.

The pattern from Lenet repeats: as the network is pruned, it learns faster and test accuracy rises as compared to the original network.

“The pattern from Lenet in Section 2 repeats: as the network is pruned, it learns faster and test accuracy rises as compared to the original network.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In terms of accuracy, the lines in Figure 31 are approximately arranged from bottom to top in increasing order of network size.

“Put another way, in terms of accuracy, the lines are approximately arranged from bottom to top in increasing order of network size.”

May combine two claims.

For Conv-2, learning rate 0.0002 has the highest initial validation accuracy, maintains both high validation accuracy and low early-stopping times for among the longest, and reaches the fastest early-stopping times.

“For Conv-2, we select a learning rate of 0.0002, which has the highest initial validation accuracy, maintains both high validation accuracy and low early-stopping times for the among the longest, and reaches the fastest early-stopping times.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The output layer of Lenet is so small (1,000 weights out of 266,000) that pruning it reaches a point of diminishing returns much earlier than the other layers.

“Since the output layer is so small (1,000 weights out of 266,000 for the overall Lenet architecture), we found that pruning it reaches a point of diminishing returns much earlier the other layers.”

May combine two claims.

Among Lenet Adam learning rates between 0.0002 and 0.002, 0.0012 and 0.002 produce the fastest early-stopping times and maintain them to the smallest network sizes.

“Of those learning rates, 0.0012 and 0.002 produce the fastest early-stopping times and maintain them to the smallest network sizes.”

May combine two claims.

For Conv-6, learning rate 0.0002 shows greater improvements in early-stopping times (6.26x when 19.7% of weights remain) but reaches lower overall validation accuracy.

“and learning rate 0.0002 shows greater improvements in early-stopping times (6.26x when 19.7% of weights remain) but reaches lower overall validation accuracy.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The second hidden layer and the output layer become increasingly bimodal, with peaks on either side of 0.

“However, the second hidden layer and the output layer become increasingly bimodal, with peaks on either side of 0.”

May combine two claims.

Adding Gaussian noise to winning ticket initializations reduces the test accuracy of a winning ticket and slows its ability to learn.

“Adding Gaussian noise reduces the test accuracy of a winning ticket and slows its ability to learn, again demonstrating the importance of the original initialization.”

May combine two claims.

Lenet SGD learning rates 0.4 and 0.8 reach early-stopping in a similar number of iterations as the best Adam learning rates (0.0012 and 0.002) but maintain performance when pruned further (to less than 1% of original size for SGD vs about 3.6% for Adam).

“SGD learning rates 0.4 and 0.8 reach early-stopping in a similar number of iterations as the best Adam learning rates (0.0012 and 0.002) but maintain this performance when the network has been pruned further (to less than 1% of its original size for SGD vs. about 3.6% of the original size for Adam).”

Probably not a substantial claim: may not pass S3 (load-bearing).

As the network is pruned, the first hidden layer maintains its distribution.

“As the network is pruned, the first hidden layer maintains its distribution.”

May combine two claims.

For Conv-4, other learning rates reach higher validation accuracy (0.0004-3.6 percentage point improvement to 79.1% accuracy when 5.4% of weights remain) or show better improvements in early-stopping times (0.0002-5.1x faster when 9.2% of weights remain) but not both.

“Other learning rates reach higher validation accuracy (0.0004-3.6 percentage point improvement to 79.1% accuracy when 5.4% of weights remain) or show better improvements in early-stopping times (0.0002-5.1x faster when 9.2% of weights remain) but not both.”

May combine two claims.

The peaks are of different sizes, with the first hidden layer favoring negative initializations and the second hidden layer and output layer favoring positive initializations.

“Just as with the adam-optimized winning tickets, these peaks are of different sizes, with the first hidden layer favoring negative initializations and the second hidden layer and output layer favoring positive initializations.”

May combine two claims.

The lowest Lenet iterative pruning rates (0.1 and 0.2) reach higher validation accuracy and maintain that validation accuracy to smaller network sizes than higher pruning rates.

“The lowest pruning rates reach higher validation accuracy and maintain that validation accuracy to smaller network sizes; they also maintain fast early-stopping times to smaller network sizes.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Upon exceedingly close inspection, winning tickets derived from initially larger Lenet networks tend to learn marginally faster than winning tickets derived from initially smaller networks, but these differences are slight.

“Upon exceedingly close inspection, winning tickets derived from initially larger networks tend to learn marginally faster than winning tickets derived from initially smaller networks, but these differences are slight.”

May combine two claims.

For Conv-2, learning rate 0.0004 has lower initial validation accuracy (65.2% vs 67.6%) but eventually reaches higher absolute validation accuracy (71.7%, a 6.5 percentage point increase, vs. 70.9%, a 3.3 percentage point increase) than learning rate 0.0002.

“Other learning rates, such 0.0004, have lower initial validation accuracy (65.2% vs 67.6%) but eventually reach higher absolute levels of validation accuracy (71.7%, a 6.5 percentage point increase, vs. 70.9%, a 3.3 percentage point increase).”

May combine two claims.

On pruned Lenet networks, SGD learning rates 0.4 and 0.8 achieve equivalent accuracy to the best Adam learning rates and maintain that high accuracy when pruned as much as the Adam learning rates.

“Likewise, on pruned networks, these SGD learning rates achieve equivalent accuracy to the best Adam learning rates, and they maintain that high accuracy when the network is pruned as much as the Adam learning rates.”

May combine two claims.

For Conv-6, learning rate 0.0004 reaches high final validation accuracy (81.9%, an increase of 2.7 percentage points, when 15.2% of weights remain) but with smaller improvements in early-stopping times.

“Learning rate 0.0004 reaches high final validation accuracy (81.9%, an increase of 2.7 percentage points, when 15.2% of weights remain) but with smaller improvements in early-stopping times”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The input layer has two peaks, one larger peak for input units with a high number of outgoing connections and one smaller peak for input units with a low number of outgoing connections.

“Indeed, the input layer has two peaks, one larger peak for input units with a high number of outgoing connections and one smaller peak for input units with a low number of outgoing connections.”

May combine two claims.

Several Lenet Adam learning rates between 0.0002 and 0.002 achieve similar validation accuracy on the original network and maintain that performance to similar levels as the network is pruned.

“According to the graph on the right of Figure 26, several learning rates between 0.0002 and 0.002 achieve similar levels of validation accuracy on the original network and maintain that performance to similar levels as the network is pruned.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For Lenet optimized with SGD with momentum (0.9), learning rate 0.025 achieves the highest validation accuracy on the unpruned network.

“Learning rate 0.025 achieves the highest validation accuracy on the unpruned network; however, its validation accuracy never increases as it is pruned, instead decreasing gradually, and higher learning rates reach early-stopping faster.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The peaks are asymmetric: the second hidden layer has more positive initializations remaining than negative initializations, and the reverse is true for the output layer.

“Interestingly, the peaks are asymmetric: the second hidden layer has more positive initializations remaining than negative initializations, and the reverse is true for the output layer.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

When initialized from a Gaussian distribution with standard deviation 0.1, the Lenet architecture maintained high validation accuracy and low early-stopping times for the longest, approximately matching the performance of the Glorot-initialized network.

“When initialized from a Gaussian distribution with standard deviation 0.1, the Lenet architecture maintained high validation accuracy and low early-stopping times for the longest, approximately matching the performance of the Glorot-initialized network.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The adam-trained winning tickets develop a much more uneven distribution of outgoing connectivity for the input layer than does the SGD-trained network.

“Interestingly, the adam-trained winning tickets develop a much more uneven distribution of outgoing connectivity for the input layer than does the SGD-trained network (Figure 24).”

May combine two claims.

For Conv-4, learning rate 0.0003 has among the highest initial validation accuracy, maintains high validation accuracy and fast early-stopping times when pruned by among the most, and balances improvements in validation accuracy (3.7 percentage point improvement to 78.6% when 5.4% of weights remain) and improvements in early-stopping time (4.27x when 11.1% of weights remain).

“For Conv-4, we select learning rate 0.0003, which has among the highest initial validation accuracy, maintains high validation accuracy and fast early-stopping times when pruned by among the most, and balances improvements in validation accuracy (3.7 percentage point improvement to 78.6% when 5.4% of weights remain) and improvements in early-stopping time (4.27x when 11.1% of weights remain).”

May combine two claims.

For Conv-6, learning rate 0.0003 improves validation accuracy by 2.4 percentage points to 81.5% when 9.31% of weights remain and improves early-stopping times by 2.61x when pruned to 11.9%.

“For Conv-6, we also select learning rate 0.0003 for similar reasons to those provided for Conv-4. Validation accuracy improves by 2.4 percentage points to 81.5% when 9.31% of weights remain and early-stopping times improve by 2.61x when pruned to 11.9%.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

For the experiments in Section 4, k = 20000 is selected for Resnet-18, which achieves the highest validation accuracy.

“For the experiments in Section 4, we select k = 20000 , which achieves the highest validation accuracy.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The Conv-2 network proved difficult to consistently train with dropout.

“The Conv-2 network proved to be difficult to consistently train with dropout.”

May combine two claims.

In Section 4, iterative pruning is unable to find winning tickets for VGG-19 and Resnet-18 at the typical high learning rate 0.1 but is able to do so at a lower learning rate 0.01.

“In Section 4, we observe that iterative pruning is unable to find winning tickets for VGG-19 and Resnet-18 at the typical, high learning rate used to train the network (0.1) but it is able to do so at a lower learning rate (0.01).”

May combine two claims.

Regardless of whether layer-wise or global pruning is used, the patterns from Section 4 hold: at learning rate 0.1, iterative pruning finds winning tickets for neither VGG-19 nor Resnet-18; at learning rate 0.01, the lottery ticket pattern reemerges; and when training with warmup to a higher learning rate, iterative pruning finds winning tickets.

“Regardless of whether we use layer-wise or global pruning, the patterns from Section 4 hold: at learning rate 0.1, iterative pruning finds winning tickets for neither network; at learning rate 0.01, the lottery ticket pattern reemerges; and when training with warmup to a higher learning rate, iterative pruning finds winning tickets.”

May combine two claims. The quote only partly states this claim.

At higher learning rates than those in Figure 32, gradients tended to explode when training the unpruned Conv-2, Conv-4, and Conv-6 networks; at lower learning rates, the networks often failed to learn at all.

“At higher learning rates than those in Figure 32, gradients tended to explode when training the unpruned network; at lower learning rates, the networks often failed to learn at all.”

May combine two claims.

In all three cases (Conv-2, Conv-4, Conv-6), pruning convolutions alone leads to higher test accuracy and faster learning.

“In all three cases, pruning convolutions alone leads to higher test accuracy and faster learning; pruning fully-connected layers alone generally causes test accuracy to worsen and learning to slow.”

May combine two claims.

When Conv-2 and Conv-4 SGD pruned networks eventually surpassed the accuracy of the original network, they reached early-stopping in about the same or fewer iterations than the original network, constituting a winning ticket by the paper's definition.

“When they eventually did surpass the accuracy of the original network, the pruned networks reached early-stopping in about the same or fewer iterations than the original network, constituting a winning ticket by our definition.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For the experiments in Section 4, k = 10000 is selected for VGG-19, as there is little benefit to larger values of k.

“For the experiments in Section 4, we select k = 10000 , as there is little benefit to larger values of k .”

May combine two claims.

For Conv-4 and Conv-6 with dropout, a slightly slower learning rate (0.0002 as opposed to 0.0003) leads to the highest accuracy on the unpruned networks in addition to the highest sustained accuracy and fastest sustained learning as the networks are pruned during the lottery ticket experiment.

“For both the Conv-4 and Conv-6 architectures, a slightly slower learning rate (0.0002 as opposed to 0.0003) leads to the highest accuracy on the unpruned networks in addition to the highest sustained accuracy and fastest sustained learning as the networks are pruned during the lottery ticket experiment.”

May combine two claims.

For Conv-4 with dropout, early-stopping times improve by up to 1.58x (when pruned to 7.6%), a smaller improvement than the 4.27x achieved by a winning ticket obtained without dropout.

“Early-stopping times improve by up to 1.58x (when pruned to 7.6%), a smaller improvement than then 4.27x achieved by a winning ticket obtained without dropout.”

May combine two claims.

In general, iterative pruning cannot find winning tickets at any learning rate above 0.01 for either VGG-19 or Resnet-18; for higher learning rates, the pruned networks with the original initialization perform no better than when randomly reinitialized.

“In general, iterative pruning cannot find winning tickets at any rate above 0.01 for either network; for higher learning rates, the pruned networks with the original initialization perform no better than when randomly reinitialized.”

May combine two claims.

For VGG-19, for values of k below 5000, accuracy improves rapidly as k increases; this relationship reaches a point of diminishing returns above k = 5000.

“For values of k below 5000, accuracy improves rapidly as k increases. This relationship reaches a point of diminishing returns above k = 5000 .”

May combine two claims.

With dropout, the unpruned Conv-6 architecture reaches an average validation accuracy of 81.3%, an improvement of 2.2 percentage points over the accuracy without dropout; this nearly matches the 81.5% average accuracy obtained by Conv-6 trained without dropout and pruned to 9.31%.

“With dropout, the unpruned Conv-6 architecture reaches an average validation accuracy of 81.3%, an improvement of 2.2 percentage points over the accuracy without dropout; this nearly matches the 81.5% average accuracy obtained by Conv-6 trained without dropout and pruned to 9.31%.”

May combine two claims.

In all three cases (Conv-2, Conv-4, Conv-6), pruning fully-connected layers alone generally causes test accuracy to worsen and learning to slow.

“In all three cases, pruning convolutions alone leads to higher test accuracy and faster learning; pruning fully-connected layers alone generally causes test accuracy to worsen and learning to slow.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The Conv-6 network exhibited the same accuracy patterns as other experiments with SGD, with validation accuracy initially increasing with pruning before eventually decreasing again.

“The Conv-6 network also exhibited the same accuracy patterns as other experiments, with validation accuracy initially increasing with pruning before eventually decreasing again.”

May combine two claims.

We could not find SGD learning rates for which the unpruned Conv-2, Conv-4, and Conv-6 networks matched the validation accuracy of the same networks when trained with Adam; at best, the SGD-trained unpruned networks were typically 2-3 percentage points less accurate.

“As Figure 33 reflects, we could not find SGD learning rates for which the unpruned networks matched the validation accuracy of the same networks when trained with Adam; at best, the SGD-trained unpruned networks were typically 2-3 percentage points less accurate.”

May combine two claims.

For Resnet-18, for values of k below 5000, accuracy improves rapidly as k increases; this relationship reaches a point of diminishing returns above k = 5000.

“For values of k below 5000, accuracy improves rapidly as k increases. This relationship reaches a point of diminishing returns above k = 5000 .”

May combine two claims.

With dropout, the unpruned Conv-4 architecture reaches an average validation accuracy of 77.6%, a 2.7 percentage point improvement over the unpruned Conv-4 network trained without dropout and one percentage point lower than the highest average validation accuracy attained by a winning ticket.

“With dropout, the unpruned Conv-4 architecture reaches an average validation accuracy of 77.6%, a 2.7 percentage point improvement over the unpruned Conv-4 network trained without dropout and one percentage point lower than the highest average validation accuracy attained by a winning ticket.”

May combine two claims.

In Section 4, adding linear warmup to the initial learning rate makes it possible to find winning tickets for VGG-19 and Resnet-18 at higher learning rates (and thereby winning tickets that reach higher accuracy).

“In Section 4, we describe how adding linear warmup to the initial learning rate makes it possible to find winning tickets for VGG-19 and Resnet-18 at higher learning rates (and, thereby, winning tickets that reach higher accuracy).”

May combine two claims.

The graphs for VGG-19 and Resnet-18 with layer-wise pruning follow the same trends as in Section 4, but the smallest winning tickets are larger than those found by global pruning.

“The graphs follow the same trends as in Section 4, but the smallest winning tickets are larger than those found by global pruning.”

May combine two claims.

The Conv-2 and Conv-4 architectures with SGD exhibited a different validation accuracy pattern: accuracy initially declined with pruning before rising as the network was further pruned, eventually matching or surpassing the accuracy of the unpruned network.

“However, the Conv-2 and Conv-4 architectures exhibited a different validation accuracy pattern from other experiments in this paper. Accuracy initially declined with pruning before rising as the network was further pruned; it eventually matched or surpassed the accuracy of the unpruned network.”

Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

For Conv-2, Conv-4, and Conv-6 optimized with SGD with momentum (0.9), at the very lowest learning rates, accuracy initially decreased before increasing to higher levels than reached by the unpruned network.

“At the very lowest learning rates (e.g., learning rate 0.001 for Conv-4 and all but the highest learning rate for Conv-2), accuracy initially decreased before increasing to higher levels than reached by the unpruned network; this is the same pattern we observed when training these networks with SGD.”

May combine two claims.

Global pruning also finds smaller winning tickets than layer-wise pruning for Resnet-18, but the difference is less extreme than for VGG-19.

“Global pruning also finds smaller winning tickets than layer-wise pruning for Resnet-18, but the difference is less extreme than for VGG-19.”

Probably not a substantial claim: may not pass S3 (load-bearing).

At higher SGD learning rates for Conv-2 and Conv-4, gradients exploded on the unpruned networks, preventing the experiments.

“Unfortunately, at these higher learning rates, gradients exploded on the unpruned networks, preventing us from running these experiments.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For Conv-2 with dropout at learning rate 0.0004, one network occasionally failed to learn.

“At learning rate 0.0004, one network occasionally failed to learn.”

May combine two claims.

The lottery ticket pattern fails to hold at the very highest learning rates for Conv-2, Conv-4, and Conv-6: early-stopping times decreased only briefly (Conv-2 or Conv-4) or not at all (Conv-6), and accuracy increased only briefly (all three networks).

“This pattern fails to hold at the very highest learning rates: early-stopping times decreased only briefly (in the case of Conv-2 or Conv-4) or not at all (in the case of Conv-6), and accuracy increased only briefly (in the case of all three networks).”

The quote only partly states this claim.

For Conv-2, Conv-4, and Conv-6 optimized with SGD with momentum (0.9), at the very highest learning rates, early-stopping times never decreased and instead remained stable before increasing.

“At the very highest learning rates (e.g., learning rates 0.005 and 0.008 for Conv-2 and Conv4), early-stopping times never decreased and instead remained stable before increasing; this is the same pattern we observed for the highest learning rates when training with Adam.”

May combine two claims. The quote only partly states this claim.

For Conv-2, Conv-4, and Conv-6 optimized with SGD with momentum (0.9), the lottery ticket pattern continues to apply, with early-stopping times decreasing and accuracy increasing as the networks are pruned.

“In general, the lottery ticket pattern continues to apply, with early-stopping times decreasing and accuracy increasing as the networks are pruned.”

May combine two claims.

Sparse architectures produced by pruning are difficult to train from the start, reaching lower accuracy than the original networks.

“However, contemporary experience is that the sparse architectures produced by pruning are difficult to train from the start, which would similarly improve training performance.”

• Also stated in: § 1 INTRODUCTION · p. 1 · plain

  “Contemporary experience is that the architectures uncovered by pruning are harder to train from the start, reaching lower accuracy than the original networks.”

• cites Li et al., 2016Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Han et al., 2015 sourceNot checked — no full text of the source

May combine two claims.

When randomly reinitialized, winning tickets perform far worse, meaning structure alone cannot explain a winning ticket's success.

“When randomly reinitialized, winning tickets perform far worse, meaning structure alone cannot explain a winning ticket's success.”

May combine two claims.

The gap between training accuracy and test accuracy is smaller for winning tickets, pointing to improved generalization.

“This means that the gap between training accuracy and test accuracy is smaller for winning tickets, pointing to improved generalization.”

May combine two claims.

The gap between test and training accuracy is smaller for winning tickets, indicating they generalize better.

“This means that the gap between test and training accuracy is smaller for winning tickets, indicating they generalize better.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Each line in each graph of Appendix H represents the average of three separate experiments, with error bars indicating the minimum and maximum value that any experiment took on at that point.

“Each line in each graph of this section represents the average of three separate experiments, with error bars indicating the minimum and maximum value that any experiment took on at that point.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Experiments in the main body of the paper are conducted five times.

“(Experiments in the main body of the paper are conducted five times.)”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Validation and test performance is only considered retroactively and has no effect on the progression of the lottery ticket experiments.

“As with the MNIST experiments, validation and test performance is only considered retroactively and has no effect on the progression of the lottery ticket experiments.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

On each iteration of the lottery ticket experiment, the network is trained for 50,000 training iterations regardless of when early-stopping occurs; no validation or test data is taken into account during the training process, and early-stopping times are determined retroactively by examining validation performance.

“On each iteration of the lottery ticket experiment, the network is trained for 50,000 training iterations regardless of when early-stopping occurs; in other words, no validation or test data is taken into account during the training process, and early-stopping times are determined retroactively by examining validation performance.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The CIFAR10 dataset consists of 50,000 32x32 color (three-channel) training examples and 10,000 test examples.

“The CIFAR10 dataset consists of 50,000 32x32 color (three-channel) training examples and 10,000 test examples.”

• cites Krizhevsky & Hinton, 2009Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The different accuracy pattern also appeared for Conv-6 networks at slower SGD learning rates, suggesting that faster learning rates for Conv-2 and Conv-4 than those in Figure 32 might cause the usual lottery ticket accuracy pattern to reemerge.

“Interestingly, this pattern also appeared for Conv-6 networks at slower SGD learning rates, suggesting that faster learning rates for Conv-2 and Conv-4 than those in Figure 32 might cause the usual lottery ticket accuracy pattern to reemerge.”

The quote only partly states this claim.

At the highest learning rates, the iterative pruning algorithm fails to find winning tickets for Conv-2, Conv-4, and Conv-6, similar to Section 4.

“This pattern is similar to that which we observe in Section 4: at the highest learning rates, our iterative pruning algorithm fails to find winning tickets.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The Resnet-18 architecture was first introduced by He et al. (2016).

“The Resnet-18 architecture was first introduced by He et al. (2016).”

• cites He et al. (2016)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Liu et al. (2019) uses an initial pruning rate of 0.1.

“Liu et al. (2019) uses an initial pruning rate of 0.1.”

• cites Liu et al. (2019)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

The version of VGG19 used here was adapted by Liu et al. (2019) for CIFAR10.

“The version that we use here was adapted by Liu et al. (2019) for CIFAR10.”

• cites Liu et al. (2019)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.