4 Tutorial on deep probabilitic modeling with Pyro

import torch
import pyro
pyro.set_rng_seed(101)

4.1 Recap on Motivation

Our goal is to understand causal modeling within the context of generative machine learning. We just examined one generative machine learning framework called Bayesian networks (BNs) and how we can use BNs as causal models.

Bayesian Networks (BNs) Framework that defines a probabilistic generative model of the world in terms of a directed acyclic graph.

causal Bayesian networks: Bayesian networks where the direction of edges in the DAG represent causality.

Bayesian networks provide a general-purpose framework for representing a causal data generating story for how the world works.

Now we will introduce probabilistic programming, a framework that is more expressive than Bayesian networks.

4.1.1 What is a probabilistic programming language?

“A probabilistic programming language (PPL) is a programming language designed to describe probabilistic models and then perform inference in those models. PPLs are closely related to graphical models and Bayesian networks but are more expressive and flexible. Probabilistic programming represents an attempt to”Unify general purpose programming" with probabilistic modeling."

-Wikipedia

A PPL is a domain-specific programming language for that lets you write a data generating story as a program. As with a causal Bayesian network, you can write your program in a way that orders the steps of its execution according to cause and effect.

4.1.2 How exactly do Bayesian networks and probabilistic programming differ?

Representation of relationships between variables. BNs restricted to representing the relationships between variables in terms of conditional probability distributions (CPDs) factored according to a DAG. Frameworks typically limit you to a small set of parametric CPDs (e.g., Gaussian, multinomial).

Just as computer programs are more expressive than flow charts, PPLs let you represent relations any way you like so long as you can represent them in code. PPL relationships can include control flow and recursion. In causal models, we will see that this allows you to be more specific about mechanism than you can with CPDs.

DAG vs. open world models. BNs restrict the representation of the joint distribution to a DAG. This constraint enables you to reason easily about the joint distribution through graph-theoretic operations like d-separation. PPLs need not be constrained to a DAG. For example (using an imaginary Python PPL package):

X = Bernoulli(p)
if X == 1:
    Y = Gaussian(0, 1)

In a DAG, you have a fixed set of variables, i.e. a “closed world”. In the above model, the variable Y is only instantiated if X==1. Y may or may not exist depending on how the generative process unfolds. For a more extreme example, consider this:

X = Poisson(λ)
Y = [Gaussian(0, 1)]
for i in range(1, X):
    Y[i] = Gaussian(Y[i-1], 1))

Here you have the total number of Y variables itself being a random variable X. Further, the mean of the ith Y is a random variable given by the i-1th Y. You can’t do that with a Bayes net! Unfortunately, we can’t reason about this as directly as we can with a DAG. For example, recall that with the DAG, we had a convenient algorithm called CPDAG that converts the DAG to a partially directed acyclic graph structure called a PDAG that provides a compact representation of all the DAGs in an equivalence class. How might we define an equivalence class on this program? Certainly, enumerating all programs with an equivalent representation of the joint distribution would be very difficult even with constraints on the length of the program. In general, enumerating all programs of minimal description that provide equivalent representations of a joint distribution is an NP-hard problem.

Inference When you have a DAG and a constrained set of parametric CPDs, as well as constraints on the kind of inference, queries the user can make, you can implement some inference algorithms in your BN framework that will generally work in a reasonable amount of time.

PPLs are more flexible than BNs, but the trade-off s that getting inference to work is harder. PPL’s develop several abstractions for inference and leave it to the user to apply them, requiring the user to become something of an expert in inference algorithms. PPL developers make design decisions to make inference easier for the user, though this often sacrifices some flexibility. One emergent pattern is to build PPLs on tensor-based frameworks like Tensorflow and PyTorch. Tensor-based PPLs allow a data scientist with experience building deep learning models to rely on that experience when doing inference.

Image

\[\texttt{Kevin Smith - Tutorial: Probabilistic Programming}\]

4.2 Introduction to Pyro

Pyro is a universal probabilistic programming language (PPL) written in Python and supported by PyTorch on the backend. Pyro enables flexible and expressive deep probabilistic modeling, unifying the best of modern deep learning and Bayesian modeling.

Our purpose of this class, pyro has “do”-operator that allows intervention and counterfactual inference in these probabilistic models.

4.2.1 Stochastic Functions

The basic unit of probabilistic programs is the stochastic function. A stochastic function is an arbitrary Python callable that combines two ingredients:

deterministic Python code; and
primitive stochastic functions that call a random number generator

For this course, we will consider these stochastic functions as models. Stochastic functions can be used to represent simplified or abstract descriptions of a data-generating process.

4.2.2 Primitive stochastic functions

We call them distributions. We can explicitly compute the probability of the outputs given the inputs.

loc = 0.   # mean zero
scale = 1. # unit variance
normal = torch.distributions.Normal(loc, scale) # create a normal distribution object
x = normal.rsample() # draw a sample from N(0,1)
print("sample: ", x)

## sample:  tensor(-1.3905)

Pyro simplifies this process of sampling from distributions. It uses pyro.sample().

x = pyro.sample("my_sample", pyro.distributions.Normal(loc, scale))
print(x)

## tensor(-0.8152)

Just like a direct call to torch.distributions.Normal().rsample(), this returns a sample from the unit normal distribution. The crucial difference is that this sample is named. Pyro’s backend uses these names to uniquely identify sample statements and change their behavior at runtime depending on how the enclosing stochastic function is being used. This is how Pyro can implement the various manipulations that underlie inference algorithms.

Let’s write a simple weather model.

4.2.3 A simple model

import pyro.distributions as dist
def weather():
    cloudy = pyro.sample('cloudy', dist.Bernoulli(0.3))
    cloudy = 'cloudy' if cloudy.item() == 1.0 else 'sunny'
    mean_temp = {'cloudy': 55.0, 'sunny': 75.0}[cloudy]
    scale_temp = {'cloudy': 10.0, 'sunny': 15.0}[cloudy]
    temp = pyro.sample('temp', dist.Normal(mean_temp, scale_temp))
    return cloudy, temp.item()
for _ in range(3):
    print(weather())

## ('cloudy', 64.5440444946289)
## ('sunny', 94.37557983398438)
## ('sunny', 72.5186767578125)

First two lines introduce a binary variable cloudy, which is given by a draw from the Bernoulli distribution with a parameter of \(0.3\). The Bernoulli distribution returns either \(0\) or \(1\), line 2 converts that into a string. So, So according to this model, \(30%\) of the time it’s cloudy and \(70%\) of the time it’s sunny.

In line 4 and 5, we initialize mean and scale of the temperature for both values. We then sample, the temperature from a Normal distribution and return that along with cloudy variable.

We can build complex model by modularizing and reusing the concepts into functions and use them as programmers use functions.

def ice_cream_sales():
    cloudy, temp = weather()
    expected_sales = 200. if cloudy == 'sunny' and temp > 80.0 else 50.
    ice_cream = pyro.sample('ice_cream', pyro.distributions.Normal(expected_sales, 10.0))
    return ice_cream

4.3 Inference

As we discussed earlier, the reason we use PPLs is because they can easily go backwards and reason about cause given the observed effect. There are myriad of inference algorithms available in pyro. Let’s try it on an even simpler model.

\[weight \mid guess \sim \mathcal{N}(guess, 1)\] \[measurement \mid guess, weight \sim \mathcal{N}(weight, 0.75)\]

def scale(guess):
    weight = pyro.sample("weight", dist.Normal(guess, 1.0))
    measurement = pyro.sample("measurement", dist.Normal(weight, 0.75))
    return measurement
scale(5.)

Suppose we observe that the measurement of an object was \(14\) lbs. What would have we guessed if we tried to guess it’s weight first?

This question is answered in two steps.

Condition the model.

conditioned_scale = pyro.condition(scale, data={"measurement": torch.tensor(14.)})

Set the prior and infer the posterior. We will use

from pyro.infer.mcmc import MCMC
from pyro.infer.mcmc.nuts import HMC
from pyro.infer import EmpiricalMarginal
import matplotlib.pyplot as plt
# %matplotlib inline
guess_prior = 10.
hmc_kernel = HMC(conditioned_scale, step_size=0.9, num_steps=4)
posterior = MCMC(hmc_kernel, 
                 num_samples=1000, 
                 warmup_steps=50).run(guess_prior)

## 
Warmup:   0%|          | 0/1050 [00:00, ?it/s]
Warmup:   0%|          | 0/1050 [00:00, ?it/s, step size=6.42e+00, acc. rate=1.000]
Warmup:   0%|          | 1/1050 [00:00, 11.38it/s, step size=1.08e+00, acc. rate=0.500]
Warmup:   0%|          | 2/1050 [00:00, 21.95it/s, step size=1.23e-01, acc. rate=0.333]
Warmup:   0%|          | 3/1050 [00:00, 26.89it/s, step size=1.68e-01, acc. rate=0.500]
Warmup:   0%|          | 4/1050 [00:00, 35.84it/s, step size=1.68e-01, acc. rate=0.500]
Warmup:   0%|          | 4/1050 [00:00, 35.84it/s, step size=2.65e-01, acc. rate=0.600]
Warmup:   0%|          | 5/1050 [00:00, 35.84it/s, step size=4.46e-01, acc. rate=0.667]
Warmup:   1%|          | 6/1050 [00:00, 35.84it/s, step size=8.00e-01, acc. rate=0.714]
Warmup:   1%|          | 7/1050 [00:00, 35.84it/s, step size=6.91e-01, acc. rate=0.750]
Warmup:   1%|          | 8/1050 [00:00, 35.84it/s, step size=1.29e+00, acc. rate=0.778]
Warmup:   1%|          | 9/1050 [00:00, 35.84it/s, step size=1.02e-01, acc. rate=0.700]
Warmup:   1%|          | 10/1050 [00:00, 35.84it/s, step size=1.93e-01, acc. rate=0.727]
Warmup:   1%|1         | 11/1050 [00:00, 35.84it/s, step size=3.66e-01, acc. rate=0.750]
Warmup:   1%|1         | 12/1050 [00:00, 35.84it/s, step size=6.89e-01, acc. rate=0.769]
Warmup:   1%|1         | 13/1050 [00:00, 35.84it/s, step size=1.22e+00, acc. rate=0.786]
Warmup:   1%|1         | 14/1050 [00:00, 35.84it/s, step size=1.03e-01, acc. rate=0.733]
Warmup:   1%|1         | 15/1050 [00:00, 35.84it/s, step size=1.96e-01, acc. rate=0.750]
Warmup:   2%|1         | 16/1050 [00:00, 44.56it/s, step size=1.96e-01, acc. rate=0.750]
Warmup:   2%|1         | 16/1050 [00:00, 44.56it/s, step size=3.57e-01, acc. rate=0.765]
Warmup:   2%|1         | 17/1050 [00:00, 44.56it/s, step size=6.67e-01, acc. rate=0.778]
Warmup:   2%|1         | 18/1050 [00:00, 44.56it/s, step size=1.24e+00, acc. rate=0.789]
Warmup:   2%|1         | 19/1050 [00:00, 44.56it/s, step size=1.15e-01, acc. rate=0.750]
Warmup:   2%|1         | 20/1050 [00:00, 44.56it/s, step size=2.13e-01, acc. rate=0.762]
Warmup:   2%|2         | 21/1050 [00:00, 44.56it/s, step size=3.92e-01, acc. rate=0.773]
Warmup:   2%|2         | 22/1050 [00:00, 44.56it/s, step size=6.74e-01, acc. rate=0.783]
Warmup:   2%|2         | 23/1050 [00:00, 44.56it/s, step size=8.90e-01, acc. rate=0.792]
Warmup:   2%|2         | 24/1050 [00:00, 44.56it/s, step size=1.60e+00, acc. rate=0.800]
Warmup:   2%|2         | 25/1050 [00:00, 44.56it/s, step size=1.67e-01, acc. rate=0.769]
Warmup:   2%|2         | 26/1050 [00:00, 44.56it/s, step size=3.02e-01, acc. rate=0.778]
Warmup:   3%|2         | 27/1050 [00:00, 44.56it/s, step size=5.38e-01, acc. rate=0.786]
Warmup:   3%|2         | 28/1050 [00:00, 54.82it/s, step size=5.38e-01, acc. rate=0.786]
Warmup:   3%|2         | 28/1050 [00:00, 54.82it/s, step size=9.52e-01, acc. rate=0.793]
Warmup:   3%|2         | 29/1050 [00:00, 54.82it/s, step size=1.67e+00, acc. rate=0.800]
Warmup:   3%|2         | 30/1050 [00:00, 54.82it/s, step size=1.91e-01, acc. rate=0.774]
Warmup:   3%|2         | 31/1050 [00:00, 54.82it/s, step size=3.34e-01, acc. rate=0.781]
Warmup:   3%|3         | 32/1050 [00:00, 54.82it/s, step size=4.76e-01, acc. rate=0.788]
Warmup:   3%|3         | 33/1050 [00:00, 54.82it/s, step size=7.42e-01, acc. rate=0.794]
Warmup:   3%|3         | 34/1050 [00:00, 54.82it/s, step size=1.27e+00, acc. rate=0.800]
Warmup:   3%|3         | 35/1050 [00:00, 54.82it/s, step size=1.59e-01, acc. rate=0.778]
Warmup:   3%|3         | 36/1050 [00:00, 54.82it/s, step size=2.64e-01, acc. rate=0.784]
Warmup:   4%|3         | 37/1050 [00:00, 54.82it/s, step size=4.51e-01, acc. rate=0.789]
Warmup:   4%|3         | 38/1050 [00:00, 54.82it/s, step size=7.64e-01, acc. rate=0.795]
Warmup:   4%|3         | 39/1050 [00:00, 54.82it/s, step size=9.19e-01, acc. rate=0.800]
Warmup:   4%|3         | 40/1050 [00:00, 54.82it/s, step size=1.69e-01, acc. rate=0.780]
Warmup:   4%|3         | 41/1050 [00:00, 54.82it/s, step size=2.85e-01, acc. rate=0.786]
Warmup:   4%|4         | 42/1050 [00:00, 54.82it/s, step size=4.49e-01, acc. rate=0.791]
Warmup:   4%|4         | 43/1050 [00:00, 66.98it/s, step size=4.49e-01, acc. rate=0.791]
Warmup:   4%|4         | 43/1050 [00:00, 66.98it/s, step size=6.73e-01, acc. rate=0.795]
Warmup:   4%|4         | 44/1050 [00:00, 66.98it/s, step size=1.59e+00, acc. rate=0.778]
Warmup:   4%|4         | 45/1050 [00:00, 66.98it/s, step size=2.29e+01, acc. rate=0.783]
Warmup:   4%|4         | 46/1050 [00:00, 66.98it/s, step size=3.87e+00, acc. rate=0.766]
Warmup:   4%|4         | 47/1050 [00:00, 66.98it/s, step size=3.82e-01, acc. rate=0.750]
Warmup:   5%|4         | 48/1050 [00:00, 66.98it/s, step size=5.17e-01, acc. rate=0.755]
Warmup:   5%|4         | 49/1050 [00:00, 66.98it/s, step size=1.11e+00, acc. rate=0.760]
Sample:   5%|4         | 50/1050 [00:00, 66.98it/s, step size=1.11e+00, acc. rate=0.760]
Sample:   5%|4         | 50/1050 [00:00, 66.98it/s, step size=1.11e+00, acc. rate=0.765]
Sample:   5%|4         | 51/1050 [00:00, 66.98it/s, step size=1.11e+00, acc. rate=0.769]
Sample:   5%|4         | 52/1050 [00:00, 66.98it/s, step size=1.11e+00, acc. rate=0.774]
Sample:   5%|5         | 53/1050 [00:00, 66.98it/s, step size=1.11e+00, acc. rate=0.778]
Sample:   5%|5         | 54/1050 [00:00, 66.98it/s, step size=1.11e+00, acc. rate=0.782]
Sample:   5%|5         | 55/1050 [00:00, 66.98it/s, step size=1.11e+00, acc. rate=0.786]
Sample:   5%|5         | 56/1050 [00:00, 66.98it/s, step size=1.11e+00, acc. rate=0.789]
Sample:   5%|5         | 57/1050 [00:00, 66.98it/s, step size=1.11e+00, acc. rate=0.793]
Sample:   6%|5         | 58/1050 [00:00, 66.98it/s, step size=1.11e+00, acc. rate=0.797]
Sample:   6%|5         | 59/1050 [00:00, 66.98it/s, step size=1.11e+00, acc. rate=0.800]
Sample:   6%|5         | 60/1050 [00:00, 66.98it/s, step size=1.11e+00, acc. rate=0.803]
Sample:   6%|5         | 61/1050 [00:00, 66.98it/s, step size=1.11e+00, acc. rate=0.806]
Sample:   6%|5         | 62/1050 [00:00, 66.98it/s, step size=1.11e+00, acc. rate=0.810]
Sample:   6%|6         | 63/1050 [00:00, 66.98it/s, step size=1.11e+00, acc. rate=0.812]
Sample:   6%|6         | 64/1050 [00:00, 66.98it/s, step size=1.11e+00, acc. rate=0.815]
Sample:   6%|6         | 65/1050 [00:00, 66.98it/s, step size=1.11e+00, acc. rate=0.818]
Sample:   6%|6         | 66/1050 [00:00, 66.98it/s, step size=1.11e+00, acc. rate=0.821]
Sample:   6%|6         | 67/1050 [00:00, 66.98it/s, step size=1.11e+00, acc. rate=0.824]
Sample:   6%|6         | 68/1050 [00:00, 66.98it/s, step size=1.11e+00, acc. rate=0.826]
Sample:   7%|6         | 69/1050 [00:00, 66.98it/s, step size=1.11e+00, acc. rate=0.829]
Sample:   7%|6         | 70/1050 [00:00, 66.98it/s, step size=1.11e+00, acc. rate=0.831]
Sample:   7%|6         | 71/1050 [00:00, 66.98it/s, step size=1.11e+00, acc. rate=0.833]
Sample:   7%|6         | 72/1050 [00:00, 66.98it/s, step size=1.11e+00, acc. rate=0.836]
Sample:   7%|6         | 73/1050 [00:00, 66.98it/s, step size=1.11e+00, acc. rate=0.838]
Sample:   7%|7         | 74/1050 [00:00, 66.98it/s, step size=1.11e+00, acc. rate=0.840]
Sample:   7%|7         | 75/1050 [00:00, 66.98it/s, step size=1.11e+00, acc. rate=0.842]
Sample:   7%|7         | 76/1050 [00:00, 66.98it/s, step size=1.11e+00, acc. rate=0.844]
Sample:   7%|7         | 77/1050 [00:00, 88.08it/s, step size=1.11e+00, acc. rate=0.844]
Sample:   7%|7         | 77/1050 [00:00, 88.08it/s, step size=1.11e+00, acc. rate=0.846]
Sample:   7%|7         | 78/1050 [00:00, 88.08it/s, step size=1.11e+00, acc. rate=0.848]
Sample:   8%|7         | 79/1050 [00:00, 88.08it/s, step size=1.11e+00, acc. rate=0.850]
Sample:   8%|7         | 80/1050 [00:00, 88.08it/s, step size=1.11e+00, acc. rate=0.852]
Sample:   8%|7         | 81/1050 [00:00, 88.08it/s, step size=1.11e+00, acc. rate=0.854]
Sample:   8%|7         | 82/1050 [00:00, 88.08it/s, step size=1.11e+00, acc. rate=0.855]
Sample:   8%|7         | 83/1050 [00:00, 88.08it/s, step size=1.11e+00, acc. rate=0.857]
Sample:   8%|8         | 84/1050 [00:00, 88.08it/s, step size=1.11e+00, acc. rate=0.859]
Sample:   8%|8         | 85/1050 [00:00, 88.08it/s, step size=1.11e+00, acc. rate=0.860]
Sample:   8%|8         | 86/1050 [00:00, 88.08it/s, step size=1.11e+00, acc. rate=0.862]
Sample:   8%|8         | 87/1050 [00:00, 88.08it/s, step size=1.11e+00, acc. rate=0.864]
Sample:   8%|8         | 88/1050 [00:00, 88.08it/s, step size=1.11e+00, acc. rate=0.865]
Sample:   8%|8         | 89/1050 [00:00, 88.08it/s, step size=1.11e+00, acc. rate=0.867]
Sample:   9%|8         | 90/1050 [00:00, 88.08it/s, step size=1.11e+00, acc. rate=0.868]
Sample:   9%|8         | 91/1050 [00:00, 88.08it/s, step size=1.11e+00, acc. rate=0.870]
Sample:   9%|8         | 92/1050 [00:00, 88.08it/s, step size=1.11e+00, acc. rate=0.860]
Sample:   9%|8         | 93/1050 [00:00, 88.08it/s, step size=1.11e+00, acc. rate=0.862]
Sample:   9%|8         | 94/1050 [00:00, 88.08it/s, step size=1.11e+00, acc. rate=0.863]
Sample:   9%|9         | 95/1050 [00:00, 88.08it/s, step size=1.11e+00, acc. rate=0.865]
Sample:   9%|9         | 96/1050 [00:00, 88.08it/s, step size=1.11e+00, acc. rate=0.866]
Sample:   9%|9         | 97/1050 [00:00, 88.08it/s, step size=1.11e+00, acc. rate=0.867]
Sample:   9%|9         | 98/1050 [00:00, 88.08it/s, step size=1.11e+00, acc. rate=0.869]
Sample:   9%|9         | 99/1050 [00:00, 88.08it/s, step size=1.11e+00, acc. rate=0.870]
Sample:  10%|9         | 100/1050 [00:00, 88.08it/s, step size=1.11e+00, acc. rate=0.871]
Sample:  10%|9         | 101/1050 [00:00, 88.08it/s, step size=1.11e+00, acc. rate=0.873]
Sample:  10%|9         | 102/1050 [00:00, 88.08it/s, step size=1.11e+00, acc. rate=0.874]
Sample:  10%|9         | 103/1050 [00:00, 88.08it/s, step size=1.11e+00, acc. rate=0.875]
Sample:  10%|9         | 104/1050 [00:00, 88.08it/s, step size=1.11e+00, acc. rate=0.876]
Sample:  10%|#         | 105/1050 [00:00, 88.08it/s, step size=1.11e+00, acc. rate=0.877]
Sample:  10%|#         | 106/1050 [00:00, 88.08it/s, step size=1.11e+00, acc. rate=0.879]
Sample:  10%|#         | 107/1050 [00:00, 88.08it/s, step size=1.11e+00, acc. rate=0.880]
Sample:  10%|#         | 108/1050 [00:00, 88.08it/s, step size=1.11e+00, acc. rate=0.872]
Sample:  10%|#         | 109/1050 [00:00, 88.08it/s, step size=1.11e+00, acc. rate=0.873]
Sample:  10%|#         | 110/1050 [00:00, 88.08it/s, step size=1.11e+00, acc. rate=0.874]
Sample:  11%|#         | 111/1050 [00:00, 88.08it/s, step size=1.11e+00, acc. rate=0.875]
Sample:  11%|#         | 112/1050 [00:00, 113.46it/s, step size=1.11e+00, acc. rate=0.875]
Sample:  11%|#         | 112/1050 [00:00, 113.46it/s, step size=1.11e+00, acc. rate=0.876]
Sample:  11%|#         | 113/1050 [00:00, 113.46it/s, step size=1.11e+00, acc. rate=0.877]
Sample:  11%|#         | 114/1050 [00:00, 113.46it/s, step size=1.11e+00, acc. rate=0.878]
Sample:  11%|#         | 115/1050 [00:00, 113.46it/s, step size=1.11e+00, acc. rate=0.879]
Sample:  11%|#1        | 116/1050 [00:00, 113.46it/s, step size=1.11e+00, acc. rate=0.880]
Sample:  11%|#1        | 117/1050 [00:00, 113.46it/s, step size=1.11e+00, acc. rate=0.881]
Sample:  11%|#1        | 118/1050 [00:00, 113.46it/s, step size=1.11e+00, acc. rate=0.882]
Sample:  11%|#1        | 119/1050 [00:00, 113.46it/s, step size=1.11e+00, acc. rate=0.883]
Sample:  11%|#1        | 120/1050 [00:00, 113.46it/s, step size=1.11e+00, acc. rate=0.884]
Sample:  12%|#1        | 121/1050 [00:00, 113.46it/s, step size=1.11e+00, acc. rate=0.885]
Sample:  12%|#1        | 122/1050 [00:00, 113.46it/s, step size=1.11e+00, acc. rate=0.886]
Sample:  12%|#1        | 123/1050 [00:00, 113.46it/s, step size=1.11e+00, acc. rate=0.887]
Sample:  12%|#1        | 124/1050 [00:00, 113.46it/s, step size=1.11e+00, acc. rate=0.888]
Sample:  12%|#1        | 125/1050 [00:00, 113.46it/s, step size=1.11e+00, acc. rate=0.889]
Sample:  12%|#2        | 126/1050 [00:00, 113.46it/s, step size=1.11e+00, acc. rate=0.890]
Sample:  12%|#2        | 127/1050 [00:00, 113.46it/s, step size=1.11e+00, acc. rate=0.883]
Sample:  12%|#2        | 128/1050 [00:00, 113.46it/s, step size=1.11e+00, acc. rate=0.884]
Sample:  12%|#2        | 129/1050 [00:00, 113.46it/s, step size=1.11e+00, acc. rate=0.885]
Sample:  12%|#2        | 130/1050 [00:00, 113.46it/s, step size=1.11e+00, acc. rate=0.885]
Sample:  12%|#2        | 131/1050 [00:00, 113.46it/s, step size=1.11e+00, acc. rate=0.886]
Sample:  13%|#2        | 132/1050 [00:00, 113.46it/s, step size=1.11e+00, acc. rate=0.887]
Sample:  13%|#2        | 133/1050 [00:00, 113.46it/s, step size=1.11e+00, acc. rate=0.888]
Sample:  13%|#2        | 134/1050 [00:00, 113.46it/s, step size=1.11e+00, acc. rate=0.889]
Sample:  13%|#2        | 135/1050 [00:00, 113.46it/s, step size=1.11e+00, acc. rate=0.890]
Sample:  13%|#2        | 136/1050 [00:00, 113.46it/s, step size=1.11e+00, acc. rate=0.891]
Sample:  13%|#3        | 137/1050 [00:00, 113.46it/s, step size=1.11e+00, acc. rate=0.891]
Sample:  13%|#3        | 138/1050 [00:00, 113.46it/s, step size=1.11e+00, acc. rate=0.892]
Sample:  13%|#3        | 139/1050 [00:00, 113.46it/s, step size=1.11e+00, acc. rate=0.893]
Sample:  13%|#3        | 140/1050 [00:00, 113.46it/s, step size=1.11e+00, acc. rate=0.894]
Sample:  13%|#3        | 141/1050 [00:00, 113.46it/s, step size=1.11e+00, acc. rate=0.894]
Sample:  14%|#3        | 142/1050 [00:00, 113.46it/s, step size=1.11e+00, acc. rate=0.895]
Sample:  14%|#3        | 143/1050 [00:00, 113.46it/s, step size=1.11e+00, acc. rate=0.896]
Sample:  14%|#3        | 144/1050 [00:00, 113.46it/s, step size=1.11e+00, acc. rate=0.897]
Sample:  14%|#3        | 145/1050 [00:00, 113.46it/s, step size=1.11e+00, acc. rate=0.897]
Sample:  14%|#3        | 146/1050 [00:00, 113.46it/s, step size=1.11e+00, acc. rate=0.898]
Sample:  14%|#4        | 147/1050 [00:00, 141.91it/s, step size=1.11e+00, acc. rate=0.898]
Sample:  14%|#4        | 147/1050 [00:00, 141.91it/s, step size=1.11e+00, acc. rate=0.899]
Sample:  14%|#4        | 148/1050 [00:00, 141.91it/s, step size=1.11e+00, acc. rate=0.899]
Sample:  14%|#4        | 149/1050 [00:00, 141.91it/s, step size=1.11e+00, acc. rate=0.900]
Sample:  14%|#4        | 150/1050 [00:00, 141.91it/s, step size=1.11e+00, acc. rate=0.901]
Sample:  14%|#4        | 151/1050 [00:00, 141.91it/s, step size=1.11e+00, acc. rate=0.901]
Sample:  14%|#4        | 152/1050 [00:00, 141.91it/s, step size=1.11e+00, acc. rate=0.902]
Sample:  15%|#4        | 153/1050 [00:00, 141.91it/s, step size=1.11e+00, acc. rate=0.903]
Sample:  15%|#4        | 154/1050 [00:00, 141.91it/s, step size=1.11e+00, acc. rate=0.903]
Sample:  15%|#4        | 155/1050 [00:00, 141.91it/s, step size=1.11e+00, acc. rate=0.904]
Sample:  15%|#4        | 156/1050 [00:00, 141.91it/s, step size=1.11e+00, acc. rate=0.904]
Sample:  15%|#4        | 157/1050 [00:00, 141.91it/s, step size=1.11e+00, acc. rate=0.905]
Sample:  15%|#5        | 158/1050 [00:00, 141.91it/s, step size=1.11e+00, acc. rate=0.906]
Sample:  15%|#5        | 159/1050 [00:00, 141.91it/s, step size=1.11e+00, acc. rate=0.906]
Sample:  15%|#5        | 160/1050 [00:00, 141.91it/s, step size=1.11e+00, acc. rate=0.907]
Sample:  15%|#5        | 161/1050 [00:00, 141.91it/s, step size=1.11e+00, acc. rate=0.907]
Sample:  15%|#5        | 162/1050 [00:00, 141.91it/s, step size=1.11e+00, acc. rate=0.908]
Sample:  16%|#5        | 163/1050 [00:00, 141.91it/s, step size=1.11e+00, acc. rate=0.909]
Sample:  16%|#5        | 164/1050 [00:00, 141.91it/s, step size=1.11e+00, acc. rate=0.909]
Sample:  16%|#5        | 165/1050 [00:00, 141.91it/s, step size=1.11e+00, acc. rate=0.910]
Sample:  16%|#5        | 166/1050 [00:00, 141.91it/s, step size=1.11e+00, acc. rate=0.910]
Sample:  16%|#5        | 167/1050 [00:00, 141.91it/s, step size=1.11e+00, acc. rate=0.911]
Sample:  16%|#6        | 168/1050 [00:00, 141.91it/s, step size=1.11e+00, acc. rate=0.911]
Sample:  16%|#6        | 169/1050 [00:00, 141.91it/s, step size=1.11e+00, acc. rate=0.912]
Sample:  16%|#6        | 170/1050 [00:00, 141.91it/s, step size=1.11e+00, acc. rate=0.912]
Sample:  16%|#6        | 171/1050 [00:00, 141.91it/s, step size=1.11e+00, acc. rate=0.913]
Sample:  16%|#6        | 172/1050 [00:00, 141.91it/s, step size=1.11e+00, acc. rate=0.913]
Sample:  16%|#6        | 173/1050 [00:00, 141.91it/s, step size=1.11e+00, acc. rate=0.914]
Sample:  17%|#6        | 174/1050 [00:00, 141.91it/s, step size=1.11e+00, acc. rate=0.914]
Sample:  17%|#6        | 175/1050 [00:00, 141.91it/s, step size=1.11e+00, acc. rate=0.915]
Sample:  17%|#6        | 176/1050 [00:00, 141.91it/s, step size=1.11e+00, acc. rate=0.915]
Sample:  17%|#6        | 177/1050 [00:00, 141.91it/s, step size=1.11e+00, acc. rate=0.916]
Sample:  17%|#6        | 178/1050 [00:00, 141.91it/s, step size=1.11e+00, acc. rate=0.916]
Sample:  17%|#7        | 179/1050 [00:00, 141.91it/s, step size=1.11e+00, acc. rate=0.917]
Sample:  17%|#7        | 180/1050 [00:00, 141.91it/s, step size=1.11e+00, acc. rate=0.917]
Sample:  17%|#7        | 181/1050 [00:00, 141.91it/s, step size=1.11e+00, acc. rate=0.918]
Sample:  17%|#7        | 182/1050 [00:00, 141.91it/s, step size=1.11e+00, acc. rate=0.918]
Sample:  17%|#7        | 183/1050 [00:00, 172.91it/s, step size=1.11e+00, acc. rate=0.918]
Sample:  17%|#7        | 183/1050 [00:00, 172.91it/s, step size=1.11e+00, acc. rate=0.918]
Sample:  18%|#7        | 184/1050 [00:00, 172.91it/s, step size=1.11e+00, acc. rate=0.919]
Sample:  18%|#7        | 185/1050 [00:00, 172.91it/s, step size=1.11e+00, acc. rate=0.919]
Sample:  18%|#7        | 186/1050 [00:00, 172.91it/s, step size=1.11e+00, acc. rate=0.920]
Sample:  18%|#7        | 187/1050 [00:00, 172.91it/s, step size=1.11e+00, acc. rate=0.920]
Sample:  18%|#7        | 188/1050 [00:00, 172.91it/s, step size=1.11e+00, acc. rate=0.921]
Sample:  18%|#8        | 189/1050 [00:00, 172.91it/s, step size=1.11e+00, acc. rate=0.921]
Sample:  18%|#8        | 190/1050 [00:00, 172.91it/s, step size=1.11e+00, acc. rate=0.921]
Sample:  18%|#8        | 191/1050 [00:00, 172.91it/s, step size=1.11e+00, acc. rate=0.922]
Sample:  18%|#8        | 192/1050 [00:00, 172.91it/s, step size=1.11e+00, acc. rate=0.922]
Sample:  18%|#8        | 193/1050 [00:00, 172.91it/s, step size=1.11e+00, acc. rate=0.923]
Sample:  18%|#8        | 194/1050 [00:00, 172.91it/s, step size=1.11e+00, acc. rate=0.923]
Sample:  19%|#8        | 195/1050 [00:00, 172.91it/s, step size=1.11e+00, acc. rate=0.923]
Sample:  19%|#8        | 196/1050 [00:00, 172.91it/s, step size=1.11e+00, acc. rate=0.924]
Sample:  19%|#8        | 197/1050 [00:00, 172.91it/s, step size=1.11e+00, acc. rate=0.924]
Sample:  19%|#8        | 198/1050 [00:00, 172.91it/s, step size=1.11e+00, acc. rate=0.925]
Sample:  19%|#8        | 199/1050 [00:00, 172.91it/s, step size=1.11e+00, acc. rate=0.925]
Sample:  19%|#9        | 200/1050 [00:00, 172.91it/s, step size=1.11e+00, acc. rate=0.925]
Sample:  19%|#9        | 201/1050 [00:00, 172.91it/s, step size=1.11e+00, acc. rate=0.926]
Sample:  19%|#9        | 202/1050 [00:00, 172.91it/s, step size=1.11e+00, acc. rate=0.926]
Sample:  19%|#9        | 203/1050 [00:00, 172.91it/s, step size=1.11e+00, acc. rate=0.926]
Sample:  19%|#9        | 204/1050 [00:00, 172.91it/s, step size=1.11e+00, acc. rate=0.927]
Sample:  20%|#9        | 205/1050 [00:00, 172.91it/s, step size=1.11e+00, acc. rate=0.927]
Sample:  20%|#9        | 206/1050 [00:00, 172.91it/s, step size=1.11e+00, acc. rate=0.928]
Sample:  20%|#9        | 207/1050 [00:00, 172.91it/s, step size=1.11e+00, acc. rate=0.928]
Sample:  20%|#9        | 208/1050 [00:00, 172.91it/s, step size=1.11e+00, acc. rate=0.928]
Sample:  20%|#9        | 209/1050 [00:00, 172.91it/s, step size=1.11e+00, acc. rate=0.929]
Sample:  20%|##        | 210/1050 [00:00, 172.91it/s, step size=1.11e+00, acc. rate=0.929]
Sample:  20%|##        | 211/1050 [00:00, 172.91it/s, step size=1.11e+00, acc. rate=0.929]
Sample:  20%|##        | 212/1050 [00:00, 172.91it/s, step size=1.11e+00, acc. rate=0.930]
Sample:  20%|##        | 213/1050 [00:00, 172.91it/s, step size=1.11e+00, acc. rate=0.930]
Sample:  20%|##        | 214/1050 [00:00, 172.91it/s, step size=1.11e+00, acc. rate=0.930]
Sample:  20%|##        | 215/1050 [00:00, 172.91it/s, step size=1.11e+00, acc. rate=0.931]
Sample:  21%|##        | 216/1050 [00:00, 172.91it/s, step size=1.11e+00, acc. rate=0.931]
Sample:  21%|##        | 217/1050 [00:00, 172.91it/s, step size=1.11e+00, acc. rate=0.931]
Sample:  21%|##        | 218/1050 [00:00, 172.91it/s, step size=1.11e+00, acc. rate=0.932]
Sample:  21%|##        | 219/1050 [00:00, 172.91it/s, step size=1.11e+00, acc. rate=0.932]
Sample:  21%|##        | 220/1050 [00:00, 205.65it/s, step size=1.11e+00, acc. rate=0.932]
Sample:  21%|##        | 220/1050 [00:00, 205.65it/s, step size=1.11e+00, acc. rate=0.932]
Sample:  21%|##1       | 221/1050 [00:00, 205.65it/s, step size=1.11e+00, acc. rate=0.932]
Sample:  21%|##1       | 222/1050 [00:00, 205.65it/s, step size=1.11e+00, acc. rate=0.933]
Sample:  21%|##1       | 223/1050 [00:00, 205.65it/s, step size=1.11e+00, acc. rate=0.933]
Sample:  21%|##1       | 224/1050 [00:00, 205.65it/s, step size=1.11e+00, acc. rate=0.933]
Sample:  21%|##1       | 225/1050 [00:00, 205.65it/s, step size=1.11e+00, acc. rate=0.934]
Sample:  22%|##1       | 226/1050 [00:00, 205.65it/s, step size=1.11e+00, acc. rate=0.934]
Sample:  22%|##1       | 227/1050 [00:00, 205.65it/s, step size=1.11e+00, acc. rate=0.934]
Sample:  22%|##1       | 228/1050 [00:00, 205.65it/s, step size=1.11e+00, acc. rate=0.934]
Sample:  22%|##1       | 229/1050 [00:00, 205.65it/s, step size=1.11e+00, acc. rate=0.935]
Sample:  22%|##1       | 230/1050 [00:00, 205.65it/s, step size=1.11e+00, acc. rate=0.935]
Sample:  22%|##2       | 231/1050 [00:00, 205.65it/s, step size=1.11e+00, acc. rate=0.935]
Sample:  22%|##2       | 232/1050 [00:00, 205.65it/s, step size=1.11e+00, acc. rate=0.936]
Sample:  22%|##2       | 233/1050 [00:00, 205.65it/s, step size=1.11e+00, acc. rate=0.936]
Sample:  22%|##2       | 234/1050 [00:00, 205.65it/s, step size=1.11e+00, acc. rate=0.932]
Sample:  22%|##2       | 235/1050 [00:00, 205.65it/s, step size=1.11e+00, acc. rate=0.932]
Sample:  22%|##2       | 236/1050 [00:00, 205.65it/s, step size=1.11e+00, acc. rate=0.932]
Sample:  23%|##2       | 237/1050 [00:00, 205.65it/s, step size=1.11e+00, acc. rate=0.933]
Sample:  23%|##2       | 238/1050 [00:00, 205.65it/s, step size=1.11e+00, acc. rate=0.933]
Sample:  23%|##2       | 239/1050 [00:01, 205.65it/s, step size=1.11e+00, acc. rate=0.929]
Sample:  23%|##2       | 240/1050 [00:01, 205.65it/s, step size=1.11e+00, acc. rate=0.929]
Sample:  23%|##2       | 241/1050 [00:01, 205.65it/s, step size=1.11e+00, acc. rate=0.930]
Sample:  23%|##3       | 242/1050 [00:01, 205.65it/s, step size=1.11e+00, acc. rate=0.930]
Sample:  23%|##3       | 243/1050 [00:01, 205.65it/s, step size=1.11e+00, acc. rate=0.930]
Sample:  23%|##3       | 244/1050 [00:01, 205.65it/s, step size=1.11e+00, acc. rate=0.931]
Sample:  23%|##3       | 245/1050 [00:01, 205.65it/s, step size=1.11e+00, acc. rate=0.931]
Sample:  23%|##3       | 246/1050 [00:01, 205.65it/s, step size=1.11e+00, acc. rate=0.931]
Sample:  24%|##3       | 247/1050 [00:01, 205.65it/s, step size=1.11e+00, acc. rate=0.931]
Sample:  24%|##3       | 248/1050 [00:01, 205.65it/s, step size=1.11e+00, acc. rate=0.932]
Sample:  24%|##3       | 249/1050 [00:01, 205.65it/s, step size=1.11e+00, acc. rate=0.932]
Sample:  24%|##3       | 250/1050 [00:01, 205.65it/s, step size=1.11e+00, acc. rate=0.932]
Sample:  24%|##3       | 251/1050 [00:01, 205.65it/s, step size=1.11e+00, acc. rate=0.933]
Sample:  24%|##4       | 252/1050 [00:01, 205.65it/s, step size=1.11e+00, acc. rate=0.933]
Sample:  24%|##4       | 253/1050 [00:01, 205.65it/s, step size=1.11e+00, acc. rate=0.933]
Sample:  24%|##4       | 254/1050 [00:01, 205.65it/s, step size=1.11e+00, acc. rate=0.933]
Sample:  24%|##4       | 255/1050 [00:01, 205.65it/s, step size=1.11e+00, acc. rate=0.934]
Sample:  24%|##4       | 256/1050 [00:01, 205.65it/s, step size=1.11e+00, acc. rate=0.934]
Sample:  24%|##4       | 257/1050 [00:01, 236.96it/s, step size=1.11e+00, acc. rate=0.934]
Sample:  24%|##4       | 257/1050 [00:01, 236.96it/s, step size=1.11e+00, acc. rate=0.934]
Sample:  25%|##4       | 258/1050 [00:01, 236.96it/s, step size=1.11e+00, acc. rate=0.934]
Sample:  25%|##4       | 259/1050 [00:01, 236.96it/s, step size=1.11e+00, acc. rate=0.935]
Sample:  25%|##4       | 260/1050 [00:01, 236.96it/s, step size=1.11e+00, acc. rate=0.935]
Sample:  25%|##4       | 261/1050 [00:01, 236.96it/s, step size=1.11e+00, acc. rate=0.935]
Sample:  25%|##4       | 262/1050 [00:01, 236.96it/s, step size=1.11e+00, acc. rate=0.935]
Sample:  25%|##5       | 263/1050 [00:01, 236.96it/s, step size=1.11e+00, acc. rate=0.936]
Sample:  25%|##5       | 264/1050 [00:01, 236.96it/s, step size=1.11e+00, acc. rate=0.936]
Sample:  25%|##5       | 265/1050 [00:01, 236.96it/s, step size=1.11e+00, acc. rate=0.936]
Sample:  25%|##5       | 266/1050 [00:01, 236.96it/s, step size=1.11e+00, acc. rate=0.936]
Sample:  25%|##5       | 267/1050 [00:01, 236.96it/s, step size=1.11e+00, acc. rate=0.937]
Sample:  26%|##5       | 268/1050 [00:01, 236.96it/s, step size=1.11e+00, acc. rate=0.933]
Sample:  26%|##5       | 269/1050 [00:01, 236.96it/s, step size=1.11e+00, acc. rate=0.933]
Sample:  26%|##5       | 270/1050 [00:01, 236.96it/s, step size=1.11e+00, acc. rate=0.934]
Sample:  26%|##5       | 271/1050 [00:01, 236.96it/s, step size=1.11e+00, acc. rate=0.934]
Sample:  26%|##5       | 272/1050 [00:01, 236.96it/s, step size=1.11e+00, acc. rate=0.934]
Sample:  26%|##6       | 273/1050 [00:01, 236.96it/s, step size=1.11e+00, acc. rate=0.934]
Sample:  26%|##6       | 274/1050 [00:01, 236.96it/s, step size=1.11e+00, acc. rate=0.935]
Sample:  26%|##6       | 275/1050 [00:01, 236.96it/s, step size=1.11e+00, acc. rate=0.935]
Sample:  26%|##6       | 276/1050 [00:01, 236.96it/s, step size=1.11e+00, acc. rate=0.935]
Sample:  26%|##6       | 277/1050 [00:01, 236.96it/s, step size=1.11e+00, acc. rate=0.935]
Sample:  26%|##6       | 278/1050 [00:01, 236.96it/s, step size=1.11e+00, acc. rate=0.935]
Sample:  27%|##6       | 279/1050 [00:01, 236.96it/s, step size=1.11e+00, acc. rate=0.936]
Sample:  27%|##6       | 280/1050 [00:01, 236.96it/s, step size=1.11e+00, acc. rate=0.936]
Sample:  27%|##6       | 281/1050 [00:01, 236.96it/s, step size=1.11e+00, acc. rate=0.936]
Sample:  27%|##6       | 282/1050 [00:01, 236.96it/s, step size=1.11e+00, acc. rate=0.936]
Sample:  27%|##6       | 283/1050 [00:01, 236.96it/s, step size=1.11e+00, acc. rate=0.937]
Sample:  27%|##7       | 284/1050 [00:01, 236.96it/s, step size=1.11e+00, acc. rate=0.937]
Sample:  27%|##7       | 285/1050 [00:01, 236.96it/s, step size=1.11e+00, acc. rate=0.937]
Sample:  27%|##7       | 286/1050 [00:01, 236.96it/s, step size=1.11e+00, acc. rate=0.937]
Sample:  27%|##7       | 287/1050 [00:01, 236.96it/s, step size=1.11e+00, acc. rate=0.938]
Sample:  27%|##7       | 288/1050 [00:01, 236.96it/s, step size=1.11e+00, acc. rate=0.938]
Sample:  28%|##7       | 289/1050 [00:01, 236.96it/s, step size=1.11e+00, acc. rate=0.938]
Sample:  28%|##7       | 290/1050 [00:01, 236.96it/s, step size=1.11e+00, acc. rate=0.938]
Sample:  28%|##7       | 291/1050 [00:01, 236.96it/s, step size=1.11e+00, acc. rate=0.938]
Sample:  28%|##7       | 292/1050 [00:01, 236.96it/s, step size=1.11e+00, acc. rate=0.939]
Sample:  28%|##7       | 293/1050 [00:01, 263.59it/s, step size=1.11e+00, acc. rate=0.939]
Sample:  28%|##7       | 293/1050 [00:01, 263.59it/s, step size=1.11e+00, acc. rate=0.939]
Sample:  28%|##8       | 294/1050 [00:01, 263.59it/s, step size=1.11e+00, acc. rate=0.939]
Sample:  28%|##8       | 295/1050 [00:01, 263.59it/s, step size=1.11e+00, acc. rate=0.939]
Sample:  28%|##8       | 296/1050 [00:01, 263.59it/s, step size=1.11e+00, acc. rate=0.939]
Sample:  28%|##8       | 297/1050 [00:01, 263.59it/s, step size=1.11e+00, acc. rate=0.940]
Sample:  28%|##8       | 298/1050 [00:01, 263.59it/s, step size=1.11e+00, acc. rate=0.940]
Sample:  28%|##8       | 299/1050 [00:01, 263.59it/s, step size=1.11e+00, acc. rate=0.940]
Sample:  29%|##8       | 300/1050 [00:01, 263.59it/s, step size=1.11e+00, acc. rate=0.940]
Sample:  29%|##8       | 301/1050 [00:01, 263.59it/s, step size=1.11e+00, acc. rate=0.940]
Sample:  29%|##8       | 302/1050 [00:01, 263.59it/s, step size=1.11e+00, acc. rate=0.941]
Sample:  29%|##8       | 303/1050 [00:01, 263.59it/s, step size=1.11e+00, acc. rate=0.941]
Sample:  29%|##8       | 304/1050 [00:01, 263.59it/s, step size=1.11e+00, acc. rate=0.941]
Sample:  29%|##9       | 305/1050 [00:01, 263.59it/s, step size=1.11e+00, acc. rate=0.941]
Sample:  29%|##9       | 306/1050 [00:01, 263.59it/s, step size=1.11e+00, acc. rate=0.941]
Sample:  29%|##9       | 307/1050 [00:01, 263.59it/s, step size=1.11e+00, acc. rate=0.942]
Sample:  29%|##9       | 308/1050 [00:01, 263.59it/s, step size=1.11e+00, acc. rate=0.942]
Sample:  29%|##9       | 309/1050 [00:01, 263.59it/s, step size=1.11e+00, acc. rate=0.942]
Sample:  30%|##9       | 310/1050 [00:01, 263.59it/s, step size=1.11e+00, acc. rate=0.942]
Sample:  30%|##9       | 311/1050 [00:01, 263.59it/s, step size=1.11e+00, acc. rate=0.942]
Sample:  30%|##9       | 312/1050 [00:01, 263.59it/s, step size=1.11e+00, acc. rate=0.942]
Sample:  30%|##9       | 313/1050 [00:01, 263.59it/s, step size=1.11e+00, acc. rate=0.943]
Sample:  30%|##9       | 314/1050 [00:01, 263.59it/s, step size=1.11e+00, acc. rate=0.943]
Sample:  30%|###       | 315/1050 [00:01, 263.59it/s, step size=1.11e+00, acc. rate=0.943]
Sample:  30%|###       | 316/1050 [00:01, 263.59it/s, step size=1.11e+00, acc. rate=0.943]
Sample:  30%|###       | 317/1050 [00:01, 263.59it/s, step size=1.11e+00, acc. rate=0.943]
Sample:  30%|###       | 318/1050 [00:01, 263.59it/s, step size=1.11e+00, acc. rate=0.944]
Sample:  30%|###       | 319/1050 [00:01, 263.59it/s, step size=1.11e+00, acc. rate=0.944]
Sample:  30%|###       | 320/1050 [00:01, 263.59it/s, step size=1.11e+00, acc. rate=0.944]
Sample:  31%|###       | 321/1050 [00:01, 263.59it/s, step size=1.11e+00, acc. rate=0.941]
Sample:  31%|###       | 322/1050 [00:01, 263.59it/s, step size=1.11e+00, acc. rate=0.941]
Sample:  31%|###       | 323/1050 [00:01, 263.59it/s, step size=1.11e+00, acc. rate=0.941]
Sample:  31%|###       | 324/1050 [00:01, 263.59it/s, step size=1.11e+00, acc. rate=0.942]
Sample:  31%|###       | 325/1050 [00:01, 263.59it/s, step size=1.11e+00, acc. rate=0.942]
Sample:  31%|###1      | 326/1050 [00:01, 263.59it/s, step size=1.11e+00, acc. rate=0.942]
Sample:  31%|###1      | 327/1050 [00:01, 263.59it/s, step size=1.11e+00, acc. rate=0.942]
Sample:  31%|###1      | 328/1050 [00:01, 263.59it/s, step size=1.11e+00, acc. rate=0.942]
Sample:  31%|###1      | 329/1050 [00:01, 263.59it/s, step size=1.11e+00, acc. rate=0.942]
Sample:  31%|###1      | 330/1050 [00:01, 287.68it/s, step size=1.11e+00, acc. rate=0.942]
Sample:  31%|###1      | 330/1050 [00:01, 287.68it/s, step size=1.11e+00, acc. rate=0.943]
Sample:  32%|###1      | 331/1050 [00:01, 287.68it/s, step size=1.11e+00, acc. rate=0.943]
Sample:  32%|###1      | 332/1050 [00:01, 287.68it/s, step size=1.11e+00, acc. rate=0.943]
Sample:  32%|###1      | 333/1050 [00:01, 287.68it/s, step size=1.11e+00, acc. rate=0.943]
Sample:  32%|###1      | 334/1050 [00:01, 287.68it/s, step size=1.11e+00, acc. rate=0.943]
Sample:  32%|###1      | 335/1050 [00:01, 287.68it/s, step size=1.11e+00, acc. rate=0.943]
Sample:  32%|###2      | 336/1050 [00:01, 287.68it/s, step size=1.11e+00, acc. rate=0.944]
Sample:  32%|###2      | 337/1050 [00:01, 287.68it/s, step size=1.11e+00, acc. rate=0.944]
Sample:  32%|###2      | 338/1050 [00:01, 287.68it/s, step size=1.11e+00, acc. rate=0.944]
Sample:  32%|###2      | 339/1050 [00:01, 287.68it/s, step size=1.11e+00, acc. rate=0.944]
Sample:  32%|###2      | 340/1050 [00:01, 287.68it/s, step size=1.11e+00, acc. rate=0.944]
Sample:  32%|###2      | 341/1050 [00:01, 287.68it/s, step size=1.11e+00, acc. rate=0.944]
Sample:  33%|###2      | 342/1050 [00:01, 287.68it/s, step size=1.11e+00, acc. rate=0.945]
Sample:  33%|###2      | 343/1050 [00:01, 287.68it/s, step size=1.11e+00, acc. rate=0.945]
Sample:  33%|###2      | 344/1050 [00:01, 287.68it/s, step size=1.11e+00, acc. rate=0.945]
Sample:  33%|###2      | 345/1050 [00:01, 287.68it/s, step size=1.11e+00, acc. rate=0.945]
Sample:  33%|###2      | 346/1050 [00:01, 287.68it/s, step size=1.11e+00, acc. rate=0.945]
Sample:  33%|###3      | 347/1050 [00:01, 287.68it/s, step size=1.11e+00, acc. rate=0.945]
Sample:  33%|###3      | 348/1050 [00:01, 287.68it/s, step size=1.11e+00, acc. rate=0.946]
Sample:  33%|###3      | 349/1050 [00:01, 287.68it/s, step size=1.11e+00, acc. rate=0.943]
Sample:  33%|###3      | 350/1050 [00:01, 287.68it/s, step size=1.11e+00, acc. rate=0.943]
Sample:  33%|###3      | 351/1050 [00:01, 287.68it/s, step size=1.11e+00, acc. rate=0.943]
Sample:  34%|###3      | 352/1050 [00:01, 287.68it/s, step size=1.11e+00, acc. rate=0.943]
Sample:  34%|###3      | 353/1050 [00:01, 287.68it/s, step size=1.11e+00, acc. rate=0.944]
Sample:  34%|###3      | 354/1050 [00:01, 287.68it/s, step size=1.11e+00, acc. rate=0.944]
Sample:  34%|###3      | 355/1050 [00:01, 287.68it/s, step size=1.11e+00, acc. rate=0.944]
Sample:  34%|###3      | 356/1050 [00:01, 287.68it/s, step size=1.11e+00, acc. rate=0.944]
Sample:  34%|###4      | 357/1050 [00:01, 287.68it/s, step size=1.11e+00, acc. rate=0.944]
Sample:  34%|###4      | 358/1050 [00:01, 287.68it/s, step size=1.11e+00, acc. rate=0.944]
Sample:  34%|###4      | 359/1050 [00:01, 287.68it/s, step size=1.11e+00, acc. rate=0.944]
Sample:  34%|###4      | 360/1050 [00:01, 287.68it/s, step size=1.11e+00, acc. rate=0.945]
Sample:  34%|###4      | 361/1050 [00:01, 287.68it/s, step size=1.11e+00, acc. rate=0.945]
Sample:  34%|###4      | 362/1050 [00:01, 287.68it/s, step size=1.11e+00, acc. rate=0.945]
Sample:  35%|###4      | 363/1050 [00:01, 287.68it/s, step size=1.11e+00, acc. rate=0.945]
Sample:  35%|###4      | 364/1050 [00:01, 287.68it/s, step size=1.11e+00, acc. rate=0.945]
Sample:  35%|###4      | 365/1050 [00:01, 298.28it/s, step size=1.11e+00, acc. rate=0.945]
Sample:  35%|###4      | 365/1050 [00:01, 298.28it/s, step size=1.11e+00, acc. rate=0.945]
Sample:  35%|###4      | 366/1050 [00:01, 298.28it/s, step size=1.11e+00, acc. rate=0.946]
Sample:  35%|###4      | 367/1050 [00:01, 298.28it/s, step size=1.11e+00, acc. rate=0.946]
Sample:  35%|###5      | 368/1050 [00:01, 298.28it/s, step size=1.11e+00, acc. rate=0.946]
Sample:  35%|###5      | 369/1050 [00:01, 298.28it/s, step size=1.11e+00, acc. rate=0.946]
Sample:  35%|###5      | 370/1050 [00:01, 298.28it/s, step size=1.11e+00, acc. rate=0.946]
Sample:  35%|###5      | 371/1050 [00:01, 298.28it/s, step size=1.11e+00, acc. rate=0.946]
Sample:  35%|###5      | 372/1050 [00:01, 298.28it/s, step size=1.11e+00, acc. rate=0.946]
Sample:  36%|###5      | 373/1050 [00:01, 298.28it/s, step size=1.11e+00, acc. rate=0.947]
Sample:  36%|###5      | 374/1050 [00:01, 298.28it/s, step size=1.11e+00, acc. rate=0.947]
Sample:  36%|###5      | 375/1050 [00:01, 298.28it/s, step size=1.11e+00, acc. rate=0.947]
Sample:  36%|###5      | 376/1050 [00:01, 298.28it/s, step size=1.11e+00, acc. rate=0.947]
Sample:  36%|###5      | 377/1050 [00:01, 298.28it/s, step size=1.11e+00, acc. rate=0.947]
Sample:  36%|###6      | 378/1050 [00:01, 298.28it/s, step size=1.11e+00, acc. rate=0.947]
Sample:  36%|###6      | 379/1050 [00:01, 298.28it/s, step size=1.11e+00, acc. rate=0.947]
Sample:  36%|###6      | 380/1050 [00:01, 298.28it/s, step size=1.11e+00, acc. rate=0.948]
Sample:  36%|###6      | 381/1050 [00:01, 298.28it/s, step size=1.11e+00, acc. rate=0.948]
Sample:  36%|###6      | 382/1050 [00:01, 298.28it/s, step size=1.11e+00, acc. rate=0.948]
Sample:  36%|###6      | 383/1050 [00:01, 298.28it/s, step size=1.11e+00, acc. rate=0.948]
Sample:  37%|###6      | 384/1050 [00:01, 298.28it/s, step size=1.11e+00, acc. rate=0.948]
Sample:  37%|###6      | 385/1050 [00:01, 298.28it/s, step size=1.11e+00, acc. rate=0.948]
Sample:  37%|###6      | 386/1050 [00:01, 298.28it/s, step size=1.11e+00, acc. rate=0.948]
Sample:  37%|###6      | 387/1050 [00:01, 298.28it/s, step size=1.11e+00, acc. rate=0.948]
Sample:  37%|###6      | 388/1050 [00:01, 298.28it/s, step size=1.11e+00, acc. rate=0.949]
Sample:  37%|###7      | 389/1050 [00:01, 298.28it/s, step size=1.11e+00, acc. rate=0.949]
Sample:  37%|###7      | 390/1050 [00:01, 298.28it/s, step size=1.11e+00, acc. rate=0.949]
Sample:  37%|###7      | 391/1050 [00:01, 298.28it/s, step size=1.11e+00, acc. rate=0.949]
Sample:  37%|###7      | 392/1050 [00:01, 298.28it/s, step size=1.11e+00, acc. rate=0.949]
Sample:  37%|###7      | 393/1050 [00:01, 298.28it/s, step size=1.11e+00, acc. rate=0.949]
Sample:  38%|###7      | 394/1050 [00:01, 298.28it/s, step size=1.11e+00, acc. rate=0.949]
Sample:  38%|###7      | 395/1050 [00:01, 298.28it/s, step size=1.11e+00, acc. rate=0.949]
Sample:  38%|###7      | 396/1050 [00:01, 298.28it/s, step size=1.11e+00, acc. rate=0.950]
Sample:  38%|###7      | 397/1050 [00:01, 298.28it/s, step size=1.11e+00, acc. rate=0.950]
Sample:  38%|###7      | 398/1050 [00:01, 298.28it/s, step size=1.11e+00, acc. rate=0.950]
Sample:  38%|###8      | 399/1050 [00:01, 298.28it/s, step size=1.11e+00, acc. rate=0.950]
Sample:  38%|###8      | 400/1050 [00:01, 298.28it/s, step size=1.11e+00, acc. rate=0.950]
Sample:  38%|###8      | 401/1050 [00:01, 312.25it/s, step size=1.11e+00, acc. rate=0.950]
Sample:  38%|###8      | 401/1050 [00:01, 312.25it/s, step size=1.11e+00, acc. rate=0.950]
Sample:  38%|###8      | 402/1050 [00:01, 312.25it/s, step size=1.11e+00, acc. rate=0.950]
Sample:  38%|###8      | 403/1050 [00:01, 312.25it/s, step size=1.11e+00, acc. rate=0.950]
Sample:  38%|###8      | 404/1050 [00:01, 312.25it/s, step size=1.11e+00, acc. rate=0.951]
Sample:  39%|###8      | 405/1050 [00:01, 312.25it/s, step size=1.11e+00, acc. rate=0.951]
Sample:  39%|###8      | 406/1050 [00:01, 312.25it/s, step size=1.11e+00, acc. rate=0.951]
Sample:  39%|###8      | 407/1050 [00:01, 312.25it/s, step size=1.11e+00, acc. rate=0.951]
Sample:  39%|###8      | 408/1050 [00:01, 312.25it/s, step size=1.11e+00, acc. rate=0.951]
Sample:  39%|###8      | 409/1050 [00:01, 312.25it/s, step size=1.11e+00, acc. rate=0.951]
Sample:  39%|###9      | 410/1050 [00:01, 312.25it/s, step size=1.11e+00, acc. rate=0.951]
Sample:  39%|###9      | 411/1050 [00:01, 312.25it/s, step size=1.11e+00, acc. rate=0.951]
Sample:  39%|###9      | 412/1050 [00:01, 312.25it/s, step size=1.11e+00, acc. rate=0.952]
Sample:  39%|###9      | 413/1050 [00:01, 312.25it/s, step size=1.11e+00, acc. rate=0.952]
Sample:  39%|###9      | 414/1050 [00:01, 312.25it/s, step size=1.11e+00, acc. rate=0.952]
Sample:  40%|###9      | 415/1050 [00:01, 312.25it/s, step size=1.11e+00, acc. rate=0.952]
Sample:  40%|###9      | 416/1050 [00:01, 312.25it/s, step size=1.11e+00, acc. rate=0.952]
Sample:  40%|###9      | 417/1050 [00:01, 312.25it/s, step size=1.11e+00, acc. rate=0.952]
Sample:  40%|###9      | 418/1050 [00:01, 312.25it/s, step size=1.11e+00, acc. rate=0.952]
Sample:  40%|###9      | 419/1050 [00:01, 312.25it/s, step size=1.11e+00, acc. rate=0.952]
Sample:  40%|####      | 420/1050 [00:01, 312.25it/s, step size=1.11e+00, acc. rate=0.952]
Sample:  40%|####      | 421/1050 [00:01, 312.25it/s, step size=1.11e+00, acc. rate=0.953]
Sample:  40%|####      | 422/1050 [00:01, 312.25it/s, step size=1.11e+00, acc. rate=0.953]
Sample:  40%|####      | 423/1050 [00:01, 312.25it/s, step size=1.11e+00, acc. rate=0.953]
Sample:  40%|####      | 424/1050 [00:01, 312.25it/s, step size=1.11e+00, acc. rate=0.953]
Sample:  40%|####      | 425/1050 [00:01, 312.25it/s, step size=1.11e+00, acc. rate=0.953]
Sample:  41%|####      | 426/1050 [00:01, 312.25it/s, step size=1.11e+00, acc. rate=0.953]
Sample:  41%|####      | 427/1050 [00:01, 312.25it/s, step size=1.11e+00, acc. rate=0.953]
Sample:  41%|####      | 428/1050 [00:01, 312.25it/s, step size=1.11e+00, acc. rate=0.953]
Sample:  41%|####      | 429/1050 [00:01, 312.25it/s, step size=1.11e+00, acc. rate=0.953]
Sample:  41%|####      | 430/1050 [00:01, 312.25it/s, step size=1.11e+00, acc. rate=0.954]
Sample:  41%|####1     | 431/1050 [00:01, 312.25it/s, step size=1.11e+00, acc. rate=0.954]
Sample:  41%|####1     | 432/1050 [00:01, 312.25it/s, step size=1.11e+00, acc. rate=0.954]
Sample:  41%|####1     | 433/1050 [00:01, 312.25it/s, step size=1.11e+00, acc. rate=0.954]
Sample:  41%|####1     | 434/1050 [00:01, 312.25it/s, step size=1.11e+00, acc. rate=0.954]
Sample:  41%|####1     | 435/1050 [00:01, 312.25it/s, step size=1.11e+00, acc. rate=0.954]
Sample:  42%|####1     | 436/1050 [00:01, 312.25it/s, step size=1.11e+00, acc. rate=0.954]
Sample:  42%|####1     | 437/1050 [00:01, 324.00it/s, step size=1.11e+00, acc. rate=0.954]
Sample:  42%|####1     | 437/1050 [00:01, 324.00it/s, step size=1.11e+00, acc. rate=0.954]
Sample:  42%|####1     | 438/1050 [00:01, 324.00it/s, step size=1.11e+00, acc. rate=0.954]
Sample:  42%|####1     | 439/1050 [00:01, 324.00it/s, step size=1.11e+00, acc. rate=0.955]
Sample:  42%|####1     | 440/1050 [00:01, 324.00it/s, step size=1.11e+00, acc. rate=0.955]
Sample:  42%|####2     | 441/1050 [00:01, 324.00it/s, step size=1.11e+00, acc. rate=0.955]
Sample:  42%|####2     | 442/1050 [00:01, 324.00it/s, step size=1.11e+00, acc. rate=0.955]
Sample:  42%|####2     | 443/1050 [00:01, 324.00it/s, step size=1.11e+00, acc. rate=0.955]
Sample:  42%|####2     | 444/1050 [00:01, 324.00it/s, step size=1.11e+00, acc. rate=0.955]
Sample:  42%|####2     | 445/1050 [00:01, 324.00it/s, step size=1.11e+00, acc. rate=0.955]
Sample:  42%|####2     | 446/1050 [00:01, 324.00it/s, step size=1.11e+00, acc. rate=0.955]
Sample:  43%|####2     | 447/1050 [00:01, 324.00it/s, step size=1.11e+00, acc. rate=0.955]
Sample:  43%|####2     | 448/1050 [00:01, 324.00it/s, step size=1.11e+00, acc. rate=0.955]
Sample:  43%|####2     | 449/1050 [00:01, 324.00it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  43%|####2     | 450/1050 [00:01, 324.00it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  43%|####2     | 451/1050 [00:01, 324.00it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  43%|####3     | 452/1050 [00:01, 324.00it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  43%|####3     | 453/1050 [00:01, 324.00it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  43%|####3     | 454/1050 [00:01, 324.00it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  43%|####3     | 455/1050 [00:01, 324.00it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  43%|####3     | 456/1050 [00:01, 324.00it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  44%|####3     | 457/1050 [00:01, 324.00it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  44%|####3     | 458/1050 [00:01, 324.00it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  44%|####3     | 459/1050 [00:01, 324.00it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  44%|####3     | 460/1050 [00:01, 324.00it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  44%|####3     | 461/1050 [00:01, 324.00it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  44%|####4     | 462/1050 [00:01, 324.00it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  44%|####4     | 463/1050 [00:01, 324.00it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  44%|####4     | 464/1050 [00:01, 324.00it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  44%|####4     | 465/1050 [00:01, 324.00it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  44%|####4     | 466/1050 [00:01, 324.00it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  44%|####4     | 467/1050 [00:01, 324.00it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  45%|####4     | 468/1050 [00:01, 324.00it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  45%|####4     | 469/1050 [00:01, 324.00it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  45%|####4     | 470/1050 [00:01, 324.00it/s, step size=1.11e+00, acc. rate=0.958]
Sample:  45%|####4     | 471/1050 [00:01, 324.00it/s, step size=1.11e+00, acc. rate=0.958]
Sample:  45%|####4     | 472/1050 [00:01, 324.00it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  45%|####5     | 473/1050 [00:01, 332.78it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  45%|####5     | 473/1050 [00:01, 332.78it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  45%|####5     | 474/1050 [00:01, 332.78it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  45%|####5     | 475/1050 [00:01, 332.78it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  45%|####5     | 476/1050 [00:01, 332.78it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  45%|####5     | 477/1050 [00:01, 332.78it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  46%|####5     | 478/1050 [00:01, 332.78it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  46%|####5     | 479/1050 [00:01, 332.78it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  46%|####5     | 480/1050 [00:01, 332.78it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  46%|####5     | 481/1050 [00:01, 332.78it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  46%|####5     | 482/1050 [00:01, 332.78it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  46%|####6     | 483/1050 [00:01, 332.78it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  46%|####6     | 484/1050 [00:01, 332.78it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  46%|####6     | 485/1050 [00:01, 332.78it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  46%|####6     | 486/1050 [00:01, 332.78it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  46%|####6     | 487/1050 [00:01, 332.78it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  46%|####6     | 488/1050 [00:01, 332.78it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  47%|####6     | 489/1050 [00:01, 332.78it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  47%|####6     | 490/1050 [00:01, 332.78it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  47%|####6     | 491/1050 [00:01, 332.78it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  47%|####6     | 492/1050 [00:01, 332.78it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  47%|####6     | 493/1050 [00:01, 332.78it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  47%|####7     | 494/1050 [00:01, 332.78it/s, step size=1.11e+00, acc. rate=0.958]
Sample:  47%|####7     | 495/1050 [00:01, 332.78it/s, step size=1.11e+00, acc. rate=0.958]
Sample:  47%|####7     | 496/1050 [00:01, 332.78it/s, step size=1.11e+00, acc. rate=0.958]
Sample:  47%|####7     | 497/1050 [00:01, 332.78it/s, step size=1.11e+00, acc. rate=0.958]
Sample:  47%|####7     | 498/1050 [00:01, 332.78it/s, step size=1.11e+00, acc. rate=0.958]
Sample:  48%|####7     | 499/1050 [00:01, 332.78it/s, step size=1.11e+00, acc. rate=0.958]
Sample:  48%|####7     | 500/1050 [00:01, 332.78it/s, step size=1.11e+00, acc. rate=0.958]
Sample:  48%|####7     | 501/1050 [00:01, 332.78it/s, step size=1.11e+00, acc. rate=0.958]
Sample:  48%|####7     | 502/1050 [00:01, 332.78it/s, step size=1.11e+00, acc. rate=0.958]
Sample:  48%|####7     | 503/1050 [00:01, 332.78it/s, step size=1.11e+00, acc. rate=0.958]
Sample:  48%|####8     | 504/1050 [00:01, 332.78it/s, step size=1.11e+00, acc. rate=0.958]
Sample:  48%|####8     | 505/1050 [00:01, 332.78it/s, step size=1.11e+00, acc. rate=0.958]
Sample:  48%|####8     | 506/1050 [00:01, 332.78it/s, step size=1.11e+00, acc. rate=0.959]
Sample:  48%|####8     | 507/1050 [00:01, 332.78it/s, step size=1.11e+00, acc. rate=0.959]
Sample:  48%|####8     | 508/1050 [00:01, 332.78it/s, step size=1.11e+00, acc. rate=0.959]
Sample:  48%|####8     | 509/1050 [00:01, 332.78it/s, step size=1.11e+00, acc. rate=0.959]
Sample:  49%|####8     | 510/1050 [00:01, 340.78it/s, step size=1.11e+00, acc. rate=0.959]
Sample:  49%|####8     | 510/1050 [00:01, 340.78it/s, step size=1.11e+00, acc. rate=0.959]
Sample:  49%|####8     | 511/1050 [00:01, 340.78it/s, step size=1.11e+00, acc. rate=0.959]
Sample:  49%|####8     | 512/1050 [00:01, 340.78it/s, step size=1.11e+00, acc. rate=0.959]
Sample:  49%|####8     | 513/1050 [00:01, 340.78it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  49%|####8     | 514/1050 [00:01, 340.78it/s, step size=1.11e+00, acc. rate=0.955]
Sample:  49%|####9     | 515/1050 [00:01, 340.78it/s, step size=1.11e+00, acc. rate=0.955]
Sample:  49%|####9     | 516/1050 [00:01, 340.78it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  49%|####9     | 517/1050 [00:01, 340.78it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  49%|####9     | 518/1050 [00:01, 340.78it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  49%|####9     | 519/1050 [00:01, 340.78it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  50%|####9     | 520/1050 [00:01, 340.78it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  50%|####9     | 521/1050 [00:01, 340.78it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  50%|####9     | 522/1050 [00:01, 340.78it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  50%|####9     | 523/1050 [00:01, 340.78it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  50%|####9     | 524/1050 [00:01, 340.78it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  50%|#####     | 525/1050 [00:01, 340.78it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  50%|#####     | 526/1050 [00:01, 340.78it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  50%|#####     | 527/1050 [00:01, 340.78it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  50%|#####     | 528/1050 [00:01, 340.78it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  50%|#####     | 529/1050 [00:01, 340.78it/s, step size=1.11e+00, acc. rate=0.955]
Sample:  50%|#####     | 530/1050 [00:01, 340.78it/s, step size=1.11e+00, acc. rate=0.955]
Sample:  51%|#####     | 531/1050 [00:01, 340.78it/s, step size=1.11e+00, acc. rate=0.955]
Sample:  51%|#####     | 532/1050 [00:01, 340.78it/s, step size=1.11e+00, acc. rate=0.955]
Sample:  51%|#####     | 533/1050 [00:01, 340.78it/s, step size=1.11e+00, acc. rate=0.955]
Sample:  51%|#####     | 534/1050 [00:01, 340.78it/s, step size=1.11e+00, acc. rate=0.955]
Sample:  51%|#####     | 535/1050 [00:01, 340.78it/s, step size=1.11e+00, acc. rate=0.955]
Sample:  51%|#####1    | 536/1050 [00:01, 340.78it/s, step size=1.11e+00, acc. rate=0.955]
Sample:  51%|#####1    | 537/1050 [00:01, 340.78it/s, step size=1.11e+00, acc. rate=0.955]
Sample:  51%|#####1    | 538/1050 [00:01, 340.78it/s, step size=1.11e+00, acc. rate=0.955]
Sample:  51%|#####1    | 539/1050 [00:01, 340.78it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  51%|#####1    | 540/1050 [00:01, 340.78it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  52%|#####1    | 541/1050 [00:01, 340.78it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  52%|#####1    | 542/1050 [00:01, 340.78it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  52%|#####1    | 543/1050 [00:01, 340.78it/s, step size=1.11e+00, acc. rate=0.954]
Sample:  52%|#####1    | 544/1050 [00:01, 340.78it/s, step size=1.11e+00, acc. rate=0.954]
Sample:  52%|#####1    | 545/1050 [00:01, 340.78it/s, step size=1.11e+00, acc. rate=0.954]
Sample:  52%|#####2    | 546/1050 [00:01, 340.78it/s, step size=1.11e+00, acc. rate=0.954]
Sample:  52%|#####2    | 547/1050 [00:01, 347.44it/s, step size=1.11e+00, acc. rate=0.954]
Sample:  52%|#####2    | 547/1050 [00:01, 347.44it/s, step size=1.11e+00, acc. rate=0.954]
Sample:  52%|#####2    | 548/1050 [00:01, 347.44it/s, step size=1.11e+00, acc. rate=0.954]
Sample:  52%|#####2    | 549/1050 [00:01, 347.44it/s, step size=1.11e+00, acc. rate=0.955]
Sample:  52%|#####2    | 550/1050 [00:01, 347.44it/s, step size=1.11e+00, acc. rate=0.955]
Sample:  52%|#####2    | 551/1050 [00:01, 347.44it/s, step size=1.11e+00, acc. rate=0.955]
Sample:  53%|#####2    | 552/1050 [00:01, 347.44it/s, step size=1.11e+00, acc. rate=0.955]
Sample:  53%|#####2    | 553/1050 [00:01, 347.44it/s, step size=1.11e+00, acc. rate=0.955]
Sample:  53%|#####2    | 554/1050 [00:01, 347.44it/s, step size=1.11e+00, acc. rate=0.955]
Sample:  53%|#####2    | 555/1050 [00:01, 347.44it/s, step size=1.11e+00, acc. rate=0.955]
Sample:  53%|#####2    | 556/1050 [00:01, 347.44it/s, step size=1.11e+00, acc. rate=0.955]
Sample:  53%|#####3    | 557/1050 [00:01, 347.44it/s, step size=1.11e+00, acc. rate=0.955]
Sample:  53%|#####3    | 558/1050 [00:01, 347.44it/s, step size=1.11e+00, acc. rate=0.955]
Sample:  53%|#####3    | 559/1050 [00:01, 347.44it/s, step size=1.11e+00, acc. rate=0.955]
Sample:  53%|#####3    | 560/1050 [00:01, 347.44it/s, step size=1.11e+00, acc. rate=0.955]
Sample:  53%|#####3    | 561/1050 [00:01, 347.44it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  54%|#####3    | 562/1050 [00:01, 347.44it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  54%|#####3    | 563/1050 [00:01, 347.44it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  54%|#####3    | 564/1050 [00:01, 347.44it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  54%|#####3    | 565/1050 [00:01, 347.44it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  54%|#####3    | 566/1050 [00:01, 347.44it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  54%|#####4    | 567/1050 [00:01, 347.44it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  54%|#####4    | 568/1050 [00:01, 347.44it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  54%|#####4    | 569/1050 [00:01, 347.44it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  54%|#####4    | 570/1050 [00:01, 347.44it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  54%|#####4    | 571/1050 [00:01, 347.44it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  54%|#####4    | 572/1050 [00:01, 347.44it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  55%|#####4    | 573/1050 [00:01, 347.44it/s, step size=1.11e+00, acc. rate=0.956]
Sample:  55%|#####4    | 574/1050 [00:01, 347.44it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  55%|#####4    | 575/1050 [00:01, 347.44it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  55%|#####4    | 576/1050 [00:01, 347.44it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  55%|#####4    | 577/1050 [00:01, 347.44it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  55%|#####5    | 578/1050 [00:01, 347.44it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  55%|#####5    | 579/1050 [00:01, 347.44it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  55%|#####5    | 580/1050 [00:01, 347.44it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  55%|#####5    | 581/1050 [00:01, 347.44it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  55%|#####5    | 582/1050 [00:01, 347.44it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  56%|#####5    | 583/1050 [00:01, 350.83it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  56%|#####5    | 583/1050 [00:01, 350.83it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  56%|#####5    | 584/1050 [00:01, 350.83it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  56%|#####5    | 585/1050 [00:01, 350.83it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  56%|#####5    | 586/1050 [00:01, 350.83it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  56%|#####5    | 587/1050 [00:01, 350.83it/s, step size=1.11e+00, acc. rate=0.957]
Sample:  56%|#####6    | 588/1050 [00:01, 350.83it/s, step size=1.11e+00, acc. rate=0.958]
Sample:  56%|#####6    | 589/1050 [00:01, 350.83it/s, step size=1.11e+00, acc. rate=0.958]
Sample:  56%|#####6    | 590/1050 [00:01, 350.83it/s, step size=1.11e+00, acc. rate=0.958]
Sample:  56%|#####6    | 591/1050 [00:01, 350.83it/s, step size=1.11e+00, acc. rate=0.958]
Sample:  56%|#####6    | 592/1050 [00:01, 350.83it/s, step size=1.11e+00, acc. rate=0.958]
Sample:  56%|#####6    | 593/1050 [00:01, 350.83it/s, step size=1.11e+00, acc. rate=0.958]
Sample:  57%|#####6    | 594/1050 [00:01, 350.83it/s, step size=1.11e+00, acc. rate=0.958]
Sample:  57%|#####6    | 595/1050 [00:02, 350.83it/s, step size=1.11e+00, acc. rate=0.958]
Sample:  57%|#####6    | 596/1050 [00:02, 350.83it/s, step size=1.11e+00, acc. rate=0.958]
Sample:  57%|#####6    | 597/1050 [00:02, 350.83it/s, step size=1.11e+00, acc. rate=0.958]
Sample:  57%|#####6    | 598/1050 [00:02, 350.83it/s, step size=1.11e+00, acc. rate=0.958]
Sample:  57%|#####7    | 599/1050 [00:02, 350.83it/s, step size=1.11e+00, acc. rate=0.958]
Sample:  57%|#####7    | 600/1050 [00:02, 350.83it/s, step size=1.11e+00, acc. rate=0.958]
Sample:  57%|#####7    | 601/1050 [00:02, 350.83it/s, step size=1.11e+00, acc. rate=0.958]
Sample:  57%|#####7    | 602/1050 [00:02, 350.83it/s, step size=1.11e+00, acc. rate=0.959]
Sample:  57%|#####7    | 603/1050 [00:02, 350.83it/s, step size=1.11e+00, acc. rate=0.959]
Sample:  58%|#####7    | 604/1050 [00:02, 350.83it/s, step size=1.11e+00, acc. rate=0.959]
Sample:  58%|#####7    | 605/1050 [00:02, 350.83it/s, step size=1.11e+00, acc. rate=0.959]
Sample:  58%|#####7    | 606/1050 [00:02, 350.83it/s, step size=1.11e+00, acc. rate=0.959]
Sample:  58%|#####7    | 607/1050 [00:02, 350.83it/s, step size=1.11e+00, acc. rate=0.959]
Sample:  58%|#####7    | 608/1050 [00:02, 350.83it/s, step size=1.11e+00, acc. rate=0.959]
Sample:  58%|#####8    | 609/1050 [00:02, 350.83it/s, step size=1.11e+00, acc. rate=0.959]
Sample:  58%|#####8    | 610/1050 [00:02, 350.83it/s, step size=1.11e+00, acc. rate=0.959]
Sample:  58%|#####8    | 611/1050 [00:02, 350.83it/s, step size=1.11e+00, acc. rate=0.959]
Sample:  58%|#####8    | 612/1050 [00:02, 350.83it/s, step size=1.11e+00, acc. rate=0.959]
Sample:  58%|#####8    | 613/1050 [00:02, 350.83it/s, step size=1.11e+00, acc. rate=0.959]
Sample:  58%|#####8    | 614/1050 [00:02, 350.83it/s, step size=1.11e+00, acc. rate=0.959]
Sample:  59%|#####8    | 615/1050 [00:02, 350.83it/s, step size=1.11e+00, acc. rate=0.959]
Sample:  59%|#####8    | 616/1050 [00:02, 350.83it/s, step size=1.11e+00, acc. rate=0.959]
Sample:  59%|#####8    | 617/1050 [00:02, 350.83it/s, step size=1.11e+00, acc. rate=0.960]
Sample:  59%|#####8    | 618/1050 [00:02, 350.83it/s, step size=1.11e+00, acc. rate=0.960]
Sample:  59%|#####8    | 619/1050 [00:02, 349.89it/s, step size=1.11e+00, acc. rate=0.960]
Sample:  59%|#####8    | 619/1050 [00:02, 349.89it/s, step size=1.11e+00, acc. rate=0.960]
Sample:  59%|#####9    | 620/1050 [00:02, 349.89it/s, step size=1.11e+00, acc. rate=0.960]
Sample:  59%|#####9    | 621/1050 [00:02, 349.89it/s, step size=1.11e+00, acc. rate=0.960]
Sample:  59%|#####9    | 622/1050 [00:02, 349.89it/s, step size=1.11e+00, acc. rate=0.960]
Sample:  59%|#####9    | 623/1050 [00:02, 349.89it/s, step size=1.11e+00, acc. rate=0.960]
Sample:  59%|#####9    | 624/1050 [00:02, 349.89it/s, step size=1.11e+00, acc. rate=0.960]
Sample:  60%|#####9    | 625/1050 [00:02, 349.89it/s, step size=1.11e+00, acc. rate=0.960]
Sample:  60%|#####9    | 626/1050 [00:02, 349.89it/s, step size=1.11e+00, acc. rate=0.960]
Sample:  60%|#####9    | 627/1050 [00:02, 349.89it/s, step size=1.11e+00, acc. rate=0.960]
Sample:  60%|#####9    | 628/1050 [00:02, 349.89it/s, step size=1.11e+00, acc. rate=0.960]
Sample:  60%|#####9    | 629/1050 [00:02, 349.89it/s, step size=1.11e+00, acc. rate=0.960]
Sample:  60%|######    | 630/1050 [00:02, 349.89it/s, step size=1.11e+00, acc. rate=0.960]
Sample:  60%|######    | 631/1050 [00:02, 349.89it/s, step size=1.11e+00, acc. rate=0.960]
Sample:  60%|######    | 632/1050 [00:02, 349.89it/s, step size=1.11e+00, acc. rate=0.961]
Sample:  60%|######    | 633/1050 [00:02, 349.89it/s, step size=1.11e+00, acc. rate=0.961]
Sample:  60%|######    | 634/1050 [00:02, 349.89it/s, step size=1.11e+00, acc. rate=0.961]
Sample:  60%|######    | 635/1050 [00:02, 349.89it/s, step size=1.11e+00, acc. rate=0.961]
Sample:  61%|######    | 636/1050 [00:02, 349.89it/s, step size=1.11e+00, acc. rate=0.961]
Sample:  61%|######    | 637/1050 [00:02, 349.89it/s, step size=1.11e+00, acc. rate=0.961]
Sample:  61%|######    | 638/1050 [00:02, 349.89it/s, step size=1.11e+00, acc. rate=0.961]
Sample:  61%|######    | 639/1050 [00:02, 349.89it/s, step size=1.11e+00, acc. rate=0.961]
Sample:  61%|######    | 640/1050 [00:02, 349.89it/s, step size=1.11e+00, acc. rate=0.961]
Sample:  61%|######1   | 641/1050 [00:02, 349.89it/s, step size=1.11e+00, acc. rate=0.961]
Sample:  61%|######1   | 642/1050 [00:02, 349.89it/s, step size=1.11e+00, acc. rate=0.961]
Sample:  61%|######1   | 643/1050 [00:02, 349.89it/s, step size=1.11e+00, acc. rate=0.961]
Sample:  61%|######1   | 644/1050 [00:02, 349.89it/s, step size=1.11e+00, acc. rate=0.961]
Sample:  61%|######1   | 645/1050 [00:02, 349.89it/s, step size=1.11e+00, acc. rate=0.961]
Sample:  62%|######1   | 646/1050 [00:02, 349.89it/s, step size=1.11e+00, acc. rate=0.961]
Sample:  62%|######1   | 647/1050 [00:02, 349.89it/s, step size=1.11e+00, acc. rate=0.961]
Sample:  62%|######1   | 648/1050 [00:02, 349.89it/s, step size=1.11e+00, acc. rate=0.961]
Sample:  62%|######1   | 649/1050 [00:02, 349.89it/s, step size=1.11e+00, acc. rate=0.962]
Sample:  62%|######1   | 650/1050 [00:02, 349.89it/s, step size=1.11e+00, acc. rate=0.962]
Sample:  62%|######2   | 651/1050 [00:02, 349.89it/s, step size=1.11e+00, acc. rate=0.962]
Sample:  62%|######2   | 652/1050 [00:02, 349.89it/s, step size=1.11e+00, acc. rate=0.962]
Sample:  62%|######2   | 653/1050 [00:02, 349.89it/s, step size=1.11e+00, acc. rate=0.962]
Sample:  62%|######2   | 654/1050 [00:02, 349.89it/s, step size=1.11e+00, acc. rate=0.962]
Sample:  62%|######2   | 655/1050 [00:02, 315.56it/s, step size=1.11e+00, acc. rate=0.962]
Sample:  62%|######2   | 655/1050 [00:02, 315.56it/s, step size=1.11e+00, acc. rate=0.962]
Sample:  62%|######2   | 656/1050 [00:02, 315.56it/s, step size=1.11e+00, acc. rate=0.962]
Sample:  63%|######2   | 657/1050 [00:02, 315.56it/s, step size=1.11e+00, acc. rate=0.962]
Sample:  63%|######2   | 658/1050 [00:02, 315.56it/s, step size=1.11e+00, acc. rate=0.962]
Sample:  63%|######2   | 659/1050 [00:02, 315.56it/s, step size=1.11e+00, acc. rate=0.962]
Sample:  63%|######2   | 660/1050 [00:02, 315.56it/s, step size=1.11e+00, acc. rate=0.962]
Sample:  63%|######2   | 661/1050 [00:02, 315.56it/s, step size=1.11e+00, acc. rate=0.962]
Sample:  63%|######3   | 662/1050 [00:02, 315.56it/s, step size=1.11e+00, acc. rate=0.962]
Sample:  63%|######3   | 663/1050 [00:02, 315.56it/s, step size=1.11e+00, acc. rate=0.962]
Sample:  63%|######3   | 664/1050 [00:02, 315.56it/s, step size=1.11e+00, acc. rate=0.962]
Sample:  63%|######3   | 665/1050 [00:02, 315.56it/s, step size=1.11e+00, acc. rate=0.962]
Sample:  63%|######3   | 666/1050 [00:02, 315.56it/s, step size=1.11e+00, acc. rate=0.963]
Sample:  64%|######3   | 667/1050 [00:02, 315.56it/s, step size=1.11e+00, acc. rate=0.963]
Sample:  64%|######3   | 668/1050 [00:02, 315.56it/s, step size=1.11e+00, acc. rate=0.963]
Sample:  64%|######3   | 669/1050 [00:02, 315.56it/s, step size=1.11e+00, acc. rate=0.963]
Sample:  64%|######3   | 670/1050 [00:02, 315.56it/s, step size=1.11e+00, acc. rate=0.963]
Sample:  64%|######3   | 671/1050 [00:02, 315.56it/s, step size=1.11e+00, acc. rate=0.963]
Sample:  64%|######4   | 672/1050 [00:02, 315.56it/s, step size=1.11e+00, acc. rate=0.963]
Sample:  64%|######4   | 673/1050 [00:02, 315.56it/s, step size=1.11e+00, acc. rate=0.963]
Sample:  64%|######4   | 674/1050 [00:02, 315.56it/s, step size=1.11e+00, acc. rate=0.963]
Sample:  64%|######4   | 675/1050 [00:02, 315.56it/s, step size=1.11e+00, acc. rate=0.963]
Sample:  64%|######4   | 676/1050 [00:02, 315.56it/s, step size=1.11e+00, acc. rate=0.963]
Sample:  64%|######4   | 677/1050 [00:02, 315.56it/s, step size=1.11e+00, acc. rate=0.963]
Sample:  65%|######4   | 678/1050 [00:02, 315.56it/s, step size=1.11e+00, acc. rate=0.963]
Sample:  65%|######4   | 679/1050 [00:02, 315.56it/s, step size=1.11e+00, acc. rate=0.963]
Sample:  65%|######4   | 680/1050 [00:02, 315.56it/s, step size=1.11e+00, acc. rate=0.963]
Sample:  65%|######4   | 681/1050 [00:02, 315.56it/s, step size=1.11e+00, acc. rate=0.963]
Sample:  65%|######4   | 682/1050 [00:02, 315.56it/s, step size=1.11e+00, acc. rate=0.963]
Sample:  65%|######5   | 683/1050 [00:02, 315.56it/s, step size=1.11e+00, acc. rate=0.963]
Sample:  65%|######5   | 684/1050 [00:02, 315.56it/s, step size=1.11e+00, acc. rate=0.964]
Sample:  65%|######5   | 685/1050 [00:02, 315.56it/s, step size=1.11e+00, acc. rate=0.964]
Sample:  65%|######5   | 686/1050 [00:02, 315.56it/s, step size=1.11e+00, acc. rate=0.964]
Sample:  65%|######5   | 687/1050 [00:02, 315.56it/s, step size=1.11e+00, acc. rate=0.964]
Sample:  66%|######5   | 688/1050 [00:02, 315.56it/s, step size=1.11e+00, acc. rate=0.964]
Sample:  66%|######5   | 689/1050 [00:02, 320.89it/s, step size=1.11e+00, acc. rate=0.964]
Sample:  66%|######5   | 689/1050 [00:02, 320.89it/s, step size=1.11e+00, acc. rate=0.964]
Sample:  66%|######5   | 690/1050 [00:02, 320.89it/s, step size=1.11e+00, acc. rate=0.964]
Sample:  66%|######5   | 691/1050 [00:02, 320.89it/s, step size=1.11e+00, acc. rate=0.964]
Sample:  66%|######5   | 692/1050 [00:02, 320.89it/s, step size=1.11e+00, acc. rate=0.964]
Sample:  66%|######6   | 693/1050 [00:02, 320.89it/s, step size=1.11e+00, acc. rate=0.964]
Sample:  66%|######6   | 694/1050 [00:02, 320.89it/s, step size=1.11e+00, acc. rate=0.964]
Sample:  66%|######6   | 695/1050 [00:02, 320.89it/s, step size=1.11e+00, acc. rate=0.964]
Sample:  66%|######6   | 696/1050 [00:02, 320.89it/s, step size=1.11e+00, acc. rate=0.964]
Sample:  66%|######6   | 697/1050 [00:02, 320.89it/s, step size=1.11e+00, acc. rate=0.964]
Sample:  66%|######6   | 698/1050 [00:02, 320.89it/s, step size=1.11e+00, acc. rate=0.964]
Sample:  67%|######6   | 699/1050 [00:02, 320.89it/s, step size=1.11e+00, acc. rate=0.964]
Sample:  67%|######6   | 700/1050 [00:02, 320.89it/s, step size=1.11e+00, acc. rate=0.964]
Sample:  67%|######6   | 701/1050 [00:02, 320.89it/s, step size=1.11e+00, acc. rate=0.964]
Sample:  67%|######6   | 702/1050 [00:02, 320.89it/s, step size=1.11e+00, acc. rate=0.964]
Sample:  67%|######6   | 703/1050 [00:02, 320.89it/s, step size=1.11e+00, acc. rate=0.964]
Sample:  67%|######7   | 704/1050 [00:02, 320.89it/s, step size=1.11e+00, acc. rate=0.965]
Sample:  67%|######7   | 705/1050 [00:02, 320.89it/s, step size=1.11e+00, acc. rate=0.965]
Sample:  67%|######7   | 706/1050 [00:02, 320.89it/s, step size=1.11e+00, acc. rate=0.965]
Sample:  67%|######7   | 707/1050 [00:02, 320.89it/s, step size=1.11e+00, acc. rate=0.965]
Sample:  67%|######7   | 708/1050 [00:02, 320.89it/s, step size=1.11e+00, acc. rate=0.965]
Sample:  68%|######7   | 709/1050 [00:02, 320.89it/s, step size=1.11e+00, acc. rate=0.965]
Sample:  68%|######7   | 710/1050 [00:02, 320.89it/s, step size=1.11e+00, acc. rate=0.965]
Sample:  68%|######7   | 711/1050 [00:02, 320.89it/s, step size=1.11e+00, acc. rate=0.965]
Sample:  68%|######7   | 712/1050 [00:02, 320.89it/s, step size=1.11e+00, acc. rate=0.965]
Sample:  68%|######7   | 713/1050 [00:02, 320.89it/s, step size=1.11e+00, acc. rate=0.965]
Sample:  68%|######8   | 714/1050 [00:02, 320.89it/s, step size=1.11e+00, acc. rate=0.965]
Sample:  68%|######8   | 715/1050 [00:02, 320.89it/s, step size=1.11e+00, acc. rate=0.965]
Sample:  68%|######8   | 716/1050 [00:02, 320.89it/s, step size=1.11e+00, acc. rate=0.965]
Sample:  68%|######8   | 717/1050 [00:02, 320.89it/s, step size=1.11e+00, acc. rate=0.965]
Sample:  68%|######8   | 718/1050 [00:02, 320.89it/s, step size=1.11e+00, acc. rate=0.965]
Sample:  68%|######8   | 719/1050 [00:02, 320.89it/s, step size=1.11e+00, acc. rate=0.965]
Sample:  69%|######8   | 720/1050 [00:02, 320.89it/s, step size=1.11e+00, acc. rate=0.965]
Sample:  69%|######8   | 721/1050 [00:02, 320.89it/s, step size=1.11e+00, acc. rate=0.965]
Sample:  69%|######8   | 722/1050 [00:02, 320.89it/s, step size=1.11e+00, acc. rate=0.965]
Sample:  69%|######8   | 723/1050 [00:02, 323.73it/s, step size=1.11e+00, acc. rate=0.965]
Sample:  69%|######8   | 723/1050 [00:02, 323.73it/s, step size=1.11e+00, acc. rate=0.965]
Sample:  69%|######8   | 724/1050 [00:02, 323.73it/s, step size=1.11e+00, acc. rate=0.966]
Sample:  69%|######9   | 725/1050 [00:02, 323.73it/s, step size=1.11e+00, acc. rate=0.966]
Sample:  69%|######9   | 726/1050 [00:02, 323.73it/s, step size=1.11e+00, acc. rate=0.966]
Sample:  69%|######9   | 727/1050 [00:02, 323.73it/s, step size=1.11e+00, acc. rate=0.966]
Sample:  69%|######9   | 728/1050 [00:02, 323.73it/s, step size=1.11e+00, acc. rate=0.966]
Sample:  69%|######9   | 729/1050 [00:02, 323.73it/s, step size=1.11e+00, acc. rate=0.966]
Sample:  70%|######9   | 730/1050 [00:02, 323.73it/s, step size=1.11e+00, acc. rate=0.966]
Sample:  70%|######9   | 731/1050 [00:02, 323.73it/s, step size=1.11e+00, acc. rate=0.966]
Sample:  70%|######9   | 732/1050 [00:02, 323.73it/s, step size=1.11e+00, acc. rate=0.966]
Sample:  70%|######9   | 733/1050 [00:02, 323.73it/s, step size=1.11e+00, acc. rate=0.966]
Sample:  70%|######9   | 734/1050 [00:02, 323.73it/s, step size=1.11e+00, acc. rate=0.966]
Sample:  70%|#######   | 735/1050 [00:02, 323.73it/s, step size=1.11e+00, acc. rate=0.966]
Sample:  70%|#######   | 736/1050 [00:02, 323.73it/s, step size=1.11e+00, acc. rate=0.966]
Sample:  70%|#######   | 737/1050 [00:02, 323.73it/s, step size=1.11e+00, acc. rate=0.966]
Sample:  70%|#######   | 738/1050 [00:02, 323.73it/s, step size=1.11e+00, acc. rate=0.966]
Sample:  70%|#######   | 739/1050 [00:02, 323.73it/s, step size=1.11e+00, acc. rate=0.966]
Sample:  70%|#######   | 740/1050 [00:02, 323.73it/s, step size=1.11e+00, acc. rate=0.966]
Sample:  71%|#######   | 741/1050 [00:02, 323.73it/s, step size=1.11e+00, acc. rate=0.966]
Sample:  71%|#######   | 742/1050 [00:02, 323.73it/s, step size=1.11e+00, acc. rate=0.966]
Sample:  71%|#######   | 743/1050 [00:02, 323.73it/s, step size=1.11e+00, acc. rate=0.966]
Sample:  71%|#######   | 744/1050 [00:02, 323.73it/s, step size=1.11e+00, acc. rate=0.966]
Sample:  71%|#######   | 745/1050 [00:02, 323.73it/s, step size=1.11e+00, acc. rate=0.966]
Sample:  71%|#######1  | 746/1050 [00:02, 323.73it/s, step size=1.11e+00, acc. rate=0.967]
Sample:  71%|#######1  | 747/1050 [00:02, 323.73it/s, step size=1.11e+00, acc. rate=0.967]
Sample:  71%|#######1  | 748/1050 [00:02, 323.73it/s, step size=1.11e+00, acc. rate=0.967]
Sample:  71%|#######1  | 749/1050 [00:02, 323.73it/s, step size=1.11e+00, acc. rate=0.967]
Sample:  71%|#######1  | 750/1050 [00:02, 323.73it/s, step size=1.11e+00, acc. rate=0.967]
Sample:  72%|#######1  | 751/1050 [00:02, 323.73it/s, step size=1.11e+00, acc. rate=0.967]
Sample:  72%|#######1  | 752/1050 [00:02, 323.73it/s, step size=1.11e+00, acc. rate=0.967]
Sample:  72%|#######1  | 753/1050 [00:02, 323.73it/s, step size=1.11e+00, acc. rate=0.967]
Sample:  72%|#######1  | 754/1050 [00:02, 323.73it/s, step size=1.11e+00, acc. rate=0.967]
Sample:  72%|#######1  | 755/1050 [00:02, 323.73it/s, step size=1.11e+00, acc. rate=0.967]
Sample:  72%|#######2  | 756/1050 [00:02, 323.73it/s, step size=1.11e+00, acc. rate=0.967]
Sample:  72%|#######2  | 757/1050 [00:02, 323.73it/s, step size=1.11e+00, acc. rate=0.967]
Sample:  72%|#######2  | 758/1050 [00:02, 330.15it/s, step size=1.11e+00, acc. rate=0.967]
Sample:  72%|#######2  | 758/1050 [00:02, 330.15it/s, step size=1.11e+00, acc. rate=0.967]
Sample:  72%|#######2  | 759/1050 [00:02, 330.15it/s, step size=1.11e+00, acc. rate=0.967]
Sample:  72%|#######2  | 760/1050 [00:02, 330.15it/s, step size=1.11e+00, acc. rate=0.967]
Sample:  72%|#######2  | 761/1050 [00:02, 330.15it/s, step size=1.11e+00, acc. rate=0.967]
Sample:  73%|#######2  | 762/1050 [00:02, 330.15it/s, step size=1.11e+00, acc. rate=0.967]
Sample:  73%|#######2  | 763/1050 [00:02, 330.15it/s, step size=1.11e+00, acc. rate=0.967]
Sample:  73%|#######2  | 764/1050 [00:02, 330.15it/s, step size=1.11e+00, acc. rate=0.967]
Sample:  73%|#######2  | 765/1050 [00:02, 330.15it/s, step size=1.11e+00, acc. rate=0.967]
Sample:  73%|#######2  | 766/1050 [00:02, 330.15it/s, step size=1.11e+00, acc. rate=0.967]
Sample:  73%|#######3  | 767/1050 [00:02, 330.15it/s, step size=1.11e+00, acc. rate=0.967]
Sample:  73%|#######3  | 768/1050 [00:02, 330.15it/s, step size=1.11e+00, acc. rate=0.967]
Sample:  73%|#######3  | 769/1050 [00:02, 330.15it/s, step size=1.11e+00, acc. rate=0.968]
Sample:  73%|#######3  | 770/1050 [00:02, 330.15it/s, step size=1.11e+00, acc. rate=0.968]
Sample:  73%|#######3  | 771/1050 [00:02, 330.15it/s, step size=1.11e+00, acc. rate=0.968]
Sample:  74%|#######3  | 772/1050 [00:02, 330.15it/s, step size=1.11e+00, acc. rate=0.968]
Sample:  74%|#######3  | 773/1050 [00:02, 330.15it/s, step size=1.11e+00, acc. rate=0.968]
Sample:  74%|#######3  | 774/1050 [00:02, 330.15it/s, step size=1.11e+00, acc. rate=0.968]
Sample:  74%|#######3  | 775/1050 [00:02, 330.15it/s, step size=1.11e+00, acc. rate=0.968]
Sample:  74%|#######3  | 776/1050 [00:02, 330.15it/s, step size=1.11e+00, acc. rate=0.968]
Sample:  74%|#######4  | 777/1050 [00:02, 330.15it/s, step size=1.11e+00, acc. rate=0.968]
Sample:  74%|#######4  | 778/1050 [00:02, 330.15it/s, step size=1.11e+00, acc. rate=0.968]
Sample:  74%|#######4  | 779/1050 [00:02, 330.15it/s, step size=1.11e+00, acc. rate=0.968]
Sample:  74%|#######4  | 780/1050 [00:02, 330.15it/s, step size=1.11e+00, acc. rate=0.968]
Sample:  74%|#######4  | 781/1050 [00:02, 330.15it/s, step size=1.11e+00, acc. rate=0.968]
Sample:  74%|#######4  | 782/1050 [00:02, 330.15it/s, step size=1.11e+00, acc. rate=0.968]
Sample:  75%|#######4  | 783/1050 [00:02, 330.15it/s, step size=1.11e+00, acc. rate=0.968]
Sample:  75%|#######4  | 784/1050 [00:02, 330.15it/s, step size=1.11e+00, acc. rate=0.968]
Sample:  75%|#######4  | 785/1050 [00:02, 330.15it/s, step size=1.11e+00, acc. rate=0.968]
Sample:  75%|#######4  | 786/1050 [00:02, 330.15it/s, step size=1.11e+00, acc. rate=0.968]
Sample:  75%|#######4  | 787/1050 [00:02, 330.15it/s, step size=1.11e+00, acc. rate=0.968]
Sample:  75%|#######5  | 788/1050 [00:02, 330.15it/s, step size=1.11e+00, acc. rate=0.968]
Sample:  75%|#######5  | 789/1050 [00:02, 330.15it/s, step size=1.11e+00, acc. rate=0.968]
Sample:  75%|#######5  | 790/1050 [00:02, 330.15it/s, step size=1.11e+00, acc. rate=0.968]
Sample:  75%|#######5  | 791/1050 [00:02, 330.15it/s, step size=1.11e+00, acc. rate=0.968]
Sample:  75%|#######5  | 792/1050 [00:02, 290.48it/s, step size=1.11e+00, acc. rate=0.968]
Sample:  75%|#######5  | 792/1050 [00:02, 290.48it/s, step size=1.11e+00, acc. rate=0.968]
Sample:  76%|#######5  | 793/1050 [00:02, 290.48it/s, step size=1.11e+00, acc. rate=0.969]
Sample:  76%|#######5  | 794/1050 [00:02, 290.48it/s, step size=1.11e+00, acc. rate=0.969]
Sample:  76%|#######5  | 795/1050 [00:02, 290.48it/s, step size=1.11e+00, acc. rate=0.969]
Sample:  76%|#######5  | 796/1050 [00:02, 290.48it/s, step size=1.11e+00, acc. rate=0.969]
Sample:  76%|#######5  | 797/1050 [00:02, 290.48it/s, step size=1.11e+00, acc. rate=0.969]
Sample:  76%|#######6  | 798/1050 [00:02, 290.48it/s, step size=1.11e+00, acc. rate=0.969]
Sample:  76%|#######6  | 799/1050 [00:02, 290.48it/s, step size=1.11e+00, acc. rate=0.969]
Sample:  76%|#######6  | 800/1050 [00:02, 290.48it/s, step size=1.11e+00, acc. rate=0.969]
Sample:  76%|#######6  | 801/1050 [00:02, 290.48it/s, step size=1.11e+00, acc. rate=0.969]
Sample:  76%|#######6  | 802/1050 [00:02, 290.48it/s, step size=1.11e+00, acc. rate=0.969]
Sample:  76%|#######6  | 803/1050 [00:02, 290.48it/s, step size=1.11e+00, acc. rate=0.969]
Sample:  77%|#######6  | 804/1050 [00:02, 290.48it/s, step size=1.11e+00, acc. rate=0.969]
Sample:  77%|#######6  | 805/1050 [00:02, 290.48it/s, step size=1.11e+00, acc. rate=0.969]
Sample:  77%|#######6  | 806/1050 [00:02, 290.48it/s, step size=1.11e+00, acc. rate=0.969]
Sample:  77%|#######6  | 807/1050 [00:02, 290.48it/s, step size=1.11e+00, acc. rate=0.969]
Sample:  77%|#######6  | 808/1050 [00:02, 290.48it/s, step size=1.11e+00, acc. rate=0.969]
Sample:  77%|#######7  | 809/1050 [00:02, 290.48it/s, step size=1.11e+00, acc. rate=0.969]
Sample:  77%|#######7  | 810/1050 [00:02, 290.48it/s, step size=1.11e+00, acc. rate=0.969]
Sample:  77%|#######7  | 811/1050 [00:02, 290.48it/s, step size=1.11e+00, acc. rate=0.969]
Sample:  77%|#######7  | 812/1050 [00:02, 290.48it/s, step size=1.11e+00, acc. rate=0.969]
Sample:  77%|#######7  | 813/1050 [00:02, 290.48it/s, step size=1.11e+00, acc. rate=0.969]
Sample:  78%|#######7  | 814/1050 [00:02, 290.48it/s, step size=1.11e+00, acc. rate=0.969]
Sample:  78%|#######7  | 815/1050 [00:02, 290.48it/s, step size=1.11e+00, acc. rate=0.969]
Sample:  78%|#######7  | 816/1050 [00:02, 290.48it/s, step size=1.11e+00, acc. rate=0.969]
Sample:  78%|#######7  | 817/1050 [00:02, 290.48it/s, step size=1.11e+00, acc. rate=0.969]
Sample:  78%|#######7  | 818/1050 [00:02, 290.48it/s, step size=1.11e+00, acc. rate=0.969]
Sample:  78%|#######8  | 819/1050 [00:02, 290.48it/s, step size=1.11e+00, acc. rate=0.970]
Sample:  78%|#######8  | 820/1050 [00:02, 290.48it/s, step size=1.11e+00, acc. rate=0.970]
Sample:  78%|#######8  | 821/1050 [00:02, 290.48it/s, step size=1.11e+00, acc. rate=0.970]
Sample:  78%|#######8  | 822/1050 [00:02, 290.48it/s, step size=1.11e+00, acc. rate=0.970]
Sample:  78%|#######8  | 823/1050 [00:02, 290.48it/s, step size=1.11e+00, acc. rate=0.970]
Sample:  78%|#######8  | 824/1050 [00:02, 290.48it/s, step size=1.11e+00, acc. rate=0.970]
Sample:  79%|#######8  | 825/1050 [00:02, 290.48it/s, step size=1.11e+00, acc. rate=0.970]
Sample:  79%|#######8  | 826/1050 [00:02, 290.48it/s, step size=1.11e+00, acc. rate=0.970]
Sample:  79%|#######8  | 827/1050 [00:02, 290.48it/s, step size=1.11e+00, acc. rate=0.970]
Sample:  79%|#######8  | 828/1050 [00:02, 290.48it/s, step size=1.11e+00, acc. rate=0.970]
Sample:  79%|#######8  | 829/1050 [00:02, 308.63it/s, step size=1.11e+00, acc. rate=0.970]
Sample:  79%|#######8  | 829/1050 [00:02, 308.63it/s, step size=1.11e+00, acc. rate=0.970]
Sample:  79%|#######9  | 830/1050 [00:02, 308.63it/s, step size=1.11e+00, acc. rate=0.970]
Sample:  79%|#######9  | 831/1050 [00:02, 308.63it/s, step size=1.11e+00, acc. rate=0.970]
Sample:  79%|#######9  | 832/1050 [00:02, 308.63it/s, step size=1.11e+00, acc. rate=0.970]
Sample:  79%|#######9  | 833/1050 [00:02, 308.63it/s, step size=1.11e+00, acc. rate=0.970]
Sample:  79%|#######9  | 834/1050 [00:02, 308.63it/s, step size=1.11e+00, acc. rate=0.970]
Sample:  80%|#######9  | 835/1050 [00:02, 308.63it/s, step size=1.11e+00, acc. rate=0.970]
Sample:  80%|#######9  | 836/1050 [00:02, 308.63it/s, step size=1.11e+00, acc. rate=0.970]
Sample:  80%|#######9  | 837/1050 [00:02, 308.63it/s, step size=1.11e+00, acc. rate=0.970]
Sample:  80%|#######9  | 838/1050 [00:02, 308.63it/s, step size=1.11e+00, acc. rate=0.970]
Sample:  80%|#######9  | 839/1050 [00:02, 308.63it/s, step size=1.11e+00, acc. rate=0.970]
Sample:  80%|########  | 840/1050 [00:02, 308.63it/s, step size=1.11e+00, acc. rate=0.970]
Sample:  80%|########  | 841/1050 [00:02, 308.63it/s, step size=1.11e+00, acc. rate=0.970]
Sample:  80%|########  | 842/1050 [00:02, 308.63it/s, step size=1.11e+00, acc. rate=0.970]
Sample:  80%|########  | 843/1050 [00:02, 308.63it/s, step size=1.11e+00, acc. rate=0.970]
Sample:  80%|########  | 844/1050 [00:02, 308.63it/s, step size=1.11e+00, acc. rate=0.970]
Sample:  80%|########  | 845/1050 [00:02, 308.63it/s, step size=1.11e+00, acc. rate=0.970]
Sample:  81%|########  | 846/1050 [00:02, 308.63it/s, step size=1.11e+00, acc. rate=0.970]
Sample:  81%|########  | 847/1050 [00:02, 308.63it/s, step size=1.11e+00, acc. rate=0.971]
Sample:  81%|########  | 848/1050 [00:02, 308.63it/s, step size=1.11e+00, acc. rate=0.971]
Sample:  81%|########  | 849/1050 [00:02, 308.63it/s, step size=1.11e+00, acc. rate=0.971]
Sample:  81%|########  | 850/1050 [00:02, 308.63it/s, step size=1.11e+00, acc. rate=0.971]
Sample:  81%|########1 | 851/1050 [00:02, 308.63it/s, step size=1.11e+00, acc. rate=0.971]
Sample:  81%|########1 | 852/1050 [00:02, 308.63it/s, step size=1.11e+00, acc. rate=0.971]
Sample:  81%|########1 | 853/1050 [00:02, 308.63it/s, step size=1.11e+00, acc. rate=0.971]
Sample:  81%|########1 | 854/1050 [00:02, 308.63it/s, step size=1.11e+00, acc. rate=0.971]
Sample:  81%|########1 | 855/1050 [00:02, 308.63it/s, step size=1.11e+00, acc. rate=0.971]
Sample:  82%|########1 | 856/1050 [00:02, 308.63it/s, step size=1.11e+00, acc. rate=0.971]
Sample:  82%|########1 | 857/1050 [00:02, 308.63it/s, step size=1.11e+00, acc. rate=0.971]
Sample:  82%|########1 | 858/1050 [00:02, 308.63it/s, step size=1.11e+00, acc. rate=0.971]
Sample:  82%|########1 | 859/1050 [00:02, 308.63it/s, step size=1.11e+00, acc. rate=0.971]
Sample:  82%|########1 | 860/1050 [00:02, 308.63it/s, step size=1.11e+00, acc. rate=0.971]
Sample:  82%|########2 | 861/1050 [00:02, 308.63it/s, step size=1.11e+00, acc. rate=0.971]
Sample:  82%|########2 | 862/1050 [00:02, 308.63it/s, step size=1.11e+00, acc. rate=0.971]
Sample:  82%|########2 | 863/1050 [00:02, 316.40it/s, step size=1.11e+00, acc. rate=0.971]
Sample:  82%|########2 | 863/1050 [00:02, 316.40it/s, step size=1.11e+00, acc. rate=0.971]
Sample:  82%|########2 | 864/1050 [00:02, 316.40it/s, step size=1.11e+00, acc. rate=0.971]
Sample:  82%|########2 | 865/1050 [00:02, 316.40it/s, step size=1.11e+00, acc. rate=0.971]
Sample:  82%|########2 | 866/1050 [00:02, 316.40it/s, step size=1.11e+00, acc. rate=0.971]
Sample:  83%|########2 | 867/1050 [00:02, 316.40it/s, step size=1.11e+00, acc. rate=0.971]
Sample:  83%|########2 | 868/1050 [00:02, 316.40it/s, step size=1.11e+00, acc. rate=0.971]
Sample:  83%|########2 | 869/1050 [00:02, 316.40it/s, step size=1.11e+00, acc. rate=0.971]
Sample:  83%|########2 | 870/1050 [00:02, 316.40it/s, step size=1.11e+00, acc. rate=0.971]
Sample:  83%|########2 | 871/1050 [00:02, 316.40it/s, step size=1.11e+00, acc. rate=0.971]
Sample:  83%|########3 | 872/1050 [00:02, 316.40it/s, step size=1.11e+00, acc. rate=0.971]
Sample:  83%|########3 | 873/1050 [00:02, 316.40it/s, step size=1.11e+00, acc. rate=0.971]
Sample:  83%|########3 | 874/1050 [00:02, 316.40it/s, step size=1.11e+00, acc. rate=0.971]
Sample:  83%|########3 | 875/1050 [00:02, 316.40it/s, step size=1.11e+00, acc. rate=0.971]
Sample:  83%|########3 | 876/1050 [00:02, 316.40it/s, step size=1.11e+00, acc. rate=0.971]
Sample:  84%|########3 | 877/1050 [00:02, 316.40it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  84%|########3 | 878/1050 [00:02, 316.40it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  84%|########3 | 879/1050 [00:02, 316.40it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  84%|########3 | 880/1050 [00:02, 316.40it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  84%|########3 | 881/1050 [00:02, 316.40it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  84%|########4 | 882/1050 [00:02, 316.40it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  84%|########4 | 883/1050 [00:02, 316.40it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  84%|########4 | 884/1050 [00:02, 316.40it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  84%|########4 | 885/1050 [00:02, 316.40it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  84%|########4 | 886/1050 [00:02, 316.40it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  84%|########4 | 887/1050 [00:02, 316.40it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  85%|########4 | 888/1050 [00:02, 316.40it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  85%|########4 | 889/1050 [00:02, 316.40it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  85%|########4 | 890/1050 [00:02, 316.40it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  85%|########4 | 891/1050 [00:02, 316.40it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  85%|########4 | 892/1050 [00:02, 316.40it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  85%|########5 | 893/1050 [00:02, 316.40it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  85%|########5 | 894/1050 [00:02, 316.40it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  85%|########5 | 895/1050 [00:02, 316.40it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  85%|########5 | 896/1050 [00:02, 316.40it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  85%|########5 | 897/1050 [00:02, 316.40it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  86%|########5 | 898/1050 [00:02, 324.67it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  86%|########5 | 898/1050 [00:02, 324.67it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  86%|########5 | 899/1050 [00:02, 324.67it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  86%|########5 | 900/1050 [00:02, 324.67it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  86%|########5 | 901/1050 [00:02, 324.67it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  86%|########5 | 902/1050 [00:02, 324.67it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  86%|########6 | 903/1050 [00:02, 324.67it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  86%|########6 | 904/1050 [00:02, 324.67it/s, step size=1.11e+00, acc. rate=0.971]
Sample:  86%|########6 | 905/1050 [00:02, 324.67it/s, step size=1.11e+00, acc. rate=0.971]
Sample:  86%|########6 | 906/1050 [00:02, 324.67it/s, step size=1.11e+00, acc. rate=0.971]
Sample:  86%|########6 | 907/1050 [00:02, 324.67it/s, step size=1.11e+00, acc. rate=0.971]
Sample:  86%|########6 | 908/1050 [00:02, 324.67it/s, step size=1.11e+00, acc. rate=0.971]
Sample:  87%|########6 | 909/1050 [00:03, 324.67it/s, step size=1.11e+00, acc. rate=0.971]
Sample:  87%|########6 | 910/1050 [00:03, 324.67it/s, step size=1.11e+00, acc. rate=0.971]
Sample:  87%|########6 | 911/1050 [00:03, 324.67it/s, step size=1.11e+00, acc. rate=0.971]
Sample:  87%|########6 | 912/1050 [00:03, 324.67it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  87%|########6 | 913/1050 [00:03, 324.67it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  87%|########7 | 914/1050 [00:03, 324.67it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  87%|########7 | 915/1050 [00:03, 324.67it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  87%|########7 | 916/1050 [00:03, 324.67it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  87%|########7 | 917/1050 [00:03, 324.67it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  87%|########7 | 918/1050 [00:03, 324.67it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  88%|########7 | 919/1050 [00:03, 324.67it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  88%|########7 | 920/1050 [00:03, 324.67it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  88%|########7 | 921/1050 [00:03, 324.67it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  88%|########7 | 922/1050 [00:03, 324.67it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  88%|########7 | 923/1050 [00:03, 324.67it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  88%|########8 | 924/1050 [00:03, 324.67it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  88%|########8 | 925/1050 [00:03, 324.67it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  88%|########8 | 926/1050 [00:03, 324.67it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  88%|########8 | 927/1050 [00:03, 324.67it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  88%|########8 | 928/1050 [00:03, 324.67it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  88%|########8 | 929/1050 [00:03, 324.67it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  89%|########8 | 930/1050 [00:03, 324.67it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  89%|########8 | 931/1050 [00:03, 324.67it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  89%|########8 | 932/1050 [00:03, 324.67it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  89%|########8 | 933/1050 [00:03, 331.43it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  89%|########8 | 933/1050 [00:03, 331.43it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  89%|########8 | 934/1050 [00:03, 331.43it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  89%|########9 | 935/1050 [00:03, 331.43it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  89%|########9 | 936/1050 [00:03, 331.43it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  89%|########9 | 937/1050 [00:03, 331.43it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  89%|########9 | 938/1050 [00:03, 331.43it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  89%|########9 | 939/1050 [00:03, 331.43it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  90%|########9 | 940/1050 [00:03, 331.43it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  90%|########9 | 941/1050 [00:03, 331.43it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  90%|########9 | 942/1050 [00:03, 331.43it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  90%|########9 | 943/1050 [00:03, 331.43it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  90%|########9 | 944/1050 [00:03, 331.43it/s, step size=1.11e+00, acc. rate=0.972]
Sample:  90%|######### | 945/1050 [00:03, 331.43it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  90%|######### | 946/1050 [00:03, 331.43it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  90%|######### | 947/1050 [00:03, 331.43it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  90%|######### | 948/1050 [00:03, 331.43it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  90%|######### | 949/1050 [00:03, 331.43it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  90%|######### | 950/1050 [00:03, 331.43it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  91%|######### | 951/1050 [00:03, 331.43it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  91%|######### | 952/1050 [00:03, 331.43it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  91%|######### | 953/1050 [00:03, 331.43it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  91%|######### | 954/1050 [00:03, 331.43it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  91%|######### | 955/1050 [00:03, 331.43it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  91%|#########1| 956/1050 [00:03, 331.43it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  91%|#########1| 957/1050 [00:03, 331.43it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  91%|#########1| 958/1050 [00:03, 331.43it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  91%|#########1| 959/1050 [00:03, 331.43it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  91%|#########1| 960/1050 [00:03, 331.43it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  92%|#########1| 961/1050 [00:03, 331.43it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  92%|#########1| 962/1050 [00:03, 331.43it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  92%|#########1| 963/1050 [00:03, 331.43it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  92%|#########1| 964/1050 [00:03, 331.43it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  92%|#########1| 965/1050 [00:03, 331.43it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  92%|#########2| 966/1050 [00:03, 331.43it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  92%|#########2| 967/1050 [00:03, 331.43it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  92%|#########2| 968/1050 [00:03, 331.43it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  92%|#########2| 969/1050 [00:03, 338.93it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  92%|#########2| 969/1050 [00:03, 338.93it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  92%|#########2| 970/1050 [00:03, 338.93it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  92%|#########2| 971/1050 [00:03, 338.93it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  93%|#########2| 972/1050 [00:03, 338.93it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  93%|#########2| 973/1050 [00:03, 338.93it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  93%|#########2| 974/1050 [00:03, 338.93it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  93%|#########2| 975/1050 [00:03, 338.93it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  93%|#########2| 976/1050 [00:03, 338.93it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  93%|#########3| 977/1050 [00:03, 338.93it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  93%|#########3| 978/1050 [00:03, 338.93it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  93%|#########3| 979/1050 [00:03, 338.93it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  93%|#########3| 980/1050 [00:03, 338.93it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  93%|#########3| 981/1050 [00:03, 338.93it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  94%|#########3| 982/1050 [00:03, 338.93it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  94%|#########3| 983/1050 [00:03, 338.93it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  94%|#########3| 984/1050 [00:03, 338.93it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  94%|#########3| 985/1050 [00:03, 338.93it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  94%|#########3| 986/1050 [00:03, 338.93it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  94%|#########3| 987/1050 [00:03, 338.93it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  94%|#########4| 988/1050 [00:03, 338.93it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  94%|#########4| 989/1050 [00:03, 338.93it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  94%|#########4| 990/1050 [00:03, 338.93it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  94%|#########4| 991/1050 [00:03, 338.93it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  94%|#########4| 992/1050 [00:03, 338.93it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  95%|#########4| 993/1050 [00:03, 338.93it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  95%|#########4| 994/1050 [00:03, 338.93it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  95%|#########4| 995/1050 [00:03, 338.93it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  95%|#########4| 996/1050 [00:03, 338.93it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  95%|#########4| 997/1050 [00:03, 338.93it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  95%|#########5| 998/1050 [00:03, 338.93it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  95%|#########5| 999/1050 [00:03, 338.93it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  95%|#########5| 1000/1050 [00:03, 338.93it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  95%|#########5| 1001/1050 [00:03, 338.93it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  95%|#########5| 1002/1050 [00:03, 338.93it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  96%|#########5| 1003/1050 [00:03, 338.93it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  96%|#########5| 1004/1050 [00:03, 339.08it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  96%|#########5| 1004/1050 [00:03, 339.08it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  96%|#########5| 1005/1050 [00:03, 339.08it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  96%|#########5| 1006/1050 [00:03, 339.08it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  96%|#########5| 1007/1050 [00:03, 339.08it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  96%|#########6| 1008/1050 [00:03, 339.08it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  96%|#########6| 1009/1050 [00:03, 339.08it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  96%|#########6| 1010/1050 [00:03, 339.08it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  96%|#########6| 1011/1050 [00:03, 339.08it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  96%|#########6| 1012/1050 [00:03, 339.08it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  96%|#########6| 1013/1050 [00:03, 339.08it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  97%|#########6| 1014/1050 [00:03, 339.08it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  97%|#########6| 1015/1050 [00:03, 339.08it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  97%|#########6| 1016/1050 [00:03, 339.08it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  97%|#########6| 1017/1050 [00:03, 339.08it/s, step size=1.11e+00, acc. rate=0.973]
Sample:  97%|#########6| 1018/1050 [00:03, 339.08it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  97%|#########7| 1019/1050 [00:03, 339.08it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  97%|#########7| 1020/1050 [00:03, 339.08it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  97%|#########7| 1021/1050 [00:03, 339.08it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  97%|#########7| 1022/1050 [00:03, 339.08it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  97%|#########7| 1023/1050 [00:03, 339.08it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  98%|#########7| 1024/1050 [00:03, 339.08it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  98%|#########7| 1025/1050 [00:03, 339.08it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  98%|#########7| 1026/1050 [00:03, 339.08it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  98%|#########7| 1027/1050 [00:03, 339.08it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  98%|#########7| 1028/1050 [00:03, 339.08it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  98%|#########8| 1029/1050 [00:03, 339.08it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  98%|#########8| 1030/1050 [00:03, 339.08it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  98%|#########8| 1031/1050 [00:03, 339.08it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  98%|#########8| 1032/1050 [00:03, 339.08it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  98%|#########8| 1033/1050 [00:03, 339.08it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  98%|#########8| 1034/1050 [00:03, 339.08it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  99%|#########8| 1035/1050 [00:03, 339.08it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  99%|#########8| 1036/1050 [00:03, 339.08it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  99%|#########8| 1037/1050 [00:03, 339.08it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  99%|#########8| 1038/1050 [00:03, 339.08it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  99%|#########8| 1039/1050 [00:03, 337.51it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  99%|#########8| 1039/1050 [00:03, 337.51it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  99%|#########9| 1040/1050 [00:03, 337.51it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  99%|#########9| 1041/1050 [00:03, 337.51it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  99%|#########9| 1042/1050 [00:03, 337.51it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  99%|#########9| 1043/1050 [00:03, 337.51it/s, step size=1.11e+00, acc. rate=0.974]
Sample:  99%|#########9| 1044/1050 [00:03, 337.51it/s, step size=1.11e+00, acc. rate=0.974]
Sample: 100%|#########9| 1045/1050 [00:03, 337.51it/s, step size=1.11e+00, acc. rate=0.974]
Sample: 100%|#########9| 1046/1050 [00:03, 337.51it/s, step size=1.11e+00, acc. rate=0.974]
Sample: 100%|#########9| 1047/1050 [00:03, 337.51it/s, step size=1.11e+00, acc. rate=0.974]
Sample: 100%|#########9| 1048/1050 [00:03, 337.51it/s, step size=1.11e+00, acc. rate=0.974]
Sample: 100%|#########9| 1049/1050 [00:03, 337.51it/s, step size=1.11e+00, acc. rate=0.974]
Sample: 100%|##########| 1050/1050 [00:03, 307.59it/s, step size=1.11e+00, acc. rate=0.974]

marginal = EmpiricalMarginal(posterior, "weight")
plt.hist([marginal().item() for _ in range(1000)],)
plt.title("P(weight | measurement = 14)")
plt.xlabel("Weight")
plt.ylabel("#")

4.3.0.1 Shapes in distribution:

We know that PyTorch tensor have single shape attribute, Distributions have two shape attributes with special meaning. * .batch_shape: Indices over .batch_shape denote conditionally independent random variables, * .event_shape: indices over .event_shape denote dependent random variables (ie one draw from a distribution).

These two combine to define the total shape of a sample. Thus the total shape of .log_prob() of distribution is .batch_shape.

Also, Distribution.sample() also has a sample_shape attribute that indexes over independent and identically distributed(iid) random variables.

      |      iid     | independent | dependent
------+--------------+-------------+------------
shape = sample_shape + batch_shape + event_shape

To know more about + , go through broadcasting tensors in PyTorch.

4.3.1 Examples

One way to introduce batch_shape is use expand.

d = dist.MultivariateNormal(torch.zeros(3), torch.eye(3, 3)).expand([5]) # expand - 3 of these Multivariate Normal Dists
print("batch_shape: ", d.batch_shape)

## batch_shape:  torch.Size([5])

print("event_shape: ", d.event_shape)
#x = d.sample(torch.Size([5]))

## event_shape:  torch.Size([3])

x = d.sample()
print("x shape: ", x.shape)          # == sample_shape + batch_shape + event_shape

## x shape:  torch.Size([5, 3])

print("d.log_prob(x) shape:", d.log_prob(x).shape)  # == batch_shape

## d.log_prob(x) shape: torch.Size([5])

The other way is using plate context manager.

Pyro models can use the context manager pyro.plate to declare that certain batch dimensions are independent. Inference algorithms can then take advantage of this independence to e.g. construct lower variance gradient estimators or to enumerate in linear space rather than exponential space.

with pyro.plate("x_axis", 5):
    d = dist.MultivariateNormal(torch.zeros(3), torch.eye(3, 3))
    x = pyro.sample("x", d)

x.shape

In fact, we can also nest plates. The only thing we need to care about is, which dimensions are independent. Pyro automatically manages this but sometimes we need to explicitely specify the dimensions. Once we specify that, we can leverage PyTorch’s CUDA enabled capabilities to run inference on GPUs.

with pyro.plate("x_axis", 320):
    # within this context, batch dimension -1 is independent
    with pyro.plate("y_axis", 200):
        # within this context, batch dimensions -2 and -1 are independent

Note that we always count from the right by using negative indices like \(-2\), \(-1\).

4.3.2 Gaussian Mixture Model

\[\texttt{Blei - Build, Compute, Critique, Repeat:Data Analysis with Latent Variable Models}\]

from __future__ import print_function
import os
from collections import defaultdict
import numpy as np
import scipy.stats
import torch
from torch.distributions import constraints
from pyro import poutine
from pyro.contrib.autoguide import AutoDelta
from pyro.optim import Adam
from pyro.infer import SVI, TraceEnum_ELBO, config_enumerate, infer_discrete
from matplotlib import pyplot
# %matplotlib inline
pyro.enable_validation(True)
data = torch.tensor([0., 1., 10., 11., 12.])
K = 2  # Fixed number of components.
@config_enumerate
def model(data):
    # Global variables.
    weights = pyro.sample('weights', dist.Dirichlet(0.5 * torch.ones(K)))
    scale = pyro.sample('scale', dist.LogNormal(0., 2.))
    with pyro.plate('components', K):
        locs = pyro.sample('locs', dist.Normal(0., 10.))
    with pyro.plate('data', len(data)):
        # Local variables.
        assignment = pyro.sample('assignment', dist.Categorical(weights))
        pyro.sample('obs', dist.Normal(locs[assignment], scale), obs=data)

4.3.3 Review of Approximate Inference

We have variables \(Z\)s (cluster assignments) and \(X\)s (data points) in our mixture model, where \(X\) is observed and \(Z\) is latent (unobserved). As we saw earlier, a generative model entails a joint distribution \[p(Z,X)\]

Inference of unknown can be achieved through conditioning on the observations.

\[p(Z \mid X) = \frac{p(Z, X)}{p(X)}\]

And for the most interesting problems, the integral for the denominator(marginal) is not tractable. \[p(X) = \int dZp(X \mid Z)p(Z)\]

So we have to directly approximate \(p(Z \mid X)\). There are two ways of approximate this posterior.

Sampling methods like Gibbs sampler.
Variational inference.

4.3.4 Variational Inference:

We can’t compute \(p(Z \mid X)\) directly, so let’s approximate with some other distribution \(q(Z; \nu)\) over Z that is tractable (for example, Gaussions or other exponential family).

Image

\[\texttt{David Blei - Variational Inference (NeurIPS 2016 Tutorial)}\]

Since q is tractable, we can play with it’s parameter \(\nu\) such that it reaches as close to \(p(Z\mid X)\) as possible. More precisely, we want to minimize the KL divergence between \(q\) and \(p\). With this trick, we just turned an inference problem to an optimization problem!

\[ \begin{align*} KL(q(Z;\nu) \mid\mid p(Z\mid X)) &= -\int dZ\ q(Z) \log\frac{P(Z\mid X)}{q(Z)}\\ &= -\int dZ\ q(Z) \log \frac{\frac{p(Z,X)}{p(X)}}{q(Z)}\\ &= -\int dZ\ q(Z) \log \frac{p(Z,X)}{p(X)q(Z)}\\ &= -\int dZ\ q(Z) \left[ \log \frac{p(Z,X)}{q(Z)} - \log p(X) \right]\\ &= - \int dZ\ \log \frac{p(Z,X)}{q(Z)} + \underbrace{\int dZ\ q(Z)}_{\text{=1}}\log p(X)\\ &= - \int dZ\ \log \frac{p(Z,X)}{q(Z)} + \log p(X)\\ \log p(X) &= KL(q(Z;\nu)\mid\mid p(Z\mid X) + \underbrace{\int dZ\ q(Z;\nu) \log \frac{p(Z,X)}{q(Z;\nu)}}_{\mathcal{L}}\\ \end{align*} \]

Note that we already observed \(X\) and we conditioned the model to get \(p(Z \mid X)\). But given \(X\), \(\log p(X)\) is constant! So, minimizing KL is equivalent to maximizing \(\mathcal{L}\).

How do you maximize \(\mathcal{L}\)? Take \(\nabla_{\nu} \mathcal{L}\).

\(\mathcal{L}\) is called variational lower bound. It is often called ELBO.

Stochastic Variational Inference scales variational inference to massive data. Just like in stochastic variational inference, you subsample the data and update the posterior!

4.3.5 Stochastic Optimization

In stochastic optimization, we replace the gradient with cheaper noisy estimate which is guranteed to converge to a local optimum.

\[\nu_{t+1} = \nu_t + \rho_t \hat{\nabla}_{\nu} \mathcal{L}(\nu_t)\]

Requirements:

Unbiased gradients, i.e. \[\mathbb{E}[\hat{\nabla}_{\nu} \mathcal{L}(\nu_t)] = \nabla_{\nu}\mathcal{L}(\nu)\]
Step-size sequence \(\rho_t\) that follows Robbins-Monro conditions.

Stochastic variational inference takes inspiration from stochastic optimization and natural graidient. We follow the same procedure as stochastic gradient descent.

4.3.6 A Rough Stochastic variational inference algorithm:

Initialize \(q\) with some \(\nu\)
Until Converge:
- subsample from Data:
  - compute gradient \(\hat{\nabla_{\nu}}\mathcal{L}_{\nu_t}\)
  - update global parameter \(\nu_{t+1} = \nu_t + \rho_t \hat{\nabla_{\nu}}\mathcal{L}_{\nu_t}\)
Return \(q(Z;\nu)\)

4.3.7 Training a MAP estimator

Let’s start by learning model parameters weights, locs, and scale given priors and data. We will use AutoDelta guide function. Our model will learn global mixture weights, the location of each mixture component, and a shared scale that is common to both components.

During inference, TraceEnum_ELBO will marginalize out the assignments of datapoints to clusters.

max_plate_nesting lets Pyro know that we’re using the rightmost dimension plate and that Pyro can use any other dimension for parallelization.

4.4 Some other Pyro vocabulary

poutine - Beneath the built-in inference algorithms, Pyro has a library of composable effect handlers for creating new inference algorithms and working with probabilistic programs. Pyro’s inference algorithms are all built by applying these handlers to stochastic functions.
- poutine.block - blocks pyro premitives. By default, it blocks everything.
param - Parameters in Pyro are basically thin wrappers around PyTorch Tensors that carry unique names. As such Parameters are the primary stateful objects in Pyro. Users typically interact with parameters via the Pyro primitive pyro.param. Parameters play a central role in stochastic variational inference, where they are used to represent point estimates for the parameters in parameterized families of models and guides.
param_store - Global store for parameters in Pyro. This is basically a key-value store.

global_guide = AutoDelta(poutine.block(model, expose=['weights', 'locs', 'scale']))
optim = pyro.optim.Adam({'lr': 0.1, 'betas': [0.8, 0.99]})
elbo = TraceEnum_ELBO(max_plate_nesting=1)
svi = SVI(model, global_guide, optim, loss=elbo)

def initialize(seed):
    pyro.set_rng_seed(seed)
    pyro.clear_param_store()
    # Initialize weights to uniform.
    pyro.param('auto_weights', 0.5 * torch.ones(K), constraint=constraints.simplex)
    # Assume half of the data variance is due to intra-component noise.
    pyro.param('auto_scale', (data.var() / 2).sqrt(), constraint=constraints.positive)
    # Initialize means from a subsample of data.
    pyro.param('auto_locs', data[torch.multinomial(torch.ones(len(data)) / len(data), K)]);
    loss = svi.loss(model, global_guide, data)
    return loss
# Choose the best among 100 random initializations.
loss, seed = min((initialize(seed), seed) for seed in range(100))
initialize(seed)
print('seed = {}, initial_loss = {}'.format(seed, loss))

## seed = 7, initial_loss = 25.665584564208984

# Register hooks to monitor gradient norms.
gradient_norms = defaultdict(list)
for name, value in pyro.get_param_store().named_parameters():
    value.register_hook(lambda g, name=name: gradient_norms[name].append(g.norm().item()))
losses = []
for i in range(200):
    loss = svi.step(data)
    losses.append(loss)
    print('.' if i % 100 else '\n', end='')

## 
## ...................................................................................................
## ...................................................................................................

pyplot.figure(figsize=(10,3), dpi=100).set_facecolor('white')
pyplot.plot(losses)
pyplot.xlabel('iters')
pyplot.ylabel('loss')
pyplot.yscale('log')
pyplot.title('Convergence of SVI');

map_estimates = global_guide(data)
weights = map_estimates['weights']
locs = map_estimates['locs']
scale = map_estimates['scale']
print('weights = {}'.format(weights.data.numpy()))

## weights = [0.375 0.625]

print('locs = {}'.format(locs.data.numpy()))

## locs = [ 0.49887404 10.984463  ]

print('scale = {}'.format(scale.data.numpy()))

## scale = 0.6514337062835693

X = np.arange(-3,15,0.1)
Y1 = weights[0].item() * scipy.stats.norm.pdf((X - locs[0].item()) / scale.item())
Y2 = weights[1].item() * scipy.stats.norm.pdf((X - locs[1].item()) / scale.item())
pyplot.figure(figsize=(10, 4), dpi=100).set_facecolor('white')
pyplot.plot(X, Y1, 'r-')
pyplot.plot(X, Y2, 'b-')
pyplot.plot(X, Y1 + Y2, 'k--')
pyplot.plot(data.data.numpy(), np.zeros(len(data)), 'k*')
pyplot.title('Density of two-component mixture model')
pyplot.ylabel('probability density');