# Using the sampler¶

`spvcm` is a generic gibbs sampling framework for spatially-correlated variance components models. The current supported models are:

- `spvcm.both` contains specifications with correlated errors in both levels, with the first statement `se/sma` describing the lower level and the second statement `se/sma` describing the upper level. In addition, `MVCM`, the multilevel variance components model with no spatial correlation, is in the `both` namespace.
- `spvcm.lower` contains two specifications, `se/sma`, that can be used for a variance components model with correlated lower-level errors.
- `spvcm.upper` contains two specifications, `se/sma` that can be used for a variance components model with correlated upper-level errors.

### Specification¶

These derive from a variance components specification:

$$ Y \sim \mathcal{N}(X\beta, \Psi_1(\lambda, \sigma^2) + \Delta\Psi_2(\rho, \tau^2)\Delta') $$

Where:

1. $\beta$, called `Betas` in code, is the marginal effect parameter. In this implementation, any region-level covariates $Z$ get appended to the end of $X$. So, if $X$ is $n \times p$ ($n$ observations of $p$ covariates) and $Z$ is $J \times p'$ ($p'$ covariates observed for $J$ regions), then the model's $X$ matrix is $n \times (p + p')$ and $\beta$ is $p + p' \times 1$.
2. $\Psi_1$ is the covariance function for the response-level model. In the software, a separable covariance is assumed, so that $\Psi_1(\rho, \sigma^2) = \Psi_1(\rho) * I \sigma^2)$, where $I$ is the $n \times n$ covariance matrix. Thus, $\rho$ is the spatial autoregressive parameter and $\sigma^2$ is the variance parameter. In the software, $\Psi_1$ takes any of the following forms:
   - Spatial Error (`SE`): $\Psi_1(\rho) = [(I - \rho \mathbf{W})'(I - \rho \mathbf{W})]^{-1} \sigma^2$
   - Spatial Moving Average (`SMA`): $\Psi_1(\rho) = (I + \rho \mathbf{W})(I + \lambda \mathbf{W})'$
   - Identity: $\Psi_1(\rho) = I$
3. $\Psi_2$ is the region-level covariance function, with region-level autoregressive parameter $\lambda$ and region-level variance $\tau^2$. It has the same potential forms as $\Psi_1$.
4. $\alpha$, called `Alphas` in code, is the region-level random effect. In a variance components model, this is interpreted as a random effect for the upper-level. For a Varying-intercept format, this random component should be added to a region-level fixed effect to provide the varying intercept. This may also make it more difficult to identify the spatial parameter.

## Softare implementation¶

All of the possible combinations of Spatial Moving Average and Spatial Error processes are contained in the following classes. I will walk through estimating one below, and talk about the various features of the package.

First, the API of the package is defined by the `spvcm.api` submodule. To load it, use `import spvcm.api as spvcm`:

In [1]:

    import spvcm.api as spvcm #package API
    spvcm.both.Generic # abstract customizable class, ignores rho/lambda, equivalent to MVCM
    spvcm.both.MVCM # no spatial effect
    spvcm.both.SESE #  both spatial error (SE)
    spvcm.both.SESMA # response-level SE, region-level spatial moving average
    spvcm.both.SMASE # response-level SMA, region-level SE
    spvcm.both.SMASMA # both levels SMA
    spvcm.upper.SE # response-level uncorrelated, region-level SE
    spvcm.upper.SMA # response-level uncorrelated, region-level SMA
    spvcm.lower.SE # response-level SE, region-level uncorrelated
    spvcm.lower.SMA # response-level SMA, region-level uncorrelated

Out[1]:

    spvcm.lower_level.sma.model.Lower_SMA

Depending on the structure of the model, you need at least:

- `X`, data at the response (lower) level
- `Y`, system response in the lower level
- `membership` or `Delta`, the membership vector relating each observation to its group or the "dummy variable" matrix encoding the same information.

Then, if spatial correlation is desired, `M` is the "upper-level" weights matrix and `W` the lower-level weights matrix.

Any upper-level data should be passed in $Z$, and have $J$ rows. To fit a varying-intercept model, include an identity matrix in $Z$. You can include state-level and response-level intercept terms simultaneously.

Finally, there are many configuration and tuning options that can be passed in at the start, or assigned after the model is initialized.

First, though, let's set up some data for a model on southern counties predicting `HR90`, the Homicide Rate in the US South in 1990, using the the percent of the labor force that is unemployed (`UE90`), a principal component expressing the population structure (`PS90`), and a principal component expressing resource deprivation.

We will also use the state-level average percentage of families below the poverty line and the average Gini coefficient at the state level for a $Z$ variable.

In [2]:

    #seaborn is required for the traceplots
    import pysal as ps
    import numpy as np
    import pandas as pd
    import seaborn as sns
    import matplotlib.pyplot as plt
    %matplotlib inline

Reading in the data, we'll extract these values we need from the dataframe.

In [3]:

    data = ps.pdio.read_files(ps.examples.get_path('south.shp'))
    data = data[data.STATE_NAME != 'District of Columbia']
    X = data[['UE90', 'PS90', 'RD90']].values
    N = X.shape[0]
    Z = data.groupby('STATE_NAME')[['FP89', 'GI89']].mean().values
    J = Z.shape[0]

    Y = data.HR90.values.reshape(-1,1)

Then, we'll construct some queen contiguity weights from the files to show how to run a model.

In [4]:

    W2 = ps.queen_from_shapefile(ps.examples.get_path('us48.shp'),
                                 idVariable='STATE_NAME')
    W2 = ps.w_subset(W2, ids=data.STATE_NAME.unique().tolist()) #only keep what's in the data
    W1 = ps.queen_from_shapefile(ps.examples.get_path('south.shp'),
                                 idVariable='FIPS')
    W1 = ps.w_subset(W1, ids=data.FIPS.tolist()) #again, only keep what's in the data

    W1.transform = 'r'
    W2.transform = 'r'

With the data, upper-level weights, and lower-level weights, we can construct a membership vector or a dummy data matrix. For now, I'll create the membership vector.

In [5]:

    membership = data.STATE_NAME.apply(lambda x: W2.id_order.index(x)).values

But, we could also build the dummy variable matrix using `pandas`, if we have a suitable categorical variable:

In [6]:

    Delta_frame = pd.get_dummies(data.STATE_NAME)
    Delta = Delta_frame.values

Every call to the sampler is of the following form:

`sampler(Y, X, W, M, Z, membership, Delta, n_samples, **configuration)`

Where `W`, `M` are passed if appropriate, `Z` is passed if used, and only one of `membership` or `Delta` is required. In the end, `Z` is appended to `X`, so the effects pertaining to the upper level will be at the tail end of the $\beta$ effects vector. If both `Delta` and `membership` are supplied, they're verified against each other to ensure that they agree before they are used in the model.

For all models, the membership vector or an equivalent dummy variable matrix is required. For models with correlation in the upper level, only the upper-level weights matrix $\mathbf{M}$ is needed. For lower level models, the lower-level weights matrix $\mathbf{W}$ is required. For models with correlation in both levels, both $\mathbf{W}$ and $\mathbf{M}$ are required.

Every sampler uses, either in whole or in part, `spvcm.both.generic`, which implements the full generic sampler discussed in the working paper. For efficiency, the upper-level samplers modify this runtime to avoid processing the full lower-level covariance matrix.

Like many of the `R` packages dedicated to bayesian models, configuration occurs by passing the correct dictionary to the model call. In addition, you can "setup" the model, configure it, and then run samples in separate steps.

The most common way to call the sampler is something like:

In [7]:

    vcsma = spvcm.upper.SMA(Y, X, M=W2, Z=Z, membership=membership,
                          n_samples=5000,
                          configs=dict(tuning=1000, adapt_step=1.01))

This models, `spvcm.upper.SMA`, is a variance components/varying intercept model with a state-level SMA-correlated error.

Thus, there are only five parameters in this model, since $\rho$, the lower-level autoregressive parameter, is constrained to zero:

In [8]:

    vcsma.trace.varnames

Out[8]:

    ['Lambda', 'Betas', 'Sigma2', 'Alphas', 'Tau2']

The results and state of the sampler are stored within the `vcsma` object. I'll step through the most important parts of this object.

# `trace`¶

The quickest way to get information out of the model is via the trace object. This is where the results of the tracked parameters are stored each iteration. Any variable in the sampler state can be added to the tracked params. Trace objects are essentially dictionaries with the keys being the name of the tracked parameter and the values being a list of each iteration's sampler output.

In [9]:

    vcsma.trace.varnames

Out[9]:

    ['Lambda', 'Betas', 'Sigma2', 'Alphas', 'Tau2']

In this case, `Lambda` is the upper-level moving average parameter, `Alphas` is the vector of correlated group-level random effects, `Tau2` is the upper-level variance, `Betas` are the marginal effects, and `Sigma2` is the lower-level error variance.

I've written two helper functions for working with traces. First is to just dump all the output into a pandas dataframe, which makes it super easy to do work on the samples, or write them out to `csv` and assess convergence in R's `coda` package.

In [10]:

    trace_dataframe = vcsma.trace.to_df()

the dataframe will have columns containing the elements of the parameters and each row is a single iteration of the sampler:

In [11]:

    trace_dataframe.head()

Out[11]:

| Lambda | Sigma2 | Tau2 | Betas_0 | Betas_1 | Betas_2 | Betas_3 | Betas_4 | Betas_5 | Alphas_0 | ... | Alphas_6 | Alphas_7 | Alphas_8 | Alphas_9 | Alphas_10 | Alphas_11 | Alphas_12 | Alphas_13 | Alphas_14 | Alphas_15
--- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---
0 | 0.940050 | 34.046017 | 1.334209 | 7.032362 | -0.502780 | 2.010436 | 4.469775 | -0.028225 | 9.839930 | -1.288980 | ... | -0.583933 | 0.924241 | 0.087917 | 0.051871 | 0.856965 | -0.090386 | 0.495369 | 1.762157 | -0.796564 | 2.272783
1 | 0.940050 | 34.215037 | 1.587374 | 5.277195 | -0.286292 | 1.834136 | 3.738277 | -0.100795 | 14.294029 | -1.221123 | ... | -1.449357 | 1.559842 | -0.018327 | 0.043533 | 1.671848 | -0.134705 | 0.897505 | 1.570762 | -0.016576 | 1.841339
2 | 0.940050 | 33.171990 | 2.780147 | 6.501418 | -0.196233 | 2.006755 | 4.027759 | -0.005111 | 5.549450 | -2.063556 | ... | -1.361113 | 2.283653 | -0.092895 | -0.374784 | 1.027415 | 0.093002 | -0.463750 | 1.537222 | 0.463958 | 1.777161
3 | 0.853473 | 31.246997 | 1.680753 | 13.216350 | -0.251791 | 1.859299 | 3.977210 | 0.119079 | -15.384568 | -3.300783 | ... | -1.440598 | 1.070437 | -1.102695 | -1.098938 | 0.954793 | -1.346576 | -1.292709 | 1.483239 | -0.950301 | 1.109188
4 | 0.853473 | 32.761820 | 3.508263 | 4.432656 | -0.215052 | 1.626291 | 4.020410 | -0.020324 | 13.994845 | -2.571416 | ... | -1.216185 | 1.413811 | -1.069306 | -1.187765 | -0.131962 | -0.378420 | -1.337283 | 1.369119 | -2.404278 | 2.538615

5 rows × 25 columns

You can write this out to a csv or analyze it in memory like a typical pandas dataframes:

In [12]:

    trace_dataframe.mean()

Out[12]:

    Lambda        0.608686
    Sigma2       32.265190
    Tau2          1.828138
    Betas_0       8.156127
    Betas_1      -0.258890
    Betas_2       1.796985
    Betas_3       3.973051
    Betas_4      -0.003564
    Betas_5       2.108436
    Alphas_0     -1.588250
    Alphas_1     -0.124204
    Alphas_2     -0.086578
    Alphas_3      0.059299
    Alphas_4     -2.261698
    Alphas_5     -0.931411
    Alphas_6     -0.400545
    Alphas_7      1.117568
    Alphas_8      0.102000
    Alphas_9     -0.169745
    Alphas_10     1.024000
    Alphas_11     0.308998
    Alphas_12     0.189192
    Alphas_13     1.873476
    Alphas_14    -0.617676
    Alphas_15     2.084861
    dtype: float64

The second is a method to plot the traces:

In [13]:

    fig, ax = vcsma.trace.plot()
    plt.show()

    /home/ljw/anaconda3/envs/py3/lib/python3.5/site-packages/statsmodels/nonparametric/kdetools.py:20: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
      y = X[:m/2+1] + np.r_[0,X[m/2+1:],0]*1j

The trace object can be sliced by (chain, parameter, index) tuples, or any subset thereof.

In [14]:

    vcsma.trace['Lambda',-4:] #last 4 draws of lambda

Out[14]:

    array([ 0.48405096,  0.48405096,  0.48405096,  0.78951104])

In [15]:

    vcsma.trace[['Tau2', 'Sigma2'], 0:2] #the first 2 variance parameters

Out[15]:

    {'Sigma2': [34.046017271846118, 34.215037006990748],
     'Tau2': [1.3342090890246048, 1.5873738276194227]}

We only ran a single chain, so the first index is assumed to be zero. You can run more than one chain in parallel, using the builtin python `multiprocessing` library:

In [16]:

    vcsma_p = spvcm.upper.SMA(Y, X, M=W2, Z=Z, membership=membership,
                            n_samples=5000, n_jobs=3, #run 3 chains
                            configs=dict(tuning=500, adapt_step=1.01))

In [17]:

    vcsma_p.trace[0, 'Betas', -1] #the last draw of Beta on the first chain.

Out[17]:

    array([[ 10.42671835],
           [ -0.18088028],
           [  1.7756331 ],
           [  3.60975713],
           [ -0.07059934],
           [ -2.7762485 ]])

In [18]:

    vcsma_p.trace[1, 'Betas', -1] #the last draw of Beta on the second chain

Out[18]:

    array([[ 6.84611721],
           [-0.41804214],
           [ 1.64652096],
           [ 4.29570284],
           [ 0.0309099 ],
           [ 6.81012182]])

and the chain plotting works also for the multi-chain traces. In addition, there are quite a few traceplot options, and all the plots are returned by the methods as matplotlib objects, so they can also be saved using `plt.savefig()`.

In [19]:

    vcsma_p.trace.plot(burn=1000, thin=10)
    plt.suptitle('SMA of Homicide Rate in Southern US Counties', y=0, fontsize=20)
    #plt.savefig('trace.png') #saves to a file called "trace.png"
    plt.show()

    /home/ljw/anaconda3/envs/py3/lib/python3.5/site-packages/statsmodels/nonparametric/kdetools.py:20: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
      y = X[:m/2+1] + np.r_[0,X[m/2+1:],0]*1j

In [20]:

    vcsma_p.trace.plot(burn=-100, varnames='Lambda') #A negative burn-in works like negative indexing in Python & R
    plt.suptitle('First 100 iterations of $\lambda$', fontsize=20, y=.02)
    plt.show() #so this plots Lambda in the first 100 iterations.

    /home/ljw/anaconda3/envs/py3/lib/python3.5/site-packages/statsmodels/nonparametric/kdetools.py:20: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
      y = X[:m/2+1] + np.r_[0,X[m/2+1:],0]*1j

To get stuff like posterior quantiles, you can use the attendant pandas dataframe functionality, like `describe`.

In [21]:

    df = vcsma.trace.to_df()

In [22]:

    df.describe()

Out[22]:

| Lambda | Sigma2 | Tau2 | Betas_0 | Betas_1 | Betas_2 | Betas_3 | Betas_4 | Betas_5 | Alphas_0 | ... | Alphas_6 | Alphas_7 | Alphas_8 | Alphas_9 | Alphas_10 | Alphas_11 | Alphas_12 | Alphas_13 | Alphas_14 | Alphas_15
--- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---
count | 5000.000000 | 5000.000000 | 5000.000000 | 5000.000000 | 5000.000000 | 5000.000000 | 5000.000000 | 5000.000000 | 5000.000000 | 5000.000000 | ... | 5000.000000 | 5000.000000 | 5000.000000 | 5000.000000 | 5000.000000 | 5000.000000 | 5000.000000 | 5000.000000 | 5000.000000 | 5000.000000
mean | 0.608686 | 32.265190 | 1.828138 | 8.156127 | -0.258890 | 1.796985 | 3.973051 | -0.003564 | 2.108436 | -1.588250 | ... | -0.400545 | 1.117568 | 0.102000 | -0.169745 | 1.024000 | 0.308998 | 0.189192 | 1.873476 | -0.617676 | 2.084861
std | 0.277064 | 1.211527 | 1.121763 | 3.249228 | 0.080627 | 0.205511 | 0.235770 | 0.084480 | 9.478349 | 0.866599 | ... | 0.825168 | 0.897640 | 0.846020 | 0.776934 | 0.860310 | 0.799908 | 0.767246 | 0.710348 | 0.868327 | 0.834200
min | -0.527474 | 27.965344 | 0.241872 | -3.912410 | -0.613551 | 0.886141 | 2.947461 | -0.295581 | -31.363481 | -4.657230 | ... | -4.289157 | -2.552218 | -2.767615 | -2.987661 | -2.569753 | -3.273487 | -3.087397 | -0.673602 | -4.875626 | -0.676062
25% | 0.445954 | 31.442746 | 1.099079 | 6.026430 | -0.312824 | 1.660085 | 3.816479 | -0.059833 | -4.274624 | -2.147018 | ... | -0.931950 | 0.544729 | -0.435577 | -0.677292 | 0.464970 | -0.201530 | -0.314159 | 1.406189 | -1.186915 | 1.528803
50% | 0.647734 | 32.253578 | 1.575177 | 8.142431 | -0.260495 | 1.797555 | 3.975418 | -0.004856 | 2.077913 | -1.553896 | ... | -0.377612 | 1.106097 | 0.089986 | -0.159915 | 1.043969 | 0.318552 | 0.185046 | 1.868410 | -0.593257 | 2.059923
75% | 0.842345 | 33.042655 | 2.256722 | 10.344981 | -0.205082 | 1.938355 | 4.129071 | 0.053324 | 8.408304 | -1.004612 | ... | 0.145648 | 1.685655 | 0.627832 | 0.346805 | 1.592591 | 0.827596 | 0.689496 | 2.330962 | -0.022296 | 2.614217
max | 0.999475 | 37.975548 | 18.930464 | 18.982719 | 0.061743 | 2.554069 | 4.799785 | 0.362505 | 36.030492 | 1.550131 | ... | 2.606743 | 4.344976 | 3.047487 | 2.684723 | 4.345115 | 2.962645 | 3.355820 | 4.745947 | 2.064849 | 6.190255

8 rows × 25 columns

There is also a `trace.summarize` function that will compute various things contained in `spvcm.diagnostics` on the chain. It takes a while for large chains, because the `statsmodels.tsa.AR` estimator is much slower than the `ar` estimator in `R`. If you have rpy2 installed and `CODA` installed in your R environment, I attempt to use R directly.

In [23]:

    vcsma.trace.summarize()

Out[23]:

|  | mean | HPD_low | median | HPD_high | std | N_iters | N_effective | AR_loss
--- | --- | --- | --- | --- | --- | --- | --- | --- | ---
Chain_0 | Lambda | 0.608686 | 0.065576 | 0.647734 | 0.999475 | 0.277064 | 5000 | 496 | 0.9008
Sigma2 | 32.265190 | 29.986869 | 32.253578 | 34.683241 | 1.211527 | 5000 | 5000 | 0.0000
Tau2 | 1.828138 | 0.321364 | 1.575177 | 3.844107 | 1.121763 | 5000 | 1007 | 0.7986
Betas_0 | 8.156127 | 1.993420 | 8.142431 | 14.698938 | 3.249228 | 5000 | 2308 | 0.5384
Betas_1 | -0.258890 | -0.413384 | -0.260495 | -0.100403 | 0.080627 | 5000 | 2279 | 0.5442
Betas_2 | 1.796985 | 1.398922 | 1.797555 | 2.194201 | 0.205511 | 5000 | 4123 | 0.1754
Betas_3 | 3.973051 | 3.514929 | 3.975418 | 4.435496 | 0.235770 | 5000 | 3187 | 0.3626
Betas_4 | -0.003564 | -0.168442 | -0.004856 | 0.157426 | 0.084480 | 5000 | 573 | 0.8854
Betas_5 | 2.108436 | -15.912772 | 2.077913 | 21.151735 | 9.478349 | 5000 | 4067 | 0.1866
Alphas_0 | -1.588250 | -3.261818 | -1.553896 | 0.132113 | 0.866599 | 5000 | 472 | 0.9056
Alphas_1 | -0.124204 | -2.814616 | -0.127997 | 2.240199 | 1.281021 | 5000 | 2959 | 0.4082
Alphas_2 | -0.086578 | -2.101419 | -0.077058 | 1.842112 | 0.989799 | 5000 | 907 | 0.8186
Alphas_3 | 0.059299 | -1.462329 | 0.068942 | 1.569032 | 0.765519 | 5000 | 386 | 0.9228
Alphas_4 | -2.261698 | -3.675698 | -2.222016 | -0.829483 | 0.731566 | 5000 | 329 | 0.9342
Alphas_5 | -0.931411 | -2.585484 | -0.909363 | 0.642910 | 0.814978 | 5000 | 286 | 0.9428
Alphas_6 | -0.400545 | -2.037184 | -0.377612 | 1.160176 | 0.825168 | 5000 | 269 | 0.9462
Alphas_7 | 1.117568 | -0.638643 | 1.106097 | 2.967069 | 0.897640 | 5000 | 485 | 0.9030
Alphas_8 | 0.102000 | -1.422613 | 0.089986 | 1.981343 | 0.846020 | 5000 | 240 | 0.9520
Alphas_9 | -0.169745 | -1.726714 | -0.159915 | 1.280813 | 0.776934 | 5000 | 419 | 0.9162
Alphas_10 | 1.024000 | -0.676837 | 1.043969 | 2.714632 | 0.860310 | 5000 | 567 | 0.8866
Alphas_11 | 0.308998 | -1.290688 | 0.318552 | 1.900305 | 0.799908 | 5000 | 391 | 0.9218
Alphas_12 | 0.189192 | -1.357847 | 0.185046 | 1.670994 | 0.767246 | 5000 | 399 | 0.9202
Alphas_13 | 1.873476 | 0.530793 | 1.868410 | 3.320154 | 0.710348 | 5000 | 234 | 0.9532
Alphas_14 | -0.617676 | -2.277931 | -0.593257 | 1.104059 | 0.868327 | 5000 | 528 | 0.8944
Alphas_15 | 2.084861 | 0.386081 | 2.059923 | 3.698922 | 0.834200 | 5000 | 419 | 0.9162

So, 5000 iterations, but many parameters have an effective sample size that's much less than this. There's debate about whether it's necesasry to thin these samples in accordance with the effective size, and I think you should thin your sample to the effective size and see if it affects your HPD/Standard Errorrs.

The existing python packages for MCMC diagnostics were incorrect. So, I've implemented many of the diagnostics from `CODA`, and have verified that the diagnostics comport with `CODA` diagnostics. One can also use `numpy` & `statsmodels` functions. I'll show some types of analysis.

In [24]:

    from statsmodels.api import tsa
    #if you don't have it, try removing the comment and:
    #! pip install statsmodels

For example, a plot of the partial autocorrelation in $\lambda$, the upper-level spatial moving average parameter, over the last half of the chain is:

In [25]:

    plt.plot(tsa.pacf(vcsma.trace['Lambda', -2500:]))

Out[25]:

    [<matplotlib.lines.Line2D at 0x7f5b64462b70>]

So, the chain is close-to-first order:

In [26]:

    tsa.pacf(df.Lambda)[0:3]

Out[26]:

    array([ 1.        ,  0.81965873, -0.00715038])

We could do this for many parameters, too. An Autocorrelation/Partial Autocorrelation plot can be made of the marginal effects by:

In [27]:

    betas = [c for c in df.columns if c.startswith('Beta')]
    f,ax = plt.subplots(len(betas), 2, figsize=(10,8))
    for i, col in enumerate(betas):
        ax[i,0].plot(tsa.acf(df[col].values))
        ax[i,1].plot(tsa.pacf(df[col].values)) #the pacf plots take a while
        ax[i,0].set_title(col +' (ACF)')
        ax[i,1].set_title('(PACF)')
    f.tight_layout()
    plt.show()

As far as the builtin diagnostics for convergence and simulation quality, the `diagnostics` module exposes a few things:

Geweke statistics for differences in means between chain components:

In [28]:

    gstats = spvcm.diagnostics.geweke(vcsma, varnames='Tau2') #takes a while
    print(gstats)

    [{'Tau2': array([-1.10621591, -0.72340167, -0.39336566, -0.6434425 , -1.07029223,
           -0.64300605, -0.55609213, -0.86085011,  0.03321543, -0.07358083,
            0.96706966,  0.2995308 ,  0.10946477,  0.72820281,  1.03346151,
            1.06694517,  1.46883481,  1.14800018,  1.28145411,  0.77522892,
            1.41288455,  1.30949693,  1.13250827,  1.04737628,  0.90399709,
            0.75503933,  0.52152212,  0.40105851,  0.79339579,  0.7520482 ,
            0.29342595,  0.35693364,  0.8404059 ,  1.31066294,  0.11754609,
           -1.09797389, -1.53398731, -0.86139733, -1.13148474, -0.75681619,
           -0.32750459,  1.06321117,  1.63930971,  1.08093095,  0.84069153,
            1.56772766,  0.24701853, -0.36473467,  0.26204387,  0.61498849])}]

Typically, this means the chain is converged at the given "bin" count if the line stays within $\pm2$. The geweke statistic is a test of differences in means between the given chunk of the chain and the remaining chain. If it's outside of +/- 2 in the early part of the chain, you should discard observations early in the chain. If you get extreme values of these statistics throughout, you need to keep running the chain.

In [29]:

    plt.plot(gstats[0]['Tau2'][:-1])

Out[29]:

    [<matplotlib.lines.Line2D at 0x7f5b5430be48>]

We can also compute Monte Carlo Standard Errors like in the `mcse` R package, which represent the intrinsic error contained in the estimate:

In [30]:

    spvcm.diagnostics.mcse(vcsma, varnames=['Tau2', 'Sigma2'])

Out[30]:

    {'Sigma2': 0.014152442945278773, 'Tau2': 0.0326037034802052}

Another handy statistic is the Partial Scale Reduction factor, which measures of how likely a set of chains run in parallel have converged to the same stationary distribution. It provides the difference in variance between between chains vs. within chains.

If these are significantly larger than one (say, 1.5), the chain probably has not converged. Being marginally below $1$ is fine, too.

In [31]:

    spvcm.diagnostics.psrf(vcsma_p, varnames=['Tau2', 'Sigma2'])

Out[31]:

    {'Sigma2': 1.000201067966211, 'Tau2': 1.0016696419316742}

Highest posterior density intervals provide a kind of interval estimate for parameters in Bayesian models:

In [32]:

    spvcm.diagnostics.hpd_interval(vcsma, varnames=['Betas', 'Lambda', 'Sigma2'])

Out[32]:

    {'Betas': [(1.9934200102974096, 14.698937734522005),
      (-0.41338424595666518, -0.10040315322431537),
      (1.3989223581965953, 2.1942011388972169),
      (3.5149294335233812, 4.4354957521824439),
      (-0.16844241462922532, 0.1574258939322527),
      (-15.91277183831169, 21.151734866467908)],
     'Lambda': (0.065576011337338092, 0.99947503163647056),
     'Sigma2': (29.986869435028119, 34.683241168680439)}

Sometimes, you want to apply arbitrary functions to each parameter trace. To do this, I've written a `map` function that works like the python builtin `map`. For example, if you wanted to get arbitrary percentiles from the chain:

In [33]:

    vcsma.trace.map(np.percentile,
                    varnames=['Lambda', 'Tau2', 'Sigma2'],
                    #arguments to pass to the function go last
                    q=[25, 50, 75])

Out[33]:

    [{'Lambda': array([ 0.44595386,  0.64773363,  0.8423446 ]),
      'Sigma2': array([ 31.44274643,  32.25357801,  33.0426554 ]),
      'Tau2': array([ 1.09907856,  1.57517696,  2.25672175])}]

In addition, you can pop the trace results pretty simply to a `.csv` file and analyze it elsewhere, like if you want to use use the `coda` Bayesian Diagnostics package in `R`.

To write out a model to a csv, you can use:

In [34]:

    vcsma.trace.to_csv('./model_run.csv')

And, you can even load traces from csvs:

In [35]:

    tr = spvcm.Trace.from_csv('./model_run.csv')
    print(tr.varnames)
    tr.plot(varnames=['Tau2'])

    ['Lambda', 'Tau2', 'Sigma2', 'Alphas', 'Betas']

    /home/ljw/anaconda3/envs/py3/lib/python3.5/site-packages/statsmodels/nonparametric/kdetools.py:20: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
      y = X[:m/2+1] + np.r_[0,X[m/2+1:],0]*1j

Out[35]:

    (<matplotlib.figure.Figure at 0x7f5b5414fac8>,
     array([[<matplotlib.axes._subplots.AxesSubplot object at 0x7f5b5441eb70>,
             <matplotlib.axes._subplots.AxesSubplot object at 0x7f5b542afbe0>]], dtype=object))

# Working with models: `draw` and `sample`¶

These two functions are used to call the underlying Gibbs sampler. They take no arguments, and operate on the sampler in place. `draw` provides a single new sample:

In [36]:

    vcsma.draw()

And sample steps forward an arbitrary number of times:

In [37]:

    vcsma.sample(10)

At this point, we did 5000 initial samples and 11 extra samples. Thus:

In [38]:

    vcsma.cycles

Out[38]:

    5011

Parallel models can suspend/resume sampling too:

In [39]:

    vcsma_p.sample(10)

In [40]:

    vcsma_p.cycles

Out[40]:

    5010

Under the hood, it's the `draw` method that actually ends up calling one run of `model._iteration`, which is where the actual statistical code lives. Then, it updates all `model.traced_params` by adding their current value in `model.state` to `model.trace.` In addition, `model._finalize` is called the first time sampling is run, which computes some of the constants & derived quantities that save computing time.

# Working with models: `state`¶

This is the collection of current values in the sampler. To be efficient, Gibbs sampling must keep around some of the computations used in the simulation, since sometimes the same terms show up in different conditional posteriors. So, the current values of the sampler are stored in `state`.

All of the following are tracked in the state:

In [41]:

    print(vcsma.state.keys())

    dict_keys(['Psi_2', 'Betas_cov0i', 'Betas_covm', 'XtX', 'Sigma2_b0', 'Lambda_min', 'DeltatDelta', 'Sigma2_an', 'N', 'In', 'Delta', 'Log_Lambda0', 'Psi_2i', 'Ij', 'M', 'X', 'J', 'Rho_min', 'Betas_cov0', 'Rho_max', 'Tau2_a0', 'p', 'Psi_1i', 'Tau2_an', 'Lambda_max', 'Psi_1', 'PsiLambdai', 'DeltaAlphas', 'Rho', 'XBetas', 'Tau2', 'Y', 'initial_values', 'Lambda', 'Betas', 'Sigma2', 'Betas_mean0', 'W', 'PsiRhoi', 'Alphas', 'Tau2_b0', 'Log_Rho0', 'Sigma2_a0'])

If you want to track how something (maybe a hyperparameter) changes over sampling, you can pass `extra_traced_params` to the model declaration:

In [42]:

    example = spvcm.upper.SMA(Y, X, M=W2, Z=Z, membership=membership,
                            n_samples=250,
                            extra_traced_params = ['DeltaAlphas'],
                            configs=dict(tuning=500, adapt_step=1.01))
    example.trace.varnames

Out[42]:

    ['Lambda', 'Betas', 'Sigma2', 'DeltaAlphas', 'Alphas', 'Tau2']

# `configs`¶

this is where configuration options for the various MCMC steps are stored. For multilevel variance components models, these are called $\rho$ for the lower-level error parameter and $\lamdba$ for the upper-level parameter. Two exact sampling methods are implemented, Metropolis sampling & Slice sampling.

Each MCMC step has its own config:

In [43]:

    vcsma.configs

Out[43]:

    {'Lambda': <spvcm.steps.Metropolis at 0x7f5b6a0544a8>,
     'Rho': <spvcm.steps.Metropolis at 0x7f5b6a054c88>}

Since `vcsma` is an upper-level-only model, the `Rho` config is skipped. But, we can look at the `Lambda` config. The number of accepted `lambda` draws is contained in :

In [44]:

    vcsma.configs.Lambda.accepted

Out[44]:

    1177

so, the acceptance rate is

In [45]:

    vcsma.configs.Lambda.accepted / float(vcsma.cycles)

Out[45]:

    0.23488325683496308

Also, if you want to get verbose output from the metropolis sampler, there is a "debug" flag:

In [46]:

    example = spvcm.upper.SMA(Y, X, M=W2, Z=Z, membership=membership,
                            n_samples=500,
                            configs=dict(tuning=250, adapt_step=1.01,
                                         debug=True))

Which stores the information about each iteration in a list, accessible from `model.configs.<parameter>._cache`:

In [47]:

    example.configs.Lambda._cache[-1] #let's only look at the last one

Out[47]:

    {'accepted': False,
     'current_logp': array([-9.40271195]),
     'jump': 2.246453562495179,
     'new_logp': array([-9.40271195])}

Configuration of the MCMC steps is done using the `config` options dictionary, like done in `spBayes` in `R`. The actual configuration classes exist in spvcm.steps:

In [48]:

    from spvcm.steps import Metropolis, Slice

Most of the common options are:

### Metropolis¶

- `jump`: the starting standard deviation of the proposal distribution
- `tuning`: the number of iterations to tune the scale of the proposal
- `ar_low`: the lower bound of the target acceptance rate range
- `ar_hi`: the upper bound of the target acceptance rate range
- `adapt_step`: a number (bigger than 1) that will be used to modify the jump in order to keep the acceptance rate betwen `ar_lo` and `ar_hi`. Values much larger than `1` result in much more dramatic tuning.

### Slice¶

- `width`: starting width of the level set
- `adapt`: number of previous slices use in the weighted average for the next slice. If `0`, the `width` is not dynamically tuned.

In [49]:

    example = spvcm.upper.SMA(Y, X, M=W2, Z=Z, membership=membership,
                            n_samples=500,
                            configs=dict(tuning=250, adapt_step=1.01,
                                         debug=True, ar_low=.1, ar_hi=.4))

In [50]:

    example.configs.Lambda.ar_hi, example.configs.Lambda.ar_low

Out[50]:

    (0.4, 0.1)

In [51]:

    example_slicer = spvcm.upper.SMA(Y, X, M=W2, Z=Z, membership=membership,
                                   n_samples=500,
                                   configs=dict(Lambda_method='slice'))

In [52]:

    example_slicer.trace.plot(varnames='Lambda')
    plt.show()

    /home/ljw/anaconda3/envs/py3/lib/python3.5/site-packages/statsmodels/nonparametric/kdetools.py:20: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
      y = X[:m/2+1] + np.r_[0,X[m/2+1:],0]*1j

In [53]:

    example_slicer.configs.Lambda.adapt, example_slicer.configs.Lambda.width

Out[53]:

    (0, 0.5)

# Working with models: customization¶

If you're doing heavy customization, it makes the most sense to first initialize the class without sampling. We did this before when showing how the "extra_traced_params" option worked.

To show, let's initialize a double-level SAR-Error variance components model, but not actually draw anything.

To do this, you pass the option `n_samples=0`.

In [54]:

    vcsese = spvcm.both.SESE(Y, X, W=W1, M=W2, Z=Z, membership=membership,
                          n_samples=0)

This sets up a two-level spatial error model with the default uninformative configuration. This means the prior precisions are all `I * .001*`, prior means are all 0, spatial parameters are set to `-1/(n-1)`, and prior scale factors are set arbitrarily.

### Configs¶

Options are set by assgning to the relevant property in `model.configs`.

The model configuration object is another dictionary with a few special methods.

Configuration options are stored for each parameter separately:

In [55]:

    vcsese.configs

Out[55]:

    {'Lambda': <spvcm.steps.Metropolis at 0x7f5b3f9a6400>,
     'Rho': <spvcm.steps.Metropolis at 0x7f5b3f9a6ba8>}

So, for example, if we wanted to turn off adaptation in the upper-level parameter, and fix the Metrpolis jump variance to `.25`:

In [56]:

    vcsese.configs.Lambda.max_tuning = 0
    vcsese.configs.Lambda.jump  = .25

### Priors¶

Another thing that might be interesting (though not "bayesian") would be to fix the prior mean of $\beta$ to the OLS estimates. One way this could be done would be to pull the `Delta` matrix out from the state, and estimate: $$ Y = X\beta + \Delta Z + \epsilon $$ using `PySAL`:

In [57]:

    Delta = vcsese.state.Delta
    DeltaZ = Delta.dot(Z)
    vcsese.state.Betas_mean0 = ps.spreg.OLS(Y, np.hstack((X, DeltaZ))).betas

### Starting Values¶

If you wanted to start the sampler at a given starting value, you can do so by assigning that value to the `Lambda` value in `state`.

In [58]:

    vcsese.state.Lambda = -.25

Sometimes, it's suggested that you start the beta vector randomly, rather than at zero. For the parallel sampling, the model starting values are adjusted to induce overdispersion in the start values.

You could do this manually, too:

In [59]:

    vcsese.state.Betas += np.random.uniform(-10, 10, size=(vcsese.state.p,1))

### Spatial Priors¶

Changing the spatial parameter priors is also done by changing their prior in state. This prior must be a function that takes a value of the parameter and return the log of the prior probability for that value.

For example, we could assign `P(\lambda) = Beta(2,1)` and zero if outside $(0,1)$, and asign $\rho$ a truncated $\mathcal{N}(0,.5)$ prior by first defining their functional form:

In [60]:

    from scipy import stats

In [61]:

    def Lambda_prior(val):
        if (val < 0) or (val > 1):
            return -np.inf
        return np.log(stats.beta.pdf(val, 2,1))

In [62]:

    def Rho_prior(val):
        if (val > .5) or (val < -.5):
            return -np.inf
        return np.log(stats.truncnorm.pdf(val, -.5, .5, loc=0, scale=.5))

And then assigning to their symbols, `LogLambda0` and `LogRho0` in the state:

In [63]:

    vcsese.state.LogLambda0 = Lambda_prior
    vcsese.state.LogRho0 = Rho_prior

### Performance¶

The efficiency of the sampler is contingent on the lower-level size. If we were to estimate the draw in a dual-level SAR-Error Variance Components iteration:

In [64]:

    %timeit vcsese.draw()

    10 loops, best of 3: 42.5 ms per loop

To make it easy to work with the model, you can interrupt and resume sampling using keyboard interrupts (`ctrl-c` or the `stop` button in the notebook).

In [65]:

    %time vcsese.sample(100)

    CPU times: user 8.3 s, sys: 103 ms, total: 8.4 s
    Wall time: 4.27 s

In [66]:

    vcsese.sample(10)

# Under the Hood¶

### Package Structure¶

Most of the tools in the package are stored in relevant python files in the top level or a dedicated subfolder. Explaining a few:

- `abstracts.py` - the abstract class machinery to iterate over a sampling loop. This is where the classes are defined, like `Trace`, `Sampler_Mixin`, or `Hashmap`.
- `plotting.py` - tools for plotting output
- `steps.py` - the step method definitions
- `verify.py` - like `user` checks in `pysal.spreg`, this contains a few sanity checks.
- `utils.py`- contains statistical or numerical utilities to make the computation easier, like cholesky multivariate normal sampling, more sparse utility functions, etc.
- `diagnostics.py` - all the diagnostics
- `priors.py` - definitions of alternative prior forms. Right now, this is pretty simple.
- `sqlite.py` - functions to use a sqlite database instead of an in-memory chain are defined here.

### The implementation of a Model¶

The package is implemented so that every "model type" first sends off to the `spvcm.both.Base_Generic`, which sets up the state, trace, and priors.

Models are added by writing a `model.py` file and possibly a `sample.py` file. The `model.py` file defines a `Base/User` class pair (like `spreg`) that sets up the state and trace. It must define hyperparameters, and can precompute objects used in the sampling loop. The base class should inherit from `Sampler_Mixin`, which defines all of the machinery of sampling.

The loop through the conditional posteriors should be defined in `model.py`, in the `model._iteration` function. This should update the model state in place.

The model may also define a `_finalize` function which is run once before sampling.

So, if I write a new model, like a varying-intercept model with endogenously-lagged intercepts, I would write a `model.py` containing something like:

    class Base_VISAR(spvcm.generic.Base_Generic):
        def __init__(self, Y, X, M, membership=None, Delta=None,
                     extra_traced_params=None, #record extra things in state
                     n_samples=1000, n_jobs=1, #sampling config
                     priors = None, # dict with prior values for params
                     configs=None, # dict with configs for MCMC steps
                     starting_values=None, # dict with starting values
                     truncation=None, # options to truncate MCMC step priors
                     center=False, # Whether to center the X,Z matrices
                     scale=False # Whether re-scale the X,Z matrices
                     ):
            super(Base_VISAR, self).__init__(self, Y, X, M, W=None,
                                             membership=membership,
                                             Delta=Delta,
                                             n_samples=0, n_jobs=n_jobs,
                                             priors=priors, configs=configs,
                                             starting_values=starting_values,
                                             truncation=truncation,
                                             center=center,
                                             scale=scale
                                             )
            self.sample(n_samples, n_jobs=n_jobs)

            def _finalize(self):
                # the degrees of freedom of the variance parameter is constant
                self.state.Sigma2_an = self.state.N/2 + self.state.Sigma2_a0
                ...

            def _iteration(self):

                # computing the values needed to sample from the conditional posteriors
                mean = spdot(X.T, spdot(self.PsiRhoi, X)) / Sigma2 + self.state.bmean0
                ...
        ...

I've organized the directories in this project into `both_levels`, `upper_level`, `lower_level`, and `hierarchical`, which contains some of the spatially-varying coefficient models & other models I'm working on that are unrelated to the multilevel variance components stuff.

Since most of the `_iteration` loop is the same between models, most of the models share the same sampling code, but customize the structure of the covariance in each level. These covariance variables are stored in the `state.Psi_1`, for the lower-level covariance, and `state.Psi_2` for the upper-level covariance. Likewise, the precision functions are `state.Psi_1i` and `state.Psi_2i`.

For example:

In [67]:

    vcsese.state.Psi_1 #lower-level covariance

Out[67]:

    <function spvcm.utils.se_covariance>

In [68]:

    vcsese.state.Psi_2 #upper-level covariance

Out[68]:

    <function spvcm.utils.se_covariance>

In [69]:

    vcsma.state.Psi_2 #upper-level covariance

Out[69]:

    <function spvcm.utils.sma_covariance>

In [70]:

    vcsma.state.Psi_2i

Out[70]:

    <function spvcm.utils.sma_precision>

In [71]:

    vcsma.state.Psi_1

Out[71]:

    <function spvcm.utils.ind_covariance>

The functions that generate the covariance matrices are stored in `spvcm.utils`. They can be arbitrarily overwritten for alternative covariance specifications.

Thus, if we want to sample a model with a new covariance specification, then we need to define functions for the variance and precision.