Using Population objects to create biased data#

import mlsim
import pandas as pd
import numpy as np
import seaborn as sns
from collections import namedtuple

Create an all default population

pop = mlsim.bias.Population()

To view the details on this population, we can use the get_parameter_description method.

print(pop.get_parameter_description())
Demographic Parameters
DemParams(Pa=[0.5, 0.5], Pz_a=[[0.5, 0.5], [0.5, 0.5]])
Target Parameters 
TargetParams(Py_az=[[[0.95, 0.05], [0.95, 0.05]], [[0.95, 0.05], [0.95, 0.05]]])
Feature Parameters 
FeatureParams(distfunc=<function <lambda> at 0x7fa2858a7ee0>, theta=[[[[5, 2], [2, 5]], [[5, 2], [2, 5]]], [[[5, 2], [2, 5]], [[5, 2], [2, 5]]]])
Feature Noise Parameters 
NoiseParams(noisefunc=<function <lambda> at 0x7fa2858bca60>, theta=[[[1.0, 1.0], [1.0, 1.0]], [[1.0, 1.0], [1.0, 1.0]]])

The instantiation just assigns values to these parameters. In order to get data, we use the sample method.

help(pop.sample)
Help on method sample in module mlsim.bias.populations:

sample(N, return_as='DataFrame') method of mlsim.bias.populations.Population instance
    sample N members of the  population, according to its underlying
    distribution
    
    Parameters
    -----------
    N : int
        number of samples
    return_as : string, 'dataframe'
        type to return as, can be pandas 'DataFrame' or IBM AIF360
        'structuredDataset'
pop_df1 = pop.sample(100)
pop_df1.head()
a z y x0 x1
0 0.0 1.0 1.0 0.476985 4.573850
1 0.0 1.0 1.0 1.819441 3.678264
2 1.0 1.0 1.0 0.793593 5.013014
3 0.0 0.0 0.0 4.801599 1.062575
4 1.0 0.0 0.0 3.898425 1.472309

Changing the type of bias#

Now demo some with various biases to create examples

# create a correlated demographic sampler
label_bias_dem = mlsim.bias.DemographicCorrelated(rho_a=.2,rho_z=[.25,.15])

# instantiate a population with that
pop_label_bias = mlsim.bias.PopulationInstantiated(demographic_sampler=label_bias_dem)
pop_label_bias_df1 = pop_label_bias.sample(100)
pop_label_bias_df1.head()
a z y x0 x1
0 0.0 1.0 1.0 2.038903 3.600771
1 0.0 0.0 0.0 4.580856 0.286881
2 0.0 0.0 0.0 4.546255 0.120677
3 0.0 0.0 0.0 6.815834 3.730704
4 0.0 0.0 0.0 3.425712 1.601802

New we’ll create a feature bias where the classes are separable for one group and not for the other.

feature_sample_dist = lambda mu,cov :np.random.multivariate_normal(mu,cov)
per_group_means = [[[1,2,3,4,3,3],[4,6,8,8,10,6]],[[3,2,3,4,4,3],[1,3,4,4,5,3]]]
D =6
shared_cov = [np.eye(D)*.75,.95*np.eye(D)]
feature_bias = mlsim.bias.FeaturePerGroupSharedParamWithinGroup(
            feature_sample_dist,per_group_means,shared_cov)
pop_feature_bias = mlsim.bias.PopulationInstantiated(feature_sampler=feature_bias)
pop_feature_bias_df1 = pop_feature_bias.sample(100)
pop_feature_bias_df1.head()
a z y x0 x1 x2 x3 x4 x5
0 1.0 0.0 0.0 5.048226 6.602369 7.579147 6.633704 8.208764 9.526087
1 1.0 0.0 0.0 2.889835 6.247248 9.765874 8.669932 9.643852 8.426092
2 1.0 1.0 0.0 0.362637 4.353561 3.513254 6.225042 6.521920 3.785695
3 1.0 0.0 0.0 3.035455 4.596863 6.124153 8.303987 11.195854 3.100999
4 0.0 0.0 0.0 2.487673 3.611788 3.717411 6.406443 3.577272 2.763359
var_list = ['x'+ str(i) for i in range(D)]
g = sns.pairplot(pop_feature_bias_df1, vars= var_list, hue = 'z')
../_images/demo_populations_15_0.png
[sns.pairplot(dffbai, vars= var_list, hue = 'z') for ai,dffbai in pop_feature_bias_df1.groupby('a')]
[<seaborn.axisgrid.PairGrid at 0x7fa283f1e2e0>,
 <seaborn.axisgrid.PairGrid at 0x7fa280178970>]
../_images/demo_populations_16_1.png ../_images/demo_populations_16_2.png