`wefe`.RNSB¶

class wefe.RNSB[source]¶

Relative Relative Negative Sentiment Bias (RNSB).

References

[1]: Chris Sweeney and Maryam Najafian. A transparent framework for evaluating

unintended demographic bias in word embeddings.
In Proceedings of the 57th Annual Meeting of the Association for
Computational Linguistics, pages 1662–1667, 2019.

[2]: https://github.com/ChristopherSweeney/AIFairness/blob/master/python_notebooks/Measuring_and_Mitigating_Word_Embedding_Bias.ipynb

__init__(*args, **kwargs)¶

run_query(query: wefe.query.Query, model: wefe.word_embedding_model.WordEmbeddingModel, estimator: sklearn.base.BaseEstimator = <class 'sklearn.linear_model._logistic.LogisticRegression'>, estimator_params: Dict[str, Any] = {'max_iter': 10000, 'solver': 'liblinear'}, num_iterations: int = 1, random_state: Optional[int] = None, print_model_evaluation: bool = False, lost_vocabulary_threshold: float = 0.2, preprocessors: List[Dict[str, Union[str, bool, Callable]]] = [{}], strategy: str = 'first', normalize: bool = False, warn_not_found_words: bool = False, *args: Any, **kwargs: Any) → Dict[str, Any][source]¶

Calculate the RNSB metric over the provided parameters.

Note if you want to use with Bing Liu dataset, you have to pass the positive and negative words in the first and second place of attribute set array respectively. Scores on this metric vary with each run due to different instances of classifier training. For this reason, the robustness of these scores can be improved by repeating the test several times and returning the average of the scores obtained. This can be indicated in the num_iterations parameter.

Parameters

queryQuery

A Query object that contains the target and attribute word sets to be tested.

modelWordEmbeddingModel

A word embedding model.

estimatorBaseEstimator, optional

A scikit-learn classifier class that implements predict_proba function, by default None,

estimator_paramsdict, optional

Parameters that will use the classifier, by default { ‘solver’: ‘liblinear’, ‘max_iter’: 10000, }

num_iterationsint, optional

When provided, it tells the metric to run the specified number of times and then average its results. This functionality is indicated to strengthen the results obtained, by default 1.

random_stateUnion[int, None], optional

Seed that allow making the execution of the query reproducible. Warning: if a random_state other than None is provided along with num_iterations, each iteration will split the dataset and train a classifier associated to the same seed, so the results of each iteration will always be the same , by default None.

print_model_evaluationbool, optional

Indicates whether the classifier evaluation is printed after the training process is completed., by default False

preprocessorsList[Dict[str, Union[str, bool, Callable]]]

A list with preprocessor options.

A preprocessor is a dictionary that specifies what processing(s) are performed on each word before it is looked up in the model vocabulary. For example, the preprocessor {'lowecase': True, 'strip_accents': True} allows you to lowercase and remove the accent from each word before searching for them in the model vocabulary. Note that an empty dictionary {} indicates that no preprocessing is done.

The possible options for a preprocessor are:

lowercase: bool. Indicates that the words are transformed to lowercase.
uppercase: bool. Indicates that the words are transformed to uppercase.
titlecase: bool. Indicates that the words are transformed to titlecase.
strip_accents: bool, {'ascii', 'unicode'}: Specifies that the accents of the words are eliminated. The stripping type can be specified. True uses ‘unicode’ by default.
preprocessor: Callable. It receives a function that operates on each word. In the case of specifying a function, it overrides the default preprocessor (i.e., the previous options stop working).

A list of preprocessor options allows you to search for several variants of the words into the model. For example, the preprocessors [{}, {"lowercase": True, "strip_accents": True}] {} allows first to search for the original words in the vocabulary of the model. In case some of them are not found, {"lowercase": True, "strip_accents": True} is executed on these words and then they are searched in the model vocabulary.

strategystr, optional

The strategy indicates how it will use the preprocessed words: ‘first’ will include only the first transformed word found. all’ will include all transformed words found, by default “first”.

normalizebool, optional

True indicates that embeddings will be normalized, by default False

warn_not_found_wordsbool, optional

Specifies if the function will warn (in the logger) the words that were not found in the model’s vocabulary, by default False.

Returns

Dict[str, Any]: A dictionary with the query name, the calculated kl-divergence, the negative probabilities for all tested target words and the normalized distribution of probabilities.

Examples

>>> from wefe.query import Query
>>> from wefe.utils import load_test_model
>>> from wefe.metrics import RNSB
>>>
>>> # define the query
>>> query = Query(
...     target_sets=[
...         ["female", "woman", "girl", "sister", "she", "her", "hers",
...          "daughter"],
...         ["male", "man", "boy", "brother", "he", "him", "his", "son"],
...     ],
...     attribute_sets=[
...         ["home", "parents", "children", "family", "cousins", "marriage",
...          "wedding", "relatives",],
...         ["executive", "management", "professional", "corporation", "salary",
...          "office", "business", "career", ],
...     ],
...     target_sets_names=["Female terms", "Male Terms"],
...     attribute_sets_names=["Family", "Careers"],
... )
>>>
>>> # load the model (in this case, the test model included in wefe)
>>> model = load_test_model()
>>>
>>> # instance the metric and run the query
>>> RNSB().run_query(query, model) 
{
    "query_name": "Female terms and Male Terms wrt Family and Careers",
    "result": 0.09223875552506647,
    "kl-divergence": 0.09223875552506647,
    "clf_accuracy": 1.0,
    "negative_sentiment_probabilities": {
        "female": 0.5543954373912665,
        "woman": 0.3107589242224508,
        "girl": 0.18710587484907013,
        "sister": 0.1787081823837198,
        "she": 0.4172419154977331,
        "her": 0.4030950036121549,
        "hers": 0.3126640373120572,
        "daughter": 0.14249529344431694,
        "male": 0.4422224610164615,
        "man": 0.4194123616222211,
        "boy": 0.20556697141459176,
        "brother": 0.19801831727151584,
        "he": 0.5577524826493919,
        "him": 0.514179075019818,
        "his": 0.5544435993736733,
        "son": 0.18711536982098712,
    },
    "negative_sentiment_distribution": {
        "female": 0.09926195811727109,
        "woman": 0.05563995884577796,
        "girl": 0.0335004479837668,
        "sister": 0.0319968796973831,
        "she": 0.0747052496243332,
        "her": 0.07217230999250153,
        "hers": 0.05598106059906622,
        "daughter": 0.02551312816774791,
        "male": 0.07917790162647549,
        "man": 0.07509385803950792,
        "boy": 0.03680582257831352,
        "brother": 0.0354542707060297,
        "he": 0.09986302166025017,
        "him": 0.09206140304753956,
        "his": 0.09927058129913385,
        "son": 0.03350214801490194,
    },
}

wefe.RNSB¶

`wefe`.RNSB¶