wefe.WEAT

class wefe.WEAT[source]

A implementation of Word Embedding Association Test (WEAT).

It measures the degree of association between two sets of target words and two sets of attribute words through a permutation test.

References

Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. Semantics derived automatically from language corpora contain human-like biases. Science,356(6334):183–186, 2017.

__init__(*args, **kwargs)
metric_name: str = 'Word Embedding Association Test'
metric_short_name: str = 'WEAT'
metric_template: Tuple[Union[int, str], Union[int, str]] = (2, 2)
run_query(query: wefe.query.Query, word_embedding: wefe.word_embedding_model.WordEmbeddingModel, return_effect_size: bool = False, calculate_p_value: bool = False, p_value_test_type: Literal['left-sided', 'right-sided', 'two-sided'] = 'right-sided', p_value_method: Literal['approximate', 'exact'] = 'approximate', p_value_iterations: int = 10000, p_value_verbose: bool = False, lost_vocabulary_threshold: float = 0.2, preprocessor_args: Dict[str, Optional[Union[bool, str, Callable]]] = {'lowercase': False, 'preprocessor': None, 'strip_accents': False}, secondary_preprocessor_args: Optional[Dict[str, Optional[Union[bool, str, Callable]]]] = None, warn_not_found_words: bool = False, *args: Any, **kwargs: Any) Dict[str, Any][source]

Calculate the WEAT metric over the provided parameters.

Parameters
queryQuery

A Query object that contains the target and attribute word sets to be tested.

word_embeddingWordEmbeddingModel

A WordEmbeddingModel object that contains a word embedding pretrained model.

return_effect_sizebool, optional

Specifies if the returned score in ‘result’ field of results dict is by default WEAT effect size metric, by default False

calculate_p_valuebool, optional

Specifies whether the p-value will be calculated through a permutation test. Warning: This can increase the computing time quite a lot, by default False.

p_value_test_type{‘left-sided’, ‘right-sided’, ‘two-sided}, optional

In case of calculating the p-value, specify the type of test to be performed. The options are ‘left-sided’, ‘right-sided’ and ‘two-sided , by default ‘right-sided’

p_value_method{‘exact’, ‘approximate’}, optional

In case of calculating the p-value, specify the method for calculating the p-value. This can be ‘exact’and ‘approximate’. by default ‘approximate’.

p_value_iterationsint, optional

If the p-value is calculated and the chosen method is ‘approximate’, it specifies the number of iterations that will be performed , by default 10000.

p_value_verbosebool, optional

In case of calculating the p-value, specify if notification messages will be logged during its calculation., by default False.

lost_vocabulary_thresholdfloat, optional

Specifies the proportional limit of words that any set of the query is allowed to lose when transforming its words into embeddings. In the case that any set of the query loses proportionally more words than this limit, the result values will be np.nan, by default 0.2

preprocessor_argsPreprocessorArgs, optional

A dictionary with the arguments that specify how the pre-processing of the words will be done. The possible arguments for the function are: - lowercase: bool. Indicates if the words are transformed to lowercase. - strip_accents: bool, {‘ascii’, ‘unicode’}: Specifies if the accents of

the words are eliminated. The stripping type can be specified. True uses ‘unicode’ by default.

  • preprocessor: Callable. It receives a function that operates on each

    word. In the case of specifying a function, it overrides the default preprocessor (i.e., the previous options stop working).

, by default {

‘strip_accents’: False, ‘lowercase’: False, ‘preprocessor’: None, }

secondary_preprocessor_argsPreprocessorArgs, optional

A dictionary with the arguments that specify how the secondary pre-processing of the words will be done, by default None. Indicates that in case a word is not found in the model’s vocabulary (using the default preprocessor or specified in preprocessor_args), the function performs a second search for that word using the preprocessor specified in this parameter.

warn_not_found_wordsbool, optional

Specifies if the function will warn (in the logger) the words that were not found in the model’s vocabulary , by default False.

Returns
Dict[str, Any]

A dictionary with the query name, the resulting score of the metric, and the scores of WEAT and the effect size of the metric.