`wefe`.MAC¶

class wefe.MAC[source]¶

Mean Average Cosine Similarity (MAC).

The general steps of the test are as follows [1].

Embed all target and attribute words.
For each target set:
- For each word embedding in the target set:
  For each attribute set:
  
  Calculate the cosine similarity of the target embedding and
  
  each attribute embedding of the set.
  
  Calculate the mean of the cosines similarities and save it in a array.
Average all the mean cosine similarities and return the calculated score.

The closer the value is to 1, the less biased the query will be.

References

[1]: Thomas Manzini, Lim Yao Chong,Alan W Black, and Yulia Tsvetkov. Black is to Criminal as Caucasian is to Police: Detecting and Removing Multiclass Bias in Word Embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 615–621, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.

[2]: https://github.com/TManzini/DebiasMulticlassWordEmbedding/blob/master/Debiasing/evalBias.py

__init__(*args, **kwargs)¶

run_query(query: wefe.query.Query, model: wefe.word_embedding_model.WordEmbeddingModel, lost_vocabulary_threshold: float = 0.2, preprocessors: List[Dict[str, Union[str, bool, Callable]]] = [{}], strategy: str = 'first', normalize: bool = False, warn_not_found_words: bool = False, *args: Any, **kwargs: Any) → Dict[str, Any][source]¶

Calculate the MAC metric over the provided parameters.

Parameters

queryQuery

A Query object that contains the target and attribute word sets for be tested.

modelWordEmbeddingModel

A word embedding model.

preprocessorsList[Dict[str, Union[str, bool, Callable]]]

A list with preprocessor options.

A preprocessor is a dictionary that specifies what processing(s) are performed on each word before it is looked up in the model vocabulary. For example, the preprocessor {'lowecase': True, 'strip_accents': True} allows you to lowercase and remove the accent from each word before searching for them in the model vocabulary. Note that an empty dictionary {} indicates that no preprocessing is done.

The possible options for a preprocessor are:

lowercase: bool. Indicates that the words are transformed to lowercase.
uppercase: bool. Indicates that the words are transformed to uppercase.
titlecase: bool. Indicates that the words are transformed to titlecase.
strip_accents: bool, {'ascii', 'unicode'}: Specifies that the accents of the words are eliminated. The stripping type can be specified. True uses ‘unicode’ by default.
preprocessor: Callable. It receives a function that operates on each word. In the case of specifying a function, it overrides the default preprocessor (i.e., the previous options stop working).

A list of preprocessor options allows you to search for several variants of the words into the model. For example, the preprocessors [{}, {"lowercase": True, "strip_accents": True}] {} allows first to search for the original words in the vocabulary of the model. In case some of them are not found, {"lowercase": True, "strip_accents": True} is executed on these words and then they are searched in the model vocabulary.

strategystr, optional

The strategy indicates how it will use the preprocessed words: ‘first’ will include only the first transformed word found. all’ will include all transformed words found, by default “first”.

normalizebool, optional

True indicates that embeddings will be normalized, by default False

warn_not_found_wordsbool, optional

Specifies if the function will warn (in the logger) the words that were not found in the model’s vocabulary, by default False.

Returns

Dict[str, Any]: A dictionary with the query name, the resulting score of the metric, and a dictionary with the distances of each attribute word with respect to the target sets means.

Examples

>>> from wefe.metrics import MAC
>>> from wefe.query import Query
>>> from wefe.utils import load_test_model
>>>
>>> query = Query(
...     target_sets=[
...         ["female", "woman", "girl", "sister", "she", "her", "hers",
...          "daughter"],
...         ["male", "man", "boy", "brother", "he", "him", "his", "son"],
...     ],
...     attribute_sets=[
...         ["home", "parents", "children", "family", "cousins", "marriage",
...          "wedding", "relatives",
...         ],
...         ["executive", "management", "professional", "corporation", "salary",
...          "office", "business", "career",
...         ],
...     ],
...     target_sets_names=["Female terms", "Male Terms"],
...     attribute_sets_names=["Family", "Careers"],
... )
>>>
>>> # load the model (in this case, the test model included in wefe)
>>> model = load_test_model()
>>>
>>> # instance the metric and run the query
>>> MAC().run_query(query, model) 
{'query_name': 'Female terms and Male Terms wrt Family and Careers',
'result': 0.8416415235615204,
'mac': 0.8416415235615204,
'targets_eval': {'Female terms': {'female': {'Family': 0.9185737599618733,
    'Careers': 0.916069650076679},
    'woman': {'Family': 0.752434104681015, 'Careers': 0.9377805145923048},
    'girl': {'Family': 0.707457959651947, 'Careers': 0.9867974997032434},
    'sister': {'Family': 0.5973392464220524, 'Careers': 0.9482253392925486},
    'she': {'Family': 0.7872791914269328, 'Careers': 0.9161583095556125},
    'her': {'Family': 0.7883057091385126, 'Careers': 0.9237247597193345},
    'hers': {'Family': 0.7385367527604103, 'Careers': 0.9480051446007565},
    'daughter': {'Family': 0.5472579970955849, 'Careers': 0.9277344475267455}},
'Male Terms': {'male': {'Family': 0.8735092766582966,
    'Careers': 0.9468009045813233},
    'man': {'Family': 0.8249392118304968, 'Careers': 0.9350165261421353},
    'boy': {'Family': 0.7106057899072766, 'Careers': 0.9879048476286698},
    'brother': {'Family': 0.6280269809067249, 'Careers': 0.9477180293761194},
    'he': {'Family': 0.8693044614046812, 'Careers': 0.8771287016716087},
    'him': {'Family': 0.8230192996561527, 'Careers': 0.888683641096577},
    'his': {'Family': 0.8876195731572807, 'Careers': 0.8920885202242061},
    'son': {'Family': 0.5764635019004345, 'Careers': 0.9220191016211174}}}}

wefe.MAC¶

`wefe`.MAC¶