Оценка и анализ моделей распознавания юмора

Добрый день! Уважаемые студенты, Вашему вниманию представляется дипломная работа на тему: «Оценка и анализ моделей распознавания юмора»

Оригинальность работы 95%

Аннотация

В последнее время в связи с развитием систем общения задача распознавания юмора приобретает всё большую актуальность. Данная работа анализирует модели распознавания юмора, обученные на нескольких наборах данных, с помощью состязательных атак. В первой части работы проведен обзор предметной области и существующих исследований по распознаванию юмора и по состязательным атакам. Вторая часть содержит описание существующих наборов данных и построения моделей распознавания юмора на их основе, а также подробный анализ их работы. В третьей части описаны применяемые состязательные атаки и их влияние на результаты классификации. Наконец,

работа завершается подведением итогов и идеями для дальнейших исследований.

Recently, in connection with the development of communication systems, the task of humor detection is becoming increasingly relevant. This paper analyzes humor recognition models trained on multiple datasets using adversarial attacks. In the first part of the paper, the research area and existing works on humor recognition and adversarial attacks are reviewed. The second part contains description of existing datasets and construction of humor recognition models, as well as detailed analysis of their results. The third part describes adversarial attacks used and their impact on classification results. Finally, the work ends with a summary of obtained results and ideas for further research.

Список ключевых слов – распознавание юмора; модель-классификатор; логистическая регрессия; эмбеддинг; Bag of Words; Word2Vec; Universal Sentence Encoder; ClassifierDL; состязательные атаки

Оглавление

Аннотация ....................................................................................................................

4

Введение.......................................................................................................................

5

Обзор литературы........................................................................................................

7

1.

Распознавание юмора ........................................................................................

7

2.

Состязательные атаки......................................................................................

17

Выводы....................................................................................................................

24

Глава 1. Данные .........................................................................................................

26

1.1

One-Liners ..........................................................................................................

26

1.2

Humicroedit ........................................................................................................

28

1.3 Unfun.me ............................................................................................................

29

1.4 PUNS..................................................................................................................

30

Выводы и результаты.............................................................................................

31

Глава 2. Модели и метрики качества ......................................................................

32

2.1

Метрики качества.............................................................................................

33

2.2

Bag of Words & Logistic Regression ................................................................

35

2.2.1

Train on One-liners ......................................................................................

37

2.2.2

Train on Humicroedit ...................................................................................

38

2.2.3

Train on Unfunme ........................................................................................

39

2.2.4

Train on Puns ...............................................................................................

40

2.3

Word2Vec & Logistic Regression .....................................................................

42

2.3.1

Train on One-Liners .....................................................................................

43

2.3.2

Train on Humicroedit ...................................................................................

44

2.3.3

Train on Unfunme ........................................................................................

45

2.3.4

Train on Puns ...............................................................................................

46

2.4

Universal Sentence Encoder & ClassifierDL ....................................................

47

2.4.1

Train on One-liners ......................................................................................

48

2.4.2

Train on Humicroedit ...................................................................................

49

2.4.3

Train on Unfunme ........................................................................................

50

2.4.4

Train on Puns ...............................................................................................

51

2.5

Сравнение моделей ..........................................................................................

52

2.5.1

One-liners (train, test) ..................................................................................

53

2

2.5.2 Humicroedit (train, test)........................................................................ 54

2.5.3 Unfunme (train, test)............................................................................. 55

2.5.4 Puns (train, test).................................................................................... 56

2.5.5 Сравнение на всех датасетах.............................................................. 57

Выводы и результаты.................................................................................... 58

Глава 3. Состязательные атаки......................................................................... 59

3.1 Метрики качества..................................................................................... 61

3.2 Slang 18+.................................................................................................. 61

3.3 Антонимы................................................................................................. 63

3.4 Рифмы....................................................................................................... 65

3.5 Сравнение................................................................................................. 67

Выводы и результаты.................................................................................... 68

Глава 4. Применение состязательных атак....................................................... 69

4.1 Slang 18+.................................................................................................. 70

4.2 Антонимы................................................................................................. 73

4.3 Рифмы....................................................................................................... 76

Выводы и результаты.................................................................................... 78

Заключение........................................................................................................ 79

Библиографический список.............................................................................. 81

Приложениe 1. Результаты классификации BOW на исходных данных......... 84

Приложениe 2. Результаты классификации Word2Vec на исходных данных. 86

Приложениe 3. Результаты классификации USE на исходных данных.......... 88

Приложениe 4. Процент изменений текстов класса «non-humorous» для каждой

атаки................................................................................................................... 90

Приложениe 5. Процент ошибок для класса «non-humorous» в исходных

результатах........................................................................................................ 91

Приложениe 6. Результаты классификации после применения атаки «Slang 18+»

92

Приложениe 7. Результаты классификации после применения атаки «Антонимы»

94

Приложениe 8. Результаты классификации после применения атаки «Рифмы» 96

Библиографический список

[1] R, Mihalcea, and C. Strapparava. “Making Computers Laugh: Investigations

in Automatic Humor Recognition.” (2005). HLT/EMNLP.

[2] Британский Национальный корпус [сайт]. URL: https://www.english-corpora.org/bnc/

[3] F, Barbieri, and H. Saggion. “Automatic Detection of Irony and Humour in Twitter.” (2014).

[4] Yang et al. “Humor recognition and humor anchor extraction.” (2015).

EMNLP.

[5] Potash et al. “#HashtagWars: Learning a Sense of Humor.” (2016).

[6] Miller et al. “SemEval-2017 Task 7: Detection and Interpretation of English Puns.” (2017). SEMEVAL2017.

[7] WordNet (2010) [сайт]. URL: https://wordnet.princeton.edu

[8] L. Chen, and Chong Min Lee. “Predicting Audience's Laughter Using

Convolutional Neural Network.” (2017).

[9] TED-talks [сайт]. URL: https://www.ted.com/talks

[10] Ermilov et al. “Stierlitz Meets SVM: Humor Detection in Russian.” (2018).

AINL.

[11] C. Chang, and C. Lin. “LIBSVM: A library for support vector machines.”

(2011).

[12] A. Cattle, and X. Ma. “Recognizing Humour using Word Associations and Humour Anchor Extraction.” (2018).

[13] S. Attardo, and V. Raskin. “Script theory revis(it)ed: Joke similarity and joke representation model.”. (1991).

[14] Zhang et al. “Exploiting Syntactic Structures for Humor Recognition.”

(2018). COLING.

81

[15] Blinov et al. “Large Dataset and Language Model Fun-Tuning for Humor

Recognition.” (2019). ACL.

[16] J. Howard, and S. Ruder. “Universal Language Model Fine-tuning for Text

Classification.”(2018). ACL.

[17] O. Weller, and K. Seppi. “Humor Detection: A Transformer Gets the Last Laugh.” (2019). EMNLP.

[18] Reddit r/Jokes thread [сайт]. URL: https://www.reddit.com/r/Jokes/

[19] P. Chen, and V. Soo, “Humor recognition using deep learning” (2018).

Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers).

[20] Devlin et al. “BERT: Pre-training of Deep Bidirectional Transformers for

Language Understanding.” (2019). NAACL-HLT.

[21] Hossain et al. “President Vows to Cut Hair : Dataset and Analysis of Creative Text Editing for Humorous Headlines.” (2019).

[22] Hossain et al. “SemEval-2020 Task 7: Assessing Humor in Edited News

Headlines” (2020).

[23] Weller et al. “Can Humor Prediction Datasets be used for Humor Generation? Humorous Headline Generation via Style Transfer.” (2020).

[24] I. Annamoradnejad, and G. Zoghi “ColBERT: Using BERT Sentence Embedding for Humor Detection” (2020).

[25] Ziser et al. “Humor Detection in Product Question Answering Systems.”

(2020).

[26] Goodfellow et al. “Explaining and harnessing adversarial examples.”

(2015). In Proceedings of ICLR.

82

[27] R. Jia and P. Liang. “Adversarial Examples for Evaluating Reading Comprehension Systems.” (2017).

[28] Rajpurkar et al. “SQuAD: 100, 000+ Questions for Machine Comprehension of Text.” (2016). EMNLP.

[29] Zhou et al. “Learning to Discriminate Perturbations for Blocking Adversarial Attacks in Text Classification.” (2019). EMNLP/IJCNLP.

[30] Y. Malkov, and D. Yashunin. “Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs.” (2016). IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31] Xiaosen et al. “Natural Language Adversarial Attacks and Defenses in

Word Level.” (2019).

[32] Zhou et al. “Fake News Detection via NLP is Vulnerable to Adversarial Attacks.” (2019).

[33] Fakebox state-of-the-art model [сайт]. URL:

https://machinebox.io/docs/fakebox

[34] Wallace et al. “Universal Adversarial Triggers for Attacking and Analyzing

NLP.” (2019). EMNLP.

[35] Singh et al. “Model Robustness with Text Classification: Semantic-preserving adversarial attacks” (2020).

[36] R. West, and E. Horvitz, “Reverse-Engineering Satire, or «Paper on

Computational Humor Accepted despite Making Serious Advances»” (2019).

Proceedings of the AAAI Conference on Artificial Intelligence.

[37] Cer, Daniel Matthew, et al. “Universal Sentence Encoder.” (2018).