A Word Embedding Based Approach for Focused Web Crawling Using the Recurrent Neural Network

Author
Keywords
Abstract
Learning-based focused crawlers download relevant uniform resource locators (URLs) from the web for a specific topic. Several studies have used the term frequency-inverse document frequency (TF-IDF) weighted cosine vector as an input feature vector for learning algorithms. TF-IDF-based crawlers calculate the relevance of a web page only if a topic word co-occurs on the said page, failing which it is considered irrelevant. Similarity is not considered even if a synonym of a term co-occurs on a web page. To resolve this challenge, this paper proposes a new methodology that integrates the Adagrad-optimized Skip Gram Negative Sampling (A-SGNS)-based word embedding and the Recurrent Neural Network (RNN).The cosine similarity is calculated from the word embedding matrix to form a feature vector that is given as an input to the RNN to predict the relevance of the website. The performance of the proposed method is evaluated using the harvest rate (hr) and irrelevance ratio (ir). The proposed methodology outperforms existing methodologies with an average harvest rate of 0.42 and irrelevance ratio of 0.58.
Year of Publication
9998
Journal
International Journal of Interactive Multimedia and Artificial Intelligence
Volume
In Press
Issue
In Press
Number
In Press
Number of Pages
1-11
Date Published
09/2020
ISSN Number
1989-1660
URL
https://www.ijimai.org/journal/sites/default/files/2020-09/ip2020_09_03_0.pdf
DOI
10.9781/ijimai.2020.09.003
Attachment