article · Big Data Mining and Analytics · 2024
Unstructured Big Data Threat Intelligence Parallel Mining Algorithm
Zhihua Li, Xinye Yu, Tao Wei, Junhao Qian
Cited by 6Open access
Abstract
To efficiently mine threat intelligence from the vast array of open-source cybersecurity analysis reports on the web, we have developed the Parallel Deep Forest-based Multi-Label Classification (PDFMLC) algorithm. Initially, open-source cybersecurity analysis reports are collected and converted into a standardized text format. Subsequently, five tactics category labels are annotated, creating a multi-label dataset for tactics classification. Addressing the limitations of low execution efficiency and scalability in the sequential deep forest algorithm, our PDFMLC algorithm employs broadcast variables and the Lempel-Ziv-Welch (LZW) algorithm, significantly enhancing its acceleration ratio. Furthermore, our proposed PDFMLC algorithm incorporates label mutual information from the established dataset as input features. This captures latent label associations, significantly improving classification accuracy. Finally, we present the PDFMLC-based Threat Intelligence Mining (PDFMLC-TIM) method. Experimental results demonstrate that the PDFMLC algorithm exhibits exceptional node scalability and execution efficiency. Simultaneously, the PDFMLC-TIM method proficiently conducts text classification on cybersecurity analysis reports, extracting tactics entities to construct comprehensive threat intelligence. As a result, successfully formatted STIX2.1 threat intelligence is established.
Cite this paper
Li, Z., Yu, X., Wei, T., & Qian, J. (2024). Unstructured Big Data Threat Intelligence Parallel Mining Algorithm. Big Data Mining and Analytics, 7(2), 531–546. https://doi.org/10.26599/bdma.2023.9020032
Li, Zhihua, et al. “Unstructured Big Data Threat Intelligence Parallel Mining Algorithm.” Big Data Mining and Analytics, vol. 7, no. 2, 2024, pp. 531–46, https://doi.org/10.26599/bdma.2023.9020032.
Li, Zhihua, Xinye Yu, Tao Wei, and Junhao Qian. 2024. “Unstructured Big Data Threat Intelligence Parallel Mining Algorithm.” Big Data Mining and Analytics 7 (2): 531–46. https://doi.org/10.26599/bdma.2023.9020032.
Li, Z. et al. (2024) ‘Unstructured Big Data Threat Intelligence Parallel Mining Algorithm’, Big Data Mining and Analytics, 7(2), pp. 531–546. Available at: https://doi.org/10.26599/bdma.2023.9020032.
@article{li2024unstructured,
title = {{Unstructured Big Data Threat Intelligence Parallel Mining Algorithm}},
author = {Li, Zhihua and Yu, Xinye and Wei, Tao and Qian, Junhao},
journal = {Big Data Mining and Analytics},
volume = {7},
number = {2},
pages = {531--546},
year = {2024},
doi = {10.26599/bdma.2023.9020032}
}Made from the index record. Check names, capitalisation and page numbers against the paper. Need another style? AI for Research has over 10,000.