The impact of the introduction of Large Language Models like ChatGPT has been tremendous. The potential applications for AI have skyrocketed with the open-sourcing of GPT (Generative Pre-Trained Transformer) models, and it cannot be overlooked that malicious programs can also be created with ChatGPT.
With the ever-growing demand for specialized Legal Language Models (LLMs), a new application has been developed that can be trained solely on data sourced from the dark web. Named DarkBERT, this achievement is credited to its South Korean creators and offers an overview of the dark web in its launch paper; click here to read further. Thus, LLMs will continue to expand and evolve with tailored data according to specific purposes.
When DarkBERT was published in 2019, it was based on the already existing RoBERTa architecture. Unexpectedly, researchers soon found that this AI model had yet to hit its peak performance levels and therefore was undertrained when initially released. This realization sparked a resurgence of interest in DarkBERT as it strived to unlock its full potential.
The research team launched their study by navigating the Dark Web with a shield of anonymity provided by the Tor network. Once the raw data was collected, refining processes like deduplication, category balancing, and pre-processing were conducted in order for them to generate an extensive database from the content sourced on the Dark Web.
Data from the Dark Web was leveraged to produce DarkBERT, a version of RoBERTa Large Language Model capable of comprehending the distinct linguistic features and encrypted data present on this part of the web. This system can then be utilized to extract valuable information from content sourced from this environment.
It may not be accurate to call English the exclusive language of the Dark Web, however, researchers believe that a particular Machine Learning Model was created specifically with this dialect in mind.
Ultimately, the researchers’ prediction was proven correct – DarkBERT demonstrated superior capabilities than other established language models. With this breakthrough, security specialists and law enforcement can further delve into the inner workings of the internet. This is where much activity takes place after all.
Even though DarkBERT has been developed similarly to other LLMs, there is still much potential for improvement with more training and tuning. What it will eventually be used for, and what information can be acquired from it, are yet to be determined.
