During the company’s research forum in January, a researcher at Microsoft Research Lab, Deependra Misra, presented an explanation on how to increase the accuracy of large language models through the technique of reducing the rank of the selective layer LASER.
By using a technique to simplify the selective layer, researchers were able to modify and replace a large weight matrix with a smaller one.
Weights are a key element that plays a crucial role in the ability of artificial neural networks to learn and predict.
Weights in artificial neural networks resemble synapses in biological neural networks.
Increasing the weight size increases the reliance of the large language model on the weight. According to Microsoft’s tests, replacing a large weight matrix with a smaller one does not affect the accuracy of the large language model.
Misra said, “It is expected that the model’s loss will increase during the intervention using the selective layer rank reduction technique in the large language model, meaning that the model’s performance may be negatively affected due to the reduction of information in it, as it was trained on large amounts of data.”
He confirmed that the loss of the large language model decreases with the correct intervention using the selective layer rank reduction technique.
Microsoft’s team succeeded in successfully using the selective layer rank reduction technique in three different large open-source language models, namely RoBERTa, Llama 2, and GPT-J.
The large language model improved by 30% in some cases. The performance of the open-source large language model GPT-J in predicting gender from resumes increased from 70.9% accuracy to 97.5% accuracy after using the selective layer rank reduction technique.
Artificial intelligence models make many real-world mistakes, making the accuracy of large language models a cause for concern.
The problem of hallucination is not dealt with through fear, as it is not related to a misunderstanding of things but rather to creating non-existent entities.
Hallucination and inaccurate artificial intelligence models have caused significant harm.
