31309206463?profile=RESIZE_400xCompanies invest large sums in training artificial intelligence (AI) neural networks and treat the resulting internal weights as valuable proprietary assets. These weights are the numerical values that determine how the network processes inputs and produces outputs. An attack known as model extraction lets an outsider query a trained model many times and build a copy network that produces similar predictions. In the paper “Extraction meets Distillation – cryptanalytic techniques to recover parameters from a Deep Neural Network,” Biprarshi Biswas, Arya Anshuman Ojha, Yubaraj Das, Devansh Khosla, Kamran Rais, Saankhya Samanta, Subhamoy Maitra, and Pantelimon Stănică investigate whether an attacker can recover substantial portions of the actual weight directions inside the network and then improve the stolen copy, not merely duplicate its external behavior.[1]

An attacker starts by gaining the ability to send inputs to a working neural network and receive the class label it predicts for each one, usually through a web service that companies provide so others can use the model. The attacker sends many different inputs, records the labels that come back, and trains a new network to produce matching answers. Success is normally measured by how often the new network gives the same labels as the original on a fresh set of test inputs.

Copying the exact internal numbers that control the original network is harder because different sets of those numbers can produce nearly the same final answers. The order of certain internal units can be rearranged, and some numbers can be made larger in one place and smaller in another without changing the network's output. The researchers focus on recovering pieces of those internal numbers and then building a complete network that keeps the recovered pieces while it is trained further.

An AI neural network learns to recognize patterns or make decisions, such as identifying objects in photos or sorting emails into categories. To do this, it contains many small calculation steps. Each step takes several incoming numbers, multiplies each by its own stored number, adds the results, and then adds one more stored number. The final total of that step is either passed forward as-is or changed according to a simple rule.

One common rule passes the total forward only if it is greater than zero; otherwise, it sends zero instead. This rule is known as a rectified linear unit (ReLU). A slight variation, called leaky ReLU, still passes a positive total unchanged but, when the total is negative, sends a small fraction of it rather than zero. Because of these rules, each calculation step has a clear dividing line: all the possible inputs that make that step’s total exactly zero. On one side of the line, the step is active and passes a positive value; on the other side, it is inactive or only weakly active. When many such steps are placed one after another, the network’s final answers change from one label to another along boundaries that are defined by the points where individual steps switch from active to inactive. Those switches cause the network’s overall labeling behavior to change direction in a predictable way that an attacker can locate and measure.

Cryptanalysis is the study of how to recover secret information from a system by examining its structure and behavior rather than by guessing every possibility. The researchers treated the neural network in the same way. They looked for the places where individual calculation steps switch from active to inactive, because those switches create measurable changes in the network’s labeling behavior. By locating and analyzing those switches, they recovered the stored numbers that control the steps, using the same kind of structural examination that is used to recover secret keys from ciphers.

Those switches create special locations that an attacker can find. At such a location, the network is exactly where one calculation step switches from active to inactive, and where its final answer is about to change from one label to another. Only that one step is switching, and the two labels involved stay the same on both sides of the location. The researchers find these locations by first identifying two inputs that receive different labels from the network, then moving step by step along the boundary between those labels while checking the boundary's direction after each move. When the boundary direction suddenly changes, that change marks a candidate location of the kind described. The attacker records that location together with a nearby point on each side of it.

Once the researchers collected several of these special locations for one calculation step (i.e., neuron), they recovered the set of stored numbers that control that step. Each special location provides information about the direction those numbers point. When several such locations are examined together, their combined information reveals that direction. The researchers do this by solving a straightforward calculation that finds the best-fitting direction for the collected locations. The direction they recover still has two unknowns: how large the numbers should be, and whether they should point in the positive or negative sense. The size can usually be adjusted later by changing the numbers in the next step; the positive or negative sense must be figured out separately.

To decide whether the recovered set of stored numbers should point in the positive or negative sense, the researchers used a simple comparison of distances. Starting from one of the special locations, they moved a short way along the boundary between the two labels in each of the two possible directions and measured how far they had to go before something new happened, such as another calculation step switching or the network’s label changing again. They recorded the direction that reached the next change sooner as the positive sense.

They tested both a basic counting method and more thorough search methods that try many possible combinations of positive and negative senses and kept the combination that best matched the labels the original network gave. Once the senses are chosen, they set the extra stored number (the bias) for each step so that the step’s total was exactly zero at the special locations already found. They then calculated the final set of stored numbers that turned the outputs of all the earlier steps into the network’s predicted labels, using the labels the original network produced.

The experiments in the paper were assisted, meaning the researchers were given extra internal information that a real attacker who only sees the final labels would not have. At several stages, they were allowed to look at details of the original network such as the exact rate of change at certain points, the intermediate results of earlier calculation steps, and the true stored numbers from previous layers. The researchers state this clearly and therefore describe their process as assisted recovery followed by reconstruction that uses only labels. The latter stages work only with the labels the original network already produced, but the directions recovered earlier depend on the extra internal information. When the researchers remove access to the original network’s stored numbers during the distance comparisons, the positive or negative senses they choose become less accurate, even though the finished network still matches the original labels most of the time.

After the researchers completed the cryptanalytic examination of the switches to recover the directions and the positive or negative senses of the stored numbers for the calculation steps, they switched to a final improvement stage. In this stage, they tried different combinations of positive and negative senses and kept the combination that best matched the labels the original network gave. They then set the last set of stored numbers so that the outputs of all the earlier steps produced the correct labels.

Next, they continued training the remaining stored numbers while deliberately protecting the numbers already recovered so they were not changed. One way to enforce the protection is to copy the recovered numbers back into place after every training update. Combining the earlier recovery of the stored numbers through cryptanalytic examination with this final training stage is the central contribution of the work.

On two small networks created for testing, the reconstructed networks matched the original labels on 98.90 percent to 99.92 percent of 5,000 inputs not used during recovery or final training. On two networks trained to classify photographs of objects, the match was 56.40 percent and 58.29 percent. In both cases, the combined process of cryptanalytic examination followed by final training produced better matches on three practical measures than a process that used only final training. A high rate of matching labels does not prove that the exact stored numbers inside the original network were recovered. Different sets of stored numbers can produce nearly the same labels, and the researchers stress that copying the network’s behavior and recovering its internal stored numbers are separate goals.

The researchers also tested the same recovery process with the leaky ReLU rule instead of the ordinary ReLU rule. With the extra internal information, the leaky ReLU version recovered the stored numbers for 51 to 57 calculation steps in the deepest layer of the largest test network across eight separate runs, while the ordinary ReLU version recovered none. That advantage did not appear in every network or every layer. Additional checks in the paper examine how reliable the distance comparisons remain with small errors, how errors in the recovered numbers affect later steps, and under what conditions the already-recovered numbers stay unchanged during the final training stage.

The research shows a practical route from methods used to attack ciphers to recovering the numbers stored inside an AI neural network. The same kind of cryptanalytic examination that finds weaknesses in ciphers can locate the special switch locations and recover the directions of the stored numbers that control the calculation steps. A final training stage can then complete and improve the partial recovery using only the labels the original network produces. Recovering the exact stored numbers remains harder than simply matching the labels, and the paper's experiments rely on extra internal information during the early stages. The researchers note that a more rigorous test using only the final labels throughout would require replacing that extra information with numbers reconstructed from earlier steps and with methods that look only at the labels.

The researchers recommend further tests that use only the final labels at every stage, a more precise count of how many label requests are required, and trials on larger networks trained for practical tasks. They also note that future studies should measure label matching and recovery of the stored numbers as separate results. The results show that cryptanalytic ideas transfer to extracting neural-network parameters and that the extracted directions can be retained and refined through distillation. The pathway is partial and assisted in the reported experiments, but it shows that the boundary geometry created by ReLU-style activations provides usable structure for an attacker willing to collect and analyze dual points.

 

This article is shared at no charge for educational and informational purposes only.

Red Sky Alliance is a Cyber Threat Analysis and Intelligence Service organization.  We provide indicators of compromise information (CTI) via a notification/Tier I analysis service (RedXray) or an analysis service (CTAC).  For questions, comments, or assistance, please contact the office directly at 1-844-492-7225 or feedback@redskyalliance.com    

Weekly Cyber Intelligence Briefings:
REDSHORTS - Weekly Cyber Intelligence Briefing
https://attendee.gotowebinar.com/register/7855487668891299929

 

[1] https://six3ro.substack.com/p/recovering-ai-neural-network-weights

You need to be a member of Red Sky Alliance to add comments!