How Hackers Exploit Protocols

When two layers on the same node communicate, or when peer layers communicate, they rely on a well-defined protocol and precisely stated message formats. The protocol states who initiates the connection, how the session is terminated, the order of messages, what to do if an error occurs, and other characteristics of the session that are necessary for the exchange to succeed. A protocol also can be thought of as an algorithm because most protocols are defined as state machines.

As with other algorithms, order is important. If you want to bake a cake, getting the right ingredients is just one part of the process. Mixing and cooking the ingredients in the right order are required to achieve an edible result. The same is true for network communications protocols. If one half of the session decides to get creative with the protocol, the results will not be guaranteed.

Getting the order of messages right is important, but you also need to format the messages properly. A baking recipe is useless if the order of the steps is clear, but the steps are not accurately defined. For example, if the recipe is incorrectly copied from a friend and asks for one unit of butter instead of one unit of flour, you will end up with a different dessert. Similarly, in network message exchanges, if message integrity cannot be guaranteed, any dialogue between the peers will not succeed.

Designing a secure distributed protocol is more of an art than a science. When someone at your site invents a new distributed security protocol, alarm bells should sound. Unless this person is knowledgeable and has studied a number of references, a better-than-average chance exists for the protocol to have weaknesses.

Here are some examples from real-world situations:

• A distributed authentication protocol was designed using a challenge-response technique, but the challenge and the response were the same value. A hacker impersonating the recipient could just replay the challenge when asked for the response.
• A protocol was designed to accept incoming messages of a fixed length. Unfortunately, the program did not check the length of incoming messages (sound familiar?) and, because the system was a public Web server, any anonymous user on the Internet could crash the site.

Both of these examples appeared in commercial products developed by well-known companies with corporate offices on the West Coast. Network security is a complex beast. The merit badge for network security is earned only after years of study and trial-and-error. Designing network protocols is tough. Designing a new network security protocol is for experienced practitioners only.