You're in the library. Panic sets in. They're missing. It might be your car keys or credit cards, but getting them back is imperative. Luckily, they're sitting on the last table you look at. Unless someone has you targeted for Mission: Impossible, there's little concern that your keys or credit cards were copied and replaced.
Unfortunately, encryption keys are harder to see and easier to copy. All electronic security systems employ some sort of key. Many employ a number of different keys. But as in J.R.R. Tolkien's The Lord of the Rings, there's "One ring to rule them all."
Developers dealing with any sort of electronics that use a communications link should be working with encryption and authentication technology. In many cases, this technology can be employed at the edge of control, such as a virtual private network (VPN) that's implemented as part of a gateway. The VPN provides a secure channel between end points, and it can alleviate the need to employ additional security features on data traveling over the VPN. Embedded developers may use a VPN, but it's rarely the only form of security that must be employed. This typically means using multiple encryption keys; hence, the need to manage and secure these keys. That's because, as any computer attacker knows, getting into a system is often a matter of finding just one key.
Unfortunately, security and encryption are complex. Though logic makes understanding them easier, gaps in a person's knowledge can make a logical conclusion inaccurate because the starting supposition is incorrect. For example, CSS (content scrambling system) encryption of movies on DVDs is relatively strong, but the ability to rip and duplicate a DVD turned out to be rather trivial using a program called DeCSS. For those who don't know, the keys to the encryption were readily available. Cracking the actual encryption was hard, but finding the keys wasn't. Of course, various commercial and political factions are trying to keep the genie in the bottle. That's a discussion I will leave to other forums, though.
It takes a good understanding of security techniques, as well as the implications of their use, to make a secure system. Get familiar with all aspects of security from encryption methods to integration with applications to deployment. The weakest link can be found anywhere along this line, so don't overlook a particular aspect because "it's not my job."
The keeper of the keys and the one key that rules them all will be well hidden in the hardware you design. Or so we hope.