Re: What's wrong with this RC4?

Andrew Pogrebennyk wrote:
> Feel really good when everything is implemented right :) Or, at least,
> there are no obvious bugs this time.

Now that you've gotten numerically correct output, you should
look at some subtler issues.

Your code seems to confuse pass-phrase / key / state. An RC4
key is sequence of octets no more than 256 octets long. For
security reasons, each new message requires a new key. An RC4
encryption state consists of a permutation of the integers
[0..255] and two integers in that same range. RC4 itself has
no notion of pass-phrase.

For practical reasons, crypto code should not assume that
the entire input message is available in memory. The usual
method is to process one piece, while updating the encryption
state to be ready to handle the next piece.

Reasonable (untested) declarations might be:

typedef struct
{
     unsigned char s[256];
     unsigned char i;
     unsigned char j;
} RC4_State;

void set_key(
         RC4_State* state,
         const unsigned char* session_key,
         size_t session_key_size);

void rc4(
         RC4_State* state,
         const unsigned char* in,
         unsigned char* out,
         size_t inout_size);

-- 
--Bryan