Thats good news Dave. The fact you have a great relationship with your brother says to me that its not really a problem or it would have caused some level of tension or separation. Perhaps because you stopped this as you became aware of what it really was, it is only you who felt the impact of it and he more easily reconciled it later in his development?
The fact you stopped, even though he still wanted to continue, when you started to gain awareness is very important here. To me that answers the question of consent because if you had continued then it 'may' be construed that you became a consenting party yet your brother remained unable to consent. This is a question for an expert in this type of law really, but I think that would be a crucial question asked.
Hanging onto these memories in a negative context is only going to keep the cycle going. Try to rationalise them as you are doing here so that you can change how you feel about them which will allow you to move beyond them.
Take care and all the best.