TID HEADCANNON

As they grew up, Will and Tessa's children never saw any Silent Brother other than their 'Uncle Jem'. As a result, when James first saw Brother Enoch he whipped out his dagger and yelled that he was an imposter and Silent Brothers never looked so hideous.