Headcanon

Christmas in Togami's family isn't happy at all.
His parents and family send eachother rich gifts, later they have a dinner but they aren't close to eachother.
They talk about work (if they talk) and they do the dinner only because of the tradition.

Therefore, Byakuya never had happy Christmas in his life