The tavern Romulus was at was a different one from the one Charles was at. He only visits it in secret, since a slave isn't supposed to have money to spend. But his master probably knew. Considering how he isn't very stealthy when drunk, he'd have been seen by a fellow slave at some point.