The Holy family didn't get to live a peaceful and quiet life in Bethlehem for very long. Mary recovered from the birth and they were there possibly up to 2 years before the wise men showed up, to the house where they were living, not the way usually portrayed in Christmas cards and manger scenes. Then King Herod attempted to kill Jesus. The Magi who came to see the Christ child bringing gifts were warned not to go back to King Herod but to return home by another route and probably told Joseph about that before they left. Then an angel told Joseph in a dream to get up, take the child (not baby) and his mother and escape into Egypt.
They had the gold from the Magi and as a Carpenter Joseph would find work wherever he went. But being a refugee is difficult, leaving the life you know, and everything familiar behind you and coping with a different language, different culture. They had to deal with being outsiders, with a different religion from those around them. Apparantly there is historical evidence of them dwelling in caves on the outskirts of towns in Egypt. Some refugees escape with nothing but their lives. Spare a prayer for all refugees today - our Lord and Saviour, with his family were once refugees, strangers in a strange land.