Prine Andrew has notably been seen with the royal family despite being stripped of his royal patronages