he is threatening to get a restraining order

Both parents have a right to custody of their children if there is no agreement in place or court order that provides otherwise. This means that you both have the right to make decisions concerning the children and decisions should be made in the children's best interests. It is unclear whether a visit to your parents would mean a short drive or an extended trip outside of the city, province, or country. This is important because if you need to travel outside of Canada, you will need your husband to sign a travel consent. If your husband's refusal to allow the children to see their grandparents is based solely on an unreasonable personal dislike of your parents, his conduct may not be in the best interests of the children. While he cannot obtain a restraining order against your parents under the Family Law Act, if he has legitimate concerns about the children's safety he could try to obtain a criminal peace bond, however he will need to show that there is a genuine concern that the children will be harmed if they are in contact with their grandparents. I strongly recommend that you speak to a lawyer about your situation in greater detail.