Whitney Houston‘s sister-in-law Pat Houston was not happy with Ray J last night at the Billboard Awards. TMZ has learned that Ray J was almost physically removed by security from the show after Whitney’s family ordered their security to get him away from them.

Minutes before the tribute to Whitney began, Ray J was seated next to the Houston family. Pat Houston, who believes that Ray J was a bad influence on Whitney, demanded her security remove Ray J from his seat but he refused to budge.

Pat then summoned hotel security, along with several uniformed officers to physically remove Ray J.

Security warned Pat that it was about to get physical and if the drama were to be caught on video  it would not be a good look. Sources say Pat had a change of heart and Ray J got to keep his seat.

Even though he got to keep his seat, he told his people: “Pat ruined what was a great opportunity to honor the woman who I loved.”