Rio de Janeiro — The USOC says American swimmers Jack Conger and Gunnar Bentz were removed from their flight to the U.S. by Brazilian authorities Wednesday night, hours after a judge ordered gold medalist Ryan Lochte and teammate Jimmy Feigen stay in the country as police investigate their claim they were robbed during the Olympics.

United States Olympic Committee spokesman Patrick Sandusky confirmed the swimmers were detained but had no other details.

Lochte and his three teammates said they were held up at gunpoint in a taxi as they returned to the athletes village from a party, several hours after the last Olympic swimming events were held.

Police have found little evidence so far to support their accounts, and say the swimmers were unable to provide key details in police interviews.

The office of Judge Keyla Blank gave the order earlier Wednesday to seize the passports of only Lochte and Feigen.

Steve Bentz, father of Gunnar Bentz, when reached by phone late Wednesday night, said: "I really don't want to say anything," and hung up.

Lochte's father, Steve Lochte, told The Associated Press by phone from his Florida home that his son called him Tuesday after arriving in the United States. The 32-year-old swimmer was going to pick up his car and buy a new wallet to replace the one that he said was stolen in the robbery.

"I'm just happy he's safe," the elder Lochte said. "It was an unfortunate experience for him and the other three. I don't know what all the controversy is. They were basically taken out of the taxi and robbed. The main thing is he's very lucky that he's safe and that all they got was his cash and wallet."

The elder Lochte said his son's Olympic credential and cell phone weren't taken during the incident early Sunday morning. He said he was sure Ryan had his passport or he would not have been allowed to board a plane.

Feigen's whereabouts could not immediately be confirmed, though he told the San Antonio Express-News he was still in Brazil.

The U.S. Olympic Committee said police went to the athletes village Wednesday morning to try to collect the passports, but the swim team had already moved out.

The U.S. State Department issued a statement Wednesday encouraging those involved to cooperate with Brazilian law enforcement.

A USA Swimming spokesman said he couldn't comment on athlete whereabouts because of security concerns. Lochte's attorney, Jeff Ostrow, did not immediately return messages seeking comment.

A police official with knowledge of the investigation told the AP that police cannot find their taxi driver or witnesses. The official spoke on condition of anonymity because the investigation is ongoing.

The group did not call police, authorities said, and officers began investigating once they saw media reports in which Lochte's mother spoke about the robbery. Police interviewed Lochte and one other swimmer, who said they had been intoxicated and could not remember what type and color of taxi they rode in or where the robbery happened, the police official said. The swimmers also could not say what time the events occurred.

Later in the day, Lochte described the incident to NBC's "Today" show.

"We got pulled over, in the taxi, and these guys came out with a badge, a police badge, no lights, no nothing just a police badge and they pulled us over," Lochte said. "They pulled out their guns, they told the other swimmers to get down on the ground — they got down on the ground. I refused, I was like we didn't do anything wrong, so — I'm not getting down on the ground.

"And then the guy pulled out his gun, he cocked it, put it to my forehead and he said, 'Get down,' and I put my hands up, I was like 'whatever.' He took our money, he took my wallet — he left my cellphone, he left my credentials."

Word of the robbery initially created confusion between Olympic and U.S. officials. An International Olympic Committee spokesman at first said reports of the robbery were "absolutely not true," then reversed himself, apologized and said he was relying on initial information from the USOC that was wrong.

Lochte told USA Today that he and his teammates didn't initially tell the U.S. Olympic officials about the robbery "because we were afraid we'd get in trouble."

"Why would anybody fabricate anything?" Steve Lochte said. "It's just ridiculous."

Lochte swam in two events at the Rio Games, winning gold in the 4x200-meter freestyle relay. Bentz and Conger also earned gold for swimming in the preliminary heats of the 4x200 relay. Feigen received gold for swimming in the heats of the 4x100 free relay, which was won by the U.S.