Washington - The National Security Agency for almost three years searched a massive database of Americans' phone call records attempting to identify potential terrorists in violation of court-approved privacy rules, and the problem went unfixed because no one at the agency had a full technical understanding of how its system worked, according to new documents and senior government officials.

Moreover, it was Justice Department officials who discovered the problem and reported it to the court that oversees surveillance programs, the documents show, undermining assertions by the NSA that self-reporting is part of its culture.

The improper activity went on from May 2006 to January 2009, according to a March 2009 opinion by Judge Reggie Walton, who serves on the Foreign Intelligence Surveillance Court.

It was one of more than a dozen documents declassified and released Tuesday in response to lawsuits by civil liberties groups and at the direction of President Barack Obama in the wake of the June disclosure by former NSA contractor Edward Snowden of the massive phone records collection.

"The documents released today are a testament to the government's strong commitment to detecting, correcting and reporting mistakes that occur in implementing technologically complex intelligence collection activities, and to continually improving its oversight and compliance processes," said James Clapper, the director of national intelligence.

The rebuke of the NSA comes less than a month after the Office of the Director of National Intelligence released a highly critical FISA court opinion that took the agency to task for its operation of a separate surveillance program. Taken together, the documents released by the office over the past month paint a picture of an agency that has sought and won sweeping surveillance powers to run complex domestic data collection without anyone having full technical understanding of the efforts, and that has repeatedly misrepresented the programs' scope to its court overseer.

Such revelations call into question the effectiveness of an oversight program that depends on accurate disclosure by the NSA to a court that acts in secret and says it lacks the resources to verify independently the agency's assertions.

"It has finally come to light that the FISC's authorizations of this vast collection program have been premised on a flawed depiction of how the NSA uses" the phone data, Walton wrote.

"This misperception by the FISC existed from the inception of its authorized collection in May 2006, buttressed by repeated inaccurate statements made in the government's submissions," he continued.

Privacy procedures "have been so frequently and systemically violated that it can fairly be said that this critical element of the overall (phone records) regime has never fully functioned effectively," he said.

The "bulk records" program began without any court or congressional approval shortly after the Sept. 11, 2001, attacks but was put under court supervision in May 2006 when American phone companies balked at providing the data solely at the request of the executive branch.

Under the program, the NSA receives daily transfers of all customer records from the nation's phone companies. Those records include numbers called, the calls' time and duration, but no customer names or the content of conversations.

Beginning on Jan. 9, 2009, Justice Department officials began notifying the court of problems, in particular that the NSA had been running an automated "alert list" on selected phone numbers without meeting the court-required standard of "reasonable and articulable suspicion" that those numbers were tied to terrorists.

Justice Department officials notified the court that the NSA had been searching the business records "in a manner directly contrary" to the court's orders "and directly contrary to the sworn attestations of several Executive Branch officials," Walton wrote in a Jan. 29, 2009, order.

NSA Director Keith Alexander suggested to the court that the violations stemmed from a belief by NSA personnel that the not all databases were covered by the same privacy rules, Walton wrote in his March opinion.

"That interpretation of the court's orders strains credulity," Walton said.

Walton also suggested that the NSA's Office of General Counsel deliberately chose to approve the use of phone numbers that did not meet the court standards because such procedures were in keeping with other NSA collection activities.

In March 2009, the court took the unusual step of ordering the government to seek approval to query the database on a case-by-case basis "except where necessary to protect against an imminent threat to human life."

Walton also expressed consternation at the NSA's inaccurate description of its use of the database, saying it slowed efforts to fix the errors. "The government's failure to ensure that responsible officials adequately understood the NSA's alert list process, and to accurately report its implementation to the court, has prevented for more than two years, both the government and the FISC from taking steps to remedy daily violations," Walton wrote.

After the problems came to light, the NSA and the Justice Department in February 2009 began a full review of the program. The review identified additional violations that needed fixing, the documents show.

For instance, the review found that "over time" more than 200 analysts from the CIA, the FBI and the National Counterterrorism Center had access to "query results" from the database that did not properly mask the identities of U.S. persons.

In September, 2009, the Justice Department reported that query results were shared with NSA analysts who were not trained in the handling of the data. Walton noted that the problems came after the government's "submission of a report intended to assure the Court that the NSA had corrected and addressed the issues."

In November 2009, Walton also expressed concern that the NSA had searched phone numbers long after the numbers had been found to be irrelevant and said he "remained concerned" that the NSA did not meet the required standard for using the numbers, exposing information about Americans who were not the subject of FBI investigations.

Walton's March ruling also expressed skepticism about the program's utility, noting the government cited only three FBI preliminary investigations opened as a result of tips gleaned from the database. "The time has come for the government to describe to the Court how, based on the information collected and analyzed during that time, the value of the program to the nation's security justifies the continued collected and retention of massive quantities of US person information," he said.

The documents were released in response to a lawsuit from the Electronic Frontier Foundation, which first sought disclosure two years ago when Congress was debating reauthorization of the statute that the government used to justify the phone collection.

"At that time, the government withheld every word, claiming that grave harm to national security would result if the court's opinions were released," said David Sobel, the foundation's senior counsel. "Now that the agency has been forced to release them, we can see that the real reason for secrecy was to conceal the fact that surveillance under the Patriot Act was far broader, and less focused, than the public and Congress had been led to believe."