home | about | documents | news | publications | FOIA | research | internships | search | donate | mailing list
The State Department's new Access to Archival Databases Web portal.

U.S. National Archives Web Site Uploads Thousands of Diplomatic Cables

A Major Step for On-Line Research

National Security Archive Electronic Briefing Book No. 188

For more information contact:
William Burr, Editor
202/994-7032

Posted - April 21, 2006
(Updated - May 16, 2006)

 

 

Washington, DC, April 21, 2006 - Last month the National Archives and Records Administration (NARA) put almost 320,000 declassified cables on-line when it opened up State Department document databases from 1973 and 1974. This is significant news for researchers, because the text of declassified diplomatic cables is now retrievable on the NARA Web site.

Beginning in 1973, the State Department began creating electronic systems for transmitting cables to and from U.S. embassies. With computerized records management becoming standard practice, only electronic copies of the cables would be saved in the State Department's Central Foreign Policy Files. Over time, NARA will put on-line State Department document databases for the years after 1974. The new databases provide extensive coverage of key events of the period, from the October War, to the conflict in Indochina, to developments in Chile surrounding the September 1973 coup against the Allende government. They also include withdrawal cards of documents that are still classified, so that they can be requested under the Freedom of Information Act (FOIA).

Unfortunately, the new electronic systems lost significant numbers of cables as they migrated through new software and hardware. Moreover, Top Secret and other tightly-controlled cables cannot be retrieved on-line. The new databases include non-cable records, but they can only be identified, not viewed, on-line.

According to National Security Archive senior analyst William Burr, the National Archives and the State Department "have taken a major step forward in transparency by making available on-line around the world important declassified historical records."

The National Security Archive's background paper on the new databases shows the strengths and weaknesses of the new on-line system as well as sample cables from 1973-1974, including items regarding:

  • a June 1973 coup attempt against Chilean president Salvador Allende,

National Archives Releases New State Department Document Databases from the Mid-1970s
Edited by William Burr *

On March 17, the National Archives unveiled newly declassified databases of State Department documents from 1973 and 1974. The databases include the State Department's cable traffic from both years, non-cable documents for 1974, and withdrawal notices for those documents that remain classified. This is a long-awaited event, which encountered delays because of the difficult and complex task of segregating documents with private (e.g., Social Security numbers) and classified information from the non-classified items in the databases. The effort resulted in a major release - nearly 320,000 declassified cables (119,356 for 1973 and 200,508 for 1974).

At their best, the cable databases can be searched, documents retrieved and viewed on-line using Adobe Acrobat. This is an extraordinary development for doing primary research on the history of U.S. foreign relations because it reflects the State Department's practice, beginning in 1973, of relying on automated systems to preserve its Central Foreign Policy Files. Rather than preserving paper copies of those files, the Department began to rely on computer databases and microfilm systems. As highly innovative as these systems were in the mid-1970s, they were nevertheless in their infancy; some records were lost and others not preserved in an orderly way. Nonetheless, the availability of these electronic systems means that some primary source research can be accomplished without traveling to the NARA facility at College Park, MD. As time passes and databases for the rest of the 1970s and later years become available, the automated State Department Central Foreign Policy Files will be a major site for historical research on U.S. diplomacy.

NARA's Access to Archival Databases (AAD) Web page provides useful information in a downloadable FAQ document on the new State Department document databases, but there is more to be said. To help new users navigate their way through the system, its various components, as presented on the AAD web-site, is described and discussed in the following pages.


Cable databases

As the AAD Web site shows, the cable traffic includes the periods July-December 1973 and January-December 1974. Each period is divided into two components: the cable databases and the withheld document cards. Almost everything on the cable databases is searchable, including possible key phrases or words, date ranges, classification, or distribution method (e.g., Limdis or Limited Distribution, or Exdis or Exclusive Distribution). Moreover, cities where the United States had embassies or consulates can be searched, e.g. cables sent "to" or "from" Paris produce many hits for the U.S. Embassy in France. Critically important for searches is that the cables can be searched by using the State Department's TAGS [Traffic Analysis by Geography and Subject] system, which enables researchers to retrieve documents by subject and geographic area, e.g. PINR for "political affairs intelligence" and PK for "Pakistan." A pull-down menu allows researchers to designate those TAGS that they want to assign to searches.

The withdrawal cards do not provide quite the same scope for searches of classified cables because some of the titles are classified. Nevertheless, it is possible to use the TAGS, from/to, date ranges, classification levels, etc. to identify possibly relevant items.

A search of the cable databases for 1973 and 1974 will produce, in many instances, electronic versions of the documents; unfortunately, the search results will often yield an "error reading text" message instead of the document. That refers to those items that the State Department lost over the years when migrating the cable databases from old to new software and hardware. Apparently NARA has not yet received a formal explanation from State of exactly what happened. Nevertheless, some of the missing telegrams may be recoverable if the page of data that follows the "error reading text" message includes a number in the Film Number field. That means that the document will show up in the P-Reels (described below) for 1974 or the following years. Unhappily, many of the un-retrievable cables do not have such numbers and may be lost forever. Thus, of the telegrams for 1973, some 8,612, or 7.2%, of the 119,356 cables, are in that category. For the 1974 telegrams, the picture is not as quite as bad: 1,085 cables, or half of one percent, of 200,508 cables, are lost. According to sources at NARA, the Department of State has done everything it can to recover the lost telegrams. The Computer Output Microfilm (COM) now on the D-Reels (see below) was produced as a backup, and many telegrams were later filmed on the P-Reels. These microfilm collections are an important source for possible recovery of the lost telegrams. It is also possible that some of the missing items will eventually show up in State Department lot files, in embassy records, or in collections of presidential papers.

Once items listed on the electronic withdrawal cards are declassified in whole or in part through FOIA or mandatory review, NARA will face the difficult task of finding ways to integrate the newly-released items into the cable databases. NARA staffers are starting to think about how to manage this problem but they have not yet tasked a software contractor to develop a solution.

The cable databases for 1973 and 1974 include documents classified through Secret but they do not include Top Secret cable traffic. The National Archives has received a carton that includes a series of telegrams supposedly representing the entire universe of TS telegrams from 1973-1974. The declassification unit at NARA has not yet processed the carton. Some archivists had been led to believe that downgraded-TS documents were to be microfilmed into the P-Reels, but have found nothing to indicate that has actually occurred.

While Top Secret cables are of interest to researchers, so are NODIS ("no distribution") cables, which had highly limited circulation. There are not many NODIS cables in the 1973-1974 cable databases but so far none are retrievable - they appear to be available only on the P-Reels for subsequent years (although some may be found in the Nixon presidential collections). Beginning in 1975, however, the State Department began producing a separate set of microfilm (N-Reels) for NODIS telegrams. The Department has already transferred copies of the 1975 N-reel microfilm to NARA where they await declassification review. The NODIS telegrams will not have a separate index and will be searchable either in the cable databases or in the P-reel indexes (including respective withdrawal card indexes).


P-Reel microfilm 1974

The P-Reel ["P" for paper] microfilm includes mostly non-cable documents: airgrams, memoranda of conversations, memos and the like. The index on the AAD Web site covers all of the P-Reel documents from 1974 that have been declassified. (A separate database covers the withdrawal cards for the P-Reel). This is an index only; the documents themselves, unlike most cables, cannot be viewed on-line. The list of documents that a productive search will yield will include document numbers that begin with the year in which the item was microfilmed (e.g. P740006-0867). To deal with this problem, NARA has printed all of the documents from 1974 for which it has the microfilm. It has also created a finding aid so that researchers visiting NARA can find them easily (see Annex A below). Unfortunately, there's a kicker: More than a few of the 1974 documents were microfilmed in subsequent years going as far forward as the mid-1980s (they have document numbers beginning with, e.g., 82). NARA does not have the P-Reel microfilm for those years; only the State Department does. Therefore, researchers will need to ask the State Department for copies, presumably using the FOIA request route.

Another element of complexity in the P-reel system is the relationship between the withdrawal cards and the print-outs. Despite appearances to the contrary not every document for which there is a withdrawal card is classified. Those documents will show up fully declassified in the P-reel print-outs. However, some documents listed in the P-reel index are actually classified; researchers will find withdrawal sheets when they look for them.

The same search methods that are available for cables can be used for the P-Reels. Searches on the withdrawal cards, however, are problematic because the titles of the documents are invariably classified. While TAGS can be used to produce possibly relevant items, some documents do not have TAGS, which limits search possibilities significantly.


D-Reels

D-Reels ["D" for digital] are computer output microfilm of the electronic telegram traffic. All the telegrams in the system were supposed to have been automatically printed to film as the permanent copy. There were problems with the equipment especially in the early years, and there were some telegrams which were never successfully filmed. I don't know of any way to quantify the extent of the problem, but I do know that the FRUS historians have been unable to find a few telegrams which they thought might be significant. The telegrams are basically unarranged and unsearchable as they appear on the reels except by the numbers in the File Number of the Message Attributes.


Changes in State Department goals for transfers of the Central Foreign Policy Files

The State Department's original plan for declassification and transfer of the Central Foreign Policy Files was to transfer everything for each year at once: electronic telegrams, electronic microfilm index entries, and paper copies of the microfilmed documents. The Department expected to schedule the review so that it could reach EO 12958's 25-year goal by the end of 2006, with each year transferred when it was fully review and declassified. That would mean the transfer of everything through 1981 by December 31 of this year. Recently, however, the State Department scaled back that goal. At the December 2005 meeting of the Historical Advisory Committee, State announced that it would curtail the production and review of paper printouts after the release of the databases for 1975. Under the Executive Order's guidelines for declassification review of records on non-paper media, the Department has the right to a three-year extension of the declassification deadline, which means that the 1976-1981 materials as well as any 1974 and 1975 documents microfilmed after 1976 may not be available until 2009. According to one NARA staffer, "The HAC did not seem to realize the significance of the December announcement, but perhaps when they do State will revisit its decision."

* The editor thanks NARA officials for invaluable information and insights on the new databases.


Sample Cables from 1973 and 1974 Databases

Document 1: U.S. Embassy Santiago Cable 4728 to State Department, "Attempt Coup," 29 June 1973, Confidential

According to this report from Ambassador Nathaniel Davis, on the morning of 29 June, elements of the military loyal to President Salvador Allende tried to arrest officers who were engaged in a coup plot against the Socialist Unity government. That attempt temporarily backfired when the plotters, led by Lt. Col. Souper, detained those who were trying to arrest them and led tanks and troops to the presidential palace and the Defense Ministry. As the cable shows, the coup attempt collapsed when loyal forces led by General Prat surrounded the plotters and forced them to surrender. The next significant coup plot, in September 1973, could not be foiled and the Allende government fell to a generals' coup, which led to a bloodbath.

Documents 2a and b: On the Eve of the October War

Document 2a: State Department Cable 194280 to Embassy Tel Aviv, "Rumored Egyptian Military Alert," 30 September 1973, Secret, Exdis

Document 2b: Embassy Tel Aviv cable 7570 to State Department, "Rumored Egyptian Military Alert and Syrian Military Activity," 30 September 1973, Secret, Exdis

In late September 1973, rumors of war in the Middle East circulated and senior U.S. officials wondered if conflict was impending. Assistant Secretary of State Joseph Sisco queried Ambassador Keating, who reported that the U.S. Defense Attaché saw a Syrian attack as improbable. None of the Embassy's sources saw anything alarming. A massive intelligence failure became apparent when Syrian and Egyptian forces attacked Israel early on 6 October 1973.

Document 3: Department of State Cable 109189 to U.S. Consulate Jerusalem, 24 May 1974, Secret, Exdis

A few days after India's nuclear test on 18 May 1974 U.S. Ambassador Daniel P. Moynihan sent the Department his assessment of the situation. Kissinger was in Jerusalem for talks with the Israelis and the Department forwarded it to him. A bit obsequious, by crediting Secretary of State Kissinger and Secretary of Defense Schlesinger with so much foresight in anticipating the Indian bomb, Moynihan supported the Department's moderate response, arguing that Washington should give "a good dose of thought to how [India's] interests and ours may be accommodated, and taking emphatic steps to bring about [an] accommodation." For Moynihan, the central problem in world politics during the 1970s was not further nuclear proliferation but finding ways to solve the slowdown of growth in the West: "whether a world system can be contrived in which the surplus capital being accumulated by the oil producing countries is invested in countries such as India, using technology and goods purchased from the West."

Document 4: Department of State Cable 114183 to JCS/SecDef/CIA, "Morning Summary of Significant Events," 31 May 1974, Secret, Exdis

As a routine service to the Pentagon and the CIA, the State Department sent a daily summary of significant information gleaned by U.S. embassies and reported in cable traffic. Some of this information may have been picked up for use in the CIA's President's Daily Brief. Information reported on 31 May included reports on governmental changes in France and Thailand, China-Soviet border talks, an Argentine-India nuclear agreement, Indonesia's role on the international commission supervising the cease-fire in Vietnam, and a summary of former Prime Minister Heath's conversations with the Chinese. Heath reported that Mao was "well," Zhou Enlai "not so well" and Deng Xiaoping was "an able replacement" for Zhou if that became necessary.

Document 5: U.S. Interests Section Baghdad cable 898 to State Department, "U.S. Policy on Iran-Iraq Conflict," 23 December 1974, Secret, Exdis

During the mid-1970s, Iran, with clandestine support from the CIA, supported the Kurdish insurgency against the Baathist regime in Iraq. With Iranian forces getting more heavily involved in the conflict, the chief of the U.S. interests section, Arthur L. Lowrie, worried that the situation might get out of hand and disrupt regional stability. He saw some possibility that the Iraqi regime "for all of its abhorrent traits" could turn out to be rather different if it followed a "twin policy of development and rapprochement." Further, "the Kurdish leadership is fighting hopeless battle," and no one wants them "to succeed." Lowrie recommended the "strongest possible demarche to Shah to determine extent to which US and Iranian interests conflict in this matter and how they might be reconciled." Within months the Shah had taken steps to end the conflict with Iraq and support for the Kurds was abruptly terminated; the cable traffic from 1975, once released, may shed light on how the agreement came about.

Document 6: Embassy Phnom Penh cable 17096 to State Department, "Lon Non's Plans," 26 December 1974, Secret, Exdis

Brigadier General Lon Non was the "little brother" of Lon Nol, who ruled Cambodia from 1970 until the collapse of his regime in the spring of 1975. As Kenton Clymer shows in his prize-winning study, The United States and Cambodia, 1970-2000, U.S. officials saw Lon Non as a bad actor; according to a 1973 U.S. embassy assessment, while Lon Non displayed "unusual energy and initiative," he was "restrained by neither good political judgment nor moral scruples." The embassy had cut him off from access to U.S. equipment apparently because he siphoned it off for his own purposes. Reportedly Lon Non had his own assassination squad and he was suspected of instigating violent attacks, including assassination attempts, on political opponents. Moreover, embassy officials thought that Lon Non was behind a 1972 attempt to murder the U.S. Chargé Thomas Enders. In 1973, as part of a U.S.-sponsored attempt to broaden the Lon Nol government, Lon Non was hustled out of the country, but when he returned the State Department worried that he would start causing trouble again. John Gunther Dean, the U.S. ambassador, dismissed the "hand-wringing telegrams" from the Department: "This problem could have been avoided if those responsible for looking after Lon Non in the States had done their job effectively." When the regime fell, Lon Nol fled but "little brother" stayed behind and the Khmer Rouge executed him.

home | about | documents | news | publications | FOIA | research | internships | search | donate | mailing list

Contents of this website Copyright 1995-2004 National Security Archive. All rights reserved.
Terms and conditions for use of materials found on this website.