E-utilities and the History server

The E-utilities API is at its most useful when you combine multiple API calls together. Constructing an E-utilities URL to use ESearch to search PubMed is more time-consuming and less effective than running the same search in the web version of PubMed. However, if you can string together multiple URLs in sequence, feeding the results of one URL into the creation of the next, you can start to tap into the real power and flexibility of E-utilities.

One way to combine multiple E-utilities URLs in sequence is by making use of the History server. Several of the E-utilities allow you to store results on this server and/or use previously-stored results to create new URLs.

Note: It is often easier and more efficient to combine multiple E-utilities URLs in sequence by using E-utilities in a programming environment, such as using the EDirect software to use E-utilities in a Unix environment. However, EDirect (and many other software packages designed to allow you to combine E-utilities URLs) use the History server behind the scenes to create links between commands. Although the software may not show you the History server parameters, it still may be making use of this feature. For more on how EDirect uses the History server, please see the “Making data pipelines with the History server” section of our EDirect overview.

Storing results to the History server using ESearch

To store the results of an ESearch to the History server for later use, when you create the ESearch URL, set the usehistory parameter to y. When this parameter is set, your ESearch results will include some additional information:

<eSearchResult>
    <Count>22968</Count>
    <RetMax>100</RetMax>
    <RetStart>0</RetStart>
    <QueryKey>1</QueryKey>
    <WebEnv>NCID_1_90045324_130.14.22.215_9001_1467896922_1344063293_0MetA0_S_MegaStore_F_1</WebEnv>
    ...

Make note of the <QueryKey> and <WebEnv> elements in the resulting XML; this is the information you will need in order to use these search results in a later E-utilities URL.

The <WebEnv> element provides a “Web environment string.” You can think of this as an identifier for your current “session.” If you want to combine multiple E-utilities URLs together, they all need be conducted in the same Web environment.

The <QueryKey> element provides a number that indicates the specific query in the specified Web environment: If the value of the element is “1”, then this is the first query conducted in this Web environment. This allows you to refer to previous queries by specifying a Web environment and query key, and also allows you to combine the results of multiple queries from the same Web environment using AND/OR/NOT by specifying which query keys you would like to combine.

Using History server data as an input

For E-utilities queries that accept a list of unique identifiers (UIDs) as an input (like EFetch), you can use the results of a previous E-utilities query as that input, as long as you have stored those results to the History server (see “Storing results to the History server”). Just specify the Web environment and query key of the previous query using the WebEnv and query_key parameters. These parameters can be used on a number of different E-utilities, including:

• EFetch: Retrieves the full records for each unique identifier (UID) in the list stored on the History server.
• ESummary: Retrieves the Document Summary for each unique identifier (UID) in the list stored on the History server.
• ELink: Retrieves LinkOut URLs or unique identifiers (UIDs) for linked or related records for each unique identifier (UID) in the list stored on the History server.

If you are using EFetch or ESummary, for large results sets, you may want to consider using retstart and retmax parameters to retrieve smaller batches, rather than your entire results set at one time. Please see our EFetch and ESummary documentation for more information about using these parameters.

Combining multiple searches with AND/OR/NOT using the History server

You can also use the WebEnv, usehistory and query_key parameters to combine multiple ESearch results sets, using Boolean operators (AND, OR, NOT). This is similar to combining searches using History in the PubMed Advanced Search Builder.

To combine searches, set the usehistory parameter to y and set the WebEnv parameter using the Web environment string from a previous E-utilities query. Once you have specified the Web environment, you have two options for specifying which searches you want to combine:

1. You can use the term parameter to incorporate the query keys for previous searches, just as you would in the PubMed Advanced Search Builder, with one key exception: instead of prefacing each query key with “#”, use the string “%23” instead (this helps avoid confusion when resolving the URL). For example, if you wanted to identify the first search in your Web environment, instead of using “#1”, you would use “%231”. You can use Boolean operators to combine multiple query keys, or to combine query keys with other search terms. For example:

   • “(#1 OR #2) NOT #3” would become &term=(%231+OR+%232)+NOT+%233
   • “#6 AND review[pt]” would become &term=%236+AND+review[pt]

2. You can combine a previous search with a new search using Boolean AND by creating a new ESearch URL and using the query_key parameter to specify the query key of a previous ESearch. The previous ESearch will be ANDed together with the search specified in the term parameter of the new search. The following two URLs are functionally the same (substitute an actual Web environment string for <webenv string>:

   • esearch.fcgi?db=pubmed&term=asthma&query_key=1&WebEnv=<webenv string>&usehistory=y
   • esearch.fcgi?db=pubmed&term=%231+AND+asthma&WebEnv=<webenv string>&usehistory=y

Both searches translate to “#1 AND asthma”.

Storing results to the History server using EPost

You can also use the EPost E-utility to upload a list of unique identifiers (UIDs) for later use. UIDs uploaded in this way can be used as an input into one of the utilities which accept the WebEnv and query_key parameters (like EFetch, ESummary, and ELink), or be combined with a search using AND/OR/NOT as described above.

For more information, see our documentation of EPost and its parameters.