Mike Davidson: Good afternoon, and thank you for joining me
for "Welcome to E-utilities for PubMed."
This is the first class in the "Insider's Guide
to Accessing NLM data" program.
My name is Mike Davidson; I'm a librarian
at the National Library of Medicine
here in Bethesda, Maryland,
and I'm here today with my colleagues
Pete Seibert and Kate Majewski.
Now, each of us have a few different jobs
here at the National Library of Medicine,
but we're all members of the NLM Training Team,
and we're all also members of the NLM Customer Service team.
So if you've ever written to NLM customer service
with a question about PubMed,
we may have been the ones writing back to you.
Before we get into today's agenda,
let me start off by saying that we know
our audience has lots of different levels of experience.
We expect that you all have some background
searching and working in PubMed, but we know that some of you
may have more technical expertise than others.
We're going to try today to offer a little bit
of something for everyone.
If you're more technically experienced,
and the stuff that we're covering at the beginning
is old news to you, please hang in there;
there is some more advanced stuff coming towards the end.
If you're a little less technically experienced
and some of the stuff goes over your head, please stick with us;
we do have some introductory stuff
and we try to sort of synthesize it all together
for those of you with a little less experience.
So for today's agenda, we're going to start off talking by
taking a look at some particularly challenging
PubMed questions.
We're then going to introduce you to a solution
that may help you answer those questions.
We'll provide a little background
on how that solution works,
to give you a good, solid foundation.
We're going to explore some practical
implementations of the solution,
including some tools you can use in the real world,
which will hopefully help us answer our questions.
And as I said just a moment ago, one of my jobs here at NLM
is on the PubMed Customer Service Team.
And our team over the last few years
has started to see an increase in a certain type of questions
that all follow the same general pattern.
How can I get a particular set of data out of
PubMed in a particular format?
So here's an example of one of those types of questions:
Which authors have a lot of recent publications
on a subject?
So, who's "hot" right now in a particular field?
And not just authors with a lot of publications overall,
but authors with the most recent publications.
Also, which organizations are funding research on this topic?
And I want this data in a format where I can analyze it,
or visualize it, like in Microsoft Excel.
Now, this can be a challenge in PubMed because,
while you can download results in a CSV format,
which you can open in Excel,
that format groups all of the authors
together in the same field.
You have to pull them out and separate them
in order to do any analysis.
That format also doesn't include funding agencies,
so you'd have to get at that information in a different way.
Here's another example.
Let's say we were trying to visualize trends on opioid abuse
broken down by specific types of opioids.
So, as I'm sure many of you are familiar,
this is a screen shot of a PubMed results page
for a very basic search just for "fentanyl abuse."
And if you look over here on the right side,
you'll see what we call the Discovery Bar,
which contains a lot of helpful information to help expand,
and provide context for, your search results.
And one of the features you'll sometimes see
is right up here at the top, this Results by Year histogram.
And I'm going to blow it up a little bit
so you can see it a little better.
And this chart displays how many of your search results
were published in each year.
And if this were a live demo of PubMed,
instead of a screen shot,
as I brought my cursor over these different bars,
it would indicate which bar represents which year,
and we could click on this Download CSV link
and pull the data behind this graphic to use however we want.
But now let's say I want to compare this trend
to a different search for "oxycodone abuse."
Now, again, we can get the histogram over here,
but if we want to compare that oxycodone abuse graph
with the fentanyl abuse graph,
we'd have to do a little bit more work.
While these results-by-year graphs are very useful,
they don't give us any options
for customization or advanced analysis.
Now, these are kind of advanced questions.
The overwhelming majority of PubMed users can got to
www.pubmed.gov and find exactly what they need.
PubMed's an incredibly useful tool
that I'm sure a lot of you use all the time with great success.
But as we've seen, for certain jobs,
advanced users like you sometimes need different tools.
It's all about having the right tool for the job.
So, as advanced users, what do you need your tools to do?
You need them to get exactly the data you need,
and only the data you need,
in the specific output format that you need it in.
And so, to that end, we want to introduce you to E-utilities.
Now, E-utilities is an API for accessing PubMed
and other NCBI databases.
Great; but what does that mean?
Let's start off with, what is an API?
API is a term that gets thrown around a lot,
but it's only rarely explained,
so if you've never used an API before,
or maybe you have used one but you still aren't 100 percent
sure what an API is, we're going to start off with a definition.
API stands for Application Programming Interface.
And an API is a set of tools, routines,
and protocols for building software applications.
Okay, that's a good start, but as a definition
it's still pretty generic.
So let's get a little bit more specific
about the E-utilities API,
which is set of tools, routines, and protocols
that allow you interact directly with the data
in over two dozen NCBO databases,
including PubMed, the MeSH database,
and PMC, or PubMed Central.
Okay, but what does that mean?
Really, it's simpler than it sounds.
The E-utilities API
is just a series of rules
for querying a database.
And for today, the database we're going to be
talking about is PubMed.
But as we mentioned, these rules work for querying
a bunch of different NCBI databases.
And for those of you who aren't familiar,
NCBI is one of the divisions
of the National Library of Medicine.
They do a lot of different things,
but one of their jobs is developing
and maintaining a variety
of databases of biomedical information, including PubMed,
and they have developed the E-utilities API
as a way of querying those databases.
The way you construct those queries is in the form
of structured URLs.
Each URL asks the NCBI server for information
and returns it in a certain format.
What information you get and what format it's in depends on
how exactly you construct the URL.
Now, in practice, you yourself
probably won't be constructing these URLs
on a regular basis.
In just a few minutes, I'll be telling you about a set of tools
that can create the URLs for you.
But in order to understand how E-utilities works,
and what your tools are actually doing,
we're going to go over the basics of creating
an E-utilities URL.
Now, this might seem a little daunting at first.
Because each E-utilities URL
is designed to ask a question to the server,
and there's basically an infinite number of questions
you can ask, there are just as many URLs you can create.
However, all of these URLs follow the same set of rules
and the same basic pattern.
So let's take a look at that pattern.
We're going to go into a lot more detail very soon,
but, in short, the three parts of an E-utilities URL
are the Base URL,
which is the address of the E-utilities server;
a Utility Name,
which is the name of the specific tool you are using;
and some Parameters,
which define the details of your query.
Now, in practice,
those three parts might look something like this:
The Base URL is this chunk of URL right here,
and that's always going to be the same.
We have the Utility Name,
which will be one of nine options
which look something like this,
and they all have this .fcgi extension
with the question mark on the end.
We'll talk more about those options in just a second.
And then we have the Parameters,
which define the details of your query.
Now, this part down here at the bottom
might look a little scary at first,
but once you start to learn the syntax,
you'll see that some familiar search terms and tags in there,
and it will get pretty easy for anyone who spent
any time using PubMed.
For example, we have a "Journal" tag right here.
We have a Boolean "AND" right here and there,
and I wouldn't get too thrown off by those pluses;
those pluses are just in place of spaces,
because you can't have a space in a URL.
Okay. So let’s go through these three parts one step at a time.
First is the Base URL.
As I said, it is always the same.
This is the beginning of every E-utilities URL,
and it identifies the address of the E-utilities server.
This is it; not much more to say about the Base URL.
But that's good, because our first part is out of the way,
so now we can move on to the second part,
which is choosing
which one of the nine specific utilities you'll be using.
Each of the utilities asks a different kind of question,
and which utility you choose
will depend on what kind of question you want to ask.
For example, ESearch searches a database
and returns a list of unique identifiers, or UIDs,
for the records that meet your search criteria.
So, if you're searching PubMed,
this would return a list of PMIDs.
Now, it's important to note here that
ESearch doesn't actually retrieve full
PubMed records; it just retrieves the PMIDs.
To retrieve full records,
we'd have to use a different utility,
like, for example, EFetch.
EFetch takes of series of unique identifiers,
like PMIDs, like, for example, the list of PMIDS
that you could get from ESearch,
and retrieves the full record,
including all of the data
in all of the fields for each of those UIDs.
Note that the records here that you're retrieving
are the exact same data
that's in the web version of PubMed, www.pubmed.gov,
so there are no differences in the content.
There are also seven other utilities
that each do different things,
but we're not going to get into all of them right now.
If you do want to see a full list,
and what they all do,
you can check out our Insider's Guide website
at dataguide@nlm.nih.gov.
The last part of a E-utilities URL is the parameters,
which, as we said, define the details of your query.
Which parameters you can use vary by utility;
not all parameters apply to all utilities.
Some are required, and some are optional.
One parameter that you will see in most E-utilities URLs
is the "db" parameter,
which specifies which database you're going to be querying,
because remember, you can use E-utilities
to query a number of NCBI databases, not just PubMed.
If you use the ESearch utility,
you'd also probably want to use a "term" parameter,
which defines your search string:
What will you be searching for?
Other parameters you might see, if you use different utilities,
include the "id" parameter,
which lets you specify a list of unique identifiers, like PMIDs,
that you might use with an EFetch.
The "retmode" and "rettype" parameters let you specify
what format you want your results returned in.
And I'll show you an example of that a little bit later on.
There are many, many more parameters than I have time
to show you right now, but, again,
there is full documentation available at our website.
Now that we've gone through these three parts,
though, we can put them together
and create an actual E-utilities URL.
So, again, our first part is the Base URL, always that same part.
Our second part is our specific choice of utility name;
in this case, it's going to be ESearch,
so we're searching a database,
and we have the .fcgi extension with a question mark at the end.
And then we have the parameters
defining the details of our query.
We have our "db" parameter,
which specifies which database
we're going to be searching on PubMed,
and we have a "term" parameter,
which defines, again, our search string.
And remember, those pluses are just in place of spaces.
And that's it. We've put the three parts together
and created an E-utilities URL.
And now that we've created this URL,
we can actually use to query the server
by just putting it in any web browser and seeing what happens.
So I'm going to hop out and demo this right now in a web browser,
and Pete is going to put the URL in a chat box as well.
But when you use these URLs, make sure you copy and paste;
don't just click on them.
WebX [spelled phonetically], for some reason,
doesn't like URLs that end in a bracket.
So copy and paste, and you should be fine.
I want to copy and paste, myself,
and paste it right up here into the URL bar,
under the destination bar up at the top of my browser.
I know it's really tiny in the WEBX window; I apologize.
I don't really have a good way of fixing that.
But, again, Pete has put that in the chat box as well.
And I'm just going to put that in there and hit Enter,
and I get my results.
As you'll note, we get a count
how many results from this database
match that particular search string;
in this case, it's six.
And we have a list of those six PMIDs right here.
Remember, this retrieves PMIDs only;
it doesn't retrieve full PubMed records.
And the stuff at the bottom that sort of looks like it might be
PubMed records is actually the query translation,
just as you would find in the Search
Details box of the web version of PubMed.
All of the query translation and Automatic Term
Mapping that happens in the web version of
PubMed also happens with ESearch,
and that translation is visible here.
Again, the exact query and what results we get
are determined by our parameters.
Now, in this case, we were searching for articles
on breast cancer,
published in the journal "Science," in the year 2008.
Now, if we wanted to change our search,
to search a different journal instead,
we could just tweak that parameter.
In fact, that's what I'm going to do.
I'm going to go up in my URL bar here
and leave most of the URL the same;
I'm just going to change the word "science" to "nature."
And when I go ahead and hit Enter,
I, of course, get different results,
because I'm looking at a different journal.
In this case, I get 12 results, and I get a list of PMIDs here
that match this particular search string.
So that's two ESearch examples.
Now let's take a look at an EFetch example.
And, again, it follows the same pattern;
we have the same three parts.
We have the Base URL, which, again, always the same.
We've changed our utility to EFetch,
so rather than searching,
we'll actually be retrieving records this time.
And our parameters, again, provide the details:
specifies the UIDs for the records we want to retrieve,
and the format that we want to retrieve them in.
So if I take that URL, and copy and paste that into my URL bar,
and, again, Pete will do that in the chat box,
we get a different type of result:
We get full records,
and we put in two PMIDs for our "id" parameter,
and we get one record right here,
and going down we can see a second one right here.
And they are in a text format,
and it's using the abstract view.
And if you've ever messed around with that format menu
in the web version of PubMed,
you might have stumbled across this format.
It's the exact same format, and it's the same exact same data,
that you would encounter in the Abstract
(text) view in the web version of PubMed.
Now, of course, we can change that formatted view
just by changing our parameters.
So I'm going to change this part at the end, here --
it's hard to see, but I'm going to change the "retmode"
and "rettype" at the end,
and they can just "retmode=xml."
And Pete will put that in the chat box as well.
And when I hit Enter this time,
I'm not actually retrieving different records;
I'm retrieving the same two records,
they're just in a different format.
In this case, they're in the full XML format,
which includes all the data structured out like this
so you can see everything that's a part of that record.
Now, we've just spent about 10, 15, maybe 20 minutes
to accomplish something that we can do in PubMed
in about five seconds.
And remember that what we said before,
it's all about the right tool for the job.
So, for some questions, going to
www.pubmed.gov is
absolutely the right choice.
But simple queries, like the ones we've been looking at,
are not really the point of E-utilities.
Where it's strongest is when you can combine multiple
URLs together using the results of one URL to create the next.
So let me give you an example of that,
and Pete will put this in a chat box as well.
This is another ESearch URL; paste that in there.
And this one is our "fentanyl abuse"
search from earlier,
and it results in about 830
PMIDs.
Now, to solve our problem from earlier,
we would then want to retrieve the full records
from these PMIDs,
so that we can look at the publication date,
and then pull out that publication date,
create that histogram,
and then compare that to our "oxycodone abuse" histogram.
Now, in order to do this, as I said,
the first thing we need is to retrieve full records,
and to do that, we need an EFetch URL.
Now, I'm going to type this EF etch URL right into the URL bar,
rather than copying and pasting, which, again,
you may or may not be able to see depending
on the resolution of your monitor;
but I will try to narrate as I go.
We're going to start, as always, with the same Base URL,
because, as we keep saying, that doesn't change.
Now we're going to be fetching records,
so we an EFetch.fcgi?utilityname,
and then we're going to add in our parameters;
we'll start with "db=pubmed,"
because we're fetching from PubMed.
Put an ampersand on there.
And then I'm going to type "id="
so we can specify which PMIDs we want to fetch.
And I'm going to go ahead and copy this first one,
paste that in there;
put a comma,
copy this one,
paste that in there; put a comma.
And this is clearly ridiculous.
There's no way that this is a good idea,
or in any way practical.
Even if it wasn't a terrible idea to hand copy 830
PMIDs into a URL bar, separated by commas,
remember that we would also have to do the same thing
for the "oxycodone abuse" search as well.
And this specifically is why we want to use E-utilities
in a programming environment.
By using E-utilities in a programming environment,
we start to unlock the real power of the API.
We don’t have to create these URLs by hand;
we can just write the rules
for creating the URL right into our program.
We can then automatically take the results of one query
to generate the next one, for example,
creating an EFetch URL based on the results of our ESearch.
We can also use programming tools to manipulate output,
for example, pulling out that publication date
and collating it together
to create the data for our histogram.
Now, ordinarily, in order to use E-utilities
in a programming environment,
you would need to complete a series of complicated,
time-consuming steps, just to get started,
before you could even get close
to actually answering your question.
But wait. We've already done a lot of that for you.
A convenient E-utilities tool kit
already exists in the form of EDirect.
Now, EDirect is software developed by NCBI.
It is a set of tools with the E-utilities URL creation
rules built in.
So you'll execute EDirect commands,
and EDirect will create URLs for you,
send them off to query the server,
and then give you the results.
EDirect also has tools to extract the exact elements
that you need from the full PubMed records you download.
And because EDirect works in a Unix environment,
there's not a lot of overhead;
it doesn't take a lot to get started.
But it also has an advantage of working with other tools
and programming languages so advanced users
can build more elaborate and powerful solutions.
Now, as I said, EDirect doesn't take a lot of overhead
to get started, but you do need to do a little bit of prep work.
In order to get started with EDirect,
you will need access to a Unix command line terminal.
Now if you're using a Mac or Linux computer,
your operating system already has one of these built in,
so you are ready to go.
If you're using Windows, you can install Cygwin,
that's C-Y-G-W-I-N,
which is a free Unix terminal emulator
that works in a Windows environment.
And that's actually what I do, and what my demos will be on.
Second, after you get that Unix command line terminal,
you'll need to actually install the EDirect software.
And there's a pre-written installation script
that you can get from our website
that runs in your Unix terminal,
which downloads, installs, and configures EDirect for you.
Now EDirect doesn't take a lot of prior knowledge to learn,
but it can help to have a little bit of Unix knowledge.
Or, you can take the route that I did when I was starting out,
which is having a knowledgeable friend
that you can pester with lots of questions.
Even if you don't have somebody like this
around your organization,
there is a lot of help online,
and it is actually pretty easy to find.
You can just go to your favorite search engine
and search for the word "Unix"
plus whatever it is you're trying to do;
somebody's probably asked that same question,
and hopefully got an answer that you can borrow.
One of the big advantages of EDirect is its ability
to extract specific data elements
from the PubMed XML format.
So it will help to have a familiarity
with the basic structure and syntax of XML
in order to get the most out of EDirect.
If you're already used to looking at PubMed data
in XML format, then you're good to go.
But if you're not, I would encourage you
to start taking a look at it.
You can always get the XML of a PubMed record
via the format menu in the web version of PubMed.
And by looking at that you'll start to get a feel
for the structure of the data
and where to find the thing
that you might want to be looking for.
It can also be helpful to have a little bit
of programming knowledge,
especially Perl or Python.
But this is absolutely not required.
However, the more that you know about Unix,
about XML, or about programming,
the more you'll be able to do with EDirect,
though a lot of this you can absolutely learn as you go.
So let me show you what EDirect looks like in action.
As I said before, EDirect works in a Unix environment,
and what that means is that we're going
to use this "terminal,"
in this case, Cygwin, to interact with our computer.
And this is our command prompt, right up here,
and we just type in the commands that you want to run
and execute them directly from here
and get our results back in our terminal.
So, for example, if I wanted to search PubMed,
I'd type in an ESearch command,
in this case, for "fentanyl abuse,"
hit Enter, and get my results back in XML.
Now, as we said at the top,
we know we have people attending today
with a wide range of experiences.
And I want to give those of you
with less programming experience fair warning.
As you've already seen a little bit,
and in the next couple of minutes,
we're going to start showing you some things and throwing around
some terms that might be a little scary,
might be a little outside your comfort zone.
Please don't panic.
Even if you're less technically proficient,
if you're willing to put in a little effort,
I promise you you will be able to pick this stuff up
and put it to use.
Over the next couple of slides, we're going to try and show
some examples of E-utilities and EDirect in action,
and what it can do for you.
If you've got a little bit more of a technical background,
some of this stuff will be old news to you,
and you'll start to see right away
how you can get started with E-utilities.
If this is all a little newer to you,
don't worry too much about the details
of how we did all of this stuff.
For now, just take a broader view of the kind of things
that we can do with E-utilities.
If this seems like something that might be useful,
you can take one of our more hands-on, interactive classes;
you can read the documentation available
on the Insider's Guide website;
or you could find the more technically-inclined person
at your organization
and have them help you through the early stages.
Now with all that said, let's take a look at the way EDirect
can quickly and seamlessly put E-utilities to work for us.
So, as you might have noticed,
EDirect commands look a little bit different
than the URLs we constructed before,
but these commands are really just a faster,
simpler way of creating those URLs.
Behind the scenes, this command on the screen
is actually creating this URL.
And if you look closely, you can see some of the similarities.
This ESearch up here is saying which utility we want to use,
just like we do in our URL over here.
The db argument says to use "pubmed"
as the "db" parameter for our URL.
And here's our search query, "fentanyl abuse,"
which is going into the "term" parameter.
It works just the same for EFetch.
So here is the command, and here's the URL.
And, again, we can see that the "efetch" command
tells us to use the "efetch" utility.
The db argument tells us to use pubmed.
We have an id argument that tells us
which PMIDs we want to fetch.
And we also have a format argument, which creates a URL
that uses the "retmode" parameter XML
to get us XML records. And when you run these commands,
you won't actually see these URLs;
EDirect creates into the background, queries the server,
and just returns you your results.
But even this doesn't get us any further than we were before.
When we had to create a URL,
then take the results of that manually
and feed them into the next URL.
Where EDirect really gets good is when you can put
multiple commands together.
Now, these two little characters here at the end tell us
to take the results of this ESearch command
and pipe them into our next command,
which we can find on the next line.
Now, you'll notice that this EFetch on this second line
is missing a few things.
It doesn't have a DB argument
because it already knows what database we're in.
We're in PubMed. We already told it that in the ESearch.
It also doesn't have an ID argument,
because it knows to take the PMIDs
that were the output of the ESearch command
on the previous line.
Now, these two lines together allow us to search PubMed
and then pull the complete XML records of all of our results.
And now that we can do that, we’re really ready
to start answering some of our questions.
We’re going to look at a few of the examples we talked about
at the beginning of the session.
And again, don’t worry about following along
with every detail right now.
This is just to give you an idea
of what E-utilities can do for you.
I wouldn't even pay too much attention
to the details of the code.
All of this sample code from our examples today
is available on our website, fully-annotated and explained,
so you can review it, tweak it, reuse it,
or repurpose it for whatever you need.
So, remembering back to our earlier examples,
we were trying to find authors
that are especially prolific in the last few years
on our chosen topic.
So, again, we're going to start at a Unix Terminal,
and then we'd put in our series of commands.
Now, you can type them in by hand if you want,
but I usually write them out in a text editor
and then copy and paste the whole script
into the terminal, like so.
Again, we're not going to go into details,
but you'll see some familiar elements.
We have an ESearch command right up here.
Our DB is PubMed and here's our search query.
We have a topic string, and it's Boolean ANDed
together with a date restriction.
We're then piping this into an EFetch to get XML
and then piping that XML into our next line.
And we haven't talked about this xtract command just yet,
and I'm not going to go into a lot of detail on it right now.
But it is a super useful command that is built into EDirect.
It extracts specific elements from XML output
and then rearranges them into a table.
It's very customizable.
You choose the output, the arrangement, the separators.
And this xtract command is one of the things
that makes EDirect really useful.
It is a little outside the scope of today's class,
but if you take our EDirect for PubMed class,
we spend a lot of time
talking about xtract, like a lot of time.
Like three out of the five sessions
are just about all of the cool things you do with xtract.
After the xtract, we have a few more lines of post-processing,
which makes sure we're getting the data
we want in the format that we want.
And once we have all our commands in there,
we just hit enter, and get our results.
Now, normally, this would take longer
than just an instantaneous click.
This is the magic of screenshots.
But it would have to query the server
and do a little processing,
so it would take a few moments to finish.
But we can see what the output would be,
which is a list of the top 10 authors
who have the most publications on this topic
in the last few years.
Now, our next example,
which was pulling out the list of the agencies
that have funded research on this topic,
is very similar, actually, to our first example.
Starting with our blank command prompt,
we put in our pre-written code
and we can see that the ESearch and EFetch lines
are exactly the same.
We're retrieving the same records.
The only difference is which
elements I'm extracting
from those records. And when I run this code,
instead of getting a list of the most prolific authors,
I get a list of the top 10 agencies
that are most frequently listed
as funding research on these records.
Now, if I'm interested in researching this topic,
these might be the agencies that I'd want to target
with grant requests.
Now, for our last big demo, we're going to try
and do a version of that Results by Year histogram,
but comparing our two opioid abuse
searches on the same graph.
And this is just an example of the kind of thing you can do
with E-utilities to give you a little more customizability
than you'd get in the web version of PubMed.
And for this demo,
I'm actually going to demo it live, so bear with me.
Because who knows exactly what's going to happen?
Hop out here and get to my command prompt.
Again, so, this is my command prompt.
And I've already pre-written my code in a text editor.
And let me actually bring that over
so you can see that as well.
So, this is what the code looks like.
And you'll notice that it is quite a bit longer.
And I will talk you through it in just a moment.
But first, I'm going to copy
and paste it into my command prompt to get it running
because it's going to take a little while to finish.
And as I paste it in,
you'll notice that only part of the code
is pasted in right away.
Because this is actually a series
of three separate mini-scripts.
And Cygwin is showing us the first part of what we pasted,
waiting until that's finished,
and then moving on to the next part.
Just like before, we're running a search,
retrieving full XML records,
and extracting data from the XML.
But rather than outputting the results right on our screen,
we're saving those results to a file.
For this example, I'm actually limiting the results
to a date range starting in 1988,
which is the time frame I'm interested
in looking at for this particular example.
For each of these blocks, there's also a few more lines
of code
to get the data into the correct format before we save it.
Then, of course, we have to do that same thing over again
for the oxycodone abuse search.
And finally, we have the last line here,
which will merge the two files together.
We can then open up that text file that we've created --
that "abuse_compare.txt" --
in a text editor and see what it looks like.
And I'll actually do that right now and there it is.
So, there is our result.
We have a list of years
and a list of the number of years results
published in each year for the search "fentanyl abuse"
and "oxycodone abuse."
We can also open that file up in software that can analyze
or visualize data, like Microsoft Excel.
And I'll go ahead and do that right now.
And I'll make that a little bigger, get that file open.
There it is, "abuse_compare."
And Excel has this fancy Text Import Wizard,
where if I just click right through,
it will give me exactly what I want to see.
And let me make that a little bigger, so you can see.
It's the same results that we had in that text file.
We just open them in Excel, so now we can use that data
as the basis for a chart
that lets us compare the two histograms.
Now, I've actually made this chart ahead of time --
very cooking show of me.
This is the one I've prepared ahead of time.
And we can see, using that same method to get the data,
this is the graph that I created based on it.
And as I said before, this is just an example,
and it's a pretty scaled down one,
so it would run fast enough to complete during this session.
You could use a different search
or extract different data elements.
You could also style or visualize this data
completely differently.
For another example, I have a chart here of
PubMed results for "heroin addiction"
in blue versus "opioid addiction" in orange.
And so, this comparison also might tell an interesting story.
Because we're using E-utilities,
we've got a more flexible
set of tools at our disposal.
Okay, before we finish up today, I want to go back to the slides
for just a minute to zoom
back out a little bit to the big picture.
So, the whole point of all of the things
that we've been talking about today
is figuring out different ways to access PubMed Data.
And as we've seen, the E-utilities API
is a great channel that you can use to get at that data.
And EDirect is a tool that can make using E-utilities easier
if you're working in a Unix Terminal environment.
However, if you're working in a different environment,
you will probably want to use different tools.
For example, let's saying you're working
with the R programming language.
Now, some of you may be familiar with R.
It's really good for doing statistical analysis
and data visualization.
Let's say that you or someone in your organization
wants to do some analysis or visualization of PubMed data,
and you want to use R to do it.
Now, in order to get at that PubMed data quickly and easily,
you'll want to use the same E-utilities API,
but because you're in a different environment,
you'll want to use a different tool.
EDirect works in Unix. It doesn't work in R.
In R, though, there are a number of packages of commands,
kind of like EDirect,
but designed to work specifically
in the R environment.
These packages, like RISmed, rentrez --
and there are several others, as well --
help make it easier to use the E-utilities API
to access PubMed data from within the R environment.
Now, if you're working in a different environment,
not Unix or R, you might have different tools altogether.
Maybe you'd even have to build your own tools.
But you'd still be using the same E-utilities API.
And in some environments, like a web browser,
you might not use special tools at all.
We can just use the URL creation rules
to use E-utilities directly.
Now, this is what makes E-utilities
so useful and flexible.
You can use E-utilities to access PubMed data
regardless of what environment you're working in.
You just have to find the right tool for the job.
Now, hopefully, today's session introduced you
to the basic idea of E-utilities and to EDirect,
but we know that there's still a lot more learn.
And fortunately, this is just the beginning
of the Insider's Guide.
If you're interested in something
a little bit more interactive,
we encourage you to sign up for one of the other
Insider's Guide classes.
EDirect for PubMed is more of a hands-on workshop
that introduces you to the basics of using EDirect
to get exactly the PubMed data you want,
in the format you want it in.
It is a five-part series,
with an additional, optional office hours session.
And that starts March 5th.
The next session of that starts March 5th.
You can sign up to register right now
and Pete's going to put a link in the chat box
to that registration page.
In addition to EDirect for PubMed,
we do hope to develop further classes
based on what we hear from you.
Please let us know via the evaluation form
that should pop up when you leave the session.
And stay tuned to that Classes page
of the Insider's Guide website
to get more announcements as they come out.
If you are interested in continuing to learn
about these tools, if you want to take our class
or even if you're not interested in that,
I would encourage you to check out the Insider's Guide website
at dataguide.nlm.nih.gov.
And I have a few minutes, so I'll give you a quick tour
of that, as well.
You probably saw it already when you signed up for this class.
But this is what it looks like.
There is, of course, the Classes page right here,
where you signed up for Welcome to E-utilities for PubMed.
I click on that and I get the --
this is where the recordings will go.
So, this is the recording of the last time we offered this,
but the next time will be appearing right here
as soon as it is edited and ready to go.
There's also a Class Materials page right here.
All of the sample code from today's class, the slides,
and the script are all available via that Class Materials page.
Now, if I go back up here to the top,
there's an E-utilities documentation section,
which again, gives full documentation
on all of the utilities and parameters
that you're likely to need when using E-utilities with PubMed.
And we also have the EDirect section here
with the Installing EDirect page.
That's how you would get
the detailed installation instructions
for EDirect.
So, in addition to that,
which should be plenty of documentation for anybody,
but you can also check out NCBI's documentation
for E-utilities and EDirect.
The links are here on this slide.
While we mostly focus on the use of these tools
to access PubMed data,
NCBI's documentation is much broader,
as it covers all of the different databases
that E-utilities works with.
Now, before we finish up today, I just have one more example
that I want you to think about.
So, say we want to know what topics our organization
is publishing on.
Now, our organization has many authors
and many different research components and divisions
and labs with a variety of names and acronyms and abbreviations.
So, how can we efficiently search for our organization
and analyze the topics of published articles
to get a sort of portfolio review for our organization?
Now, this is the kind of question
that we get at Customer Service quite frequently,
and it is absolutely doable with E-utilities.
I'm not going to demo it right now
because we don't really have time,
but we do have some annotated sample code on the website
that I just showed you that you can check out.
If this is the kind of question that you have,
or folks in your institution have,
I'd encourage you to take a look at our sample code
and see if it's something you could adapt to meet your needs.
Even if this isn't something that's exactly relevant to you,
I would consider taking a look at the code anyway.
See if it is something that you could adapt into something
that would be more relevant to you.
This might be similar to a problem
that you've had trouble solving in the past
because you didn't have the right tools for the job.
And I'm sure many of you have some jobs that E-utilities
is the right tool for,
so I encourage you to get out there
and put E-utilities to work.