welcome back to EDirect for
PubMed.
My name is Mike Davidson.
I'm a librarian at the national
library of medicine in Bethesda,
Maryland.
This is the final of five
sessions in the EDirect for
PubMed series, so thanks so much
for sticking with us so far.
Today is really about putting
together all of the people we've
put together so far and talk
about how we can develop and
build solutions to real world
problems.
For today's agenda we're going
to start off with a quick recap
of part four.
We're then going to spend a few
minutes discussing how we can
put all of the tools we learned
about to use and go over some
methods for developing
solutions.
Then we're going to finish off
and spend most of today's class
with a deep dive into a few real
world case studies with a look
at the whole process from
beginning to end.
So for our recap, hopefully you
remember that last Thursday we
spent pretty much all of the
class talking about extract
conditional arguments.
We use the if argument to limit
our output based on if a pattern
or block met a certain
condition.
All by itself limit our output
base on the presence of a
particular element or attribute.
We use if when the equals
argument when it equaled a
particular spring and then use
if with contains on whether an
element contained a certain
string.
We also talked about the two
Boolean conditional argument,
the or required at least one of
the conditions we specified must
be met and the and argument
required that both of the
conditions must be met.
We did a quick discussion of the
position argument and let us
include a specific block in our
output based on its position in
a pattern so whether it's the
first block or second block or
last block etc.
Finally we talked about one
argument that isn't one of the
conditional arguments, the deaf
argument which lets us specify a
place holder for whether there
would be blank cells in our
output table.
I'm going to pause for a moment
and take a look to see if we
have any questions.
Looking clear so far.
Okay.
Again, feel free to keep sending
those questions to all
participants and we will address
them from time to time to the
class but let's get moving and
talk about sort of where we've
come over these last four
sessions.
We spent a lot of time
introducing you to new tools and
techniques.
Some of these tools are new ways
of doing things that we can
already do in PubMed such as
Esearch which is a search of the
data base and the Efetch which
includes XML.
Some of the new tools click
Xtract which can extract
specific data from XML and
arrange it in a specific table.
How do we actually do it?
How can we put these tools to
use.
Learning how to use these tools
is only part of a challenge.
You need to figure out when to
use which tools and in what
order?
And a lot of this comes from
practice, experience.
I can offer you some advice
based on the hours and hours I
spent trying and failing and
ultimately succeeding in
building scripts to do specific
tasks.
I'm going to walk you through my
process but these suggestions
may not work for everyone and
that's okay.
If you find a different way of
working through this process
that works for you, I would
encourage you to do that.
That said, here are some of the
things that I think about when
I'm trying to develop a solution
in EDirect.
Before you start to build a
script it can be helpful to
think about your goal and make
sure you have a good idea of
what you're trying to
accomplish.
And the first thing you might
want to do is identify your
input.
In other words, what do you
already know?
Maybe you have a PubMed search
strategy, maybe you have a list
of Pmids in a while that you
want to analyze.
What type of input you have is
going to determine how you build
the first steps of your script.
Next identify your output.
So what is it that you want to
know?
What should be the end product
of whatever process you come up
with?
Do you want a list of Pmids.
Which field specifically are you
trying to output, author name,
mesh headings, DOIs?
What is it you want to know.
Then identify your output
format.
How do you enter your data
arranged.
If you want to table instead of
what order should the columns be
in that table?
Do you want to save your output
to a file?
If so, what type of file?
Figuring out these details will
help you refine and polish your
script to get the results in the
exact same format that you need.
Now we just spent four sessions
teaching you EDirect but you
want to make sure EDirect is
actually the right tool for the
job.
When I go out and learn about a
new tool, I'm always very
excited and I'm tempted to go
out and look for projects.
You want to make sure the tool
you choose serves the goal of
the project, not the other way
around.
EDirect can be very useful but
it isn't always the most
practical solution to a problem.
If you can go the job faster by
going to PubMed and getting your
answer that way, them you should
absolutely consider that option
instead.
Like ways if your project
requires access to an incredibly
large amount of data, like, for
example, tens of thousands of
records, you may also run into
trouble.
To avoid overloading the
servers, NCBIasks limits to
limit usage to offpeak hours.
You can find more on NCBI's
website.
The guidelines are a little
dense so I suggest you wait
until after class to take a look
at that.
Incredibly high volume scripts
take a lot to run.
I would direct you to the NLM
data distribution program which
provides bulk amount of records.
Until recently we used to
mention the NLM distribution
program as an alternative who
folks who want to work with the
entirety of PubMed.
Researchers who had the
infrastructure to take the bulk
downloads, pull out the XML and
file into their data bases so
they can work with them.
We know that not all EDirect
representers have their
resources.
However, starting with feet 8.0
there's a new feature that has
the best of both worlds.
It puts it to work with the
entire methods of PubMed.
This also you build a local copy
of PubMed on your own drive so
you don't have to wait for
Eutility to fetch these records
at the same time.
Rather than fetching the XML for
Eutilities, it can fetch if your
own drive.
Now there are a few caveats to
this process.
First in order to actually see
any speed benefit, you need to
have an external solid state
drive.
And you'll need that drive to be
at least 500 gigabytes.
It could take anywhere from six
to over 30 hours the first time
you build it.
As you can see, this might be a
useful option where you're
analyzing huge amounts of data.
For any projects where you're
retrieving less; however, if
you're interested in pursuing
this, there are some links in
the hand out.
I'll be happy to address
questions on it either during
the end of today's class or on
Thursday during office hours.
There is one other way to get
the best of both worlds.
You may also want to remember
that Xtract works on any XML
using the input argument.
Even if you don't get your XML
from Efetch but get it from the
download option or from some
other source, you can still use
all you know about Xtract on
that FML.
So while we're talking about
data, you want to make sure that
just pause we're using an
exciting new tool you don't
forget everything or ignore
everything you know about PubMed
data.
The solutions are only going to
be as good as the data you're
retrieving and as good as your
knowledge of that data.
If you remember back to the
second session of this class,
Sarah showed you how there are
nine different year elements on
a single record and all of them
mean different things.
In order to make an informed
choice, we needed to know the
difference between the different
dates PubMed uses and recognize
that public date was the right
one nor that situation.
Keep in mind that data is or
isn't available in PubMed.
For example, PubMed doesn't have
full text, only citation to see
articles.
In additionally not all fields
are populated for all records.
Also as we've talked about
previously, mesh headings are
only attached to Medline
records, and Medline records
make up the bulk of PubMed but
because it takes time, you'll
find that a lot of recent
publications don't have mesh
headings because they haven't
yet been fully indexed.
You'll want to of course about
what that data looks like.
For example, author affiliation
data is supplied by the
publisher and is not controlled
or reviewed by NLM.
Understanding the data will help
you figure out what is hopping
and will also highlight issues
that you need to address.
Bottom line, the better at
searching and work in the web
version of PubMed you are and
the better understanding you
have of understanding PubMed
data, the more effective your
Escripts will be.
If you want to improve your
searching skills, NLMoffer a
variety of tutorials and
training opportunities, and if
you want to get more familiar
with the data remember that XML
documentation as Sarah showed
you in the second documentation.
That's a great place to start.
One of the advantages of EDirect
as we talked about is it lets
you automate jobs.
As we mentioned you want to
consider whether the efficiency
you gain is worth the time
investment.
One area where you want to
schedule how much of your time
investment is worth it figuring
out when to stop development.
Many times with EDirect their
always multiple ways of doing
what you want to do.
There's almost always a way to
accomplish 100% of your goal una
single script.
How far, there are usually ways
of accomplishing 90% or 50% of
your goal and doing the other
10% manually or using a
different tool like excel.
So you'll have to decide whether
the additional time andest it
will take to get your script
from 90% or 100% is more or less
efficient than doing the
remaining 10% manually.
This brings us another . is
sometimes a solution will
require actual human judgment
for evaluating results,
determining relevance or
determining some of the data
issues that we were talking
about before.
Figuring out if and where a
human needs to fit into your
task should be something to
think about the development
process.
Before we go on, I want to under
line the 90% to 10% process and
it's something I think about a
lot.
If your 90% solution saves you
time, them you might not need to
go further.
It's another way of saying don't
let the perfect be the enemy of
the good.
It is definitely worth
reiterating here.
Go slow.
Build one step at a time and
then combine things.
You'll be much better able to
spot issues in your development
testing and troubleshooting as
you go.
To show you more of hey mean,
we're going to spend the rest of
the session look at case studies
but before I go I want to touch
base to see if there are any
questions that need to be
addressed.
Hang on one second.
Let me just pause here and take
a look.
It looks like we do have a
couple of questions in there but
it looks like we're going to
circle back to those a little
later on.
So for our case studies today,
we're not just going to break
down the code of the finished
product.
Instead we're going to develop
each solution from the ground up
looking at how we choose, build
and test each part, each
command.
And for our first case study,
we're going to start thinking
about our goal.
In this case we want a list of
articles about breast cancer
that were published in the last
year in our link to entry in
clinicaltrial.GOV.
For each article we want the
PMID, NCTnumber or numbers if
there are multiple ones, the
first author's name and journal
that that right article was
published in.
That's our over arching goal.
We'll start with input and it's
going to be a PubMed search
string.
The search string that I'm going
to use is very simple, breast
cancer and clinicaltrials dove.
So we'll look for breast cancer
as a subject and
clinicaltrials.GOV in the
identification field.
For my purposes this is going to
work fine.
We know that we want just the
last year so we're going to
limit it by date from March 2017
throughout end of February 2018.
Search string, date restriction.
This to me sounds like we're
going to be starting with
Esearch.
So now that we know -- we know
what we know, we know our input,
now we need to know what we want
to know, identifier output.
We want PMID and we should all
know by this point that we need
to specify the correct PMID
using parent-child construction.
This one we had to look up.
The asession number element
which is a child of the data
bank elements.
However, the data bank elements
is a little tricky because it
can also contain links to other
data banks.
We will have to make sure that
we only get session numbers when
their NCTnumbers and not other
numbers.
That's our first two.
We still needed two more.
We wanted first aller.
For that we'll take last night
and initials but only for the
first author.
This is going to be a good
opportunity to use that position
argument we were talking about.
Finally journal which we used a
couple of times over the course
of these sessions is Iso
abbreviation.
Now let's talk about your
format.
How do we want our output to
look?
Each article is going to have
its own row and then four
columns.
We're going to have PMID, then
the journal name and then the
NCTnumber or numbers.
The reason I want to put that
NCTnumber last is it's
repeatable.
I want everything else to be in
nice, neat columns.
Separate the multiple NCTnuns
is the record has more than one.
Finally we want to save this to
a file so we can open it to
excel, read and visualize it for
specifically so that's another
aspect of our format we have to
talk about.
So now that we've thought about
our goal, thought about what
with want, what we have, let's
actually build this thing and so
I'm going to hop over into my
terminal here.
Make sure that you all can see
it.
A little bit bigger.
That's pretty good.
Okay.
So let's start off with the easy
part.
We know that we're going to need
an Esearch.
Esearch is pretty easy.
It's easy to build, easy to
test.
So let's start off with that.
Esearch, DBPubMed query.
Just going to put in any search
string.
Clinicaltrials.GOV.
Close by quotes, run that.
So 1465.
Now I'm going to pop over to
PubMed real quick and make sure
that's the same as I get over
here.
Obviously it should be but it
doesn't hurt to test.
You can put it in the search
string itself or use the date
arguments.
I'm not going to use the date
arguments because I'm going to
be messing with my dates and
tweaking them a little bit here
and there as I go to get smaller
sets of data.
But I'll start off with the real
date restriction I will be
using.
We wanted 2017/03/01.
I think that's right.
Looking good.
Run that.
So now that's down to just the
number.
So will the me change it to from
2017 to the 2018.
Now I'm down to 50 records.
And actually the first thing I'm
going to do -- I'll put it on my
screen and take the same test
search that I just ran, that
should gave me about 50 records.
But before I go and just back a
bunch of XML, so I'm just going
to grab Pmids.
I'll use format UID, and that
will give me the list of UIDs
that is retrieved by each
search.
It is a decent list of PMIDs.
Much like we've done in previous
examples is I would build a
testing Efetch.
And then I would copy and paste
a couple of these Pmids but into
my testing command here and grab
the XML just of those few
records.
That is what I do.
And so mine looks like this.
Doesn't quite get one line,
that's okay.
And if I run that, I should get
a whole bunch of XML, goes by
really fast.
We're going to use these few
records for building our Xtract
command.
Cut out one of my Pmids, make
day little bit more manageable
on this zoomed in screen.
That's better.
Not going to quite fit.
Okay.
So now I need to build my
Xtract.
We know that we want a list of
citations or pattern, going to
be PubMed articles.
And a couple of these elements
are pretty easing too.
We know that we want PMID which
I need to be like that to make
sure we get the correct PMID and
Iso abbreviation, also pretty
simple.
When I execute that, we get --
okay, so we get our two elements
in our two columns.
Doing good so far, two columns
out of the way.
Now I'm going to copy and paste
that part back in.
There we go.
Make my Efetch even a little bit
shorter.
There we go.
Now we need to work on the first
author.
Remember to use first author we
need to use position.
We're going to use block author
to better explore our author
information.
We're going to look through one
author at the time but only use
the first one.
So we block author and position
first.
I can use position one as well.
And then the first aller
information that we needed was
last night and initials but I'm
again going to use the trick.
I'm going to put those two in
the same column separated by a
space so it will format just
like it formats in PubMed with
the last NATO space initials.
When I execute that.
My column alignment is a little
off here because of the way the
tab spaces and journal names are
not all the same length but
that's okay.
We're getting three column here
are the data that we want.
Okay.
So now we need to do the harder
part which is the NCnumbers.
I'm going to take the same batch
of stuff I just had.
There we go.
Now we can see it better.
And now we knew that our session
numbers are going to be within
the -- our session number almost
within the data bank element but
we only want some of those data
banks, not all of them.
So I'm going to use another
block to explore data bank by
data bank and then our element
is going to be a session number
but we don't want every session
number, right, so we need to go
in and pick a session number
here.
Them we're going to have our
element session number and
actually I'll going to throw in
another separating here.
We want it separated by spaces.
Let's see what that looks like.
Did I get it all?
Okay.
I got it all.
I don't know for sure that that
stuff work at session numbers
because I don't have any session
numbers here.
I think this is if.
PMID, journal name, aller name.
So now we can sort of put all of
this together.
I'm going to go ahead and put it
all together with the Esearch
and Efetch, run it on the whole
set of data.
I'll clear my screen, going to
copy that whole thing.
So we have this, so we have our
search, we have our date
restriction, we have our
fetching XML, pattern, first
aller name, there we go and our
session number's right there.
But we wanted to save this to a
file.
I'm going to save this to our
greater plan sign.
Go ahead and run that.
It's 500 records should take a
little while but not super long.
Okay.
Good.
And that's all set and I'm just
going to go ahead and use that
LFcommand that Kate showed you.
And there it is right there.
There's our file and we can go
hop into excel and make sure
it's giving us what we want.
We use the text import wizard to
make sure we're tabbed limited
here.
Finish that.
Zoom in a little bit.
And it's readable.
Make sure our columns are spaced
out so we can see everything.
Honestly, it looks like we did a
great job.
We have PMID, journal title
abbreviation, author last night
and initials and NCTnumber and
a great example with the numbers
separated by pipes.
That's it.
We've accomplished our goal.
We have the information that we
need.
Are there any questions about
that case study?
I'm not seeing any right now.
I know I'm going through this
kind of fast.
Again, a lot of this does take
time to develop these solutions
which again you'll only
experience with more practice.
But speaking of more practice,
let' take a look at another case
study, a little bit of a
different vibe at this time.
Don't need this anymore.
Go back to my slides.
Amy is saying it always makes
sense of you do it but it takes
me five times as long to go
through my thought process.
Don't expect you're going to be
solving these at the rate I'm
presenting.
I'm still learning and trying to
figure out the best solution to
some of these cases.
Let's take a look at this next
case.
One common question we get is
about analyzing authorship and
co-authorship.
In this case we have a author,
Ericspace Peterson.
That's ger I go, E-R-I-C.
I mention that because it's
going to be important later.
I did a little research on this.
He's at duke.
Looks like we have a little bit
of lag in the video.
Hang on a second.
Let's just try and get this back
here.
Our video is lagging out a
little bit.
Hopefully his web page will come
up in just a second.
There he is.
Now we're caught up.
So this is Eric Peter sun.
He looks like he specially eyes
in cardiology at duke and if we
go down here to the list of
publications, we see a handful
here and then we see a whole
bunch more page.
This is a very prolific fellow.
And we want to know who he's
worked with and how frequently.
So that's sort of our general
objective here.
It's our mission.
Okay.
My video's caught back up, which
is great.
So again following our input
outpit put formatting structure,
in this case it's just a name,
Eric DPeterson.
That's what we know.
Our output in this case is a
list of coauthors.
Ideally with their affiliations
and number of papers they've
co-authored with him.
This is more exploratory than
the last one.
We don't have concrete goals but
sort of a list of co-authors
with the number of people that
co-authored is a good starting
point.
We'll see where this goes and
one of the reasons we're looking
at this particular case study is
while that structure that I told
you about can be useful, don't
feels like you always need to
stick to it.
You will find things that you
wouldn't get otherwise.
And before we go any further,
there is a word, I do want to
have a word about the data here.
Author disam biggation is a
tricky topic in PubMed.
It's up to the user to make sure
that you're looking at the
correct author.
We'll have to find some methods
to ensure that our data contains
only the correct Eric
DPeterson.
PubMed doesn't have everything.
Not every article in the world
is in PubMed so you have to use
caution when you're doing true
bibliometric research.
Finally affiliation data as we
mentioned before isn't
controlled.
It's sort of free text submitted
by the publishers, not really
reviewed on the NLM side so
we're going to have to contend
with that as well.
But let's again get building and
see how we go.
I'm already here.
Clear my screen.
So we're going to get started
with a basic Esearch just to
sort of see how many records
we're dealing with and how big
of a case this is.
So I'm just going to start with
the basic Esearch, PubMed,
query, Peterson, EDtagged with
AU.
That's the way that you make
sure you get most possible
results for this author because
both full names are not
necessarily required and up
until the last ten or 15 years
only the first author could even
have their full name submitted.
So let's run this.
Dealing with 1200 records.
That's not a lot for us to deal
with overall but a lot to deal
with for a demo some of these
searches are going to take a
little while to run.
Now we're going to try to do
some disambiguation to see who
are our Eric DPeterson and who
is unaccounted for.
We'll do our search, type that
into an Efetch, grab some XML.
We're going to take a look at
each of these 1200 citations,
figure out who the Peterson
EDlisted on them actually is.
So we'll start with pattern
PubMed article and we'll start
with PMID.
It's a good starting point.
And now we want to find the
author information for the
Peterson EDthat's on each of
these records, and this is a
little bit of that tricky
exercise we did towards the end
of last class.
So what I'm going to do is use a
block to look through each
author on each record but only
pull out the author information
for Peterson ED, and so the way
that we did that last time and
I'm going to do the same thing
that I did last time here is
with a condition.
If last name equals Peterson and
initials equals ED.
So that should only give me
author information for the
Peterson EDon each of these
records.
I limited myself to that, I
needed to decide what
information I'm going to
actually look at for ed Ford to
disambiguate.
Identifier is an obvious choice,
if it's present, so we'll
definitely do that.
But before that I want to do
some same stuff.
We'll put last name and forename
because forename is the full
first name, if it's present.
Affiliation could help us as
well.
I'm going to save that for a
little bit later.
While this is processing, again,
what we'll hopefully see here is
some very easy first steps that
we can take to correct whatever
is in this result that is not
what we want.
As I said, 1,200 records takes a
little while to run.
Asking and should not be
capitalized.
Yeah, should not be capitalized.
All of the stuff that comes
right after the dash, all lower
case.
Okay.
So here's our result set.
So we got a bunch of Peterson
ed.
They have no forename, no
identifier.
We can't really do much about
those without looking at each
one.
Let's go way up to the top
first.
One thing I'm not seeing here,
except for right there, are
orchids.
So orchid are kind of a
relatively new introduction to
the data base.
So they are not required, not
every publisher submits them,
not every author has them and
even authors who have them now
didn't have them earlier in
their career.
That's not super helpful.
We'll take it off on the next
draft.
We have a bunch of Eric Dspace
Petersons.
Keep on moving for now.
We have a couple of blanks here.
Let's take a look at one of
those just to see sort of what's
going on with that.
Hop over into PubMed and look at
this one record because, based
on our search, each record
should have an ed Peterson.
But we need to figure out why
the ed Peter sun is missing for
the output up these records, and
if we look at the collaborator's
link right here, we see he
listed her as a collaborator.
The author tag for searching
does by default includes both
authors and collaborators and
investigators but our if
condition was specifically look
at the author Lexington.
That's a little tricky.
I did see something else that
needed a fix as well and it's
right down here, Eric can a K,
probably not our same Eric
Peterson but let's take a look.
Grab that PMID, hop over into
PubMed, pull it up and, yeah,
this looks like more orthopedic
rather than cardiology related
so this E-R-I-KPeterson is
probably not who we're looking
for.
That would be a good next step,
get rid of the ERIKSwith a K.
I did make a note of that PMID.
That's the PMID.
We're going to take that basic
structure and piece that in and
copy it.
Sorry.
There we go.
So this is what we had last time
but I'm going to add in another
condition pause we want to make
sure that forename is not Eric
with a Kspace D, forename is
not Eric can a K.
When I was building this case
study, I actually got a little
stumped here and I went to look
at the documentation, and I went
over to a documentation site and
I went to EDirect documentation.
I knew that I was building a
condition.
I was building something to
limit my data so I went down to
Xtract to see if there was
anything useful that I could
find in here.
I was scrolling through here and
I came across this is not
argument.
That's exactly what I want,
right?
So I'm going to hop back over
here, going to add in and and
replace my if or in addition to
another one forename is not Eric
D.
Go ahead and run this and we'll
see how it goes.
Still the same number of
records.
The ERIKwith a Kthat we want
to get rid of.
That's not right.
It's still in there.
We blanked out the Eric Dpart
but it mixed them in with the
collaborators.
I didn't take them out of the
search.
I just didn't show the blocks.
While this worked technically,
it just ended up making more
work for me.
It did what I asked but not what
I wanted.
That's part of the process.
Something better I thought of,
which I probably should have
thought of in the first place is
modifying any original search.
I can explicitly exclude
Peterson ERIKwith a Kas a full
author name.
Let's do that.
Peterson, ERIK, FAU.
So that should exclude all of
the Peterson ERIKs.
That helps me.
It moves me a step in the right
direction.
So I'm going to run that
modified script or basically the
same disrupt that I rap before.
Hang on a second.
Try that again.
This is basically the same
script that I ran before.
I'm not Ewing the identifier
because that wasn't helping me.
I announced out those ERIKs
with a K.
The next time hopefully those
won't even be in the list.
Okay.
Let's get it back up to where
that was.
And 18780023, I don't see it
there.
I think we got rid of it.
So that's good.
Dealing with these collaborator
and investigators, there are
still these blanks in here where
Eric Peterson is an
investigator, not an author.
The first question we have to
ask is do we want to include
these record at all?
If not we can modify our search
again to explicitly removing
them but that might run the risk
that it removes when he is an
aller and investigator.
We can just take this list of
Pmids that we have here, remove
the bad ones, remove the ones we
don't want and use the Epost
command and Efetch to retrieve
only the specific records that
we want and Xtract the
coauthors.
If we are doing more intensive
checking on these records and we
find other records that we want
to remove, this might be the
preferable solution and if we do
want to include these records,
we still need to make sure it's
not E-R-I-Kas an investigator.
We need to make sure we're
seeing that Peterson name
information and they are not
just showing up as blanks.
I'm going to do another version
of that same script that I just
ran, but this time I'm going to
try to find -- sorry.
Clear that again.
Try this one more time.
There we go.
This is the same script we ran
last time in our modified search
giving us our aller first name
and forename.
Right after the author I'm going
to run block investigator and do
the same thing.
Actually I'm going to copy and
paste this part because it's
exactly the same thing as it was
up above.
Oop, an extra space in there.
This is what I get for copying
and pasting wrong.
Hopefully this should deal with
those blanks and our list should
now be giving us author
information for EDPeterson.
Again, let's make sure that that
is the case once it finishes
running.
Apparently Eric Peterson is
listed as an author multiple
times.
Hang on.
We got one here that's still
blank.
Let's take a quick look at that
in PubMed, see what's going on
with that one.
We got a Peterson E.
But we specified Peterson ed.
Let's take a look at the XML,
see what's happening there.
Find him in the author section
right here.
It's ed, ed Peterson.
We could modify our search to
get rid of ed Peter sun but this
might be an arm for annually
cure rating our list.
Looking at affiliation data
would be probably the next step
to vet those records to see if
there's any mass you have
outliars in the affiliation that
we can get rid of.
We're running a little low on
time.
So I'm going to stay enough
disambiguous information.
Move on to that step back down
to here.
So let's say that we feel
comfortable that this search is
retrieve only the correct
record.
We probably feed in a curated
list of PMID for any post but
I'm going to pretend we're
satisfying with this search.
Next we need to figure out how
to get the co-allers.
Let me grab that search, paste
it in there.
Efetch, format XML, so we'll get
all of these records gun and now
we'll try to find the coauthors,
right.
So we want a list of authors.
Whenever we want a list of
something, that's usually our
pattern.
Then let's say element -- well,
actually not yet because we
don't want every author.
We want every author except Eric
Peter sun.
We just want his coauthors.
I'm going to use that is not
argument I found before.
So we're going to say if last
night, make sure I get this
right, is not Peterson and
initials is not ed, out.
Them we're going to -- let's
just get last name and initials
now.
Last name.
We can put them in separate
columns.
I can change my tab to a single
space so I'll get last name
space initials so I don't have
any information that I need
here.
And then we're going to use --
I'm going to use unique count
rank.
That should give me a list of
authors aren't ed Peterson on
all of these record that ed
Peterson was on and give me
basically the top ten list,
right, who are the most frequent
coauthors.
It will take a little while to
run.
Okay.
So big list of authors.
Get back up to the list here.
Big list of author.
These are our most frequent
collaborators.
Our frequent coauthors but of
course this doesn't entirely
account for co-author disam
biggation.
What if Eric Peterson
collaborated with people with
the same last name or initials.
It was probably cop fusing for
him but could be for us.
Where do eric Peterson's
co-authors work.
In order to do that we'll
probably take a lot of human
evaluation or some very
elaborate scripting work on our
part.
This is where that 90-10 rule
might come into place.
If we get 90% of the way there,
that might be good enough and we
can do the rest of it by hand.
I do want to save some time for
the next case study so I'll let
you experiment with that
further.
You have the cold block on that
supplemental handout.
Want to run on because we're
running a little bit out time
with our third case study.
Emily, it absolutely would.
That's one of the reasons I
didn't show it to you is it
would just look like a bunch of
garbage.
You see a lot of ones and twos.
Even if people are using more or
less the same affiliation, if we
have a space off here or there,
it's going to give wildly
different information.
That's what I'm talking about
acquiring that sort of hand
curated artisanal affiliation
checking because otherwise
you're going to sort of -- the
machine is only as -- as we can
make it.
That's a very good point.
I'm going to move on to our
third case study which is kind
of interesting.
It's different than anything
we've looked at in this class so
far and it was suggested by
another form per student.
Usually we're analyzing PubMed
data but this time we're
actually going to analyze a
PubMed search strategy, and the
strategy that we'll be analyzing
is one that an organization uses
to find new veterinary articles
being added to PubMed.
They run the search every week
with a different date rain
believe and that strategy
searches a list of about 300
different veterinary journals
and that list is updated
periodically and it also
searches a handful of mesh
subject terms as well.
Those mesh terms are also
periodically updated bought they
are actually due for an update.
So our goal -- I jumped ahead a
little bit.
Our goal is to see if we can
find other terms that we can add
to the search to improve
retrieval.
One approach we can take is to
look at the result for this
search currently and them
analyze the subject terms that
occur frequently in the current
result set.
If there are terms that aren't
in the current search stream,
they might be good candidates to
add that strategy.
Again, walking through the
method that I outlined earlier,
our input is going to be a
search strategy and it's a
pretty long one.
I'll show it to you in a second
so we probably want to retrieve
it continue a file so we can
tweak it overtime but it will
keep our code further.
Our output is going to be a list
of descriptors.
And then for our format we want
a list of unique subject terms
sorted by frequency in the
result set with the accounts of
how frequently each appeared.
So this is classic example.
It should look a lot like what
we just ran before and it's
going to be a pretty big list of
subject returns.
Let's start with piling by
taking a look at the search
strategy itself.
You can see we have a big long
list of journal IDs here and
some mesh terms at the bottom
and some date restrictions down
at the very bottom.
So let's start by running --
we're just going to run the
search without the date
restriction so we're going to
run everything else except for
the date restriction.
I see what size results here
we're dealing with.
My advance search builder, paste
in my giant search.
Okay.
So we got a quoted phrase, not
found issue but more importantly
we have two million results.
That's a lot of results.
That's too many results.
I'm going to get rid of that.
I'm going to go with just the
journals.
I'm going to look at the journal
part because presumably if we
include the mesh terms it will
get a bias towards that, towards
those mesh terms.
Let's just see what's in the
journals.
Go ahead and paste that in
there.
Now we're down to 600,000.
That is still a lot, a lot of
search results.
Now I'm going to destroy a
publication.
Just going to do 2017, find that
filter.
All right.
Now we're down to about 25,000
records which is still a lot.
It's probably okay for us to
work with in terms of a demo or
in terms of our actual analysis
but for a demo it would take an
Esearch and an Efetch per
iteration to run this.
It's just such a huge result
set.
First I'm going to hop over into
my terminal again.
I'm going to get down to the
bottom, clear my screen and
start building my Esearch.
Getting my note back up over
here where it doesn't have to
obscure your view.
For this part I'm going to start
with my Esearch because I'm
going to be pulling in the
search strategy each time.
I saved that file, that search
string with just the journal
tags, just the journal IDs
into a separate file that I
called vet journals dot TXT.
I can throw on my date
restriction to see make sure
I've done this right.
They should line up with what I
did in the web version of
PubMed.
Yeah, same result.
So this is the search that I
want to analyze.
But before I go too crazy with
those 25,000 records, I am going
to get a couple of PMIDs just to
work with.
And the way I'm going to do that
is by piping this into an
Efetch, Efetch, we reaching
format UID.
You can piece it together.
This should just give me the top
ten PMIDs.
And so I can take those top ten
PMIDs and build an Efetch for
testing purposes.
Again, I'm not going to waste
your time with copy and pacing
one PMID at a time.
I have a very short testing
Efetch that I can run here.
Should get me XML.
I'm sorry I'm speeding up a
little bit.
I'm just trying to get through
all of this before we run out of
time, still leave some time for
your questions.
This is our Efetch that we're
going to use to build the
extract and eventually use on
the jumbo data set.
So let's build the extract.
So we want a list of mesh
headings so I'm going to use my
pattern.
It's going to be mesh heading.
You could also use pattern
descriptor name that would work.
I'm not going to because I know
what the next part of the demo
is going to be.
Descriptor name would be as
well.
They be our element descriptor
name is going to be here.
You should run that and I get a
nice handy, dandy list of
descriptor names which is what
I'm looking for so that I can
eventually use them with any
sort of count rank.
Okay.
So then what I would do is I
would do something like this.
Which is taking that same
search, pairing it with the
Xtract, pairing that with the
Xtract that I just came up with,
unique count rack on the end and
then saving it all to a file.
I'm not going to run this and
you shouldn't run it either, and
the reason why is it will take
way too long to run.
However, the long part of this
is the downloading of the XML.
So the Xtract part runs pretty
quickly.
It's getting the 248 mega bytes
of XMLin this set down that
take over ten minutes.
If we know we are going to do
several Xcontracts on the same
large set of data, we can just
use the XMLonce and keep
refusing it.
The way I would do that is
running a modified version of
this Esearch and Efetch or up
modified version of Esearch and
Efetch.
I'm just going to save the whole
XMLinto a file.
I ran it earlier and it took too
long.
But once I have this on my
computer, I can use this with
the Xtract input argument to
keep pulling it in and doing
different slices of the data.
Let me do a quick LS.
We can see it's right up here,
and actually let me hop over
into window and I'll show you,
that file is right here.
It's very large, it's 248 mega
bytes because it's a year's
worth of results for that
search.
So I have that and now that I
have that, I can run my Xtract
without having to pull that file
each time.
So let's do input vet dot XML
that.
Will pull in that XML file from
my local computer.
I've done the same Xtract that I
wanted to before.
Throw on a unique count rank and
I'll save that to a file.
Shouldn't take too long.
No, see, that's it.
Had to turn over 248 mega bytes
in XML, but it didn't have to
download it.
Here's what the output of that
file looks like.
So it's basically a big, long
list of mesh headings sorted by
frequency.
You are you'll notice getting a
lot of these animals, females,
humans, cattle, these check tags
which basically any time they
are mentioned in an article have
to be assigned but they are not
necessarily what it's about,
what the article is mostly
about.
To get down to more major
topics, we might have to look at
what the major topics are
assigned to each citation.
So we can try and do that.
That's going to be a little
trickier but you actually had a
homework assignment that was
fairly similar to this exact
process.
We're going to use that same
downloaded XML and we're going
to keep the same pattern, and
I'm not going to keep that same
element just yet because I think
we need to make a few tweaks.
I mistyped my pattern.
Before we go any further, I want
to take a look at an actual
record and how those mesh
headings are set up in the
XMLbecause there's tricky stuff
we need to contend with.
And let's look at the XML of
this, and the reason I want to
do this, scrolling down to the
mesh headings past this giant
list of authors.
You'll notice that there's this
major topic YNattribute and
this theoretically tells us
whether a heading is a major
topic, yes or no.
That's why it's called major
topic YN.
When index is assigned Indian
Ocean a record, they assign
headings or descriptors.
They can also asign sub headings
or qualifiers to further
describe or focus on a
particular aspect of one of
those descriptors.
Now if we look at how the
attributes is set on the record,
we can see how it's more
complicated.
I'm going to look at a
descriptor that has no
qualifier, mutation right here.
You can see there's no
qualifier, it's just the
descriptor and the attribute is
set to yes.
Now let's look at a descriptor
to as a one qualifier.
How about compliment system
proteins right here.
It has a descriptor and
qualifier metabolism.
We can see when it's a
descriptor qualifier combination
that's major, the attribute is
set on the qualifier but not on
the descriptor.
You can see the metabolism
attribute is set to Y, but it's
set to N.
This one right here, only one of
these qualifiers genetics right
sheer tagged as major.
The descriptor itself is not
tagged and the other sub heading
is not tagged.
So if we want a list of the
major topics, we have to be very
careful.
If we only grab elements with
the major attribute set to yes,
we're not going to get the
descriptor actual qualifying.
For my list I don't want to look
at qualifiers, it's just too
granular for what I'm looking
at, I just want to see all of
the descriptors that are major
top you can see.
I know that's a little bit
confusing but we're going to try
and figure that out, how to do
that.
This was actually an assignment
on your homework.
There's more information about
it on the sample code from last
night's class where the rest of
the answers are for the
homework.
But the way that I'm going to
figure this out is I'm going to
basically check through each
mesh head and apply a condition
and I'm only going to include
the mesh headings where --
forgot to -- what did I do
wrong?
There we go.
I just had an extra space in
there.
So I'm going to only include the
mesh heading so basically if the
descriptor name -- if descriptor
name -- descriptor name
attribute, descriptor name major
topic YNattribute equals yes,
then we want to output the
descriptor name.
So that's -- we definitely want
that.
Or if any of the qualifier names
in that mesh heading have the
major topic YNattribute set to
yes, if either of those things
are true, then we want the
descriptor name if that mesh
heading.
I think I did that right.
Did I do that right?
I need unique count rank at the
end.
That would be bad.
Throw that in major topics.
File, run that.
Again, shouldn't take too long.
And we could see that my list is
very different now.
It's elevating things that are
more major topic of articles and
not just male, female.
So that's a good list.
And it's definitely a good place
to start.
But these are mesh headings and
are therefore based on a sub set
of our results from that search
because I actually checked the
list of 308 journals that are
being searched here.
Only about half of them are
Medline journals so the rest of
them are not indexed and won't
any mesh terms on them.
We can capture the mesh terms a
little bit of a different way
not use mesh but instead using
author keywords.
I'm going to run through this a
little bit quickly, take that
same input.
Pattern should be keyword.
Almost should be keyword.
Unique count rank.
And this is another way of
splicing the same data but this
time we're looking at keywords
instead of mesh headings.
Hopefully I did that right.
Bring that over here and see
what we're looking at with these
keywords.
Hopefully this will give us some
text words instead of just
limiting ourselves to mesh.
But you can see that the author
keywords are not controlled.
They are just submitted by the
publisher.
So this will require more
analysis on our part but there
might be something of value
here.
Absolutely more analysis is
needed here put we have the raw
data.
Because we downloaded that XML,
we can keep doing different
alternatives, keep doing
different slices, to show us
what the data is.
I'll address those questions in
just a minute.
I want to do some last,
finishing thoughts here before
you all flee.
We are running out of time and I
do want to thank you very much
for sticking with us over the
past five classes.
I do want to make a point to say
though if you feel like you
still don't really know how to
use EDirect after five 90 minute
sessions, I do not blame you at
all.
It certainly took me a lot
longer to learn this stuff.
There's a lot of absorb here and
you're probably only going to be
able to learn it by working with
he'd audio yourself.
I hope we were able to show you
where EDirect was so useful.
I hope you will take time to
review the sample time on the
website, look at the records and
poke through the documentation.
It does take time and practice
to really master these tools and
figure out how and whether to
use them.
Give them a try, experiment.
Get comfortable in the command
line environment.
We do have one final assignment.
It gives you an opportunity to
solve some real world problems
with some of the stuff you
learned over the past few weeks.
I encourage you to use it as
another opportunity to roll up
your sleeves and really dig in
to EDirect.
Finally I would also Ian courage
to you keep your eyes open for
the next time a project that
would be a good fit for EDirect
comes along.
In the meantime I have one more
resource to show you that you
might find useful.
NCBIhas released or started up
I should say an EDirect cook
book.
It's a list of pre made scripts
that can accomplish certain
tasks.
I would definitely encourage to
you take a look at this
resource.
There's a link in your handout
and I'll show you what it looks
like real quick.
It looks like this.
And it's actually been published
through gift hub.
If you haven't ever used Git hub
you don't need to know anything
about it in order to use the
cook book.
You can just copy and paste them
right into your terminal and
give them a try.
For now most of these script are
focused on the molecular
biology.
There are a few PubMed scripts
up there including a few from
the script.
The nice thing about this being
on Git hub, I'm going to scroll
back up to the top here.
If you need a script but there
isn't one that's in this cook
book already, you can use the
tab right up at the top here to
request a script basically and
somebody in the community might
be able to help you out and come
up with something for you.
On the flip side, if you have a
script that you want no share,
you can do that too on Git hub.
This will let you develop an
EDirect community of practice
that will help people share
their knowledge.
Getting back to the slides here.
Make sure you sign up for the
utilities announce web serve.
It's where we post our new
classes.
It's not a lot, maybe once a
month.
There's some good changes to the
utilities.
We don't have anymore insider's
guide classes listed on the
website right now.
You folks are going to get a
special sneak peak we will be
announcing later next week as
part of the insider's guide.
On Thursday at this time there
will be a special office hour
session for anyone with
lingering EDirect questions.
That's limited to just the
people enrolled in this class.
On April 25th ear going to be
holding our first public EDirect
office hours for alumni for
PubMed and other intermediate to
advance EDirect users.
We're going to kick it off with
a more advanced presentation it.
It's still pretty useful I
think.
The rest of the time though will
sort of be an open forum to
answer any questions from the
audience about EDirect.
How does a certain command work,
how can I accomplish a certain
task using EDirect.
This is a new type of program
for us and we're not sure how
it's going to go.
We know people have questions
about how to apply EDirect in
real world situations.
If this works out we might
continue doing these down the
road.
If you're curious, we do hope
you'll stop by.
Announcements will be going out
to that and we'll make sure
you're getting an announcement
for that as well.
For those of you curious about
how to claim credit as soon as
you submit your final exam.
The final exam for this class is
Again, thank for sticking with
us for all of these classes.
I hope we'll see you at office
hours or again at one of our
future classes.