>>> Good afternoon and thank you
for joining us for EDirect for
office hours for April 201.
This is our first ever EDirect
office hours session, so thank
you for joining us for this and
for being a part of this.
We're very excited to be here
today to answer your questions
and maybe teach you a few
EDirect tricks.
I'm Mike Davidson.
I'm one of the instructors and
lead designer.
In the room with me today is
Kate Majewski.
Also on the line we have Peter
and Sara who are a couple of
other EDirect PubMed instructors
and we're pleased to be joined
today by Peter Cooper.
Developing and supporting
PubMed, Eutilities and EDirect.
Peter and his team also offer
training on Eutilities and
EDirect but they offer more
molecular biologists and
scientists.
So we're happy to have him here
today to bring perspective on
these topics.
As I said most of today is going
to be about answering your
questions and using variables in
extract.
Then we're going to have a few
brief announcements about some
recent and upcoming EDirect
changes, and the rest of the
time is for you.
So, as I said, before we get
into your questions, we want to
introduce you to a new topic for
many of you because we do not
cover it in the topic of PubMed,
and this is EDirect.
Those of you who have taken
EDirect for PubMed might
remember some questions that we
didn't answer at the time or
examples that we didn't address
saying that they were
technically feasible using
EDirect but that they advanced
techniques out of scope for the
class.
The reason we don't cover this
for PubMed because it's kind of
tricky to wrap your head around
how variations would work and
when you want to use them.
You might find they don't work
exactly the way you expect.
The top level summary of
variables is they are a method
of storing information in one
part of an abstract and
retrieving it in another part.
I'm going to start with an
example.
I'm going to start with an
example you may have seen in
EDirect for PubMed especially if
you've taken the class in one of
our offerings.
Let's say we want to do the
search for an author in PubMed
and analyze the different
affiliations with that author in
PubMed, and we can use this to
track an author's career over
time but it might be more useful
for disambiguation.
It's up to the user to make sure
the results they are looking at
are for the correct author.
This is especially true for
authors with more common names.
So one name you can try to
disambiguate authors, if you
know an author specifically
works in China, for example, you
can probably any named in
Germany are attributable to
somebody else.
So I'm going to hop out of my
flies and go over to my Unix
terminal to look at some code.
And this is a block of code we
showed in many more recent
offerings, and we use this to
accomplish a similar task.
I'm going to walk through this
line by line and get us warmed
up.
First we use Esearch to search
PubMed for the authors Smith BH,
it's Smithabout the H.
We're going to limit our search
by dates, by publication dates
specifically to articles
published from 2014 and 2018.
We're going to use Efetch to
retrieve the XML.
And then we're going to send
that XMLinto our extract
statement, and for this project
we want to look at each record
that has an author of BHSmith
and see what affiliation is
listed for the BHSmith on that
record.
We'll have extract put the PMID.
Using element to you put the
PMID in the first row.
Then we need to find the
affiliation data for the
BHSmith for each record.
I'm going to step back and just
look at some PubMed XML and set
a sense of how the author data
is structured.
If you can see right here,
here's our author list, we can
see a second one there and third
one there, and for each record
on a record, child elements of
last name, initials, affiliation
is optional, identifier
aoptional as well.
What we're going to do is use
block to look at each of these
author elements for the record
one at a time but only display
information for the author
element where the author is
actually BHSmith.
Looking at our code, as extract
goes through each author block,
it checks if last name equals
Smith and if initials equals BH.
If both of those things are
true, we will output data for
the author.
And each of these records should
have one BHsmudge on them
because that's how we determine
which records to include in our
search.
Now obviously last name and
initials here are always going
to be Smith BH.
Our second column should all be
Smith, and our third column
should be BH.
We really don't need those in
the output.
I'm going to keep them there to
make sure you will all are
working correctly.
When we run it, we will get our
list of PMIDs in the first
column and then the affiliation
data for BHSmith in the last
column, if it is in fact
present, and that's mostly what
we get here, and it's good as
far as it goes.
But we have a bunch of rows here
that don't have any affiliation
data.
We have one here, here, here,
etc.
As you probably know,
affiliation is not a required
field in PubMed.
Up until 1988 we didn't have
affiliation data at all.
Up until about five years ago
only the first author's
affiliation data was accepted.
Again, it is not required.
If we are only interested in
looking at BHSmith's
affiliation data and we don't
care about the records where
BHSmith has no affiliation
data, can we exclude the records
with no affiliation information?
Probably the most obvious way
would be to get this output into
another program like excel,
either by saving a file or
copying and paste.
We can use the tools to filter
out any row.
We're here to learn about
EDirect so let's see if we can
find a way to accomplish this
without using excel.
I'm going to clear my screen
here and paste in a modified
code block and see if that does
the trick.
So what we're doing here is
anding in another condition.
We'd like to make it when we
only see input for an author
when there's an affiliation date
with that author.
However, of we execute this, we
will we will discover this did
not do exactly what we wanted.
Now the second and third column
are also blank.
We kept the PMIDs but we blanked
out the rest of the row which is
not exactly what we're looking
to do.
It prevented us from seeing the
last names and initials if there
was any affiliation data but it
didn't prevent us from seeing
the PMID where that wasn't true.
It's again not doing what we
want.
Going to clear my screen again
and look at a third version of
this code.
Hopefully getting a little
closer.
In this case I moved that
affiliation copy up into the
pattern rather than in the
block.
As you're about to see, this is
not going to cut it because the
problem with this condition is
it only excludes roads if there
were no affiliations at all.
Those records somewhere in that
PubMed record have an
affiliation element, just not
specifically for BHSmith.
So what we would really like to
do, and I'm going to paste that
code in one more time so I can
show you.
Copy that code and paste it in
again.
What we really like to do is put
the PMID down here inside the
element inside this and them use
the affiliation down here.
The problem is we can't do that
because PMID once we're inside
the block, we can only access
data that's part of the block.
PMID is not a child or
descendant of author, we have to
be pulling in the PMID up at the
pattern level.
So again that's what we'd like
to do and there is in fact a way
that we can accomplish this.
As we may have suspected, we can
do it using variables.
So here's another code block
that's going to come a lot
closer to what we want.
Got a couple new things in here
we haven't talked about before.
This time in our extract
command, we're actually going to
save the PMID into a variable
named VAR.
We do that by making up a new
custom argument with the name of
the variable we want used and
that's right here.
We have this dash and dash and
the name of the variable which
is VARin all caps and one.
This puts the PMID into a
variable named VAR.
This doesn't have to be called
VAR1.
I'm calling it that to avoid
confusion.
We can cawed it UIDin all caps
or PMID if I want to, we could
call it Bob.
The only rules are the variable
name has to be all capital
letters and digits.
No lower case letters, no
symbols.
Once we have this PMID into this
variable, we can start our block
as we did before.
From each record we're going to
check each author so that the
last name equals Smith and that
the authority had the
information data.
We would use the element output
to content data.
We do this by using quotes and
an ampersand.
We put ampersand VAR1 all
surrounded by quotes.
Ordinarily we couldn't put the
PMID in here because it's not a
child author but we saved it,
it's now available for us.
We're also going to use, again,
the last name, initials and
affiliation.
And when I execute this, we will
see that every row has a PMID,
last name, officials and
affiliation data.
We filtered out the rows that
don't have affiliation record
data.
We never found a block that met
all three of our conditions.
So we never output anything.
Because our only element
argument is in our block of our
record didn't have a block that
met our conditions, we just
skipped it entirely and the
result is only the rows that
BHSmith has affiliation data
are output.
I know that that's sort of a lot
to take un.
I'm going to go through some
syntax with nuts and bolts here.
Phenomenon we have some time at
the end of today's session, we
might go through another
example.
As I've said, we've already seen
the syntax a little bit.
Names can be digit and capital
letters.
Instead of using the dash
element, you use a dash and the
name of the variable you want to
store the name into.
Like we said before, if you're
in a block author, extracts can
only access information from
within the author element.
If you declare a variable inside
a block or pattern, can only
access that to store it into the
variable.
Use an element argument just
like you normally would.
The variable name should be
preceded by an ampersand and
then you put the whole thing in
quotes.
There's a couple handy
shortcuts, handy tricks what I
found when working with
variables.
Of course you can put multiple
values in the same variable.
Just like you can put multiple
values in the same column, you
do the same thing when you're
creating a variable.
So in this example, our variable
date has both the year and the
months both in that same
variable one after the other.
This can help you organize your
abstract command a little
better.
Second you can output multiple
variables in the same element
argument just like you can
output multiple elements or
attributes in the same element
arm by separating them with a
space.
As we've already seep in our
example mix and match various
attributes all in the same
element argument.
Finally you can also output the
same variable multiple times.
Like I said, you can make good
advantage of this to streamline
your code if you need to output
the same amount of elements
multiple times, you can save all
of those elements to a single
variable and output to the
variable as many times as you
need.
All right.
Any questions about variables,
any of that stuff?
I see that we have a question
here that's not related to this,
so I'll get back to that in just
a moment when we get into an
open question part.
I don't see any questions right
now, but, if you have them,
please put them in there.
I will move on to some other
announcement related things and
we can circle back to them if
anybody has any questions a
little bit later on.
Okay.
So until recently of people
asked about doing programs on
hundred of records, we used to
mention the bulk data exports as
an alternative to folks who
wanted to work with the entirety
of the public all at once.
That program is not necessarily
practical for everyone.
Many of our data distribution
customers, our companies, our
researchers who have the
infrastructure to tick these
bulk downloads, file the records
into their own home grown data
bases.
We know that not all EDirect
users have those resources.
However, starting with EDirect
version 8 there is a new feature
that can help you get the best
of both worlds.
It puts it to work as the entire
58 million records of PubMed.
So this new feature lets a build
a local copy of PubMed on your
local drive so you don't have to
wait for it to be built every
time.
Rather than watching the
XMLwith Eutility, you fetch it
with your local copy.
There are a few caveats with
this process.
First, in order to see any speed
benefit, you'll need to have an
external solid state drive.
A spinning disc hard drive is
probably not going to cut it,
and you'll want that drive to
have enough capacity.
Depending on your system it
could take anywhere from six to
over 30 hours the next time you
build it.
This might be a useful option
for some projects where you're
analyzing huge amounts of data.
But for many projects where
you're retrieving less than
10,000 records, you're probably
better the old fashioned way.
We have a documentation page on
the insider's guide website.
You can get there via the link
that's on the screen.
Also if you go to our home page,
there's a link on the lower
right that will take you
directly there.
Also coming on may 30th I
believe, the developers of
EDirect will be presenting an
NCBIwebinar.
There's a scheduling change to
that.
As I said, the developers of
EDirect will be presenting an
NCBIminute webinar to talk
about this and some of the other
features of EDirect.
So if you're interested in that
stuff, please be sure to check
it out.
All right.
With that, that's most of the
schpeal that I have for you.
So I'm going to pause for a
moment.
If you have any questions and
haven't put them in there, go
ahead and put them in here.
I'm going to speak with the
folks in the room and make sure
I haven't missed anything and
we'll start answering some of
your questions.
it looks like we have one
question so far.
About the recent announcement in
PubMed users can now search on
or pepped searches to find
articles with related data links
or the linkout, other literature
resources field.
Kim was asking whether EDirect
can fetch ideas returned on
PubMed as having that data
filter.
And Peter points out, yes, of
course you can do this, but
basically the way you would do
it is with any Esearch.
Putting that data tagged with
filter would work, putting it in
your Esearch query would also
work as well.
So that would look something
like this.
Hop over, demo that.
Got to make sure we're
specifying which data base we're
searching, and this would be
like that or more likely and
search term and -- however you
were searching.
It looks like that mostly
answered Kim's question, but, if
that prompts any other questions
from anybody else, feel free to
ask them.
Asking about APIuse.
I don't know if you were able to
tune in to the NCBIwebinar that
happened earlier today.
It was all about the plan for
rolling out the APIkeys when
they are necessary, when they
are desired, when they are
useful.
If you miss that, there's a lot
of great information in there
and the recording will be posted
soon.
The short version of your, to
answer your question is, you do
not have to get an APIkey, but
depending on your usage, you may
really want to get an APIkey.
[inaudible]
>> And we're just having a quick
discussion in the room here to
verify that that's true.
Basically if you were to get an
APIkey to use with EDirect you
could -- you could set that as
an environment variable that
will give you access to
basically an expanded rate for
APIqueries.
EDirect is compatible with that
and is ready to go with that, as
that rolls out.
And there was information about
that in the NCBIminute from
today so I would encourage you
to check that out.
>> Asking where to see all of
the filter elements.
All of the different -- you're
asking basically where to see
all of the different filters
that work with that filter tag
in PubMed.
Is that what you're asking?
>> Okay.
Hang on a second here.
Let me discuss this here.
We had a quick discussion here
and there are a couple of ways
to find out that information.
None of them are EDirect
specific actually.
One way you could go about it,
I'll hop over to PubMed and I'll
go to advance search here and
find the filter box, and you can
actually look at the index list
by clicking show index list that
will show you everything that is
indexed with that filter tag.
That is one approach that of
course doesn't tell you exactly
what any of these filters are,
but it does tell you what the
available options are.
You can also find more filters
in My NCBI by going to My NCBI
and the filters that are would
ordinarily show up, they are not
showing here right now, and I'm
not into My NCBI so I'm going to
back out of here real quick.
But the manage filters link
right here that would allow you
to manage what filters show up
on the sidebar, a lot of those
PubMed filters that are relevant
and work with that filter in
brackets tag, you can also add
them as those filters in the
sidebar there and you can get a
little more information about
them by managing the filters in
My NCBI.
Does that more or less sort of
answer your question?
Any of those filters once you
find them, you can use them in a
search, and Esearch using
EDirect.
Hopefully, that more or less
answers your question.
If you have followups, please,
let me know.
Let's see, we've got a message
here from Jonathan who is the
developer of EDirect who is
working with us today who says
that you can specify the
APIkey, you have two options
and I'm going to type these in
right here.
You can either set an
environment variable of -- let
me type it in here so that I can
make sure it is -- so you'd set
your APIkey that you get from
My NCBI, you'd set that into
this environment variable, that
would be one option.
And he he's actually added
another, basically an argument
that allows you to specify an
APIkey on the fly.
Then you do APIunder score key
and you put your APIkey in on
that argument and that can
specify it on a line by line at
a time command basis.
Again, I do encourage you to go
back and check out that webinar
from earlier today.
It's only half and hour but
there's a lot of valuable
information in there and sort of
some time line about when this
stuff is going to take effect,
how the testing is going to
work.
If you are a particularly heavy
user or even if you're just
interested, make sure you swing
pack and check that stuff out.
Any other questions about
EDirect or any related topics
that you would like us to
address?
I'm going to mute for just a
moment but please feel free to
keep typing your questions in,
if you are typing them, or, if
them.
all right.
Actually we were just discussing
here pause we wanted to make
sure that we address all of the
questions fully and I think that
we covered most of what's out
there right now.
What time is it?
So I'm trying to decide whether
we should go into another
variables related example or --
so if you do have other
questions that occur to you,
please put them in there.
I'm going to go through this
other example because I think
it's more of a practical
application of variables and I
think that you might find it
interesting and maybe find it
something that you could make
use of later on, and then we'll
still have more time at the end
for questions about that or
anything else that occurs to you
if you are still mulling what
kind of question you want to
ask.
Give me just one moment.
I want to get back over into my
slides and we'll talk about just
one other example.
Give me one moment.
okay.
So our other example for
variables is a technique that
you can use to help with
affiliation searching.
And this is again one way to do
this, not necessarily the only
way or the best way, but it's a
fun thing that I found that's a
really good, strong use for
variables.
Again, you might find it useful
as well.
So we already talked a little
bit about affiliation data.
Going a little bit further,
affiliation data in PubMed is
supplied by the publisher of the
article and is free text, not
structured data.
Different authors can report
their affiliation data
differently which makes it
difficult to search for
effectively.
For example, let us say you are
trying to look for papers
written by researchers at the
center for translational
medicine at Thomas Jefferson
University in Philadelphia,
Pennsylvania.
So that's your follow.
-- goal.
And so I did a search and tried
to find authors, and that was
just a few of the different ways
that papers written by authors
represent their affiliation.
There's spelling variations,
what order is it in there.
Finding an exact affiliation
stream to search is pretty
difficult.
And it's very difficult to do
the search with good precision
and recall.
If you just searched for the
phrase translational medicine
and the phrase Thomas Jefferson,
you'll get many institutions
with translational medicine
centers or many different
components of Thomas Jefferson
University.
A different strategy would be to
search for each of these
different variations in their
entirety plus all of the other
variations you find all ored
together.
To be a little bit more precise
but also practical because how
would you possibly know that
you're getting all of the
variations?
So another common approach is
searching for two portions of
the affiliation anded together.
Like, for example, transitional
medicine tagged with ADfor
affiliation and Thomas Jefferson
tagged with ad.
This should return all authors
with Thomas generous and Thomas
Jefferson with affiliation.
For example, it will get you
record like this PMID which has
multiple authors on it.
One of the listed authors on
that record works at the
research center for research
medicine in Shanghai and a
different author on the same
record is from the department of
microbiology in Philadelphia so
this technically meet our search
criteria but it isn't what we
want.
We want both of these records
where both are present in the
same affiliation field for the
same author and that's what it's
designed to do.
Hope back over here, get by
notes back up and ear going to
start with that basic search
that I just described,
translational medicine and
Thomas Jefferson both in the
anded field.
Those are two unique phrases,
combined together should be
pretty unique and it will get us
all of the records from authors
and it of also get us records
where translational medicine and
Thomas Jeffs are the affiliation
for two different officers.
That's okay.
We will filter this down and get
just the record that we want.
But I just ran the search to see
how many record we get to this
search and we get 252.
That's our base line.
Again, that's more records than
we actually warrant.
So now I'm going to take that
search, throw it into an Efetch,
get some XML off of that, and
them what I'm going to do is
start my extract.
We want to get a list of PMIDs
that have at least one author
with both of our affiliation
streams in there.
A good way to do that is use
pattern, PubMed article.
As we already talked about in
order to find that PMID, extract
needs to be able to see it.
It needs to be able to be in the
-- the information needs to be
in the portion of the record
that extract is exploring.
So purchase we start looking at
authors or affiliations, we need
to save the PMID up here at the
PubMed article level so we'll
have it later.
We do that again with our
variable.
You can use any combination of
capital letters and digits.
Save that PMID in there, and now
we can use the block to check
each affiliation element one at
a time, and I could use block
author instead to check each
author one at a time, but this
way if an author has a multiple
affiliation listed, we'll make
sure both of our strings are in
same affiliation.
So now get my conditions, and
let me copy and paste those
because they are lengthy.
Too many spaces.
So what this will do again is
it's going to check each
affiliation and make sure that
it contains both translational
medicine and Thomas Jeffs.
Now that it's done that, we can
actually output the contents of
our variable.
Typing in the wrong place using
elements and our variable name
right here.
Quotes with the ampersand.
So thinking about what this
means, we can see if no
affiliation records have no
strings, we won't output
anything at all.
We'll just skip down to the next
record, so that record will be
skipped in I entirety it.
We will see if we have multiple
authors that meet the criteria
on a record.
For example, if several
colleagues at an institution
print something together.
All of these authors, each one
of those PMIDs represent a ditch
affiliation element on the same
record that met our criteria.
But fortunately these make our
PMID IDlist a little more
unusual.
But I have a better way to do
that or at least a different way
to do that which introduces
another new technique we didn't
talk about either.
Here's my block of code.
Again, this is exactly the same
as the last code except I've
added this tab argument here and
by making the tab slash ed which
is the short happened for the
new line character, we're asking
it to separate not with tabs but
with new lines.
So it will actually be sort of
new line down on to the next
line on its own.
So when we execute this we'll
get a big, long list of PMID.
Again, we're not going to have
the same things on the same row.
You will of those PMIDs that
were previously stretched out to
the one row are appearing
sequential hee one row after the
other.
But that's okay due to our next
step.
Obviously there's ways you to do
a unique -- use the unique
command to separate out.
We don't even need to do that
because of what I'm going to do
next.
Once we have this list of PMIDs
for records that meet our
criteria.
If you don't remember Epost,
it's the command that lets you
upload a list of PMID to the
history server so you can
retrieve those with Efetch.
Will upload that list I'm typing
in to the history server.
It doesn't care about the
duplicates.
It's only going to post each one
of those PMIDs once.
My next step would be to type
the Epost into an Efetch,
retrieve the records and then
pipe those records into an
extract to get the data that I
want from this filtered set of
records with authors that have
the correct affiliation.
Before I do that though, I'm
just going to run through this
command up until this point, up
until the Epost which will show
us how many PMID are actually in
our list.
You can see it's 210 right here.
So between this Esearch and this
Epost, we filtered out more than
50 records because they didn't
actually meet the search
criteria we were looking for and
we were able to filter this out
because of this technique using
this variable.
As I said, we can continue the
script by piping that Epost into
an Efetch and then into an
extract and them retrieving
whatever information we want.
I've added a little extra script
here, one we talked about in
class before.
This will get us the most
frequent granting agencies
attached to articles published
by authors.
So we can see transitional
medicine, HLVIis their number
one by a long shot is their most
common granting agency in the
way it appears on public
information.
But you could do whoever
whatever you wanted on this
extract now that you filtered
down your results into just the
specific records that you're
looking for.
Okay.
Any questions about that?
I know I sort of went through it
a little fast.
The sample code is available on
the website.
Also the slides and the script
as I said are also on the white
so you can replay through those
or the recording later if you
want.
Again, I'm more than happy to
take any questions about that or
the other Edirect stuff that's
on your mind.
Folks are quiet today.
It's okay.
Hopefully you got a little
something out of the new content
anyway, if you didn't have any
specific, burning questions.
For me I'll often have the
question when I'm working on the
stuff and not necessarily when I
have an opportunity to ask the
question.
So I totally understand that.
But, again, if you do have
questions, we'll be more than
happy to answer them.
And, if you don't have questions
right now but do have questions
at a later time, you can always
reach out to us using the
contact link on our website.
You can go to contact us right
up here and then click on
customer support.
All of that gets routed directly
to the training team.
If you have questions about
EDirect or any of the insider's
guide stuff or suggestions or
comments, just click on this
customer support button right
here, and that will go directly
to us and we'll be able to look
up your question an hopefully
get you an answer back.
All right.
Make sure I didn't miss
anything.
If I missed your question,
please let me know as well.
All right.
That seems like all we have.
That all the content I have for
you.
That's all the questions we
have.
Kate's put the evaluation link
in there.
As she points out, it will also
pop up automatically when you
close the session.
Please fill that out and let us
know how we're doing and let us
know if you have suggestions or
for changes and improvements so
that we can make this better in
the future, and with that I
think we'll call it a day.
Thanks very much for joining us.