Welcome back to EDirect for
PubMed.
Today is part four, xtract conditional arguments.
My name is Mike Davidson, I'm a librarian here at the
National Library of Medicine in Bethesda, Maryland.
I'm here today with Pete Seibert, who is
answering questions in the chat
as well as Kate Majewski and Sarah Helson who are also helping out with that.
As you know, this is part four
out of five for the EDirect for PubMed series.
So we're over halfway through.
Good job sticking with us.
In the first session, Pete talked to you about
getting PubMed data using esearch and efetch. In the second session, Sarah showed you
how to extract specific data elements from PubMed XML and arrange
them in a custom tabular format using the xtract command.
Last time, Kate told you more about Xtract
including how to customize separators and group elements together
using block and covered some
Unix tools for working with
files and today we'll finish up
some of the staff we ran out of
time for last time as well as
talking about conditional
arguments which will help us
refine our output to get
precisely the data we want.
For today's agenda, we're going
to start with a quick recap of
part three, covering epost as
well we didn't get to fully
cover last time.
Then we're going to dive to the
xtract conditional arguments, starting with the -if argument
we're then going to talk about how to use multiple conditions together
at the same time with -and and -or
We'll finish up the conditional discussion
with the -position argument, which lets you limit your output
by an elements location within the XML record,
And finally we'll look at one of the other xtract arguments that
lets you define placeholders for blank cells,
which I believe was a question that somebody
asked in a previous class,
so this is where your answer to that comes up.
So for our recap of last time,
last time Kate talked to you
about different way to see group
and format your output and she
started off with the tab
argument which lets you
customize the separator between
columns in your output, and the sep argument
which lets you customize the separator between multiple values in the same column.
She also talked about block which is used to
select and group child elements of the same parent, and if you recall,
she used that with author data, to group the author last names with the corresponding initials belong to the same author.
She also talked a little bit
about working with files
including the greater than sign
to save your output to a file.
She also mentioned cat which she
used to read the contents of a
file, like a search string and
pull that search string out of
the text file into Esearch
and finally she started to talk about epost,
which she used to save a list of PMIDs to the history server, so we could retrieve
the records with efetch. The way Kate described that was,
If you keep a bunch of PMIDs in a
spreadsheet somewhere and want
to pull up all of the records,
this is how you would do that.
Because we sort of ran out of
time to cover that in full on
Monday, I'm going to cover some
of that right now.
We'll step back just a little
bit.
When we introduce the pipe,
going back to Pete in the first
session, we were basically
talking with Esearch and Efetch
and said we were piping the
one command and using them as the basis for another.
If you want to think like that, that is totally fine.
But remember, when we do an esearch, we aren't actually getting out a list of PMIDS.
We're getting this XML snippet
with the number of PMID that's
met our search criteria but also
the WebEnv and query key.
The results of the Esearch
including the full list of PMIDs
is being stored to the NCBI history server
The history server is what keeps track of your previous searches. Just like Advanced Search Builder in the
web version of PubMed, the history server here
works exactly the same way.
The WebENV, or web environment,
tracks your session to identify
your specific history and not
some other users and the query
key indicates which query in
your history you're actually
talking about, so what specific
query.
And when you pipe something from
Esearch to Efetch, what the pipe
is actually doing is piping the
web environment and query key
information from the Esearch to
the Efetch.
The Esearch is also saving the
results on the history server
including both the PMIDs and the
fact this they come from PubMed.
And then the web environment and
query key we piped over to
Efetch tells which results to pull down
from the history server.
Most of the time you can think of
the pipe as piping PMIDs from one command to another, but the fact that it actually works this way
lets us do some more interesting
and powerful things, because Esearch isn't the only command
that can put PMIDs, on the history server.
The EDirect command can be used to upload a series of PMIDs to the history server
just like esearch can, but you just specify the PMIDs
you want to upload.
The piping part works exactly
the same way.
The data base and PMIDs go up to
the history server and web
environment and query goes over
to the Efetch which knows which
information to pull down from
the history server.
So let me demo how this actually
works in a terminal, in using
the actual commands.
So let me make sure my window's
big enough here.
Going to make it a little bit
bigger.
There we go.
That's readable.
And so I'm just going to start
off with a basic Epost command
and this command is actually in
your -- is in your handout from
part three.
So I'm just going to copy and
paste that.
And it's just Epost, we have to
specify our data base so we know
what data base these IDs these
are referring to and specify
which PMIDs we want to post.
And when we execute that, we get
again the same sort of snippet
that we got out of an Esearch
and saying we're putting these
two PMIDs up, here's the web
environment.
Here's the query key.
We can take that same command
and pipe it into an Efetch just
like we would have an Esearch
and we can retrieve it in
whatever format we want.
I'm going to use abstract
because it's a little more
condensed right now.
We don't need to specify the
data base or IDs.
And when I execute that, I'm
retrieving the abstracts from
those two PMIDs.
But of course that doesn't
actually get us anything that
Efetch can't do on its own.
You can specify the database and IDs directly in efetch, which would save you an extra command.
However, I'm going to clear my
screen now.
What we really want to do is use
another Epost argument input,
dash input.
You might recall that Sarah
mentioned that briefly that you
can use the input argument with
the xtract command and you can use that to pull XML from a file
on your computer
rather than piping it in from a
Efetch.
And Epost has an input argument
that works the same way but
you're inputting a file with a
list of PMIDs rather than
inputting a file with XML.
If you recall from last time, we
actually had you run a command
that output a list of PMIDs, one
per line, and saved that list to
a file.
I'm going to use the ls command
that lists the files in my home
directory.
I probably have a few more than
you did because of other demos
and stuff we've run here.
We can see this special
name.CSV.
That was the special file that
had the PMIDs in it, one per
line.
And if I run the cat command,
should be able to see what's in
it.
Yeah, I see the four PMIDs in
it.
That's the file we want to get
the PMIDs out of.
What we can do is just do an
Epost input, specialname.CSV.
We still need to specify the
data base.
And then pipe that to an Efetch.
We're in business.
We basically pulled those PMIDs
out of that file just like if
you have any file, any
spreadsheet of PMIDs lying
around.
You can pull those out of the
file and get the information for
each of those records using
Efetch.
You can use it do an abstract or
do them in XMLand pipe that
into an Xtract.
All right.
So that's the basics of Epost.
I did a little fast, I know.
But we have a lot of other stuff
to cover today.
I'm going to go back to the
slides now and see if there are
any questions from last class,
from the homework, if you have
any questions about Epost, go
ahead and put those in the chat
box.
Before I go back into the
questions, I'm actually going to
point out one thing that one of
your classmates mentioned to us
which is that we made a mistake
in the answers to last time's
homework.
And specifically the mistake was
in the answer to question three.
Now the answer to question three
was a perfectly good answer, it
was just to a different
question, not to a question we
asked you.
So we updated the correct answer
and it's now on the website.
This is the sample code question
three for session three.
What the problem with the answer
was was the Esearch and Efetch
were fine.
It was just Xtracting the wrong
information to give you a list
of grant agencies.
It was giving you a list of
authors instead.
We updated it there.
If you still have any questions
about it, we'd be happy to
answer them.
Thank you to Amy for pointing
that out.
Any questions I missed?
Barry has a question: It does not seem to matter whether the file of PMIDS is a CSV
or a text file, is that right? That's a great question, Barry.
For the most part Unix doesn't
care what file extension you
have on a file.
It's not really something that
Unix cares about in most cases.
So CSVs, text files, even
XML files they are all just plaint text files, that we put these extensions on because
Window or Mac applications can then
figure out how to treat the
content of those files.
To answer your question
specifically, does it matter
when the list of PMIDs is a text
file, it does not because they
are both plain text file
formats.
You can't use an XLS or XLSX Excel
file because those have special
proprietary formatting on there.
We also have another question
here, does not seem to matter
the order we write Epost.
That's true.
I believe for Epost you can put
the arguments in a variety of
different orders.
So that should be -- that should
answer that question.
And Brian asked us will it work
with windows end of line format?
I'm not 100% sure about that.
I think it also depends on
exactly what shell and what
flavor of Linux and what flavor
of Unix you're working with.
I would test that.
I don't have a good answer on
that.
Asking for a link to the
corrections from last session,
somebody will put that in there.
For now I think we're going to
move on because we have, as I
said, a lot of stuff to cover
today it.
Bear with me just one moment.
Okay.
So moving on to some new stuff.
Again, remember our theme for
this entire class.
We keep saying EDirect is a tool
to help you get the data you
need and only the data you need
in the exact format that you
need it in.
And how do you do that with
EDirect?
We already told you about
Esearch and Efetch which gets
you the data you need.
What Kate talked about last
time with tab and sep was getting you the format you need.
But in order to filter your results to just the data you need,
you'll want to use a special set
of Xtract arguments known as
conditional arguments which is
what we're going to talk about
today.
And the conditional argument
you'll probably use the most is
if.
Now if lets you limit the data
included in the table Xtract
creates included in your output
table depending on certain
conditions.
So data will only be included in
your output if it matches the
conditions you specify.
And for those of you who have
done even a little bit of computer
programming, you'll be familiar
with the if-then statements.
The idea is if a condition is
true, then follow a specified
set of instructions.
If the condition is not true,
then don't.
So for xtract's if argument ,we're
going to follow the same
structure.
If a condition is met for
a given pattern, then create a
new row for the pattern and
populate the specified columns.
If the condition isn't met for a
pattern, we'll skip that one, go
on to the next one and check
that one.
Let me walk you through an
example of when and why you
might want to do this.
Let's say we want a list of
authors in our result set with ORCID IDs
and we want the names and IDs of each author.
I have an efetch here that is also on your handout,
and I know that some of these records have ORCIDs, so that will be a good test.
The first thing we need to do is
look at some PubMed XML to
figure out where to get that ORCID ID.
To do that, I'm going to hop out of my slides again, and I'm going to take those PMIDs from my efetch
I'm going to search them in
PubMed and take a look at the
XML.
So we should be getting pretty
familiar with this by going up
to the format here and XML.
I could also find this information by looking through the documentation that
Sarah pointed you to last
week at this time.
Instead I'm going to search down
here and look for ORCID which is
right here and this looks like
an ORCID ID, and we can see that
it is in the identifier element.
And I know that for the most
part almost exclusively the only
information in an identifier
element anywhere in PubMed is
ORCIDs.
So if there's any identifier
element on a record, it pretty
much has what we want.
So knowing that we're going to
go in and start a rough draft of
this command.
I'm going to start off by
clearing my screen and then copy
my Efetch and start building my
Xtract.
For this we said we wanted a
list of authors, not a list of
PubMed records.
We've done this a couple times
before.
Usually our pattern is PubMed
article but this time I want it
to be -pattern Author, because I want it to be a list of authors, one author per
row.
And then in my element argument
I'm going to specify my columns
here.
I'm going to do last name and
initials and identifier, but I'm
going to use the trick that Kate
taught you last time which is
I'm going to group last names
and initials together in the same column, using a
comma.
I'm going to put a SEP of a space in there.
So my last name and initials should be in the first column, separated by a space, then we'll have a tab, then we'll have our ORCID ID.
My results look pretty good.
I'm getting the right layout of
columns here, the right layout
of data.
We're getting authors that have
ORCIDs but we're also getting
authors that don't.
I could make a quick change.
I'm going to clear my screen
again and I'm going to take that
same set of code but I'm going
to go in and add something else.
Right after the pattern argument
I'm going to add my if argument.
And so, as I said before, if
tells us to include in our
output any pattern if the
condition we define is met for
that pattern.
In this case the condition is
the condition is the existence of an identifier
element inside this author,
inside this pattern.
And when I execute this, I'm
down to just those records that
I wanted, just those authors
that I wanted, the ones that
have ORCIDs.
So I'm going to go back to our
if-then structure for those of
you that think differently, a
little bit of presentation of
the same data.
If the pattern has an identifier
element, so if the author has an
identifier element, then create
a new row for the pattern with
two columns of data, last name
and initials in the first column
and identifier in the second
column, again, if the pattern
does not have an identifier
element, Xtract will skip it and
move on to the next one.
So to visualize how this works,
I want to go back to that triple
box format that Kate showed you
last time.
If you remember we had our input
XML on the left, output of our
command on the right and actual
command is down on the bottom.
Working through this step by
step, as always Xtract looks for
the first occurrence of our
pattern author which is right up
here at the top.
Rather than creating a new row
right away, first we check to
see whether that author has an
identifier element in it, whether
the pattern meets our condition.
This one does so we're going to
create a new row with the last
name, initials, and identifier
for that author.
Once that's all done, we'll move
on to the next author and check
to see whether this one has an
identifier.
This one doesn't have an identifier element,
so we're just going to skip it. We're not going to
create a new row, we're going to go right on down to the next author.
And this one does have an
identifier again so we will
create another row this time
with last name, initials and
identifier.
Any questions about the basic
concept of if?
This is a fundamental thing that
a lot of the stuff today builds
on, so if you have any
questions, this is a great time
to ask, and we're also about to
start an exercise.
I'll start explaining the
exercise.
If you have any questions, go
ahead and finish typing them in.
For our first exercise, what we
want to do is write an Xtract
command that only includes
PubMed records in the output if
they have mesh headings on them.
So your output's going to be one
row for PubMed record and it's
going to have two columns, PMID
in the first column and citation
status in the second column.
We have an Efetch here that you
can use to test.
I'll give you a few minutes to
work on this it.
One way you might want to
approach this is writing the
code that will get you the
output you would want.
Then go back in and edit back in
to add your conditional.
We'll give you a few minutes to
work on this.
The answers are at the bottom.
When you are all set, please
give me a green check in the
participants panel.
All right.
I'll give you a few minutes.
>>> Try to wrap up what you're
doing.
Amy was mentioned she was
getting the right data but was
having problems.
It doesn't matter exactly how
you look as long as you're
getting the right stuff and not
getting the stuff you shouldn't
be getting.
It's fine.
We're going to walk through it
together.
I'll show you how I would walk
through this and then if you
still have questions, we can go
through them.
Let me clear out the screen, get
my Efetch for the exercise.
Okay.
So, like I said, what I'm going
to try to do is start off by
getting all of the output that I
want and then go back and do my
limiting.
So in that case I'm going to
start off with my Xtract pattern
is going to be PubMed article
because we want one row for
record.
That should be pretty familiar,
and then the data we're trying
to get in this case is the PMID
and the citation status.
Both of these are ones we should
be pretty familiar with.
We can get PMID using parent
child construction and we can
get citation status using the
attribute status on the Medline
citation record using that at
sign.
Let me run that and see what
that comes up with.
Okay.
So that's getting the PMID and
the citation status in the two
columns.
That's what I wanted.
Now the next thing that I need
to figure out is how to limit it
down to the PubMed records that
have mesh headings on them.
So I'm going to take the same
code that I just ran, paste them
back in there.
Oops, copied too much.
Sorry about that.
Paste that in there and I'm
going to go back in and add my
conditions.
So in this case my condition,
and there are a couple of
different ways you could do that
this.
The way I'm going to do it is by
putting in an -if MeshHeading
condition.
So basically what that's saying
is only include PubMed records
in my output if they have a mesh
heading element.
You could also use descriptor
name.
That could work just as well
which is a theme for a lot of
the stuff we're going to work on
today.
If you get the right answer, it
works.
If you get something that works,
that is correct.
So in this case, I run that and
my output is limited down to
just those three records.
So that has done what I wanted it
to do, so that works for me, and
you'll notice that the status of
all of these three records is
Medline which makes sense if
you're familiar with Medline
status and Medline records in
PubMed because you probably know
that only Medline status records
have mesh headings attached to
them.
So this exercise was one of the
ways of limiting your results to
only Medline records if that is
what you're interested in doing.
And again following that same
theme that I was mentioning,
there are plenty of other ways
to accomplish this same task.
If you're using an Esearch you
could include a tagged mesh
heading in your Esearch query
which would limit yourself to
just Medline records.
You could add an explicit restriction to the MEDLINE subset
by adding Medline tagged
with SB in brackets, to your search query.
And the feature that I'm about to show you gives you yet another way to include
only MEDLINE records, if that's what you want to do.
I'm going to clear your statuses.
We're going to move on from this exercise
If you are still working on it,
I'm going to ask you to stop and
follow along with this one and we can circle back a little bit later.
Serdar's asking what happens if the MeshHeading area is present but empty.
That should not happen.
The only time you'll see a
MeshHeading is if it has a
DescriptorName in it.
Let's talk about another
possible way to limit yourself to only
only to Medline records.
If you did want to do it, here's
a way you could do it.
I'm going to clear that screen
again.
In that exercise our condition
was the presence of an element in a pattern.
So the presence of the MeshHeading element.
If there was one there, we would
include the record.
If there wasn't the mesh heading
there, we wouldn't include
the record.
However, we can also include
data based on the value of an
element or attribute, as opposed to its simple presence.
Let me switch back to the slides, and talk to you about -equals.
Like before we can use the if
argument to specify an element
but now we'll follow that up
with an equals argument to
specify what the value of that element should be equal to.
In this case, we will include the pattern if
ISOAbbreviation equals JAMA
Again following back to our same
structure that we've been
following before.
If the element Iso abbreviation
equals JAMA, them we will create
a new row for the pattern with the
two columns, volume and issue.
This highlights another useful
thing about if that you may have
figured out but it doesn't hurt
to reiterate it.
The data in your columns doesn't
have to be from the same element
used in your condition.
So our -if element here was ISOAbbreviation, but we're not outputting
ISOAbbreviation.
We're outputting Volume and Issue, two completely
different elements.
I will also point out for those
of you mentioning Epost works in
different orders with these
condition, you do want to make
sure that the order is correct.
You do have to have the iffirst
and then the equals.
If you flip the things around
you're not going to like your
outputs.
And sort of expanding a little
bit on that if equals concept,
we can also use attributes in
our conditions.
Just like we did before the with
the at sign to specify the
element name and then the at
sign and attribute name and you
can use that in an if just like
you would an element name.
In this example if the attribute
status for the Medline citation
element equals Medline, has the
value Medline, then create a new
row for that pattern, in this
case the one column, PMID.
If there is no attribute status, or if the attribute status
doesn't equal Medline, then
we'll just skip it.
As I said, this is another way
that you could restrict your
results to only Medline records.
Lots of different ways to
accomplish the same task.
And that equals argument that we
were talking about does work
great if you know exactly what
the element in your if argument
should be equal to, but you also
do have some other options to
work with if.
If you're looking for a partial
match and so the contains
argument, the starts with
argument, the end with argument
all work in place of an equals.
You work them with an if to get
different kinds of matches.
So for an example with contains,
we'll follow our same format,
try to drum this into you, this if-then idea,
that's sort of the key element
here in this conditional, for
most of these conditional
arguments.
This example is a little
different because our pattern,
we're going back to author, not
PubMed article, so each row
should have one author rather
than one record, and in this
case, if the element affiliation
contains the word Japan, then we
will create a new row for that
author with the last name and
initials in the first column and
the affiliation in the second
column, and as always if the
condition is not met, Xtract
skips the pattern and moves on
to the next author. This -contains is
especially useful for
affiliation elements, as not
everyone from the same institution always formats the
affiliation data the exact same
way so you can use this to
capture variations to use
contains to get a partial match.
Any questions about any of the
stuff that we've talked about so
far?
>> Mike, Emily has a question in
chat.
She wants to know if you can or
multiple things?
>> That's a great question,
Emily.
You are jumping ahead about five
slides.
So hang on.
I will answer that in just a few
minutes.
But first we're going to have
another exercise.
Oh, is capitalization important?
That's also a good question.
I believe that the matching on
that would be case sensitive.
I'm not 100% sure on that.
I haven't played around with
that as well.
In this case, for contains it
would be, and I know we
mentioned Esearch,
capitalization is not important.
It is case insensitive.
That's because that's using that
PubMed search engine where case
doesn't matter.
Because this is using a Unix
command, I believe case would
matter for contains here.
Okay.
We're going to get started with
the exercise.
If you have more questions,
please keep them coming.
This exercise a little bit more
complicated than the previous
one we did, sort of building on
the previous things.
We want again an extract command
that only includes PubMed
records for articles published
in one of the JAMA journals.
So this could be JAMA itself or
it could be JAMA Cardiology, JAMA oncology,
JAMA dermatology, etc.
We know that the journal title abbreviation is going to start with
JAMA.
Again, a couple of ways that you
can solve this one, I'm
interested to see how you folks
work it out, for our output we
want one row for PubMed record
and we want two columns, PMID
and Iso abbreviation is going to
be our other element which
you're probably familiar with
by now is the element for Journal Title Abbreviation.
You can use the efetch that's here to get your sample data.
It's also on your handout.
I'll give you a few minutes to
work on this and you can put
that green check mark in the
participant's panel when
you're all set.
If you get real stuck, please feel
free to look at the hint in the
chat box or the answer at the
bottom.
There's absolutely no problem in
doing that.
I'll give you a few minutes to
work on this.
All right.
Let's walk through this one
together and I see a couple of
you are asking some questions
which are actually some very
good questions which we're going
to address in just a minute.
Let's walk through a answer and
see what some other answers
could be.
Get back out to here, going to
grab my Efetch.
As per usual, I'm going to build
most of my command first to make
sure that I'm getting the output
that I want and I'm going to
limit it with my condition.
All right.
So this is giving you one row
for PubMed record and it is
giving me the two columns that I
want for my output.
Take a look.
Looks good.
I have five records here and two
of them start with JAMA.
In my ideal output, I should be
getting just those two.
This is another nice way you can
go about testing conditions.
You don't have to have the
element you're conditioning on
in your output, but for testing, it's often
nice to double check
and make sure you got that
part right.
First let us do the actual
solution to this problem.
And we're going to go back and
add in our condition.
In this case we wanted to only
include the records where the
Iso abbreviation started with
JAMA.
So if Iso abbreviation and I use
starts with JAMA.
Which you pointed out, you could
also use contains provided that
met your particular goal in that
particular case.
So I'll execute this just
so we can see it.
Two records of output with just
the JAMA at the beginning.
If there is a record that had
JAMA somewhere else in it and
you did not want to include
that, you'll want to use starts
with.
It's all about solving a problem
you want to solve and there are
often many different ways to do
that.
Okay.
And any questions that we need
to address.
All right.
Pat has a question having
trouble keeping track of which
spaces are required and when
they are not.
I've been using the cheat sheet
for the appendix.
Any questions on how to keep
this straight or figure it out.
So I'm trying to figure out what
-- it depends on what you mean
by when spaces are needed.
Okay.
So if you're looking at -- if
you're looking at my previous
command here, right, so there's
a space between the command and
the argument name and them
between the argument name and
the actual stuff that's in the
argument, with the exception
right here of ID, right, because
IDhas all of these things
grouped together with commas or
like in Esearch where there's
quotes around a bunch of stuff,
so Unix has a couple of
different ways for grouping
information in general.
When you have an argument you're
generally specifying one
argument separated in spaces, so
we have ID, space, the value of the
argument, the stuff that I want,
the ID's that I want to
retrieve and then a space.
If you still have more questions
about that, please go ahead and
ask them.
I'm going to move on for now and
come back and circle around and
make sure I got you covered in a
little bit.
>>> We are going to move on to a
slightly more complicated sort
of elaboration on what we've
talked about so far and so if
you are still working on that
previous exercise, I'd ask you
to stop for right now because
this next part's a little tricky
is grasp and it's going to be
really important because so far
all of our conditional examples
have been in a pattern and we
are only including the pattern
in our output if that condition
is met.
However, we can also put an if
argument inside a block, and if
you remember back to last time,
Kate showed you how to use block
when dealing with authors to
keep the corresponding last name
and initials together.
And sort of a quick refresher,
what Xtract did is look for the first occurrence of the -block
Author within the pattern,
pulled the last name and initials from inside that block,
then moved onto the next -block Author, LastName and Initials
and continued on through that pattern until there
are no more blocks in the
pattern.
So that's again quick refresher
on how block works.
However, we can then sort of add
on to that by putting an if
argument inside a block to limit
and make sure we only include
certain blocks in our output
based on if they meet the
condition.
Instead of filtering out entire
patterns we're going to include
every pattern but only filter
out the blocks that don't meet
our condition.
For example, let's say we want
to see the PMIDs for all of the
records in our Efetch and we
also want to see the DOI, if
there is one for that record.
So first we need to figure out
where the DOI is, and I'm going
to hop over to some XML.
I'm going to scan through here
and see if I can find some
DOIs.
There we go.
That looks like a DOI right
here.
So we have a DOI, and we can see
that it's in the articleId
element.
But we also see that there are
three other article IDelements
that aren't necessarily DOIs
and we only want the DOIs.
So in this case we only want
ArticleId elements where the
IDtype is DOI.
But this is tricky because if
we made our condition for the
entire pattern based on this
attribute, it would still
include this record but include
all of our article IDs in our output.
We're going to use an -if in a block to
check through each of these ArticleIds one at a time,
and only include the ones that we
want.
I'm going to copy and paste the
code for this one because I
always type it wrong.
Paste that in there.
Okay.
So we've got a standard Efetch
here.
In our first line we're going to
create a new row for every
PubMed record.
We're going to have the PMID for
each record because we don't have
any condition before the
-element argument, so every pattern will be included.
On our next line we have our
block argument of articleID,
which allows us to focus on
the contents of one ArticleID element at
a time. Inside that block
we have an if and an equals and
then our element.
So Xtract is going to check each
articleID block in the pattern
to see if the IDtype attribute
equals DOI.
If it does, we will put the
article IDelement in our second
column.
If it doesn't, we're going to
skip that block and check the
next one and keep checking each articleID
to see if there's any DOIs, in there, but we're only going to add the ArticleID
to the output if it meets our condition.
So this is what that actually
looks like.
The end results should be one
row per PubMed record and each
row will have a PMID in the
first column and the second
column of each row will be
either an article ID, if there's
an article IDwith the DOI, or
it will be blank.
I know that's kind of
complicated.
It's kind of an important one
though.
So if you have any questions, go
ahead and ask them now.
I'm going to be checking the
chat here.
Okay.
Doesn't look like we have any
questions about that.
Hopefully you all are still with
me.
If you're really struggling,
please ask questions so we can
make sure that we get you
answers.
Would you use the same approach
to pull a free text URL from
PMC?
Brian, good question.
I am not 100% familiar with how
the data is structured with PMC.
I might circle back with you
maybe on Monday.
I might have an answer for you
on that.
Pat, I'm going to come back to
you at the end because we'll
deal with the space question a
little bit later on.
Okay.
I think we're all go on that,
which is good, because we're
going to make it even more
complicated with something that
many of you have been asking
about for weeks now which is
combining multiple conditions.
Sometimes you may want to
include multiple conditions in
the same Xtract and include only
patterns or blocks that meet at
least one of your different
conditions or meet all of your
conditions.
In order to do that, you would
want to use the or and and arguments.
Those of you who were already asking about this, and probably
many of the rest of you are familiar with Boolean logic.
So with an -or, at least one of the conditions you specify must be true in order to include
that pattern or block.
With and, all of the conditions you specify must be true.
So the syntax on this is just a
little bit more complicated with
what we've been doing.
We're going to start down here
with our first condition.
It's just the same as it was
before.
We have an if argument here and
we can have an equal or contains
or anything else to specify what
that should be equal to or
contain.
For our second condition we're
not going to start it with an
if.
We're going to start it with an
or, we have or and then the
attribute name and equal or
contains.
We don't need to have those if
we don't want.
We can use those if we need
them.
So working through this using
our same sort of general pattern
that we've been using, if the
IDtype attribute for the
article IDelement equals DOIor
equals PMC, if either one is
true, then we'll put the value
of that article IDin the second
column.
If neither of those conditions
are true, we're going to skip
the articleID block and move on to the next articeID block, check
whether that one meets either of these
conditions and keep going
through to see if any of the
other article IDs meet any of
the conditions.
If they do, we'll include them.
If they don't, we'll skip them.
So when we execute that -- I'll
copy and paste that as well.
This is just the same code as we
were looking at on the slide.
And you see our output is just a
little different than before.
We have a PMID for every row but
you can see because this process
repeats for every ArticleID in the pattern,
if a PubMed record has both a DOI and a PMCID,
then both the DOI and the PMICID will be presented in that row,
like in the third row here, we
have both of those because they
both met the condition because
they were both included.
Any questions about or?
And if you have questions about
and, hold on a second because
that's what we're doing next but
do you have any questions about
or so far?
Okay.
Good.
And works very similarly to or,
at least in terms of the syntax.
Obviously it's the reverse
though because this pattern, the
pattern or block must satisfy all of
the conditions that are anded
together in order to be
included.
So for this example here, we
have our if, last name here is
followed by equals but our and
is not.
Just like with if, you don't
need an equals if you're just
checking for the presence of a
particular element.
So in this one, if the pattern
has a last name element with the
value KAMAL, and if the pattern
has any affiliation element, any
affiliation element is present
for that author, then we will
create the row for that author, with two columns:
Last name/initials in the first column, and affiliation in the second.
If either of those things is not
true, we're just going to skip
it.
I have one more and example that
I want to show you, and for this
one we only want to include
records that have any mesh
heading that contains the words
Zika virus and to have the mesh
heading microcephaly.
So for this xtract, if the
pattern has a descriptor name
that contains the string Zika
virus and the pattern has a
descriptor name element that
equal microcephaly, if both are
true, we will create a new row
for the new pattern with PMID
and article title.
If either of those things is not
true, we're going to skip the
pattern and move on.
I want to take a closer look at
what's happening here with some
mockup XML.
This is showing some of the mesh
headings for a PubMed record,
and this particular record would
be included in our results
because it has a descriptor name
element that contains the string
Zika virus, and it also has a
descriptor name element that
equals microcephaly.
Both of those things are true.
So it would be included in our
result, and this is why we use
contains for Zika virus instead
of equals.
We wanted to make sure we
included records like this one
that has Zika virus infection
but doesn't have the heading
Zika virus.
Okay.
Any questions about and or or or
anything that we need to address
right now before we go into our
conditional and-or exercise?
This one's tricky so I'm going
to give you a little bit of time on it.
I'm going to take a little time
time to explain it because it's kind of complicated.
This is another one of those examples, that Kate did
one with you last time, and
Sarah did one at the end of her
session.
This is combining everything
that we've done so far.
So we're starting at the
beginning.
We're not giving you an Efetch
on this.
Our goal is to do a search for
BHSmith as the author and see
the different affiliations that
are listed for that author.
We're going to limit it to the
last five years, 2012 through
2017.
And in this particular case,
this is only a search that we
have run in previous classes and
I think somebody mentioned that
it looked like we were dealing
with multiple BHSmiths in our
out put.
This is a really great tool
using this affiliation data to
help disambiguate those authors,
to see the difference between a
BHSmith in one area versus
another one somewhere else.
For this particular exercise we
only want to see the affiliation
data for BHSmith, regardless of
how many authors are on each
citation.
We don't care about any
of the other coauthors.
We just care about BHSmith and
we want our output to be a table
of citations for each data.
We're going to have a PMID for
every record and author last
name and initials for BHSmith. So that's always going to be "Smith BH".
That is the proper PubMed
format, last name followed by
initials.
It's always going to be Smith
BH, because that's the only author we're going to be looking at.
Our conditions should limit us to that, but it will help us make sure we're on the right rack.
Finally we want affiliation data
as well because that's sort of
the whole point about this.
I want you to come up with the
whole script.
I'm going to give you plenty of
time to work on this.
It is pretty tricky.
I will say feel free to look at
the hints in the chat box and
feel free, look at the answers
at the end.
No shame in that.
Clear your status and give me a
green check when you are all
set.
all right.
I gather that some of you are
struggling.
I don't blame you.
This is a tough one.
We're trying to synthesize a lot
of different pieces together
here.
I'm going to show you my
solution.
There might be multiple
different ways to do this.
We'll talk about what's going on
here and how and why we did what
we did.
If I can remember how to copy
and paste here.
Sorry about that.
All right.
Let me clear my screen, paste in
my answer.
Actually, I'm going to clear my
screen again and paste in my
answer slightly nicer so you can
actually read all of it.
There you go, and then we can
talk about why we did what we
did.
Okay.
So as I suggested we started off
with an Esearch, and this is
actually an Esearch that we've
run a couple of times already
searching for an author, Smith
space BH, tagged with author.
You should all be good searching at searching for authors by now.
Then we have the -datetype PDAT.
We have the Min date and Maxdate to specify my date range.
That's good so far, that gives me the records that I want to look at.
I piped that over to Efetch to
get the XML, and I piped that into the Xtract.
We've been doing that since
session two.
When we get into our Xtract, even our first line of our xtract is pretty straightforward.
We want a table of citation.
So pattern, PubMed article is
going to be doing what we're
going to be doing and our PMID
we wanted for every record.
Then we wanted to just look at
the affiliation data for
BHSmith.
In order to do that, we needed
to check each author one at a
time to see if that author is
BHSmith.
That means taking block.
We take a block author which
lets us look one author at a
time and we're only going to
include the author in the output
if that author's last name
equals Smith and their initials
equals BH.
So we should only be including
the BH Smith author data in
our output.
All of the other authors we're
going to ignore entirely.
What data are we going to get
from BHSmith, last name and
initials grouped in a column.
We just want to make sure and
then we're going to get the
affiliation.
I'm using the -sep space comma
trick which is how I like to do
the authors.
If you had no SEPthere and just
those in separate columns or even in
the same column, that would be
fine.
That would work just as well.
Your formatting would be
slightly different.
It will take a few moments to
complete, probably not as long
as folks who are trying to
contact us from farther away.
But hang on a second.
It might be our servers are a
little slow right now.
Five years might be a lot.
There we go.
It took a little bit.
But we got our Pmids, our Smith
BHs and we get our affiliation
data, and if we look through
this affiliation data, you can
see we have a couple of
affiliations from South
Carolina.
We have a couple of affiliations
from Scotland so maybe these are
multiple BHSmiths and we need
to disambiguate.
And that's why we can do this.
Brian pointed out that not all
of these BHSmiths have
affiliation data.
Affiliation is not a required
field, and up until recently
affiliation data could only be,
in the last 10 or 15 years,
affiliation data could only be
accepted for the first author
which probably explains why some
of these are missing that
affiliation data.
We didn't necessarily tell it to
exclude records that didn't have
affiliation data for that
author.
Again, I know that this is
tricky.
This is a sort of complex
research project packed into a
ten minute exercise.
So if you are still struggling
with this, I do encourage you to
go back and take another look at
it after class.
Let me take a look at the
questions because I do want to
move on to something, changing
gears a little a bit.
All right.
Kate's got that so I'm going to
move on because we are a little
short on time.
Clear my screen here, and we're
going to change to a slightly
different topic, again still
conditional but just a little
bit of a different thing.
Get my slides organized here.
So as I said there's one more
conditional argument that we
haven't talked about as yet
which is still very useful.
It can be pretty useful for
isolating specific parts of your
data.
So for this example let's say
you want to include author data
that is in your table but you
only want the first author and
you can't really do this with
equals or contain unless you
know the name of the first
author for each record.
That's just not what equals or
contains can do.
You need to know the value you
wanted to match.
However, you can do this using
the position argument.
And you use position in
combination with block.
And earlier today if we talked
about if and block, we are
including a block in our output
only if it met our condition,
and with position we can include
a block in our output only if it
is the first occurrence of the
block or the second occurrence
of the block or the third
occurrence of the block.
So if we want to include only
the first block, we would add a
position first argument right
after the block.
If we wanted only the last block
we could add position last.
This is especially useful in
this last part since we can
specify that we want the last
block even if we don't know how
many blocks there are.
When using position you can also specify a -block by number, if you want.
So -position and then the number 1 works out to the same thing as -position first.
I'll show you what that looks
like in practice.
So I'm going to start off with a
script that Kate ran last time
or a version of a script Kate
ran last time and what you'll
see is that it's basic Efetch.
This should be in your handout as well.
It should give you one row per PubMed record
with the PMID,
and then we are looking at authors one at a time, group the
last name with initials,
separating them with a space.
We get a list of each author
for the PMID with the last
name and initials grouped
together.
That was the basic use of block.
However, if instead of all of
the authors, we only want the
first author, we can add the
position argument.
The way we do that, I paste in
the same code, we add it right
after the block.
And this makes it so block works
the same way.
It works author by author but it
only is going to output the
first one.
When we do that, we get a nice
neat list of PMIDs and we get
last name and initials of just
the first author.
We take that same code and
modify it so get just the last
author by changing this first to
a last.
We can also specify an arbitrary
number, we can specify we just
want the third author or 16th
author by changing the last to a
number.
Any question on that?
Looks like Kate's handling that
one.
All right.
So we are going to move on to
our last topic today.
We're just about done but I do
want to show you one more
argument, and this one is not
technically a conditional
argument.
It's more closely related to the
other formatting argument like
tab and SEPthat Kate showed you
last time.
It is a way of adjusting your
output automatically based on
your input, and actually
somebody was asking for this
specifically, so make sure you
pay attention.
Clear my screen but go back to
that previous example.
Looking at just the first
authors from this group of
PubMed records but this time I'm
also going to add in another
piece of information.
I'm going to add in the
identifier.
So looking at this code we can
see -- again we're looking
author by author just doing the
first author, putting last name
and initials and the identifier
for that first author.
When I execute this, I get my
three columns of data, I get
PMID, first author last name and
initials, and I get ORCID ID.
This is all well and good but if
we now want to add a fourth
column after the identifier we
might get into trouble because
of the blanks.
I'm going to go ahead and paste
in that same code and add in
another element here.
I'm just going to add in last
name again just to sort of
demonstrate what I'm talking
about because when I execute
this, we can see that when a
column would be blank because
there's no data from the input
to put there, Xtract just skips
it.
This is causing alignment
problems in our tables because
where there's a blank in the
third column, xtract
puts the data that should good
in the fourth column right into that blank, and
everything gets shifted over and
your table becomes out of
alignment.
And the solution to this is the
def argument.
And the def argument lets you
specify a default value, def for default,
for the cells in your table. So you're specifying a
place holder.
Wherever a blank would be,
Xtract would put the default
instead.
So following up on our last
example, going to paste that in
again and we're going to put a
default of Na and I'm going to
do it right next to the tab in
SEP.
It's a formatting argument so
we'll put inside the block and after our condition, just like tab or sep.
Just stick it right here, slip
that in there.
And when I enter now our blanks
aren't blank anymore.
We have that place holder in
there.
Your columns may still look a
little bit out of alignment
depending on your terminal.
However, if you sent this to a
file, everything would look nice
and neat because you'd have
something in every column.
You would open it up in excel so
it would look great.
I see we have a couple of
questions.
I just want to do the wrapup
stuff first and then we'll dig
into a couple of the questions
that I saw up there.
All right.
Thank you, first of all, very
much for sticking with us so
far.
We're over the hill.
We're on the down swing.
That was by far the most
complicated stuff you'll have to
learn as part of this class.
Next Monday is our last session.
I'll be back again.
You're stuck with me.
We're going to try to put all of
this stuff together and look at
some strategies and techniques
for building solutions for real
problems from the ground up and
look at case studies suggested
by former students of this
class.
As always, the homework
questions are at the bottom of
the handouts for today's class.
The answers are on the sample
code page.
I think they are all correct
this time.
Sorry about that from last time.
I know that we haven't been
asking you to submit them.
We really do encourage you to do
them because they will, first of
all, help you expand your
knowledge and expand your skills
of these things.
Trying them out is the only way
you're going to get better at
this stuff but it sets you up
for the exam which we do ask you
to submit.
As always you can check out the
website for more documentation
and you can ask any of us
questions, if you have any
questions about the homework,
about any of the course content,
you can e-mail
NLMtrainers@NIH.GOV.