Kate Majewski: Welcome back to EDirect for PubMed.
Today is part three, Formatting Results and Using Unix Tools.
My name's Kate Majewski.
I'm a librarian here at NLM.
I'm here with Sarah Helson, who you will remember from
part two, and Mike Davidson, the creator of the series.
Remember our theme?
Getting the PubMed data you need, and only
the data you need, in the format you need.
Last week we started the process by getting
PubMed data using ESearch and EFetch.
And in the next session we will be restricting
our output to only the data we need.
Today we're going to format our data the way we want.
And we'll also talk about working with files so that
you can start to think of EDirect in the context
of the other tools that you might want you use.
Today we'll do a quick recap of part two.
Then we'll talk about customizing separators
with tab and sep, grouping elements with block,
saving results to a file, and reading from a file.
Last class, Sarah talked about extract which pulls
data from XML and arranges it into a table.
We used the pattern argument to create rows
for our table.
And we used the element argument to create
the columns for our table.
When using these or any of the extract arguments
that specify particular parts of an XML document,
we specify an XML element by using its name.
And we do so carefully, making sure your spelling
and capitalization are correct.
We can identify specific elements that are
in a particular location in the hierarchy
using the parent-child construction.
And we used the example of PMID that lives within
the Medline citation element.
And we can also identify attributes using
the at sign.
And we used the used the example of the status
attribute of the Medline citation element.
Let's review really quickly where to find
the course materials on the data guide site.
So, if you start at dataguide.nlm.nih.gov,
remember you can go to Classes and here's our set
of classes.
At the bottom of the screen you can find the class
materials linked.
So, on this page you're going to find the handouts,
the codes, slides, and the script.
And I want to particularly point out the sample
code right here.
So, this goes through the concepts taught in class.
It walks you through the examples.
And at the bottom you're going to find the annotated
homework solutions.
Speaking of homework, I'd like to -- hopefully you did the
homework and you got some practice finding XML elements
in the PubMed records.
And so, while I'm waiting for some questions --
I'm hoping you're going to ask me some
questions in chat.
But while I'm waiting for that, there was a question
from last class that I'd like to address.
And the question was whether you can extract MeSH UIDs
from PubMed records.
So, let's take a quick look at an example in PubMed.
And I will put the PMID in chat if you want
to follow along.
Okay.
So, as you practiced in homework, you can always
take a look at the XML in PubMed records
by changing the format.
And as we look through the data here for this example,
we can see that there's a section called
MeSH Heading List.
So, note in the XML MeSH headings are
labeled "descriptor name."
And subheadings are labeled "qualifier name."
And each of these have a unique identifier.
And here they are.
So, let's say I wanted two columns in my output,
the PMID in the first column and a list
of the MeSH descriptor UIDs in the second.
So, I want you ignore the qualifiers for now because
we're actually going to address that this class.
But just using the same sample set as last time,
and I'm going to give you the whole code here
up to my question.
Hopefully that pasted properly.
Okay.
So, using this same EFetch that we used last time,
asking for format XML I asked for two columns
in my output -- the PMID and then
the MeSH descriptor UIDs.
So, in your homework there was a similar question.
So, I'm just going to ask you what would be the next
thing I would type in to get that second column?
Go ahead and put it in chat.
And let me move -- no, I don't need to move, actually.
You can see it right below on the bottom of my screen.
Okay.
So, one of the things that we covered last time was how
to pull out an attribute.
And Amy, thank you very much [laughs].
Okay, Amy's got it.
So, what you want to do is, because we're looking for an
attribute, this is very similar to this Medline citation status.
And I think you did a homework question like this.
Okay.
So, let me type it in.
Okay, make sure that I got this right.
Okay.
All right.
So, I don't blame you if you didn't get that right away.
Because, you know, we did just cover that last time.
But did you understand what I just did?
Give me a green check.
All right, great.
Wonderful.
Okay, there may be a few who did not.
That's okay.
I will just ask you to review the homework from
last time again.
And then take a look at the PubMed XML one more time
and see if you can get it before next class.
All right.
So, I'm going to clear your status now.
Okay.
Oh, there's the homework question.
Oh, okay.
So, the question that's in chat is about what happens
if you want to reflect in your output that there
is no data in the record?
And we're going to get to that -- we haven't
got to that yet, have we?
Okay, yeah.
We're going to get to that.
So, just hold that question and we will answer it.
Okay.
All right.
So, we now know the proper syntax for pulling
out data from XML.
And today we're going to look at formatting that data cleanly.
So, I'm just going to stay here in my Cygwin,
my Unix terminal, to show you a new example.
So, let's say you want to retrieve a few
PubMed records and extract the PMID, the ISSN,
and the last name of all the authors for each record.
So, we're going to start with an EFetch, the same EFetch --
let's just keep it simple -- using that same example
from -- and it's also in your handout.
So, if you don't have your handout open, make sure
you get it open.
Okay.
So, here's our familiar EFetch.
We're using just a sample set.
Just to remind you if you haven't been thinking about it,
this is just a sample set that we're using to test to make sure
that we're getting the data we need.
Obviously, you're going to be replacing this with the i.d.s,
or a search, or something to be able to pull new data.
But we want a sample set to make sure that we're getting what
we're asking for -- or what we want, I should say.
We always get what we ask for.
We don't always get what we want.
All right.
And actually, before I even do this let me just pause another
second and say when you're copying and pasting from the
handout, or from anywhere, be sure that you're copying
from a clean code.
In other words, if you try to copy something from Word or
another program, a program like Word is going to introduce some
different characters into your code.
That's not going to work very well with Unix.
Unix is very particular.
Every character has a meaning.
You don't want extra spaces.
You're probably going to see me make the mistake.
You got to make sure that your code is very clean.
So, this is why we've made the handout -- which you should have
open right now -- available in a plain text file.
Because we want to make sure that we're not introducing any
characters that we don't want.
All right, sorry.
Back to the example.
So, we want one row per PubMed record.
So, we know what our pattern is going to be.
So, again, this should be familiar.
We want one row per record so we're using PubMed article as
our pattern.
One row per record.
And then we're going to define our elements.
Okay.
So, again now, it's been a while, so I got to remind you
what we're asking for.
So, first we want the PMID.
And PMID, the one that we want, lives in the Medline
citation element.
So, PMID.
Okay, then we want the ISSN in the second column.
That one seems pretty straightforward, let's put
ISSN in there.
And then we want the last name of the author.
Okay.
So, we could be looking through our PubMed XML, and finding the
elements we need, and putting this in.
And let's try it and let's see what happens.
Okay.
So, when we look at the output, yes, we got what we asked for.
We got PMID, the first column.
We got ISSN for second column.
And then we got the set of last names for each record.
So, this is good output.
But it's kind of hard to tell where each column begins and
ends because the columns are all separated by tabs.
But so are the multiple values in the Last Name column
separated by tabs.
So, it makes it a little bit hard to read and deal with.
We can fix this using some special formatting arguments.
So, I have some slides to help you understand this.
Okay.
The formatting arguments that we're going to discuss are tab
and sep.
Tab and sep let you control what extract will use as separators
for your data.
Tab changes what character is used as a separator
after each column.
And sep changes the separator between multiple values
in the same column.
Note that I worded those two definitions a little
differently.
The tab is a separator after each column.
The sep is a separator between values in the same column.
And this distinction is going to be important.
But for now, just remember that the tab separates columns, and
the sep separates multiple values in the same column.
Now, by default both of these separators are a tab character,
which is what we've seen in our example so far.
So, let's look at what happens to our output when we change
the tab and sep arguments.
So, first we're going to start with no tab and no sep.
And this is what we just saw in my Unix terminal.
Here's our output.
And remember that extract creates a new column for each
different element that we specify.
And so, this might help you visualize it a little better.
So, we have actually three columns in our data.
We have a column for PMID, a column for ISSN, and a column
for last name.
But our column for last name has multiple values in it.
And we can see that there's a tab between each of our columns
-- PMID, ISSN, Last Name.
And we can see that there's a tab between each of the values
in our Last Name column.
Now, if we specified our tab and sep, and we used a backslash t,
we're going to get the same thing.
We don't see any changes.
And that's because this is the default.
We've defined both our tab and our sep with this backslash
t in but we've defined it as the default.
So, it doesn't actually change anything.
But what we can do is we can now -- now that we have the right
syntax -- we can change the characters within
these arguments.
So, let's change the sep argument from a tab to
a blank space.
And you can see the output at the bottom of the slide.
So, we can see that the columns are still separated by tabs.
But the values in our Last Name column are now separated by
single spaces instead.
Let's try something else.
Let's leave the sep the same but change our tab argument
to the pipe character.
Now our columns are separated by pipes instead of tabs.
Let's play some more.
We'll switch the sep to comma space.
So, as you can see, using the tab and sep we can customize
our output format.
So, here my columns are now separated by pipes.
But the multiple author last names are separated
by comma space.
Please note that customizing these separators can be
a little tricky.
So, keep a couple things in a mind.
First, when you use a tab or a sep, it only affects the part
of the command that comes after it.
So, in this example, even though we set our tab to a pipe, the
separator after the first column is still the tab because
we didn't change it until after that PMID element.
Second you can overwrite an earlier tab or sep later
on in the line.
So, again, if we look at this example, we start off with a
default tab.
And then we change it to a pipe between the second
and third columns.
And then change it again to a colon between columns three
and four.
And you can think of it as a linear process where we are
engaging different formatting instructions as we go.
Does anybody have questions about tab and sep?
Okay, when you don't have questions I'm going to give you
an exercise.
Here you go.
Okay.
All right.
Don't worry, we'll give you plenty of time for this.
All right.
And we want to refer you to your handout because you'll want to
copy and paste the EFetch command.
What I'd like you to do is write an extract command that has a
new row for each PubMed record.
You know how to do that by now.
You have columns for PMID, journal title abbreviation,
and author-supplied key words.
I'd like you to separate each column with a pipe.
And I'd like you to separate the multiple key words in the last
column with commas.
So, your output looks like the bottom of this slide here.
I'm going to give you about five minutes to do that.
What I'd like you to do, though, is to make sure to put up a
green check when you're done so that we know when we
can move on.
And as always in this class, there's no shame in checking the
answers if you're confused and you need to understand where
we are.
So, go ahead.
Give it a try.
And I think we might have a couple of hints along the way if
you want to check the chat.
All right, go.
Okay, let's go over this together for those of you who
are having trouble.
I'm going to open up my terminal here and clear it.
Okay.
This is always tough because I have to remember what
I asked for.
All right.
So, what we're looking for is the PMID -- let me go back to
here -- PMID, journal title abbreviation, and author
supplied key words.
We want the columns separated by pipe and the data within columns
separated with commas.
Okay.
All right.
So, using that same EFetch example, okay.
Again, one line per PMID -- one row per PMID, I mean.
We're going to extract with that same PubMed article pattern.
Okay.
Now I'm going to start -- usually what I do is start
with the data first and then formatting afterwards.
So, hopefully you all got the right elements.
I'm going to kind of take a risk here because I'm going
to have to move backwards.
But so, the elements that we're asking for from the PubMed data
are the PMID, Medline title abbreviation which you might
have found in the XML as ISO abbreviation -- this is going
to be tricky -- and then key word is the XML for
the author keywords.
It's not going to work.
Okay.
So, that's the data we're asking for.
But now what we need to do is add the separators.
So actually, let me just do -- let me just run this, make sure
that's correct.
All right.
So, I see a PMID, I see what looks like a Medline title
abbreviation, and I see some keywords.
So, that looks like that ran properly.
So, now let me recall that using that magic up arrow.
And now I need to add the separators.
So, to do so I want to go before that element and add tab.
So, between columns I wanted a pipe and sep.
And between data within the same column I wanted a comma.
So, that was the answer.
Hopefully I can prove it to you by running this.
All right.
Think that worked.
Love live demos.
All right.
So, the question -- okay, Brian has the first question.
And that's the difference between keyword lists, key word,
and keyword.
Well, let's look at the XML.
Ah, probably need a different record from this one.
Let me pull up one of the sample records that makes sense to me.
So, I'm going to look at this one.
I'll put it in chat so you'll know which one I'm looking at.
Okay.
And I'm going to look at this in XML.
Okay.
So, I'm looking for the author keywords, that's what
I asked for.
Am I going too far?
Okay.
Here we go.
No, that's not it.
All right.
So, yeah.
Okay.
So, you asked what's the difference between keyword list,
keyword, and keyword.
Okay.
So, here you can see keyword list is an element
that wraps around keywords.
So, this is similar to the ideas that we've been exploring so
far where we've had actually a couple examples where we
have a parent element and a child element.
So, if you defined -- if you're looking for these keywords right
here, you can ask for keyword -- just keyword -- as long as there
aren't other keywords somewhere in the XML.
But if there were other keyword elements in the XML, you might
need to specify the parent to be sure that you get
the right list of keywords.
So, there is no difference.
It's just a matter of specifying a parent element if you need to.
So, yeah, you'll have to look at the XML and figure
out whether you need to specify a parent element.
Or sometimes you discover it [laughs] the hard way by trying
to extract something and you end up with the wrong data.
So, that's why you always have to use a sample set.
Be familiar with the records in your sample set so that you
know, when you test your code, you know that you're
getting the right data from the right place.
And then, you know, once you've tested it
and you've checked, that's when you pull the data
from whatever records you need.
Hope that makes sense.
Okay.
And then Nathan had a question.
Would also work -- I think you mean add the sep later if the
previous columns don't have multiple values.
Okay.
So, yeah.
Okay, I think I see what you're asking.
Does the -- can you put in your formatting arguments in
different places of your code depending on where you want
the formatting.
And that's kind of what I was alluding to earlier is it's
read, kind of, left to right.
So, whenever you want to change your formatting you can enter
your formatting arguments.
But test it and make sure that it works.
So, let me see.
I mean, we could test the code that you've written here and see
if it works.
If it is what you want it to.
Mike Davidson: Do that.
Kate Majewski: Yeah.
Let's see.
I'm not sure if this is going to copy and paste very
well from chat.
But I'm going to try it.
Actually, let me clear.
Let's try this here.
See what happens.
Have some fun.
Mike Davidson: You're going to be adventurous.
Kate Majewski: Oh.
Yeah.
Guess I'm going to need EFetch [laughs].
All right.
Let me try this.
Hang on.
Using a handy control C to cancel out of that.
All right.
Okay.
So, we'll use the sample set.
And let's see if I can do this.
All right.
So, we're extracting a new row for each record.
We're adding a tab, which is a pipe between columns.
And then we're putting a PMID, the ISO abbreviation,
and then we're adding a -- with a comma space
before the keyword.
Let's try it.
Okay.
Well, that didn't work.
Mike Davidson: So, you just need the extra element keyword.
Kate Majewski: Okay.
So, yeah.
We could fix this.
Do it like this.
Does that look good?
Mike Davidson: [affirmative]
Kate Majewski: Let's try it.
Yay.
It worked.
So, yeah.
You could do it, but you have to re-add that element
to make it work.
All right.
Now we're having fun.
Okay.
So, let's put what we've done so far aside for a moment,
and let's move on.
I want to talk about a little bit more complicated
formatting here.
All right.
Let's say we want to pull all of our authors for each citation
in our results set.
And we want one row per record.
And we want to include the PMID.
And here, I'm going to -- this time -- let me clear again.
This time we're going to make things interesting.
We're going to put the initials in and see what happens.
Okay.
All right.
So, extract hold all the data from the last name element, and
then it pulled all the data from the initials element.
And if you look in my code, that's actually what -- exactly
what I asked it to do.
but it's probably not what I wanted.
We want to retain the relationship between an
individual's last name and the corresponding initials.
So, let me show you what's happening here.
All right.
So, take a look at this slide.
On the left we have some dummy XML, and on the right is going
to be the output of our command.
At the bottom is our code.
All right.
Mike Davidson: It's lagging.
Kate Majewski: Oh.
We have a lag in my display?
All right.
I'm going to just stop for a quick second because
I want to make sure that you guys can follow this.
It's important.
Okay.
All right.
So, let's take a look at the bottom.
This is the code.
It's what we're asking for.
So, here's how -- here's how it works.
First, extract finds the first instance in the XML of our
pattern element.
In this case that's PubMed article -- because remember
we wanted one row per PubMed citation.
Then extract looks for all instances of the element or
elements identified in the element argument
and outputs them.
So, first it's going to look for Medline PMID,
which there should be one per record.
Okay?
So, it finished.
It found that.
It's done.
Now it looks for the next element.
It's going to look for last name.
So, it finds one and it outputs that.
But then, rather than giving us the initials, it's going to keep
looking for last name.
It gives us the next last name.
And keeps going until there are no more last name elements, and
only then does it go to the following element, initials,
and then it outputs all of the initials.
It finds the first initial and outputs that.
Second initial, outputs that, et cetera.
Because all extract is doing is it's scanning through the XML,
it's finding the string that identifies the element we're
looking and outputting the contents of that element.
So here, that's what it's doing.
Sorry -- oops.
To fix this, we can use an argument called block.
Block is one of a series of extract arguments -- they call
them exploration arguments -- which help us identify and group
the elements that we want to output.
And this is -- what we want to do, we want to group each
related pair of last name and initials.
So, go ahead to PubMed right now.
You should have a record open.
We were just looking at one.
Look in any record in XML and look at where last
name and initials live.
So, do you see the element that wraps, or blocks together, the
last name and initials for each individual author.
Actually, let me clear your status.
Give me a green check when you see that.
Sorry.
It might take you a moment to get over to PubMed and look.
But what I'm -- what I'm asking you to look for is look for the
last name element and the initials element, and then
identify what element wraps those together for any
individual author.
Okay.
So, once you've found the element, that's going
to establish that relationship between those
two different elements.
It's going to wrap around them.
Then you know what your block is going to be.
Okay.
So, if you've found that -- and I can look back at my slide --
take a look at the code here.
So, the first part's familiar.
We've done this lots of times.
Pattern PubMed article, element, and we got the PubMed.
But this next part, the second line here, is what I want you to
pay attention to.
We're going to block by the author element, and then we're
going to ask for it to spit out the last name and the initials
for each block before it goes on to the next block.
Okay.
So, let's try this.
Okay.
So, here's the same code.
Blocking by author.
Last name.
Initials.
All right.
I'm going to clear your status.
Okay.
All right.
I'm going to go back to my slide because I want
to show you what it's doing.
All right.
So, if you're looking elsewhere, please look at my slides
right now [laughs].
I want to show you how block works.
Okay.
So, just like before, extract finds the first instance
in the XML of our pattern PubMed article.
Then extract looks for all instances of the element or
elements identified in the element argument
and outputs that.
So, this time -- okay.
That's just Medline citation, PMID.
Then extract is going to find the first instance of the
element identified in the block argument.
So, it's going to author.
Now, extract is going to look within that first author element
and output the element or elements identified in the
element argument following.
So, this should be one last name and initials.
Okay.
So, then it's done with that block.
Now extract is going to look for the next instance
of the block element, the second author element,
and output the elements contained within.
Last name.
Initials.
And then it's going to go to the next block and output
last name, initials.
And it's going to keep doing that block by block
until there are no more elements left in the pattern.
Oop.
I keep clicking one more -- one too many times.
Okay.
So, once it's reached the end of a line, it goes back to the
beginning, finds the next instance of the element
identified in the pattern.
Okay.
I'm going to pause for a moment, make sure everybody
is with me, more or less.
Could you give me a green check if you understand
at least conceptually what block is doing?
Even if you're not exactly sure if you can apply it yet.
If you get the gist of what block does,
give me a green check.
Great.
Most of you.
Not everybody --
There are a couple of questions.
Some of the questions that are coming in, we may have to put
off until a later class because they get into a little bit more
complexity than we plan to today [laughs].
Yeah.
Okay.
So, let me -- let me ask you, those --
Amy and Sarah [phonetic sp].
We're going to have to hold off because we're going to -- yeah,
we're going to get to this.
But I don't want to totally confuse those of us who are just
trying to understand what block is doing.
Okay.
All right.
Okay.
So, this is great.
Block is wonderful, a great tool.
So, now we've pulled the last name and initials next to each
other by blocking them, using their parent element author.
And we've got all the authors for each record
on a single line.
We've got the authors last name and initials in the order that
we want.
Last name, initials.
But everything is separated by tabs.
So, you know, wouldn't it be nice if we could group the last
name and initials together in our output to reflect
their relationship?
And we can do that, too.
Okay.
So, all right.
So, from the beginning of the class, we've talked about using
pattern to create rows, and element to create columns.
And so far, each of our columns have had data from a single
element or attribute in them.
And if you look at the three columns of this output table,
you can see that happening.
The first column has data from the PMID element, the second
column has data from the ISSN element, and the third column
has multiple values.
But each of those values is from different occurrences
of the same element, the last name element.
But with what we now know about TAB and SEP, we can actually
specify different characters to separate between columns versus
between multiple values in the same column.
But with this author-last name-initials example, what we
actually want to do is output values from two different
elements into the same column.
And once we do that, we can use our SEP character
to separate them.
Okay.
We can do that with a comma -- oops.
I think I skipped over my last slide a little bit.
Okay.
So, we can do this with a comma.
So, instead of separating the multiple elements with a space,
we use a comma, and we can put them both in the same column.
So, we're actually telling extract not to consider these
separate columns.
And because the data is not considered in separate columns,
extract will use the SEP character to separate he last
name and the initials.
Okay.
So, we do this, and we combine it with block.
We still have our column separated by tabs, but our last
name and our initials are grouped together in the same
column with a space separating the two values.
And this makes the author information much easier to read.
Okay.
A little aside that might reduce some confusion as your building
your extract command.
Whenever you have a block argument in your command, it's
actually going to reset your TAB and SEP to the default.
So, if you use block, you want use the TAB and SEP within
the block.
So, you remember that the block is going to reset
your TAB and SEP.
So, looking at this example, we define our TAB as a pipe.
And so, our first two columns are separated by the pipe.
But once we start creating new columns for each of these author
blocks, our TAB is actually going to reset to it's default.
And this is because that block argument, like I said,
has reset.
Our SEP still works because we've defined
it after our block.
So, if we want all of our columns separated by pipes, we
need to go in and add another TAB argument after our
block argument.
All right.
So, I realize that's a lot.
But we're going to try this out.
I'm going to give you another exercise.
Actually, I'm going to clear your status.
All right.
And here's the second exercise.
And this exercise combines the TAB, the SEP, the block,
and the comma, which is kind of a lot, but it's hard
to do this without combining those.
So, what I'd like you to do is to write an extract command that
has a new row for each PubMed record, has a column for PMID;
okay, that part's easy.
You know how to do that.
Then, what I'd like you to do is list all
of the MeSH headings, separated by a pipe.
And if a heading has sub-headings, separate the
heading and each of the sub-headings with
a forward slash.
Okay.
Don't panic.
I want to suggest your process on this one because I realize
that this is a lot.
So, take a deep breath, and do it this way [laughs].
Start by puling up at least one of these records
in PubMed in XML.
And then identify the elements you want to extract.
So, start there.
What's the data you want?
Okay.
That shouldn't be too tough.
You just want to look through the XML and find the elements
you want to extract.
Then, think about what elements have parent elements
that you want to group using block.
And after that, consider what elements you might like
combined into the same column using a comma.
All right.
Finally, after all that, then consider what characters you'd
like to use to separate the columns using TAB.
And then, the data within a column using SEP.
And that might require a little bit of experimentation.
Okay.
So, give me a green check when you've done it, whether you've
done it yourself or copied it from the handout, which is
totally fine.
And we'll give you some hints along the way.
So, I'm going to give you a good amount of time to work on this.
But again, try to do it step-by-step, like I suggested,
so you don't get overwhelmed by the process.
Okay.
Go.
Okay.
I know not everyone finished that.
And that's okay.
Let's look at this together.
And actually, for this -- to review this, I'm going to use
Notepad so I can make it nice and big, and play with
the code a little bit, and explain what we're doing here.
Okay.
So again, the challenge was -- actually, let me just show
the slide again.
The challenge was to write an extract command that has a new
row for each PubMed record, has a column for PMID, and then
it lists all the MeSH headings, separated by a pipe, and has
the sub-headings attached, separated with a forward slash.
Okay.
So, I asked you to kind of do this in steps.
So, we're going to use that same old EFetch, using those
three records, so that we know when we look at our data
if we got the right stuff.
Then the next part should be familiar to you.
We want one row per PubMed record, so we're using
that same pattern of PubMed article.
Okay.
Then the question is what elements do we want?
Well, I asked for PMID.
So again, that should be familiar from our
previous examples.
The Medline citations, slash, PMID.
Okay.
And then I asked for the MeSH headings and the sub-headings.
So, if you looked through the PubMed XML, you probably
found descriptor name and qualifier name.
Okay.
So, that's the data we want.
But, like with that author example with the last name and
initials, what this is doing is asking for one column with PMID,
one column with descriptor name, and one column
with qualifier name.
And it doesn't establish any relationship between the
descriptor and the qualifier.
So, the next part requires us to understand that descriptors and
qualifiers are different element, but they are wrapped
in a parent element called MeSH heading.
So, similar to the way that last name and initials are wrapped in
an author element, these are wrapped in a MeSH
heading element.
So, what we want to do is use that block argument to keep that
relationship between each descriptor and its qualifiers.
So, what we want to do is insert the block --
so, let me just do this.
Here.
Oops.
Even Notepad wasn't the best choice here [laughs].
All right.
There we go.
So, we'll add a block per MeSH heading, and then we want the
descriptor name and the qualifier name.
So, now the qualifiers will be associated with the right
MeSH descriptors.
And this is probably the most important part of this.
The next part is that we want to display the qualifiers with
their associated descriptors in the same column.
Does anyone remember how to output data from different
elements into the same column?
Go ahead and put it in chat.
How do you output different elements, data from different
elements, into the same column?
Anybody?
Yay.
Okay.
Nina sort of got it.
Comma.
Yes.
So, you would add a comma to output the qualifiers into the
same column as the descriptor.
Okay.
So, that's going to look pretty good.
But we also asked you to clean it all up with TAB and SEP.
We want a pipe between columns and a slash within the column
for descriptors and qualifiers.
So, we can start off here, right after our pattern.
Before the element we can add -- oops -- a pipe between
our columns there.
But then remember that block is going to reset the TAB and SEP.
So, after the block and before -- oops.
I forgot the element.
Why didn't you say something [laughs]?
Kate Majewski: Sorry.
Before the element but after the block we're going to specify our
TAB as pipe, and our SEP -- oops -- SEP as forward slash.
So, unless I introduced some bad characters there that should be
our answer, I'm just going to take the risk.
I don't know.
We'll see what -- we'll see if this works.
All right.
If I hadn't added that element, I would learn right now [laughs]
that that wasn't what I wanted.
But let's see.
Okay.
Let's see if we got what we asked for.
We have a PMID.
We have MeSH headings separated with pipes -- all the columns
separated with pipes -- and then we have sub-headings
attached to the MeSH headings with a forward slash.
I think we did it right.
All right.
What do you think?
So, okay.
I just want to say if you've figured out the block on that
one, you did great.
If you figured out the block and the comma, I think you're doing
really great.
If you figured out the block, the comma, and the TABs and the
SEPs in the time that I gave you, honestly, I bow to you.
You did far better than I could the first time I looked
at this exercise.
It is complicated.
If you didn't figure out any of that on your own, but you looked
at the answer, and you sort of, kind of get the idea of what
we're doing, that's about what I'd hope for at this point.
I think that this takes a little time to absorb it.
But if you -- if you follow -- if you fall into any of the
categories I just mentioned, either you figured it out or you
looked it up.
You know, all the way to looking it up and kind of getting the
gist of things, can you give me a green check so I can identify
anybody who just might be totally lost [laughs]?
And while you're doing that just, you know, if you're
still having trouble, take a breath.
Honestly, make peace with the idea that you're going to need
to review this and practice it.
I needed to sleep on it at least a couple of times and let my
brain work through it before I could really kind of get the
gist of this.
So, you'll be okay.
All right.
Yeah.
So, I think there are quite a few in the class who might need
to sleep on this.
But this is why we have the practice exercises and why we
break this over many days.
Okay.
All right.
So, at this point in the class, I'm actually going to change
gears and talk about something completely different -- well,
not completely different, but very different.
So, okay.
We'll all take a deep breath.
And this part's not hard, so stick with me.
Okay.
I'm going to talk about Unix.
Unix is fun.
All right.
I'm going to clear your status.
All right.
Looking for -- I'm looking at the chat.
Okay.
All right.
Okay.
So, looking at Unix.
I'm going to talk about just a few Unix tips that will be
useful in using EDirect.
We don't expect anyone to be Unix experts at the end of this,
but we want you to have the tools that you need to use this
-- to use EDirect properly.
So, you may notice, depending on the size of your terminal and
your screen, that we have a lot of line wrapping here.
I think someone just mentioned that about the line wrapping.
It gets kind of ugly.
There's only so much data manipulation that we want to do
here in EDirect.
And so one of our goals here is to be able to extract our data
neatly, and cleanly, and well-organized so that we can
import it into Excel, or a text editor, or some other familiar
environment where we can play with it, manipulate it, do other
things with it.
And as you might have discovered by now, copying and pasting is
not really all that practical, especially if we have things
that go off the screen.
So, fortunately, saving your output from here to a file is
actually really simple.
So, let me show you how.
I'm going to clear my screen.
Okay, so let's say I do an Efetch, and I want to output
to a file.
What I would do -- let's see.
Just going to stick an Efetch in here.
Okay, and I'm going to just ask for the UIDs.
So, the PMIDs.
To put these in a file, I use a greater-than sign, and then
I give it a file name, whatever file name I want.
Let's call it -- oops.
I have to type properly, testfile.txt.
Okay.
So that's really all there is to it.
I don't see anything happening, but I didn't get an error
message, so that's good.
So, how do I know that it worked?
Well, here's another Unix command.
It's LS for list.
So, LS lists all the files in your current directory.
So voila, here I have created a file called "testfile.txt," and
when I list the files on my directory, there it is.
Okay, there it is.
Well, where is that?
This is a little bit strange, because I'm looking at files,
but I don't actually know where they are.
So, in -- if this class is the first time that you've used the
Unix terminal and your machine, you may be wondering, you know,
"Where is this mystical, magical folder that we're reviewing?"
So, let me give you a couple of tips on how to find
it outside of Unix.
One command you can use is PW -- oops.
Got to be in the right place.
PWD for print working directory.
And that just asks Unix, "Hey, where am I?"
So, PWD, go ahead and do that, tells you what directory you're
currently working in.
Okay, so, I can see that I'm in Home and then a folder called
something like my username, Majewski.
So, depending on how your software is set up, this folder
is mostly likely on your hard drive somewhere.
Now, if you're using Cygwin like I am, we have
an extra easy cheat.
There is actually a command that you can use that will tell you
where this is on your computer.
So, it's this.
It's cygpath, C-Y-G-P-A-T-H, space, minus W, space, tilde.
So, this command tells me where in Windows my Cygwin
directory lives.
And so, if I hit enter, it tells me that it's on C,
it's a folder called Cygwin 64 in a folder called Home
and a folder called Majewski.
So -- and if I go to my folders in Windows, here they are.
I can find that because it told me where to look.
Okay, C, Cygwin 64, there it is.
Cygwin 64 and then Home, here's Home.
And then that's me.
Here it is.
So, there's some other stuff here I don't care about,
but I do recognize the folders that I -- or the files
that I saw listed when I did that LS command.
So, you can see it's still up on my screen.
When I do this LS in Cygwin, it's telling me the files
that are here actually sitting on my drive on C.
So, let's look at the file that I just created
a moment ago using this, testfile.txt.
Let's see.
Open this file, look, three PMIDs.
That's what I did.
Okay.
There it is.
Okay.
All right.
So, you can save in whatever file format you want.
I could do an Esearch then use an Efetch to get a list
of PMIDs, and I could use the greater-than sign
to redirect the output to a text file.
I could use Efetch to download the full XML
and save the output to an XML file.
And actually, let me just show you what that looks like.
Okay, so I'm fetching these three records in XML format
using that greater-than sign and putting that into
a file called "testfile.xml."
And I could look at these in Unix if I wanted.
I can also look at them in my home environment,
like I just showed you.
Okay, so, to recap that.
For Cygwin users, your working directory is probably a
subfolder of the folder where you installed Cygwin, and you
can use this cygpath minus W tilde.
If you're using a Mac, your working directory is probably
in the Users folder, like this Users slash and then
your user name.
If you still can't find your files, another trick is to use a
distinctive file name so that you can search your computer for
it using your operating system's native search feature.
And actually, I'd like all of us to do this right now.
So, go ahead to your terminal.
I'm going to clear my terminal screen again.
What I'd like you to do is a quick Efetch, and I want you
to create a file with a special name.
So, here's the command.
So Efetch these four IDs.
I'm just going to take the UIDs and put them in a file
called "specialname.csv."
So the point -- what I'm doing here is I'm just
creating a file with a funny name so that
I can use my home operating system to search for it,
and I can find where this folder is.
And again, we can use LS to kind of prove to ourselves
that it's there.
So, here it is, testfile.xml.
Again, please do that, because we're going to use
that file in a minute.
Okay.
So, I've already found mine, but if I wanted to, I could've used
my windows down here to search for that specialname.csv file.
You would have to use whatever your operating system
is to find that.
Okay.
Can you give me a green check if you've found your files?
Okay.
So, if you haven't found your files, frankly, we will
probably refer you to your IT staff for help.
If you've got an IT staff, you might need to call them
in for this.
But if you can, if you're still having trouble finding your
files, and you know what kind of system you're using, go ahead
and put it in chat so that we at least have an idea of where the
problems are so that we can maybe try to figure out some
other guidance to provide.
So again, if you haven't given me a green check to tell me that
you have found your files, then please put in chat what kind
of system you have so that we can at least have an
idea where the problem is.
All right.
I see a little chat going on.
Oh, there was -- Pat asks, "What happens
when you save to a filename that's already
in there?"
If you save to an existing filename, it's going to
overwrite that file.
So, as Mike says, be careful.
All right.
Okay, so, I guess we can jump to the exercise, since it relates
directly to this.
What I'd like you to do is get the XML of all articles about
the relationship of Zika virus to microcephaly in Brazil.
I want you to save your results to a file.
And that might take a little bit, because this is a
big search.
Okay?
I think I'm going to just go over the answer.
Don't worry if you hadn't finished.
You can always take a look at these exercises.
That's why we give you all the information on the website.
Okay, so, the question was, "How do you --"
I get the full XML of all articles about the
relationship of Zika virus to microcephaly in Brazil.
We want you to save that to a file.
And -- so this one requires you to remember all
the way back to Esearch.
We haven't done that in a while.
And here it is.
So, we're searching PubMed.
I put in a simple query.
You might -- your query might've varied from mine.
It doesn't matter that much for the purposes
of our class today.
If you have questions about your query, you might want
to attend a PubMed for Librarians class or another
one of our PubMed training.
But you can -- I just used a simple query here.
And here's the part that's new.
We fetched in a format XML, and we used a greater-than sign,
and saved it to a file called zika.xml.
And you could've called your file whatever you want.
And like I said, it does take a little while, because
this is a lot of records.
But that's how you would do it.
And to show that, I can use my LS command, remember list
to list the files in my directory, and there's
my new file right here, zika.xml.
Okay.
And so, you're probably wondering,
"How can I look at this?
What am I doing with it?"
That's what I want to cover next.
But if you're still having trouble with finding your file
or with that last exercise, you know, we can try
to answer some questions at the end if we have time.
But we're going to move on, because I want to cover
just a couple more things.
So, please stop working on that, and let's look at the
next thing that I wanted to cover, and that is cat.
Okay, so, Unix has a lot of tools designed just for
manipulating files.
And we're not going to really get into them much, but here's
an exception, and that's cat.
So, cat is short for concatenate.
It's -- it is used for concatenation, but it's also
used to open files and display them on the screen.
So, it's kind of a handy, multi-purpose tool, cat.
So, let's take a look at using cat both for displaying files
and for concatenating them or combining them.
All right.
So, here I have just done a list of my current directory,
so I can see I have a bunch of files here.
If I wanted to take a look at them, I can do so using cat.
So, actually, for this next -- actually, let me just do this.
All right.
So, let's say I wanted to look at the file "test file."
I can do that.
I can type "cat testfile.txt." All right, so there -- those
three PMIDs that I put in there earlier.
I can look at another file.
Let's say I wanted to look at file1.txt.
So, it just displays the text of the file right here
on the screen.
I can see what I'm working with.
So, maybe I can look at my file2.txt.
Just an easy way to view the file.
But I also mentioned that you can use cat for concatenation.
In other words, putting files together.
And to do that, what you do is use cat and then put your first
filename, and then your second filename, and then use that
greater-than sign again to create a new file and give it
another filename.
So here, what I'm asking it to do is to take file1
and take file2 and put them together and make file3.
So now, if I list my files, you can see that I have a new file
called "file3." And if I wanted to look at it, again, I can use
that handy, multi-purpose cat command to take a look at it.
Oops.
Where did it -- oh.
Sorry.
Typo.
There you go.
Okay, so that's fun, but how does that help us?
And okay, for this next demo, I'm actually going to ask you
not to follow along with me.
I just want you to watch for a moment, because I'm going
to use a file that you don't have.
Okay.
So, let's say that you have a long PubMed search string that
you want to use in Esearch.
So, I have one here.
I've pulled it into my directory.
So, I'll just use cat and show you what it looks like.
But you don't have this file, so you won't be able to do this.
So, cat, and I have a file called "searchstring.txt."
Okay.
So, here's a great big, long search string.
It's an effort to capture all intramural rEsearch coming out
of one of the other national institutes of health, the
National Institute of Allergy and Infectious Diseases, NIAID.
So, NIAID has lots of different names, and labs,
and subcomponents, and there are over 100 rEsearchers
who work there.
So, I want to keep this string, but I don't want to be making
changes to a search strategy typing it into Esearch.
I don't want this to be part of my command, I want to
be able to edit it and then pull it into a search.
So, what you can do is start your Esearch normally.
Okay, so, I'm using PubMed.
And now, because I have this file available to me, I can use
query and then, in parentheses, a dollar sign,
an open parens, cat, and then my filename.
And then close parentheses and close quotes.
So, this dollar sign parentheses gives us
the value of what's inside the parentheses.
So, the value of cat searchstring.txt is the output
of reading the searchstring.txt to the screen.
So, I'm pulling this into my search.
Okay.
So, you can do something like this in Efetch, as well.
Let's say that you had a list of PMIDs in a file,
and you wanted to fetch them all.
You could do so.
And this is really useful if you want to maintain
a list of citations outside of PubMed that you want
to retrieve instead of just retrieving search results.
So, if the list was a long string of PMIDs separated by
commas, you could do the exact same thing, because that
follows the syntax that you can use in Unix in EDirect.
But let's say you have a file like a CSV file with
one PMID on each line.
If you do it this way, you're going to get an error.
So, for this, we actually want to use another
EDirect command that we haven't talked about yet.
And for that, I need to delve a little bit deeper into something
that we've already talked about a lot, which is the pipe.
Okay.
So, when we introduced the pipe early in the class,
we were talking about Esearch and Efetch.
And we said we were just piping the results
of one command and using them as the basis for another.
And that's a useful way to think about it, but remember,
when we do an Esearch, we're not actually getting
a list of PMIDs.
We get an XML snippet with a number of PMIDs that meet our
search criteria plus a web environment and a query key.
And what is happening is that the results of the Esearch,
including the full list of PMIDs, is being stored to the
history server, and the history server is what keeps track
of your previous searches.
And it's kind of like what you see on the advanced
search page.
The web environment tracks your session to identify your
specific history and not some other user's history, and the
query key indicates which query in your history this is.
So, what the pipe is actually doing is piping the web
environment and query key from Esearch to Efetch.
The Esearch is saving the results on the history server,
including both the PMIDs and the fact that they come from PubMed,
and the web environment and query key tell Efetch
which results to pull down from the history server.
Now, again, most of the time, you can just think of the pipe
as piping PMIDs from one command to another.
But the fact that this actually works this way allows you some
powerful flexibility, because Esearch isn't the only
thing that can put PMIDs on the history server.
And I realize what time it is, and I'm probably going
to go over a minute or two.
Can you give me a red X if you have to go?
Yeah, okay.
All right.
So, I may have to finish this up next class.
All right, so let me just finish what I'm saying right
now, and then we will say "to be continued."
Okay.
All right.
So, the EDirect command epost can be used to upload
a series of PMIDs to the history server.
So, piping the results of epost to Efetch works exactly
the same way as Esearch.
All right.
So, I think we're going to do a demo of epost.
All right, I'm going to check with Mike to see how
he wants to handle that.
We will get -- we will get to that somehow.
All right.
Have only like two minutes more material, but I know
some folks have to go.
So -- all right.
So, the command for this, you can find in your handout
so you can try it out.
Try out using epost, and there's even a shortcut
that you can find in your handout for today.
So, maybe we'll do a quick review at the beginning
of next class.
All right.
So, I know folks have to go.
So, coming next time, we're going to continue our discussion
of extract and talk about some more advanced arguments
that help us limit our output to the exact data we need.
And we're also going to cover some ways to integrate
all of this into some real-life practical solutions
to problems.
So, in the meantime, please do your homework
and practice, and don't forget to send us a use case.
Something that you want to do with EDirect so that
we can try to work through it with you.
As always, go to our website for more
information, documentation.
Please, please, please, if you were boggled a little bit
by the block today, or combining the block with a comma,
please do the exercises and make sure that you've
got that before next time.
All right.
So, I'll say goodbye to the folks that have to go,
and I'll hang around for folks who want to ask questions.