WELCOME BACK TO EDIRECT FOR PUB MED.
TODAY IS PART 2 EXTRACTING DATA FROM XML.
I AM SARAH HELSON, A LIBRARIAN HERE AT THE
NATIONAL LIBRARY OF MEDICINE.
TODAY I'M HERE WITH KATE MAJEWSKI AND MIKE DAVIDSON,
WHO ARE GOING TO BE ANSWERING QUESTIONS IN THE
CHAT BOX FOR US TODAY.
SO THIS IS THE SECOND OF FIVE SESSIONS IN THE
EDIRECT FOR PUB MED SERIES.
SO WE TALKED LAST TIME ABOUT USING EDIRECT
COMMANDS TO SEARCH FOR AND RETRIEVE PUB MED
RECORDS IN A VARIETY OF FORMATS, INCLUDING XML.
TODAY WE'RE GOING TO START TALKING ABOUT THE
EXTRACT COMMAND, WHICH LETS YOU EXTRACT SPECIFIC
DATA ELEMENTS FROM THE PUB MED XML AND ARRANGE
THEM IF A CUSTOM TABULAR FORMAT.
SO EXTRACT IS A PRETTY BIG TOPIC AND WE'RE GOING
TO SPEND THE NEXT THREE CLASSES DISCUSSING
DIFFERENT ASPECTS OF IT BEFORE WE FINISH UP WITH
A BROADER DISCUSSION OF EDIRECT.
WE'RE GOING TO START TODAY WITH A QUICK RECAP
OF LAST SESSION AND ANSWER ANY QUESTIONS THAT
YOU HAVE ABOUT THE HOMEWORK.
SO IF YOU HAVE ANY QUESTIONS ABOUT THOSE,
TYPE THEM IN THE CHAT BOX NOW.
WE'RE THEN GOING TO DO A BRIEF REFRESHER COURSE ON
THE BASICS OF XML.
SO WE CAN ACTUALLY FIGURE OUT WHAT DATA WE WANT TO
EXTRACT FROM IT.
WE'RE GOING TO TALK ABOUT CREATING BASIC TABLES
WITH EXTRACT, AND THEN WE'RE GOING TO FINISH UP
WITH A DISCUSSION OF THE FEW COMMANDS THAT CAN
HELP TO YOU SORT OF POST-PROCESS AND CLEAN UP
YOUR OUTPUT.
SO JUST A FORMATTING NOTE FOR TODAY.
THE FIRST HALF OF TODAY'S SESSION GOING IS GOING TO
BE VERY CONTENT-HEAVY AND THEN WE'RE GOING TO GET
TO A LOT MORE OF THE HANDS-ON PART OF IT TODAY
LATER IN THE CLASS.
SO DON'T WORRY.
WE WILL GET THERE.
I PROMISE.
IT WILL NOT BE POWER POINT THE ENTIRE DAY.
SO LAST TIME WE TALKED ABOUT ESEARCH, WHERE WE
SAID THAT ESEARCH WILL SEARCH A DATABASE.
WHEN WE'RE LOOKING AT PUB MED, THAT MEANS WE ARE
SEARCHING FOR PM IDS.
THIS RETRIEVES RECORDS FROM THE DATABASE IN A
VARIETY OF FORMATS AND TODAY WE'RE GOING TO BE
LOOKING AT THE FORMAT XML A LOT.
AND THEN THAT PIPE CHARACTER WILL PIPE YOUR
RESULTS FROM ONE COMMAND TO THE NEXT.
SO SOME TIPS FOR THE USERS OUT THERE.
JUST REMEMBER THAT CONTROL C ISN'T GOING TO
COPY.
WITH PASTE, WE RECOMMEND SHIFT SO THESE
ARE ADJUSTABLE IN THE CYGWIN OPTIONS AS WELL
AND YOU CAN COPY AND PASTE THAT WAY.
SO SOME TIPS FOR ALL THE USERS OUT THERE.
SO CONTROL C IS GOING TO CANCEL.
IT'S A GREAT WAY TO GET OUT OF A MISTAKE QUICKLY.
IF YOU'VE GOT A LONG RUNNING SCRIPT, THIS IS A
GREAT WAY TO GET OUT OF IT.
THE UP-AND-DOWN ARROWS WILL ALLOW YOU TO CYCLE
THROUGH YOUR HISTORY.
SO IF YOU REALIZED THAT LAST COMMAND WAS GREAT
BUT NOW I WANT TO ADD SOMETHING ON TO IT, THIS
IS A GREAT WAY TO GET THAT COMMAND AND THEN AND
EDIT IT FROM THERE RATHER THAN RETYPING THE ENTIRE
THING.
THEN THE CLEAR COMMAND WILL CLEAR YOUR SCREEN
BUT IT DOESN'T CLEAR YOUR HISTORY SO YOU CAN STILL
USE THOSE UP-AND-DOWN ARROWS.
SO ARE THERE ANY QUESTIONS FROM THE LAST
CLASS OR HOMEWORK?
I AM GETTING NODS SAYING YES, THERE ARE.
LET'S TAKE A LOOK.
WE'VE GOT ONE QUESTION ASKING OR SAYING THAT
THEY DO NOT USE THE CRTD TYPE BUT USE THE PDAT
TYPE IN SETTING UP THE SAME RESULTS.
SO THE CRDT IS THE CREATE DATE AND THAT'S DATE THAT
THE ARTICLE'S CREATED IN PUB MED.
AND THERE ARE ACTUALLY -- MIKE IS LETTING ME KNOW
THAT THERE IS A SLIGHT DIFFERENCE.
CREATE DATE IS GOING TO GET YOU 369 RESULTS,
WHEREAS P DOT'S GOING TO GET YOU 96 RESULTS.
VERY, VERY EASY TO CONFUSE, AS A MISSPOKE
WHEN I SAID IT.
SO THERE IS A LITTLE BIT OF A DIFFERENCE THERE AND
SO THAT'S THE P DOT IS GOING TO SEARCH FOR THE
PUBLICATION DATE.
SO WHILE OFTENTIMES THESE ARE GOING TO BE SIMILAR,
THEY AREN'T QUITE THE SAME.
DATES IN PUB MED CAN GET A LITTLE TRICKY.
AND THEN, THE IS IT NECESSARY OR NOT TO ADD
BULLION OPERATORS BETWEEN TERMS?
SO THAT IS SIMPLY UP TO YOU AND THE SEARCH THAT
YOU ARE DOING.
SO IN THE EXAMPLE THAT IS PUT IN THE CHAT, DID USE
THE OPERATORS?
I BELIEVE IN OUR SAMPLE CODE PAGE, WE DON'T USE
BULLION OPERATORS.
IT'S SIMPLY UP TO YOU AS TO HOW YOU WANT TO
CONSTRUCT THAT SEARCH ITSELF.
THEN LET'S SEE.
SO WE'RE ALSO GETTING A QUESTION ABOUT HOW LONG
IT TAKES BEFORE ARTICLES ARE ADDED INTO PUB MED?
THAT'S ACTUALLY KIND OF OUT OF SCOPE FOR THIS
CLASS, SO I AM GOING TO RECOMMEND THAT YOU GO
CHECK OUT THE PUB MED HELP AND THAT SHOULD HELP
TO ANSWER YOUR QUESTION.
ANYTHING ELSE?
ALL RIGHT, SO I AM NOT SEEING ANYTHING.
DO YOU HAVE QUESTIONS, PLEASE CONTINUE TO PUT
THEM INTO THE CHAT BOX.
WE'VE GOT, LIKE I SAID, MIKE AND KATE ARE HERE
HELPING ANSWERING QUESTIONS AS WELL.
I AM NOT SEEING ANYTHING ELSE, SO WE WILL GET
STARTED.
SO ANOTHER THING THAT WE TALKED ABOUT LAST CLASS
WAS THE THEME FOR THIS COURSE "GETTING EXACTLY
THE PUB MED DATA YOU NEED IN THE EXACT FORMAT THAT
YOU NEED IT IN."
SO FAR WE'VE TALKED ABOUT HOW TO
GET DATA BY PUB MED BY USING ESEARCH TO SEARCH
FOR RECORDS AND ESEARCH TO RETRIEVE THEM.
WE KEEP PROMISING THE POINT OF EDIRECT IS TO BE
ABLE TO CUSTOMIZE YOUR OUTPUT AND SELECT OUT
ONLY THE DATA ELEMENTS THAT YOU NEED FROM THE
XML.
SO IN ORDER TO DO THAT, WE NEED TO MAKE SURE JUST
AS FAR AS THE BASIC STRUCTURE AND SYNTAX OF
XML GO.
HOPEFULLY THIS IS GOING TO BE OLD FOR MOST OF
YOU.
BUT WE WANT TO REFRESH YOUR MEMORY AND LIKE I
SAID GET EVERYONE ON THE SAME PAGE.
SO XML OR EXTENSIBLE MARKUP LANGUAGE IS USED
FOR STORING AND TRANSPORTING DATA.
AND SO XML IS DESIGNED AND STRUCTURED TO BE READ
AND USED BY COMPUTERS BUT FORTUNATELY FOR US, IT
CAN ALSO BE READ AND UNDERSTOOD BY HUMANS.
XML DOCUMENTS ARE ALSO COMPOSED OF XML ELEMENTS
AND WE'LL BE TALKING A LOT ABOUT THE ELEMENTS
TODAY.
SO TO JUMP RIGHT INTO THAT, THIS IS AN XML
ELEMENT.
IT'S GOT AN OPEN TAG AND A CLOSED TAG AND IT'S GOT
THE NAME OF THE ELEMENT IN IT.
SO IN THIS CASE "YEAR." AND THIS ALSO CONTAINS
SOME DATA.
IN THIS CASE, 2015.
SO SOME XML ELEMENTS ALSO HAVE ATTRIBUTES, WHICH
PROVIDE MORE INFORMATION ABOUT THE ELEMENT.
SO ATTRIBUTES ARE INSIDE THE OPEN TAG AND THEN
HAVE BOTH A NAME AND A VALUE.
SO YOU CAN SEE IN THIS EXAMPLE, WE HAVE THE
ELEMENT IS MEDLINE CITATION AND THEN THE
ATTRIBUTE'S NAME IS STATUS AND THEN THE VALUE
OF THE ATTRIBUTE IS MEDLINE.
SO HERE WE HAVE A FRAGMENT OF AN XML
DOCUMENT.
SO THIS IS BASED LOOSELY ON A PUB MED XML
DOCUMENT.
IT NOT COMPLETE AND IT IS MISSING A BUNCH OF
REQUIRED ELEMENTS, BUT WE WANTED TO MAKE SURE IT
FIT ON THE PAGE FOR YOU.
SO THE FIRST THING THAT WE SEE HERE IS THE OPEN
TAG OF THE ELEMENT PUB MED ARTICLE SET.
SO THE CLOSED TAG IS DOWN AT THE BOTTOM OF THE
SCREEN.
THE REST OF THE DOCUMENT IS CONTAINED WITHIN THIS
ELEMENT.
SO THIS DEMONSTRATES ANOTHER IMPORTANT CONCEPT
OF XML.
INSTEAD OF CONTAINING TEXT, LIKE OUR PREVIOUS
EXAMPLE, XML ELEMENTS CAN ALSO CONTAIN OTHER
ELEMENTS.
SO THIS CREATES A HIERARCHAL STRUCTURE
WHERE ELEMENTS ARE NESTED INSIDE A PARENT ELEMENT.
SO FOR EXAMPLE, THE PUB MED ARTICLE SET HAS A
CHILD ELEMENT PUB MED ARTICLE, AND THEN PUB MED
ARTICLE HAS ITS OWN CHILDREN.
T MAD, CREATED, JOURNAL, ARTICLE TITLE AND
ELOCATION ID.
SO SOME OF THESE TABS LIKE ARTICLE TITLE YOU
CAN SEE THE TITLE HERE.
BUT THEN OTHERS, LIKE DATE CREATED, HAVE
CHILDREN OF THEIR OWN.
SO THIS HAS THE CHILD ELEMENT, YEAR, MONTH AND DAY.
AND WE ALSO HAVE A FEW ELEMENTS WITH ATTRIBUTES.
SO ISSN HAS A TYPE ATTRIBUTE WITH A VALUE
ELECTRONIC.
AND THEN ELOCATION ID HAS TWO ATTRIBUTES -- EID
TYPE AND VALID YN.
SO THIS IS ANOTHER MARKUP AND THIS INVOLVES A
STRUCTURE YOU ARE GOING TO BE SEEING A LOT TODAY,
WHICH IS THAT STRUCTURE YOU ARE GOING TO GET WHEN
YOU USE EFETCH TO GET RECORDS FROM PUB MED.
SO AGAIN, THE WHOLE DOCUMENT IS WRAPPED IN
THIS PUB MED ARTICLE SET ELEMENT, WHICH CONTAINS
SEVERAL PUB MED ARTICLE CHILD ELEMENTS.
SO EACH PUB MED ARTICLE ELEMENT CONTAINS ALL OF
THE INFORMATION FOR A SINGLE PUB MED RECORD.
SO THIS ALSO DEMONSTRATES ANOTHER IMPORTANT FACT
ABOUT XML ELEMENTS.
SOME XML ELEMENTS CAN BE REPEATED WITHIN THE SAME
DOCUMENT.
SO YOU ARE GOING TO SEE THIS WHEN YOU STACK
MULTIPLE RECORDS.
YOU ARE GOING TO SEE ELEMENTS LIKE AUTHOR OR
MESH HEADINGS REPEAT SINCE AN ARTICLE CAN HAVE
MULTIPLE AUTHORS OR MULTIPLE MESH HEADINGS.
SO NOW CONVERTING THIS XML STRUCTURE INTO A
TABLE CAN BE REALLY TRICKY.
WE NEED TO MAKE SURE THAT WE GET THE RIGHT DATA
FROM THE RIGHT PARTS OF THE HIERARCHY AND ARRANGE
IT IN A WAY THAT MAKES LOGICAL SENSE.
SO BEFORE WE JUMP INTO THAT, ARE THERE ANY
QUESTIONS?
I AM NOT SEEING ANYTHING.
GETTING INFORMATION IN AN EASY-TO-USE FORM IS WHY
WE WANT TO USE EXTRACT.
SO EXTRACT IS A BIG PART OF WHAT'S USEFUL ABOUT
EDIRECT.
SO IT'S A COMMAND THAT EXTRACTS SPECIFIC
ELEMENTS FROM XML AND ARRANGES THEM IN A CUSTOM
TABULAR FORMAT.
BUT LIKE MOST COMMANDS, YOU CAN CONTROL THE
DETAILS OF HOW EXTRACT WORKS BY PROVIDING
ARGUMENTS.
SO WHICH ARGUMENTS YOU PROVIDE ARE GOING TO
DETERMINE WHAT THAT FORMAT LOOKS LIKE.
SO EXTRACT IS A PART OF EDIRECT BUT IT'S NOT
REALLY A UTILITY COMMAND SO IT DOESN'T USE THE ETI
TO CONTACT THE SERVER.
IT WORKS ON AN XML DATA AND SO THIS MAKES IT A
VERY IMPORTANT TOOL FOR PROCESSING DATA THAT YOU
GET FROM THE OTHER EDIRECT COMMANDS, WHICH
DO CONTACT THE SERVER.
ONE OF THE ADVANTAGES OF EXTRACT IS THAT BECAUSE
IT'S NOT ACTUALLY CONTACTING THE UTILITY
SERVER TO GET ITS INPUT, IT CAN ACTUALLY WORKING
ON ANY XML.
SO FOR MOST EXAMPLES, OUR XMLS ARE GOING TO BE THE
OUTPUT OF AN EFETCH THAT WE TYPE INTO EXTRACT AND
WHEN WE DO THAT, WE NEED TO MAKE SURE THAT WE'RE
USING THAT FORMAT XML ARGUMENT.
BUT YOU CAN ALSO USE EXTRACTS' INPUT ARGUMENT
TO SPECIFY AN XML FILE THAT'S ALREADY ON A
COMPUTER.
SO WE'RE GOING TO DO THAT TODAY BUT WE'LL BE
TOUCHING ON THAT IN A LATER CLASS.
SO THERE ARE A LARGE NUMBER OF XML ELEMENTS IN
PUB MED.
AND WHILE YOU WILL PROBABLY BE ABLE TO
EASILY DETERMINE WHICH ELEMENTS REFER TO
INFORMATION THAT YOU SEE IN THE WEB VERSION, WE
ALSO HAVE DOCUMENTATION TO HELP WITH THAT.
SO I AM GOING TO OPEN UP MY BROWSER AND TAKE A
LOOK AT THESE.
SO THESE LINKS SHOULD BE ON YOUR HANDOUT AS WELL.
SO THIS DOCUMENTATION, THIS MEDLINE PUB MED XML
ELEMENTS DESCRIPTIONS AND THEIR ATTRIBUTES PAGE --
A MOUTHFUL RIGHT THERE -- LISTS THE DIFFERENT
ELEMENTS THAT YOU WILL FIND IN PUB MED AS WELL
AS THEIR ATTRIBUTES.
SO THIS FIRST PAGE THAT YOU SEE LISTS THE
ELEMENTS THAT YOU WOULD IN THE ORDER THAT YOU
WOULD SEE THEM IN THE PUB MED XML.
SO IF I SCROLL DOWN PAST THIS INTRO, YOU CAN SEE
THAT WE START TO GET A LIST OF THOSE ELEMENTS.
IF I GO BACK UP TO THE TOP, YOU CAN SEE UNDER
THE HEADING THE ELEMENTS AND THEIR ATTRIBUTES
SECTION, THERE IS A LINK TO AN ALPHABETICAL LIST
OF ALL THE ELEMENT NAMES.
SO I FIND THIS LIST TO BE THE MOST HELPFUL IF I AM
LOOKING AT AN XML RECORD AND STILL WANT TO KNOW
WHEN A CERTAIN ELEMENT OR ATTRIBUTE IS REFERRING TO.
SO I CAN JUST LOOK IT UP HERE.
IT'S A LITTLE BIT EASIER TO FIND BECAUSE IT IS IN
THAT ALPHABETICAL LIST.
SO WE DO ALSO HAVE A NEW RESOURCE FOR XML ELEMENT
DOCUMENTATION, AND THIS IS THAT SECOND LINK ON
THE HANDOUT.
AND THIS IS LINKED TO THE PUB MED DTT AND WHILE IT
DOES HAVE THE SAME INFORMATION AS THE OTHER
PAGE, THIS IS STILL VERY, VERY NEW AND IT'S STILL A
LITTLE BIT UNDER DEVELOPMENT SO WE'RE
STILL GOING TO BE ADDING MORE INFORMATION TO IT AS
WE GO.
SO IF I OPEN UP THIS ELEMENT LIST HERE, YOU
CAN SEE WE'VE GOT A LOT OF THE ALPHABETICAL
ELEMENTS JUST LIKE IN THAT LAST PAGE.
LOOK AT THE ABSTRACT.
IT'S GOT A DEFINITION AND INFORMATION AS WELL AS AN
EXAMPLE.
AND ANOTHER HANDY TIP HERE IS THE ATTRIBUTES
AREA CAUSE THAT SHOWS YOU ALL OF THE DIFFERENT
ATTRIBUTES.
FOR INSTANCE, IF I CLICK THE MAJOR TOPIC YN, YOU
CAN SEE WHERE IT'S ALLOWED AND THE ALLOWABLE
VALUES AS WELL.
SO I JUST FIND THAT THIS CAN BE A LITTLE BIT
HELPFUL, MORE HELPFUL THAN IN -- WHEN
EVERYTHING IS IN AN ALPHABETICAL LIST.
SO ILL GOING TO HEAD BACK TO MY SLIDES TODAY AND
YOU WILL SEE ME JUMP BACK AND FORTH BETWEEN
DIFFERENT FILES A LOT TODAY.
SO WHENEVER WORKING WITH EXTRACT, WE ARE GOING TO
BE DIRECTING THE COMMAND TO SELECT SPECIFIC
ELEMENTS AND ATTRIBUTES FROM SPECIFIC PARTS OF
THE XML RECORD.
SO TO HELP US CHOOSE THE RIGHT ELEMENT, WE HAVE TO
UNDERSTAND HOW THE XML IS STRUCTURED.
TO HELP US CHOOSE -- OR THE EASIEST WAY TO DO
THIS IS TO ACTUALLY JUST HAVE THE XML OF SAMPLE
PUB MED RECORDS OPEN IN A WINDOW THAT WE CAN
CONSULT.
SO TO DO THIS, I AM ACTUALLY GOING TO JUMP
BACK TO THE WEB BROWSER AND I AM GOING TO HEAD TO
PUB MED.
SO I AM GOING TO CLOSE THESE TWO.
AND THEN I AM GOING TO DO A SEARCH FOR THREE PM IDS.
SO THESE PM IDS ARE IN THE HANDOUT.
AND THERE WE GO.
I AM GOING TO JUST DO A QUICK SEARCH FOR THEM IN
PUB MED.
AND SO NOW THAT I HAVE THIS HERE, I CAN CHANGE
MY FORMAT FROM WHATEVER IT'S CURRENTLY SET TO TO
ABSTRACT.
NOT ABSTRACT, EXCUSE ME.
TO XML.
I AM SO USED TO CLICKING ABSTRACT FOR MY REGULAR
WORK THAT I AUTOMATICALLY WENT THERE.
SO ONCE YOU HAVE CHANGED TO XML, YOU CAN SEE WHAT
IT LOOKS LIKE IN XML.
AND THIS IS WHAT YOU ARE GOING TO BE EXTRACTING
FROM.
SO THIS IS THE EXACTLY WHAT YOU WILL SEE IF YOU
USE THAT FORMAT XML ARGUMENT.
SO WE'RE GOING TO KEEP THIS OPEN AND REFER BACK
TO IT LATER JUST SO WE CAN FIGURE OUT WHERE THE
DATA IS THAT WE'RE LOOKING FOR WITHIN THAT
XML STRUCTURE.
SO I AM GOING TO HEAD BACK TO MY SLIDES.
SO YOU ARE GOING TO WANT TO DO A LOT OF TESTING
AND FINDING OF YOUR EXTRACT COMMANDS.
SO IN ADDITION TO LOOKING AT EXAMPLE XMLS TO
UNDERSTAND THE STRUCTURE, IT'S ALSO HELPFUL TO GET
A SMALL SET OF RECORDS THAT YOU CAN USE TO TEST
YOUR XML STATEMENTS.
SO IDEALLY YOU WANT A HANDFUL OF RECORDS THAT
ARE REPRESENTATIVE OF THE KIND OF RECORDS YOU ARE
GOING TO BE EXTRACTING FROM FOR REAL.
SO YOU WANT TO MAKE SURE THOSE SAMPLE RECORDS HAVE
THE SAME KINDS OF DATA AS YOUR LIVE DATA SET.
SO FOR EXAMPLE, IF YOU KNOW YOU ARE TRYING TO
EXTRACT FUNDING INFORMATION, MAKE SURE
YOU'VE GOT A COUPLE TEST RECORDS THAT DO HAVE
FUNDING INFORMATION ON IT TO MAKE SURE EVERYTHING
IS WORKING.
SO AS WE JUST MENTIONED, THERE ARE A LOT OF WAYS
TO GET XML INTO AN EXTRACT.
IN ORDER TO MAKE SURE YOU CAN EASILY RECREATE MY
EXAMPLES, I AM GOING TO BE USING THE SAME EFETCH
TODAY FOR ALMOST EVERY EXAMPLE.
AND THEN WE'RE GOING TO PIPE THAT INTO EXTRACT.
SO THAT EFETCH IS LISTED ON THE HANDOUT AND IT'S
GOING TO USE THOSE SAME THREE PM IDS THAT I'VE
JUST SEARCHED FOR.
SO I'M GOING TO OPEN UP MY TERMINAL SEG WIN
TERMINAL AND TAKE A LOOK AT THAT.
SO I AM GOING TO CREATE MY EFETCH BY SPECIFYING
THE DATABASE, WHICH IS PUB MED AND PUT IN MY ID
ARGUMENT AND PASTE IN THOSE THREE PM IDS.
I AM GOING TO SPECIFY THAT I WANT TO BE IN
FORMAT XML.
SO THEN I AM JUST GOING TO HIT ENTER AND LET MY
COMMAND RUN.
AND YOU CAN SEE WE HAVE THAT SAME XML THAT WE
JUST SAW IN PUB MED.
ALL RIGHT, SO SOMEONE WAS ASKING FOR THE LINK TO
THE HANDOUTS?
I BELIEVE EARLIER TODAY KATE HAD PUT IN A LINK TO
THE CLASS MATERIALS PAGE AND THAT CLASS MATERIALS
PAGE HAS ALL OF THE HANDOUTS, THE SCRIPTS,
THE SLIDES, EVERYTHING THAT YOU CAN NEED FOR
THAT.
AND HOPEFULLY KATE CAN STICK THAT IN THERE FOR
US.
IT'S GOING TO TAKE JUST A SECOND SO WE CAN GET THAT
FOR YOU.
ALL RIGHT, THERE IT IS.
ALL RIGHT.
SO LET ME GO BACK TO MY SLIDES.
>> SO BECAUSE EXTRACT SOS POWERFUL AND HAS SO MANY
OPTIONS, IT CAN SOMETIMES BE A LITTLE OVERWHELMING.
SO WE'RE GOING TO START WITH A VERY SIMPLE
EXAMPLE.
SO WE HAVE A SET OF PUB MED RECORDS IN XML.
WE JUST DID THAT.
BUT WHAT WE WANT IS A TABULAR LIST WITH THE PM
ID, THE JOURNAL TITLE ABBREVIATION, OR JOURNAL TA.
AND THE ARTICLE TITLE.
SO SOUNDS PRETTY BASIC, RIGHT?
PRETTY SIMPLE.
BUT EVEN IF YOU JUST DOWNLOADED A CSV
FROM PUB MED, THAT JOURNAL TA IS GOING TO BE
BURIED WITHIN YOUR DETAILS FIELD SO YOU ARE
STILL GOING OF OTHER TO DIG THAT OUT AND DO A
LITTLE PROCESSING ON THAT.
SO THIS IS OUR GOAL.
LET'S SEE HOW WE DO THAT.
SO WE WANT TO MAKE A TABLE.
SOUNDS SIMPLE, RIGHT?
THE FIRST THING WE NEED TO DO IS TO PLAN OUT WHAT
OUR TABLE'S GOING TO LOOK LIKE.
SO THIS IS OUR SAMPLE TABLE.
AND THE FIRST THING WE WANT TO CONSIDER IS
WHAT'S GOING TO CONNECT THE DATA IN EACH ROW.
SEEMS OBVIOUS BUT IF THE DATA IN EACH ROW ISN'T
ALL RELATED, THE TABLE'S NOT GOING TO TELL
US VERY MUCH.
SO IN THIS EXAMPLE, THE CONNECTION IS THAT ALL OF
THE DATA IN A ROW IS FROM THE SAME PUB MED RECORD.
SO THAT'S THE RELATIONSHIP.
THEN WE NEED TO FIGURE OUT HOW MANY ROWS
WE NEED.
WE WANT AS MANY ROWS AS WE HAVE PUB MED RECORDS
IN OUR XML.
AND HOW MANY COLUMNS DO WE NEED?
AND WHAT DATA IS IN EACH COLUMN?
SO LIKE ALL EDIRECT COMMANDS, WE SPECIFY
THESE THINGS BY CAREFULLY CHOOSING OUR ARGUMENTS.
SO WE DETERMINE WHAT CONNECTS OUR DATA BY
USING THE PATTERN ARGUMENT.
WE SPECIFY AN XML ELEMENT FOR THE PARTNER ARGUMENT
AND ALL OF THE DATA IN A SINGLE ROW WILL COME FROM
THE DESCENDANTS.
SO THE CHILD ELEMENTS, THE CHILDREN OF CHILDREN,
ET CETERA, OF A SINGLE OCCURRENCE OF THAT
PATTERN ELEMENT.
EXTRACT WILL ALSO SCAN THE XML DOCUMENT YOU
INPUT UNTIL IT FINDS AN OCCURRENCE OF THAT
PATTERN.
AND WHEN IT FINDS AN OCCURRENCE OF THE
PATTERN, IT'S GOING TO CREATE A NEW ROW.
SO AGAIN, THIS IS A DUMMY XML DOCUMENT, AND YOU CAN
SEE AT THE TOP HERE, WE SAID OUR PATTERN IS GOING
TO BE PUB MED ARTICLE.
SO EXTRACT IS GOING TO SCAN THROUGH UNTIL IT
FINDS THE FIRST OCCURRENCE OF THAT
ELEMENT.
AND ONCE IT DOES, IT'S GOING TO CREATE A NEW ROW
IN OUR OUTPUT TABLE.
SO ALL OF THE DATA IN THAT ROW WILL COME FROM
WITHIN THAT PATTERN, WITHIN THIS ONE PUB MED
ARTICLE.
ONCE EXTRACT REACHES THE END OF THE PATTERN, IT
LOOKS FOR THE NEXT OCCURRENCE OF THAT
PATTERN AND CREATES ANOTHER ROW.
SO ALL OF THE DATA IN THAT SECOND ROW WILL COME
FROM WITHIN THAT SECOND PATTERN.
AND SO ON AND SO FORTH.
UNTIL WE REACH THE END OF THE DOCUMENT AND CAN FIND
NO MORE OCCURRENCES OF THE PATTERN.
THEN WE STOP.
SO THIS ALSO ANSWERS OUR SECOND QUESTION, HOW MANY
ROWS?
SO WE WILL HAVE ONE ROW PER OCCURRENCE OF THE
PATTERN IN THE XML INPUT.
SO IF OUR PATTERN IS PUB MED ARTICLE, OUR EXTRACT
COMMAND WILL CREATE AS MANY ROWS AS THERE ARE
RECORDS IN OUR XML DOCUMENT.
SO THIS BRINGS UP AN INTERESTING QUESTION.
YOU MIGHT BE THINKING WELL, WOULDN'T MY PATTERN
ALWAYS BE PUB MED ARTICLE AND IF SO, WHY AM I
BOTHERING TO SPECIFY THIS?
WELL, MOST OF THE TIME YES.
ALL OF THE DATA IN THE SINGLE ROW IS GOING TO
COME FROM THAT ONE PUB MED RECORD BECAUSE YOU
HAVE SPECIFIED THIS BECAUSE THIS IS WHAT
ALLOWS YOU TO TABULATE PUB MED RECORDS AND SEE
TRENDS IN THE DATA.
HOWEVER, YOU COULD ALSO CHANGE THAT PATTERN TO
SEE DIFFERENT TYPES OF RELATIONSHIPS.
SO FOR INSTANCE, IF YOU ARE LOOKING AT AUTHOR
INFORMATION OR GRANT DATA, YOU MIGHT WANT TO
USE THE AUTHOR OR GRANT ELEMENT AS YOUR PATTERN.
SO WE'RE GOING TO ACTUALLY LOOK AT AN
EXAMPLE OF THIS LATER IN CLASS AS WELL.
SO THAT COVERS YOUR ROWS.
NOW LET'S TALK ABOUT CREATING COLUMNS.
SO YOU DO THAT USING A DIFFERENT ARGUMENT
ELEMENT.
YOU SPECIFY ONE OR MORE XML ELEMENT OR ATTRIBUTE
IN THE ELEMENT ARGUMENT.
IN EACH ELEMENT OR ATTRIBUTE YOU SPECIFY
WILL CREATE A NEW COLUMN.
AND HOW MANY ELEMENTS OR ATTRIBUTES YOU SPECIFY
DETERMINES THE NUMBER OF COLUMNS.
WELL, MOST OF THE TIME.
LIKE ANY RULE, THERE ARE ALWAYS EXCEPTIONS, AND
WE'RE GOING TO TALK A LITTLE BIT MORE ABOUT
THOSE IN THE NEXT CLASS.
BUT FOR TODAY WE'RE JUST GOING TO ASSUME THAT THIS
IS TRUE.
SO IF WE GO BACK TO OUR PATTERN, REMEMBER THAT
ALL OF THE DATA IN A ROW IS GOING TO COME FROM
WITHIN THAT SAME PATTERN.
SO EXTRACT LOOKS INSIDE THE PATTERN FOR THE
ELEMENTS AND ATTRIBUTES YOU SPECIFIED IN THE
ELEMENT ARGUMENT.
SO THE VALUE OF EACH OCCURRENCE OF YOUR FIRST
ELEMENT GOES INTO THE FIRST COLUMN.
THE EXTRACT MOVES ON TO THE SECOND COLUMN AND
PUTS THE VALUE OF THE SECOND ELEMENT IN THE
SECOND COLUMN.
ET CETERA.
SO HERE'S WHAT THAT LOOKS LIKE.
SO WE'VE GOT OUR SAME DUMMY XML AND WE'RE STILL
USING THAT SAME PUB MED ARTICLE.
AND YOU CAN SEE AT THE TOP WE HAVE SPECIFIED OUR
ELEMENT AS PM ID, YEAR, TITLE AND ARTICLE.
SO EXTRACT WILL FIND OUR FIRST PATTERN AND THEN
LOOKS WITHIN THAT PATTERN FOR THE FIRST ELEMENT IN
OUR ELEMENT ARGUMENT, WHICH WE SAID IS PM ID.
SO FIND THE PM ID ELEMENT AND PUTS THE FIRST
CONTENT IN THE FIRST COLUMN.
THEN IT FINDS A YEAR ELEMENT AND PUTS THAT
CONTENT INTO THE SECOND COLUMN.
THEN IT FINDS AN ARTICLE TITLE ELEMENT AND PUTS
THE CONTENT INTO THE THIRD COLUMN.
THEN THERE ARE NO MORE COLUMNS AND IT'S REACHED
THE END OF THE PATTERN SO IT MOVES ON TO THE SECOND
PUB MED ARTICLE.
CREATES A NEW ROW AND REPEATS THE PROCESS,
LOOKING FOR THE PM ID, YEAR, AND ARTICLE TITLE.
SO AGAIN, THIS PROCESS IS GOING TO REPEAT UNTIL WE
RUN OUT OF PATTERNS.
SO HOPEFULLY THIS IS ENOUGH WALK-THROUGH.
AND WE'RE GOING TO SEE WHAT THIS ACTUALLY LOOKS
LIKE IN PRACTICE.
SO YOU'VE ALREADY SEEN A LITTLE SNEAK PEAK OF THE
SYNTAX THAT EXTRACT USES BUT THIS IS A BASIC
EXTRACT COMMAND.
PRETTY SIMPLE.
SO HERE IS HOW IT WORKS.
OUR EXTRACT COMMAND HAS A PATTERN OF PUB MED
ARTICLE, AND AN ELEMENT OF ARTICLE TITLE.
SO LET'S GO TO CYGWIN AND SEE A DEMO OF THIS.
FIRST THING I AM GOING TO DO IS SIMPLY CLEAR MY
SCREEN SO YOU GUYS CAN SEE WHAT I AM DOING.
AND I AM GOING TO USE MY UP ARROWS AND GO BACK TO
EFETCH THAT I DID A COUPLE MINUTES AGO WITH
THOSE THREE PM ID THAT WE'RE GOING TO BE USING
FOR ALMOST EVERY EXAMPLE TODAY AND IT'S ALSO IN
YOUR HANDOUT.
SO I AM GOING TO TAKE THIS AND I AM GOING TO
PIPE IT INTO MY EXTRACT COMMAND.
I AM PUTTING THAT ON A SEPARATE LINE USING THAT
SLASH JUST SO THAT I CAN READ EVERYTHING A LITTLE
BIT EASIER TODAY.
AND DO EXTRACT AND THEN I SAID MY PATTERN WAS PUB
MED ARTICLE.
AS LONG AS I SPELL IT RIGHT.
AND THEN I SAID MY ELEMENT WAS ARTICLE TITLE.
SO NOW I'VE COMBINED THAT EFETCH WITH MY EXTRACT
AND I AM GOING TO HIT ENTER AND RUN THIS
COMMAND.
SO NOW YOU CAN SEE THAT WE GOT OUR RESULTS.
WE HAVE ONE ROW PER ARTICLE BECAUSE OF THAT
PATTERN PUB MED ARTICLE.
WITH A SINGLE COLUMN THAT DISPLAYS THE ARTICLE
TITLE.
I WILL NEED THIS LOOKS LIKE I ALL ABOUT IT HAVE
FOUR TITLES HERE, BUT THAT FIRST TITLE IS JUST
SO LONG THAT IT WRAPS ALL THE WAY AROUND.
SO THERE IS REALLY ONLY THREE ARTICLE TITLES
HERE.
SO THERE IS OUR FIRST EXTRACT.
WE MADE IT THROUGH.
ALL RIGHT, SO BUT LET'S SAY THAT INSTEAD OF
ARTICLE TITLE, I WANTED TOO SEE A LIST OF -- I
WANT TO SEE AUTHORS FROM THESE THREE PM IDS IN ONE
COLUMN.
SO LET'S CHANGE OUR EXTRACT TO USE A
DIFFERENT PATTERN TO DO THIS.
SO I AM GOING TO GO UP JUST AND THEN PIPE THIS
INTO MY EXTRACT.
AND THEN I SAID I WAS GOING TO CHANGE MY
PATTERN AND I SAID THAT I WANTED TO LOOK AT THE
AUTHORS FOR ALL OF THIS.
WHICH MEANS I WANT ONE AUTHOR PER LINE.
SO I AM GOING TO USE THAT AUTHOR ELEMENT HERE, TO
SPECIFY MY PATTERN AS AUTHOR.
AND THERE ARE GOING TO BE MULTIPLE AUTHORS ON EACH
RECORD MOST LIKELY BECAUSE MOST RECORDS IN
PUB MED HAVE MORE THAN ONE OF THOSE.
AND WE ALSO COULD HAVE DUPLICATES SO YOU MAY END
UP SEEING MULTIPLE THINGS LISTED IN HERE AS WELL.
SO WE HAVE OUR AUTHOR AND THEN I AM JUST GOING TO
SAY THAT I WANT MY ELEMENT TO BE LAST NAME.
JUST GOING TO PICK THE MOST BASIC ONES SO I GET
A LIST OF ALL THE LAST NAMES.
AND HIT ENTER AND RUN THIS.
AND SO NOW YOU CAN SEE WE'VE GOT MORE THAN THREE
LINES HERE BECAUSE WE ARE USING THAT DIFFERENT
PATTERN.
WE'VE SAID FOR EVERY AUTHOR ELEMENT -- OR FOR
EVERY AUTHOR IN THE XML, WE WANT A SEPARATE LINE
FOR IT AND WE HAVE PUT THAT LAST NAME HERE.
SO MOST OF THESE EXAMPLES THAT WE'RE GOING TO BE
SEEING TODAY WE'LL USE THAT PATTERN PUB MED
ARTICLE.
BUT I WANTED TO MAKE SURE THAT I SURE YOU HOW
EXTRACT CAN WORK FOR YOU AND CAN REALLY EXTRACT
THE INFORMATION THAT YOU WANT.
SO THIS IS THE BASICS OF EXTRACT.
AND OF COURSE, WE DO KEEP TRACK OF HOW EXTRACT
GIVES US THESE GREAT TABULAR OUTPUT AND WE'RE
SAYING THIS IS REALLY WHAT WE WANT.
AND THIS IS REALLY JUST A LIST FOR BOTH OF THESE.
IT'S JUST A LIST OF AUTHORS OR ARTICLE TITLES.
SO WHAT WE REALLY NEED ARE MULTIPLE COLUMNS.
SO BACK TO MY SLIDES.
THERE WE GO.
SO TO GET MULTIPLE COLUMNS, WE CAN JUST
SPECIFY MORE THAN ONE ELEMENT IN THAT ELEMENT
ARGUMENT.
SO YOU SIMPLY SEPARATE THEM WITH SPACES AND YOU
DON'T NEED TO REPEAT THAT ELEMENT NOTATION EITHER.
SO YOU CAN SEE IN THIS EXAMPLE, WE'RE EXTRACTING
BASED ON THAT PATTERN PUB MED ARTICLE AND WE'RE
CREATING TWO COLUMNS -- ONE FOR AGENCY AND ONE
FOR GRANT ID.
SO ARE THERE ANY QUESTIONS OUT THERE,
BEFORE WE GET INTO OUR FIRST EXERCISE?
LOOKS LIKE MIKE HAS BEEN -- MIKE AND KATE HAVE
BEEN TAKING CARE OF THESE.
IF YOU HAVE QUESTIONS, STICK THEM IN THE CHAT
BOX AS WE GO.
AND LET'S JUMP RIGHT INTO OUR FIRST EXERCISE.
SO I WOULD LIKE YOU GUYS TO WRITE AN EXTRACT
COMMAND THAT CREATES A TABLE.
WE WANT ONE ROW PER PUB MED RECORD AND THEN EACH
ROW SHOULD HAVE TWO COLUMNS -- VOLUME AND
ISSUE NUMBER.
SO WE'VE GIVEN YOU GUYS THE EFETCH HERE.
IT'S THE SAME ONE WE'VE BEEN USING FOR OUR DEEMOS
AND IT'S ALSO IN THE HANDOUT.
SO I WANT YOU GUYS TO TRY IT, TEST IT OUT AND THEN
GIVE ME A GREEN CHECK MARK WHEN YOU THINK
YOU'VE GOT IT.
SO WE'RE ALSO GOING TO GIVE YOU A COUPLE HINTS
IN THE CHAT BOX AS YOU WORK THROUGH THIS.
SO IF YOU WANT TO TRY THIS WITHOUT ANY HINT
WHATEVER, JUST IGNORE THE CHAT BOX FOR THE NEXT
COUPLE MINUTES.
I AM GOING TO GIVE YOU A COUPLE MINUTES AND ONCE
AGAIN, GIVE ME A GREEN CHECK MARK WHEN YOU'RE
FINISHED.
>> ALL RIGHT.
SO IT LOOKS LIKE MOST OF YOU HAVE FINISHED THIS,
SO I AM GOING TO GO AND WALK THROUGH THIS THE
SOLUTION WITH YOU RIGHT NOW.
SO I AM GOING TO ESCAPE.
AND I AM GOING TO CLEAR MY SCREEN FIRST IN CYGWIN
JUST SO YOU GUYS CAN SEE WHAT I AM WORKING ON.
SO WE'RE GOING TO START WITH THAT FROM THE
HANDOUT.
I AM GOING TO USE MY UP ARROWS ONCE AGAIN TO GET
TO THAT.
AND THEN I AM TYPING THAT INTO MY EXTRACT.
SO I SAID THAT I WANTED A TABLE WITH ONE ROW PER
PUB MED ARTICLE.
SO THAT'S GOING TO TELL ME THAT I WANT MY PATTERN
TO BE PUB MED ARTICLE.
SO I AM GOING TO PUT IN PATTERN, PUB MED, ARTICLE.
AND THEN I SAID THAT I WANTED EACH ROW TO HAVE
TWO COLUMNS.
SO THAT'S GOING TO SPECIFY MY ELEMENT ARGUMENT.
ELEMENT.
AND THEN I KNOW FROM EITHER LOOKING IN PUB MED
AT THE XML OR LOOKING IN THE CHAT BOX THAT THOSE
ARE GOING TO BE VOLUME AND ISSUE.
SO I CAN THEN JUST HIT ENTER AND RUN MY COMMAND.
SO YOU CAN SEE HERE, I'VE GOT THOSE TWO COLUMNS.
I'VE GOT ONE FOR VOLUME AND ONE FOR ISSUE.
SO HOPEFULLY, EVERYBODY WAS ABLE TO FOLLOW ALONG
WITH THAT.
SO THIS IS A PRETTY BASIC SOLUTION A PRETTY BASIC
QUESTION.
BUT DEPENDING ON WHAT TYPE OF RECORDS WE'RE
WORKING WITH AND WHAT ELEMENTS WE WANT TO
EXTRACT, WE COULD GET INTO TROUBLE.
SO I AM GOING TO TELL YOU THAT IF YOU ARE STILL
WORKING ON THAT LAST EXERCISE I AM GOING TO
ASK YOU TO PUT THAT ASIDE FOR RIGHT NOW.
JUST CAUSE I WANT YOU GUYS TO BE ABLE TO
CONTINUE TO FOLLOW ALONG AND WORK ON THIS NEXT
EXAMPLE WITH ME.
SO JUST AS A REMINDER, ALL OF THE SOLUTIONS TO
THE EXERCISES ARE AT THE BOTTOM OF THE HANDOUT AS
WELL.
SO LET'S RUN A NEW EXTRACT ON JUST ONE PM ID
TO SEE THE PM ID AND YEAR FOR THE CITATION.
SO THIS IS GOING TO BE THE ONE-TIME TODAY WE'RE
GOING TO LOOK AT A DIFFERENT PM ID.
SO WE'RE GOING TO WRITE MY EFETCH AND DO DATABASE
PUB MED AND I AM GOING TO SPECIFY I WANT MY ID TO
BE THIS ONE.
27101380.
SO IT'S GOING TO BE ON YOUR HANDOUT AND KATE
JUST PUT THAT IN THE CHAT BOX FOR YOU AS WELL.
SO THEN I AM GOING TO SAY THAT I WANT THIS TO BE
INFORMANT XML SO THAT I CAN USE MY EXTRACT TO
WORK WITH IT AND I AM GOING TO TYPE IT INTO
THAT EXTRACT.
AND THEN I AM GOING TO PUT IN MY PATTERN PUB MED
ARTICLE CAUSE THERE IS ONLY ONE OF THEM.
SO THAT'S GOING TO ALREADY CONNECT
EVERYTHING TOGETHER.
THEN I SAID I JUST WANTED TO SEE THE ELEMENTS PM ID
AND YEAR.
PRETTY SIMPLE, RIGHT?
I AM EXPECTING TO SEE ONE PM ID AND ONE YEAR.
BUT AS YOU CAN SEE, THAT IS CLEARLY NOT WHAT WE
GOT HERE.
WE GOT A WHOLE MESS OF STUFF RIGHT HERE.
WE'VE GOT A BUNCH OF POLITICAL IDS AND WE'VE
GOT A BUNCH OF YEARS TOO, WHICH IS NOT WHAT WE ARE
HOPING FOR.
PM IDS.
SO EVEN THOUGH WE'RE ONLY EXTRACTING FROM ONE PM
ID, WE'VE GOT WAY MORE DATA AND MORE PM IDS THAN
WE WANT.
SO LET'S GO INTO PUB MED AND SEE IF WE CAN FIGURE
OUT WHAT'S GOING ON WITH IT.
I AM GOING TO OPEN UP PUB MEDDED AND GO BACK SO
THAT I CAN SEARCH FOR THIS PM ID.
SO I'VE GOT THE RECORD UP, AND I AM GOING TO
CHANGE THE FORMAT TO XML.
I AM GOING TO PICK THE RIGHT, I PROMISE.
SO WE'VE GOT OUR XML FOR THIS ONE RECORD.
SO IF WE LOOK AT THE VERY TOP, WELL, WE'VE GOT THAT
PM ID RIGHT HERE, LOOKS PRETTY GOOD, RIGHT?
BUT THEN YOU CAN ALSO SEE RIGHT UNDERNEATH THAT
WE'VE GOT A YEAR FOR UNDER DATE COMPLETE.
WE'VE GOT A YEAR UNDER DATE REVISED.
WE'VE GOT ANOTHER YEAR UNDER PUB DATE AND THAT'S
THE ONE I WAS EXPECTING TO SEE, RIGHT?
YOU CAN ALREADY SEE WE'VE GOT A FEW YEARS COMING UP.
BUT WHERE ARE THOSE PM IDS COMING FROM?
I AM USING CONTROL F TO FIND THIS AND GOING TO
SEARCH FOR PM ID TO SEE WHAT I GET.
SO YOU CAN SEE HERE, I'VE GOT A GIANT LIST OF ALL
THESE PM IDS AND THIS IS ALL COMING FROM THE
COMMENTS AND CORRECTIONS AREA.
SO THESE ARE OTHER PM IDS THAT REFERENCE THE ONE
WE'RE LOOKING AT.
AND SO THAT'S WHY WE'RE CONNECTING THEM TOGETHER
IN THE XML.
BUT UNFORTUNATELY, THAT MEANS IT'S A LITTLE HARD
FOR US TO EXTRACT JUST THAT ONE PM ID THAT WE'RE
LOOKING FOR JUST USING THAT PM ID ELEMENT.
SO ESPECIALLY WHEN WE'RE USING THIS PUB MED
IDENTIFIER AS OUR UNIQUE IDENTIFIER.
SO WHAT WE NEED IS A WAY TO ISOLATE THE SPECIFIC
ELEMENTS THAT WE WANT TO LOOK AT.
SO LET'S HEAD BACK TO MY SLIDES.
AND WE'RE GOING TO ACCOMPLISH THIS BY USING
WHAT WE CALL "PARENT-CHILD CONSTRUCTION".
SO THIS RETRIEVES ONLY THE ELEMENT THAT IS THE
CHILD OF A SPECIFIC PARENT.
SO THIS WILL HELP US TO ISOLATE OBJECTS THAT HAVE
THE SAME NAME, BUT ARE IN DIFFERENT LOCATIONS IN
THE HIERARCHY.
SO ALL YOU NEED TO DO IS CONNECT THE PARENT AND
THE CHILD WITH THE SLASH, WITH A FORWARD SLASH.
SO YOU CAN SEE IN THIS EXTRACT, WE'VE SAID WE
WANT THE ELEMENT PM ID THAT IS THE CHILD OF
MEDLINE CITATIONS.
SO WE'VE GOT THE PARENT ELEMENT SLASH CHILD ELEMENT.
SO WE NOW HAVE ALL THE TOOLS THAT WE NEED TO
SOLVE OUR EXAMPLE THAT WE HAD TALKED ABOUT.
SO TO REFRESH YOUR MEMORY, OUR GOAL WAS TO
HAVE THIS BIG TABULAR LIST WITH THE PM ID,
THE JOURNAL A ABBREVIATION AND THE ARTICLE TITLE.
SO LET'S TAKE A LOOK AT SOLVING THIS,
USING CYGWIN.
SO I AM GOING TO HEAD BACK INTO THAT.
GOING TO CLEAR MY SCREEN.
SO WE'RE GOING USE THAT SAME EFETCH THAT WE'VE
BEEN USING AND PIPE THAT INTO THE EXTRACT.
SO I AM GOING TO GO WAY BACK TO THE TOP OF MINE.
SO WE'VE GOT THOSE THREE PM IDS AGAIN.
WE'RE GOING BACK TO THAT ONE.
WE'VE IGNORED THAT ONE PM ID THAT WE'VE JUST
LOOKED AT.
SO WE'RE GOING TO TYPE THAT INTO OUR EXTRACT.
SO THE FIRST THING WE NEED TO DO IS TO IDENTIFY
OUR PATTERN.
SO SINCE WE KNOW THAT WE WANT ONE ROW PER PUB MED
RECORD -- OR PER RECORD, WE WANT TO USE THAT
PATTERN PUB MED ARTICLE AGAIN.
AND ADD THAT IN.
OOPS.
MAKE SURE YOU DON'T CAPITALIZE THE M IN PUB
MED.
I ALWAYS WANT TO BUT NOT IN THIS INSTANCE.
THE NEXT THING WE NEED TO DO IS DETERMINE WHAT
ELEMENTS WE WANT TO INCLUDE.
SO WE'RE LOOKING FOR THE PM ID, THE JOURNAL P.A.,
AND THE ARTICLE TITLE.
SO LET'S TAKE A LOOK AT THE PUB MED XML.
SO I AM GOING TO JUMP BACK INTO PUB MED AND
THEN I AM JUST FOR CONSISTENCY SAKE, I AM
GOING TO GO BACK JUST A COUPLE SCREENS SO THAT I
HAVE THAT -- THE RESULTS FROM THE SAME EFETCH THAT
WE'RE DOING.
THE SAME THREE PM IDS.
AND NOW I AM GOING TO CHANGE THAT BACK TO XML.
JUST THIS WAY WE'RE ALL LOOKING AT THE SAME
INFORMATION.
SO WE CAN SEE ONCE AGAIN THAT THAT PM ID THAT WE
WANT TO USE IS RIGHT UP HERE AT THE TOP.
BUT AS WE JUST SAW, WE CAN'T -- WE SHOULD BE
USING THIS PARENT-CHILD CONSTRUCTION.
SO WHAT WE'LL DO, DO THE PARENT MEDLINE CITATION,
SLASH PM ID, WHICH IS THE CHILD.
SO ADD THAT INTO OUR XML.
OR INTO OUR CYGWIN COMMAND.
MEDLINE CITATION.
SLASH PM ID.
SO THEN THE NEXT ELEMENT THAT -- NEXT COLUMN THAT
WE WANTED WAS THE JOURNAL TA.
SO I AM GOING TO GO BACK TO PUB MED.
AND THEN I HAPPEN TO KNOW JUST FROM DOING A LOT OF
PUB MED SEARCHING AND LOOKING AT THIS XML A LOT
THAT THAT ELEMENT IS THIS ISO ABBREVIATION RIGHT HERE.
AND IF YOU DIDN'T KNOW THAT, THERE ARE A COUPLE
WAYS YOU COULD FIGURE THIS OUT.
THE FIRST IS TO LOCK AT THE CITATION BACK IN THE
ABSTRACT VIEW OR SUMMARY VIEW AND SAY OKAY, WELL I
KNOW THIS IS THAT TITLE ABBREVIATION I WANT.
WHERE IS THAT LISTED?
WHERE IS THAT VALUE LISTED IN THE XML DATA?
ANOTHER OPTION WOULD BE TO USE SOME OF THAT PUB
MED DOCUMENTATION -- THAT XML WE WERE SHOWING
YOU EARLIER.
SO WHILE I THINK THAT THIS IS PROBABLY THE ONLY
TIME THAT THIS OCCURS IN THE XML, I CAN ALSO USE
THAT PARENT-CHILD CONSTRUCTION JUST TO PLAY
IT SAFE.
SO IF I DID THAT, I WILL PUT JOURNAL SLASH
ISO ABBREVIATION.
AND I CAN SEE HERE WE'VE GOT THAT ARTICLE TITLE
ELEMENT AND THAT WILL BE OUR THIRD COLUMN.
SO I AM GOING TO ADD THAT INTO CYGWIN HERE.
SO I SAID JOURNAL SLASH ISO ABBREVIATION.
AND THEN I WANTED TO ADD IN THAT LAST COLUMN FOR
ARTICLE TITLE.
SO WE'VE GOT OUR WHOLE EXTRACT COMMAND HERE.
SO I AM GOING TO HIT ENTER AND LET THAT RUN.
AND SO NOW YOU CAN SEE WE'VE GOT THE RESULTS AND
WE'VE GOT THAT TABULAR LIST THAT WE WERE LOOKING
FOR.
IT'S A LITTLE BIT MESSY, AND AS YOU CAN SEE, THAT
FIRST ARTICLE TITLE IS REALLY WRAPPING AROUND
AGAIN.
SAME WITH THAT LAST ONE, MAKING IT LOOK A LITTLE
BIT NOT QUITE AS NICE AS WE'D PREFER, BUT WHAT WE
HAVE, THOUGH, IS STILL THAT TABULAR LIST.
WE'VE GOT THAT FIRST COLUMN WITH THE PM ID,
AND WE'VE GOT A SECOND COLUMN WITH THE JOURNAL
TA, FOLLOWED BY THE ARTICLE TITLE.
ARE THERE ANY QUESTIONS SO FAR?
I AM NOT SEEING ANY IN THE CHAT RIGHT NOW.
BUT SO ONCE AGAIN, IF YOU HAVE QUESTIONS, STICK
THEM IN THERE, AND WE WILL CONTINUE TO MONITOR THAT.
ALL RIGHT.
SO LET ME HEAD -- OOP.
ARE THESE SCRIPTS CASE-SENSITIVE?
THEY ARE.
EVERYTHING IS VERY, VERY SENSITIVE.
SO ACTUALLY, YOU MAY HAVE HEARD ME SAY JUST A
SECOND AGO WATCH OUT WHEN YOU TYPE PUB MED, BECAUSE
TYPICALLY WHEN WE TYPE "PUB MED," IT'S GOT A
CAPITAL P AND A CAPITAL M.
BUT HERE, YOU HAVE TO DO THE LOWER CASE M.
AS YOU CAN SEE RIGHT HERE.
AND THAT'S BECAUSE THAT'S THE NAME OF THE ELEMENT,
AND IF YOU DO IT -- SO FOR INSTANCE, ACTUALLY,
LET ME JUST FOR AN EXAMPLE.
LET ME RUN THIS WHOLE EFETCH ONE MORE TIME AND
JUST PASTE IT IN.
LET ME SCROLL ALL THE WAY BACK TO THE PATTERN AND
JUST SHOW YOU FOR A SECOND WHAT WOULD HAPPEN
IF YOU DO JUST MISS ONE CASE OR ONE LETTER.
THEN HIT ENTER AND RUN THIS, NOTHING'S GOING TO
HAPPEN.
SO YES, THIS IS VERY SENSITIVE.
SO AND OFTENTIMES IT DOESN'T ACTUALLY --
DOESN'T TELL WHAT YOU WENT WRONG.
SO FORTUNATELY, I KNOW WHAT IT WAS THAT I JUST
CHANGED.
BUT YES, EVERYTHING IS VERY SENSITIVE, AS KATE
POINTS OUT, SPACE-SENSITIVE TOO.
IF YOU FORGET A SPACE, SOMETHING UP WILL POP UP
OR YOU WILL GET WEIRD ERROR MESSAGES AND IT'S
FUN TRYING TO DETERMINE WHY DID I GET THAT ERROR?
THAT'S PART OF THE FUN OF UNIX.
IT'S AN ADVENTURE.
ALL RIGHT.
ARE THERE ANY OTHER QUESTIONS?
ALL RIGHT.
SO LET ME HEAD BACK TO MY SLIDES.
SO AS WE SAID EARLIER, XML ELEMENTS CAN ALSO
HAVE ATTRIBUTES.
AND IN THE PUB MED XML, THOSE ATTRIBUTES CAN
CARRY SOME IMPORTANT INFORMATION.
YOU GET THESE ATTRIBUTES BY USING THE@SIGN.
WE'VE GOT AN EXAMPLE WHERE WE SAY WE WANT
EXTRACT JUST THAT INFORMATION FROM THE
ATTRIBUTE, MAJOR TOPIC YN AND THAT MAJOR TOPIC IS
THE ATTRIBUTE OF THE ELEMENT DESCRIPTOR NAME.
WHAT I THINK IS CONNECT THAT ELEMENT
NAME@ATTRIBUTE.
SO THAT'S HOW I TEND TO THINK OF IT JUST FOR ME.
LET'S SEE.
ALL RIGHT.
SO THERE IS A QUESTION ABOUT HOW DO I TRANSFER
DATA TO AN XML TABLE?
WE'RE ACTUALLY GOING TO BE TALKING MORE ABOUT
DIFFERENT FILE FORMATS AND EVERYTHING LIKE THAT
IN A LATER CLASS.
SO I AM GOING TO ASK YOU TO SORT OF PONDER THAT
FOR RIGHT NOW AND THEN WE WILL ADDRESS THAT IN
LATER CLASSES, I PROMISE.
AND THEN NATHAN IS SAYING HE'S GETTING AN ERROR
WITH EXTRACT.
I AM GOING TO POINT YOU RIGHT TO MIKE, BECAUSE HE
IS THE EXTRACT AND EDIRECT WIZARD, HELP WITH
ALL THE TECHNICAL ISSUES SEW IS RESPONDING TO YOU
RIGHT NOW, I BELIEVE.
ALL RIGHT.
SO LET'S HEAD INTO EXERCISE NUMBER TWO.
SO I'D LIKE YOU GUYS TO WRITE AN EXTRACT COMMAND
THAT HAS ONE ROW PER PUB MED RECORD IN THE DATA
SET AND INCLUDES THE PM ID, THE JOURNAL ISSN, AND
THE CITATION STATUS.
SO ONCE AGAIN, JUST USE THAT SAME EFETCH THAT
WE'VE BEEN USING WITH THOSE THREE PM IDS, THE
SOLUTIONS ARE AT THE BOTTOM OF THE HANDOUT,
AND WE'RE GOING TO PUT SOME MORE HINT IN THE
CHAT BOX AS WE GO.
SO ONCE AGAIN, IF YOU DON'T WANT TO USE THOSE,
JUST IGNORE THE CHAT BOX FOR ANOTHER MINUTE OR TWO.
AND THEN GIVE ME A GREEN CHECK MARK WHEN YOU ARE
FINISHED.
EVEN IF YOU HAVE USED THE SOLUTIONS AT THE BOTTOM,
JUST SO THE WE KNOW WHERE EVERYBODY IS.
SO I'LL GIVE YOU GUYS A COUPLE MINUTES TO WORK ON
THAT.
ALL RIGHT.
SO LET'S TAKE A LOOK AT THIS.
AND JACK HAD A QUESTION SAYING THAT HE USED
MEDLINE CITATIONS SLASH ARTICLE SLASH JOURNAL
SLASH ISSN IT DIDN'T WORK.
THAT'S BECAUSE WHEN YOU ARE TRYING TO USE THAT
PARENT-CHILD CONSTRUCTION, IT'S JUST
THE SINGLE PARENT AND SINGLE CHILD.
SO YOU ARE GOING TO ONLY USE TWO.
SO FOR THIS YOU WOULD USE THAT JOURNAL/ISSN.
AND I'LL DEMO THAT IN JUST A SECOND.
AND ALL RIGHT SO LET'S JUMP INTO CYGWIN AND TAKE
A LOOK AT THIS.
SO I AM GOING TO USE THAT SAME PATTERN -- OR NOT
SAME PATTERN -- EXCUSE ME.
THAT SAME EFETCH THAT WE'VE BEEN USING.
ACTUALLY FIRST WHAT I AM GOING TO DO IS SIMPLY
CLEAR MY SCREEN JUST SO YOU GUYS CAN SEE
EVERYTHING FROM THE BEGINNING.
SO I'VE GOT MY EFETCH.
AND THEN I AM GOING TO PIPE THIS INTO MY
EXTRACT.
NOW I HAD SAID THAT I WANTED ONE ROW PER PUB
MED RECORD, SO THAT'S GOING TO TELL ME THAT I
WANT TO USE PATTERN PUB MED ARTICLE.
EXCUSE ME.
AND THEN I SAID THAT I WANTED THREE COLUMNS.
SO THAT'S GOING TO SPECIFY MY ELEMENTS.
SO ELEMENT.
AND THEN SAID PM ID WAS GOING TO BE THE FIRST
ONE.
SO IF I TAKE A LOOK AT THE XML, SO WHAT WE'RE
GOING TO DO IS WE'RE GOING TO USE THAT
PARENT-CHILD CONSTRUCTION TO SAY TO SPECIFY I WANT
THIS ELEMENT THAT IS THE CHILD OF A PARENT
ELEMENT.
SO THAT'S GOING TO BE THE DIFFERENCE BETWEEN THE
QUESTION THAT BRIAN JUST ASKED ABOUT MEDLINE
CITATION SLASH PM ID AND@STATUS.
SO ONE OF THOSE IS CONNECTING TWO ELEMENTS
-- THE PARENT AND THE CHILD ELEMENTS.
AND THEN THE OTHER ONE, MEDLINE CITATION STATUS
IS GOING TO CONNECT THE ELEMENT AND THAT SPECIFIC
ATTRIBUTE.
SO WE'RE CONNECTING TWO DIFFERENT SETS OF THINGS
IN THAT.
SO WITH MEDLINE CITATION, SLASH PM ID WE'RE SAYING
WE WANT JUST THIS PM ID ELEMENT.
WE WANT THE VALUE OF THAT, JUST THE ONE THAT
IS THE CHILD OF THE MEDLINE CITATION.
SO WE'RE GOING TO ADD THAT INTO MY EXTRACT.
AND THEN WE SAID THAT WE WANTED JOURNAL ISSN.
SO I DID SORT OF ALLUDE TO THIS A MINUTE AGO WHEN
I WAS ANSWERING JACK.
BUT WHAT I AM GOING TO DO IS I AM GOING TO LOOK FOR
THE ISSN IN HERE AND THAT'S GOING TO BE UNDER
JOURNAL.
SO WHILE I THINK THAT ISSN'S PROBABLY GOING TO
BE THE ONLY ISSN IN HERE, WHAT I CAN DO IS DO THAT
PARENT-CHILD CONSTRUCTION.
AND LIKE I SAID, YOU ARE GOING TO USE JUST THE ONE
PARENT AND CHILD.
SO THE JOURNAL SLASH ISSN AND THEN THAT WILL GIVE
ME THE ISSN VALUE.
THAT WILL BE WHAT IS OUTPUT.
SO LET'S ADD THAT INTO CYGWIN.
AND THEN OUR LAST ELEMENT, THAT LAST
COLUMN THAT WE WANT IS THAT CITATION STATUS.
SO AS I SAID, WHAT WE'RE TRYING TO DO IS TRYING TO
-- WE'RE GOING TO LOOK JUST AT THE ONE ELEMENT,
MEDLINE CITATION, AND CONNECT THE ATTRIBUTE OF
THE ELEMENT STATUS.
SO WHAT THAT IS DOING IS THE WAY THAT I TEND TO
THINK OF IT IS JUST SIMPLY FOR THIS INSTANCE
ESPECIALLY IS TAKE THAT ELEMENT.
I AM GOING TO PUT THAT@SIGN WHERE THAT
SPACE IS AND THEN TAKE THAT STATUS ATTRIBUTE.
SO LET'S CREATE THAT IN CYGWIN.
AND THEN @STATUS.
SO WE'VE GOT OUR ENTIRE EXTRACT COMMAND READY TO
GO RIGHT HERE.
SO NOW I AM GOING TO HIT ENTER AND RUN THIS.
ALL RIGHT.
SO I'VE GOT MY THREE COLUMNS THAT I WAS
LOOKING FOR.
THE FIRST ONE IS PM ID.
THE SECOND IS THAT ISSN.
AND THAT THIRD ONE IS THE STATUS OF THE CITATION
STATUS OF THE ARTICLE.
AND FOR THESE THREE, IT'S JUST GOING TO BE MEDLINE.
LET'S TAKE A LOOK AT SOME OF THESE QUESTIONS.
SO LET'S SEE.
LET'S SEE.
WHILE CONSTRUCTING PARENT-CHILD
CONSTRUCTION, I CAN ONLY WRITE ONE PARENT AND
CHILD?
YES.
IF YOU ARE INTERESTED IN THE PARENT-CHILD, LET'S
TAKE A LOOK AT THE PUB MED AT THE XML IN HERE.
LET'S LOOK FOR AN EXAMPLE.
SO I THINK -- LET'S -- LET ME KNOW IF I AM
MISINTERPRETING WHAT YOU ARE SAYING.
YOU ARE SAYING FOR INSTANCE YOU WANTED TO
GET THIS YEAR, WHICH IS THE CHILD OF THE MEDLINE
CITATION.
IS THAT WHAT YOU MEAN BY PARENT-CHILD?
YES.
OKAY.
SO WHAT YOU WOULD DO FOR THAT IS YOU WOULD DO DATE
COMPLETED SLASH YEAR BECAUSE THAT'S ALREADY
GOING TO BE SPECIFIC ENOUGH TO GET TO THAT
SPECIFIC DATE.
AND THEN LET'S SEE.
SO WE ALSO GOT A QUESTION ABOUT SAYING -- SOMEBODY
ADDED IN A FOURTH PM ID TO THIS AND SAID WHY AM I
NOT GETTING ANY RECORDS FOR THIS?
THAT'S ACTUALLY BECAUSE THAT SPECIFIC PM ID WAS A
BOOK, NOT AN ARTICLE.
SO THIS IS GOING TO BE LOOKING JUST AT THE PUB
MED ARTICLES.
SO THIS IS WHAT YOU ARE GOING TO SEE RIGHT HERE
WHEN YOU SEE PUB MED ARTICLE.
SO JUST MAKE SURE YOU ARE LOOKING -- SO IF YOU ARE
ALSO THAT'S ONE THING TO SEE.
IF YOU ARE SAYING WHY AM I NOT GETTING SOMETHING
RIGHT OR WHY IS THIS NOT REALLY WORKING?
ALWAYS LOOK TO MAKE SURE THAT YOU ARE LOOKING AT
WHAT YOU THINK YOU'RE LOOKING AT.
CAUSE THINGS LIKE BOOK CITATIONS AND ARTICLES
CITATIONS CAN LOOK VERY SIMILAR IN PUB MED.
SO ANOTHER COUPLE QUESTIONS.
CAN WE USE THE SAME CONTENT TO GET ANY OF THE
ATTRIBUTES IN XML?
NOT QUITE.
SO AMY IS SAYING CAN I USE -- LET'S SEE WHERE IS
THAT?
I KNOW IT'S IN HERE SOMEWHERE.
OH, HERE WE GO.
SO IT'S SOMETIMES A STRUGGLE TO FIND THINGS
IN THE XML WHEN YOU'RE ACTUALLY LOOKING FOR IT
ON THE SPOT.
AMY HAD ASKED COULD I USE THE SAME CONTENT TO GET,
SAY, ELECTRONIC TO BE THE OUTPUT HERE IF I USED
ISSN?
SO FOR THIS WHAT YOU WOULD DO IS YOU WOULD
CONNECT THE ATTRIBUTE THAT IS RIGHT HERE.
BECAUSE THE ISSN TYPE IS THE ATTRIBUTE THAT IS
APPLIED TO THE ELEMENT ISSN.
SO YOU'D CONNECT TO THIS -- SO THE WAY TO GET THE
ELECTRONIC OUTPUT IS TO DO ISSN@ISSN TYPE.
DOES THAT ANSWER YOUR QUESTION, AMY?
LET'S SEE.
THEN -- SO WHEN YOU DO PARENT-CHILD, DOES IT
ONLY GIVE THE FIRST RESULT?
WHAT IF THERE ARE MORE OF THE SAME PARENT-CHILD
LATER IN THE RECORD?
SO THAT'S GOING TO BE SOMETHING WE WERE GOING
TO TALK ABOUT LATER IN THE CLASS ACTUALLY, NOT
IN THIS CLASS BUT LATER IN THE COURSE AS A WHOLE.
SO I AM GOING TO SAY MAYBE HOLD ON TO THAT ONE
FOR NOW AND WE'RE GOING TO TOUCH ON THAT LATER ON.
OKAY.
GREAT.
ANY OTHER QUESTIONS?
ALL RIGHT.
SO I AM NOT SEEING ANY.
BUT LIKE I SAID LIKE I SAID, PUT THEM IN THE
CHAT BOX.
WE'RE MORE THAN HAPPY TO GO OVER ALL OF YOUR
QUESTIONS.
OKAY.
SO NEXT UP IS ANOTHER EXERCISE.
SO WE'RE GOING TO DO A LITTLE BIT MORE OF THIS
NEXT WEEK.
BUT I WANT YOU GUYS TO TAKE A SHOT AT BUILDING
AN ENTIRE EDIRECT SCRIPT FROM BEGINNING TO END.
AND IT'S NOT QUITE AS SCARY AS I MADE IT SOUND,
I PROMISE.
SO WHAT WE WANT IS TO FIND OUT WHICH AUTHORS
HAVE BEEN WRITING ABOUT TRAUMATIC BRAIN INJURY IN
ATHLETES.
AND THEN WE ONLY WANT THE MOST CURRENT RESEARCH.
SO JUST 2016 AND 2017.
AND WE ONLY WANT THE AUTHOR NAMES AND WE'D
LYING TO SEE ONE AUTHOR NAME APPEAR ON EACH LINE.
AND THEN WITH THAT, WE WANT LAST NAME AND
INITIALS.
WE WANT THE WHOLE SCRIPT, NOT JUST THAT EXTRACT
COMMAND.
SO MAKE SURE YOU CAN WRITE THE ENTIRE THING.
AND MAKE SURE THAT IT WORKS HOPEFULLY.
SO I AM GOING TO GIVE YOU ADVICE THAT I ALWAYS TRY
TO USE WHILE BUILDING EDIRECT SCRIPTS, WHICH IS
TO TRY AND BUILD ONE PART AT A TIME.
START SMALL AND MAKE SURE THAT THAT'S WORKING
BEFORE YOU MOVE ON TO THE NEXT ONE.
SO I AM GOING TO GIVE YOU GUYS A COUPLE MINUTES TO
WORK ON THIS.
GIVE ME A GREEN CHECK MARK WHEN YOU ARE
FINISHED.
ONCE AGAIN, WE'RE GOING TO BE PUTTING HINTS INTO
THE CHAT BOX.
SO GO AHEAD, TEST IT OUT AND TRY IT OUT AND LET ME
KNOW IF YOU'RE GETTING IT RIGHT.
>> OKAY.
SO I KNOW THAT A LOT OF YOU GUYS ARE STILL
WORKING ON THIS BUT JUST SO WE'RE NOT GOING TO GET
THROUGH CONTENT OF THIS.
ONCE AGAIN, ALL OF THE SOLUTIONS ARE AT THE VERY
BOTTOM OF THE HANDOUT.
IF YOU AREN'T ABLE TO GET THROUGH THIS RIGHT NOW, I
WOULD REALLY RECOMMEND GOING THROUGH AND LOOKING
AT THIS TONIGHT OR SOMETIME BETWEEN,000 AND
THE NEXT CLASS JUST BECAUSE IT'S REALLY
HELPFUL TO SORT OF HAVE THIS UNDERSTANDING OF
PUTTING IT ALL TOGETHER BEFORE YOU MOVE ON TO THE
NEXT CLASS.
SO LET'S TAKE A LOOK AT THIS.
SO I AM GOING TO GO BACK INTO CYGWIN TO START
WITH, GOING TO CLEAR MY SCREEN.
AND THEN I SAID THAT I WANTED TO LOOK AT WHICH
AUTHORS HAVE BEEN WRITING ABOUT TRAUMATIC BRAIN
INJURY IN ATHLETES.
SO TO ME THAT WAYS I WANT TO DO A SEARCH.
SO TO DO THAT I AM GOING TO START WITH AN ESEARCH.
AND SPECIFY MY DATABASE IS PUB MED.
AND I AM GOING TO PUT IN MY QUERY AS JUST PRETTY
SIMPLE "TRAUMATIC BRAIN" -- I DON'T THINK I
SPELLED IT CORRECTLY "ATHLETES".
AND I AM JUST KEEPING MY QUERY VERY SIMPLE.
YOU GUYS COULD BE GETTING VERY FANCY WITH YOUR
SEARCHES.
I AM GOING TO KEEP IT SIMPLE TODAY.
AND I SAID I WANTED TO LIMIT PUBLICATIONS FROM
2016 TO 2017.
SO TO DO THAT I AM GOING TO USE MY DATE TYPE
ARGUMENT AND THEN I AM GOING TO SAY MY MEND DATE
IS 2016 AND MY MAX DATE IS 2017.
MAX DATE IS 2017.
SO NOW JUST TO MAKE SURE THAT THIS SEARCH IS
ACTUALLY GOING TO WORK AND THAT I AM GOING TO
GET RESULTS, ILL GOING TO HIT ENTER TO RUN MY
ESEARCH.
SO I CAN SEE THAT I GOT 425 RESULTS HERE, WHICH
IS GREAT.
THAT TELLS ME IT'S WORKING THE EXACT WAY
THAT I WANT TO DO IT.
BUT FOR ME PERSONALLY, 425 RESULTS IS A LITTLE
BIT OF A BIG SET FOR ME TO WORKING ON MY EXTRACT.
SO WHAT I AM GOING TO DO IS I AM GOING TO TAKE
THIS ESEARCH AND PIPE IT INTO AN EFETCH TO GET A
LIST OF PM IDS SO I CAN CREATE MY SAMPLE SET.
SO I AM GOING TO TAKE THAT COMMAND THAT I JUST
RAN AND PIPE THAT INTO MY EFETCH.
AND EFETCH BASED ON FORMAT U ID AND THIS IS
GOING TO GIVE ME A LONG LIST OF PM IDS.
BUT EXCEPT THAT I MISTYPED SOMETHING.
ALL RIGHT.
I AM GOING TO USE THAT CLEAR TO GET BACK OUT OF
THAT OR CANCEL.
I AM GOING TO GO BACK AND SEE WHAT -- OH, I SEE
WHAT I DID.
I ADDED IN A DASH IN FRONT OF UID, AND THAT
NEGATED THE PM ID PART AND SAID GAVE YOU 400
CITATIONS IN ABSTRACT FORMAT, WHICH IS NOT
QUITE WHAT I WAS LOOKING FOR.
SO I AM GOING TO ENTER AND RUN THIS AGAIN.
THAT'S WHAT I WAS LOOKING FOR.
A GIANT LIST.
EVEN IN THE LIVE DEMO, WE'RE GOING TO MAKE
MISTAKES AND PUT DASHES WHERE THERE SHOULDN'T BE
THERE.
JUST ALL PART OF EDIRECT AND LEARNING TO LIVE WITH
IT, UNFORTUNATELY.
TYPOS.
SO I AM NOW GOING TO CREATE THE SAMPLE SET.
I AM GOING TO LEAVE THIS HERE AND TYPE IN MY
COMMAND RIGHT AT THE VERY BOTTOM.
BUT I WILL GO BACK TO AN EASIER WAY TO VIEW THIS
IN JUST A SECOND, I PROMISE.
GOING TO CREATE MY EFETCH, DATABASE PUB MED.
AND I AM GOING TO PUT IN MY ID.
I AM GOING TO GRAB A COUPLE FROM HERE.
IT DOESN'T MATTER WHICH ONES I AM LOOKING FOR AND
ALSO CTRL C IS NOT GOING TO COPY IN EDIRECT.
I AM FULL OF TYPOS TODAY.
ALL RIGHT.
THE BENEFITS OF A LIVE DEMO, RIGHT?
GOING TO PASTE THAT WITH THE RIGHT COMMANDS THIS
TIME.
I'VE GOT ANOTHER ONE.
PASTE THAT IN.
AND I AM GOING TO GRAB JUST ONE MORE JUST FOR
PROSTERITY.
IT DOESN'T MATTER WHICH ONES I AM USING.
I WANT TO GET A SMALL SET JUST SO THAT I AM NOT
WORKING WITH EVERY COMMAND THAT I RUN, 425.
SO I AM GOING TO GO HIT ENTER AND RUN THIS.
AND SO NOW I'VE GOT THIS LOVELY XML THAT WE'VE
BEEN USED TO SEEING ALL DAY.
SO I AM GOING TO CLEAR MY SCREEN SO THAT I CAN
START WORKING BACK AT THE TOP BECAUSE IT'S A LITTLE
EASIER TO READ.
I AM GOING TO USE MY UP ARROW TO GET MY SAMPLE SET.
SO FROM HERE I SAID I WANTED TO EXTRACT INFORMATION.
SO I AM GOING TO PUT IN MY EXTRACT AND I AM GOING
TO PICK MY PATTERN.
SO I SAID THAT I WANTED TO SEE JUST THE AUTHOR
NAMES AND JUST ONE AUTHOR PER LINE.
SO WHAT THAT IS TELLING ME, THE ONE PER LINE IS
THAT I AM GOING TO USE A DIFFERENT PATTERN THAN
NORMAL.
I WANT TO USE PATTERN AUTHOR.
SO I AM GOING TO PUT THAT IN.
AND THEN THE NEXT THING I SAID I WANTED TO LOOK FOR
WAS THE LAST NAME AND INITIALS.
SO IF I WENT INTO THE XML, SO FOR INSTANCE, IN
PUB MED AND THIS IS JUST OLD SAMPLE DATA BUT THIS
IS JUST GOING TO TELL ME WHAT TO LOOK FOR.
SO I CAN SEE THAT THOSE ELEMENTS ARE LAST NAME
AND INITIALS.
SO I CAN SIMPLY ADD THOSE INTO MY SEARCH.
INTO MY COMMAND.
SO I AM GOING TO ADD IN LAST NAME.
AND INITIALS.
AND I THINK THAT IS ALL THAT WE ARE LOOKING FOR, RIGHT?
SO WE WANT ONE AUTHOR NAME PER LINE AND LAST
NAME AND INITIALS.
SO I AM GOING TO HIT ENTER AND LET THIS RUN.
SO NOW YOU CAN SEE THAT IS EXACTLY HA WE GOT.
WE GOT A SMALL LIST OF LAST NAMES AND AUTHORS
AND THEIR INITIALS.
SO I KNOW THAT THIS EXTRACT IS GOING TO WORK.
SO NOW THAT I KNOW THIS, WHAT I CAN DO IS I CAN
COMBINE IT WITH MY ORIGINAL ESEARCH.
SO I AM GOING TO CLEAR MY SCREEN AND THEN I AM
GOING TO SIMPLY PASTE IN MY COMPLETED CODE.
THERE WE GO.
SO NOW YOU CAN SEE I AM CREATING MY ESEARCH AT
THE TOP.
I'VE GOT MY QUERY, MY DATE TYPE, PARAMETERS,
AND THEN I AM PUTTING THAT INTO THE EFETCH TO
GET IT INTO XML.
AND THEN FROM THERE, I AM ABLE TO ADD IN MY EXTRACT
TO EXTRACT THE DATA THAT I AM LOOKING FOR, WHICH
IS ONE AUTHOR PER LINE, WITH LAST NAME AND
INITIALS.
SO I AM GOING TO HIT ENTER AND LET THAT RUN.
SO BECAUSE IT IS GOING THROUGH 425 RESULTS, IT'S
GOING TO TAKE A SECOND OR TWO TO RUN.
SO LET'S SEE WHEN THIS FINISHES UP.
SO THERE WE GO.
SO NOW WE HAVE OUR RESULTS, WHICH IS GREAT.
SO I AM GOING TO -- SO IF YOU GUYS HAVE NOT
FINISHED WORKING ON IT, I AM GOING TO ASK THAT YOU
SET IT ASIDE JUST FOR NOW SO I CAN SHOW YOU TWO
USEFUL COMMANDS.
LIKE I SAID, THE SOLUTIONS AT THE BOTTOM
AS WELL OF THE HANDOUT AND IF YOU WEREN'T ABLE
TO GET THIS TODAY, I WOULD RECOMMEND JUST TRY
IT EITHER TONIGHT OR JUST SOME TIME BEFORE THE NEXT
CLASS.
BECAUSE IN OUR NEXT CLASS, WE'RE GOING TO
TALK A LITTLE BIT MORE ABOUT DIFFERENT WAYS OF
FINE-TUNING YOUR OUTPUT FORMAT TO MAKE YOUR
RESULTS AS CLEAN AND USEFUL AS POSSIBLE.
SO BEFORE WE GO, THOUGH, I WANT TO SHOW YOU TWO
COMMANDS THAT MIGHT HELP YOU OUT A BIT IN THAT
REGARD.
SO RIGHT NOW THIS IS JUST A LIST OF AUTHORS.
AND THIS IS PROBABLY A DUPLICATE, ESPECIALLY IF
AUTHORS ARE PUBLISHING MULTIPLE ARTICLES ON THE
SAME TOPIC.
SO IF FOR INSTANCE, I WANTED TO SEE THE MOST
PROLIFIC AUTHORS ON A GIVEN TOPIC, WE CAN USE
ANOTHER HANDY EDIRECT COMMAND CALLED SORT
UNIQUE COUNT RANK.
SO I AM GOING TO CLEAR MY SCREEN AND THEN IF I
PASTE IN THAT SAME COMMAND AS BEFORE, I CAN
PIPE THIS INTO THAT COMMAND THAT I JUST
MENTIONED -- SORT, UNIQUE, COUNT, RANK.
SO WHAT THIS IS GOING TO DO IS IT WILL SHOW US
EVERY UNIQUE VALUE FROM OUR LIST OF AUTHORS.
ALONG WITH HOW MANY TIMES EACH OF THOSE AUTHOR
NAMES APPEARED AND SORTED BY FREQUENCY.
SO I AM GOING TO HIT ENTER TO LET THIS RUN.
AND SO IN A SECOND WHEN IT DOES SO -- AND LIKE I
SAID THIS CAN TAKE A MINUTE OR TWO FOR
EVERYTHING TO RUN.
SO NOW YOU CAN SEE WE'VE GOT THIS LIST.
WE'VE GT A NUMBER AND A NAME.
SO LET'S SCROLL UP TO THE TOP OF THIS.
THERE WE GO.
YOU CAN SEE THAT THIS HAS HELPED US TO QUANTIFY
OUR RESULTS.
SHOWING YOU HOW FREQUENTLY EACH AUTHOR
NAME OCCURS IN OUR RESULTS.
BUT EVEN AFTER THIS LIST HAS BEEN DEDUPED AND
SORTED, YOU CAN SEE AT THE BOTTOM WE STILL
HAD A LOT OF AUTHORS WHO'VE ONLY
WRITTEN ONCE OR TWICE ABOUT THIS TOPIC.
AND WHILE I'M SURE THEIR ARTICLES ARE ABSOLUTELY GREAT,
THAT'S NOT QUITE WHAT WE WERE
LOOKING FOR WHEN WE SAID WE WANTED THE MOST
PROLIFIC AUTHORS ON THIS TOPIC.
SO I AM GOING TO SHOW YOU ONE MORE COMMAND THAT
SHOULD HELP WITH THIS A LITTLE BIT.
ALL RIGHT.
SO I AM GOING TO CLEAR MY SCREEN.
AND I AM GOING TO PASTE IN A NEW COMPLETED SCRIPT.
SO YOU CAN SEE WE'VE GOT THAT SAME EFETCH.
WE PUT IT INTO COUNT, UNIQUE RANK AND ADDED
THIS NEW COMMAND CALLED HEAD.
WHICH MEANS WE JUST WANT TO SEE THE TOP 10 ROWS OF
THAT OUTPUT THAT WE'RE GETTING FROM THAT UNIQUE
COUNT RANK.
SO THIS IS GOING TO SHOW US THE TOP 10 MOST
FREQUENTLY OCCURRING AUTHORS.
SO AS YOU CAN SEE HERE, NOW WE'VE GOT A WILL THE
OF 10 NAMES.
THERE ARE THE AUTHORS WHO HAVE -- APPEARED MOST
FREQUENTLY IN THIS QUERY THAT WE DID IN OUR
ESEARCH TO BEGIN WITH.
ALL RIGHT, SO LET ME SEE IF WE'VE GOT ANY
QUESTIONS ABOUT THIS.
SO SORT UNIQUE COUNT RANK IS GOING TO HELP TO YOU
GET RID OF THOSE DUPLICATES BECAUSE IT
WILL SAY WELL, EVERY TIME THIS NAME APPEARS, IT
WILL JUST SORT OF GIVE A TALLY MARK.
AND ONCE IT GETS THROUGH THAT WHOLE LIST, IT WILL
SAY HOW MANY OF THOSE TALLIES WERE THERE.
FOR INSTANCE, IN THIS LIST HERE, YOU CAN SEE
THAT IVERSON APPEARED 18 TIMES IN THAT INITIAL
GIANT LIST OF AUTHORS.
SO WHAT IT DID IS SAID EVERY TIME I COUNT, IT
SAW THAT NAME IT SAID OKAY, THERE IS ONE MORE
INSTANCE OF IT.
AND THEN IT COMBINED THAT INTO THAT ONE LIST WHERE
WE SAID EACH AUTHOR HAS APPEARED X NUMBER OF
TIMES IN THIS LIST.
SO THAT'S THE GREAT WAY TO FILTER OUT THOSE
DUPLICATE AUTHORS.
ALL RIGHT.
SO I THINK THAT'S ALL THAT I AM SEEING FOR
QUESTIONS.
SO LET'S HEAD BACK TO MY SLIDES.
SO NEXT WEEK, WE'RE GOING TO CONTINUE OUR
DISCUSSION OF EXTRACT AND TALK ABOUT SOME MORE
ADVANCED ARGUMENTS THAT WILL ALLOW TO YOU FORMAT
THAT DATA THAT YOU HAVE EXTRACTED.
WE'RE ALSO GOING TO COVER WORKING WITH FILES AND
USING CONDITIONAL ARGUMENTS IN YOUR
COMMANDS.
WE'RE ALSO GOING TO GIVE YOU SOME TIPS FOR
DEVELOPING PRACTICAL SOLUTIONS TO REAL-WORLD
PROBLEMS.
SO AS ALWAYS, WE ENCOURAGE YOU TO VISIT
OUR WEBSITE FOR MORE INFORMATION AND
DOCUMENTATION ON EDIRECT, AS WELL AS THE SCRIPTS,
SLIDES AND RECORDING AS SOON AS WE CAN GET THAT UP.
SO WE ARE ALWAYS GOING TO SUGGEST TO YOU TO JOIN
THAT UTILITIES ANNOUNCEMENT MAILING LIST.
IT'S A GREAT WAY TO GET INFORMATION ABOUT WHAT'S
GOING ON WITH UTILITIES AND EDIRECT.
AND FROM THAT SAME -- YOU CAN GET TO THAT FROM THE
CONTACT BUTTON.
THE LINK IS HERE ALSO AND YOU CAN ALSO SEND US ANY
QUESTIONS THAT YOU HAVE IN THE MEANTIME.
SO AT THE BOTTOM OF THE HANDOUT FROM TODAY, WE
HAVE THE EXERCISES FOR YOUR HOMEWORK.
THOSE ARE -- WE'RE NOT GOING TO CHECK THOSE.
IT'S JUST AN ON AN HONOR SYSTEM TO HELP YOU REALLY
APPLY WHAT IT IS THAT YOU'VE BEEN LEARNING.
THEN ANOTHER THING WE'D LIKE YOU GUYS TO DO IS TO
SEND US OUR CASE STUDIES.
SO WE HAVE AN E-MAIL ADDRESS HERE
NLMTRAINERS@NIH.GOV.
IT'S GOING TO GO DIRECTLY TO US AND SEND US
EXAMPLES OF THINGS WHERE YOU THINK YOU KNOW, MAYBE
EDIRECT COULD WORK ON THIS.
MAYBE THAT'S SOMETHING THAT COULD HELP ME WITH
THIS REALLY, WEIRD COMPLICATED SEARCH THAT I
AM DOING IN PUB MED AND TRYING TO GET CERTAIN
INFORMATION OUT.
SO WE'RE LOOKING FOR ANYTHING THAT YOU THINK
WELL, MAYBE THIS COULD BE -- I COULD USE EDIRECT ON
THIS.
MAYBE EXTRACT WOULD HELP ME GET THE INFORMATION I
WANT.
SEND THOSE TO US.
WE'RE ALWAYS LOOKING FOR MORE USES FOR EDIRECT STAT.
PROLIFIC AUTHORS THAT WE JUST TALKED ABOUT, CAME
FROM A QUESTION THAT WE GOT FROM A USER.
THEY SAID HOW AM I GOING TO DO THIS?
THEY SAID HEY THAT WORKS REALLY WELL IN EDIRECT.
PLEASE SUBMIT YOUR EXAMPLES AND WE WILL DO
OUR FIFTH CLASS I BELIEVE IN A WEEK AND A HALF, WE
ARE GOING TO GO AHEAD AND STRUCTURE OUR CLASS
AROUND THOSE CASE STUDIES.
AND THEN WITH THAT, I'D BE HAPPY TO TAKE ANY
QUESTIONS THAT YOU HAVE.