WELCOME BACK TO
EDIRECT FOR PUBMED. TODAY IS
PART 5, DEVELOPING AND BUILDING
SCRIPTS. I'M MIKE DAVIDSON, A
LIBRARIAN AT THE NATIONAL
LIBRARY OF MEDICINE AND I'M HERE
WITH KATE MAJEWSKI AND SARAH
HELSON AND PETER SEIBERT. THIS
IS THE FINAL OF FIVE SESSIONS,
SO THANKS FOR STICKING WITH US.
TODAY IS ALL ABOUT PUTTING ALL
THE PIECES WE LOOKED AT SO FAR
TOGETHER AND TALKING ABOUT HOW
WE CAN DEVELOP AND BUILD
SOLUTIONS TO REAL-WORLD
PROBLEMS. AS ALWAYS, WE WILL
START WITH A QUICK RECAP OF LAST
TIME AND SPEND A FEW MINUTES
TALKING ABOUT STRATEGIES,
DISCUSSING HOW WE CAN PULL ALL
THE TOOLS-- PUT ALL THE TOOLS
WE LEARNED ABOUT TO USE AND GO
OVER METHODS FOR DEVELOPING
SOLUTIONS. WE WILL FINISH OFF
WITH A DEEP DIVE INTO REAL-WORLD
CASE STUDIES, INCLUDING SOME
SUGGESTED BY MEMBERS OF THE
CLASS, AND LOOK AT THE WHOLE
PROCESS FROM BEGINNING TO END.
WE WILL START WITH OUR RECAP.
HOPEFULLY YOU REMEMBER LAST
THURSDAY WE SPENT MOST OF THE
CLASS TALKING ABOUT CONDITIONAL
ARGUMENTS, STARTING WITH IF, IF
A PATTERN OR BLOCK MET A CERTAIN
CONDITION. THEN WE DID A BUNCH
OF VARIATIONS ON THAT. WE USED
IF BY ITSELF TO LIMIT BASED ON
THE PRESENCE OF A PARTICULAR
ELEMENT OR ATTRIBUTE, WE USED IF
WITH EQUALS. AND WE ALSO USED
IF WITH CONTAINS TO LIMIT BASED
ON WHETHER AN ELEMENT CONTAINED
A CERTAIN STRING. WE THEN MOVED
ON TO TALK ABOUT THE BOOLEAN
ADDITIONAL ARGUMENTS OR REQUIRES
ONE OF THE CONDITIONS WE SPECIFY
MUST BE TRUE, AND AND, WHICH
REQUIRES BOTH CONDITIONS MUST BE
MET. WE DID A QUICK DISCUSSION
OF POSITION, WHICH LETS US
SPECIFY-- INCLUDE A SPECIFIC
BLOCK BASED ON ITS POSITION IN A
PATTERN, SO WHETHER IT IS THE
FIRST BLOCK, SECOND BLOCK, LAST
BLOCK, ET CETERA. AND WE TALKED
ABOUT ONE ARGUMENTS NOT STRICTLY
SPEAKING CONDITIONAL, THE DEF
ARGUMENT, WHICH ALLOWED US TO
DEFINE A PLACEHOLDER FOR PLANK
CELLS. BEFORE WE MOVE ON, DID
ANYONE HAVE QUESTIONS ABOUT THE
HOMEWORK OR ANYTHING FROM ANY OF
THE PREVIOUS FOUR CLASSES?
LOOKS PRETTY GOOD. IF YOU HAVE
ANY, THROW THEM IN THERE. WE
WILL ADDRESS OTHER QUESTIONS AS
THEY COME UP. LET'S GET GOING
WITH NEW STUFF. OTHER THESE
SESSIONS, WE SPENT A LOT OF TIME
INTRODUCING YOU TO NEW TOOLS AND
TECHNIQUES. SOME OF THE TOOLS
ARE NEW WAYS OF DOING THINGS WE
CAN ALREADY DO IN PUBMED, LIKE
ESEARCH, WHICH SEARCHES A
DATABASE, AND EFETCH, WHICH
RETRIEVES FULL RECORDS IN XML.
SOME OF THESE TOOLS ALLOW US TO
DO NEW THINGS WE CAN'T DO IF
PUBMED, LIKE EXTRACT, WHICH LETS
US EXTRACT DATA FROM XML AND
ARRANGE IT IN A TABLE. WE HAVE
ALL THESE TOOLS THAT LET US
ANSWER ADVANCED MUST BE MED
QUESTIONS, BUT HOW DO WE
ACTUALLY DO IT? HOW CAN WE PUT
THESE TOOLS TO USE? LEARNING
HOW TO USE THESE TOOLS IS ONLY
PART OF THE CHALLENGE. YOU ALSO
NEED TO FIGURE OUT WHEN TO USE
WHICH TOOLS AND IN WHAT ORDER.
BASICALLY, WE SORT OF NEED TO
LEARN HOW TO BE SOFTWARE
DEVELOPERS. A LOT OF THIS COMES
THROUGH EXPERIENCE AND
EXPERIMENTATION, HOWEVER, I CAN
OFFER YOU SOME ADVICE BASED ON
THE HOURS AND HOURS I SPENT
TRYING AND OFTEN FAILING TO
BUILD SCRIPTS TO DO SPECIFIC
TASKS. I WILL WALK YOU THROUGH
A PROCESS I USE SOMETIMES, BUT
THESE SUGGESTIONS MAY NOT WORK
FOR EVERYONE OR IN ALL
SITUATIONS, AND THAT'S OKAY. IF
YOU FIND A DIFFERENT WAY OF
THINKING ABOUT THIS PROCESS THAT
WORKS FOR YOU, GO WITH THAT.
THERE'S NO ONE RIGHT ANSWER
HERE. THAT SAID, LET ME TELL
YOU SOME OF THE THINGS I THINK
ABOUT WHEN I AM TRYING TO
DEVELOP A SOLUTION IN EDIRECT.
BEFORE STARTING TO BUILD A
CRYPT, IT CAN BE HELPFUL TO
THINK ABOUT YOUR GOAL, AND MAKE
SURE YOU HAVE A GOOD IDEA OF
WHAT YOU ARE TRYING TO
ACCOMPLISH. THE FIRST THING YOU
WANT TO DO IS IDENTIFY YOUR
INPUT. IN OTHER WORDS, WHAT IS
IT THAT YOU ALREADY KNOW? MAYBE
YOU HAVE A NAME OR SEARCH TERM
OR FULL PUBMED SEARCH STRATEGY,
MAYBE YOU HAVE A LIST OF PMIDS
OR PUBMED RECORDS YOU WANT TO
RETRIEVER AND ANALYZE. MAYBE
YOU ALREADY HAVE A BUNCH OF
PUBMED RECORDS IN XML. WHAT
TYPE OF INPUT YOU HAVE IS GOING
TO DETERMINE HOW YOU BUILD THE
FIRST STEPS OF YOUR SCRIPT.
NEXT, IDENTIFY YOUR GOAL. WHAT
IS IT YOU WANT TO KNOW? DO YOU
WANT JUST LIST OF PMID'S, DO YOU
WANT SPECIFIC DATA WITHIN A
RECORD? WHICH FIELDS
SPECIFICALLY ARE YOU TRYING TO
OUTPUT? WHEN THINKING ABOUT
THAT, YOU CAN ALSO THINK ABOUT
YOUR OUTPUT FORMAT. HOW DO YOU
WANT YOUR DATA ARRANGED? IS RAW
XML GOOD ENOUGH, OR DO YOU WANT
A TABLE INSTEAD? IF YOU WANT A
TABLE, WHAT ORDER SHOULD THE
COLUMNS BE IN? HOW SHOULD THE
COLUMNS BE SEPARATED? DO YOU
WANT YOUR OUTPUT SAVED TO A
FILE? IF SO, WHAT TYPE OF FILE?
THESE DETAILS WILL HELP YOU
POLISH YOUR SCRIPT TO GOAT THE
RESULTS THIS THE EXACT FORMAT
YOU WANT. WE JUST SPENT FOUR
SESSIONS TEACHING YOU EDIRECT,
BUT YOU WANT TO MAKE SURE
EDIRECT IS ACTUALLY THE RIGHT
TOOL FOR THE JOB. I KNOW WHEN I
LEARN ABOUT A NEW TOOL, I AM
TEMPTED TO GO OUT AND LOOK FOR
PROJECTS TO USE IT ON. YOU WANT
TO MAKE SURE THE TOOL SERVES THE
PROJECT. EDIRECT CAN BE USEFUL,
BUT IT ISN'T ALWAYS THE MOST
PRACTICAL SOLUTION TO A PROBLEM.
IF YOU CAN DO THE JOB FASTER BY
JUST GOING TO PUBMED, THEN YOU
MAY WANT TO CONSIDER DOING JUST
THAT. IF YOUR PROJECT REQUIRES
ACCESS TO AN INCREDIBLY LARGE
AMOUNT OF PUBMED DATA, LIKE FOR
EXAMPLE-- YOU MAY FIND EDIRECT
IS NOT THE BEST CHOICE. TO
AVOID OVERLOADING THE SERVERS,
NCBI SET FORTH USAGE GUIDELINES,
WHICH ASK USERS TO LIMIT
HIGH-VOLUME USAGE TO OFF-PEAK
HOURS. YOU CAN FIND OUT MORE ON
NCBI'S WEB SITE. THERE'S A URL
IN YOUR HANDOUT. I SUGGEST YOU
WAIT UNTIL AFTER CLASS TO LOOK
AT THEM. ADDITIONALLY,
INCREDIBLY HIGH-VOLUME SCRIPTS
TAKE A LONG TIME TO RUN, SO IT
MIGHT NOT BE PRACTICAL. IF YOUR
PROJECT REQUIRES ACCESS TO PAUL
OF PUBMED, I WOULD INSTEAD
DIRECT YOU TO THE NLM DATA
DISTRIBUTION PROGRAM, WHICH
PROVIDES BULK DOWNLOADS OF
RECORDS IN XML. THERE'S A URL
IN THE HANDOUT. YOU MAY ALSO
WANT TO REMEMBER THAT XTRACT IS
WORK ON ANY XML USING THE INPUT
ARGUMENT. EVEN IF YOU DON'T GET
YOUR XML FROM EFETCH OR FROM
USING THE PUBMED SEND TO FILE
OPTION TO DOWNLOAD XML FROM THE
WEB VERSION OF PUBMED, YOU STILL
BE ABLE TO USE-- MAY BE ABLE TO
USE XTRACT. YOU WANT TO MAKE
SURE JUST BECAUSE WE ARE USING
AN EXCITING NEW TOOL, YOU DON'T
FORGET OR IGNORE EVERYTHING YOU
KNOW ABOUT PUBMED DATA. THE
SOLUTIONS YOU BUILD WITH EDIRECT
ARE ONLY GOING TO BE AS GOOD AS
THE DATA YOU ARE RETRIEVING AND
AS GOOD AS YOUR KNOWLEDGE OF
THAT DATA. REMEMBERING BACK TO
OUR SECOND SESSION, WHERE SARAH
SHOWED YOU NINE DIFFERENT YEAR
ELEMENTS ON A SINGLE RECORD. IN
ORDER TO MAKE AN INFORMED CHOICE
ABOUT WHICH ONE TO EXTRACT, WE
NEEDED TO KNOW THE DIFFERENCE
BETWEEN THE DIFFERENT DATES
PUBMED USES AND RECOGNIZE THAT
PUB DATE WAS THE CORRECT DATE
FOR THAT SITUATION. ALSO, KEEP
IN MIND WHAT DATA IS AND ISN'T
AVAILABLE IN PUBMED. FOR
EXAMPLE, AS YOU KNOW, PUBMED
DOESN'T HAVE FULL TEXT, ONLY
CITATIONS TO DATA.
ADDITIONALLY, NOT ALL FIELDS ARE
POPULATED FOR ALL RECORDS. MESH
HEADINGS ARE ONLY ATTACHED TO
MEDLINE RECORDS. MEDLINE
RECORDS MAKE UP THE BULK OF
PUBMED, BUT BECAUSE THE INDEXING
PROCESS TAKES TIME, YOU WILL
FIND A LOT OF RECENT
PUBLICATIONS DON'T HAVE MESH
HEADINGS, BECAUSE THEY HAVEN'T
YET BEEN FULLY INDEXED. EVEN
WHEN DATA IS AVAILABLE, YOU WANT
TO THINK ABOUT WHAT THAT DATA
LOOKS LIKE. AUTHOR AFFILIATION
DATA IS SUPPLIED BY THE
PUBLISHER AS FREE TEXT AND IS
NOT CONTROLLED BY NLM. THERE
CAN BE INCONSISTENCIES IN GRANT
IDENTIFIERS SOMETIMES.
UNDERSTANDING THE DATA WILL HELP
YOU FIGURE OUT WHAT IS POSSIBLE
AND HIGHLIGHT ISSUES YOU MAY
NEED TO ADDRESS IN YOUR
DEVELOPMENT. THE BOTTOM LINE,
THE BETTER AT SEARCHING AND
WORKING IN THE WEB VERSION YOU
ARE AND THE BETTER UNDERSTANDS
YOU HAVE OF PUBMED'S UNDERLYING
DATA, THE MORE EFFECTIVE YOUR
EDIRECT SCRIPTS WILL BE. IF YOU
WANT TO IMPROVE YOUR SEARCHING
SKILLS, NLM AND THE NATIONAL
NETWORK OF LIBRARIES OF MEDICINE
OFFER A VARIETY OF TUTORIALS AND
TRAINING OPPORTUNITIES. IF YOU
WANT TO GET MORE FAMILIAR WITH
THE DATA, YOU CAN REMEMBER THE
PUBMED XML DOCUMENTATION SARAH
SHOWED YOU IN THE SECOND
SESSION, AS THAT IS A GOOD PLACE
TO START. WE ARE ALSO TRYING TO
COOK UP NEW DATA TRAINING
CLASSES, CLASSES SORT OF
SPECIFICALLY ABOUT THE PUBMED
DATA ITSELF. THOSE ARE STILL IN
THE WORKS. WE DON'T HAVE
ANYTHING TO ANNOUNCE NOW, BUT
YOU COULD ALSO LOOK IN THE CHAT
BOX FOR THE LINK ON ADDITION
TRAINING OPPORTUNITIES THAT KATE
PUT IN THERE. ONE OF THE BIG
ADVANTAGES OF EDIRECT IS IT LETS
YOU AUTOMATE REPETITIVE AND
TIME-CONSUMING JOBS. HOWEVER,
DEVELOPING AN EDIRECT SCRIPT CAN
ALSO TAKE TIME. AS WE
MENTIONED, YOU WANT TO CONSIDER
WHETHER THE EFFICIENCY YOU GAIN
IS WORTH THE TIME INVESTMENT.
OF COURSE, BY WORKING WITH
EDIRECT AND LEARNING, YOU GET
THE ADDED BENEFIT OF LEARNING
ABOUT HOW IT WORKS AND GETTING
BETTER AT IT, SO YOU SHOULD
FACTOR THAT IN AS WELL. BUT ONE
AREA YOU WANT TO MAKE SURE YOU
CONSIDER HOW MUCH YOUR
INVESTMENT IS WORTH IS WHEN
FIGURING OUT WHEN TO STOP
DEVELOPMENT. AS WE HAVE SAID
MANY TIMES WITH EDIRECT, THERE
ARE ALMOST ALWAYS MULTIPLE WAYS
OF DOING WHAT YOU WANT TO DO,
AND IT-- IN MANY CASES, THERE'S
A WAY TO ACCOMPLISH 100% OF YOUR
GOAL IN A SING SCRIPT. HOWEVER,
THERE ARE USUALLY ALSO WAYS OF
ACCOMPLISHING 90%, 75% OR 50% OF
YOUR GOAL IN A SCRIPT AND DOING
THE REMAINING 10, 25 OR 50%
EITHER MANUALLY OR USING A
DIFFERENT TOOL, LIKE EXCEL. YOU
WILL HAVE TO DECIDE WHETHER THE
ADDITIONAL TIME AND EFFORT IT
WILL TAKE TO GET YOUR SCRIPT
FROM 90% TO 100% IS MORAL
EFFICIENT THAN DOING THE
REMAINING 10% MANUALLY. THIS
ALSO BRINGS UP ANOTHER POINT,
WHICH IS SOMETIMES A SOLUTION
WILL REQUIRE ACTUAL HUMAN
JUDGMENT FOR EVALUATING RESULTS,
DETERMINING RELEVANCE OR DEALING
WITH SOME OF THOSE TRICKY DATA
ISSUES. YOU CAN AUTOMATE THE
REPETITIVE PARTS, WHICH WILL
FREE UP MORE TIME TO PERFORM THE
TASKS THAT ACTUALLY REQUIRE A
HUMAN. FIGURING OUT IF AND
WHERE A HUMAN FITS INTO YOUR
DEVELOPMENT PROCESS SHOULD BE
SOMETHING YOU THINK ABOUT.
BEFORE WE GO ON, I REALLY WANT
TO UNDERLINE THAT 90/10 CONCEPT,
AS SOMETHING I THINK AND TALK
ABOUT A LOT, AS THE OTHER
INSTRUCTORS CAN TELL YOU. ONE
OF THE MAIN POINTS OF AUTOMATION
IS TO SAVE YOU TIME. SO IF YOU
90% SOLUTION FAVORS YOU TIME,
YOU MAY NOT NEED TO GO FURTHER.
BASICALLY, DON'T LET THE PERFECT
BE THE ENEMY OF THE GOOD. WE
HAVE MENTIONED THIS A NUMBER OF
TIMES, BUT IT IS WORTH
REITERATING. YOU WILL BE BETTER
ABLE TO SPOT ISSUES IF YOU
CREATE COMMANDS ONE AT A TIME,
TESTING AS YOU GO. TO SHOW YOU
MORE OF WHAT I MEAN ABOUT THIS,
WE ARE GOING TO SPEND THE REST
OF THIS SESSION LOOKING AT SOME
CASE STUDIES. WE AREN'T GOING
TO BREAK DOWN THE CODE OF THE
PRODUCT. INSTEAD, WE ARE GOING
TO DEVELOP THE SOLUTION FROM THE
GROUND UP, LOOKING AT HOW WE
CHOOSE, BUILD AND TEST EACH
COMMAND. HERE'S OUR FIRST CASE
STUDY, AND THIS IS THE GOAL OF
OUR FIRST STUDY. WE WANT A LIST
OF ARTICLES ABOUT BREAST CANCER
WHAT WERE PUBLISHED IN 2016 AND
THE FIRST HALF OF 2017 IN OUR
LINK TO ENTRIES IN
CLINICALTRIALS.GOV. FOR EACH
ARTICLE, WE NEED THE PMID, THE
NCT NUMBER OR NUMBERS, THE FIRST
AUTHOR AND THE JOURNAL THAT
ARTICLE WAS PUBLISHED IN. THIS
IS A PRETTY STANDARD EDIRECT
CASE. IT'S ASKING US TO GET
INFORMATION IN A CERTAIN FORMAT,
IN A CERTAIN ARRANGEMENT AND
IT'S INFORMATION WE CAN'T EASILY
GET FROM THE WEB VERSION OF
PUBMED. GOING TO OUR FRAMEWORK
I JUST INTRODUCED YOU TO, WE
WILL START BY IDENTIFYING OUR
INPUT. OUR INPUT IS GOING TO BE
A PUBMED SEARCH STRING. WE ARE
GOING TO SEARCH BREAST CANCER
AND CLINICALTRIALS.GOV TAGGED
WITH SI IN BRACKETS. IF YOU ARE
NOT AS FAMILIAR WITH PUBMED
SEARCHING, IT IS SEARCHING FOR
BREAST CANCER AS A SUBJECT, AND
ALSO REQUIRING THAT EVERYTHING
HAS CLINICALTRIALS.GOV IN THE
SUPPLEMENTAL INFORMATION FIELD.
WE ALSO KNOW WE WANT TO LIMIT
OUR RESULTS BY DATE FROM JANUARY
2016 TO JUNE 2017, SO PUTTING
THAT TOGETHER, THAT GIVES US A
STRONG INDICATION WE WILL BE
STARTING THE SCRIPT WITH AN
ESEARCH. LET'S TALK ABOUT OUR
OUTPUT. WE HAD FOUR ELEMENTS WE
WANTED IN OUR OUTPUT. PM ID,
WHICH YOU SHOULD BE FAMILIAR
WITH USING THAT PARENT-CHILD
CONSTRUCTION, NCT NUMBER, WHICH
IS A LITTLE TRICKY. THE XML
ELEMENT IN A PUBMED RECORD THAT
CONTAINS THE NCT NUMBER IS VIA
ACCESSION NUMBER ELEMENT, A
CHILD OF THE DATA BANK ELEMENT,
BUT WE HAVE TO BE CAREFUL. THAT
ACCESSION NUMBER ELEMENT CAN
CONTAIN OTHER INFORMATION. WE
ONLY WANT TO USE THE NCT
NUMBERS, SO WE'LL HAVE TO MAKE
SURE WE ARE ONLY USING ACCESSION
NUMBERS FROM DATA BANKS, WHERE
THE DATA BANK NAME IS
CLINICALTRIALS.GOV. CONTINUING
OUR DISCUSSION OF OUTPUT FOR THE
FIRST AUTHOR, WE WANT TO LAST
NAME AND INITIALS, BUT ONLY FOR
THE FIRST AUTHOR. THIS WOULD BE
A GOOD OPPORTUNITY TO USE THE
POSITION ARGUMENT WE TALKED
ABOUT LAST TIME. THEN THE
JOURNAL TITLE ABBREVIATION,
WHICH WILL BE IN THE ISO
ABBREVIATION FIELD, WHICH WE
ALSO TALKED ABOUT LAST TIME.
LET'S TALK ABOUT OUR FORMAT.
EACH ARTICLE IS GOING TO HAVE
ITS OWN ROW, AND WILL HAVE FOUR
COLUMNS. WE WILL DO PMID FIRST,
THEN JOURNAL, THEN FIRST AUTHOR,
THEN NCT NUMBER. I PUT THE NCT
NUMBER LAST, BECAUSE IT IS
REPEATABLE, AND THERE'S KIND OF
AN ORDER OF DIFFICULTY, SO IT
WILL MAKE IT A NICER
CONVERSATION HERE. WE CAN
SEPARATE OUR COLUMNS BY TABS,
AND WE'LL SEPARATE MULTIPLE NCT
NUMBERS BY PIPES. FINALLY, WE
WANT TO SAVE THIS TO A FILE, SO
WE CAN OPEN IT IN EXCEL AND
READ, VISUALIZE OR ANALYZE THE
DATA MORE EASILY. WE HAVE DONE
SOME GOOD PLANNING. NOW IT'S
TIME TO START BUILDING. WE ARE
GOING TO START BUILDING ONE STEP
AT A TIME WITH THE EASIEST PART
FIRST, WHICH IS THE ESEARCH. AS
YOU MAY REMEMBER FROM PETE'S
CLASS, THE FIRST SESSION,
ESEARCH COMMANDS ARE PRETTY
SIMPLE TO WRITE AND THEY RUN
PRETTY QUICKLY AND THEY ARE EASY
TO EVALUATE IF THEY ARE WORKING
BY COMPARING THEM TO THE PUBMED
VERSION. LET'S START WITH THAT.
WE ALREADY KNOW OUR SEARCH
STRING. LET'S JUST SEARCH THAT
FIRST, GET ABOUT 1200 RECORDS,
AND WE CAN COMPARE AND CONTRAST
WITH PUBMED WITH THAT SAME
SEARCH. SO OUR ESEARCH IS
CORRECT. WELL, THE PART OF IT
WE HAVE DONE SO FAR IS CORRECT.
REMEMBER, WE WANT A DATE
RESTRICTION AS WELL. THIS GOES
BACK TO THE THING THAT PETE
SHOWED YOU, AND WE TALKED ABOUT
A COUPLE OF TIMES. THERE'S
MULTIPLE WAYS TO DO YOUR DATE
RESTRICTION. YOU CAN PUT IT IN
YOUR SEARCH STRING, IF YOU WANT,
BUT I LIKE TO KEEP IT OUT, USING
THE DATE-TYPE AND MIN DATE
ARGUMENTS. I WILL SHORTEN THEM,
TO EXPERIMENT WITH, WITH MY
XTRACT, SO I FIND IT EASIER TO
DO THAT WHEN THEY ARE IN THE
ARGUMENTS SEPARATELY. MAX DATE
IS 2017/6/30. DOWN TO 232
RECORDS. THAT'S A MUCH SMALLER
CHUNK TO WORK WITH. THAT
ESEARCH, AS IT IS NOW, WITH
THOSE DATE RESTRICTIONS IS WHAT
I WILL BE USING EVENTUALLY, BUT
FOR NOW, I WILL MAKE IT EVEN
SMALLER. I WILL TAKE IT JUST
THE FIRST SIX MONTHS OF 2016,
SEE WHAT WE GET FROM THAT.
THAT'S JUST 128 PMIDS, SO I HAVE
A SMALL SET I CAN WORK WITH.
I'M GOING TO TAKE THAT AND PIPE
THAT ENTER AN EFETCH, BUT BEFORE
I GRAB THE XML, BECAUSE 128
RECORDS ARE STILL A BIT MORE
THAN I WANT TO WORK WITH FOR MY
EARLY STAGES OF TESTING. I WILL
USE FORMAT UID TO GRAB A COUPLE
PMIDS. NOW I HAVE A BIG LIST OF
PMID'S AND I CAN CREATE ANOTHER
EFETCH I WILL USE FOR TESTING
BASED ON SOME OF THESE. I WOULD
START WRITING MY EFETCH AND COPY
AND PASTE. I HAVE DONE THAT
ALREADY AHEAD OF TIME, SO I HAVE
AN EFETCH THAT LOOKS LIKE THIS.
FOUND SIX RECORDS. IT IS
MANAGEABLE SET. THERE'S MY SIX
RECORDS. THIS IS THE PROCESS
THAT I WOULD TAKE WHY WOULD
DEVELOPING THIS SOLUTION. I
WOULD START WITH A LARGE SEARCH,
SCALE IT DOWN TO A SMALLER
SEARCH, GRAB A COUPLE PMID'S,
USE THAT AS AN EFETCH, THEN I
HAVE SOMETHING TO WORK WITH TO
BUILD MY XTRACT. WE WILL PIPE
THIS XML RIGHT INTO OUR XTRACT
AND WE WILL START BUILDING WITH
THE EASY SUCH. THE PATTERN IS
GOING TO BE PUBMED ARTICLE. OUR
FIRST TWO ELEMENTS ARE ALSO
PRETTY EASY, AT LEAST AS FAR AS
WE-- AT THIS POINT, THEY SHOULD
BE PRETTY EASY. WE HAVE USED
THESE A COUPLE OF TIMES. IF WE
RUN THAT, WE HAVE TWO COLUMNS,
SO WE ARE HALFWAY THERE. WE
HAVE OUR PMID AND OUR JOURNAL
TITLE ABBREVIATION. I WILL COPY
AND PASTE THAT BACK IN THERE, SO
I HAVE SOMETHING TO WORK WITH
FOR MY NEXT PART, WHICH IS THE
FIRST AUTHOR. REMEMBER, WE
WANTED TO USE THE POSITION
ARGUMENT. IN ORDER TO USE THE
POSITION ARGUMENT, WE NEED TO
USE BLOCK, BECAUSE BLOCK IS
GOING TO CHECK EACH AUTHOR ONE
AT A TIME, BUT ONLY OUTPUT THE
FIRST ONE, BECAUSE WE HAVE USED
THIS POSITION FIRST ARGUMENT.
FOR AUTHOR NAME, I WILL DO WHAT
I LIKE TO DO, SEP SPACE, AND PUT
THE LAST NAME AND INITIALS IN
THE SAME COLUMN, BY USING A
COMMA. I HAVE THREE COLUMNS,
WHICH IS GOOD. IF I WANTED TO,
I COULD TAKE A COUPLE OF THESE
PMID'S AND MANUALLY CHECK THEM
EITHER IN PUBMED OR BY FETCHING
A SINGLE RECORD ONE AT TIME AND
MAKE SURE THE INFORMATION IS
CORRECT, BUT I FEEL CONFIDENT
THIS IS CORRECT AT THIS POINT.
I AM GOING THE CLEAR MY SCREEN
AND WE'LL MOVE ON TO THE
TRICKIER PART. THE NEXT PART,
WE WERE TALKING ABOUT THE NCT
NUMBERS. THIS IS WHERE WE KNOW
THE NCT NUMBERS IN THE ACCESSION
NUMBER ELEMENT, SO WE CAN DO
THAT. WE KNOW IT IS IN THAT
ACCESSION NUMBER ELEMENT IS A
CHILD OF THE DATA BANK ELEMENT.
BUT WE ALSO KNOW WE DON'T WANT
EVERY ACCESSION NUMBER. WE ONLY
WANT THE NCT NUMBERS. SO WE
WILL DO THAT WITH A CONDITION,
JUST LIKE LAST TIME. WE ONLY
WANT THE DATA BANK IF THE DATA
BANK NAME EQUALS
CLINICALTRIALS.GOV, THEN WE WILL
OUTPUT THE ACCESSION NUMBER IF
THAT'S TRUE. IF NOT, WE WILL
SKIP IT AND CHECK THE NEXT ONE.
WE ALSO WANTED TO SEPARATE
MULTIPLE NCT NUMBERS, IF THEY
OCCURRED WITH PIPE. AND
REMEMBER, OUR BLOCK WIPES OUT
OUR PREVIOUS FORMATTING
CONDITIONS, SO WE NEED TO PUT
THAT STEP IN ANOTHER TIME. I
EXECUTE THAT, I GET MY FOUR
COLUMNS. PMID, JOURNAL TITLE
ABBREVIATION, AUTHOR--FIRST
AUTHOR, LAST NAME, INITIALS AND
NUMBERS. NOW WE CAN PUT IT ALL
TOGETHER. WE CAN GRAB OUR
ESEARCH BACK, PUT ALL THESE
PARTS TOGETHER IN ONE BIG
SCRIPT. SO THE SEARCH PIPES
RIGHT INTO THE EFETCH, WHICH
GETS THE XML, WHICH PIPES RIGHT
INTO THAT EXTRACT WE JUST BUILT.
BEFORE I RUN THIS, I WILL
REDIRECT IT TO A FILE. I WANT
TO OPEN THIS IN EXCEL. AND THIS
MIGHT TAKE A MOMENT OR TWO TO
RUN, BECAUSE THERE'S HALF THE
AMOUNT OF DATA IN THERE. NOT
TOO LONG. SO NOW, THREE LET
CLAY, WE SHOULD HAVE OUR DATA,
OUR TABLE IN THIS TEXT FILE, AND
I'M GOING TO GIVE THAT LS
COMMAND THAT KATE SHOWED US ON
SESSION 3 TO CHECK AND MAKE SURE
YES, CLINICAL TRIALS.TXT. THE
FILE IS THERE. NOW LET'S LOOK
AT IT IN EXCEL AND SEE IF WE
CAN-- SEE IF IT IS WHAT WE
NEEDED IT TO BE. I WILL OPEN UP
EXCEL. GOING TO GO FILE, OPEN,
GOING TO FIND MY HOME DIRECTORY.
IT IS NOT SHOWING UP HERE NOW,
BECAUSE I NEED TO SWITCH FROM
ALL EXCEL FILES TO ALL FILES IN
THIS LIT FORMAT MENU. THEN I
CAN FIND IT RIGHT HERE. IT IS
CLINICAL TRIALS.TXT. WHEN I
OPEN THAT, IT IS PROJECTING ME
WITH THIS TEXT IMPORT WIZARD.
IT MIGHT BE A LITTLE SMALL ON
YOUR SCREEN, IF YOU WERE VIEWING
IT AT 50%. I CAN'T SOLVE THAT
WITH THIS WINDOW, BUT I CAN
SOLVE IT WITH OUR OUTPUT. WHEN
YOU ARE OPENING TEXT FILE IN
EXCEL, YOU CLICK NEXT TO THE
SECOND PAGE, CHECK YOUR
DELIMITERS TO MAKE SURE TAB IS
SELECTED AND YOU WILL SEE A
PREVIEW HERE. THEN I CAN CLICK
FINISH, ZOOM IN A LITTLE BIT.
AND NOW WE HAVE OUR RESULTS.
AND WE HAVE PMID, JOURNAL TITLE
ABBREVIATION, FIRST AUTHOR, LAST
NAME AND INITIALS AND NCT
NUMBERS. WE SCROLL DOWN A
COUPLE, WE HAVE ONE THAT HAS
MULTIPLES AND THEY ARE ALL
SEPARATED BY PIPES. WE HAVE
ACCOMPLISHED OUR GOAL. WE HAVE
THE INFORMATION WE NEED, BUT
ONLY THE INFORMATION WE WANTED
IN EXACTLY THE FORMAT THAT WE
WANTED IT IN.
ANY QUESTIONS ABOUT THIS CASE
STUDY OR ANY OF THE STUFF WE
HAVE TALKED ABOUT SO FAR? TAKE
A LOOK AT THE CHAT, GIVE YOU A
MOMENT, IF YOU ARE STILL TYPING.
EITHER YOU ARE ALL WITH ME OR
ALL ASLEEP. EITHER WAY, LET ME
CLOSE THIS. CAN YOU SEE THE
SCRIPT ONE MORE TIME? SURE.
I'M GOING TO CLOSE MY FILE.
THIS SCRIPT IS THE IN YOUR
HANDOUT. THE-- ALL THE
INFORMATION AND ALL THE STEPS
FROM THIS CASE STUDY SHOULD BE
IN THAT HANDOUT FOR SESSION 5
TODAY. BARB ASKS HOW WOULD YOU
OPEN THE FILE TO FIX THE
CHARACTERS THAT DON'T DISPLAY
CORRECTLY? SOME OF THOSE
CHARACTERS ARE NOT SUPPORTED BY
EXCEL, SO IT DEPENDS ON WHAT YOU
ARE DOING, AND WHAT PROGRAMS YOU
WOULD WANT TO USE IT IN MOVING
FORWARD. I DON'T HAVE A GREAT
ANSWER FOR THAT RIGHT NOW, BUT
IT ALSO DEPEND ON WHAT YOU ARE
DOING WITH THIS DATA DOWN THE
ROAD. IF YOU ARE JUST LOOKING
AT IT, YOU COULD GO IN AND
FIGURE OUT A WAY TO FIX THOSE
MANUALLY. THERE'S ALSO SOME
SORT OF MORE ADVANCED TECHNIQUES
YOU CAN USE TO SANITIZE THOSE
CHARACTERS FOR EXCEL, BUT IT IS
A LITTLE BEYOND THE SCOPE OF
WHAT WE ARE TALKING ABOUT HERE.
ANY OTHER QUESTIONS? OKAY, WE
ARE GOING TO MOVE ON TO OUR
SECOND CASE STUDY, BECAUSE I
THINK IT'S REALLY INTERESTING,
AND IT'S SOMETHING THAT A NUMBER
OF YOU HAVE ASKED ABOUT IN ONE
CAPACITY OR ANOTHER WHEN WE WERE
SOLICITING IDEAS FOR CASE
STUDIES. SEVERAL PEOPLE ASKED
ABOUT ANALYZING AUTHORSHIP AND
CO-AUTHORSHIP INFORMATION.
JAMIE WAS KIND ENOUGH TO GIVE US
A SPECIFIC AUTHOR WE COULD USE
AS AN EXAMPLE, SO THANK YOU VERY
MUCH. WE WILL START WITH OUR
GOAL. AND WE HAVE AN AUTHOR,
ERIC D. PETERSON. IT'S SPELLED
ERIC, BECAUSE IT WILL BE
IMPORTANT LATER. BEFORE I DUG
INTO THIS TOO FAR, I DID A
LITTLE DIGGING ON THIS GUY, JUST
TO MAKE SURE I KNEW WHO HE WAS
AND TO GET A LITTLE INFORMATION
TO INFORM MY FUTURE DEVELOPMENT,
SO I FOUND HIS WEB SITE OR PAGE
ON THE DUKE UNIVERSITY SCHOOL OF
MEDICINE WEB SITE. HIS FIELD IS
CARDIOLOGY, AND I WAS LOOKING
DOWN, AND HE GOT HIS M.D. IN
1988, SO HE'S BEEN GOING FOR--
HE'S BEEN PRODUCING FOR A NUMBER
OF YEARS. IF YOU SCROLL DOWN
FURTHER, YOU HAVE 164 PAGES OF
PUBLICATIONS. THIS IS A VERY
PROLIFIC AUTHOR, AND WHAT WE
WANT TO KNOW IS WHO HE'S WORKED
WITH AND HOW FREQUENTLY. THAT'S
SORT OF THE BASELINE GOAL.
TRYING TO GET A SENSE OF WHO HIS
CO-AUTHORS ARE. GO BACK TO MY
SLIDES. PEOPLE WERE ASKING
ABOUT THE SPELLING OF ERIC
PETERSON. SORRY ABOUT THAT.
ERIC D. PETERSON. SO I WILL
START TO WORK US THROUGH THAT
FRAMEWORK I TALKED ABOUT, WITH
THE INPUT/OUTPUT FORMAT. THE
INPUT IS JUST THE NAME. OUR
OUTPUT IS PROBABLY A LIST OF
CO-AUTHORS, IDEALLY WITH MAYBE
THEIR AFFILIATIONS OR NUMBER OF
PAPERS THEY CO-AUTHORED WITH
ERIC D. PETERSON, BUT WE AREN'T
SURE YET. FOR FORMAT, WE ARE
NOT EXACTLY SURE WHAT THE BEST
FORMAT FOR THIS OUTPUT IS GOING
TO BE. ONE OF THE REASONS WE
ARE LOOKING AT THIS CASE STUDY
SPECIFICALLY IS TO SHOW YOU
WHILE THAT FRAMEWORK I TOLD YOU
ABOUT CAN BE USEFUL, PLEASE
DON'T FEEL LIKE YOU ALWAYS NEED
TO STICK TO IT. YOU WILL
SOMETIMES FIND THINGS THROUGH
EXPLORATION AND EXPECTATION YOU
WOULDN'T GET TO OTHERWISE JUST
BY FOLLOWING THOSE STEPS. I
WOULD ENCOURAGE YOU TO THINK
ABOUT THE DATA, AND THERE'S A
COUPLE DATA ISSUES THIS CASE.
ONE IS THAT AUTHOR
DISAMBIGUATION IS KIND OF
TRICKY. PUBMED DOESN'T DO IT.
THERE'S IDENTIFIER INFORMATION
YOU CAN USE TO FIGURE OUT THE
DIFFERENCE BETWEEN TWO AUTHORS
WITH THE SAME NAME AND INITIALS,
BUT WE HAVE TO FIND SOME METHODS
TO ENSURE THAT OUR DATA CONTAINS
INFORMATION FOR JUST THE CORRECT
ERIC D. PETERSON. I WOULD
CAUTION YOU PUBMED DOESN'T HAVE
EVERYTHING. I WOULD ENCOURAGE
YOU TO USE CAUTION WHEN DOING
THE WORK, UNLESS YOU ARE SURE
THE UNIVERSAL CITATIONS YOU ARE
LOOKING AT ARE ALL IN PUBMED.
FINALLY, FOR THIS DISCUSSION,
BUT THERE MAY BE OTHER ISSUES WE
NEED TO APPROACH, AFFILIATION
DATA IS NOT CONTROLLED. SO
QUANTIFYING THAT WILL BE TRICKY,
BUT WE'LL COME BACK TO THAT IN A
SECOND. LET'S START WITH THE
BASICS, A BASIC SEARCH AND
FIGURE OUT HOW MANY RECORDS WE
ARE DEALING WITH.
LET ME HOP HERE. WE ARE GOING
TO DO A BASIC SEARCH, ESEARCH,
PETERSON, ED, THE PROPER AUTHOR
FORMAT IS LAST NAME, SPACE
INITIALS, TAGGED WITH AUTHOR. I
RUN THAT. OOPS, I FORGOT THE
QUERY ARGUMENT. SO IF YOU HAVE
EVER SEEN THIS ERROR WHEN YOU
HAVE BEEN MESSING AROUND WITH
STUFF, YOU PROBABLY EITHER
FORGOT AN ARGUMENT OR FORGOT A
DASH, LIKE I JUST DID. NOW WE
HAVE-- WE KNOW HOW MANY
PETERSON ED RECORDS ARE IN
PUBMED, 1171. BUT AS WE HAVE
TALKED ABOUT, WE NEED TO DO SOME
DISAMBIGUATION. THERE COULD BE
MANY PETERSON ED'S. SO WE'LL
GET A LITTLE MORE INFORMATION
ABOUT WHICH SPECIFIC PETERSON ED
IS LISTED. SO I WILL TAKE THAT
SEARCH, PIPE IT INTO AN EFETCH.
1,100 RECORDS IS A LITTLE MUCH
TO BE DOING THIS WITH. SOME OF
OUR CODE EXAMPLES WILL TAKE A
LITTLE LONGER TO RUN. I WOULD
TEST IT WITH A SMALLER SET, BUT
I THINK WE'RE RIGHT ON THE EDGE
OF WHERE WE CAN DO THIS
FUNCTIONALLY IN A DEMO. WHAT WE
HAVE WANTED WAS TO LOOK AT EACH
OF THE RECORD AND MAKE SURE THEY
ARE REFERRING TO THE CORRECT
PETERSON ED, THE CORRECT ERIC D.
PETERSON. WE WILL START OFF
WITH A PATTERN OF PUBMED
ARTICLE, THEN WE WANT THE PMID.
THAT PART WE KNOW. THAT LOOKS
GOOD SO FAR. NOW WE WANT TO
LOOK AT THE AUTHOR INFORMATION
FOR THE PETERSON ED ATTACHED TO
EACH OF THESE RECORDS AND MAKE
SURE IT IS THE CORRECT PETERSON
ED. THIS IS SIMILAR TO EXERCISE
3 FROM LAST CLASS. WE ARE GOING
TO USE A BLOCK AUTHOR TO CHECK
EACH AUTHOR, BUT WE ARE ONLY
GOING TO LOOK AT THE INFORMATION
WHERE THE LAST NAME EQUALS
PETERSON AND THE INITIALS EQUALS
ED. THAT'S GOING TO LIMIT US SO
WE'LL HAVE ONE ROW PER RECORD
WITH JUST THIS AUTHOR'S
INFORMATION. WHAT INFORMATION
WOULD HELP US FOR-- WOULD HELP
US WITH DOING THIS
DISAMBIGUATION. SO A COUPLE
THINGS WE CAN DO-- THE
IDENTIFIER MIGHT HELP. AS WE
NOTICED, THIS AUTHOR STARTED--
GOT HIS DEGREE IN 1988, SO
CHANCES ARE THAT PROBABLY NOT
ALL OF HIS PUBMED RECORDS-- NOT
ALL THE CITATIONS HAVE HIS
IDENTIFIER AND POSSIBLY-- IT IS
NOT REQUIRED, SO THAT MAY OR MAY
NOT HELPFUL, BUT WE'LL PUT IN A
DEF ARGUMENT OF NA, JUST TO
PLACEHOLD, IN CASE THE
IDENTIFIER IS NOT THERE. WE CAN
ALSO USE AFFILIATION. THAT
MIGHT HELP. AFFILIATION IS NOT
CONTROLLED, BUT IF YOU SEE
SOMETHING BLATANTLY INCORRECT,
THAT WOULD INDICATE SOMETHING,
BUT I WILL HOLD OFF ON THAT FOR
NOW. I AM GOING TO DO LAST NAME
AND FORENAME. FORENAME IS WHEN
PRESENT, THE FULL NAME OF THE
AWE-- FIRST-- THE FULL FIRST
NAME AND THE MIDDLE INITIAL OF
THE AUTHOR. IF THERE IS NO
FIRST-- FULL FIRST NAME, IT
WILL JUST BE THE INITIALS. THIS
MIGHT TELL US SOMETHING. I AM
GOING TO RUN THIS, BECAUSE IT
WILL TAKE A MOMENT OR TWO TO
RUN. WHAT WE WILL DO IS SORT OF
LOOK THROUGH THESE RECORDS AND
SEE IF THERE'S ANY WE CAN
SPECIFICALLY OMIT. LOOKING AT
THE QUESTIONS. MATT IS DOING A
GOOD JOB OF ANSWERING-- OH,
THERE WE GO. THANK YOU, MATT.
I WAS NOT FAMILIAR WITH THAT.
NANCY SAYS AT LEAST ONE
CO-AUTHOR DROPPED PETERSON'S
MIDDLE INITIAL. YES, AND THAT
IS ANOTHER QUESTION TOO. SO
THIS IS SPECIFICALLY TARGETING
E.D. PETERSON, BUT LET'S LOOK AT
OUR RESULTS. WE ARE GOING TO DO
A QUICK ANALYSIS TO SEE IF
THERE'S ANY RECORD HERE
BLATANTLY BY THE WRONG PERSON.
RIGHT NOW I'M NOT SEEING A LOT
OF IDENTIFIER INFORMATION, SO
THAT MIGHT NOT END UP BEING SO
USEFUL, BUT WE WILL KEEP
CHECKING ON THAT. PART OF THE
REASON WE ARE NOT SEEING HERE IS
BECAUSE THESE PMID'S ARE SORTED
FROM MOST RECENT TO LEAST
RECENT. BY PMID IN DESCENDING
ORDER. THE OLDEST STUFF IS AT
THE BOTTOM OF OUR LIST. THESE
PETERSON E.D.'S MIGHT TAKE MORE
WORK. THAT MIGHT BE WHERE OUR
HUMAN INTERVENTION COMES IN.
KEEP SCROLLING UP HERE AND SEE
WHAT ELSE WE GOT. WE HAVE ERIC
DAVID, THAT'S FINE. WELL, WE
HAVE AN ERIK. HERE'S A BIT OF
AN ISSUE. WE HAVE A COUPLE HERE
WITH NO PETERSON E.D.
INFORMATION. SO I WILL GRAB
THIS ONE AND TAKE A LOOK IN
PUBMED AND SEE WHAT'S GOING ON
WITH THIS ONE. I DON'T SEE
PETERSON E.D. LISTED AS AN
AUTHOR, BUT IF I OPEN UP THE
COLLABORATORS TAB, I SEE HIM
RIGHT HERE, WHICH MEANS HE WOULD
BE LISTED IN THE INVESTIGATOR
FIELD, INSTEAD OF AUTHOR FIELD.
I KNOW, FROM HAVING LOOKED AT
THIS AHEAD OF TIME, THERE ARE A
COUPLE PLANKS OF THAT NATURE IN
HERE. WE WILL COME BACK TO
THOSE, THE
COLLABORATOR/INVESTIGATOR ISSUE.
I WANT TO DEAL WITH THIS ERIK
WITH A K HERE. THIS IS
18780023. WE HAVE PUBMED HERE.
WE MIGHT AS WELL LOOK AT SEE
WHAT IS GOING ON. THIS IS A
RECORD FROM 2008 AND IT LOOKS
LIKE IT IS ORTHOPEDIC
SURGERY-RELATED, SO UNLESS OUR
ERIC D. PETERSON IS MOONLIGHTING
AS A SURGEON, THIS IS PROBABLY
NOT WHAT WE WANT. THERE'S ONE
HERE, THERE'S ONE THERE,
PROBABLY A FEW MORE IN THERE
ELSEWHERE, SO I WILL TAKE NOTE
OF THIS PMID. IN MY NEXT DRAFT,
I WILL TRY TO GET RID OF THE
ERIK, WITH A K, PETERSONS. SO I
WANT TO START OFF JUST THE SAME
WITH MY ESEARCH, MY EFETCH, MY
XTRACT WITH THE PUBMED ARTICLE,
THEN LOOK THROUGH THE BLOCK
AUTHORS, ONLY INCLUDE AUTHORS IF
LAST NAME EQUALS PETERSON AND
INITIALS ED. THAT'S GOOD, BUT I
ALSO WANT TO EXCLUDE IT WHEN THE
FORENAME IS ERIK W A K. SO HOW
AM I GOING TO DO THAT? WHEN I
GET STUCK LIKE THIS, I GO TO THE
DOCUMENTATION. I WILL GO TO OUR
WEB SITE, WHICH I BELIEVE MOST
OF YOU HAVE LOOKED AT, AND GO TO
THIS EDIRECT MENU, GO TO EDIRECT
DOCUMENTATION, WHICH HAS A LOT
OF THE DOCUMENTATION ON ALL THE
STUFF WE TALKED ABOUT SO FAR.
SCROLL DOWN TO THE EXTRACT
SECTION, RIGHT HERE, AND WE ARE
WORKING ON A CONDITIONAL
ARGUMENT. GOING TO SCROLL DOWN,
BECAUSE I THINK WHAT I WANT IS
THIS. THIS IS NOT ARGUMENT,
WHICH WILL LET ME SAY I WILL
ONLY INCLUDE THESE AUTHORS IF
THE FORENAME IS NOT ERIK, WITH A
K, SPACE D. SO JUST THROW THAT
ON HERE, JUST ANOTHER BOOLEAN.
FORENAME IS NOT ERIK WITH A K,
SPACE D. THE IDENTIFIER WASN'T
GETTING ME MUCH, SO I WILL SKIP
THAT. ALSO, BECAUSE I'M
SKIPPING THAT, I CAN ALSO SKIP
MY DEF AND GO STRAIGHT TO THE
ELEMENT PART. LAST NAME,
FORENAME. I SHOULD HAVE PMID,
LAST NAME, FORENAME AND I SHOULD
HAVE GOTTEN RID OF MY ERIKS WITH
K'S. HOPEFULLY THAT
DOCUMENTATION WILL PROVE USEFUL
TO YOU, AND HOPEFULLY YOU WILL
BE-- IF YOU GET STUCK ON SOME
STUFF, BROWSE THROUGH THERE AND
SEE IF THERE'S SOMETHING YOU ARE
FORGETTING OR MISSING THAT MIGHT
HELP YOU SOLVE A PARTICULAR
PROBLEM. REMEMBER, I WAS
LOOKING FOR 18780023, WHICH WE
SHOULD HAVE GOTTEN RID OF, WITH
THAT CONDITION. MY PMID'S ARE
IN THE CORRECT ORDER, SO I CAN
SCROLL UP UNTIL I FIND-- OKAY,
SO THAT'S NOT QUITE RIGHT. WHAT
THAT DID WAS BLANKED OUT THE
AUTHOR INFORMATION FOR THAT
PARTICULAR AUTHOR. BUT THAT
MIXED IT IN WITH MY
COLLABORATORS, INVESTIGATORS, SO
WHILE THIS QUOTE, UNQUOTE
WORKED, IT DID WHAT I ASKED IT
TO DO, IT ENDED UP MAKING MORE
WORK FOR US. THIS IS THE EXACT
PROCESS I WENT THROUGH WHEN
DEVELOPING THIS CASE STUDY. I
DID ALL THIS WORK TO GET THAT
CONDITION IN THERE, I THOUGHT I
WAS SO CLEVER, THEN REALIZED IT
WAS A DUMB IDEA, BECAUSE AT THIS
TIME DIDN'T WORK THE WAY I
WANTED IT TO WORK. IT DIDN'T
HELP ME AT ALL AND I HAD TO
THINK OF SOMETHING BETTER. THIS
HAPPENS ALL THE TIME IN THIS
SORT OF DEVELOPMENT PROCESS, BUT
THEN I DID THINK OF SOMETHING
BETTER, WHICH WAS PROBABLY
SOMETHING THAT YOU FIGURED OUT
ALREADY, BUT I WAS BEING DUMB,
WHY DON'T WE JUST MODIFY THE
SEARCH. WE ALREADY HAVE AN
ESEARCH THERE. WE CAN JUST
MODIFY IT TO EXCLUDE THOSE ERIKS
WITH K'S. LET ME CLEAR THE
SCREEN, GET-- THIS WAS OUR
ORIGINAL SEARCH, AND WE GOT 1171
RECORDS. I AM GOING TO TAKE
THAT SEARCH AND NOT OUT PETERSON
ERIK WITH A K, TAG WITH FULL
AUTHOR NAME. SO IT SHOULD
EXCLUDE THOSE RECORDS THAT HAD
ERIK WITH A K, AND THEY DO. A
MUCH SIMPLER WAY TO DO SOMETHING
THAT ACTUALLY WORKS, INSTEAD OF
A COMPLICATED WAY THAT DIDN'T
WORK. LET'S TAKE THAT SEARCH--
I WILL COPY AND PASTE THIS. SO
NOW WE ARE GOING TO TRY AND
SEE-- I AM GOING TO CANCEL
THAT. WE ARE GOING TO DEAL WITH
THE INVESTIGATOR QUESTION. WE
COULD DO THIS IN A FEW WAYS.
THE FIRST THING WE HAVE TO
DECIDE IS DO WE WANT TO EXCLUDE
THOSE RECORDS? SO IF ERIC AT
PETERSON IS AN INVESTIGATOR, ARE
THOSE RECORDS RELEVANT TO OUR
SEARCH? IF NOT, IF WE THINK WE
ONLY WANT WHEN HE'S AN AUTHOR,
WE COULD EXCLUDE THOSE RECORDS
ENTIRELY BY MAYBE MODIFYING OUR
SEARCH, BUT THAT MIGHT RUN THE
RISK OF REMOVING RECORDS WHERE
HE IS LISTED AS BOTH, SO WE HAVE
TO BE CAREFUL ABOUT THAT.
ANOTHER SOLUTION IS JUST MAKE A
NOTE OF THE PMIDS AND EXCLUDE
THEM FROM OUR LIST. WE COULD
SORT OF JOT DOWN WHICH ONES ARE
AFFECTED AND IF WE ARE DOING
MORE INTENSIVE CHECKING OF THESE
RECORD AN WE FIND OTHER RECORDS
WE MAY WANT TO REMOVE FOR OTHER
REASONS, THIS MIGHT BE HOW I
WOULD DO IT, IF I WAS DOING THIS
FOR REAL. I WOULD TAKE MY BIG
LIST, REMOVE THE BAD PMID'S AND
HAVE THAT LIST, FEED THEM INTO
AN EPOST, USE EFETCH TO FETCH
THE RECORDS, THEN I AM ONLY
USING THE RECORDS I WANT AND
EXTRACTING THE CO-AUTHORS. THAT
MIGHT BE A VALID WAY--PLACE
FOR THE HUMAN INTERACTION TO
COME IN. BUT MAYBE WE DO WANT
TO INCLUDE THESE RECORDS. MAYBE
WE DO-- IF HE IS AN
INVESTIGATOR, THAT'S FINE.
THAT'S RELEVANT TO OUR INTEREST,
BUT ONLY IF IT IS THE CORRECT
ERIC. WE STILL HAVE TO DO THE
DISAMBIGUATION ON THAT. I AM
GOING TO START WITH WHAT I HAD
BEFORE, WITH MY NEW AND IMPROVED
ESEARCH, BUT NOT GOING TO STOP
THERE. THIS ONLY CAPTURES IF HE
IS AN AUTHOR. I AM GOING TO DO
BASICALLY THAT SAME BLOCK AND
THE SAME TWO LAST LINES OF CODE,
BUT I WILL ADD IT FOR IF HE'S AN
INVESTIGATOR. IT WILL CHECK
EACH AUTHOR TO SEE IF PETERSON
ED IS THERE. IF NOT THERE,
WE'LL CHECK EACH INVESTIGATOR,
OR EVERYBODY IF HE IS THERE. SO
ONE I RUN THIS, IT SHOULD DEAL
WITH MOST OF THOSE BLANKS AND WE
SHOULD BE ABLE TO GET MORE
INFORMATION ON ALL THESE
AUTHORS. ONCE WE HAVE DONE
THAT, WE CAN ADD IN MAYBE
AFFILIATION AND CHECK THE
AFFILIATION TO SEE IF EACH ERIC
PETERSON IS THE CORRECT ERIC
PETERSON. THAT IS NOT GOING TO
BE FOOL-PROOF AND IT WILL TAKE
TIME, BUT USING EDIRECT, WE CAN
GET THE INFORMATION. LET'S MAKE
SURE THIS WORKED FIRST. I AM
SEEING FEWER BLANKS, WHICH IS
GOOD. WE HAVE ERIC D. PETERSON
LISTED A NUMBER OF TIMES AS
INVESTIGATOR. WE GOT RID OF OUR
ERIK WITH A K, BUT WE STILL ARE
A BLANK HERE. COPY THIS OVER,
TAKE A LOOK AT THIS ONE. SO
THIS ONE RETRIEVED PETERSON E.
LET'S SEE WHAT HAPPENS GOING ON
WITH THIS ONE. SO IT'S ED
PETERSON THAT WE GO. SO WE CAN
EXCLUDE ED PETERSON WITH A
MODIFIED SEARCH. THERE'S A LOT
OF WAYS TO DO THIS. IT DOES
TAKE SOME WORK TO DO THIS
DISAMBIGUATION, AND I THINK WE
SPENT ENOUGH TIME ON
DISAMBIGUATION. I WANT TO GET
TO ANOTHER CASE STUDY, SO I WILL
CHANGE GEARS A LITTLE BIT.
WE'LL PRETEND WE DID ALL OUR
DISAMBIGUATION AND PRETEND THAT
THIS ESEARCH THAT WE HAVE IS
GOOD ENOUGH, AND THAT'S GOING
OBJECT OUR-- WE DID ALL OUR
DISAMBIGUATION, WE GOT A SET OF
RESULTS, SO NOW LET'S TRY TO
FIGURE OUT WHO ERIC PETERSON'S
CO-AUTHORS ARE. THIS IS
STRAIGHTFORWARD ACTUALLY. WE
WILL DO AN EXTRACT AND WANT A
LIST OF HIS CO-AUTHORS, SO WE'LL
DO PATTERN AUTHOR, BUT WE HAVE
TO MAKE SEWER WE EXCLUDE ERIC
PETERSON NOW. WE ONLY WANT HIS
CO-AUTHORS. THE WAY TO DO THAT
IS USING THAT NOT TRICK. THIS
IS ACTUALLY A GOOD PLACE FOR IT.
SO IF LAST NAME IS NOT PETERSON
AND INITIALS IS NOT ED, THAT'S
WHEN WE INCLUDE THE COAUTHOR
NAME. LET'S PUT A SEP IN HERE.
GETTING AHEAD OF MYSELF.
ELEMENT LAST NAME, COMMA,
INITIALS, PUTS THOSE IN THE SAME
COLUMN. WE WANTED A TABULATED
AND QUANTIFIED LIST OF
CO-AUTHORS, SO WE'LL PIPE THAT
INTO A SORT, UNIQUE COUNT RANK
AND WE WILL GET A LIST OF THE
CO-AUTHORS, HOPEFULLY, SORTED BY
FREQUENCY. THIS IS GOING TO
TAKE ALMOST JUST AS MUCH TIME TO
RUN, BECAUSE WE ARE STILL
CHURNING OVER THAT SAME XML.
NOW WE HAVE A BIG LONG LIST OF
CO-AUTHORS. SCROLL ALL THE WAY
TO THE TOP. MAYBE I MIGHT WANT
TO USE THAT HEAD COMMAND THAT
SARAH SHOWED YOU TO SHORTEN THIS
LIST NEXT TIME, BUT HERE WE GO.
NOW WE ARE AT THE TOP, WE HAVE
THE MOST FREQUENT COLLABORATORS.
THE TRICK HERE, THOUGH, IS THAT
WE HAVE DONE AUTHOR
DISAMBIGUATION, BUT WE HAVEN'T
DONE CO-AUTHOR DISAMBIGUATION.
IF ERIC PETERSON COLLABORATED
WITH TWO PEOPLE WITH THE SAME
INITIALS-- I THINK WE ARE
RUNNING A LITTLE LOW ON TIME.
WE MIGHT LOOK AT THEIR
AFFILIATIONS AND SEE IF THERE
ARE AFFILIATIONS, WHERE ARE THEY
PUBLISHED FROM? QUANTIFYING
THAT WILL BE TRICKY, BECAUSE
THAT INFORMATION IS NOT
CONTROLLED IN ANY WAY, SO IT
WOULD PROBABLY TAKE A LOT OF
HUMAN EVALUATION OR SOME VERY
ELABORATE SCRIPTING, BUT
CONSIDERING THAT 90/10 RULE I
TALKED ABOUT, IS IT ACTUALLY
EASIER TO FIGURE OUT A PERFECT
SCRIPT, OR EASIER TO LOOK
THROUGH THEM. I WILL LEAVE THAT
AS AN EXERCISE TO THE READER TO
EXPERIMENT WITH THAT. I'M GOING
TO CHANGE GEARS INTO A NEW CASE
STUDY, BUT IF THERE ARE
QUESTIONS ABOUT THIS ONE BEFORE
I MOVE ON-- KATE SAYS IT WOULD
TAKE LESS TIME TO RUN IF YOU
WERE USING THE HISTORY SERVER.
NOT EXACTLY, BECAUSE THE HISTORY
SERVER JUST SAVES PMID'S, AND
THE ISSUE HERE IS NOT GETTING
THE PMID'S, IT IS GETTING THE
XML FROM THE SERVER ONTO OUR
COMPUTER, BUT YOU SET ME UP
NICELY FOR THE NEXT CASE STUDY,
SO THANK YOU VERY MUCH FOR THAT.
ANY OTHER QUESTIONS? OKAY,
GOOD. IF YOU THINK OF ANY, I AM
GOING TO HOPEFULLY RESERVE TIME
AT THE END FOR QUESTIONS TOO.
CLEAR THAT AND WE WILL MOVE ON
TO OUR OTHER CASE STUDY. THIS
CASE STUDY WAS SUGGESTED BY
SUSAN.
THANK YOU, SUSAN. IT IS AN
INTERESTING EXAMPLE OF AN
OUTSIDE THE BOX USE FOR EL
DIRECT. INSTEAD OF ANALYZING
PUBMED DATA, WHICH IS WHAT WE
OFTEN DO, WE WILL ANALYZE A
PUBMED SEARCH STRATEGY. THIS IS
NOT SOMETHING I REALLY THOUGHT
ABOUT BEFORE LOOKING INTO THIS
CASE STUDY. AND STRATEGY WE ARE
GOING TO BE ANALYZING ONE THAT
SUSAN'S ORGANIZATION USES TO
FIND NEW VETERINARY ARTICLES
BEING ADDED TO PUBMED. THEY RUN
THE SEARCH EVERY WEEK WITH A
DATE RANGE, I BELIEVE, AND THE
SEARCH, WHICH WILL LOOK AT IN
DETAIL, SEARCHES A LIST OF ABOUT
300 VETERINARY JOURNALS AND THAT
LIST IS UPDATED AS NEW JOURNALS
EMERGE THAT ARE RELEVANT TO THE
TOPIC. IT ALSO SEARCHES A
HANDFUL OF MESH SUBJECT TERMS.
THAT LIST IS ALSO PERIODICALLY
UPDATED, BUT MAYBE IS DUE FOR AN
UPDATE, WHICH IS SORT OF THE
PURPOSE HERE. OUR GOAL WITH
THIS PROJECT IS TO SEE IF WE CAN
FIND OTHER TERMS THAT AREN'T
CURRENTLY BEING SEARCHED, THAT
WE CAN ADD TO THIS SEARCH TO
IMPROVE RETRIEVAL. ONE WAY TO
APPROACH THIS IS LOOK AT THE
RESULTS THIS SEARCH IS
RETRIEVING NOW, AS IS, THEN
ANALYZE THE SUBJECT TERMS THAT
OCCUR FREQUENTLY ON THE
CITATIONS IN THAT RESULTS SET.
IF THERE ARE TERMS THAT POP UP
IN A LOT OF THESE RESULTS THAT
AREN'T IN THE SEARCH STRATEGY,
MAY MIGHT BE GOOD CANDIDATES TO
ADD TO A FUTURE VERSION OF THIS
STRATEGY. LET'S WALK THROUGH
THIS FRAMEWORK WE TALKED ABOUT
BEFORE. THE INPUT/OUTPUT
FORMAT. THE INPUT IS A SEARCH
STRATEGY. A PRETTY LONG
STRATEGY. WE WILL START WITH
ESEARCH, BUT WE WANT TO KEEP
THAT STRATEGY IN A FILE, SO IF
WE WANT THE REPEAT THIS IN THE
FUTURE, WE CAN TWEAK THE
STRATEGY IN THE FILE AND LEAVE
OUR CODE THE SAME. THIS WILL
ALSO HELP KEEP OUR CODE SHORTER.
OUR OUTPUT IS GOING TO BE A LIST
OF SUBJECT TERMS AND PROBABLY
WE'LL USE MESH DESCRIPTORS, SO
WE'LL PROBABLY WANT TO USE THAT
DESCRIPTOR NAME ELEMENT, BUT
WE'LL SEE. WE COULD BE OPEN TO
NEW IDEAS, AS THEY COME UP. IF
OUR FORMAT, THIS IS MORE CUT AND
DRIED. WE WANT THE-- LIST OF
UNIQUE SUBJECT TERMS, WITH
COUNTS. THIS IS CLASSIC COUNT
RANK TERRITORY. IT WILL BE A
BIG LIST. WE ARE GOING TO START
BY TAKING A LOOK AT THE SEARCH
STRATEGY. PULL IT UP IN HERE.
THAT'S A LITTLE TOO BIG. YOU
CAN SORT OF READ THAT, I THINK.
THIS IS THE SEARCH STRATEGY.
IT'S NOT A BUNCH OF JOURNALS AT
THE TOP, A BUNCH OF MESH
HEADINGS AT THE BOTTOM, AND IT'S
GOT A DATE RESTRICTION DOWN
HERE. BEFORE WE DO ANYTHING, I
AM GOING TO RUN SEARCH, BUT
WITHOUT THE DATE RESTRICTION.
WE ARE TRYING TO SEE SORT OF
WHAT THIS BODY OF RESULTS IS
LIKE, SO WE DON'T WANT TO LIMIT
IT TO NEW STUFF. WE WILL LOOK
AT EVERYTHING, AT LEAST FOR NOW.
THEN WE'LL SEE HOW BIG OF A
PROBLEM THAT'S GOING TO BE. GO
BACK TO PUBMED. INSTEAD OF
GOING INTO THE NORMAL SEARCH
BOX, I WILL CLICK ON THE
ADVANCED LINK HERE UNDER THE
SEARCH BOX TO GET INTO ADVANCED
SEARCH BUILDER. SOME BROWSERS
DO NOT LIKE EXTREMELY LARGE
SEARCH STRINGS IN THE NORMAL
SEARCH BOX. BUT IN THE ADVANCED
FILTER, YOU SHOULDN'T HAVE A
PROBLEM. GO AHEAD AND SEARCH.
WE GET ABOUT 2 MILLION RESULTS,
WHICH IS DEFINITELY TOO MANY FOR
US TO WORK IN THIS DEMO. IT IS
ALSO WAY TOO MANY TO LOOK AT FOR
TESTING. WHAT WE ARE GOING TO
DO IS BEFORE WE GET TOO
RESTRICTIVE WITH THE DATES, WE
ARE GOING TO LOOK AT MAYBE JUST
THE JOURNALS. I AM GOING TO HOP
BACK IN HERE AND GRAB JUST THE
JOURNALS. THE SUBJECT TERMS WE
ARE LOOKING TO IMPROVE ANYWAY.
LET'S SEE WHAT THE JOURNALS HAVE
IN THEM AND START FROM THERE.
WRONG WINDOW. THERE WE GO.
BACK IN ADVANCED. SO NOW WE ARE
DOWN TO 600,000, STILL PRETTY
LARGE. FOR TESTING PURPOSES, I
WILL THROW A DATE RESTRICTION ON
THERE. YOU MAY CONSIDER A DATE
RESTRICTION. YOU MAY NOT BE
INTERESTED IN EXTREMELY OLD
RESULTS. YOU MAY WANT TO LOOK
AT THE SUBJECT TERMS MORE
RECENTLY. JUST THROW A 2017
DATE RESTRICTION ON THERE, GET
IT DOWN TO SOMETHING MORE
MANAGEABLE. 15,000 RECORDS.
STILL KIND OF LONG, BUT WE MAY
HAVE A SOLUTION. LET'S START
BUILDING OUR ESEARCH AND WE'LL
SEE WHERE WE GO FROM THERE. WE
HAVE CLEARED OUR SCREEN, SO
WE'LL START WITH AN ESEARCH. I
HAVE ALREADY SAVED JUST THE
SEARCH WITH THE JOURNAL LIST,
LIKE EXCLUDING THE MESH TERMS
INTO A SEPARATE FILE. SO I HAVE
THAT READY TO GO. THE I RUN A
QUICK LS HERE, WE CAN SEE VET
JOURNALS.TXT IS HERE. SO THAT
IS READY TO GO. ESEARCH-- I
WILL REMEMBER TO PUT IN MY QUERY
ARGUMENT. AND PULLING OUR
SEARCH STRING FROM FILE, SO
WE'LL USE THE TRICK KATE SHOWED
YOU WITH THE DOLLAR SIGN,
PARENTHESES AND CAT, THE NAME OF
THE FILE. WE WANTED A DATE
RESTRICTION ON THAT TOO, SO
WE'LL DO DATE-TYPE PDAT AND MIN
DATE IS 2017 AND MAX DATE IS
2017. WE SHOULD GET ABOUT
15,000 RESULTS. ABOUT THE SAME
AS WHAT WE GOT IN PUBMED. NOW
WHAT I WOULD DO IS TRY TO GET
PMID'S TO TEST WHAT MY EVENTUAL
OUTPUT IS GOING TO BE. 15,000
IS TOO MUCH TO TEST WITH,
BECAUSE IF WE MAKE MISTAKES, IT
WILL TAKE FOREVER TO FIGURE OUT
WE MADE MISTAKES. SO I WILL
TAKE THAT SEARCH. RATHER THAN
GETTING THE FULL XML, I WILL
JUST GRAB A LIST OF PMID'S.
WILL ALSO TAKE A LITTLE WHILE,
BECAUSE THERE'S A LOT OF PMID'S
IN THERE, BUT THEY ARE A LOT
FASTER TO DOWNLOAD THAN XML.
NOW I NEED AN EFETCH FOR TESTING
PURPOSES. I WILL GRAB A HANDFUL
OF PMID'S HERE. PUBMED, AND
TAKE THIS ONE AND PASTE THAT.
TAKE THIS ONE, PASTE THAT. TOO
MANY COMMAS, AND I THINK I WILL
PROBABLY DO ONE MORE, JUST TO
MAKE SURE I GOT SOME MESH
HEADINGS IN THERE. MY FORMAT IS
GOING TO BE XML. NOW I HAVE MY
EFETCH THAT I CAN USE TO TEST.
LET ME CLEAR MY SCREEN, GET MY
EFETCH.
LET'S TALK ABOUT OUR EXTRACT.
THIS WILL BE A PRETTY SIMPLE
EXTRACT. WE WANT A LIST OF MESH
HEADINGS. SO SIMILAR TO THE WAY
WE WANT OUR LIST OF AUTHORS
BEFORE, WE USE PATTERN MESH
HEADING, AND MESH HEADING, WE
WILL LOOK AT IT IN MORE DETAIL,
THE MESH HEADING ELEMENT HAS
TRIAL ELEMENTS FOR THE
DESCRIPTOR NAME, WHICH IS THE
HEADING, AND QUALIFIER NAME, IF
THERE IS ONE, WHICH IS FOR THE
SUBHEADING. WE JUST WANT THE
ACTUAL HEADING, WHICH IS THE
DESCRIPTOR NAME. THEN WE SHOULD
BE GOOD TO GO. LET'S SEE WHAT
THAT DOES. SO THAT GIVES US A
LIST OF MESH HEADINGS A LIST OF
DESCRIPTORS. THAT'S WHAT WE
WANTED. WHAT WE WILL PROBABLY
DO IS THROW A COUNT RANK AT THE
END OF THIS THING, SO WE GET
SOMETHING THAT WOULD LOOK LIKE
THIS, WHICH I'M NOT GOING TO
RUN. THE REASON I AM NOT GOING
TO RUN IT, IT WILL TAKE ABOUT
FIVE MINUTES TO RUN, JUST TO
COMPLETE. AND THE REASON WHY,
IS THE XML PART. DOWNLOADING
THE XML, THE XML ISSUE THERE--
IT IS GOING TO BE ABOUT 130
MEGABYTES, IT WILL TAKE ABOUT
FIVE MINUTES TO DOWNLOAD. BUT
IF WE KNOW WE ARE GOING TO DO
SEVERAL DIFFERENT EXTRACTS AND
SEVERAL SPLICES ON THE SAME
LARGE DATA SET, WE CAN JUST
DOWNLOAD THE XML ONCE AND KEEP
REUSING IT. SO RATHER THAN
RUNNING THAT SCRIPT I JUST HAD
UP THERE, I WOULD RUN SOMETHING
LIKE THIS, WHICH OPPORTUNITY DO
ANY OF THE EXTRACT STUFF. IT
JUST TAKES THE WHOLE XML AND
SAVES IT AS A FILE, A BIG FILE.
AND IT WILL STILL TAKE FIVE
MINUTES TO DO THIS PART,
HOWEVER, IT WILL SPEED
EVERYTHING ELSE UP DOWN THE
ROAD. I RAN THIS EARLIER, SO I
AM NOT GOING TO RUN IT NOW. WE
DON'T HAVE FIVE MINUTES TO
SPARE, BUT IF I DO MY LS AGAIN,
YOU CAN SEE RIGHT HERE, WE HAVE
VET.XML READY TO GO. NOW LET'S
DO SOME STUFF WITH IT. WE CAN
DO STUFF PRETTY EASILY BY USING
THIS INPUT ARGUMENT IN THE
XTRACT COMMAND. IT IS JUST AS
IF I'M TYPING IT IN FROM AN
EFETCH. WE WILL USE THE SAME
XTRACT WE JUST BUILT, WITH OUR
SORT UNIQUE COUNT RANK, AND SAVE
IT TO A FILE. WHEN I RUN THIS,
RATHER THAN TAKING FIVE MINUTES,
IT TAKES LESS THAN FIVE SECONDS.
THERE'S A HUGE ADVANTAGE THERE,
ESPECIALLY IF I AM GOING TO DO
MULTIPLE DIFFERENT SPLICES OF
THIS DATA, WHICH I WILL BE. SO
I AM GOING TO OPEN UP THAT FILE,
SO WE CAN LOOK AT IT. NOW WE
HAVE A LIST OF MESH HEADINGS.
SOME OF THESE ARE NOT
PARTICULARLY INTERESTING, LIKE
ANIMALS, FEMALE, MALE, HUMANS.
WE ARE GETTING A LOT OF THESE
CHECK TAGS IN HERE. THERE'S A
LOT OF JUNK IN HERE WE ARE TO
THE GOING TO BE INTERESTED IN
ADDING TO OUR SEARCH.
MAYBE THAT'S GOOD ENOUGH.
REMEMBER THE 90/10 RULE. MAYBE
THE REST OF THE 10% WE COULD DO
BY LOOKING THROUGH THE LIST AND
FINDING THE GOOD STUFF, BUT I
WANT-- WE SHOW OFF A LITTLE AND
LOOK AT MAJOR TOPICS, TRY TO GET
JUST THE MAJOR TOPICS, RATHER
THAN ALL THE MESH HEADINGS IN
HERE. WE WILL DO THAT,
SOMETHING SOMETHING, WHICH YOU
DID ON YOUR HOMEWORK, HOPEFULLY.
WE WILL START BY LOOKING AT SOME
XML FOR A PARTICULAR RECORD.
LET ME GET THAT RECORD HERE. WE
CAN LOOK AT ANY RECORD THAT HAS
MESH HEADINGS. I WILL DO
28657829, AND JUST LOOK AT THE
XML TO SEE WHAT THE STRUCTURE OF
THE MESH HEADINGS IS. I GET
DOWN HERE, REMEMBER, I AM
LOOKING FOR THE HEADINGS THE
INDICES HAVE DETERMINED ARE THE
MAJOR TOPICS. I CAN SEE HERE,
IF WE LOOK AT-- THIS MESH
HEADING ELEMENT DOESN'T HAVE ANY
QUALIFIERS. IT JUST HAS
DESCRIPTORS. IT JUST HAS THE
DESCRIPTOR. THAT ATTRIBUTE IS
SET TO YES RIGHT ON THE
DESCRIPTOR ELEMENT. HOWEVER, IF
WE LOOK UP HERE AT COMPLEMENT
SYSTEM PROTEINS, WE CAN SEE THE
MAJOR TOPIC YN ATTRIBUTE IS SET
TO YES ON THE QUALIFIER, NOT ON
THE DESCRIPTOR. IN ORDER TO
MAKE SURE WE GET ALL THE MAJOR
TOPICS, WE HAVE TO SORT OF THINK
A LITTLE CREATIVELY HERE. WHAT
WE WANT IS IF THE DESCRIPTOR
NAME FOR A MESH HEADING HAS A
MAJOR TOPIC ATTRIBUTE SET TO
YES, WE SET THE DESCRIPTOR NAME.
WE DON'T CARE ABOUT THE
QUALIFIER NAMES, UNLESS THEY ARE
TELLING US THE DESCRIPTOR IS A
MAJOR TOPIC. THIS IS ACTUALLY
ALSO EXPLAINED IN THE SAMPLE
CODE FROM LAST TIME, IN THE
HOMEWORK, BUT I WILL WALK
THROUGH IT QUICK.
I WILL PROBABLY COPY AND PASTE
IT. WE ARE GOING TO USE OUR
XTRACT, INPUT THE SAME FILE,
SAME PATTERN, BUT WE ARE ONLY
GOING TO INCLUDE MESH HEADINGS
IF THE MAJOR TOPIC YN ATTRIBUTE
FOR DESCRIPTOR NAME EQUALS Y, OR
SAME THING FOR QUALIFIER NAME.
AND EITHER WAY, IF EITHER OF
THOSE THINGS ARE TRUE, THE
OUTPUT IS GOING TO BE THE
DESCRIPTOR NAME. TYPE THAT INTO
A SORT UNIQUE COUNT RANK, SAVE
IT TO A NEW FILE, AND I KNOW I'M
RUSHING A LITTLE BIT, BUT I WANT
TO GET THROUGH ALL THIS STUFF,
MAKE SURE WE HAVE TIME FOR
QUESTIONS. THEN LOOK AT THE
RESULTS NOW. WHICH ARE A LITTLE
MORE SPECIALIZED HERE. A LITTLE
MORE INTERESTING. WE SORT OF
WEEDED OUT SOME OF THE STUFF WE
WERE LESS INTERESTED IN. THIS
IS A GOOD LIST, HOWEVER, THESE
ARE ALL MESH HEADINGS AND BASED
ON A SUBSET OF OUR LARGER
RESULTS SET. I CHECKED THAT
LIST OF 308 JOURNALS. ONLY
ABOUT HALF ARE MEDLINE JOURNALS.
THE REST ARE NOT INDEXED. SO
THEY ARE NOT GOING TO HAVE ANY
MESH TERMS ON THEM, SO I'M NOT
GOING TO DOWNLOAD THIS NEXT PART
NOW, BUT HOW COULD WE CAPTURE
THOSE SUBJECT TERMS ON THESE
JOURNALS. WE COULD USE AUTHOR
KEY WORDS, WHICH ARE A DIFFERENT
KIND OF SUBJECT TERM. THEY ARE
LESS CONTROLLED, SO IT WOULD
REQUIRE MORE ANALYSIS ON OUR
PART, BUT THERE MIGHT BE
SOMETHING OF VALUE IN THERE.
THE ADVANTAGE TO HAVING
DOWNLOADED THAT XML AHEAD OF
TIME IS PAUSE WE DID THAT, WE
CAN EXPLORE ANY SLICE OF THAT
DATA WE WANT. WE CAN EXPLORE
KEY WORDS, OTHER WAYS OF LOOKING
AT MESH HEADINGS, WITHOUT
WAITING TO RE-DOWNLOAD THAT
STUFF EACH TIME. ANY QUESTIONS
ABOUT THAT CASE STUDY? OR
ANYTHING ELSE, WHILE I GET READY
TO GO INTO A COUPLE OF FINAL
THOUGHTS HERE. IF THERE'S
ANYTHING ELSE, I WILL ADDRESS IT
IN A MOMENT. A COUPLE OTHER
IMPORTANT THINGS TO TALK ABOUT.
THAT IS ABOUT MOST OF WHAT I HAD
FOR YOU TODAY. I REALLY WANT TO
THANK YOU FOR STICKING WITH US
OVER THE PAST FIVE CLASSES. I
WANT TO SAY, IF YOU FEEL LIKE
YOU STILL DON'T REALLY KNOW HOW
TO USE EDIRECT AFTER FIVE
75-MINUTE SESSIONS, IT TOOK ME A
LOT LONGER TO LEARN THIS STUFF,
THIS IS A LOT TO ABSORB HERE AND
YOU PROBABLY WILL ONLY BE ABLE
TO LEARN IT BY WORKING WITH
EDIRECT YOURSELF. I HOPE WE
WERE ABLE TO SHOW YOU WHY IT'S
SO USEFUL AND INTRODUCE YOU TO
TOOLS TO HELP YOU PUT IT TO USE,
BUT I ALSO HOPE YOU WILL TAKE
TIME TO REVIEW THE SAMPLE CODE
ON THE WEB SITE, LOOK AT THE
RECORDINGS, POKE THROUGH
DOCUMENTATION, EXPERIMENT WITH
EXERCISES. IT DOES TAKE TIME
AND PRACTICE TO MASTER THESE
TOOLS, SO PLEASE GIVE THEM A
TRY, PLAY AROUND WITH SOME OF
THE COMMANDS. GET COMFORTABLE
IN THE COMMAND LINE ENVIRONMENT.
AND WE HAVE ONE FINAL ASSIGNMENT
FOR THIS CLASS, WHICH GIVES YOU
THE OPPORTUNITY TO SOLVE SOME
REAL-WORLD PROBLEMS WITH SOME OF
THE STUFF YOU LEARNED OVER THE
PAST FEW WEEKS. I ENCOURAGE YOU
TO USE IT TO REALLY DIG INTO
EDIRECT. FINALLY, KEEP YOUR
EYES OPEN FOR THE NEXT TIME A
PROJECT THAT WOULD BE A GOOD FIT
FOR EDIRECT COMES ALONG. I HAVE
ONE MORE RESOURCE I WANT TO SHOW
YOU. IT'S STILL PRETTY NEW, BUT
YOU MIGHT FIND IT USEFUL. NCBI
STARTED UP WHAT THEY CALL AN
EDIRECT COOKBOOK. IT IS A LIST
OF PREMADE SCRIPTS TO ACCOMPLISH
SIMPLE TASKS. LOOKING AT AND
MODIFYING OTHERS' EXAMPLES IS
THE BEST WAY TO LEARN EDIRECT,
IT IS A GOOD RESOURCE TO BE
FAMILIAR WITH. I WILL SHOW IT
TO YOU REAL QUICK. IT IS ALSO
IN YOUR HANDOUT AND THE COURSE
MATERIALS PAGE. IF YOU WANT TO
LOOK AT IT THERE. THIS HAS BEEN
PUBLISHED IN GET HUB, AN ONLINE
REPOSITORY FOR PEOPLE OR SHARE
AND WORK ON CODE TOGETHER.
DOESN'T SEEM TO BE-- IF YOU
HAVEN'T EVER USED GET HUB, DON'T
WORRY. YOU DON'T NEED TO KNOW
ANYTHING ABOUT IT TO USE THE
COOKBOOK. IT HAS A NICE FRONT
END ON IT NOW. JUST GO TO THE
SITE, WHICH IS ALSO IN THE CHAT
BOX, AND SCROLL DOWN AND YOU
WILL SEE ALL THESE EXAMPLES
HERE. JUST COPY AND PASTE ANY
OF THE CODE, HOPEFULLY THE-- A
LOT OF THEM ARE DOCUMENTED IN
TERMS OF WHAT THEY DO, AND FEEL
FREE TO PLAY AROUND WITH THEM,
DO WHATEVER YOU WANT WITH THEM.
FOR NOW, MOST OF THE SCRIPTS ARE
MORE FOCUSED ON THE MOLECULAR
BIOLOGY, BUT THERE ARE SCRIPTS
THERE FROM PUBMED, AND I AM
TRYING TO GET MORE AS WELL, BUT
THE NICE THING ABOUT GET HUB,
CLICK ON THIS VIEW IN GET HUB
BUTTON-- THE NICE THING IS IF
YOU NEED A SCRIPT THAT DOES A
PARTICULAR THING, BUT THERE
ISN'T ONE HERE ALREADY, YOU CAN
USE THE ISSUES TAB, WHICH IS
RIGHT HERE TO REQUEST A SCRIPT.
AND HOPEFULLY SOME MEMBER OF THE
COMMUNITY WILL HELP YOU OUT. ON
THE FLIP SIDE, IF YOU HAVE THESE
SCRIPTS YOU WANT TO SHARE, YOU
CAN DO THAT TOO. THIS IS PRETTY
NEW. THEY ARE STILL WORKING OUT
SOME OF THE KINKS, BUT HOPEFULLY
IT WILL LET US DEVELOP A
COMMUNITY OF PRACTICE. AS WE
MENTIONED BEFORE, MAKE SURE YOU
SUBSCRIBE TO THE
UTILITIES-ANNOUNCE LIST SERVE.
THEY DON'T SPAM YOU. IT'S ONCE
A MONTH OR TWO AT THE MOST. WE
DON'T HAVE CLASSES SCHEDULED
NOW, BUR AS SOON AS NEW
OFFERINGS ARE AVAILABLE, WE WILL
ANNOUNCE THEM ON THE WEB SITE
AND THROUGH THE LIST SERVE. FOR
THOSE OF YOU KERR US OBJECT MLA
CREDIT, WE WILL SEND YOU
INFORMATION ABOUT HOW TO CLAIM
THAT CREDIT, AS SOON AS YOU
FINISH YOUR FINAL ASSIGNMENT.
AGAIN, THANKS
FOR STICKING WITH US FOR ALL
THESE CLASSES, AND HOPE TO SEE
YOU SOON AT ONE OF OUR FUTURE
CLASSES.