Embed Size (px)
Citation preview
"Methodologies"for
research in computer science
Lionel Brunie LIRIS lab
National Institute of Applied Sciences (INSA) Lyon, France
Menu
DisclaimerDisclaimer Discovery vs. invention What is a research result in computer science ? What is a research paper in computer science ? General
structure(s) of a research paper in CS Searching for black boxes, traps and black holes How is research evaluated ? And now ?
Disclaimer This is NOT a lecture !This is NOT a lecture !
I apologize not to be an epistemologistI apologize not to be an epistemologist
I am just a practitioner of computer science that have co-authored ~100 I am just a practitioner of computer science that have co-authored ~100 research papers and read/evaluate several hundreds onesresearch papers and read/evaluate several hundreds ones
This open discussion is absolutely NOT objectiveThis open discussion is absolutely NOT objective
Goal : open the windows and discuss how we ACTUALLY work (and Goal : open the windows and discuss how we ACTUALLY work (and not how we should work…)not how we should work…)
All questions, remarks, contradictions are welcome !!!All questions, remarks, contradictions are welcome !!!
Menu
Disclaimer
Discovery vs. inventionDiscovery vs. invention What is a research result in computer science ? What is a research paper in computer science ? General
structure(s) of a research paper in CS Searching for black boxes, traps and black holes How is research evaluated ? And now ?
Discovery vs. invention
My feeling is that there are two main ways of practicing science : My feeling is that there are two main ways of practicing science : discovery vs. inventiondiscovery vs. invention
Biologists, physicists, chemists, researchers in psychology… are Biologists, physicists, chemists, researchers in psychology… are discoverersdiscoverers
Computer scientists, researchers in nanotech, researchers in process Computer scientists, researchers in nanotech, researchers in process engineering or in industrial engineering… are inventorsengineering or in industrial engineering… are inventors
Mathematicians… are mathematicians :-)Mathematicians… are mathematicians :-)
Discovering (just two words)
Understanding the world : what are atoms constituted of, why a Understanding the world : what are atoms constituted of, why a disease in inherited, why do people have dreams, etc.disease in inherited, why do people have dreams, etc.
Understanding means first asking questions, then observing, inquiring, Understanding means first asking questions, then observing, inquiring, modeling, evaluatingmodeling, evaluating
At the end of the research process, one has an At the end of the research process, one has an answeranswer to the initial to the initial questionquestion
An answer is the most often not definitive. It is an explanation of a An answer is the most often not definitive. It is an explanation of a small piece of the natural world under some hypothesessmall piece of the natural world under some hypotheses
Inventing Software computer science produces Software computer science produces inventionsinventions
Computers do not exist by themselves. They have been created by Computers do not exist by themselves. They have been created by human beings => there is nothing to human beings => there is nothing to discoverdiscover in a computer or in a in a computer or in a softwaresoftware
The objective of research in CS is “just” to make computers and The objective of research in CS is “just” to make computers and computer networks more efficient more easy to use, more reliable, computer networks more efficient more easy to use, more reliable, more powerful… i.e. more useable/usefulmore powerful… i.e. more useable/useful
As a consequence, a research result in CS has no (real) intrinsic value. As a consequence, a research result in CS has no (real) intrinsic value. It has only the value that the research community and/or the society It has only the value that the research community and/or the society gives to it. A useless invention has no value !gives to it. A useless invention has no value !
Menu
Disclaimer Discovery vs. invention
What is a research result in computer What is a research result in computer science ?science ?
What is a research paper in computer science ? General structure(s) of a research paper in CS
Searching for black boxes, traps and black holes How is research evaluated ? And now ?
A difficult answer…
The endless debate about fundamental research, basic research, The endless debate about fundamental research, basic research, applied research, finalized research…applied research, finalized research…
A first and definitive answer : a result has some scientific value as A first and definitive answer : a result has some scientific value as soon as it has been accepted by a scientific publication (a scientific soon as it has been accepted by a scientific publication (a scientific committee)committee)
A second answer : a result has some scientific value if it interests A second answer : a result has some scientific value if it interests people and if it is novel (original) (and if it has some genericity)people and if it is novel (original) (and if it has some genericity)
The most difficult (and so the most interesting) problems have often The most difficult (and so the most interesting) problems have often been raised by applications, e.g. medical applications. Investigating been raised by applications, e.g. medical applications. Investigating these problems have gone through theoretical scientific developments.these problems have gone through theoretical scientific developments.
Menu
Disclaimer Discovery vs. invention What is a research result in computer science ?
What is a research paper in computer What is a research paper in computer science ? General structure(s) of a research science ? General structure(s) of a research paper in CSpaper in CS
Searching for black boxes, traps and black holes How is research evaluated ? And now ?
Introduction-MotivationsIntroduction-Motivations State of the artState of the art
ProposalProposal ExperimentsExperiments DiscussionDiscussion
Conclusion and future workConclusion and future work
Structure of a research paper in CS
2 (+1) key sections (1/3) Having 1 very good idea in a scientist’s life is… very good ; having two is Having 1 very good idea in a scientist’s life is… very good ; having two is
exceptionalexceptional One paper => One paper => one problemone problem and and at most one (two) contribution(s)at most one (two) contribution(s) to defend to defend
A first key section : the state of the artA first key section : the state of the art A state of the art must not be “flat” or general purposeA state of the art must not be “flat” or general purpose It must be focused on your scientific and applicative targets and “critical and It must be focused on your scientific and applicative targets and “critical and
compared”compared” Critical : the contribution/limitations of each cited paper should be analyzed wrt Critical : the contribution/limitations of each cited paper should be analyzed wrt
your problemyour problem Compared : papers should be organized in taxonomiesCompared : papers should be organized in taxonomies
The state of the art establishes/describes the scientific basis wrt which your work The state of the art establishes/describes the scientific basis wrt which your work should be compared, your should be compared, your //// will be evaluated will be evaluated
Ideally, all pertinent papers should be analyzed. Practically, it is often untracktableIdeally, all pertinent papers should be analyzed. Practically, it is often untracktable ““Making the bibliography” is a key component of the scientific processMaking the bibliography” is a key component of the scientific process
2 (+1) key sections (2/3)
A second key section : the discussionA second key section : the discussion Goal of the discussion : analyzing your contribution/proposal :Goal of the discussion : analyzing your contribution/proposal :
Wrt to your initial objective/motivationWrt to your initial objective/motivation Wrt to the state of the artWrt to the state of the art Wrt the experiments you have run… and have not runWrt the experiments you have run… and have not run Wrt the implementation choices you have madeWrt the implementation choices you have made Wrt genericityWrt genericity
A discussion is A discussion is notnot a conclusion a conclusion The discussion is often distributed over the paper :-(The discussion is often distributed over the paper :-( Do not believe authors : make your personal discussion !Do not believe authors : make your personal discussion !
2 (+1) key sections (3/3)
A third obvious section : your contributionA third obvious section : your contribution Must be “sound”Must be “sound” Must be comprehensive and completeMust be comprehensive and complete It should allow people to reproduce the experiments you useIt should allow people to reproduce the experiments you use It must be self containedIt must be self contained
Introduction - Motivation
Inventing some thing that interests nobody has no real Inventing some thing that interests nobody has no real sensesense
Inventing some thing that addresses a problem that has an Inventing some thing that addresses a problem that has an existing satisfactory solution has a limited interestexisting satisfactory solution has a limited interest
To be interesting, your work/paper should address a To be interesting, your work/paper should address a problem :problem : With a high (scientific or economic or social or…) valueWith a high (scientific or economic or social or…) value And no solutionAnd no solution
A paper should list the criteria wrt which authors consider A paper should list the criteria wrt which authors consider that their proposal is to be evaluatedthat their proposal is to be evaluated
Experiments (1/2) The often dark side of research in CSThe often dark side of research in CS A first reason : cultural/educational : computer scientists have an A first reason : cultural/educational : computer scientists have an
unsatisfactory education in statistics/experiment planning/performance unsatisfactory education in statistics/experiment planning/performance evaluationevaluation
A second reason : real life experiments are often intractable because A second reason : real life experiments are often intractable because comparing two inventions is often intractablecomparing two inventions is often intractable Concurrent software are often not availableConcurrent software are often not available Concurrent complex systems/softwares cannot be re-develop from scratch Concurrent complex systems/softwares cannot be re-develop from scratch
at a reasonable priceat a reasonable price Evaluation criteria are differentEvaluation criteria are different Users of your software are reluctant to spend/waste their time on alpha Users of your software are reluctant to spend/waste their time on alpha
versionsversions It often exists no benchmarksIt often exists no benchmarks Research works often address very small range/specific issues that are Research works often address very small range/specific issues that are
part of a complex systems but integration is often not realizedpart of a complex systems but integration is often not realized Research teams have neither the manpower nor the competence to Research teams have neither the manpower nor the competence to
develop “products” and limit their development to demonstrator or develop “products” and limit their development to demonstrator or prototypes that cannot be tested by real end usersprototypes that cannot be tested by real end users
Experiments (2/2)
However experiments are mandatory to evaluate the However experiments are mandatory to evaluate the pertinence of your workpertinence of your work
Experiments should allowExperiments should allow To evaluate the performance/pertinence/efficiency of your To evaluate the performance/pertinence/efficiency of your
proposal wrt your initial objective and wrt all the possible contexts proposal wrt your initial objective and wrt all the possible contexts (not only in the best case)(not only in the best case)
To evaluate the influence of the various parameters on the To evaluate the influence of the various parameters on the performanceperformance
To compare your proposal wrt the state of the artTo compare your proposal wrt the state of the art
Experimental conditions must be completely explicitedExperimental conditions must be completely explicited
Menu
Disclaimer Discovery vs. invention What is a research result in computer science ? What is a research paper in computer science ? General
structure(s) of a research paper in CS
Searching for black boxes, traps and black Searching for black boxes, traps and black holesholes
How is research evaluated ? And now ?
Reading a paper as an investigation
A key behavior : criticismA key behavior : criticism
A first obvious step : analyze what is written : is the A first obvious step : analyze what is written : is the proposal scientifically sound, are the performances good, proposal scientifically sound, are the performances good, is the problem really important, is the state of the art is the problem really important, is the state of the art exhaustive, etc.exhaustive, etc.
A second step : analyze what is A second step : analyze what is notnot written written
Black boxes and assumptions
Computer systems/applications/softwares are too complex/too large so Computer systems/applications/softwares are too complex/too large so that a single team/paper can address the entire problem that a single team/paper can address the entire problem => Focus on a small part or specific issue=> Focus on a small part or specific issue
=> Assumptions on the other parts=> Assumptions on the other parts
+ inputs : it exists/will exist (soon) some “system” can can provide your + inputs : it exists/will exist (soon) some “system” can can provide your prototype with the data/information it needsprototype with the data/information it needs
+ outputs : the data produced by your prototype could be/will (soon) be + outputs : the data produced by your prototype could be/will (soon) be used by the end-user side of the global systemused by the end-user side of the global system
Assumptions are Assumptions are not alwaysnot always explicit explicit
Always ask if these assumptions are valid/realistic, what do they implyAlways ask if these assumptions are valid/realistic, what do they imply
Analyzing experiments (1/2)
The border lineThe border line ““Science” definitely condemns treachery and falsification : Science” definitely condemns treachery and falsification :
modifying the result of an experiment is beyond the “red line”modifying the result of an experiment is beyond the “red line” Experimental measurements that are presented in a paper are Experimental measurements that are presented in a paper are
(supposed to be) true (at least I have supposed it)(supposed to be) true (at least I have supposed it) The question is about omitted/”forgotten” resultsThe question is about omitted/”forgotten” results From a theoretical scientific point of view, a negative result is From a theoretical scientific point of view, a negative result is
often as interesting as a positive resultoften as interesting as a positive result However, as CS has to deal with invention, a negative result is However, as CS has to deal with invention, a negative result is
commonly considered as a depreciated factcommonly considered as a depreciated fact
Analyzing experiments (2/2)
So the temptation to “avoid” including negative measurements often exists : So the temptation to “avoid” including negative measurements often exists : “this measurement is not really important but as I do not have enough space to “this measurement is not really important but as I do not have enough space to explain it, it is more important, from a scientific point of view (of course), to explain it, it is more important, from a scientific point of view (of course), to discuss in deep other more important experiments”, “The situation that leads to discuss in deep other more important experiments”, “The situation that leads to this bad result never happens in the real life”, “With a slight modification of this bad result never happens in the real life”, “With a slight modification of the prototype, it is obvious that one would never get such a result, so let us the prototype, it is obvious that one would never get such a result, so let us focus on the results concerning the core of the prototype”, etc. are reasons that focus on the results concerning the core of the prototype”, etc. are reasons that a scientist may use to self-persuade him that presenting this experiment is not a scientist may use to self-persuade him that presenting this experiment is not pertinentpertinent
Where is the yellow line ? Where is the red line ?Where is the yellow line ? Where is the red line ?
Always analyze if experiments are exhaustive. If not, always analyze if the Always analyze if experiments are exhaustive. If not, always analyze if the missing experiments are important to evaluate the performance of the missing experiments are important to evaluate the performance of the prototype and try to infer what could be the behavior of the prototype wrt these prototype and try to infer what could be the behavior of the prototype wrt these experimentsexperiments
Menu
Disclaimer Discovery vs. invention What is a research result in computer science ? What is a research paper in computer science ? General
structure(s) of a research paper in CS Searching for black boxes, traps and black holes
How is research evaluated ?How is research evaluated ? And now ?
Evaluation of a research paper
Is the topic of the paper in the domain of interest of the Is the topic of the paper in the domain of interest of the conference/journal ?conference/journal ?
(Is the bibliography complete ?)(Is the bibliography complete ?) Is the contribution original ?Is the contribution original ? Is the paper technically sound/correct ?Is the paper technically sound/correct ? Is the paper easy to read ?Is the paper easy to read ? Is the contribution significant ?Is the contribution significant ? What is the level of expertise of the evaluator / what is the What is the level of expertise of the evaluator / what is the
confidence of the evaluator in his judgementconfidence of the evaluator in his judgement
Discussion within the program or scientific committeeDiscussion within the program or scientific committee
Menu
Disclaimer Discovery vs. invention What is a research result in computer science ? What is a research paper in computer science ? General
structure(s) of a research paper in CS Searching for black boxes, traps and black holes How is research evaluated ?
And now ?And now ?
What conclusion ?
Read, read, read, read !Read, read, read, read !
Be positively criticalBe positively critical
Never admit supposed evidences. Always doubt (Descartes)Never admit supposed evidences. Always doubt (Descartes)
All questions are goodAll questions are good
Be modestBe modest
Bring your creativity !Bring your creativity !
