Research is about asking good questions and formulating precise experiments to answer those questions.
Conversation with Sasha
in that context, I'd say a self-contained question is a question phrased such that it defines the terms in the question or they are self-evident. apparently one of the hardest things in research is formulating good questions, so I'm still learning about this.
once you have a properly self-contained question with an unknown answer, you're pretty far along
probably easier to think about through example
unfortunately, a self-contained question loaded with definitions would probably only make sense to someone who is a world-expert on that research topic
thinking about the Manhattan project, on the surface it seems like there were two open questions:
Historical research is to find out whether anyone has answered those questions in the past.
In short, a given researcher is successful if they have identified a self-contained question and answered it sufficiently to be a building block for further work, either as a positive or negative result.
From Ask HN: Anybody Started a Research Institute, specifically this comment.
Q: What would you consider "real output", especially in contrast to "vanity metrics"?
This is an important question, and something we have considered when building our group.
My background is largely in experimental physical sciences, and one of the vanity metrics in my fields is publication in a Nature- or Science-family journal, or equivalent high-impact journal. In some cases publishing there has led to high-quality peer review and publication, but the pursuit of these papers drives many to over-sell or over-complicate the work. I tend to focus more on smaller studies which can be reasonably completely described, versus anthology publication.
But more directly, we measure output in three categories, in order of importance: people, validated concepts, and publication/dissemination.
For people, we want to develop a sense of integrity and curiosity, which means some mixture of freedom to roam and an ability to focus on defining and answering a concrete question. We want people to leave our group as good citizens and scientists.
Validated concepts take several forms. In the physical sciences it is often expensive to recreate an experiment, so performing and reporting the correct set of controls and validations is critical. A successful experiment is one which does not need to be repeated, because there is sufficient clarity and design to assure the next person that what was found is correct. Often this means obtaining a proper negative result which shuts down entire lines of inquiry. Especially since many projects require months or years of build-up, we must ensure that the foundation is solid.
As for publication and dissemination, our goal is that someone else would look at the work and consider building on top of it. In order for someone else to accept the work there needs to be a real trust or means for validation. We are open with our data, code, and methods, but much of the challenge lies in ensuring the dataset is valid and complete in the first place.
You are right to note that popularity drives academia, in the same way that charisma and a good pitch can get a startup off the ground. High citation counts, big-name grants, fancy titles, and time spent at big-name institutions are the standard metrics for success. I come from a somewhat outside perspective since I spent some time in start-up prior to this position, where our focus was more on actionable outcomes than positive results. But for the students and postdocs, these pressures for advancement are real and not easy to reckon with. I have no good answer, apart from attempting to impart personal ethics and aligning interests where possible.
The longevity and source of funding is a key factor in fighting the popularity contest. For our institute this is largely solved by solid political capital, which gives us some cover to pursue things without significant concern about short-term popularity. Long-term we want to be known for producing useful research, which others incorporate into their own work and become more productive.
In short, a given researcher is successful if they have identified a self-contained question and answered it sufficiently to be a building block for further work, either as a positive or negative result.