10

The following suggestions address the case where users treat the forum like a social media site (eg, reddit or slashdot), and post "opinion" answers, with no attempt to suggest that they have any grounds or basis for their assertions (example, example, example, example, example). Related to this, and possibly covered by the same option, are "data point" (anecdotal) answers that merely share personal experience (example, example, example).

Note: This case is different from the one mentioned here - the latter implying that answers make assertions based on unsourced / uncited evidence (and are not recommended for deletion), which is perhaps due to laziness rather than lack of scientific rigour, whereas the option I'm proposing is for answers that make assertions based on personal opinion / experience / belief, with no indication that supportive evidence even exists (and should be recommended for deletion).

I notice that psych.meta is heavily focused on question quality, and has virtually no discussion on answer quality. This is exemplified by @StevenJeuris's excellent comprehensive summary (and continuing discussion), that links to dozens of posts on questions. I suspect this focus is at least partly due to the way psych.SE's performance is tied to reducing the ratio of questions to answers, so there is incentive to reduce low quality / unanswered questions, but less incentive to reduce low quality / bad answers.

However, the purported benefit of reducing questions for attracting professionals to the community is only speculative. The science-minded of us know that to really find out how much it would benefit the community to tighten down on questions is to test it. I suggest that the quality of answers may also play a part in attracting or repelling professionals. That said, I hope this post might be useful for starting a discussion.

Note that the example answers I include below are still there as of this writing (several are upvoted and one of them is even an accepted answer!), so I don't think the current system of flagging, downvoting, and moderating is as effective as it could be. I myself have been discouraged from flagging or recommending for deletion non-scientific answers on the grounds that they don't violate any of the forum's policies, and would like to see that changed. This is my proposal:


The "flag" link for Answers has the following options:

  1. spam
  2. rude or abusive
  3. not an answer
  4. in need of moderator intervention

Additionally, when reviewing Low Quality Posts (ie, flagged answers), the "Recommend Deletion" button has the following options:

  1. No comment needed
  2. This is commentary on another post, not an answer
  3. This is a “thank you” comment
  4. This is an “I’m having this problem, too” comment
  5. This is a different question posted as an answer
  6. This is a link-only answer (and not spam)

My suggestions:

  1. I would like to see a "flag" option: "not a science-based answer" - in addition to or in place of the current "not an answer" option - to discourage / remove unscientific opinion answers.
  2. "Recommend Deletion" options 2, 3, and 4 are basically the same (this is a comment, not an answer - common for new users without sufficient reputation to comment), and should be merged as there is little reason to differentiate between them.
  3. In their place, I would like to see the same option: "not a science-based answer" - to support the proposed "flag" option.

Thoughts?

  • 1
    My thoughts are "yes please", but as usual I'm interested to hear what the rest of the community thinks. . . – Krysta Aug 3 '15 at 14:29
  • 1
    I always place badly referenced question under the "This is commentary on another post, not an answer" option. So far I've had no troubles with doing it this way. However, I can understand that a clearer categorization may be beneficial. A "yes please" for me too :) – Robin Kramer Oct 9 '16 at 11:16
  • By the way, sometimes people give a case-study like answer, by telling how they (being the subject of interest) behave in such situations (e.g. cogsci.stackexchange.com/a/14160/11318 or cogsci.stackexchange.com/a/16028/11318). Although these answers are not scientific based I do think they are very valuable, assuming they are real. What do you think about these kind of answers? – Robin Kramer Oct 9 '16 at 11:20
  • 2
    @RobinKramer These kinds of answers are exactly the kind I would like to see done away with. Clearer categorization is not just for convenience, it is also for making it clearer what kinds of answers are discouraged (which is apparently not obvious, as your comment suggests). Anecdotal answers (in whatever form) are problematic for many reasons, including placebo effect, lack of control, personal biases, and not being open to peer review. If we are seriously discussing attracting professionals to the site, then I do think they should be removed. – Arnon Weinberg Oct 9 '16 at 16:46
  • 1
    Good point. Case study answers should (and can) also be cited indeed. If more people agree, I'd say Let's make work of this. – Robin Kramer Oct 9 '16 at 16:54
  • Just for the sake of completeness, @RobinKramer also added a collection of meta posts on answer quality, which simply did not get incorporated into the 2016 review yet. – Steven Jeuris Jan 9 '17 at 22:53
  • The first personal experience example contains an accepted answer from a high rep mod here where there is just anecdotal evidence given and no citations. 100 bounty is also given. If this person were to have their answer removed, their rep would drop by 330 points. What do we do here considering also that this answer is from 2013 when (I believe) this was not as strictly enforced? – Chris Rogers Feb 5 at 16:03
  • @ChrisRogers Good question. I believe it is common at other stacks to close questions that no longer meet new standards introduced since they were active. This way users keep their scores, the post remains for historical reference, but is marked (preferably with an appropriate comment) as no longer fitting the site. Since you have potentially a more general question about posts that no longer meet current standards, maybe it's worth posting it separately from this thread? – Arnon Weinberg Feb 5 at 18:29
1

I agree that this issue needs sorting in some way, but there is a problem using the low-quality flag approach when we have answers upvoted by others as the low-quality flag isn't given as an option.

The question arises whether a separate flag can be created in the first selection list for sites like us and MedicalSciences without affecting non-scientific sites which don't have the references requirements.

If that is possible, then I would go down that route and have that put in.

1

Unscientific Opinion:

The problem with opinion should be obvious: Everyone has one, and that doesn't help users get answers. We already ban questions that solicit opinions, so it stands to reason that we should ban opinion-based answers.

One question that has come up is: Why are the opinions of experts more valid than the opinions of lay people? The answer, I believe, is that strictly-speaking, they are not. A good answer relies on evidence. In practice, experts are more likely to be familiar with available evidence, and rely on it to construct their views, but any lay person who does the same can provide equally valid answers. In fact, this is exactly what allows a stack like this to exist, with contributions from both professionals and lay people.

Now, just having references or citations is neither necessary nor sufficient for good questions or answers, but evidence-based answers can at least be peer-reviewed and scrutinized for accuracy by other users, where opinion-based answers cannot. It is still important for users posting answers to vet their sources, for example to ensure that they actually support their claims, but even if they fail to do this, or fail to provide references at all, at least evidence-based answers can be validated.

Anecdotal Evidence:

One common argument is: Perhaps personal anecdotes are not useful on their own, but surely in aggregate they provide useful evidence upon which to base conclusions? The answer, I believe, is absolutely not. One only needs to look at the huge amount of medical quackery, and corresponding mental health quackery, to see why.

If anecdotes are bad evidence, then multiple anecdotes are not better evidence - in fact they are worse evidence - they simply multiply the bad effect. The result is the mistaken belief that good evidence has been collected to determine effect, when in fact, good scientific evidence shows that the effect is possibly the complete opposite.

There is a massive gap between scientific evidence and aggregated anecdotal evidence. Probably the most important distinction is the lack of controls in anecdotes. For example, a person evaluating a particular treatment has no comparison point to base their evaluation against - ie, the outcome of a no-treatment condition. For all they know, the no-treatment condition could have had better results. Good scientific evidence is controlled and randomized, and ideally double-blinded, multi-sited, and pre-registered. Additionally, the statistical analysis that researchers apply to their data is far more complex than just aggregating data points - it involves controlling for confounding variables and many other nuances.

Action Items:

Regardless of whether we add a new flag for this scenario or not, opinion-based and anecdotal answers can be downvoted, and/or flagged as "not an answer" or "very low quality", and be deleted accordingly.

Please vote up this answer if you would like to see the suggestion proposed in the question implemented; downvote if you don't.

  • 1
    So where there is an opinion-based answer without citations, we flag it as not an answer even if it is? In other words, the question may provide a direct answer to the question but there are no citations backing claims – Chris Rogers Feb 5 at 15:40
  • @ChrisRogers For now, yes. If there is support for a new "not a science-based answer" as proposed, then we would use that instead, but for now, our options are limited. – Arnon Weinberg Feb 5 at 18:23

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .