[ODE] <Survey>: ODE - Present and Direction.

[Mike Reinstein]

gl:

Yes, its true that you have to go through more work to actually READ the 
results of polls. But thats where polls and surveys are most helpfull. By 
adhering to an automated poll survey collection system you lose a lot of 
data. These automated polls/surveys are the hottest thing right now as 
demonstrated by slashdot, along with every portal/ web content management 
based web site. But that doesnt mean that the polls are accurate or 
representative.


If you want to take a defensive stance on your poll, thats your choice. It's 
just a warning to everyone that the results of the poll are only as good as 
the design of the questions and the format in which you allow people to 
answer them.  I worry that major contributors on this project will use the 
survey for considering the near/long term project roadmap; if they doit will 
be based on unstructured questions be based on poorly accumulated 
statistics.

I'm not speaking as a casual observer of the phenomenon.At work,I'm 
responsible for managing our support system which includes several user 
forums, content management systems, weekly technical articles and 
phone/email based support. Over 1,000 developers frequent the site and our 
company has experienced the down side of relying on poorly planned 
polls/surveys too heavily on a first hand basis.

I implore ALL people who might use the  results of these surveys to consider 
what other questions need to be asked, and spend a little time with 
this...In much the same way adequate design needs to be done to design 
software before coding it,you need to spend adequate time designing polls 
(especially the ones that are factored into important decisions) There are 
entire semester long courses dedicated to poll/survey question creation, 
design and analysis, and there's a reason for it. Its not a joke and its not 
trivial. This isn't to say it requires a lot of formal training to do it 
adequately, but it does seem to indicate that throwing together an auto 
poller that just adds numbers isnt going to get you very far.


kind regards,

Neko


>From: "gl" <gl at ntlworld.com>
>To: <ode at q12.org>
>Subject: Re: [ODE] <Survey>: ODE - Present and Direction.
>Date: Wed, 12 May 2004 20:54:38 +0100
>
>
> > Question 1: The middle ranges are too qualitative; While 
"perfect" and
> > useless are clear indicators of how someone feels about ode, the 
middle
> > ranges are ambiguous and meaningless.. for example,  how do you 
quantify
>the
> > difference between "needs some work" and "needs 
plentyof work"?
> > suggestions for improvement:
>
>I disagree - the differentiation is clear to me.  Looking at the 
results, it
>also seems clear to the participants.  'Needs some work' means it's 
mostly
>living up to expectation, 'needs plenty of work' means it isn't.
>
> > Add a short answer section that allows people to comment on why 
and how
>ode
> > is good/bad. While this does not allow you to tally results in a 
nice
> > looking progress bar % meter, it lets people generate meaningful, 
tangible
> > feedback that can be quantified. (eg  "Needs Some work, 
capped cylinder
> > supoort is blah blah blah [insert comment here]")
>
>Sure - if you're willing to read through all the replies manually, and 
write
>up a summary : ) - I'm not.
>
> > Question 2:  This is probably ok, but I might add a section 
allowing
>people
> > to specify which specific platform they are using for the last 
option.
>
>See above  : ).  Seriously, text input would be useful is someone were
>willing to study it all.  But then we might aswell pose questions on the
>mailing list, and study all the answers.  The whole point was to cut 
through
>the long-winded answers, and often resulting discussions, and just 
quickly
>find out what the deal is in general.  If that then generates 
discussion, at
>least we have some basics out of the way.
>
> > Question 3: The wording here is misleading..."Where does ODE 
failthe
>most?"
> > implies that a single answer response is expected. Limit this to 
allowing
> > only 1 entry to be chosen.
>
>Sure, but then you only get a single failure.  I could have done the 
'top 3
>failures' thing, but I wanted to keep it short.  Again the answers seem
>pretty clear to me.
>
> > As with question 1, allowing people to provide information 
describing WHY
> > that particular item fails the most is important because it allows 
people
>to
> > quantify and explain their choice.
>
>Above.
>
> > Question 4: This question should be eliminated, it is redundant 
w.r.t.
> > questions 5-7
>
>Kinda true, but I wanted an easy way to see if certain things weren't of
>interest at all, no matter what the priority.
>
> > Question 5: Change the wording to "which future development 
is top
>priority > for you?"
> > Questions 6,7: See q5
>
>I don't think anybody got confused.
>
> > Question 8: Split this up into 2 questions; q8 contains list of 
existing
> > primitives that are important, q9 includes list of future 
primitives that
> > are important (be sure to provide a brief explanation of future 
primitives
> > so more people will know what they are selecting)
>
>Why the seperation?  The questions is basically saying 'in an ideal 
world,
>what primtives would you use?'  Whether they exist or not isn't the 
point.
>
>Re. description, the text limit is pretty harsh - I actually had to cut 
a
>few lines down.
>
> > Question 9:(will be question 10) this is fine, although I dont see 
this
> > providing much insight into anything...
>
>It shows that plenty of people would like to see 'beyond rigid-body 
core'
>features.  Again, it's just an overview - there's plenty of scope for
>getting into detail in future surveys or on the list.  Hey, I wasn't 
even
>sure anybody would bother filling this one out : ).
>
> > This may seem like nitpicking, but I assure you it's not. 
Carefully
> > developing a survey is critical because otherwise you accumulate 
feedback
>on
> > an issue, attempt to satisfy the needs derived from the survey 
results,
>and
> > that in turn inevitably leads to converging on the wrong 
solution(s).
> > If the survey system you are using does not allow for these types 
of
>survey
> > collection controls,I would argue that we should go with a another 
survey
> > system that does allow information to be collected in this 
fashion. (There
> > are dozens of free systems to choose from, or we could write 
one...we are
> > talking about to mysql tables and a handfullof php scripts)
>
>Look, if you want to design an exhaustive survey, carefully tuned (and I
>agree you can tune this very carefully), then please do.  What I wanted 
to
>do is make a very quick, easy to use, survey that (often busy) people 
would
>actually fill out, covering some of the basics - keeping it simple was
>deliberate.
>--
>gl
>
>_______________________________________________
>ODE mailing list
>ODE at q12.org
>http://q12.org/mailman/listinfo/ode

_________________________________________________________________
Stop worrying about overloading your inbox - get MSN Hotmail Extra Storage! 
http://join.msn.com/?pgmarket=en-us&page=hotmail/es2&ST=1/go/onm00200362ave/direct/01/