The rankings tab

bodawei
June 26, 2010, 05:43 AM posted in General Discussion
Some people have said that they could do without the 'rankings' tab - my view is that more data about site use would be better than less. A focus on the top ten users is of limited use to anyone. Anyway, for those that have more than a smidgin of interest, the first half of 2010 has seen a 100% increase in comments over 2009. Unfortunately, my sense is that the majority of the increase is debate over changes to the site. Some other matters that the data shows (assuming it is accurate, and noting that we only have data for the top ten users in any period): 1. Heavy users tend, on average, to stay around for about two years. Clearly it takes its toll. :) 2. Teachers/other ChinesePod staff are very light users compared to the poddies. Very light. Clearly ChinesePod does not put much instructional effort into the community, and perhaps that is a good thing. I think the podcasts are the real strength of this site. 3. While ChinesePod staff have little time on the community boards, a couple - Amber (before my time) and 'PoemsWithPete' Pete made a big impact. Amber just scrapes into the 'top 10' of the past 4 years, and Pete didn't even make it. [Note that the data is incomplete - this only counts your posts if you made it into the top 10 list for any given year.] But anyway, they provided quality if not quantity. I wonder if poddies can make other observations about these results.
Profile picture
pretzellogic
June 26, 2010, 10:19 AM

10 people are responsible for between 70-95% of the comments. (educated guess)

We don't know if frequent posting drives loyalty, or if those loyal to the site are frequent posters.

Profile picture
zhenlijiang
June 26, 2010, 12:10 PM

Bodawei, on "heavy users"--there must be many more heavy users than heavy commenters.

Yes for me giving feedback about changes to the site has definitely increased the number of my comments.
But the main cause is actually threading. Whereas before threading I would often address one comment to two or three different people, now I Reply to each one individually almost always. This is especially true when I have started a post myself. Threading has created many little pockets (out of some of those grow subthreads) in every discussion.

I don't know that it's clear that ChinesePod are not putting much instructional effort into the community; I do think they are (naturally) putting much instructional effort in the teacher services, which to me explains why so many changes that appear to some of us to compartmentalize and fragment the community have been made recently.

There are the podcasts, and there is this site. And now expanding teacher services. It's a different business model from the days Amber was here.

Profile picture
chris

Hi Zhenlijiang. I agree with your first sentence. I consider myself a 'heavyish' user again in the last month or so (my usage ebbs and flows depending on other commitments), probably averaging at least 2 hrs a day on the site, working through my backlog of lessons. But I'm a very light commenter, with maybe only 10 posts in the last couple of months. I suspect there are many out there like me and probably many more that don't post at all. Chris.

Profile picture
pretzellogic
June 26, 2010, 12:50 PM

we also don't know why people drop off the list of top posters. Reasons include (but not limited to):

- dissatisfaction with cpod

- cpod accomplished its goal and the site is no longer needed

- can't afford the subscription price

- other priorities are now higher that learning Chinese

- learning Chinese with cpod, but deciding posting is no longer a priority.

Profile picture
go_manly

Or banned from posting!

Profile picture
bodawei

You are right, there are at least a couple of 'heavy commenters' (thanks Zhenlijiang for that clarification) who have reputably had their posting rights revoked: one ranked second in 2008 (before my time) and another ranked 4th in 2009.

Profile picture
pretzellogic

I didn't see anything untoward from the 2009 frequent poster's comments that would suggest he needed his posting rights revoked. Interesting.

Profile picture
paulinurus

The often declared intention of Cpod for making all these changes was to encourage the people apparently intimidated by the heavy commenters to come out of the wood work and participate in the community. Not sure to what extent this objective was accomplished. Probably not much since there has not been an influx of new posters, especially on the community conversation boards, and certainly insignificant in the 100% increase. It'll be interesting to see the stats for the 2nd half this year when I suspect the numbers will come down due to tiredness of the heavy commenters to assist in "tweaking" the technology (as I noticed two previously heavy commenters have now cut down.)

Profile picture
bodawei

I don't know the story in either case. I guess that there are a number of criteria that ChinesePod would apply such as damaging the site's reputation, and being abusive to other poddies. But we are not likely to be told the specifics. It's a shame about the 2009 case - he gave a lot of Chinese language learning value to the site. Helped me a lot.

Profile picture
pretzellogic
June 27, 2010, 03:05 AM

paulinurus' comment made me want to do a quick spreadsheet analysis:

(assumptions:

- 5 lessons per week for 52 weeks a year.

- posters post at a constant rate)

The top 10 posters in each year created:

2007 -8964 comments

2008 -12867 comments

2009 - 13072 comments

through ~ 6 months of 2010 -10947 comments

for 2007-2009, divide by 2 to get a sense of number of comments through July 1 and then compare to current performancefor 2010:

2007 -4482 comments

2008 -6433 comments (43.5% increase)

2009 -6536 comments (1.6% increase)

2010 - 10947 comments (67.5% increase)

Significant increases in the number of2008 and 2010 comments could be driven by many things. But what is interesting is that with almost 11000 comments in 2010, it's easy to see that even if there are significant numbers of new posters, their comments could be buried in even 10% of the total number of comments for 2010. In other words, let's assume that to date, there are a total of 12000 comments on the site.With frequent posters making 10972 comments, that would leave ~1000 people to make one comment (unlikely, but conceivable) or maybe 100 people to make 10 comments (possible). We wouldn't really know for sure what happened until we went through the site lesson by lesson and counted the number of people commenting, and the number of comments each individual made.

Ok, let me tweak the above: take out about 500 comments, as they've been made by people we know, but don't see that often (call them infrequent posters; people like John and the rest of the cpod staff, and also people like a1pi2, chris, tgif, etc.). Further guestimate is that these people are responsible for 10 comments or more. This starts pointing to the idea that frequent posters are only responsible for maybe 50-75% of the comments on the site, and maybe cpod's strategy worked.

Profile picture
bodawei

Hi pretzellogic.

You could be right in your assertion that 'maybe cpod's strategy worked' but unfortunately we don't have the data. At least we don't have it neatly summarised in the Rankings tab - hence you had to make a number of brave assumptions. You would need to track through all of the Lessons and community boards manually counting posts (no-one is that crazy.)

If the idea was to get more people posting, and these newcomers generally don't make it to the top 10 posters board, we are only guessing what was happened pre- and post- the ChinesePod initiative called threading.

If it was a success story there is no reason why ChinesePod should not tell us the story (although there is a cost in compiling the information, and we have already been critical about the obsession with non-Chinese-learning activity.)

Profile picture
go_manly

I did a quick survey. I say survey, because I didn't check for all comments in each thread. Instead, I took the most recent comment (the one showing on the Conversations page) for the 300 most recent threads. This takes us back to May 25, and I think it is a worthwhile sample.

From the posters of these 300 comments, I excluded:

1. Frequent posters. These are those that occur in the top 10, and those that have posted more than 500 comments in total. I have 17 users in this list.

2. CPod staff. (47 comments of the 300)

3. Users whose first and last comments were separated by 7 days or less. This is to exclude trial members. (30 comments of the 300)

That left 52 users, which I don't think is too bad from a sample of 300 comments (reduced to 223 by the removal of groups 2 & 3 above).

This is how long those 52 users have been posting comments:

1. more than 2 years - (13)

2. between 1 and 2 years - (12)

3. 6 months to 1 year - (11)

4. 3 to 6 months - (8)

5. 1 to 3 months - (3)

6. 1 week to 1 month - (5)

I'll let you draw your own conclusions from that,

Only 22 of those 52 have made more than 50 posts.

Profile picture
bodawei

go_manly

The most obvious conclusion to draw from these numbers is that 'post threading', participation by new people is fairly insignificant. ('New' defined as people who have been around for 3 months or less.) Not a surprising result - if you came recently to the site you would be overwhelmed (or underwhelmed perhaps). Not whelmed, anyway.

Though, if the idea is to measure the impact of the ChinesePod initiative called threading (which was supposed to encourage the lurkers and more generally to boost business) - we need to compare 'with threading' to 'without threading'.

Thanks mate for your systematic approach to this - adds a bit without doubt.

Profile picture
paulinurus

A small sample may also be telling, however I think stats breakdown of users associated with the 300 comments is required to draw some conclusions. For instance, if I estimate 82 users ( groups 1 to 3 :17 frequent posters, 8 staff, and 5 trial users) associated with the 300 comments, and 70% of the 300 comments are from groups 1 to 3, this leaves 90 comments to the 52 users, an average of 1.7 comments per person. Out of the 52, I would estimate that people who have been around 6 months or longer would not be the ones "intimidated to post due to presence of frequent posters", which leaves a suspect of maximum 16 people who have now come out of the woodwork to post after gathering courage from the implementation of threading. I doubt that 100% of the 16 people are the so called "timid posters. Lets say 50%, or 8 people.

So Cpod's added 8 people to post who wouldn't have done so if not for the threading initiative i.e. a success rate of 8/82 9.7% of people participation, and 8x1.7/300 4.5% of increase in commentaries.

What's pointing is that the trial users (were not afraid to post) contributed 30 comments out of the 300 i.e. 10%, which could mean that my above 50% estimate is too high, and should be more closely be 4 people (based on the relationship of comments per person) freed up by the threading implementation and coming out of the woodwork to post. If this is the case, the success rate then is 4.8% of increased people participation and 2.3% increased commentaries.

Profile picture
go_manly

I'm not going to attack or defend your comment above, because I don't really get the gist of it, or the number work. But there are a couple of points I should make.

"90 comments to the 52 users, an average of 1.7 comments per person"

Don't forget, that is 1.7 comments from the sample of 300. If there were say 3000 comments made in total in that period, that would equate to 17 comments per person amongst these 52.

"the trial users ... contributed 30 comments out of the 300"

I made an assumption that all people who posted over a period of 7 days or less were trial users. This is unlikely to be the case, and may in fact be way off base.

Another stat, which may be interesting: These 52 users have an average of 15 months since their first comment, and have averaged 69 comments each over their time at the site (total comments, not a sample). So each one averages about 1 comment per week.

I'm not sure what to make of that, except that I wasn't aware there were so many hangers-on. And of course, there would be other infrequent posters I didn't pick up in my sample.

Profile picture
xiao_liang
June 30, 2010, 10:16 AM

I got lost in all the analysis above, but my one comment (which I made directly to chinesepod as well) was that I really would like to see Chinesepod employees contribute more to our discussions. 

 

There are some really dedicated posters (most recently Catherine), and what is often missed by us is that a very high percentage of directly asked chinese-related questions in lesson threads are answered (at least, mine have been), usually by Lu Jiaojie, Connie etc. 

 

However it is possible, especially if you spend time reading the archives in old lessons like I do, to get the impression that the team doesn't really want to participate in the community in the way that they did previously. It's understandable I guess when posting megaliths like Pete and Amber leave, but no-one seems to want to pick up the slack.

Maybe we're too whiny and not enough fun any more ;-) 

I would like to see Jenny post a bit more. I'm sure I'm not alone. I'm sad that her corporate duties have taken away her time so much that she's not able to input as much!