Experimental results on large-scale cyber-physical hybrid discussion support

Abstract

Purpose

This paper aims to present a preliminary experimental result on a large-scale experiment on a cyber-physical hybrid discussion support environment in a panel discussion session in an international conference.

Design/methodology/approach

In this paper, the authors propose a hybrid (cyber-physical) environment in which people can discuss online and also offline simultaneously. The authors conducted a large-scale experiment in a panel discussion session in an international conference where participants can discuss by using their online discussion support system and by physical communications as usual.

Findings

The authors analyzed the obtained date from the following three viewpoints: participants’ cyber-physical attention, keywords cyber-physical linkage and cyber-physical discussion flow. These three viewpoints indicate that the methodology of the authors can be effective to support hybrid large-scale discussions.

Originality/value

Online large-scale discussion has been focused as a new methodology that enable people to discuss, argue and make consensus in terms of political issues, social complex problems (like climate change), city planning and so on. In several cases, the authors found that online discussions are very effective to gather people opinions and discussions so far. Moreover, this paper proposes a hybrid (cyber-physical) environment in which people can discuss online and also offline simultaneously.

Citation

Ito, T., Otsuka, T., Kawase, S., Sengoku, A., Shiramatsu, S., Ito, T., Hideshima, E., Matsuo, T., Oishi, T., Fujita, R., Fukuta, N. and Fujita, K. (2017), "Experimental results on large-scale cyber-physical hybrid discussion support", International Journal of Crowd Science, Vol. 1 No. 1, pp. 26-38. https://doi.org/10.1108/IJCS-01-2017-0003

1. Introduction

Much attention has recently been focused on the experiments that gather large-scale opinion gathering (Malone et al., 2009; Klein, 2012). Research interest continues to increase in online crowd decision-making, which might become one of the next generation methods for open and public forums.

This paper presents preliminary experimental results on a large-scale experiment on a cyber-physical hybrid discussion support environment in a panel discussion session at an international conference. Our research group has been studying supporting technologies for online large-scale discussions. Online large-scale discussion has been focused as a new methodology that enable people to discuss, argue and make consensus in terms of political issues, social complex problems (like climate change), city planning and so on. By social experiments that collaborate with some town meetings introducing the Web-based forum system, we found that online discussions are very effective to gather opinions from the participants and discussions so far. Moreover, in this paper, we propose a hybrid (cyber-physical) environment in which people can discuss online and also offline simultaneously.

2. Background: large-scale discussion support system

To harness large-scale discussion intelligently, there are several critical factors including facilitation, incentives and understanding. These factors can make the entire discussion be held in fruitful ways and avoid negative behaviors that encourage “flaming”. “Flaming” means a hostile and insulting interaction by Wikipedia.

An open Web-based forum system called COLLAGREE (Ito et al., 2014; Sengoku et al., 2016a) has facilitator support functions and an incentive mechanism for the large-scale opinion gathering. They held a two-week long online town meeting, Nagoya Next Generation Total City Planning, where people in Nagoya City, Japan, used COLLAGREE to discuss city-planning to operate the municipal administration of Nagoya from fiscal years 2014 to 2018. In the two weeks, COLLAGREE gathered 266 total registered participants, 1,151 opinions, 3,072 visits and 18,466 views. The results demonstrated that COLLAGREE succeeded in gathering many opinions, while people understood the importance of facilitators.

Figure 1 shows a typical user-interface used by both facilitators and participants. The following are its typical functions, and we especially adopted ①, ② and ③ to support facilitators. ① shows agreement or disagreement analysis for a comment is shown. Facilitators can understand whether a discussion thread is positive or negative. ② shows keywords are highlighted so that facilitators can understand what keywords are being focused on and which are important. ③ shows facilitation tab from which facilitators can input their instructions to participants. ④ shows searching and reordering opinions and discussions. ⑤ displays issue tags that participants can add to each opinion and comment so that they can search for it afterwards. ⑥ is e-mail reminders for participants as well as reminders when related events happen.

Nagoya in Aichi Prefecture has over three million people. After three months of preparation with its city officers, they created an internet-based town meeting about the Nagoya city planning. Mayor Takashi Kawamura announced this project in newspapers and on TV as one actual town meeting of the Nagoya Next Generation Total City Planning for 2014-2018.

The experiment ran on COLLAGREE system during a two-week period from 12.00 on November 19, 2013 to 12.00 on December 3, 2013 with nine expert facilitators from the Facilitators Association of Japan. The participants discussed about their ideal city based on the Nagoya Next Generation Total City Planning 2014-2018.

As preliminary results over the two weeks, COLLAGREE gathered 266 registered participants, 1,151 opinions, 3,072 visits and 18,466 views. The total of 1,151 opinions greatly exceeded the 463 opinions obtained by previous real-world town meetings. From the questionnaires, both participants and facilitators realized the importance. However, facilitators had difficulty managing such large-scale discussions because this was their first experience (Ito et al., 201).

In the work (Takahashi et al., 2016), they have proposed an incentive mechanism for large-scale collective discussions, where the discussion activities of each participant are rewarded based on their effectiveness. With these incentives, we encourage both the active and passive actions of participants. Active actions include posting opinions, replying and agreeing and should be done for warming up discussions. Passive actions, which include getting replies and gaining agreement from others, are more highly rewarded in our system. Such passive actions suggest that one’s opinions have received interest or are supported by others. In other words, they submitted opinions that did not lead to impassioned responses from other participants.

Further, they extended their incentive mechanism so that the mechanism can take the quality of opinions into account (Takahashi et al., 2016) by using a natural language processing technique called BM2.5. By measuring the quality of opinions, we successfully incentivized participants to submit different opinions at the different phases in a discussion.

Discussion Tree (Sengoku et al., 2016a, 2016b) is a tree diagram that visualizes the flow of a discussion on the basis of the reply relationships in the conversations to make the discussion more efficient. A major difference of Discussion Tree from the argumentation map used in Deliberatrium (Gurkan et al., 2010) is that the Discussion Tree is generated automatically from chunk texts submitted freely by participants on a discussion forum. In addition, our Discussion Tree uses text-mining techniques to present the important keywords in discussion contents. These features avoid imposing a load on participants, while the argumentation map requests participants to manually create a logical argumentation structure.

3. Cyber-physical discussion support and metrics

The experimental results show the online discussion support worked well. Moreover, we found that a hybrid approach to support discussion seems also work well. In the experiment in the Aichi design league in 2015 explained above, we found that people were very excited to discuss online and also offline simultaneously.

Thus, as one methodology to support large-scale discussion, we propose the cyber-physical discussion support methodology. This approach could influence each other between the physical world and the online world.

In discussion, while some people can say their opinions physically, the other people tend to hesitate to say their own opinions. If the discussion is large, such silent people might be in majority. Our hybrid approach becomes a possibility to resolve that silent majority can say something online.

This paper proposes the following three metrics which represent how discussion has been supported physically and virtually in our hybrid environment:

1. Participants’ cyber-physical attention: This metric represents how participants can participate in cyber discussion and also in physical discussion simultaneously by measuring how participants’ attention relates the number of views and postings in online discussion.

2. Keywords cyber-physical linkage: This metric measures how contents are interrelated between virtual and physical discussions by measuring how keywords appeared in both discussions.

3. Cyber-physical discussion flow: This metric measures how discussion flows online and offline by measuring relations temporal behaviors between virtual and physical discussions.

These metrics are currently preliminary and need to be discussed and improved. However, as an initial attempt, it is quite new to propose this kind of metrics as far as we know.

4. A large-scale experiment and results

4.1 Setting

We conducted an experiment in the panel session in the international congress on advanced applied informatics (AAI 2016):

• Conference name: 5th International Congress on Advanced Applied Informatics (AAI 2016).

• Session name: International Forum on Collective Intelligence and ICT Future

• Date: 2016.07.12, 2.30 p.m.-4.30 p.m.

• Location: Kumamoto City International Center, Kumamoto, Japan.

We have one facilitator who is in charge of facilitating physical discussion and four panelists who discuss about the following themes. The third theme was not discussed due to the time limitation:

1. Artificial intelligence (AI) is taking the place of human intelligence, e.g. AlphaGo. how does AI impact human intelligence?

2. AI is used in economy and government administration. How does AI impact the social evolution?

3. AI is applied to our infrastructures, i.e. control of distributing electricity. Is AI robust enough? What are the conditions for AI’s robustness? (this theme was not discussed due to the time limitation.)

A commentary participant was encouraged to make postings to online discussion. He is a kind of the leading participants who lead the others’ discussion.

Table I shows the actual timeline of this panel discussion.

4.2 Three metrics for supporting hybrid discussions

We conducted an experiment to validate the efficiency of cyber-physical discussion support by using the proposed metrics: participants’ cyber-physical attention, keywords cyber-physical linkage and cyber-physical discussion flow.

Participants’ cyber-physical attention: This metric represents how participants can participate in cyber discussion and also in physical discussion simultaneously by measuring how participants’ attention relates to the number of views and postings in online discussion.

We will compare the participation of the real-world discussion with the number of views and postings in the virtual discussion, and found that there is correlation between them. We will show the details of the results in the experimental results session. Here, we explain the experimental settings.

To measure the attention of participants, we installed several high-quality video cameras in the discussion room so that we can record the whole participants’ behaviors. Figure 2 shows the concrete arrangement of the cameras. We installed one camera for recording the stage and three cameras for recording participants.

We combined these three videos recoded by the three cameras with the software Final Cut Pro X by Apple so that we can easily recognize the situations both of panelists and participants. Also, we put time stamps to enable temporal analysis and comparisons between posting/viewing in the virtual world and discussion in the physical world. Figure 3 shows a one-shot of the combined movie-file.

We extracted pictures for each 20 min from this movie-file.

The participant is defined as the person who gives attention, that is, is attending, to real-world discussion if he/she satisfies one of the following conditions:

• he/she is looking ahead on the stage where the facilitator or panelists are there; and

• he/she is looking at the questioner when there is a person who is asking a question.

Also, we assume the participant is participating in the virtual world discussion except for the above situations. We counted the above situations for each 20 s in the video, and sum up for each 5 min. Figure 4 shows the rate of the number of participants who are attending the real-world discussion, namely, looking forward or making a comment to discussions in the real world.

We compared the above participation of the real-world discussion with the number of views and postings in the virtual discussion and found that there is correlation between them. We will show these results in the experimental results session.

4.3 Keywords cyber-physical linkage

This metric measures how contents are interrelated between virtual and physical discussions by measuring how keywords appeared in both discussions in this paper. Ideally, this interrelation should be moderate, and different ideas from different perspectives should be generated online and offline.

We counted the frequency of the appeared keywords from the discussion among panelists recorded as texts, and also from the contents in the online (virtual) discussion. We extracted keywords manually while ignoring non-sense words and same-meaning words and ranked top 50 keywords by using BM25 algorithm (Robertson and Zaragoz, 2009). Table II shows the ranking of the keywords.

Based on the above scores, we found that there is efficient correlation between real-world and online discussions. The details will be shown in the experimental result session.

4.4 Cyber-physical discussion flow

This metric measures how discussion-flows interconnected online and offline by measuring relations temporal behaviors between virtual and physical discussions. In this paper, this is called “Cyber-physical discussion flow”. We have been analyzed several types of relations between real-world and virtual discussion. Then, we found that there is some correlation between the number of people who are looking ahead, and the number of views after 5 min after 5 min. Figure 5 shows the temporal data about the number of participants who are looking ahead and the number of views online. The details of the analyzed results will be shown in the experimental result session.

4.5 Evaluation and analysis

We evaluate the proposed three metrics: participants’ cyber-physical attention, keywords cyber-physical linkage and cyber-physical discussion flow shown in Section 4.2, 4.3 and 4.4, respectively, by calculating correlations based on the gathered data.

The parameters for calculating correlation coefficients are described as follows:

• Looking ahead: The number of participants who are looking ahead in the real-world.

• No. of views: The number of views online.

• No. of postings: The number of postings online.

• Length of No. of characters per post: The average number of characters per a post online.

Table III shows the Person correlation coefficients and significance probabilities (both sides) for each pair of the above parameters.

Based on the calculated results in Table III, we analyze the three metrics as follows.

4.6 Participants’ cyber-physical attention

This metric represents how participants can participate in cyber discussion and also in physical discussion simultaneously by measuring how participants’ attention relate to the number of views and postings in online discussion.

In Table III, the correlation between looking ahead and views is negatively significant. This means the number of views of Collagree system increases when the participants do not give any attention to real-world discussion, i.e. they do not look ahead, or vice versa. Namely, the participants always gave attention to real-world discussion or online discussion. From this result, we can conclude that the participants continuously attended the real-world or online discussion.

In the classic style panel discussion, i.e. only physical discussion, participants tend to be difficult to keep their attention or incentive to participate in the discussion if the discussion theme does not fit to their interest. Our methodology can overcome this situation and succeed to keep the participants’ motivation and attentive during this discussion session.

4.6.1 Keywords cyber-physical linkage.

This metric measures how contents are interrelated between virtual and physical discussions by measuring how keywords appeared in both discussions. The correlation value of top 52 keywords between online and real world is r = 0.339 (p = 0.024), and it is significantly correlated. Further, the top 54 keywords in online and real-world keywords is r = 0.342 (p = 0.045), and it is also significantly correlated.

These results show that the keywords in online and real-world are correlated. But the value of correlation coefficient is not higher. This means that discussion contents were somehow related but not completely the same. Namely, we can say that the discussion contents in virtual world were different from that in the real world. This contributed to the above participants’ cyber-physical attention as well.

4.6.2 Cyber-physical discussion flow.

This metric measures how discussion flows online and offline by measuring relations temporal behaviors between virtual and physical discussions. Table III shows the temporal changes of the number of looking ahead (real world) and the number of views (online). We can say that the number of views increases 5 min after the number of looking ahead decreases. But, we cannot find the opposite situation. Namely, there is not the case that the number of looking ahead increases 5 min after the number of views increases. This implies the following story: when the participants are interested in the real-world discussion, they look ahead. And then, they tended to look into the virtual world discussion (the number of looking ahead decreases). Then, after 5 min, the number of views (online) increases.

Also, we analyzed the relation between the number of looking ahead (real-world) and the number of posting (online). The correlation value of the number of looking ahead and the number of posting at the same time is –0.275 (no significance). The correlation value of the number of looking ahead and the number of posting before 5 min is positively significant (r = 0.504, p = 0.055). There is no correlation between the number of looking ahead and the number of posting after 5 min (r = –0.329, n.s.). Namely, it can be said that the participants give attention to the real-world 5 min after posting online. But, there is no relation in the opposite case. This implies that the participants have interest to see how their posting make effect to the real-world discussion. Also, it implies that real-world discussion did not incentivize posting activities online. This could imply that the cyber-physical discussion flow would be asymmetric relation, and further investigation would be required. Also, we found that looking ahead activity often happens after posting online. These preliminary results demonstrate the possibility that there are cyber-physical discussion inter-connected flows.

5. Conclusion

In this paper, we proposed a hybrid (cyber-physical) environment in which people can discuss online and also offline simultaneously. We conducted a large-scale experiment in a panel discussion session in an international conference where participants can discuss by using our online discussion support system and by physical communications as usual. We analyzed the obtained date from the following three proposed metrics: participants’ cyber-physical attention, keywords cyber-physical linkage and cyber-physical discussion flow.

We found that our methodology succeeded to keep the participants’ attention active and continuous during this discussion session by measuring the participants’ cyber-physical attention. Also by measuring keywords cyber-physical linkage, we found that the keywords in online and real-world are correlated and somehow linked. But discussion contents were somehow related but not completely the same. Namely, we can say that the discussion contents in virtual world were different from that in the real world. By measuring cyber-physical discussion flow, we found that the number of views increases 5 min after the number of looking ahead decreases. A possible explanation would be that when the participants are interested in the real-world discussion, they look ahead. Then, they tended to look into the virtual world discussion (the number of looking ahead decreases). And, then, after 5 min, the number of views (online) increases. We found that looking-ahead activity often happens after posting online as well. These preliminary results demonstrate the possibility that there is cyber-physical discussion inter-connected flows. These are the preliminary results, and we need to do more investigations as future work.

Figures

Figure 1.

User-interface

Figure 2.

Camera arrangement

Figure 3.

Panel discussion recorded for analysis

Figure 4.

The rate of the participants who are attending the real-world discussion

Figure 5.

Temporal data about # of looking ahead and # of views

Table I.

Timeline of the panel discussion

15.48-15.59	12 mins	System explanation
15.59-15.15	15 mins	Asked Theme 1 and responded by a panelist (Katsuhide)
15.15-15.21	6 mins	Asked Theme 2 and responded by a panelist (Andrew)
15.21-15.33	12 mins	Asked Theme 3
15.33-15.41	8 mins	QA by participants, question from the participants and facilitator encouraged to write questions on Collagree
15.41-16.04	15 mins	Panelists discussed about the opinions posted in Collagree
16.04-16.06	2 mins	Explanation of discussion
16.06-16.15	9 mins	Panelists’ final comments and facilitator wrapped up

Table II.

Scoring result of keywords

Ranking	Physical space	Value	Cyber space	Value
1	Question	8.081766	Emotion	0.128389
2	Example	7.271388	Name	0.128389
3	Opinion	7.232191	Being	0.125664
4	Point	7.02837	Human	0.111402
5	People	6.513154	Problem	0.110833
6	Being	6.360155	Idea	0.110833
7	Issue	6.307973	People	0.095043
8	Emotion	5.833828	System	0.09154
9	Robot	5.210628	Something	0.09154
10	Intelligence	5.194714	Robot	0.085709
11	Future	4.600353	Use	0.085709
12	Responsibility	4.554622	Art	0.084179
13	Answer	4.508041	Person	0.077811
14	System	4.422075	Post	0.077811
15	Thing	4.392325	Rule	0.072855
16	Job	4.365512	Point	0.072855
17	Research	4.127652	Company	0.063957
18	Government	3.968191	Job	0.063957
19	Brain	3.911277	Law	0.063957
20	Infrastructure	3.784719	Work	0.063957
21	Problem	3.609552	Ethic	0.063957
22	Something	3.609552	student	0.063957
23	Function	3.546963	Future	0.063957
24	Comment	3.546963	Research	0.059806
25	Country	3.509786	Economy	0.059806
26	View	3.495502	Responsibility	0.059806
27	Thread	3.46965	Situation	0.059806
28	Law	3.46965	Brain	0.059806
29	Human	3.461004	Thing	0.054063
30	Sort	3.395105	Issue	0.054063
31	Datum	3.34411	Advantage	0.054063
32	Course	3.310993	Government	0.054063
33	Word	3.306047	Account	0.054063
34	Machine	3.19705	Partner	0.054063
35	Panelist	3.086104	Utility	0.054063
36	Information	3.033029	AI	0.054063
37	Nation	3.033029	Talk	0.054063
38	Feeling	3.033029	Topic	0.054063
39	Accident	3.033029	Electricity	0.054063
40	Floor	2.987705	technology	0.049937
41	Place	2.975766	Software	0.049937
42	Impact	2.905097	Creation	0.049937
43	Application	2.905097	Imagination	0.049937
44	Knowledge	2.905097	Novel	0.049937
45	University	2.905097	Profit	0.049937
46	Purpose	2.799984	Function	0.042917
47	Life	2.799984	Threat	0.042917
48	Factory	2.742671	World	0.042917
49	Technology	2.710613	Drive	0.042917
50	Game	2.61704	Car	0.042917

Table III.

Results

	Looking ahead	Views	Postings	Length
Looking ahead
PC	1	−0.449	−0.149	0.027
Sig		0.071	0.569	0.918
Views
PC	−0.449	1	0.235	0.073
Sig	0.071		0.363	0.782
Postings
PC	−0.149	0.235	1	0.646**
Sig	0.569	0.363		0.005
Length
PC	0.027	0.073	0.646**	1
Sig	0.918	0.782	0.005

Notes: PC = Pearson Correlation; Sig = Significance (both sides)