Do you think your project would benefit from a real world deployment? Why or why not? What obstacles and opportunities are there for design in the world and with participants related to your project?
Permanent link to this article: https://www.gillianhayes.com/Inf231F12/r8-design-in-the-real-world/
27 comments
Skip to comment form ↓
Armando P.
November 28, 2012 at 7:29 pm (UTC 0) Link to this comment
My group project would absolutely benefit from a real world deployment. Since the project aims to improve the fitness of users and fitness is something that only improves over long periods of time, a field deployment would be essential to evaluate and improve the effectiveness of the project’s artifact (in this case a website). While usability testing can help evaluate and refine the website’s interface and workflows, it can do very little about measuring whether it is actually accomplishing the goal of improving the user’s fitness.
A real world deployment would allow us to monitor how often the website is actually used, the social implications it has (both within and outside the website), and hopefully changes in users’ fitness. The nice thing is that because the project’s artifact is a website, a lot of usage data can be captured automatically and changes throughout the deployment are easy to do. Additionally having data about the website’s usage along with “offline” data (e.g. repeated interviews, surveys, etc.) can yield some interesting insights into how the website affects users’ behavior.
However there would be some interesting challenges. One important one would be measuring fitness level and whether it has changed positively during the deployment. Given privacy considerations and expertise requirements, determining fitness level would be difficult. Additionally, it might be difficult to determine the sustainability of the change since participants might only be actively engaged during the deployment. Then there is the challenge of starting the deployment. The website’s benefit comes from building a sort of social network, which means the site needs a relatively large number of initial users to be beneficial and thus have its impact be measurable and/or observable. Similarly once the deployment is completed, there is the ethical consideration to make of whether to allow users to keep using the prototype or to take the site down despite the benefits it might be providing.
Surendra Bisht
November 29, 2012 at 5:08 am (UTC 0) Link to this comment
I agree with Armando’s view and would like to add the following.
Our project is a website based on a concept of social interaction to motivate people to exercise and improve their health condition. The evaluation through real world deployment will certainly benefit our project. It will provide us the insights related to the actual usage by the target users and how the website affects their exercise routine. Real world deployment will also help us to identify and understand the constraints in users’ actual context which would be missing in a laboratory setting. The possible constraints could be users daily schedule, privacy, health, or other social concern. The field study is also important to gauge the long term adoption and usage of the website. Users’ interaction with various features of the website can be automatically collected without burdening the users. Online survey at regular intervals posted on the website could be used to measure whether the website meets its goal.
Major challenge for the project is to encourage users to participate by registering and then ensure that they use the website regularly. We would require a group of users who are co-located in a city so that they can interact for exercise, sports, and other re-recreational activities.
Another challenge for the project is to deploy a robust website which otherwise will discourage the users from using it regularly. Deploying a robust web site will require substantial effort and time up front which could be a major obstacle for us. Timely maintenance and troubleshooting of the website during the period of study is also a big challenge.
Parul Seth
November 29, 2012 at 1:40 am (UTC 0) Link to this comment
Our project will benefit tremendously from a real world deployment. Despite the number of solutions that exist in the market for personal finance management, a lot of people, particularly students, are not using them. Based on our user research results, currently they mostly rely on their memory and spreadsheets for managing budget and expenses. But all are current requirements have been elicited based on the research from a subset of users and market studies. There is a huge scope for further contextualizing and improving our design. The real world will let us know the real worth of our product. This litmus test will surface the specific services of our product that people actually use, and the features in the design that are only serving as add-on. The perceived-theoretical usefulness, usability and overall experience of our system can be then transformed into practical, socially-aware solutions for actual problems in a real-world environment.
I understand doing a real world deployment is challenging in more than one ways. Pilot study or eating our own dog food seems attractive and do-able for our project, but the real test will be going out “in the wild”. For this we need to make the design robust enough for actual test by the untainted users. Another challenge arises from a component of our design which requires participation from various stores for tracking itemized expenses, we know this will be tricky, but it is really useful, creating dummy participant stores is one option, but whether this option is good for deployment in the real world is still questionable. Additionally, we will need to figure out the best possible way to collect, analyze, report and act upon the data that is carefully filtered out from the messiness and chaos of this kind of deployment. Finally, analysis of the fact that whether the tool is actually changing the user behavior, by making people spend and save prudently, is subjective and will be only possible over the course of time. For this, employing the verbatim principle, by collecting and showcasing direct user responses will offer credibility and may be useful for partly overcoming this challenge.
In summary, it is the synergy of this joint collaboration between patterns of real use and analysis of the needs that will guide us with further development to enhance sustainability of our product. Additionally, these conversations with real users will be both rewarding and fun.
Jianlin
November 29, 2012 at 8:46 am (UTC 0) Link to this comment
I agree that field deployment will be one of the ideal design processes for our project. Because our product is designed to be used in daily life, there are a large amount of factors, such as distractions, concurrent activities, and social and technical constrains, that can affect the interaction between our product and the participants. For example, a user may find out his mobile phone has no signal in a certain store, or run out of money to get the internet connection. Although we may mimic in lab situation, but it will be valuable to locate when and where the user encountered these problems, and how he works around in situ.
If possible, I would prefer deploy “in the wild” instead of convenience deployment. I mean “in the wild” is still in the scope of the targeted users, the college students including undergraduates and graduates. Because in my plan the study goals will include many objective questions, such as “how do you think about the notification frequency?”, “How do you think the sufficiency of the summary?” and so on. Study under convenient deployment could bring unexpected bias. On the contrast, if we study in the wild, we can get more real feedbacks. Of course, this approach has difficulties in participant recruitment. Especially in this class project we don’t have any funding support to provide compensations for the participants. However, we can still plan the study in this way like a real design process.
Jacob Heller
November 29, 2012 at 5:04 am (UTC 0) Link to this comment
Short Answer: Yes
Long answer: Definitely
Really Long answer: Our product needs real world testing. We’re working on system that lets the teacher know when students aren’t paying attention. If the teacher’s using a dry-erase maker, it changes color to indicate a loss of attention. If they’re using a powerpoint, the background changes color. Since this is an HCI course not an algorithms class, let’s assume that technology component works well. That is, the loss-of-student-attention detector is working correctly. But we still have no idea how the teacher or the students will react when this system is in place. The idea is that a teacher can look at the white board and see, visually, where the students zoned out. Perhaps a particular math concept is really hard, or the part of class about General Lee’s pet goat didn’t quite engage the students. Would the teacher find it useful to see the color change on the board? Would the students start snickering and turn the color changing into a game? Thus, completely undermining the system? Would the teacher feel jarred and distracted by the sudden changes in colors? We have no idea.
It’s important to remember that our product is especially intertwined with human behavior, and when children are involved, behavior analysis becomes a lot more interesting. And by interesting, I mean difficult. We may be able to test in a lab if this thing works mechanically. But does it really help a teacher do her job? Only a real world test tell us that.
Martin
November 29, 2012 at 6:58 am (UTC 0) Link to this comment
I have to agree with Jacob on this.
We want to provide live feedback to teachers on attention levels in their classroom. In particular, as Jacob mentioned, we are leveraging color to demarcate attention levels across the class. While we personally agree that it is a novel concept that converges with a need identified in our research, it’s not clear how the system would be received until we test it with real users. In the case of our tool, it is essential to use the tool in situ for any measure of meaningful understanding. It is likely that useful design insights would emerge from these a live deployment as well.
To our knowledge, no comparable live feedback tool exists. One challenge for a live deployment is that we may wait weeks before a teacher can make naturalistic use of the tool and students cease to treat it as a novelty. Additionally, we were expecting to leverage color to suggest attention and comprehension levels among students; it may turn out that color does not make sense. For example, what if a classroom does not find any particular color scheme intuitive? Are there more deeply-seated usability issues? Alternatively, teachers use white boards, projectors with transparencies, black boards, and so on; a real-world deployment would immediately suggest the limitations of our tool in any given context. Assuming the tool works, it also becomes necessary to have actual students affecting and reacting to the tool for simple questions of practicality. We assume that it will help teachers maintain their classroom more effectively, but it is also possible that the tool is a major distraction or otherwise adversely affects attention to the lesson itself. A live deployment could help to clarify these and other unforeseen issues.
Matthew Chan
November 29, 2012 at 9:03 am (UTC 0) Link to this comment
I have to agree with both Martin and Jacob on this.
Challenges we might face are schools enduring budget cuts and can’t afford our product. Moreover, there could be an element of corruption unknown to is. In particular, i’m thinking of the taxi service Uber in which users can hail a cab from a touch of their button on their smart phone. The service expanded from SF to many large cities, but one-by-one, politicians, attorney generals, taxi commissioners, etc. started squeezing Uber out of the city. A simpler example: CD’s and USBs replacing Floppy disks.
Going back to our live feedback system, is there an incumbent in the educational system that we don’t know about, whether it’s the whiteboard and dry-erase marker manufacturers or even the projector-overhead makers who might feel threaten. All in all, a great idea and product can be killed because of the status quo and corruption.
Not every idea and product will carry the weight of Apple and just enter the Music, Phone, Tablet or any other industry it chooses and come out fine.
Chuxiong Wu
November 29, 2012 at 5:14 am (UTC 0) Link to this comment
I believe that our project would benefit from a real world deployment. Basically, according to our user research, the issue of information overloads and information aggregation emerged as a universal issue. Simply speaking, the essential obstacle appears to be the lack of cohesion between event information across a variety of platforms. Besides, many platforms have event patterns; students feel those sources relevant to their interests, or lack of filter system to search information. I think the real world deployment can be really helpful and efficient for us to evaluate our design and to discover more potential issues.
Since our project is closely linked with students, the deployment in real world supports us to test our prototype. Especially, we can collect empirical data from the real world deployment. Based on our design requirements, event information aggregation will be the most important part for us to do the field study. Our novel project can gather feedback that how users feel about the way we provide that a convenient and comprehensive information resource center.
I think one of the challenges is a scope problem. Essentially, our scope should be the balance point in the relationship of information, reminders, decision filtering. The design problem might reflect to our filed study. So we might have problem to determine our boundary that where is the line of satisfaction about our project. In this case, the data collection and analysis may also have potential troubles. Thereby, our challenge is finally determine the scope for a validate boundary and how we can analyze these data from our deployment study. Also, appropriate research questions for deployment study will be listed carefully. Our project aim to serve students as a daily use tool, it’s another challenge for us to test if our design can support a long-term use and provide a convenient method to users.
Xinning Gui
November 29, 2012 at 5:25 am (UTC 0) Link to this comment
A real world deployment definitely would benefit our project. Our project is about helping UCI students to discover and organize events. If we adopt a real world deployment, we can gain empirical date from real world settings (our campus), which can help us learn the UCI students’ realistic needs and evaluate our prototype within the intended context of use. We can find out how our design can actually support UCI students’ everyday school life in real world. We can find new problems and debug over time.
Luckily, since our target populations are UCI students as ourselves, it is relatively easy for us to find a field setting to deploy our system, recruit participants, and build close relationships with our participants. It may also be easy for us to get permission from administrators to conduct the research on campus, considering that our project doesn’t involve many ethic issues.
However, challenges do exist. Some of our participants may leave the site because of graduation or transferring to another university, this may be disruptive. Also, how to promote and maintain our participants’ continuous interests about using our system, how to critically engage with our field site and how to collect and analyze the data neatly and properly may cause challenges. If we choose “in the wild”, our prototype needs to be very robust and may even be at “beta” testing level to enable us to collect data in real world. This is very challenging for us, too.
Chunzizheng
November 29, 2012 at 5:28 am (UTC 0) Link to this comment
Our project is to provide a way to protect apartment’s safety by developing a smart key which can identify the owner’s identity. And the smart key can be monitored by mobile app, so people can cancel the function of a smart key after it is lost. I think our project about smart key will benefit from the field deployment, because we can find out possible problems in the interaction between users and our product, and we can learn how our product support users’ need by observing users’ speculative reactions. For example, some users may find it hard to understand the instructions and operations; users would be a little bit hesitate when there ‘s lack of instruction to a specific operation.
I think the in the wild deployment fit our project well, since our product is a daily-used product that almost every people have to use it (for almost everyone have to live in some place and need to open their door by key). Participants in the wild deployment are people don’t know anything about our research, so participants could just act in their own way. And in this way, participants would provide us the real world usage statistics, which could help us to learn more about user’s real need and usage patterns. However, participants who are familiar with our group, such as our friends or families could be an obstacle to our research, because they may won’t tell truth, for they don’t want to be mean to us.
Timothy Young
November 29, 2012 at 5:28 am (UTC 0) Link to this comment
Real world deployment would allow us to fill our application and database with real world event information, and monitor how our users would actually navigate through this information. For now, most of our tests are based on fabricated pieces of data, and our users are responding more to the interaction and UI elements. While this is valuable information, a big part of our event finding application revolves around how users can respond to real world events.
Along with the associated costs of setting up a database, website, and smart phone applications, real world deployment would also cost a substantial amount of time to continue testing and reiterating the design as we see how users react to events a few days in advance, weeks in advance, and even months in advance. One could imagine this being the case for applications that involve users to track their behavior over time, such as a fitness and health tracker. Depending on the development process, time can be a very scarce and valuable resouce.
Jie
November 29, 2012 at 5:29 am (UTC 0) Link to this comment
Our group project will definitely benefit from a real world deployment. Field deployment enables researchers to study the interactions with users in real contexts, which is a good way to learn about users’ realistic needs and usage patterns in a real world and then help researchers to evaluate their products. Our project aims to improve people’s wake-up experiences and our project artifact is supposed to be a mobile application. Therefore, deploying the application prototype in the real world would be a good opportunity for us to know users’ feelings about the artifact and the interaction with them. This will also uncover problems of the application’s usability, such as inconvenient to use, lack of necessary functions, or poor navigation design.
Also, we face several challenges. First, the prototype should be implemented sufficiently robust; otherwise, participants might be less interested in the artifact. Second, because our design is more like a social alarm, which needs users to share their wake-up time in the application and on the web, users might have some privacy considerations. But I think those challenges would be quite useful for us to evaluate and improve our design. Actually I really want to do a field deployment of our application, because I want to learn about whether users feel our design interesting and useful, whether they can keep using it for a long time, and whether people want to participate in the social interaction. I think these questions, also challenges, will be answered after a real world testing.
Anshu Singh
November 29, 2012 at 5:29 am (UTC 0) Link to this comment
Real world deployment would be best form of usability testing for our project. From the conception stage of this project, every step has evolved around real world expectations. Most of our user research was also conducted in situ. Accordingly, testing our prototype in the wild would be the most advantageous step for the project. In my view, the type of field deployment that will best suit our initial prototype would be Semi-Controlled Studies. I think we can test our first high-fidelity prototype with intentionally selected/recruited participants. The knowledge that we gain from these experiments can help us built a better system. Next, the final product can be beta-tested. For example, we could do semi-controlled experiments with UCI community, and take it to next level by conducting robust beta testing with non UCI members.
Starting from user research phase we have faced couple of unique obstacles, and these obstacles most probably will appear during real world deployment as well. First, we might end up collecting a lot of messy data because our past experience has shown, many users give vague answers, nothing can be summed up from their answers. Second, participants’ participation depends on their own interest. While doing our research, at couple of occasions we observed that in spite of users wanting a system like the one we are building, they were completely disinterested in providing any helpful suggestions or feedback. However, I am guessing this might be the issue with any research project.
On the other we get to learn a lot from participants. They help us define the meaning of our project. Their requirements form the bases of our design principles. I think, in case of in situ deployment, participants would be more open to give honest feedback, unless we keep approaching same participants every time, they might end up being irritated
XIaoyue Xiao
November 29, 2012 at 5:29 am (UTC 0) Link to this comment
I think our project will be benefit from a real world deployment. The project of our group is an idea to control a smart key of apartment and make it safer than current traditional physical ones. And in order to reach the goals of this initial goal, we designed an mobile application having functions of opening door, managing authorities, controlling facilities of the apartment etc. It is essential to evaluate the application in real world, so that we can test the usability of it, especial to test how it works among low age and high groups of people, since we did not find appropriate interviewees of these groups during the requirement phrase.
Another feature in our project that I think will benefit from real world deployment is the sensor on windows. That is an idea that to set a camera and sensor on the window of apartment, and when someone try to break the window to get into the room will be record and a message will be sent to the owner of the key. We need to deploy this device in real world to test is there any privacy issues will be involved in such like to test will the camera record private things in the owner or neighbors’ room.
Jinelle D'souza
November 29, 2012 at 5:45 am (UTC 0) Link to this comment
The two key elements of deployments are that they seek to evaluate the impacts of particular new technologies on particular populations, activities and tasks and they seek to perform such evaluations within the intended context of use. I think our project would benefit from real world deployment because when users test it by integrating it into their daily life; we will have valuable user results to work with. If they find it useful, engaging, simple and fun, they will use it frequently and encourage others to do so. The age demographic, reasons behind their usage, time spent for it will differ. We would try the ‘in the wild’ approach because we want to test it on people who did not feel the need to learn a new language, but after viewing our application decided to give it a try. While designing our project, we felt that we may have covered the obstacles prevalent for easy language learning. But by this deployment method, we can use the users’ feedback to refine our design to meet most of the unmet user needs.
One of the major obstacles for our project is recruiting a large volume to people to use it. Only those who are determined to learn a new language on their own might try it. Also, the age demographics between 13-30 years may be more willing to give it a try. So, testing it outside this age demographic will be difficult. Another problem would be that since language learning is a gradual process, (the time will vary according to the time the user dedicates for this) the field deployment will take a long time to generate a substantial amount of useful data. This would make it difficult to decide when to terminate the deployment process. Also, we cannot monitor the time the users spent on each level and whether they are using other language learning applications or only ours. If there are users who wish to learn Spanish, they will find an application that only teaches Spanish. So, there is a huge possibility that a more specialized application will be more suited to their needs, whereas our application would target those wanting to learn multiple new languages.
Sreevatsa Sreeraman
November 29, 2012 at 8:21 am (UTC 0) Link to this comment
In the process of finding the ways in which a person learning a language would use other than taking traditional classes, we figured that there are a lot of technological solutions like Rosetta Stone. But, from our results of the survey, we found that people use a variety of ingenious methods to learn, like watching a television show in the new language. Another trend was allocating time to an informal activity was difficult. So, we figures that a solution which can act as entertainment and learning may help users learn better. These being just hypothesis based on limited user input, we have to understand if the solution of accompanying learning with entertainment will lead to learning, or cause a distraction to users. The effectiveness of the solution is based on maximizing learning while minimizing disruption. This can only be gathered from a study ‘in the wild’.
Two obstacles to deployment in the wild is the diverse set of targeted users and the solution is expected to be used with other forms of learning. Since the targeted users are diverse, we need to come up with a set of users who can provide us with well rounded results. Given that the solution may be used infrequently and accompanied with other forms of learning, evaluating the effectiveness of the solution can become a little tricky. We would have to come up with a baseline of learning speeds in order to capture data better.
Anirudh
November 29, 2012 at 5:50 am (UTC 0) Link to this comment
Our project would definitely benefit from real-world deployment. The goal of our project is to create an online marketplace for UCI community. In our user research we have found that the primary stakeholders of our project are students, staff and faculty. The need of each of these groups is different. Our project primarily targets UCI student community. Our understanding of user requirements that we gathered as part user study indicate that there is a high demand for the services offered by our website.
Our user study was limited to survey, interviews and competitive product survey (Uloop, Amazon, Ebay, Craigslist). Though this was good enough to provide us with the right direction we definitely need users to evaluate our project based on the prototype that we are building as part of our next milestone. Real-world deployment is absolutely necessary as it facilitates “connecting” with the end users of our system.
Though we are tempted to go for a “Convenience Deployment” (or eating our own dog food) we should ideally be conducting a mix of convenience deployment and “In the wild” deployment. Convenience deployment will help us understand how the user are using the system and what are the frequent problems that our stakeholders face while using our website. However this method suffers from the drawback that the users are “forced” to perform an action (buying, selling or renting) during the lab session. This “forced action” effect has the tendency to negate our aim of understanding the users’ problem as users are not using the system under normal circumstances. “In the wild” deployment offers a lot of advantages when compared to “lab deployment” as the users interact with the system without any forced activity. This is when most of the problems will be discovered, as a user might require the system to perform a task as he/she thought the system supported when compared to what we (designers) had in mind while creating the prototype. This deployment will allow us to gather “real-world” statistics and will enable us to make the system robust.
Finally, assuming that our target audience for real-world deployment is 100 in number. It would be good if we could split the users in a manner so that roughly 20 users are involved in lab deployment and 80 of them are involved in “in the wild” deployment. Also, in lab deployment roughly 60% of the users (12 users) should be picked from the user-pool we studied in user research phase as it will give us a correlation between user’ understanding of our website’ services during that phase versus the services offered by the prototype.
Ishita Shah
November 29, 2012 at 5:51 am (UTC 0) Link to this comment
Our project would definitely benefit from a real world deployment. Currently, there are a number of platforms on which UCI events are broad-casted and our application aims to aggregate all of this information and keep users informed via one single channel. Deployment would help us evaluate whether or not our application is proving to be useful in the real world, whether or not it is meeting the everyday needs of the our target users and how it is integrating with their daily activities. We would also be able to understand how our potential users interact with the application in the real world settings as opposed to a lab environment. The data collected from a deployment could be used to draw out improved designs and versions of application.
Since our target population is UCI students, I think we would go ahead with a convenience deployment. It would give us an easy access to participants. The challenge would be developing the working prototype and supporting it until the deployment ends. It requires a considerable amount of time and commitment. Another challenge would be choosing the appropriate analytic methods to analyze the data sets that the deployment would provide us with.
Pushkar
November 29, 2012 at 6:04 am (UTC 0) Link to this comment
I feel that my project would definitely benefit from a real world deployment. In our case the final end product of our project, which is a website designed to help students, professors and other staff at UCI to buy and sell their used products, is the artifact. A real world deployment of this artifact would give us an insight into how our intended audience would use the product. Since it is a website, we can collect statistics about the features which are used the most and those that are not used extensively. This will help us in organizing our website. We can add popular features to the home page of our site and remove unnecessary features as well. We can also add an option using which users can give feedback about our product and specific features in it. In case a user finds a feature missing or if the user interface of a feature is not intuitive, he/she can report such features and we can fix it at that stage itself. Another advantage of a real world deployment is that we can perform a load test of our website. A convenience deployment (or eating our own dog food), where our product is deployed to a small network of users, would help us in performing this load testing. By testing in such an environment, we can check if we have the right infrastructure to serve all our intended audience after the complete launch of the website.
The main obstacle for us would be to setup the infrastructure and deploy the system to a small set of the users. Limiting the scope of the audience in the initial stages would be a challenge. However this can be done by deploying the website just within a department like Informatics and computer science. We can allow only students and faculty from our department to login and use the system. This would be very useful in figuring out the usability issues in our website.
Dakuo
November 29, 2012 at 7:47 am (UTC 0) Link to this comment
Our project is going to deploy a web site to encourage busy people engage in physical exercises. I think our project fall into the practical AR type. We are trying to provide a potentially technological solution to motivate busy people involve in exercises and maintain a health body.
The benefits real world deployment can bring our project with credibility of data and validity through multiple perspectives. Users can voice concerns and comment on our project. We can explain and understand both our perspectives and participants. More complex viewpoints can only be addressed by multiple accounts. The credibility and validity can ensure workability of our solution, which means the real ability to address real problems in the real lives of the participants. According to AR method, our project would first deploy into our communities. We would like to engage our community partners as participants of our project. Since the academic and IT industrial professionals are the target busy people population.
The challenge of our deployment may be the social value theory is our fundamental theory and we expect people use their social relation benefits as incentive. They can share their exercise plan and make new friends while executing the suggested exercise plan. I would suggest that if the incentive result is not satisfied enough from the real world deployment, we should find other incentive method to encourage users to keep using our web site. I think sustainable maintenance would be a challenge for us although it’s basic HTML technology. The high quality of exercise plan and attractive activity coordination could be a hard thing to do.
Yao
November 29, 2012 at 8:10 am (UTC 0) Link to this comment
I think a real-world deployment is indispensible in our project. Because our project is related to people’s sleep, it might be hard for us to evaluate our design through a lab-based testing or evaluation. Like what we did in P1, we have conducted some interviews with our stakeholders, and what we can get is mostly what they felt or remembered about their sleep, which might be different from their actual daily life. I think in our project, we should have a heuristic evaluation or usability test first, and we have to deploy our design to real world to see what will happen in people’s daily sleep. I believe we can and should collect useful data in this way rather than recruiting people to a place to do some testing. Also, because our current design has something to do with social interaction, we have to put our prototype in real world to observe the usage over time and test if our model is working or not.
If we are going to have a real-world deployment, one challenge would be about privacy issue. In fact, this issue plays an important role in our whole project. If we give users our prototype, how much information would they want to share about their sleep? If some functions take place while users are sleeping, how can they evaluate the design if they are sleeping? We can get information by asking the participants about how they think and feel about the prototype, but it might be difficult if we want to “observe” them in order to have a better understanding on their problems. As a result, we might have to consider other methods rather than traditional ways to collect useful information, which must be carefully designed to achieve good data collection while being aware of people’s privacy.
Jared Young
November 29, 2012 at 8:15 am (UTC 0) Link to this comment
I do think our project would benefit from a real world deployment. By nature, the prototype we are designing is meant to be used in the real world. We can not predict how users will react to or use our new technology when it is introduced into their lives. We might be able to deduce findings from convenient studies with people that we know. However, data about usage over an extended period of time or data about usage of our prototype within the context of local/regional user groups is best collected from real world deployment.
This brings up one obstacle in the real world deployment of our project. Our prototype of our project will be a web based application/website that encourages people to engage in physical fitness (as stated by Dakuo above). For us to do real world deployment, this requires technical support for our website as well as web hosting to provide service to our clients. This costs man power/time as well as money for every day of the deployment. The longer the deployment is, the more costlier it is.
A great benefit to having an online-web prototype is the accessibility for everyone. We will be able to reach out to virtually anyone to come test our website. If we deploy software tools/functions to observe usage of the site, this will give us the benefit of collecting quantitative data that can be minimally invasive.
Xinlu Tong
November 29, 2012 at 9:38 am (UTC 0) Link to this comment
I think that our project can benefit from real world deployment. Our project involves the interaction between different users and facilities. Users can use our proposed system to post questions about language learning and even record some audio material in the real world conversation environment. So we need users use our system in a real context with different language and see if they will and how they will use our system when they meet some problems. Also, one idea about our project is to detect user’s state to determine if user is free and send notification of language learning to the user. This should be done in a real environment, and we can see many cases that vary due to the complexity of reality.
We can also get good opportunities from design in the world with other participants. Actually our project need advices from experts in different areas like linguistics and education. Our design can make things easier, but it may not accord to principles in those areas. Moreover, language learner using different aspects can provide us with valuable suggestions. We also need help from legal and marketing experts, since some of our designs may be against some intellectual properties and may not be successful as we think.
Karen
November 29, 2012 at 10:18 am (UTC 0) Link to this comment
I believe that our project, like most projects, would benefit from real world deployment given that we possessed the time and resources necessary for this. Not only would real world deployment help us to further refine our product design and account for otherwise unanticipated issues, it could also help us to develop stakeholders for the product. For our project, we are leaning on developing a form of live feedback for teachers (feedback *during* their actual teaching). This feedback would let teachers know whether their students are understanding the concepts they are currently teaching. For instance, if students are confused, marker ink or Powerpoint text would change colors. Thus the teacher would know to persist on the particular concept rather than to move on.
However there are many possible issues with such a product, and real world deployment would surely help to identify these problems. For one, what is the best way for us to identify confusion/lack of understanding in students? Importantly, would the method we design be significantly more accurate than a teacher’s own abilities to detect confusion (traditionally, teachers can ask students questions, or observe students’ faces)? We went from considering self-report by students (e.g., through electronic “understanding dials”) to a facial-recognition system, but each of these methods of detecting confusion has drawbacks. For instance, students may not be willing to use dials (though they would in essence be forced to if the dial was set at “Completely confused”, since the teacher would not move on) or may find them cumbersome. For facial-recognition systems, there are possible issues with camera set-up and with accurate detection of facial expressions. Real world deployment would surely help to identify how serious these problems are.
There are several other reasons that real world deployment would be important. For a product like ours, interaction between teachers and students is key, and there is no way a lab-based evaluation could account for such complex interactions. This method would also allow us to measure student performance (as Matt mentioned in class) and teacher satisfaction after use of this product. Importantly, we would be able to examine if teachers would actually want to use this product, or if they would find it annoying or troublesome. Such a product could have unintentional consequences–students might ridicule a teacher if nothing the teacher says is “objectively” understandable, and/or the teacher might experience anxiety as a result of the product. The design could also be modified such that the feedback was only visible to the teacher, if such consequences were discovered. Also, it is possible that unruly students would try to steal the marker.
Even though I have taught classes before, this product is so novel that I do not know whether I would want to use the product–I would have to try it first. I feel that real world deployment is crucial in the evaluation of our design.
Jeffrey
November 29, 2012 at 10:18 am (UTC 0) Link to this comment
Our project would greatly benefit from a real world deployment. The goal of our project is to design a tool that will assist users in managing their budget. Usability tests and competitive comparisons with other solutions existing in the market can provide us with feedback to refine our budget managing tool’s interface and its usability. However, such methods will not allow us to measure whether people are actually using the budget managing tool and whether such a tool is actually accomplishing its goals of facilitating the user’s budget managing task. Carrying out a real world deployment will provide us with critical information that will allow us to further comprehend users’ needs, while allowing us to further evaluate our budget tool within the intended context of use. Additionally, real world deployment will allow us to monitor the types of users using the the tools and how often such users use it, as well as the impact such a tool can have on the users’ monthly spending/saving habits. Although real world deployments will be valuable for our project, there will of course be obstacles for design in the world. Analyzing whether the tool is actually affecting the users’ spending/saving behavior will be difficult as it can be subjective. Furthermore, such analysis will also require a great deal of time as examining users’ expenditure over a short period of time would be ineffective to our studies.
Ramraj
November 29, 2012 at 1:05 pm (UTC 0) Link to this comment
My project is to develop a smart key. My project would definitely get benefited from real world deployment. We proposed three solutions 1) Mobile application 2) Window sensor ,and 3) A smart key.
Real world deployment enable us to study the real time situations and the actual context when the user is using the application. We followed the quantitative method to collect the data i.e. survey.We took a sample of 35 people.We can follow three deployment methods depending on the solution 1) Convenience deployment for the smart key and window sensor 2) Semi-controlled studies for window sensor, and 3) In the wild for mobile application because we need to know the different scenarios.We can follow the standard steps for real world deployment and I am considering one scenario i.e. mobile application.
1) Finding a field setting in which to deploy the system
2) Defining the goals of a field deployment study
3) Participant recruitment
4) Conducting the field study
5) Ending the deployment
6) Analyzing the data
We can repeat the above steps for other two solutions as well.
In case of window sensor we can deploy it in house by following the above steps and it is beneficial to do so to reduce the deployment cost,but for smart key and mobile application we need to recruit participants to capture the real time contexts. After conducting the study we will have an advantage to modify the model if needed based on the data collected.
Shih Chieh Lee
November 30, 2012 at 3:38 pm (UTC 0) Link to this comment
For our project, a real world deployment would be the best way to figure out the feasibility. The project is about waking up someone effectively and interesting, so the field deployment could find out whether it’s effective or not. Although we did a lot of researches and interviews, the goal of the system is really unique so that it’s hard to work only on paper or in labs. The major obstacle for our design is to put it into field and record. Because people usually feel unconscious or confused during waking up, recording the response, reaction, and the user’s emotion would become a tough work. If we still want to control the situation or set up the condition like the lab, it would lose the meaning to do a real world deployment. Besides, if the tester was told that every actions would be recorded or videotaped, he or she might not be as nature as usual and we would get the real result in the field.
However, there are opportunities. A real world deployment would give us an opportunity to test with different stakeholders, which helps us to examine the stakeholders that we aim to design for. Furthermore, there is another opportunity, which is also an important issue mentioned before. The actions and reactions that happen during waking up could only be captured and observed with a real world deployment. Hence, I think deploy our design into the reality could solve some problems that we could only imagine now.