Experience Capture, Composition & Reuse

Experience Capture, Composition & Reuse


How do we capture + represent personal experience through media collections and computational tools? How can these long-term personal archives, e.g. lifelogs and social media, enable new insights and understanding of personal, societal or global significance?


In working with John Saduaskas, we are exploring the use of lifelogging approaches within the context of K-12 education to inspire and support students in developing creative writing outcomes. Preliminary investigation includef user-centered inquiry through interviews, focus groups, workshops, and participatory design sessions with both teachers and students. This has yielded design guidelines and requirements in support of the development of a computational support tool. The developed online environment aids students in generating personally relevant writing topic ideas. Through writing-performance-based evaluation, the tool is currently being evaluated with 90 K-12 students and is embedded in real-world curricular contexts.


Working with Dr. Aisling Kelliher, we identified opportunity to develop new capture and presentation tools suited to recording process within situated and shared events. This documentation framework was inspired by the lifelogging vision and deployed within the context of the Emerge event at ASU in March 2012. This novel and highly dynamic event precipitated the need for novel documentation approaches. Our conversational framework was developed by co-opting techniques from Bill Gaver’s cultural probes, social networking strategies, and participatory documentation methodologies. This framework mixed elements of traditional recording apparatus (videography, photography and audio recorders), social media contribution, and novel capture technologies (wearable passive capture, time-lapse video, self-documentation, and experience capture installations) to more fully describe the event. This mixed initiate approach to the documentation of shared experienced crafted a highly detailed record and afforded a unique lens on the events, workshops and experiences felt by attendees. This is an active research area which has lead to the development of an online participatory community for the dissemination and continued discourse around this coordinated account of event content.


Noting that highly-experiential 'life-sampling' methods often do not integrate a first-person, qualified or reflective accounts, we developed a novel recording apparatus, the Probotron. This is an experience recording installation that can be deployed during situated events. We use this novel situated installation as an explorative means to investigate the potential of integrating reflective, subjective accounts into the highly chronological records that might be found in a lifelog collection. It has gathered 624 responses from 3 distinct deployments. It explores the experience of experience capture and asks what a first person account can contribute to data-centered records. A reworked version for distributed and personal use is planned.


The potential for lifelog collections to move beyond the scope of the individual and offer opportunities for discovery to motivated third parties is an exciting avenue for research and one which I have collaborated with Dr. Aisling Kelliher to investigate. We were interested to discovering how and if third parties could infer meaning and purpose from a large-scale lifelog collection. We evaluated the examination of lifelog data and construction of storied interpretations from this content in a provocative learning environment where the overall focus of study was precisely this type of considered, mediated activity. It saw a single 9-month dataset distributed to over 100 participants within a pedagogical context. The participants were versed with the skills necessary to conduct the evaluation through a semester long 'Media-Editing' module in which the evaluation was situated as the final assessable component. This enabled exploration of the affordances, constraints and considerations involved when presenting such voluminous rich and detailed personal archives to third parties. We determined a variety of reflective strategies and approaches that could benefit from additional computational support including data discovery, ethical considerations and creative opportunities.


An important challenge, especially in multimodal lifelog collections, is identifying the right content to present to the user in response to their information need. Beyond providing access to these large scale collections, tools must enable and support the user in way-finding, interpreting and constructing meaning from these archives so that they might gain value from them. In response to this, I created Orison, a custom storytelling tool for lifelog data. The tool supports the exploration of lifelog content and its arrangement into a story-based layout. The requirements for representation, sense-making and storytelling were designed based on investigations into practices surrounding media use, management and composition within three contexts: family practice, genealogical investigations, and hobbyist scrapbooking. Studies with the scrapbooking community most heavily influenced the tool and the method of digital storytelling enabled within the tool closely resembled that observed in these studies: album based two-dimensional layouts. Additionally, the tool’s workflow was intended to parallel their observed patterns and creative practices, albeit in a digital environment. While the tool enables wholly manual and effortful construction of storied interpretations, given the size, scope and complexity of the archives employed computational generation of narrative accounts was developed for the system. A fully automatic (computationally controlled), semi-automatic (computationally guided) and manual variation of the story generation functionality was evaluated with three collection owners who had amassed lifelog collections over a 2.5 year period. Outcomes supports and confirms prior work by Kevin Brooks which suggests a semi-automatic approach is favored as it maintains the authorial control of the user while reducing the effort to prepare a narrative outcome. Additionally, insights and reflections on the utility, use and opportunities for these storied interpretations of past personal experience were uncovered through qualitative inquiry with the participants.


Within this work, we explored the applicability of semantic concept detection, a method often used within video retrieval, on the domain of visual lifelogs. Our concept detector models the correspondence between low-level visual features and high-level semantic concepts (such as indoors, outdoors, people, buildings, etc.) using supervised machine learning. By doing so it determines the probability of a concept’s presence. We apply detection of 27 everyday semantic concepts on a lifelog collection composed of 257,518 SenseCam images from 5 users. The results were evaluated on a subset of 95,907 images, to determine the accuracy for detection of each semantic concept. We conducted further analysis on the temporal consistency, co-occurance and relationships within the detected concepts to more extensively investigate the robustness of the detectors within this domain.


Beyond the challenge of employing lifelogging devices in research investigations, the collections often have unique properties owing to the nature and affordances of the capture device. While they create unique and engaging perspectives on past events, they present complexities when analyzing and utilizing the content. We conducted assessments of the nature and composition of long-term multimodal collections to explore the challenges, opportunities and features of these novel collections, as well as developing considerations for their use in longitudinal research.