dialog schema builder

overview

Schema Builder is an application that allows users to build dialog schema, which are then passed into models trained on a dataset for transfer learning. From September to December 2020, I designed and developed Schema Builder under the guidance of Shikib Mehri and Dr. Maxine Eskenazi (DialRC, Language Technologies Institute, CMU).

The completed application allows users to drag and drop components and arrows to create a flowchart-style graph representing their schema diagram, and subsequently offers users the ability to test their dialog implementation by interacting with the system through a chatbox.

The main goals of the project were designing a feasible prototype, developing functional user interfaces, creating a useful data representation, developing the API, and finally integrating the frontend and backend together with a model. Under the guidance of my mentor, I successfully completed all project milestones in three months.

role

full stack intern, UI designer

skills

JavaScript   ∙   HTML + CSS   ∙   Python   ∙   API design & integration   ∙   prototyping

tools

React JS   ∙   Tornado   ∙   Postman   ∙   Git   ∙   Figma

more

Github repo   /   Concluding paper

motivation: STAR dataset

STAR is a schema-guided, task-oriented dialog dataset that facilitates transfer learning. With the development of novel schema-guided dialog models that use an explicit description of known tasks to generalize to unknown tasks, Schema Builder allows users to build arbitrary schemas, pass them to learning models, and potentially create a dialog model for a new domain.

Preliminary testing of the schema-guided models for model generalizability and task transfer shows potential for performance improvement on unseen tasks, which provides a baseline for future research in the field. I built a user interface that can, in the future, work in conjunction with zero-shot learning models.

prototyping

Link to Figma prototype

I used Figma to prototype the main features and proposed navigation of the application. Two frames from the prototype are shown below: first is the landing page, and second is a snapshot of the dialog builder in action. In this design, I focused on incorporating elements of material design to form a clean and intuitive UI.

responsibilities

• Design & prototype application


• Develop chat UI


• Develop drag & drop UI


• Build backend to interpret drag & drop schema and pass to model


• Integrate drag & drop, backend, and chat functions together

frontend

The UI could be broken down into two components: the drag and drop dialog builder area, and the messaging area. For the drag-drop area, I built upon the React Flow library, which contains useful functions that aided during the data representation phase of the project. Retrospectively, during this phase, one of the main challenges was navigating the React framework, but having the hands-on learning experience was helpful.

data representation

After the majority of frontend development was complete, I needed to design a data representation that would be useful for passing to the backend. Given a sample data object from my mentor, I logged my output in the console to ensure that it matched the target output. Data I needed to keep track of included message exchange history, nodes and edges in the graph, exact connections between nodes, and unique user IDs.

backend

Before beginning any API development, I needed to read documentation on the framework my mentor suggested using: Python Tornado. When I began the project, one of my personal goals was to improve my understanding of backend development and gain hands-on experience developing my own API. The API I developed for Schema Builder stores the user and system message exchange history as well as the saved schema. Each user is defined by a unique ID generated on the frontend, which helps the backend track what data belongs to each user.

integration

incorporating the model

concluding thoughts

The final product is a complex, multifunctional, and intuitively-designed application that both serves the goal of the project and follows the prototype design I had initially imagined.

Working on the Schema Builder helped me gain experience in not only full stack development, but also design thinking. By the nature of React and the complexity of what I needed to create, there were several times where I had to find clever ways to pass data, restructure components, and use states to accomplish my task. Working with my mentor on issues I could not resolve helped me understand different ways of thinking about problems; as a result, deep design thinking went into each component I created and developed for Schema Builder.

The entire project allowed me to exercise many of my existing skills and learn many new ones. I put my design knowledge into practice by creating the prototype; I gained familiarity with React, which is a useful tool that I can add to my frontend skillset; I learned about backend development and built my first API; I improved my understanding of how frontend and backend integrate and coexist; I also gained a basic understanding on the state of current research in dialog systems. Overall, I learned to view web development from varying perspectives at different times in the development process. These skills and experiences will invariably prove useful as I progress forward in my career.