search menu icon-carat-right cmu-wordmark

Grace A. Lewis

Principal Researcher

Grace A. Lewis is the lead for the SEI Tactical and AI-enabled Systems initiative and the principal investigator for three research projects in the field of Software Engineering for Machine Learning (SE4ML):

Characterizing and Detecting Mismatch in ML-Enabled Systems
A problem with deployment of ML systems in production environments is that their development and operation involve three perspectives, with three different and often completely separate workflows and people: the data scientist builds the model; the software engineer integrates the model into a larger system; and then operations staff deploy, operate, and monitor the system. Because these perspectives operate separately and often speak different languages, there are opportunities for mismatch between the assumptions made by each perspective with respect to the elements of the ML-enabled system, and the actual guarantees provided by each element. We conducted a study with practitioners to identify and validate mismatches and their consequences. In parallel we conducted a multi-vocal literature study to identify best practices for software engineering of ML systems that could address the identified mismatches. The result of the gap analysis between the outcome of the two tasks is a set of machine-readable descriptors that contain attributes of system elements that need to be made explicit and shared between stakeholders to avoid mismatch. These descriptors can be used by system stakeholders for information awareness and evaluation activities; and by automated mismatch detectors at design time and run time for cases in which attributes lend themselves to automation.
Automating Mismatch Detection and ML Component Testing in ML Systems
This project is a continuation of the previous project and is building on the set of developed descriptors for elements of ML-enabled system, to validate these descriptors at larger scale and provide automation to improve consistent and timely detection of mismatches. A suite of tools is being developed to (1) automate mismatch detection, and (2) demonstrate how the descriptors can be extended to support testing of ML components, particularly for production-readiness.
Predicting Inference Degradation in Production ML Systems
The inference quality of deployed ML models changes over time due to differences between characteristics of training and production data. State of engineering practice in industry relies on periodic retraining and model redeployment strategies to evade inference degradation, as opposed to monitoring for inference degradation. However, this strategy is not appropriate for all types of systems, environments, and organizations. The goal of this project is to develop a set of empirically-validated metrics that are better predictors of when a model’s inference quality will degrade below a threshold due to different types of data drift and requires retraining. Develop a set of empirically-validated metrics that are predictors of when a model’s inference quality will degrade below a threshold due to different types of data drift, and therefore requires retraining. These metrics will be able to determine (1) when a model really needs to be retrained so as to avoid spending resources on unnecessary retraining, and (2) when a model needs to be retrained before its scheduled retraining time so as to minimize the time that the model is producing sub-optimal results.

Grace also led the following research projects related to Edge Computing and IoT Security:


Kalki: High-Assurance Software-Defined IoT Security
This project developed an IoT security platform that operates, with high assurance, in a resilient and trustworthy manner even in the presence of a powerful and realistic attacker who can compromise IoT devices, control nodes, and other intermediaries. The software-defined IoT security platform is composed of (i) a high-assurance control node that monitors security-relevant events and alters the "security postures" of IoT devices to enforce specific policies; (ii) trusted data nodes that execute these security postures for each IoT device using "micro-middleboxes." The Kalki IoT security platform is available on GitHub.
Authentication and Authorization of IoT Devices in Tactical Environments
This project evaluated adapted, and implemented an IETF proposal for authentication and authorization in constrained environments (ACE) such that it is resilient to high-priority threats of tactical environments (e.g., node impersonation and capture) that are currently not addressed in ACE. The SEI-ACE implementation is available on GitHub.
Tactical Cloudlets
Tactical cloudlets are forward-deployed, discoverable computing nodes that can be hosted on vehicles and other platforms to provide computation and data services at the edge, where there is limited (or no) connectivity to the cloud. The tactical cloudlet software is available as KD-Cloudlet on GitHub. Tactical cloudlets include a novel mechanism for secure key generation and exchange for establishing trust between cloudlets and mobile clients, as well as secure service migration between cloudlets.

Professional Background

Grace Lewis is a Principal Researcher and lead of the Tactical and A-nabled Systems (TAS) initiative at the Software Engineering Institute at Carnegie Mellon University.

Before joining the SEI, Grace was Chief of Systems Development for Icesi University, where she served as project manager and technical lead for the university-wide administrative systems. Other work experience includes Design and Development Engineer for the Electronics Division of Carvajal S.A. where she developed software for communication between PCs and electronic devices and embedded software on devices microcontrollers.

At the SEI she has worked in the areas of Commercial-of-the-Shelf (COTS) Based Systems, Legacy System Modernization, Systems of Systems Engineering, Service-Oriented Architecture (SOA), and Cloud Computing, where she has a vast number of publications.

Grace has teaching experience at the graduate and undergraduate level. She serves various roles in the IEEE Computer Society, including member of the Board of Governors for the 2020-2022 period, and VP of Technical and Conference Activities (T&C). Grace also supports multiple activities related to promoting STEM and diversity in computer science and software engineering.

Grace hold a BSc in Software Systems Engineering from Icesi University in Cali, Colombia; a Post-Graduate Specialization in Business Administration from Icesi University; a Master of Software Engineering from Carnegie Mellon University in Pittsburgh, PA USA; and a PhD in Computer Science from Vrije Universiteit Amsterdam, Netherlands.

Grace's current areas of expertise and interest include software engineering for AI/ML systems, edge computing, software architecture (in particular the development of software architecture practices for systems that integrate emerging technologies), and software engineering in society.

Publications

Dissertation

Grace Alexandra Lewis. Software Architecture Strategies for Cyber-Foraging Systems. June 2016. ISBN: 978-94-6295-483-0

Publication Sites

Selected External Publications

SEI Publications

Books

Additional Publications by Grace A. Lewis

Teams

Tactical and AI-Enabled Systems (TAS)

Automating Mismatch Detection and Testing of ML Components in ML Systems

Education

  • PhD, Computer Science, Vrije Universiteit Amsterdam
  • MS, Software Engineering, Carnegie Mellon University
  • Post-Graduate Specialization, Business Administration, Icesi University
  • BS, Software Systems Engineering, Icesi University

Professional Memberships

  • IEEE Computer Society Technical & Conference Activities (T&C) Vice-President (2021-2022)
  • IEEE Computer Society Board of Governors (2020-2022)
  • IEEE Computer Society Technical Council on Software Engineering (TCSE), Diversity and Inclusion (D&I) Vice-Chair
  • IEEE Senior Member
  • ABET CSAB Board Alternate (Representative of the IEEE Computer Society) and Program Evaluator (Computing Accreditation Commission)

Current Professional Activities

Conference Organization

Steering Committee Member
ICSA - IEEE International Conference on Software Architecture
Most Influential Paper Co-Chair
ICSA 2023 - 20th IEEE International Conference on Software Architecture - March 2023 - L'Aquila, Italy
Program Committee Co-Chair
ECSA 2022 - 16th European Conference on Software Architecture - September 2022 - Prague, Czech Republic
Finance Chair
ICSE 2022 - 44th IEEE/ACM International Conference on Software Engineering - May 2022 - Pittsburgh, PA USA

Current Program Committees

ICSE 2023 - Software Engineering in Practice (SEIP) Track
45th IEEE/ACM International Conference on Software Engineering - May 14-20, 2023 - Melbourne, Australia
ESEC/FSE 2022 - Tutorials
ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering - November 14-18, 2022 - Singapore, Singapore
ASE 2022
37th IEEE/ACM International Conference on Automated Software Engineering - October 10-14, 2022 - Ann Arbor, MI USA
SAML 2022
2nd International Workshop on Software Architecture and Machine Learning @ ECSA 2022 - September 19-20, 2022 - Prague, Czech Republic