mirror of
https://github.com/harvard-edge/cs249r_book.git
synced 2026-03-12 02:06:14 -05:00
121 lines
4.9 KiB
Plaintext
121 lines
4.9 KiB
Plaintext
# Data Engineering
|
|
|
|
## Introduction
|
|
|
|
Explanation: This section establishes the groundwork, defining data engineering and explaining its importance and role in Embedded AI. A well-rounded introduction will help in establishing the foundation for the readers.
|
|
|
|
- Definition and Importance of Data Engineering in AI
|
|
- Role of Data Engineering in Embedded AI
|
|
- Synergy with Machine Learning and Deep Learning
|
|
|
|
## Problem
|
|
|
|
Explanation: This section is a crucial starting point in any data engineering project, as it lays the groundwork for the project's trajectory and ultimate success. Here's a brief explanation of why each subsection within the "Problem Definition" is important:
|
|
|
|
- Identifying the Problem
|
|
- Setting Clear Objectives
|
|
- Benchmarks for Success
|
|
- Stakeholder Engagement and Understanding
|
|
- Understanding the Constraints and Limitations of Embedded Systems
|
|
|
|
## Data Sourcing
|
|
|
|
Explanation: This section delves into the first step in data engineering - gathering data. Understanding various data types and sources is vital for developing robust AI systems, especially in the context of embedded systems where resources might be limited.
|
|
|
|
- Data Sources: crowdsourcing, pre-existing datasets etc.
|
|
- Data Types: Structured, Semi-Structured, and Unstructured
|
|
- Real-time Data Processing in Embedded Systems
|
|
|
|
## Data Storage
|
|
|
|
Explanation: Data must be stored and managed efficiently to facilitate easy access and processing. This section will provide insights into different data storage options and their respective advantages and challenges in embedded systems.
|
|
|
|
- Data Warehousing
|
|
- Data Lakes
|
|
- Metadata Management
|
|
- Data Governance
|
|
|
|
## Data Processing
|
|
|
|
Explanation: Data processing is a pivotal step in transforming raw data into a usable format. This section provides a deep dive into the necessary processes, which include cleaning, integration, and establishing data pipelines, all crucial for streamlining operations in embedded AI systems.
|
|
|
|
- Data Cleaning and Transformation
|
|
- Data Pipelines
|
|
- Batch vs. Stream Processing
|
|
|
|
## Data Quality
|
|
|
|
Explanation: Ensuring data quality is critical to developing reliable AI models. This section outlines various strategies to assure and evaluate data quality.
|
|
|
|
- Data Validation
|
|
- Handling Missing Values
|
|
- Outlier Detection
|
|
- Data Provenance
|
|
|
|
## Feature Engineering
|
|
|
|
Explanation: Feature engineering involves selecting and transforming variables to improve the performance of AI models. It's vital in embedded AI systems where computational resources are limited, and optimized feature sets can significantly improve performance.
|
|
|
|
- Importance of Feature Engineering
|
|
- Techniques of Feature Selection
|
|
- Feature Transformation for Embedded Systems
|
|
- Embeddings
|
|
- Real-time Feature Engineering in Embedded Systems
|
|
|
|
## Data Labeling
|
|
|
|
Explanation: Labeling is an essential part of preparing data for supervised learning. This section focuses on various strategies and tools available for data labeling, a vital process in the data preparation phase.
|
|
|
|
- Manual Data Labeling
|
|
- Ethical Considerations (e.g. OpenAI issues)
|
|
- Automated Data Labeling
|
|
- Labeling Tools
|
|
|
|
## Data Version Control
|
|
|
|
Explanation: Version control is critical for managing changes and tracking versions of datasets during the development of AI models, facilitating reproducibility and collaboration.
|
|
|
|
- Version Control Systems
|
|
- Metadata
|
|
|
|
## Optimizing Data for Embedded AI
|
|
|
|
Explanation: This section concentrates on optimization techniques specifically suited for embedded systems, focusing on strategies to reduce data volume and enhance storage and retrieval efficiency, crucial for resource-constrained embedded environments.
|
|
|
|
- Low-Resource Data Challenges
|
|
- Data Reduction Techniques
|
|
- Optimizing Data Storage and Retrieval
|
|
|
|
## Challenges in Data Engineering
|
|
|
|
Explanation: Understanding potential challenges can help in devising strategies to mitigate them. This section discusses common challenges encountered in data engineering, particularly focusing on embedded systems.
|
|
|
|
- Scalability
|
|
- Data Security and Privacy
|
|
- Data Bias and Representativity
|
|
|
|
## Promoting Transparency
|
|
|
|
Explanation: We explain that as we increasingly use these systems built on the foundation of data, we need to have more transparency in the ecosystem.
|
|
|
|
- Definition and Importance of Transparency in Data Engineering
|
|
- Transparency in Data Collection and Sourcing
|
|
- Transparency in Data Processing and Analysis
|
|
- Transparency in Model Building and Deployment
|
|
- Transparency in Data Sharing and Usage
|
|
- Tools and Techniques for Ensuring Transparency
|
|
|
|
## Licensing
|
|
|
|
Explanation: This section emphasizes why one must understand data licensing issues before they start using the data to train the models.
|
|
|
|
- Metadata
|
|
- Data Nutrition Project
|
|
- Understanding Licensing
|
|
|
|
## Conclusion
|
|
|
|
Explanation: Close up the chapter with a summary of the key topics that we have covered in this section.
|
|
|
|
- The Future of Data Engineering in Embedded AI
|
|
- Key Takeaways |