The Deduplicate Component ensures that every record in your output stream is unique. This is essential when merging multiple data sources that might contain overlapping information.

⚙️ Configuration

Property	Type	Description
`Keys`	`string`	A comma-separated list of field paths that define uniqueness (e.g., `id, email`).

🛠️ How it Works

Composite Key Generation: For every record, the component generates a unique "fingerprint" by concatenating the values of the specified keys.
First-In Policy: The component keeps the first record it encounters for any given fingerprint and discards subsequent matches.
Deep Path Support: You can use dot notation (e.g., user.profile.external_id) to deduplicate based on nested fields.

🚀 Best Practice

Always include at least one unique identifier (like an ID or email) in your keys to avoid accidentally removing valid records that happen to have similar properties.