ETL (Extract Transform Load) Pipeline

1. ETL Pipeline ne zaman lazım olur ?

- Bir projeyi ilk defa ayağa kaldırırken mevcut veriyi alıp kendi veri modelimize çevirirken kullanırız.

- Çalışan bir başka sistemin değişen deltasını alıp kendi sistemimize ekleyerek, senkronizasyonu sağlamak amaçlı kullanabiliriz.

- Data warehouse sistemini çeşitli kaynakları kullanarak beslemek için kullanabiliriz.

2. ELED Extract–Load–Eligibility-Deliver Pipeline

Aslında delivery pipeline demek daha doğru. ELED benim kendi uydurmam. Açıklaması şöyle

An ELD (Extract–Load–Deliver) delivery pipeline with eligibility, acknowledgment, retry, and penalty processing.

Burada

Extract = obtain candidate work from the source.

• Read pending transmission records from DB
• Receive new transmission requests
• Fetch configuration/parameters needed to decide eligibility

Load = bring that work into the dispatcher's working set.

• Put records into the actor's queue/buffer
• Restore pending/in-flight state
• Build the in-memory transmission items

Eligibility = decide what can be transmitted now.

• Is it enabled?
• Is the penalty expired?
• Is the destination available?
• Is the item within its transmission window?
• Has it already been sent/in-flight?

Deliver = actually perform reliable transmission.

• Send
• Wait for ACK
• Retry
• Apply penalty
• Complete/remove the item