re: Debunking Kimball Effective Dates
I agree with John about focusing on a robust ETL process instead of trying to minimise the data redundancy at price of less clarity on physical model, although I found the debates around the topic are...
View Articlere: Debunking Kimball Effective Dates
so I attempted to build this as a POC for a client who is experiencing this overlapping type2 issue. My tables in question are relatively small (under 400k records), so VIEW performance does not scare...
View Articlere: Debunking Kimball Effective Dates
"I think the author did not understand Kimball in First place"Well you're entitled to your opinion. As am I. I think I understand Kimball's prescriptive methods pretty well and am well qualified to...
View Articlere: Debunking Kimball Effective Dates
I think the author did not understand Kimball in First place. The date ranges in the example are overlapping and that is not correct. You can not have a record ending on 27/11/2009 and next record...
View Articlere: Debunking Kimball Effective Dates
If you want IsCurrent instead of IsLatest, you could do something like:, CONVERT(BIT,CASE WHEN GETDATE() BETWEEN MIN(dim1.[SCDStartDate]) AND MIN(dim2.[SCDStartDate]) THEN 1 ELSE 0 END) AS...
View Articlere: Debunking Kimball Effective Dates
And this is one of the reasons I am giving up on Kimball approaches. The "Fact & Dimension table" model only really works for immutable historical data about point-in-time events (one of the...
View Articlere: Debunking Kimball Effective Dates
So Jamie, curious does this approach work when dealing with large dimension table and using Hive or Impala? Seems like the join may be too expensive...
View Articlere: Debunking Kimball Effective Dates
Hihave a little case here that I hope to solve using Your code With a twistHere is the case:I am collecting data from several Sources about Companies, but let us a dummy exampleI can get several rows...
View Articlere: Debunking Kimball Effective Dates
Kimball usually leaves implementation details up to the implementer. I would simply say you've demonstrated one implementation approach to accomplish the same end goal. I've never implemented a DW...
View Articlere: Debunking Kimball Effective Dates
My thinking is that, in a data warehouse, row level constraints (ie: insuring that two rows for the same product or customer don't have overlapping effective periods) and computed columns (ie: deriving...
View Articlere: Debunking Kimball Effective Dates
Jamie is absolutely right... we could introduce a contradiction if we redundantly store a point in time twice, one to represent the end of an interval, and then again to represent the start of the next...
View Articlere: Debunking Kimball Effective Dates
This is a pretty dated article, but the concept is fundamental to traditional data warehousing. I was browsing around and felt like I should comment on this for anyone new looking around for an...
View Articlere: Debunking Kimball Effective Dates
I may be a bit late into this conversation however I noticed the issue as soon as I looked at the End date of the first record and the start date of the second record. The dimension records in SCD II...
View Articlere: Debunking Kimball Effective Dates
It's always nice to run into a five year old blog post with an ongoing discussion!I have a few points to add. For one, I like using the BETWEEN join (when a surrogate key look-up is unavailable, for...
View Articlere: Debunking Kimball Effective Dates
Hi PMerrill,Thanks for the comment."I'm a big fan of questioning our assumptions, so thanks for this conversation"I'm glad at least one person out there appreciates it :)"what about the ETL Queries?...
View Articlere: Debunking Kimball Effective Dates
I'm a big fan of questioning our assumptions, so thanks for this conversation."The primary purpose of SCD start/end dates is not for end user querying, it is for the purposes of doing lookups during...
View Articlere: Debunking Kimball Effective Dates
Hi Ivan,Thanks for the comment."I think what has been lost in all this discussion is that the purpose of the start/end dates is to make QUERYING the data easier."I disagree. The primary purpose of SCD...
View Articlere: Debunking Kimball Effective Dates
I think what has been lost in all this discussion is that the purpose of the start/end dates is to make QUERYING the data easier. a one-time hit during ELT (if there is a performance hit) is...
View Articlere: Debunking Kimball Effective Dates
Oops - just realized - no DEFAULT required on the RowVer field as it gets handled by the insert logic. :)
View Articlere: Debunking Kimball Effective Dates
Hi folks,Just thought I'd post in an alternative approach which guarantees the correct dates, and also provides a row-versioning metric. For the record - this approach is optimized for processing the...
View Articlere: Debunking Kimball Effective Dates
p.s. I should also add that in Peoplesoft, in addition to effective date, there is of "effective sequence." In other words multiple rows with the same effective date (no time component), ordered by...
View Articlere: Debunking Kimball Effective Dates
While I agree that it's hard to enforce the end effective date reliably -- that's why they are never recorded in ERPs like PeopleSoft -- on the other hand, my job as a DW designer is to give...
View Articlere: Debunking Kimball Effective Dates
Kyle Hale,I am not sure why you are referring to get "next row's Start date" with Windowing function but to get the "next row's End date":SELECT [Id] ,[NId] ,[Name] ,[HomeTown] ,[MaritalStatus]...
View Articlere: Debunking Kimball Effective Dates
Can we get an update to this blog using a windowing function to get the next row's start date to avoid the join?Just to render that whole "INNER JOIN" argument moot.
View Articlere: Debunking Kimball Effective Dates
Jamie,Your concept overlays a bit with the Data Vault methodology. I think what is missing in much of the comments is "EDW" versus "MRT". In Kimball's world the marts are most often copies or...
View Articlere: Debunking Kimball Effective Dates
"profanely ignorant". Wow. Is it "profanely ignorant" to have an opinion that differs from that of ... ummm... someone else?
View Articlere: Debunking Kimball Effective Dates
That is profanely ignorant reasoning against Kimball's Type 2 SCD handling. Kimball calls for a data time stamp that provides a distinct time range for each dimensional change for a related record set....
View ArticleUsing Composable DML to maintain entity history [T-SQL]
I recently inherited a data warehousing SQL Server solution from my good buddy James Rowland-Jones (you
View Articlere: Debunking Kimball Effective Dates
Thank you Vishy, for pointing me to the site, as I really like Jamiet's scripts and the comments. The topic will definitely win a medal in Kimball forum.I think I remember seeing similar structure (one...
View Articlere: Debunking Kimball Effective Dates
Hello Vishy,I'm not sure I understand, the blog post above does not talk about issuing UPDATEs. Can you explain a bit more perhaps?JT
View Articlere: Debunking Kimball Effective Dates
I followed this but it updates all the previous rows with the startdate of most recent row.
View Articlere: Debunking Kimball Effective Dates
Update.... I have just used and tested this method and it was easy to implement and worked with my unsual scenario (BusinessKey > 1 in one load) perfectly.It also has extra benefit in my case of...
View Articlere: Debunking Kimball Effective Dates
Hi Emil,Thanks for the reply. Your situation sounds exactly like the sort of scenario that can be prone to error.I find it interesting that you are prioritising the validity of your data ahead of ETL...
View Articlere: Debunking Kimball Effective Dates
Jamie,I agree with your post completely. Even that we have a view on top of the table with the logic it is just much simplier and less error prone comparing to trying to handle it in ETL. I'm not...
View Articlere: Debunking Kimball Effective Dates
Hi,I am new to DW,This really good.But I would like to know ,What will be the Fact table structure,If I use SCD 2 with Start date and End date and Which date(start date or end date) will be pointing to...
View ArticleDebunking Kimball Effective Dates part 2 – Windowing Functions
Reigniting the [SCDEndDate] debate In November 2009 I wrote a blog post entitled Debunking Kimball Effective
View Articlere: Debunking Kimball Effective Dates
Kevin,,Interesting. I'm not really concerned because typically I would use this technique in BI systems which only get written to during an overnight batch. Definitely would be an issue for a highly...
View Articlere: Debunking Kimball Effective Dates
We use a similar schema in a production database storing only the start date. Recently, we ran into an issue where the data was changing in the table between the first read of the table and the second...
View Articlere: Debunking Kimball Effective Dates
Interesting thread. In response to some of the posts I'd like to point out that the view does not impact the majority of the end users query's. During actual usage of the data warehouse, the majority...
View Articlere: Debunking Kimball Effective Dates
Hi AaronBertrand,In reference to I/O being expensive. Using the proposed logic of having to join back to the table to find the successor record, ir going to cost you allot more than the storage of the...
View Articlere: Debunking Kimball Effective Dates
No problem. The thing people miss is that even in a denormalized DM, speed is a *secondary* consideration. It's easy to get fast results if you don't care how wrong they are. In every DM (at least...
View Articlere: Debunking Kimball Effective Dates
Hey Mike,Thanks for the replies.I guess it comes down to making the trade-off between perf and accuracy - I seem to spend my whole work life wondering exactly where to make that trade-off, y'know. In...
View Articlere: Debunking Kimball Effective Dates
@JaggedEdge - Don't be skurred.If GROUP BY skurrs you, then try a CTE with a windowing function (ROW_NUMBER) instead.Not sure why you'd find GROUP BY skurry. If something were to keep me up at night...
View Articlere: Debunking Kimball Effective Dates
Using Jamie's method of storing just a start date you can *accurately* infer the end dates. As has been pointed out already here, accurately enforcing end dates when you store the start and end dates...
View ArticleMind that collation : SSAS
I’ve been using Analysis Services a lot lately in my current day job and recently encountered an issue
View Articlere: Debunking Kimball Effective Dates
Wow, that enormous group by does not look very appealing. Not only is there another layer of abstraction, but when columns get added I have to update a view, and the group by is gonna make it crawl.
View Articlere: Debunking Kimball Effective Dates
Stuart,All good points. I'd like to ask you something about the following:"the ETL process should not govern the design of the warehouse based on updates"Is that still the case even if the columns in...
View Articlere: Debunking Kimball Effective Dates
I don't like the view option. Firstly, this is larger overhead than just getting the data directly from the table. Secondly, the ETL process should not govern the design of the warehouse based on...
View Articlere: Debunking Kimball Effective Dates
I will second the suggestion that this technique is not suitable for cases where dimensions are expired when no facts arrive (monthly snapshot paradigm, for example). In this case, there is no place...
View Article
