{"id":184,"date":"2009-08-19T21:40:34","date_gmt":"2009-08-20T02:40:34","guid":{"rendered":"http:\/\/www.ssas-info.com\/VidasMatelisBlog\/?p=184"},"modified":"2009-08-19T21:40:34","modified_gmt":"2009-08-20T02:40:34","slug":"part-2-gemini-%e2%80%93-how-it-performs-with-bigger-tables","status":"publish","type":"post","link":"http:\/\/www.ssas-info.com\/VidasMatelisBlog\/184_part-2-gemini-%e2%80%93-how-it-performs-with-bigger-tables","title":{"rendered":"Part 2 &#8211; Gemini \u2013 how it performs with bigger tables"},"content":{"rendered":"<p>Yesterday I posted about <a href=\"http:\/\/www.ssas-info.com\/VidasMatelisBlog\/177_sql-server-2008r2-self-service-bi-gemini-how-it-performs-with-bigger-tables\" target=\"_blank\">my tests working with Gemini and bigger tables<\/a>. I realized myself and <a href=\"http:\/\/cwebbbi.spaces.live.com\/\" target=\"_blank\">Chris Webb<\/a> also suggested that my method of generating new records by simply duplicating them probably affected my results.\u00a0So I ran\u00a0more tests with different data. <!--more--><\/p>\n<p>I generated a new table by adding random number to the existing numeric fields (dimension keys and amounts). From the new table I was able to load just about 17mln rows into Gemini.\u00a0I\u00a0was getting the same memory error message when I was\u00a0trying to load more records.\u00a0My load speed was about 3.5mln rows per minute.\u00a0Saving\u00a0the Excel workbook took 75 seconds this time, and the xlsx file\u00a0was much larger\u00a0&#8211; 448MB. While working with my data set, Excel was using 740MB of RAM. This time opening an\u00a0existing Excel workbook took me 85 seconds.<\/p>\n<p>But although\u00a0with random data some operations were slower, I still was able to confirm that after loading data,\u00a0all filtering\/sorting operation were very fast and all pivot queries were returning results almost instantly. So duplicate or not duplicate data, if you\u00a0are able to fit it into memory, then Gemini will handle it with amazing speed.<\/p>\n<p>During my tests I realized that the amount of data you will be able to load into Gemini will depend entirely on your data. During my initial data &#8220;randomization&#8221; attempt I did not rounded my numeric results and I had numbers like 1.234567890. With such data I was able to load into Gemini just 4mln rows and the size of Excel workbook was about 580MB. After applying rounding to the same fields I was able to load 4 times (!) more data &#8211; 17mln rows. So when you will build you Gemini models, make sure that for bigger fact tables you load just the fields that are necessary for your analysis and make sure you round your numeric values for any calculations. There are no miracles &#8211; every character uses memory space and you need to minimize usage of that space as much as possible.<\/p>\n<p>I am still learning Gemini and I am still impressed with results.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Yesterday I posted about my tests working with Gemini and bigger tables. I realized myself and Chris Webb also suggested that my method of generating new records by simply duplicating them probably affected my results. So I ran more tests with different data. <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[17],"tags":[],"aioseo_notices":[],"_links":{"self":[{"href":"http:\/\/www.ssas-info.com\/VidasMatelisBlog\/wp-json\/wp\/v2\/posts\/184"}],"collection":[{"href":"http:\/\/www.ssas-info.com\/VidasMatelisBlog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.ssas-info.com\/VidasMatelisBlog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.ssas-info.com\/VidasMatelisBlog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.ssas-info.com\/VidasMatelisBlog\/wp-json\/wp\/v2\/comments?post=184"}],"version-history":[{"count":3,"href":"http:\/\/www.ssas-info.com\/VidasMatelisBlog\/wp-json\/wp\/v2\/posts\/184\/revisions"}],"predecessor-version":[{"id":187,"href":"http:\/\/www.ssas-info.com\/VidasMatelisBlog\/wp-json\/wp\/v2\/posts\/184\/revisions\/187"}],"wp:attachment":[{"href":"http:\/\/www.ssas-info.com\/VidasMatelisBlog\/wp-json\/wp\/v2\/media?parent=184"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.ssas-info.com\/VidasMatelisBlog\/wp-json\/wp\/v2\/categories?post=184"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.ssas-info.com\/VidasMatelisBlog\/wp-json\/wp\/v2\/tags?post=184"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}