Was playing with Malloy and DuckDB and had this silly idea, can I just reproduce something Like PowerBI Desktop Just using Open Source Tools, obviously I am talking about the Overall architecture, not exact functionalities.įor a Typical Data Analytical Workload, Broadly Speaking you get something like this : Traditionally Big Data means Data that don’t fit into RAM, nowadays, Big Data is how much Data can fit in your SSD, and that’s a welcome change.

The duration will depend mainly on your CPU and SSD speed, DuckDB consume little memory, in that example less than 2 GB, the secret sauce is parallel scan of Parquet files.

Select L_RETURNFLAG,L_LINESTATUS,count(*) from In PowerQuery you need just to insert a python script, you don’t need to know anything about python, it is just a standard SQL Query, PowerQuery is smart enough to know that df is a dataframe, and it will show the result in the next step. In this example I have a folder of 100 GB of Parquet files. Let’s say, you have a folder with 100 GB of parquet files, and you want to aggregate some results for further analysis in PowerBI, Obviously importing 100 GB maybe not be a great idea, in this scenario, from what I can see, people recommend stuff Like PySpark or Dask, which I guess are great option, but if you want a lightweight option with a simple pip install then DuckDB is amazing !!įor testing, first write your SQL Query using only 1 parquet file, when you are happy with the results, change the path from “folder/x.parquet” to “folder/*.parquet” It is just a POC that showcase you can do a lot just using a decent laptop and python. In this way, users who share project files with other parties will keep schedule integrity and protect sensitive data, which is one of the most important aspects of any Primavera P6 oriented organizations.Maybe not known enough, but you can run a python script inside PowerQuery in the desktop, and deploy it to PowerBI service using a personal gateway. Convert Global to Project activity codes.

Other data categories that can be removed too are: I showed an example for removing costs such as project funding, budget changes, spending and benefits plans, project expenses and assignments costs as this is the most usual case. What else can you clean in XER project file? Moreover, user can setup and reuse Templates that contain different modifying settings to a single or group (Batch Clean) of XER files.

This was the automatic way to clean and prepare XER files for distribution and sharing. Step 2: Select the categories to be deleted Step 3: Your “Cleaned” file is ready

Primavera sample project files designs