Showing posts with label SQL. Show all posts
Showing posts with label SQL. Show all posts

Friday, November 29, 2019

Bulk Insert With Dapper - The Right One

One of my favorite online trainers, Tim Corey, introduced Dapper for me two years ago. It was only the last week when I stumbled upon his advanced Dapper video, and implemented the correct version for a bulk insert, using this micro-ORM.

Prerequisites:
- install Dapper nuget package;
- intermediate C#/with MS SQL database;

The task of our bulk insert will be to insert the following data into an MS SQL database:

a. Table structure in SQL, defined by the query:

CREATE TABLE [dbo].[InvoiceSummary](
[id] [int] IDENTITY(1,1) NOT NULL,
[Inv_Number] [int] NOT NULL,
[IssueDate] [datetime] NOT NULL,
[BillingCode] [nvarchar](100) NOT NULL,
[EntityName] [nvarchar](200) NOT NULL,
[BUName] [nvarchar](100) NOT NULL,
[Value] [float] NOT NULL,
[VAT] [float] NOT NULL,
[TotalValue] [float] NOT NULL,
[Currency] [nvarchar](50) NOT NULL,
[ExchangeRate] [float] NOT NULL,
[Entity_Id] [int] NOT NULL,
[BU_id] [int] NOT NULL,
[UpdatedBy] [nvarchar](100) NOT NULL,
[UpdatedAt] [datetime] NOT NULL,
[PeriodID] [nvarchar](50) NOT NULL,
[Comments] [nvarchar](200) NOT NULL,
[Comment] [nvarchar](200) NOT NULL,
[Status] [nvarchar](50) NOT NULL,
[AttentionOf] [nvarchar](300) NOT NULL,
[CC] [nvarchar](300) NULL,
[HFMCode] [nvarchar](100) NULL,
 CONSTRAINT [PK_InvoiceSummary] PRIMARY KEY CLUSTERED
(
[id] ASC
)WITH (STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF) ON [PRIMARY]
) ON [PRIMARY]
GO

ALTER TABLE [dbo].[InvoiceSummary] ADD  CONSTRAINT [DF_InvoiceSummary_UpdatedAt]  DEFAULT (getdate()) FOR [UpdatedAt]
GO

b. There is a type defined, for the table-valued parameter, see SQL statement below:

CREATE TYPE [dbo].[BasicUDT] AS TABLE(
[id] [int] NOT NULL,
[Inv_Number] [int] NOT NULL,
[IssueDate] [datetime] NOT NULL,
[BillingCode] [nvarchar](100) NOT NULL,
[Entity_Id] [int] NOT NULL,
[BU_id] [int] NOT NULL,
[EntityName] [nvarchar](200) NOT NULL,
[BUName] [nvarchar](100) NOT NULL,
[Value] [float] NOT NULL,
[VAT] [float] NOT NULL,
[TotalValue] [float] NOT NULL,
[Currency] [nvarchar](50) NOT NULL,
[ExchangeRate] [float] NOT NULL,
[Comments] [nvarchar](200) NOT NULL,
[AttentionOf] [nvarchar](300) NOT NULL,
[CC] [nvarchar](300) NULL,
[HFMCode] [nvarchar](100) NULL,
[UpdatedBy] [nvarchar](100) NOT NULL,
[UpdatedAt] [datetime] NOT NULL,
[PeriodID] [nvarchar](50) NOT NULL,
[Comment] [nvarchar](200) NOT NULL,
[Status] [nvarchar](50) NOT NULL
)
GO

c. Stored procedure, to insert into our table, using table valued parameter:

CREATE procedure [dbo].[spInvoiceSummaryInsertSet]
@invsummary BasicUDT readonly
as
begin
set nocount on;
INSERT INTO [dbo].[InvoiceSummary]
           ([Inv_Number]
           ,[IssueDate]
           ,[BillingCode]
           ,[Value]
           ,[VAT]
           ,[TotalValue]
           ,[Currency]
           ,[ExchangeRate] 
           ,[Entity_Id]
           ,[BU_id]           
           ,[UpdatedBy], UpdatedAt
           ,[PeriodID]
           ,[Comment]
           ,[Status],
    [Comments]           
           ,[AttentionOf]
           ,[CC]
           ,[HFMCode],
   [EntityName],
   [BUName]
   )
SELECT [Inv_Number]
           ,[IssueDate]
           ,[BillingCode]
           ,[Value]
           ,[VAT]
           ,[TotalValue]
           ,[Currency]
           ,[ExchangeRate] 
           ,[Entity_Id]
           ,[BU_id]           
           ,[UpdatedBy], CURRENT_TIMESTAMP
           ,[PeriodID]
           ,[Comment]
           ,[Status],
    [Comments]           
           ,[AttentionOf]
           ,[CC]
           ,[HFMCode],
   [EntityName],
   [BUName]
from @invsummary


end;

d. Our original list in C# will contain the values we need to insert into the table:

// myInvList is a list of InvoiceSummary items defined by the class above at a)
InvoiceSummaryInsertSet(myInvList)

e. Our list is getting inserted into SQL Server DB using the table valued parameter

        public void InvoiceSummaryInsertSet(List<InvoiceSummary> myInvList)        {

            var dt = new ExcelServices().ConvertToDataTable(myInvList);

            using (IDbConnection connection = new System.Data.SqlClient.SqlConnection(connectionString))
            {
                var p = new
                {
                    invsummary = dt.AsTableValuedParameter("BasicUDT")
                };

                connection.Execute("dbo.spInvoiceSummaryInsertSet ", p, commandType: CommandType.StoredProcedure);
            }
        }

f. helper function to transform a list into datatable, used above, is implemented as below:

        public System.Data.DataTable ConvertToDataTable<T>(IList<T> data)

        {
            PropertyDescriptorCollection properties = TypeDescriptor.GetProperties(typeof(T));

            System.Data.DataTable table = new System.Data.DataTable();

            foreach (PropertyDescriptor prop in properties)
            {
                table.Columns.Add(prop.Name, Nullable.GetUnderlyingType(prop.PropertyType) ?? prop.PropertyType);
            }

            foreach (T item in data)

            {

                DataRow row = table.NewRow();

                foreach (PropertyDescriptor prop in properties)
                {
                    row[prop.Name] = prop.GetValue(item) ?? DBNull.Value;             

                }

                table.Rows.Add(row);
            }

            return table;

        }

That's all, it works ! Hopefully it helped some of you.

Thursday, November 28, 2019

The Challenges of Writing Business Apps/Tools


As mentioned in my previous posts on this blog, my developer journey began as an intersection between the corporate world (accounting) and programming. I used to support my own accounting processes and help others with various VBA/Excel, Access, SQL tools and that's how the journey began.

Outgrowing my role as a Management Accountant in this technological sense, in 2019 I began studying and writing full time additional applications for business/accounting problems.

The basic idea of the tools is automating Excel-based processes in a more coherent database-app approach. There are a lot of struggles in my journey, but and I have learnt a lot, especially .Net technologies and SQL, and in the meantime, the web-stack using JavaScript and lately ASP.NET Core/Razor pages.


Clearly, the biggest challenge in all these is the mapping of real world processes (based on Excel and manual work) to an application. What's needed for this to be successful? In no particular order, my conclusions are:
1. very good understanding of the processes. This takes time and lots of communication/questions.
2. Understanding inputs and outputs. Data structure, types, formats, steps, validation, internal checks of the applications, stages of completion, reports, application logic and flow all come from inputs and outputs.
3. Thinking about the entity/DB structure and program flow.
4. From the mental mapping above come the DB structure, and then flow. Usually the flow is the succession of operations: input data, review data, edit, delete, search, etc, add more data, filter, prepare intermediary reports and then further add data, filter, edit if needed and then generate final outcome (reports, files). This cycle above can have multiple stages and checks/reports built within.
5. Validate the proposed process with the customer.
4. Discussing old methods vs new optimized process: lessons learned and areas to improve on in the new system based on feedback.

My big projects in the last 2 years are:
1.  Manufacturing Reporting/Statistics tool in MS ACCESS, on key phases of the production line. (manufacturing shop floor).
2.  Internal recharge system of the company, for distribution of indirect/administrative costs. Technology: C#, .Net, WPF/MVVM and SQL Server
3. My current project is an invoicing software that collects and invoices costs by employees. Using same technology as above. + Asp.net Core Razor pages
4. Another project for 2020 will be a VAT reporting tool to automate an Excel-based approach inside the accounting department, consolidating, filtering, automating data collection from several internal business units (big departments), and preparing statutory reports for Value Added Tax.

Some clear lessons to change the Excel-Based approach into a more efficient application are:
1. Using one database instead of many files, on a central server/Azure, for each app.
2. Mapping the entities from the excel files => as database tables/ and POCO entities.
3. All the automation of the repetitive processes go to C# code = > helper, data manipulation functions, using LINQ with lists etc.
4. Report generation, using ReportViewer tool, or ClosedXML to export excel files upon request;
5. Code re-usability.

Some architectural conclusions of the journey are:
1. need to manage the users of the application
2. implementing a reporting system, with parametrised queries, using SQL/Dapper/Reportviewer and/or simple Excel reports
3. implementing a concept of "Period": the accounting month in which all transactions take place. Afterwards the period is closed, and next month is opened for another series of transactions.
4. Implementing different types of Tables / entities by functionality:
  • tables with reduced number of updates such as business units, customers, users
  • tables with some  more frequent updates (such as employees, price lists)
  • transaction tables (allocation keys calculated, monthly cost information, monthly invoice details, etc)
5. For transaction tables, insert the user and transaction date/time 
6. In some special cases, implement a history table for some entities for which the change log is important  (such as employee change history).
What do you think, what are the main points of producing such business/accounting tools? Would you do something differently from an architecture standpoint?


Thursday, November 7, 2019

Bulk Insert in Dapper into MS SQL

I am collecting costs by employee from 6 sources and then insert them into a a GeneratedDetail table. But the collected data has 1000+ rows, and the insertion one by one row is very slow. I needed to find a bulk insert solution.
SQL Server DevOps
I am using Dapper as ORM in my C# project, and MS SQL Database.
According to research, my approach would be as below. It works now quite fast, I am satisfied with the speed of execution. Feel free to comment and propose a better solution.
note: spGeneratedDetailInsert  is a stored procedure with many parameters.

        public  void GeneratedDetailInsertBulk1(List<GeneratedDetail> DetailList)
        {
            using (IDbConnection connection = new System.Data.SqlClient.SqlConnection(connectionString))
            {
                var sql = "";           
                foreach (var mydet in DetailList)
                {               
                    sql = sql + "Exec spGeneratedDetailInsert '" + mydet.BillingCode + "', " +
                        mydet.Emp_id + ", '" +
                        mydet.Emp_name + "', " +
                        mydet.HrId + ", " +
                        mydet.Entity_id + ", " +
                        mydet.BU_id + ", '" +
                        mydet.Currency + "', '" +
                        mydet.Item + "', " +
                        mydet.ExchangeRate + ", " +
                        mydet.InitValueRON + ", " +
                        mydet.InitValueCurr + ", " +
                        mydet.MUValueCurr + ", " +
                        mydet.TotalRon + ", '" +
                        mydet.ServiceType + "', '" +
                        mydet.CostCategory + "', '" +
                        mydet.UpdatedBy + "', '" +
                        mydet.UpdatedAt + "', '" +
                        mydet.PeriodId + "' ; ";                                   
                }
                 connection.Execute(sql);
            }
        }
Note: I am not using Dapper Plus, and I don't want to.

Dapper Plus | Learn how to use Dapper Bulk Insert with ...

Tuesday, November 5, 2019

Programming Is The Easy Part

I was wondering lately what is the real challenge in my current project, and what is the easy part. Do I manage the challenges the right way? Can I find some optimizations?

My answer to the challenge question is yes, there is a challenge. Understanding really well customer requirements is. What is the business process behind? What are the entities and properties?
What tables to I have as an input? Fields set as a flag? Certain situations require different treatment?
How much freedom has the user, to manipulate or corrupt data?

Yes, processing the real data that goes in the application is the real challenge, mapping it to entities that can survive the changes of the life of the application.

And what to do when the input data has some flaws? Data duplication, bad codification, violation of uniqueness... how to bridge this in my database?

Writing code is the easy part.

1. there is some routinely written code: SQL queries  - 80% are repetitive, following the same pattern (not exactly same commands). Writing the entity classes and the ORM functions that map DB procedures to entities, almost totally a routine.
2. Some semi- routinely written code - linking the WPF layout with bindings to entities, setup the ViewModel and View, draw basic XAML (copy-paste, than change)... Oh how I hate this XAML.

3. The interesting part is to write some algorithmic functionality: saving files and creating emails, generating views with filtered data, writing some more complex LINQ queries, generating invoices, creating Outlook Emails, Reports with Reportviewer...

What is then the hard part?

I think understanding the requirements really well and mentally mapping out a process how to tackle the solution step by step. Visually, View by View, Tabs, Tables, Reports, and a general succession.

Then comes the execution, test, populate with data, correction, again test, again addition of data, etc.


And finally the customer feedback, and again, corrections, optimization, refactoring, test, submission to the customer.

And after several attempts, there is the prototype that can be used at least on the short term, before the next iteration.

That's my approach. What do you think, what's your process?

Lots of SQL today

As part of my big project, today I dealt with some table creation and query writing. Mentioned earlier, I use Dapper as ORM and this means no EF queries are generated automatically.
So all Insert, Update, Select etc queries have to be written manually and returned and mapped to POCO entities (simple classes) via Dapper.



I think 2 hours was spent today on this.

I created 3 big tables (10+ fields) and wrote all CRUD queries as mentioned above.

Then, using a helper class, the ORM mappings are written in one place, just to call these functions within the application, when needed.

Some demo, from Tim Corey: https://youtu.be/QVkpzuiiVtw

An the foundation, for Stored Procedures, I used this: https://youtu.be/Sggdhot-MoM

What do you think? IS EF much more efficient?

By the way, my small prototype project in .Net core Razor Pages uses EF Core, which I wanted to learn step by step to get accustomed to it. It is nice, too, I like it! For a simple web project, it is enough for my purposes.

Sunday, November 3, 2019

My first big project (.NET) - and technologies used.



In the summer of 2017, together with my former finance manager, I began to design an internal recharge system for the company.
The choice was WPF for Windows (https://docs.microsoft.com/en-us/dotnet/framework/wpf/getting-started/walkthrough-my-first-wpf-desktop-application) using the old .NET, and an MS SQL Database.

We began to learn, exercise and think about the tools to use:
Our inspiration was: https://www.iamtimcorey.com/
1. IDE: Visual Studio Community 2017
2. WPF - for the visual part, front-end
3. ORM - Dapper - I think this is the best discovery ever! Tim speaks about this in his Videos.
https://www.iamtimcorey.com/blog/137806/entity-framework 

Please, check out his videos, about SQL, C# and lots of serious tutorials and courses.
 (https://stackexchange.github.io/Dapper/)
See also: https://dapper-tutorial.net/
4. MS SQL - I needed to review my SQL knowledge. Did a lot of exercises, and build many stored procedures for the application - dapper maps the entities to the data returned by those stored procedures, and I also used SP to insert and update entity data. Even if this was very time consuming in the beginning, proves to be very efficient and goal oriented.
5. lots of company data about expenses and the algorithm to create allocation keys - business logic designed together with Finance Manager.

The journey was hard. It took me probably 1 year to really get some speed, and show something to my Finance Manager colleague.

Luckily, I had my finance background, to understand the process we tried to automate, as there was a working excel macro/formula model as well used before the application.

But, I needed to fix my knowledge of C# (also using MVVM the first time for WPF) and SQL/Dapper. This meant weekly 2-3 hours of coding for 1 year to get some traction.
Main source for C# was google and Tim Corey, and my big discoveries were:
  • lists/collections of items 
  • LINQ and the foreach 
  • breaking down the tasks to smaller chunks and functions
  • writing a services class to export lists to excel using OpenXML
  • first steps in async/await

A special challenge was how to integrate the good old WinForm control ReportViewer into WPF and MVVM. I will detail these maybe in another blog post.