Where should data processing and transformation happen? [closed]

Question

I'm using Angular 4 as my front-end framework and basically just asp net core as an API for the client to get data from and occasionally post data.

I'm making a dashboard to display some statistical data. The data that I have in the database is not directly usable for this dashboard as it needs to be transformed and further processed. I'm not returning the model entity to the API but a View Model which is basically the same as the model entity with some ids removed etc.

What I have right now is that my Angular DashboardComponent is using a DataService to construct a HTTP GET request which points to my already existing API endpoint GetOrders. This will get all orders from the database and return it to the DataService and eventually the DashboardComponent which then needs to extract data and do a whole lot of processing and transformation so the data is ready to be given to Google Chartsand meaningful charts can be generated. Then the user can choose to see another chart on the dashboard which then is not enough to just have the Orders but another entity must also be retrieved from the database, so repeating above API call using a different end point.

My question boils down to:

• Is it best to do all of the processing and transformation on the server side and return data to the client which is almost ready to be presented, or is it fine to leave the client with this task?
• In case of server side processing and transformation, should I construct it in such a way that I have API endpoints that corresponds to the type of data that I want to show in the dashboard? So, selecting for example "Orders in the last 3 months in the UK" would send a request to an API endpoint that specifically takes parameters such as country and period?
• Maybe the data should already be in the shape I can use for the dashboard via some sort of datawarehouse? Is this the way to go with this?

In general I'm looking for some design patterns on this or just best practices, but haven't really been able to find something.

Answer 1

• Is it best to do all of the processing and transformation on the server side and return data to the client which is almost ready to be presented, or is it fine to leave the client with this task?

  Prefer for the server side processing and transformation. That way if your object model/entities changes, the client can be made agnostic that something changed. There are more reasons but that one should be one of the bigger ones.

• In case of server side processing and transformation, should I construct it in such a way that I have API endpoints that corresponds to the type of data that I want to show in the dashboard? So, selecting for example "Orders in the last 3 months in the UK" would send a request to an API endpoint that specifically takes parameters such as country and period?

  You would have to decide on this depending on the use cases that you're going to deal with and balance that with performance considerations and reusability (e.g. should you go for 1 method that will return a big object with lots of data or multiple methods that return small objects with minimal data). No clear cut rules here.

• Maybe the data should already be in the shape I can use for the dashboard via some sort of datawarehouse? Is this the way to go with this?

  No clear cut rules here either. Scale of data and budget will define this. You could shape your data using .Net code inside your .net core project or you can have a massive data warehouse scaling Petabytes of data...both are perfectly valid and ridiculous answers depending on your use case.

General "gut-check" things that could help you determine what to put where...

• You would want your view to have as close to zero data-related calculation, all the logic in your view should be view specific

  e.g.

  • OK to have in view: if something is true/false hide this/that and disable this/that

  • Belongs outside of view: order_tax = tax_rate_for_state * total;

• If someone else wants to write a view to display the statistical heavy lifting that you did, would your current api (asp.net controller method) isolate the person writing the view from knowing anything additional? A yes here is what you should be striving for. e.g. If an iPhone app calls your api GetOrderTotal(orderNumber), that iPhone app should not be aware of your discount structure.

• When you have the controller method returning a shaped result, is the code to shape the data in a single place? Having some of the work done at one area and some at a different area just makes debugging a lot harder. An example that you would not want to happen is

  [HttpGet]       
  public double GetOrderTotal(string orderNumber)
  {
      // lets pretend here that we have a stored proc that returns an orderTotal
      var orderTotal = _orderService.GetOrderTotalUsingStoredProc(orderNumber);
  
      // this if else if block should be pushed at to the _orderService method 
      // or the OrderTotal stored procedure, otherwise any other code that relies
      // on _orderService.GetOrderTotalUsingStoredProc might not know that discounts
      // are applied to certain threshold amounts
  
      if (orderTotal > 1000)
      {
          orderTotal = orderTotal * 0.9
      } 
      else if (orderTotal > 10000)
      {
          orderTotal = orderTotal * 0.85
      }
  
      return orderTotal;
  }