Migrating bad system into our current system with tons of data [closed]

Question

I inherited of a system when my company bought another company. This system is a mix of LAMP and .NET.

1. 1 Windows server running asp.net that controls third-party validation used only for apis and webservice - (let's call it WIN)
2. 8 LAMP servers (web, reports, cron, repositories etc...) - (let's call it NEW)

Our current environment:

14 LAMP servers (web, mail, repositories, etc...) - (let's call it CURRENT)

The good news is the NEW code is pretty straigt foward. A few millions lines of code (which most are apis, third party) and I can convert it to the CURRENT system. The NEW and CURRENT use both CentOs which will make the transision easy, except the Windows server that I have no idea what to do now.

Now the bad news. The NEW system database schema is not good at all. It is not normalized and the queries are slow (database queries and the code too). My first idea is to redesign them to a more normalized structure that matches the CURRENT code but I don't work. The tables from the NEW system are gigantic. The NEW system has 7 databases, over 10000 tables, the smallest tables have over 100k rows and some tables have over 500 millions rows. One of the database has most of the tables with over 25 millions rows each.

Is it safe to migrate or should I keep both running? If I should migrate I want to know what will the safest solution for me in order to migrate the Windows and NEW system into my CURRENT system?

Answer 1

First of all, moving the WIN + NEW systems into the CURRENT system will take time. so you have to make sure when you start migrating/converting everything, you know where you going. Migrating might not be an easy task and you can run into issues you never thought after you started the process.

Your idea of migrating the NEW system has pros and cons and you need to make sure it goes smoothly in order to get a good and reliable product at the end.

Pros:

• only one system to maintain: you don't want to maintain 3 systems;
• one code/database environement: PHP vs ASP.NET and MSSQL vs MySQL;
• centralized code/database;
• one coding (code and database) standard;
• save/sell equiments (you will migrate the code to your 14 servers, maybe you don't need the other 9 (WIN+NEW), so you can sell or keep them for the next projects)

cons:

• higher risk (crash, incompatibility, unknown feature you need to understand etc..);
• cheaper than migrating or re-designing everything
• risk of failure is lower than migration (since you already know both system works)
• planning, control, implementation, quality assurance: very bad if one of these fail;
• expensive: migrating can be expensive in time and money;

This is a big database, changing/optimizing this will take a significant investment in labor hours. This is not something you can easily do in few hours. This can take weeks maybe months in order to successfully migrate data to the CURRENT system without errors. If you can, you can start by migrating common or similarities from both database schema, like clients or products. This way, you import data that the CURRENT system can run without errors and also your code will recognise. The users of your CURRENT system can immediately start managing these items/records without issue. As of new or records your CURRENT system does not recognise, you can simply re-design these tables and migrate them to the CURRENT system (then update your current code).

As of code migration, if the code from the NEW system is good enough and match your standard you can keep it. This will save time in development, just make sure you update the queries and servers connections. On the other hand, if it's like spaghetti code, you will have to understand what the code do. This can also take a significant investment in labor hours. I can recommend here to standardize this one and organize your code the same way you organize it on the CURRENT one. You can centralize your code in one common folder using a common file and folder structure. You can put all your common libraries, 3rd parties etc.. there so when you call the CURRENT and the NEW code, it loads the same PHP class. This will make your ease your transitions from the NEW to CURRENT system. This way, you know all your required files at in the same place and very easy to maintain. Specially if your code requires required files that requires files. If you code is all around your servers, you can create an NFS if you like this idea.

Now, what I can suggest is to start with a Parallel Adoption . This way you make sure all systems are working properly and are healthy. Then slowly migrate data/code to the CURRENT system until everything is completed. This won't be easy and you have to identify which part of the NEW + WIN systems you have to migrate first. My recommendation will be to migrate the WIN system. Because this is independent of the CURRENT and NEW systems, as long you display the same output you should be fine. Search for open source or similar validations in PHP or if you can't find any, build them. This way this WIN system can easily be migrated to your current organization structure and coding standards. Performing tests and quality assurances will be easy and you can be completed very quickly.

Once this WIN is migrated, you need to identify what you need to transfer first to the CURRENT system. For example, if the NEW and CURRENT system has "clients", gather all the information from the NEW system and move them to the CURRENT system using a script (manually or scripted). Then, you can migrate clients elements like products, billing statement or any other records that are related to these clients). Repeat these steps until all the data are migrated. This way, you don't have to re-design any tables or change any codes from the NEW system, everything is saved on the CURRENT system and everything works correctly.

I won't recommend the big bang adoption for this case .

Answer 2

The should i migrate is not really your decision but more you're bosses decision.

For the how, it depends on the amount of data crunching you need to do. I would obviously recommend working the data from SQL to SQL using inter-database statements but that is not always feasible.

The first reason is that you might need to transform really complex data into another form and can't do it using pure SQL.

The second, would be that if coding a transfer script in PHP or any other language would take as much effort than doing it in pure SQL, you're better off making it in PHP so that when you have something to change you're sure it will be possible and not be stuck with the fact that you are in SQL and need to recode everything.

Overall, you have a huge analysis to do to understand all the data you are dealing with and plan your transfert. With the number of tables and rows you are talking about you have one hell of a trip ahead of you, i think tylenol will be your best friend.

Good luck