Multi-project logic reuse and monorepo

Multi-project logic reuse and monorepo

This article is synchronized on the personal blog shymean.com , welcome to follow

I took over a new project last year, which is mainly divided into mobile terminal and PC terminal. In the early stage of the project, both ends assumed different business responsibilities. In the later iteration process, some identical and similar business logics appeared one after another. Is a lot of similar code. Recall that before, I have experienced similar scenarios, such as splitting a to B project and then splitting a to C project. The two projects have business or platform differences, but they also have many similarities.

This article mainly researches through

monorepo
To deal with the logic reuse problem in this kind of multi-project.

Single item

For a single project, the same logic is often based on files encapsulating modules or components, and then common logic can be reused by introducing modules.

Below is a common single-project directory structure

App.vue main.js api assets components router store util views Copy code

Multi-item

There may be inextricable connections between multiple projects, such as similar development environments, the same business logic, etc.

scaffold

If multiple projects use similar technology stacks or development environments, they can be implemented by encapsulating scaffolding cli, you can refer to

vue-cli
,
create-react-app
Wait

Logic reuse

With the gradual increase of development projects, we will find that certain business scenarios will be frequently encountered. The easiest way is to copy it and use it with a little modification.

Experience tells us that it is definitely unwise to do so, so how to deal with the common logic before multiple projects?

Encapsulating common logic into modules is the easiest way. Because it is a cross-project module, you need to find a place to host it. That's right, it's the npm package.

Publish modules to the module repository, and install the corresponding module dependencies for each project. This is the most common practice of front-end modularization. There are countless modules in the npm repository.

However, in actual development, this approach has some flaws.

Debug local modules

The first is the deterioration of the development experience. It is undoubtedly the most convenient to put the source code together for debugging; but if it is based on modules, you must take the path of node modules module, develop module -> package module -> release package -> re Installing a new module -> Using a new module is the same as changing the code locally and then publishing it online for debugging. It is troublesome to think about it, and it violates the meaning of engineering.

npm provides a way to install local packages,

npm pack
Package the module directory into tgz, and then you can pass
npm install xxx.module.tgz
, So you can install the local module package directly, the disadvantage is that you need to manually package and then reinstall the dependencies, which is not flexible

Can you bypass the manual packaging step? You can install the module directly through the relative path,

npm install path/to/module
, This can also install the local module, but when the module changes, it needs to be reinstalled

When the local module is changed, can I not reinstall it? can use

npm link

# Execute the link command in the module directory to link to the global module cd path/to/module1 npm link # Execute l in the project directory cd path/to/project npm link module1 # Remove link npm unlink module1 Copy code

The link itself is just a soft link to the directory. If you change the development environment, you need to do it again, so it is limited to single-person local debugging of local modules.

Maintain dependent projects for each module

The version of the npm package follows

SemVer
Specification, i.e.
XYZ
In the format, X is the major version number, Y is the minor version number, and Z is the revision number. Each element must be incremented by value.

  • Major version number (major): When you make an incompatible API modification
  • Minor version number (minor): When you make a downward compatible functional addition
  • Revision number (patch): When you make a downward compatibility problem correction.

Check the installed library of package.json

  • XYZ
    , Install the specified version directly
  • ~XYZ
    , Will install the latest version in the current minor version (that is, the number in the middle), such as
    ^3.1.0
    The last version actually installed may be
    3.1.10
    , But will not update to
    3.2.x
  • ^XYZ
    , The latest version in the current major version (that is, the first digit) will be installed, such as
    ^3.1.0
    The last version actually installed may be
    3.10.0
    , But will not update to
    4.xx
    , It is currently the default caret of npm install

This operation of adding caret by default without specifying a specific version may bring some unexpected results. If all modules follow the standard

SemVer
It is of course the best to update the version number according to the specification, but if the wrong version is used, such as a completely incompatible version, the code will throw an exception.

In order to solve the problem of installing different packages in different environments, there has been

package-lock
And other programs.

If we do not trust this automatic update mechanism, we can install all modules in a fully specified version.

This will bring a new problem. After the public module is released and updated, if you want to update the project that depends on this module, you need to manually modify the version number of the module in each dependent project.

In other words, although we have encapsulated the public module, we still need to maintain all projects that depend on this public module. Imagine that a popular public module is dependent on 100 projects with a deadly version number. Modification is also a headache.

monorepo

If there is a new module introduction method that can immediately perceive the changes of local modules like npm link, and automatically synchronize the dependent versions after the local module version is updated, will it be able to solve the above two problems?

Think about it in a single project, we haven't even considered these issues. This is because there is nothing easier to debug than putting the module source code and the project source code together.

The way to put the code of different projects together is called

monorepo
. Suppose there are two projects,
app-mobile
with
app-pc
, They rely on
mod1
,
mod2
,
mod3
, The solution adopted by monorepo is to put these 5 projects in the same warehouse

Therefore, large-scale front-end projects gradually adopt monorepo as the management method of project code. Its main feature is to manage multiple npm packages in a single warehouse, such as

React
,
Vue
And other projects currently adopt this structure.

In addition to open source projects , monorepo is also very convenient for the multi-project logic reuse discussed earlier in this article .

lerna

reference:

Lerna provides a way to quickly build a monorepo repository, and optimize the workflow of the monorepo project by using git and npm. lerna provides a lot of instructions, such as creating modules, managing module dependencies, publishing modules, etc.

New monorepo project

# Initialization, lerna.json configuration file will be generated lerna init # Create module 1, which is located in the packages/test1 directory lerna create test1 # Create module 2 lerna create test2 # Add test1 as a dependency of test2 lerna add test1 --scope test2 # Submit code git add. git commit # Modify the git tag, update the version numbers of all modules, and publish the package to the corresponding npm warehouse. For testing, you can publish to the local private warehouse through verdaccio lerna publish Copy code

If you are pulling a monorepo project, you need

# Install the dependencies of each package, then automatically establish links for the interdependent packages, and finally execute npm prepublish lerna bootstrap Copy code

The following are some problems encountered in use.

Question 1: When the local code is only modified without committing,

lerna updated
Unable to view package changes. You need to submit before you can see the changes. If there is a package change, and then continue to modify other packages, other packages are not submitted in time, you can also view the corresponding package changes.

for example:

  • Modify mod1, when not submitted, lerna updated returns empty
  • Modify mod1, and then submit it, no release, lerna updated back to mod1
  • Modify mod2 again at this time, without submitting and publishing, at this time lerna updated returns mod1 and mod2

You can study the detection principle of lerna updated.

Problem 2: Unable to implement custom publishing scenarios for certain modules

Refer to issue1691 , issue1055 , based on lerna's feature of using git tags, either full release or none at all.

lerna will not publish private packages, but it also needs to update the corresponding version when publishing,

Question 3: The version of each package is the same, although it was not updated in a certain commit

in

Fixed/Locked
(Default) In the mode, the version numbers of all packages are the same, which is maintained
lerna.json
of
version
In, you can pass
lerna init --independent
select
Independent
Independent mode, each package can have its own version number, and each package has its own version number
package.json
maintain

Question 4: If multiple modules are dependent on third-party modules, when leran add is used, it will be installed once under node_modules of each module

The figure below shows that both mod1 and mod2 depend on axios. When lerna bootstarp is executed, axios will be repeatedly installed.

This is because lerna, as a multi-module process management tool, thinks that the

package.json
Are independent, so they will be called repeatedly during installation
yarn install
Many times, each module will repeatedly install some common modules.

Question 5: In a project that depends on other partial modules, running yarn install alone will destroy the link of lerna itself and install the wrong module

The figure below shows

lerna add mod2 --scope=mod3
after that,
cd packages/mod3 && yarn
, It will be installed to the wrong mod2 (on npm a wrong package with the same name ) instead of what we expected
packages/mod2

In addition to lerna, there are multiple monorepo management tools such as Bit and Bazel. The details can be moved to: 11 excellent tools to manage the Monorepo code base in 2021 .

yarn workspace

reference

Questions 4 and 5 mentioned above are flaws of lerna, lerna itself only manages the module establishment, association and release process, not like

yarn
,
npm
Same as participating in the actual package installation, the work of repeated installation and intelligent link identification and installation should be handed over to the package management tool, so yarn implements this function:
yarn workspaces

Yarn workspaces mainly solves the problem of installing dependencies from different package.json in multiple subdirectories. The following is a workspace project

package.json

{ "name" : "workspace_demo" , "version" : "1.0.0" , "main" : "index.js" , "license" : "MIT" , "private" : true , "workspaces" : [ "packages/*" ] } Copy code

All modules are placed

workspaces
Under the directory, follow
lerna
the same. Usage practice: As yarn and lerna have a lot of overlap in function, we use yarn's official recommendation method, use yarn to deal with dependency problems, and lerna to deal with release problems.

Execute in a workspaces project

lerna init
, It will be automatically converted to lerna workspace mode

{ "packages" : [ "packages/*" ], "version" : "0.0.0" } Copy code

Then use lerna to handle module creation and release, and yarn worksapce to manage dependencies

# Quickly create a module lerna create mod1 -y lerna create mod2 -y # Axios will be installed under the project root node_modules, which avoids the above problem 4: Repeated installation of the same dependency yarn workspace mod1 add axios yarn workspace mod2 add axios # Declare dependency, mod2 adds dependency mod1, it should be noted that the local package version number needs to be specified here yarn workspace mod2 add mod1@0.0.0 # Local development and one-pass modification, mod2 will instantly obtain the content of mod1 module, local development and debugging is very convenient # Follow-up lerna publish Copy code

Practice in the project

In actual development, it is not necessary to publish the split modules to npm or private warehouses. Our original purpose is to reuse the logic across projects. After using monorepo, make a project centralized under the same git warehouse, so that you can easily carry out logical reuse, development, debugging, and release.

Assuming that the mobile and PC projects mentioned in the title now need to be refactored, except for the difference in UI, many of the logics are the same. After adopting monorepo, the general structure of the entire project is as follows

package.json packages mobile # Mobile terminal business module pc # PC-side business module common-util message-box request storage track-log tsconfig.json yarn.lock Copy code

Advantage 1: Split modules through workspace, and the code in a single workspace is easy to transplant, so that you don t need to think too much about the division of projects in the early stage of development. In a monorepo project,

  • Multiple reused npm packages, including common components and logic, etc.
  • Business deployment project, which depends on other npm packages

Write code in the business module, and when it meets the common logic of multiple modules, it is split into independent modules. The split modules can also be easily migrated to other public component libraries. It is difficult for us to divide the responsibilities of each module at the beginning of the project. In monorepo, this way of splitting modules at any time can ensure that we are more convenient to transplant and refactor during the iteration of the business.

Advantage 2: Because it is under the same project, the PC terminal and the mobile terminal can use the same development environment, and the local public modules can also use the same development environment.

Based on this, each module can even publish the source code, without the need to build an independent development environment for each module, such as configuring ts, configuring babel, configuring rollup packaging, etc. (unless our modules need to be published to other warehouses for other different The personnel of the development environment use), which greatly improves the efficiency of development and debugging.

A demo

Based on this idea, I wrote a monorepo demo, github address , and introduced the reuse of conventional modules and jsx component modules in multiple react projects.

The entire project only completed the part of local multi-module development, and did not supplement the part of building and publishing to private warehouses such as verdaccio. It seems that it should be able to meet my initial needs.

The following are the specific steps to build the entire project.

First create the directory

mkdir monorepo-demo && cd monorepo- demo copy the code

initialization

package.json

yarn init -y duplicated code

Modify package.json, configuration

workspaces
Field

{ "name" : "monorepo-demo" , "version" : "1.0.0" , "main" : "index.js" , "license" : "MIT" , "private" : true , "workspaces" : [ "packages/*" ] } Copy code

Initialize lerna

lerna initcopy code

Create the first public module mod1

# The default version created is 0.0.0 lerna create mod1 -y Copy code

Modify the source code of mod1

packages/mod1/lib/mod1.js

module .exports = mod1; function mod1 () { console .log( 'mod1 ' ) } Copy code

Then create an application module, here use

create-react-app
create

# Wait for him to install it for a while cd packages && create-react-app react-app # Startup project yarn workspace react-app start Copy code

Install the mod1 module just created for the react-app project module

yarn workspace react-app add mod1@0.0.0 duplicated code

Try to modify the code,

packages/react-app/index.js

Import MOD1 from 'MOD1' mod1() Copy code

The work is done, we created the first usable module

Next create a react component module

u-button

lerna create u-button -y duplicated code

This time we export a button component,

packages/u-button/lub/u-button.jsx
(Remember to modify the main entry field of the u-button module package.json)

Import React from 'REACT' const the Button = () => { const text = 'Hello' return ( < Button > {text} </Button > ) } Export default the Button Copy the code

Do the same, add u-button dependency to react-app,

yarn workspace react-app add u-button@0.0.0 duplicated code

Then introduce,

Import UButton from 'U-Button' const Node = < UButton/> copy the code

Not surprisingly, you will see the following error

File was processed with these loaders: * ../../node_modules/@pmmmwh/react-refresh-webpack-plugin/loader/index.js You may need an additional loader to handle the result of these loaders. | const Button = ()=>{ | const text ='hello' > return (<button>{text}</button>) |} | Copy code

This is because

create-react-app
The default babel configuration excludes node_modules, we need to let go of its restrictions

yarn workspace react-app add customize- cra react-app-rewired -D duplicated code

on

customize-cra
with
react-app-rewired
The specific use is not expanded here in detail, the general steps

  • in
    react-app
    Created in the project directory
    config-overrides.js
const {babelInclude} = require ( 'customize-cra' ) const path = require ( 'path' ) module .exports = ( config, env ) => { //Each workspace directly outputs the original code, so you need to add babel babelInclude([ path.resolve( '../../packages' ), ])(config) return config } Copy code
  • use
    react-app-rewired
    replace
    react-app
    The relevant instructions under the package.json
"scripts" : { "start" : "react-app-rewired start" , "build" : "react-app-rewired build" , "test" : "react-scripts test" , "eject" : "react-scripts eject" } Copy code

Then restart the project

yarn workspace react-app start duplicated code

modify

u-button
The content in the module can also experience the hot update development experience, perfect

Let's create a new react project and name it

react-app2
,

cd packages && create-react-app react-app2 # Re-process dependencies and upgrade third-party dependencies to root node_modules yarn Copy code

Both react-app and react-app2 can use the public modules mod1, u-button, and even

customize-cra
Covered development environment

yarn workspace react-app2 add mod1@0.0.0 duplicated code

Next, write the code happily~

The problem

Monorepo solves the problems of multi-module development and debugging, but putting all projects under the same warehouse will also bring new problems.

Question 1: The number of files will increase rapidly according to business iterations

New colleagues may have to wait a long time when initializing the pull project~

Question 2: The projects are all placed in the same warehouse for version maintenance, when multiple people are developing collaboratively

  • It may pollute the commit history of a single project. Various features, fixes and even reverts flood the entire history.
  • Project permissions are not easy to control. It may happen that colleagues modify public modules and cause all projects to collapse

Question 3: Release and deployment of business modules

Release and deployment based on a single project is relatively simple, only need to install dependencies, package, and release in CI; for monorepo, different services need to be distinguished and deployed, and the partial modules involved depend on installation and increase release. Etc. need to be reconsidered

Micro front end

Now that monorepo is mentioned, I feel that we can mention the micro front end again.

When a single technology stack project becomes larger and larger, it is difficult to adjust the direction, such as upgrading the basic library, changing a framework, changing a language, etc. It is not easy to write, to test, and to deploy. Good deployment.

Faced with this problem, the back end has been micro-services practice and the like, as long as the project is small enough, it is easy to upgrade or replace; front-end project is the same reason, for example, a variety of promotional pop, this It should not be maintained by the code of the business project, but should be handled by a separate activity module.

The core idea of the micro front end is to deploy projects by modules, mainly to achieve code isolation and team isolation for each module. Each module can use an independent technology stack or be maintained by a different team.

It can be seen that multiple business modules are essential to the micro front end, so how to manage these business modules? The monorepo mentioned above is useful.

However, in the actual project development, the author has not encountered a scenario that requires the use of a micro front end. If there are similar business scenarios in the future, you can add more.

summary

This article first discusses some of the problems encountered in the development, debugging and update based on the independent npm package, and then studies the solution of monorepo to maintain multiple modules in the same project

  • Creation and release based on lerna management module
  • Manage dependencies between multiple modules based on yarn workspaces

Finally, whether it is multirepo or monorepo, in the final analysis, it is the management of modules. Compared with this, the more important thing is how to write reliable and easy-to-extensible modules. Code is the soul of the entire project.

Since the author is just beginning to use this kind of scheme in the project, if there is any misunderstanding in the text, please correct me and discuss it.