Depending on the mods you have installed, it can indeed cause issues when trying to join the server. Fortunately you can test this easily.
Create a new folder at a location of your convenience.
Move your mods out of the Module folder to the new folder. Make sure to keep the default modules;
- Bannerlord Online
- BirthAndDeath
- CustomBattle
- Multiplayer
- Native
- SandBox
- SandBoxCore
- StoryMode
Verify your game files through Steam or Epic Games Store. (In case some mods overwritten default game files)
Launch Bannerlord Online.
Then whenever you want to play single player, you can just move the mods back to the Modules folder.