The lawless and chaotic dark era
Therefore, for the sake of your safety, it is best to continue residing in the castle honestly
If a noble's castle and territory are seized by a neighbor, it is best not to expect that merely going to the palace to engage in a court case will suffice to reclaim the territory... Unless you have a strong relationship with the king that can compel him to send troops to fight for you, it would be quite fortunate to merely receive a position at court that provides a meager livelihood. Therefore, in that era, every lord had to firmly guard their castle and could not afford to be away for extended periods.
Since living in the castle is so uncomfortable, why not simply not live in the castle? Instead, we could build a wooden vacation cottage in a nearby area with developed transportation and beautiful scenery. After all, as a noble lord, this amount of money is always manageable. If necessary, we could even conscript the serfs for free labor.
At that time, Europe was in utter chaos, with no unified nation-states. Whether it was England, France, Germany, or Italy, they were all characterized by a multitude of feudal lords. The authority of the monarchs in each country was largely confined to the capital and royal territories, and they were utterly powerless against the private conflicts of their noble vassals
Each lord is scheming to expand their territory, seeking excuses for external expansion and aggression. Often, it is merely due to a minor border dispute, a personal attack, an unintentional remark, or a failed marriage proposal. Sometimes, even a gesture, an expression, or a hand sign can ignite a medieval war. The knights of the Middle Ages were quite violent and brutal; one should not imagine them as polite gentlemen. Medieval poetry and literature record many of their shameless deeds. For instance, a certain noble knight once proposed to a wealthy widow, but upon being rejected, he brazenly led an attack on the castle owned by the widow until she was forced to open the gates and kneel in submission to marry him.
Therefore, in that terrifying era of chaos and rampant banditry, where one could not guarantee their safety from day to day, you chose to abandon the sturdy and reliable castle to pursue some semblance of a pleasurable life, moving instead to a picturesque villa that was ill-suited for defense...! ... It is likely that within a few days, a large group of assassins and bandits would come knocking at your door, either to kidnap you and your wife and children for ransom, or, having received commissions from other nearby lords, to take your life directly
Furthermore, even if we can establish a tight defense on land, there are still the elusive Viking pirates at sea
In addition to the ill-intentioned neighboring lords, the forests of medieval Europe were also home to many ruthless bandit groups that roamed like swarms of locusts, occasionally attacking and plundering various estates—Robin Hood of England is one such representative. Just look at how Robin Hood and his men can breach even small castles; how long do you think your exquisite vacation cottage could withstand such raiders?
Indeed, this practice had become quite rare among the nobility's private duels. During the relatively peaceful late Middle Ages, it should have been feasible. As mentioned earlier, many lords had already moved out of their castles, returning only when the threat of war loomed near to reside once again in their fortified strongholds
Moreover, the territories of medieval lords were generally small, lacking strategic maneuvering space. It often took less than half a day to ride from one castle to another, providing such a short warning time that there was simply no opportunity for evacuation or relocation
As for preemptively warning against the actions of these enemies, moving back to the castle before they arrive... In medieval Europe, the land was vast and sparsely populated, with wilderness accessible to small troops everywhere. How many scouts do you have under your command to secure the entire territory?
However, if your vision is too ahead of its time, and you were to act this way during the early and mid-Middle Ages... you would essentially be seeking your own demise