A side quest is usually a smaller but still important quest that the protagonist needs to accomplish before they can face their greatest challenge. A great example of this is Luke Skywalker going to Dagobah to seek out Yoda. Luke had no idea why he was going, or what he would find, but he was driven to go. While the training Luke received was crucial to his character progression, it wasn’t the main goal of the series, to stop the Empire. Another side quest is rescuing Han from Jabba’s Palace. Without Han they may have never destroyed the shield generator, and the Empire would still dominate the galaxy. Those side quests are essential to achieving the overall goal of the series.
Side quests are essential to storytelling, even if they are very small. But sometimes a side quest is so important, or so interesting, that it demands its own storyline. Lately this has become much more acceptable and even welcomed. Origin stories for favorite characters, spinoffs for beloved side characters, just consider a Yoda origin story. Sometimes a side quest happens outside the storytelling. Past events that lead up to the action the reader is suddenly thrown into. What if we never knew what Gandolf did when he left the dwarven party? Sometimes these need to be told, and sometimes the sudden appearance of the result is best, like the sudden appearance of the dwarven kin to aid in the fight against the dark horde. Could the gathering of that reinforcement army be its own book? Absolutely. Would it have made the Hobbit better? Absolutely not. That sudden appearance brought sudden hope, which was essential in the emotional roller coaster of that part of the book.
While doing my experiment of short stories for character backgrounds, I realized that their pasts were so interesting they were begging for their own books. At first I dismissed the idea thinking that my imagination was just wanting to play and I shouldn’t allow it to run away with itself. But the more i thought about it, the more I saw these character driven quests from vastly different perspectives which lead into the books I was wanting to write. Basically my imagination made its argument and it found validation.
These stories erupted from a natural fleshing out of the idea I find so engaging for the story. Why did this happen, how did these characters get here and why, etc. What I discovered when I began outlining these stories, is that filling in these gaps in the story has explained and filled in parts of the original. Also, once I allowed myself to let go of a specific structure or trilogy, quartet, series, etc. and just tell the stories, I found the ideas started to flow once again as if I had burst a dam. I was stifling my creativity in attempting to contain it in some preconceived structure to which it didn’t belong.
So in closing, find that balance between a flowing imagination, and a structured story. Don’t assume a story has to be any certain way, let it find itself. This can be done in the outlining, world building, character development, or writing itself for those pure pantsers out there.