This is just another myth!
With the exception of turkey vultures, who have a fabulous sense of smell, most other birds have a poorly developed sense of smell. Even if they are sitting nearby and see you look in the nest, this is still not usually enough to make them abandon it. And birds that are on the ground near a nest have probably fledged (left the nest on purpose.) Baby birds haven’t practiced flying before leaving the nest of course so must learn after they have fledged. The mama is likely close by watching and will continue to feed and care for the bird until it can be on its own. Picking it up and returning it to the nest will not cause the mama to abandon it, but usually the baby will just fledge again!
However, sometimes your scent will actually attract other animals, especially raccoons, which, if they can get to the nest, will surely feast on the chicks or the eggs. This is a good reason to not disturb birds when they are nesting. Installing a baffle (shown below) on the tree or pole will prevent some predators from being able to reach the box.