Masking is using masking/cropping tools to cut out characters or objects to be moved around or placed in random locations. The masking tool is only avalible in pro versions of Sony Vegas.
Masking In Sony Vegas
You are looking at the Sony Vegas timeline. You right click on the video you put onto the track and you access your Pan/Crop tool. Check the "Mask" box under Position and use the "Anchor Creation Tool" to start cutting out the image/frame you wish to edit. Be cautious with your point creation.
Masking in Final Cut Pro X
Select a clip you want to apply effects to; Next, click on Effects (the picture with two squares) then click on Masks. The best way to freely mask is using Draw Mask on the selected clip. To do the masking frame-by-frame, split the clip into one frame length segments. Be careful if you decide to mask without splitting or else you get a messy mask.
Usage In Poop
Sometimes the masking tool is used to crop out a character and replace the backround. Sometimes the backround is left completely normal while the character or object has a bunch of effects unloaded on it. Other times the object is repeted across the screen to imitate a glitch.
Usage Outside YouTube Poop
Same as most YTPs, but without the chaoticness. This technique is common in some Adult Swim shows (Space Ghost Coast to Coast, Sealab 2021), Abridged series, notably Yu-Gi-Oh the Abridged Series, Kids WB adverts, etc.