The Science Of Perception

I was born and raised in the Audio-Visual and Show business, given my father’s line of work. From loudspeakers, crossovers, amplifiers, mixers, PAs, equalizers, compressors, microphones, instruments, to music bands, stages, discotheques, studios, auditoriums, theaters, sound checks, cameras, lighting, edition... all were part of my daily childhood. All of it!

In the process, I learned and became a fan of what I like to call “the Science of Perception”, a mix Physics, Engineering, Anatomy, Neurology, Chemistry and Psychology.

Basically, the way humans perceive the world is based on sensorial systems: vision, hearing, touch, taste, smell and balance.

These are all processed by a massive computer in our brain called sensorial nervous system that combines all the sensorial Information and generates an overall perception.

So perception is just an interpretation of data, which means it can be manipulated, generating virtual experiences.

Think on a roller coaster Virtual Reality Simulator. If you visualize the movement, hear the sound of wheels and feel the ups and downs of the car, your brain will be tricked and you will feel afraid.

The same principles are also used in cinema experiences. Dolby Sound, Dolby Atmos, Imax, MX4D, etc. all these systems manipulate and estimulate different body sensors to enable a desired perception to the audience.

So let me explain you how each sensorial system works.

VISION

With Vision we process light intensity, color, dimension, depth and movement.

Our vision processes up to 90 frames per second (Hz). Beyond 90Hz, a group of individual sequencial images is perceived as motion. This is the reason why some LCD TVs have good resolution quality, but for some reason you feel some jerky motion. Check the frequency specifications. It needs to be 100Hz to be a good TV.

Our vision needs at least 13 milliseconds to perceive the presence of something and each frame has an average time exposure of 100 to 400 ms. So, for example, if you have two sequencial flash lights of 10ms each, the first yellow and the second blue, you will see the mix of both and perceive it as one single flash of green color.

Color vision is the ability to distinguish different objects based on the wavelengths of the light they reflect, emit or transmit. 

Each color relates with one specific wavelength or frequency. A person's perception of color is a subjective process whereby the brain responds to the stimuli that are produced when incoming light reacts with the several types of cone cells in the eye. Different people see the same illuminated object or light source in different ways.

Black is the most interesting color. Black is the absence of light. The black level help us perceive depth. The darkest, the deepest.

True black with reflectance equal to zero is impossible to achieve, except in astronomic black holes. When we create black paint or black objects, light is in fact still reflected, either in all directions in matte or smoothly in shiny black objects, making it never a true black.

Do this test: at night turn off all the lights at home and put a black image in your smartphone screen. You will find that you can illuminate the room with that black image. Now turn off the screen. This is the pure black - the absence of light.

Black level is extremely important among screen technology. The black level of a TV helps to determine the level of picture quality. Generally, the darker the black level, the better the quality of the visuals. Different TV technologies such as cathode ray tube (CRT), liquid crystal display (LCD), light emitting diode (LED) and plasma manage their black levels differently, but advanced technologies such as LED have better black levels and are able to show visuals much better.

Moreover, TVs that produce three-dimensional images are dependent on good black levels to produce depth. Plasma TVs tend to produce better quality black levels because the phosphors use less power to produce the darker scenes on the screen. LCD and LED TVs utilize more power to produce the required levels of black, because in order to produce true black colors they must adjust the individual crystals in the pixels to block light that would otherwise come through.

Depth is the most curious visual perception. We can only perceive depth because we have two eyes with space between them. It’s pure mathematics. When our nervous system combines two images from each eye perspective and makes triangulation calculations, based in the space between our eyes, it perceives depth.

Try this test. Close one of your eyes now. Try to put your index finger aligned with your screen, but on the side. When you believe the finger point is aligned, move it to the screen surface and confirm if it’s a perfect match. Uhm... Not a perfect match? Don’t worry, it’s the beauty of lack of depth perception.

BALANCE

With Balance we process acceleration. It relates with Newton’s first law:  “every object will remain at rest or in uniform motion in a straight line unless compelled to change its state by the action of an external force”.

So Balance help us understand any change in direction, speed, external force or mass.

How can we manipulate Balance perception? Easy. Sit on the floor. Close your eyes and now bend quickly your back 45deg to the rear side. No hands on the floor! Feel it? It’s the same feeling you have when you are falling down in a roller coaster. By changing your gravity center, you introduce an acceleration that tells your brain you change direction and speed.

HEARING

Hearing is my preferred sensorial system, for obvious reasons. With hearing, we process sound level, balance, frequency, echo, reverb, delay and, by using two independent hearing systems with special devices called auricles, we process movement and relative spacial position in three dimensions.

Our ears are like microphones. They are sensitive to sounds from 20Hz to 20kHz. For non-geek people, this means from a very low base (subwoofer) sound to a very high treble sound.

Below 20Hz and above 20kHz we, humans, don’t hear. These are called infrasounds and ultrasounds, respectively.

Infrasounds are commonly used in seismology and geophysical surveys, since they relate with the frequency of earthworks. Infrasounds travel very well in high density materials, like rocks. Some animals are capable of detecting ultrasounds and can identify in advance for example earthquakes before humans can feel it.

Ultrasounds on the other hand travel fast and far, therefore they are commonly used in radars, medical equipment or by night animals, like bats. All of them follow the same principle: you generate an ultrasound and use a kind of microphone to detect any ultrasound reflection on a body or object. The time that the ultrasound wave to travel from point of origin and come back defines the distance of the object, given that each sound frequency travels at a known speed.

Echo and delay help us perceive distance just like bats using ultrasounds. The longest it takes to hear back the echo, the furthest away the object is.

Reverb help us perceive first the type of space. An open space has no reverb. An enclosed space generates reverb, which is basically the sound being amplified by the multiple reflections in the surrounding surfaces. The bigger the enclosed space, more reverb will be sensed.

Also the materials and shapes/textures of the surfaces where the sounds reflect change the frequency and level of the sound reflection. A flat ceramic surface generates a strong clean reverb. A wood surface reduces reverb and reflects a warmer sound with less treble. Soft materials tend to absorb the sounds and generate almost no reverb. Texturized/shaped surfaces spread the sound reflections in multiple directions reducing reverb.

Knowing how to use materials and shapes to control and manipulate sound is a science by itself. Acoustic engineers are able to design an Auditorium, a Theatre or a Recording Studio to provide the best sound experience to its users.

Echo, reverb and delay can also be synthesized with sound devices. This is commonly used in the music industry by Sound Engineers to reproduce the perception of a specific room context.

Movement and relative spacial position are perceived by the difference in sound level, sound frequency and sound balance between both ears at each instant.

A low level and base sound relate with a sound source located at a larger distance. A high and treble sound relates with a shorter distance. You can clearly identify this while listening to a car at the street.

Relative left and right position of a sound source is perceived based on the sound level difference measured in each ear. A sound that is listened louder in the left ear than in the right ear is indicative that the sound source is located towards the left side of your face. The bigger the difference, the bigger is the distance towards the left perceived. This is called Stereo Sound - the most basic and important tool of a sound recording engineer.

So imagine you are a sound engineer and want to record a jazz band with voice, piano, base, drums and chorus. You record the musicians individually in a studio room, but you want your listeners to visualize the band playing in a small concert hall, spread in the stage, while siting in the centre front row seat. So, how do you do it?

Small concert halls are characterized by its short and smooth reverb and very short delay. So, add this effect to your overall sound.

Now you want to position the musicians on stage. Singer and drums at the centre, base on the left, piano and chorus on the right. So what do you do it? Leave the Singer’s sound balanced, exactly similar on both left and right sides. For the base, change the sound balance 30% towards the left. For the chorus, change the sound balance 15% towards the right. For the piano change the sound balance 30% towards the right. The drums are a bit more tricky. Normally they are recorded with multiple microphones. So each mic shall be treated as a separate instrument. Drum base remains neutral. For the snare and right plates change the sound balance 5% towards the right. Other drums and plates on the left side, change the sound balance 5% towards the left.

Et voilá! Welcome to your first single recorded in Stereo Sound.

TASTE AND SMELL

Taste and smell are two related senses. Both work by interpretating the presence of molecules either on our mouth or in the air, respectively.

Sweet, sour, salty, bitter, and umami are all senses of taste. Taste is an experience cobbled together as much from scent, suggestion and memory, as from the nerve endings in the mouth and tongue.

Most of what we perceive about food is actually smell and not taste. Molecules of the food you are eating go into the air in your mouth. They move through the back of the throat and up into the nose to reach olfactory nerve endings in the posterior part of the roof of the nose. The molecules bind to these nerve endings, which then signal the olfactory bulb to send smell messages to the brain.

Try to drink or eat with your nose blocked. It will not taste the same!

The brain and emotion play a big part in how we perceive taste and smell. Marketing and advertising people know this better than anyone else and use it to manipulate others. 

The most famous example is Coca-Cola: all the marketing and advertisement rotates around one subject - good feelings and good memories. The brand have worked out so hard and in such a consistent fashion over dozens of years, that they were able to manipulate how a significant number of generations perceive their product flavor. This has been proven multiple times when having a taste test with their rival Pepsi. In a blind test, almost everyone prefers Pepsi, but in a normal test, as soon as people visualize the brand logo, most people prefer Coca-Cola. The logo image triggers immediately in your brain all the good memories while drinking coke and... bang... they got you manipulated. It’s like being hipnotized.

Also, people can taste the same dish differently, depending on their genes.

Smell, in particular, has an enormously emotional effect. We all have a smell - or a series of smells - that, when you smell them, they put you back in some happy place that you used to be, usually as a child. And those emotions are so strong that, regardless of your humor status, it will still make you happy by default. In my case, for example, I have multiple triggering memory smells. The smell of ocean and Berlin balls cake, take me to Algarve beach in Portugal when I was 6 years old. The smell of vegetables soup and red wine takes me to my grandparents’s house in childhood. The smell of yellow glue takes me to my father’s garage where I used to help him manufacture loudspeakers. The smell of giz takes me to my mother’s primary school. The smell of chocolate milk takes me to my mother’s “arriving home” moment. The smell of “francesinhas” (famous Oporto dish) takes me to great dinner moments with best friends.

TOUCH

Our skin is equipped with different cutaneous sensory receptors: mechanoreceptors, nociceptors and thermoreceptors.

These receptors together with the network of nervous endings that communicate the information to the brains are called the somatosensory system.

With these receptors we collect a massive amount of data: pain, pressure, tension, temperature, texture, shape, weight, contours and vibrations.

This data is extremely important since it provides an understanding of the body surrounding environment and provides our brains critical information to identify any danger that can lead to injury, disease or death and helps our body react immediately to ensure survival.

Inside our body, in tendons, muscles and joint capsules, we have additional receptors called proprioceptors. These are very interesting sensors. They sense the position of the different parts of the body in relation to each other and the surround environment. Without these sensors we would not be able to do fundamentals activities such as feeding or clothing ourselves.

If you want to understand how would be your life without these sensors just remember the last time you woke up in the middle of the night with a numb arm! A numb arm is basically our sensors turned off due to lack of blood circulation. When you have a numb arm you have the feeling that the arm is not part of your body. You can move it, but feel nothing. Horrible feeling, right?

SIXTH SENSE

Since we are talking about sensorial systems, let me explain the most controversial one - the so called Sixth Sense.

This is an intangible sense. There is no actual sensor. It’s more like a Machine Learning kind of thing. Our brain collects and processes a massive amount of data along our life span. During this period our brain identifies patterns and creates primary sub-routines that help us predict outcomes on a subconscious level, when faced with specific scenarios.

It’s an amazing capability humans have. This sense helps us take immediate decisions based on previous experiences without a conscious understanding. 

I hope you learned something new today and I was able to convert you into another Science of Perception fan! ;)