The color of diamonds depends on physical factors. The different wavelengths they absorb is due to their atomic composition.
Here is a quick reminder of how colours appear:
Light, wavelengths and colour, and everything we perceive through our eyes, are intricately related. There are many different wavelengths around, ranging from 400 nanometers to 800 nanometers. And each wavelength gives us a perception of a different colour.
Of all these different wavelengths/colours, there are three primary colours and each of these primary colours has one secondary colour:
Types of Color
Primary and Secondary colours cancel each other out: a blue wavelength accompanied by a yellow wavelength of the same intensity will give us the perception of seeing white.
Combinations of Primary and Secondary Colors
However, if the blue wavelength is unaccompanied by its yellow counterpart, then we will have the perception of seeing blue.
What would make the colour "Yellow" disappear? Absorption by objects: when light hits an object, the object will, depending on its composition, absorb some of the wavelengths and in varying degrees. If all the light that hits the object manages to escape, then we see white; and if none of the light manages to get out, then we see black.
So basically what we perceive as the colour of an object is actually the part of the light that escaped from the object: by playing around with all the wavelengths and all the different degrees of absorption for each of them, we see the infinite variety of colours around us.