Traditional motor assessment is carried out by clinicians using standard clinical tests in order to have objectivity in the evaluation, but this manual procedure is liable to the observer subjectivity. In this article, an automatic assessment system based on the Box and Blocks Test (BBT) of manual dexterity is presented. Also, the automatic test administration and the motor performance of the user is addressed. Through cameras RGB-D the execution of the test and the patient’s movements are monitored. Based on colour segmentation, the cubes displaced by the user are detected and the traditional scoring is automatically calculated. Furthermore, a pilot trial in a hospital environment was conducted, to compare the automatic system and its effectiveness with respect to the traditional one. The results support the use of automatic assessment methods of motor functionality, which in combination with robotic rehabilitation systems, could address an autonomous and objective rehabilitation process.