Objective assessment of motor function is an important component to evaluating the effectiveness of a rehabilitation process. Such assessments are carried out by clinicians using traditional tests and scales. The Box and Blocks Test (BBT) is one such scale, focusing on manual dexterity evaluation. The score is the maximum number of cubes that a person is able to displace during a time window. In a previous paper, an automated version of the Box and Blocks Test using a Microsoft Kinect sensor was presented, and referred to as the Automated Box and Blocks Test (ABBT). In this paper, the feasibility of ABBT as an automated tool for manual dexterity assessment is discussed. An algorithm, based on image segmentation in CIELab colour space and the Nearest Neighbour (NN) rule, was developed to improve the reliability of automatic cube counting. A pilot study was conducted to assess the hand motor function in people with Parkinson’s disease (PD). Three functional assessments were carried out. The success rate in automatic cube counting was studied by comparing the manual (BBT) and the automatic (ABBT) methods. The additional information provided by the ABBT was analysed to discuss its clinical significance. The results show a high correlation between manual (BBT) and automatic (ABBT) scoring. The lowest average success rate in cube counting for ABBT was 92%. Additionally, the ABBT acquires extra information from the cubes’ displacement, such as the average velocity and the time instants in which the cube was detected. The analysis of this information can be related to indicators of health status (coordination and dexterity). The results showed that the ABBT is a useful tool for automating the assessment of unilateral gross manual dexterity, and provides additional information about the user’s performance.