The risk screening of non-native species that are likely to be invasive in a defined risk assessment area is crucial for implementing strategies of rapid response and mitigation to protect native biodiversity and socio-economic activities. However, for successful risk-ranking of the screened species, scientifically defensible evidence in support of the screening outcomes must be provided, and computation of a correctly calibrated threshold to distinguish between medium-risk and high-risk species must be achieved. This paper reviews published applications of the “second-generation” Weed Risk Assessment-type decision support tools (i.e. the Aquatic Species Invasiveness Screening Kit and the Terrestrial Animal Species Invasiveness Screening Kit) and evaluates them in terms of the above two requirements. Several procedural errors were identified that involved: i) lack of provision of the report with details of the species-specific screenings; ii) incomplete justifications for the responses in the toolkit questionnaire; iii) incomplete details of the protocol used for the a priori categorisation of the screened species for threshold computation; iv) unaccepted or non-existent taxonomic names for the screened species (including typographical errors). Guidelines are provided for both assessors and reviewers to ensure that these procedural errors are avoided in future applications of these risk screening toolkits.