1. What is the role of an evaluator?

The main role of an evaluator is to read the project proposal carefully and express opinion about its value. Briefly, the evaluator elaborates whether the project proposal should or should not receive funding.

Evaluator forms the opinion on the grounds of two data sets. The first data set is the proposal itself, which is read with no additional instructions, consultations, information or other explanations. The second data set is the evaluator’s expertise. This comprehends the state-of-the-art in the field in terms of the recent results and achievements, up to date trends and current research and commercial projects, plans and strategies for the future development, which are defined by EU and finally, knowledge about leading scientific-research institutions and companies. Therefore, it is of major importance that the evaluator is updated with all the information from the specific scientific field so that he or she could do the evaluation procedure properly.

2. Which skills evaluator needs to possess?

The evaluation procedure consists of two different parts. The first one is individual evaluation. This is the time when the evaluator reads the proposal and writes a formal report, elaborating the opinion about the proposal. The time is very limited, thus, excellent reading comprehension is required, based on which the evaluator needs to draw conclusions and judgements. Furthermore, the objectiveness and making conclusions with no personal emotions interfering in the process of evaluation are essential. Additionally, the evaluator needs to be able to express the opinion concisely, precisely and straight to the point in written form and orally alike. The approach towards each proposal needs to be positive and optimistic. This means that, when starting to read, evaluator needs to believe that the project is good and meaningful. Only while reading it, evaluator discovers its drawbacks, if any. Consequently, the evaluator, even though strict, has good intensions and marks the proposal unbiasedly.

The second part of the procedure is group evaluation, which comprises two or three stages. The first one is the consensus group, the second one panel meeting and finally third one, which is done only when needed, is called hearing, roughly explained. In all three stages a group of evaluators discuss the proposal with each other, forming the final status of the project (approved for financing or rejected). During this step of the evaluation it is of key importance that the evaluators explain their point of view briefly and to the point. Moreover, each evaluator ought to be confident and prepared to defend expert opinion about the project, but also to be patient to hear out the others and adapt when needed (most common mistakes are biased judgement of some parts of the proposal or simply omitting to read certain part of the text). To conclude, evaluator needs to be fluent in oral communication and good listener, open-minded individual, ready to debate and accept different opinions.    

3. How does someone become an evaluator?

Formally, it is rather easy to become an evaluator. The “only” thing to do is to apply to the European Commission list and wait to be called. In the application form apart from usual personal data, fields that refer to the qualifications need to be filled out. Additionally, the potential evaluator needs to check the field (or more fields) for which an expert wants to apply. Surely, this field depends on one’s personal expertise. The next step is to submit your CV and prove your personal qualifications, in terms of scientific results in the field. Not only the application in the EC database needs to be updated regularly, but also applicant must follow all the calls for proposals, respond to various questionnaires and attend occasional meetings where new fields and calls are discussed and presented. Practically, the constant communication with European Commission and capacity to respond timely is required.

4. Are there any and if yes what are the benefits of being an evaluator?

Benefits of working as an evaluator are numerous. First and the most straightforward benefit is a fee, since the evaluation is being paid. Surely, in the long run this benefit is the least important one.

For the people and institutions outside the EU, getting acquainted with the system in terms of project evaluation, forming the scores, making decisions about who will get the funding is of great relevance. Furthermore, being evaluator is a great opportunity to make new acquaintanceships that can bring future cooperation. Additionally, being informed about new ideas, possibilities and trends in the development of the specific field is highly beneficial. This is extremely important because the evaluators themselves are experts who also tend to participate in the projects from the field, thus, this creates great opportunities for them to hear about new ideas and find out useful information. During the process of evaluation they have the chance to get a first-hand experience about strengths and weaknesses when it comes to applying certain ideas or specific approach to problem solving. To simplify, all the relevant information is available to the evaluator which consequently brings better results and more efficient approach to the important issues from the specific scientific field.

Finally, by reading proposals, writing the reports and participating in the discussion with other evaluators, one gains the ability to differ good from badly written proposals. This is greatly helpful when writing your own proposal, since it happens often that a great idea with a significant scientific contribution gets rejected for funding due to poorly defined dissemination or tasks division between the consortium partners. Poorly written proposals never receive the funding. Also, it is significant to mention that all the segments of the project proposal are equally relevant. Project idea, implementation, task division among partners, timeframe and dynamics of the implementation, relevance for the science and the economy, dissemination, all this is important and brings points to the application, which implies that the evaluator as well, needs to have this in mind when doing the evaluation. With all that said, I have to emphasize that stealing ideas, industrial espionage and similar activities are utterly unacceptable and severely punitive.

5. How intensive is the evaluation process, meaning how many proposals you need to evaluate and what are the time constrains?

The process of evaluation is extremely intensive because in the very limited time the evaluator needs to complete a serious and extensive work. To be precise, it is expected to do one project proposal per day or 5-6 proposals per week. If we take into consideration that the average proposal has around 100 pages sometimes even double and that the time is limited, the expert who is doing the evaluation needs to form his or her opinion after the first read already. Apart from not having enough time for reading specific parts twice or more, evaluator has to skim through the proposal swiftly which demands very good concentration and focus for a longer period during the day. Nevertheless, the evaluators are people who are in a very good shape when it comes to writing and reading, but even so, the evaluation is very stressful due to the possible mistakes in the procedure. However, this is why the group evaluation exists; especially in consensus group the evaluator’s mistakes are easily noticed and corrected.

6. After being an evaluator do you find it easier to write a project proposal?

Of course, by reading, evaluating, discussing with other evaluators and finally going through the list of rejected and approved project proposals, the knowledge about how the proposals should be written in order to receive funding is being gained.

7. What is the crucial aspect which makes one project better than the other and if the two projects have equal score how is the ranking done?

The project evaluation consists of three parts. First one is scientific excellence or technological breakthrough that the project is supposed to make in the specific field. Second part of the evaluation is the quality and efficiency of the implementation (consortium members, task division, work packages and timeframe). The final, third part is impact or a potential influence of the results on the science and the economy, wider community, practical usage of the outcomes, dissemination and communication to diverse target groups. The most often it happens that the score in the first segment is the highest while in other parts, implementation and impact, score declines due to greater focus of the researchers on the scientific excellence of the application, which is also quite logical as this part is their real expertise. The part of the proposal referring to the consortium members, task division, work packages and job organization is usually done correctly, but not as carefully and attentively as the previous segment. The part of the proposal that elaborates the results implementation and dissemination usually gets the lowest marks, since most of the researchers tend to write very general sentences, as they believe that this part of the project is the least relevant one. What a fallacy!  A few projects were rejected precisely because of having poorly or not at all defined dissemination activities.

Based on the overall mark the ranking of the projects is done. When more projects receive the same amount of points a problem emerges. There is no specific procedure that is formally recommended in such situations as there is in football, for instance. In football if there are teams with same number of points, goal difference is the parameter that decides, if it looks the same, the total number of scored goals is what matters, and if it also looks the same in the Bundesliga for example the greater number of goals scored away will decide and so on. As previously mentioned, in the project evaluation procedure, all parts of the project are equally relevant; therefore, when this happens, the decision which project will have higher ranking is made at the Panel meeting. Here every project is discussed and eventually voting is what decides which projects will get the funding. This is why it is very important to write each part of the project with equal focus and devotion. Consequently, only those who remain the highest level of their writing skills through the entire proposal can expect to have a positive outcome of the evaluation procedure.

8. What are the most common errors that you find in project proposals?

Poor job division among partners in the consortium; overlapping or lack of commitment in certain tasks; generic composition of work packages and humble interaction among them; poorly written dissemination, with very few or no details, no operational plan or additional explanation.

9. Since you were the evaluator of FP7 projects, what in your opinion, were the drawbacks in terms of evaluation procedure and do you know if this has changed in H2020?

The entire job is very well defined and organized. One of the major issues was a very short time for individual evaluation. Additionally, certain steps in the procedure which ought to be followed were causing the unnecessary waste of time. For instance, certain calls depending on the size of the grant comprehend the step in the evaluation known as “hearing”. This is the meeting where the project team presents in more details their ideas while evaluators draw final conclusions and suggest final decision regarding the funding. On the one hand, all this makes sense if there are plenty of equally good project ideas and the selection has to be done, or if a good project’s budget is high above the grant value. On the other hand, if one proposal is positively evaluated and the budget is in order, then “hearing” is entirely pointless. I am mentioning this since this is exactly what had happened once. Ten of us from all parts of Europe had to travel to Brussels because of a half an hour meeting. We spent three working days there. On top of that, it was winter time, and the entire Europe was engulfed by snowstorm, thus the flights were cancelled and entire air traffic was stopped. I am still not familiar with the procedures and rules in H2020. I honestly do not know whether certain things have been changed or not.