Posts Tagged ‘research’

Data analytics vs. intuition

Monday, June 20th, 2011

MIT Sloan Management Review, in partnership with the IBM Institute for Business Value, conducted in 2010 a survey of nearly 3,000 executives, managers and analysts working across more than 30 industries and 100 countries, aiming to help organizations understand the opportunity of information and advanced analytics.

The report “Analytics: The New Path to Value” presents the main findings of the research:

  • Top performing organizations say analytics is a differentiator, putting analytics to use in the widest possible range of decisions, large and small. They were twice as likely to use analytics to guide future strategies, and twice as likely to use insights to guide day-to-day operations;

Analytics: The New Path to Value

Source: MIT Sloan Management Review & IBM Institute for Business Value (2010)

  • Organizations that know where they are in terms of analytics adoption are better prepared to turn challenges into opportunities; (more…)

Tags: , , , ,
Posted in Data visualization, Performance Measurement, Report | No Comments »

Performance Measurement Maturity Model – assessing organizational performance measurement capabilities

Monday, November 15th, 2010

Performance Management and Measurement

In scientific management performance is associated with two key processes, performance management and performance measurement which cannot be separated from one another. Performance management both proceeds and follows performance measurement.

Performance measurement appears as a sub process of performance management that mainly focuses “on the identification, tracking and communication of performance results, by the use of performance indicators” (Brudan, 2010).

Performance Measurement roles and importance

Increasingly, authors and commentators are discussing the multiple roles of measurement, as it is recognized that performance measurement allows managers to do far more than simply check progress (Neely, 2002).

The performance measurement system is a process supporting continuous learning in which feedback is used for identifying achievements and making adjustments to agreed-upon strategies or initiatives to ensure continued excellence of activities and services, and to progress for the attainment of organizations mission, vision and objectives. It also can provide a balanced and systematic attempt to assess the effectiveness of organizations operations from different points of view: financial, business performance, clients and employees.

In this setting performance measurement is a must and it is imperiously required to support the performance management system. One way to assure that a streamlined and mature performance measurement process is in place, is to assess the current measurement capabilities against a Performance Measurement Maturity Model.

About Maturity Models

In a previous blog, describing the Performance Management Maturity Model, we’ve presented the main reasons that stay behind why organizations might choose to use a maturity model in order to assess their current capabilities, such as:

  • Gaining a better understanding of strengths and weaknesses in order to enable improvement to happen
  • Gaining recognition of service quality in order to support proposals
  • Justifying investments in process improvement
  • Providing a map for continual progression and improvement
  • Focusing on the organization’s maturity rather than specific initiatives

Thus, maturity models prove to be important tools to help organizations align their processes to a common standard, and guide their process maturity, step by step following a structured and predefined path.

Performance Measurement Maturity Model

The Performance Measurement Maturity Model proposed below is informed by an academic research project (Brudan, 2009) supported by further insights from the experience gained as implementers of Performance Management Systems and developers of smartKPIs.com, one of the most comprehensive repositories in the world with Key Performance Indicators (KPIs).

The model is built on five key dimensions each describing an important characteristic or step of a performance measurement process.

Source: eab group, 2010

  • KPIs Identification & Selection – The selection of right Key Performance Indicators (KPIs) has a major impact on the organizational strategic directions. Indicators should focus on actions and services provided at each organizational level in order to achieve the organizational strategic objectives. Most important it must be considered at all times the measurement of what is important and not necessarily easy.
  • KPIs Documentation and Alignment – The process of KPIs documentation secures that the selected indicators are actionable, by establishing definitions, purpose, calculation formulas, targets, methods of data collection and reporting and data owners and custodians.
  • KPIs Collection and Interpretation – The process of KPIs collection and interpretation needs to be supported by a well defined and optimized system solution that responds to the performance measurement collection needs of the organization. Responsible for data gathering must be identified and it must be assured data availability for each KPI tracked in the Performance Scorecards and Dashboards.
  • KPIs Reporting & Visualization – Regardless of the parties involved in the performance reporting process it must be assured that the process is effective and efficient. An inefficient and ineffective reporting system will not generate any significant positive effect on improving the performance of an organization. Another important part of the measurement process is data visualization. Data visualization involves processing information in a graphic description of it in order to be understood and transmitted more easily, faster and more efficiently.

  • KPIs Feedback and Re-alignment – This process is one of the most important parts of the performance measurement cycle. It secures that the KPIs are kept on track and aligned with the organizational strategy, in order to provide with valuable information that will allow improved organizational practices.

The Performance Measurement Maturity Model uses a five level maturity framework which is adapted from The Capability Maturity Model Integration (SEI, 2001) and Portfolio, Programme and Project Management Maturity Model (OGC, 2008). Using this framework, organizations can assess the maturity level of their performance management practices in each of the five dimensions of the model against the 5 maturity levels.

Source: eab group, 2010

The model presented above represents a new tool that organizations can employ for assessing the maturity of  their performance measurement capability. For more details about how such a model can generate value in an organizational context, contact eab group.

References

  • Brudan, A.N. (2010), Rediscovering performance management: systems, learning and integration, Measuring Business Excellence, Vol. 14, Iss. 1, pp. 109-123
  • Brudan, A. (2009), Performance Management Maturity Level in  Business Organizations, Master thesis, Arhus School of Business, Denmark
  • Neely, A. (2000), Business Performance Measurement: Theory and Practice, Cambridge University Press, UK
  • Office Of Government Commerce – OGC, (2008), Portfolio, Programme and Project Management Maturity Model (P3M3), Pubic Consultation Draft, available at www.p3m3-officialsite.com (accessed  15 November, 2010)
  • Software Engineering Institute – SEI (2001), Capability Maturity Model Integration  (CMMI), Carnegie Mellon University, available at www.sei.cmu.edu/cmmi (accessed 15 November, 2010)

Additional resources

Tags: , , , , , , ,
Posted in Performance Management, Performance Measurement | No Comments »

Individual performance management and the use of metrics in the world of scientific research

Friday, August 13th, 2010

Image by David Parkins, @ 2010 Macmillan Publishers Limited

In a recent blog post, ‘Metrics in science – Performance Measurement and the world of scientific research’, we have outlined the most important performance measures for the scientific field as presented by the researcher Richard van Noorden (2010) in the weekly international scientific journal ‘Nature’. The post concluded that the provision of metrics that invaded the field of science in the last decades needs closer review before being used, as oftentimes simple measures can better reflect scientific performance aspects than more sophisticated and complicated measures. Acknowledgment for the need of reflection and consolidation in the field of scientific performance measurement was also emphasized. Continuing this line of inquiry, a new question is explored in more detail below:

How does performance measurement influence the careers of researchers in regards to matters such as hire, promotion or tenure?

The premise that “no scientific career can be summarized by a number” is the starting point of the research study initiated by the Nature magazine. It argues that there are other things that can recommend a good scientist,  which need to be taken in consideration when making an individual performance assessment in addition to:

• # Published papers

• # Impact factor of the journals published in

• # Citation frequency of the published papers

• $ Amount of grant money earned, or

• # H-index

However, from the 150 researchers that responded to the research poll, a vast number considered that the metrics of scientific performance represent a major factor in hiring decisions, tenure decisions, promotions or performance reviews / appraisal.

Source: Nature, Do metrics matter?, @ 2010 Macmillan Publishers Limited

The study also revealed that the most important criteria for scientist evaluation, in the poll respondent’s view, were grants and income, number of publication in high impact journals and citations of published research. These findings are doubled up by a high proportion of respondents that were unsatisfied by the way some of the measures are used in performance assessments, considering that too much credit is given to objective measurement and less to qualitative reviews, such as letters of recommendation from people in the field, or reviews of the work by peers outside the department or institution.

Surprisingly however, when asked to rate the most important criteria that should be used to evaluate researchers, the poll revealed that number of publication in high impact journals, grants earned, training and mentoring students and the number of citations on published research stood out in the respondents choice.

On the other side of the barricades, the administrators and evaluators, when interviewed in regards with the same matters insisted that metrics don’t matter nearly as much for hiring, promotion and tenure as the poll respondents seemed to think.

Despite these mixed opinions, the bottom line, as revealed by the Nature study remains that:

• 51% of respondents said that they have changed their behavior because of the way they are evaluated;

• 71% of respondents said that they are concerned their colleagues can ‘game’ or ‘cheat’ the systems for evaluation in their institutions.

Summing all these findings, one conclusion that can be reached at is that the problem doesn’t necessarily sit with the use of metrics in researcher’s performance evaluation process but more with the way these measures are used. Thus the challenge, as it is acknowledged also by the study authors is not to reduce the reliance on metrics, but to give more clarity, consistency and transparency to the performance appraisal process.

Integrating the individual performance evaluation in a more comprehensive context of a performance management system and aligning the objectives from the individual level with those from the operational and strategic level can be one of the solutions to strengthen and give more clarity, consistency and direction to the individual performance appraisal process viewed from an integrated, organizational performance perspective.

Question such as: Do metrics matter? What can be done to streamline the performance monitoring  and appraisal process at the individual level? continue to generate interest in both researcher and practitioner communities.

References

Additional resources

Tags: , , , , , , , ,
Posted in KPI, Library of KPI examples, Measure, Performance Measurement, smartKPIs.com | No Comments »

Metrics in science – Performance Measurement and the world of scientific research

Wednesday, August 4th, 2010

In a recent article published in the weekly international scientific journal ‘Nature‘, under the section ‘Science metrics’ the use of performance measurement in science is thoroughly investigated. According with the article, suggestive entitled ‘A profusion of metrics‘, performance measurement in science has captured an increased role since it was first started to be used in this particular area of human activity and interest (Richard van Noorden, 2010).

As van Noorden (2010) acknowledges, it was American psychologist James McKeen Cattle who first popularized the idea of ranking scientists according with their performance, 100 years ago. In one of his works published in 1910 ‘American Men of science: A Biographical Directory‘, the scholar acknowledged the importance of measuring the science researchers performance for promoting and encouraging the development of the field in a systematic and scientific way.

Image by David Parkins, @ 2010 Nature Publishing Group

The first performance measurement attempts were based on a simple survey in which the experts were asked to rank the best performers in a specific scientific field by merits. In comparison, today a different situation revolves.

It seems that in the last few decades, the field of science witnessed an increased attraction, in terms of the use of performance measurement. A new set of objective metrics and indicators was developed that help quantify and capture almost all the aspects in the scientific field from a diversity of perspective: quality, impact or prestige. Even more, the technological advancements and the development of the online databases such as Web of Science from Thomson Reuters, Scopus from Elsevier or Google Scholar further fueled the development of ever more sophisticated measures (van Noorden, 2010). Some of the most important and used measures are shortly presented below:

• # Citations: Measures the number of times a researcher or research paper is cited by other authors

o # Citations in top journals

o # Citations per publication

o # Citations by scientific field

• # H-index: Measures the number of publications authored that are cited by at least a specific number of times

• # Impact factor: Measures the average frequency with which an article published in a journal gets cited

• # Research paper online accesses: Measures the total number of times a research paper is accessed online

Van Noorden (2010) doesn’t stop at simply describing the metrics that have impacted the most the measurement of the scientific field, but he further argues that there is a need for a thoroughly investigation on how this performance measures can be best used for monitoring different aspects of the scientific field and scientists activity.

While some of the measures, like the ‘Impact factor’ are best fitted for measuring the popularity of a journal, other measures like the ‘H-index’ best captures the individual performance of the scientists. Same happens in all scientific fields. One has to acknowledge that not all metrics best fit to capture the performance in all scientific fields and that a thoroughly selection and analysis must be made before proceeding with measuring certain aspects. More than that, the amalgam and mixture of old and new metrics, of simple or more sophisticated ones, need a thoughtful selection of the ones that can best capture and reflect the efficiency and effectiveness of scientists or scientific field.

Most of the times, simple measures can better reflect specific performance aspects than more sophisticated and complicated measures. Also when having a large number of metrics at your disposal it can become easy to monitor same thing with more measures. As van Noorden (2010) acknowledges, in the scientific field today there are ‘many metrics that correlate strongly with one another, capturing much of the same information about the data they describe’. Accordingly, more and more researcher voices are asking for reflection and consolidation in the field in regards with performance measurement aspects.

To explore further examples of metrics that capture research & development aspects, innovation or knowledge management visit Key Performance Indicators for Knowledge and Innovation Functional Area, available at smartKPIs.com.

References:

Tags: , , , , , , , , ,
Posted in KPI, Library of KPI examples, Measure, Performance Measurement, smartKPIs.com | No Comments »

R&D Investment and Performance around the world – a European Commision report

Sunday, July 11th, 2010

The Science, Technology and Competitiveness key figures report 2008/2009, released by the European Commission, presents a thorough analysis and overview of the evolution of  R&D investment and performance in Europe. The report is considered an attempt to monitor the progress and efficiency of implementing performance measures by the European research system.

Research is considered a key competitive asset in a global world as science, technology and patent applications are more widely distributed every year. Between 2000 and 2006 the investment in R&D increased in Europe by 14,8%, comparable with the US and Japan, where figures show a growth of 10.1 % and 21.9 % respectively.

Source: European Commission, 2008

The analysis of the participation shares in global R&D indicates that almost 80 % of researchers work outside the EU, 75 % of gross domestic expenditure on R&D (GERD) is executed in other world regions, and 69 % of patent applications are made outside the EU. This indicates a declining world share of GERD and patent applications, both for the US and for the European Union (European Commission, 2008).

Source: European Commission, 2008

The main conclusions of the report are:

  • Despite encouraging progress on increasing the amount of investment in R&D, the R&D intensity of EU-27 has remained unchanged, as countries with increasing R&D intensities do not have very high shares of EU-27 GDP.
  • Higher returns for private investment in R&D favor structural change, such as high-growth SMEs and higher demand and a single market for research-intensive products (European Commission, 2008).
  • European Research Area (ERA) integration is a key competitive factor for increasing the effectiveness of the European research system, due to its cost-effectiveness and framework conditions attractiveness.

The Key Performance Indicators used within this report are:

  • % R&D intensity (GERD as a % of GDP)
  • % Gross Domestic Expenditure on R&D (GERD – % Real Growth)
  • # Patents applications
  • # Researchers

For further examples of performance indicators, explore the R&D KPI examples section of the library of KPI examples available on smartkpis.com (smartKPIs.com, 2010).

References:

European Commission (2008), The Science, Technology and Competitiveness key figures report 2008/2009, available at http://ec.europa.eu/research/era/pdf/key-figures-report2008-2009_en.pdf (accessed 1 July 2010).

smartKPIs.com (2010), R&D KPI examples, available at http://www.smartkpis.com/kpi/functional-areas/knowledge-and-innovation/r-d/ (accessed 1 July 2010).

Tags: , , , , ,
Posted in Government, KPI, Measure, Performance Measurement | No Comments »

“The truth about Performance Management” … as revealed by a SAS survey report

Friday, June 11th, 2010

According with a 2007 SAS report on performance management issues, alignment is the most important benefit of performance management efforts. The report was based on survey data gathered online from 1143 respondents from cross-industry organizations across the globe (SAS, 2007). The report presents a detailed picture over the use of performance management tools, frameworks, systems and practices in the worldwide organizations.

Among the most important findings that were outlined from the survey (SAS, 2007) are:

• Performance Management practices have spread over most of the organizational functional areas. According the SAS survey findings, the operations function is most likely to drive the effort followed closely by the finance and human resources departments.

• Even though most of the performance management practices are multi-departmental, only a third of them are aligned across all departments.

• Most companies are looking for performance management initiatives that could boost their competitiveness.

• Cultural resistance, the human factor, is the primary factor to achieving performance management success.

• Organizations have problems in integrating multiple systems employed to solve problems across their structure.

• Companies who are following a sequential approach to integrating performance management practices across all organizational levels are the most likely to succeed.

• Technology plays an important role in the performance management success.

• The more a company is mature in the use of information, the more successful will be in nurturing successful performance management practices.

Additionally, a table with the most significant performance management findings from the survey can be visualized below:

Source: The truth about Performance Management, A report of survey findings, SAS 2007

Although the SAS report presents a comprehensive picture of the use of performance management practices across organizations, other surveys available for the larger public might present different results. Each study uses different research tools and attributes different rigor in data collection and interpretation. A compendium of such statistics on performance management practices can be found in the smartKPIs.com Performance Architect update 20/2020.

References:

Additional resources:

Tags: , , ,
Posted in Performance Management, Performance Measurement | No Comments »

Assessing organizational performance management capability – The Performance Management Maturity Model

Tuesday, April 20th, 2010

A model is a simplified representation of the world. According with The American Heritage Dictionary (2001), a model can have multiple significations, one of the most used  being “a schematic description of a system, theory, or phenomenon, that accounts for its known or inferred properties and may be used for further study of its characteristics”.

In line with this definition, a maturity model is a process that describes the development of an entity over time and has the following characteristics:

• The development process is described by using a limited number of levels

• Each level is characterized by specific requirements, which must be achieved in order to pass to the next level

• Levels are sequentially ordered, from an initial level up to an ending level, and

• The development process requires a progress from one level to the next one, no levels being allowed to be left out (Klimko, 2001)

One of the most popular examples of a maturity model is the Maslow Pyramid. Maslow (1943) suggested that there is a hierarchy of human needs starting from physiological needs up to the self actualization needs, and that during the progress through the hierarchy, ideally all the levels must be achieved by individuals. However, similar adapted patterns can be applied in order to assess the development stages of almost every entity, no matter of its nature or form. Such an entity could be today’s modern organization.

Over the last decades, management models designed to assess performance and identify different opportunities for improvement have began to be identified by the organizations as important strategic tools for building capabilities and achieve competitive advantages. In line with these developments, the specialists from the Office of Government Commerce (OGC), UK (2008) consider that maturity models, in particular, have become essential tools in assessing organizations’ current capabilities which help in their processes to implement change and improvements in a structured way. According to OGC, there are a number of reasons why organization might choose to use a maturity model in order to assess their current capabilities, such as:

Source: Adapted  from P3M3 (2008), draft paper

The model proposed below is a Maturity Model for Performance Management and is the result of an academic research project using three data sources:

  • review of specialized literature in performance management and measurement,
  • review of highly acknowledged maturity models, and
  • insights from performance management practitioners.

The proposed model is built on seven dimensions each describing an important characteristic of a Performance Management System:

Source: Adrian Brudan, 2009

The Performance Management Maturity Model uses a five level maturity framework which is adapted from The Capability Maturity Model Integration (CMMI, 2001) and Portfolio, Programme and Project Management Maturity Model (P3M3 Models, 2008). Thus, organizations can assess the maturity level of their performance management practices in each of the seven dimensions of the model against the 5 maturity levels  identified and described below.

Source: Adrian Brudan, 2009

Source: Adrian Brudan, 2009

The model presented above represents a new tool that organizations can employ for assessing the maturity of  their performance management capability. For more details about how such a model can generate value in an organizational context, contact eab group.

References:

  • Adrian Brudan (2009) - “Performance Management Maturity Level in  Business Organizations“, Master thesis, Arhus School of Business, Denmark
  • Klimko, G. (2001) – Working paper. Knowledge management and maturity models. Building common understanding. Second European Conference of Knowledge Management, pp 269-278
  • Maslow, A (1943) – A theory of human motivation. Psychological Review, Vol. 50, pp 370-396
  • P3M3 – Portfolio, Programme and Project Management Maturity Model (2008) – Pubic Consultation Draft, Office of Government Commerce, available at www.p3m3-officialsite.com (accessed 16 April, 2010)
  • SEI -Capability Maturity Model Integration -CMMI- (2001) – Carnegie Mellon University, Software Engineering Institute, available at www.sei.cmu.edu/cmmi (accessed 16 April, 2010)
  • The American Heritage Dictionary (2001), 4th Edition, American Heritage Publishing Company

Tags: , , , , ,
Posted in Performance Management, Performance Measurement | No Comments »

Featured products

[no image]
$99 USD
 
[no image]
$99 USD
 
[no image]
$35 USD
 
[no image]
$45 USD
 
[no image]
$289 USD
 
[no image]
$99 USD
 
[no image]
$183 USD
 
[no image]
$1800 USD