Content page
From [Institution name]
Evaluation of the Atlantic Canada Opportunities Agency’s Innovation Programming
2013-2014 to 2018-2019
Evaluation and Advisory Services
Atlantic Canada Opportunities Agency
January 2020
Table of contents
- Acronyms
- Figures
- Tables
- Acknowledgements
- Summary
- 1.0 Introduction
- 2.0 Innovation programming
- 3.0 Evaluation approach
- 4.0 Findings
- 5.0 Conclusions
- 6.0 Recommendations
- Appendix A: Innovation programming logic model
- Appendix B: Summary of findings, conclusions and recommendations
- Appendix C: Management action plan
- Appendix D: Innovation evaluation framework
- Appendix E: Enterprise Development evaluation terms of reference overview
- Appendix F: Evaluation advisory committee
Acronyms
ACOA Atlantic Canada Opportunities Agency
AIF Atlantic Innovation Fund
BAIs business accelerators and incubators
BDC Business Development Bank of Canada
BDP Business Development Program
DRF departmental results framework
EAC evaluation advisory committee
EAS Evaluation and Advisory Services
ED Enterprise Development
FAA Federal Accountability Act
FTE full-time equivalent
GAC Global Affairs Canada
GBA+ gender-based analysis plus
GC Government of Canada
G&Cs grants and contributions
IRCC Immigration, Refugees and Citizenship Canada
ISED Innovation, Science and Economic Development
MAP management action plan
NRC-IRAP National Research Council of Canada -– Industrial Research Assistance Program
O&M operations and maintenance
PAF project approval form
REGI Regional Economic Growth through Innovation
RDA regional development agency
R&D research and development
SMEs small and medium-sized enterprises
TBS Treasury Board Secretariat
Figures
- Figure 1: Client survey – Impacts of no ACOA support
- Figure 2: Client survey – Organizations involved in projects beyond ACOA
- Figure 3: Client survey – Satisfaction with ACOA service features
Tables
- Table 1: Overview of ACOA programming
- Table 2: Annual Innovation programming expenditures by fiscal year, 2013 to 2019
- Table 3: Implementation context
- Table 4: Evaluation questions
- Table 5: Summary of data collection methods
- Table 6: Evaluation limitations and mitigation strategies
- Table 7: Description of major funding for innovation
Acknowledgements
This evaluation provides the Atlantic Canada Opportunities Agency's (ACOA) management with objective, neutral evidence on the relevance and performance of the Agency's Innovation programming. ACOA's Evaluation and Advisory Services unit completed this evaluation of Innovation programming with advice from an evaluation advisory committee (EAC).
I offer deep gratitude to the members of the EAC for their advice and support throughout the study. Their contributions helped to ensure the relevance and usefulness of the evaluation. Very special thanks go to the external members of the EAC: Matt MacPhee, Senior Project Manager, Saint Mary's University Entrepreneurship Centre; Beth Mason, CEO, Verschuren Centre, Cape Breton University; and, Maurice Turcot, Senior Evaluation Analyst, Canada Economic Development for Quebec Regions. Internal members of the committee include Joe Cashin, Michael Dillon, Kalie Hatt-Kilburn, Josh Jenkins, Anne-Marie Léger, Jeff Mullen, Hélène Robichaud, Daniel Scholten and Karl Tee.
The evaluation team is grateful to the external key informants as well as the many ACOA employees from across Atlantic Canada who provided their time and knowledge in support of this evaluation.
Finally, to the evaluation team members – Deanna Slattery-Doiron, Anouk Utzschneider, Nicole Saulnier, Jasmine Arsenault and Gaétanne Kerry – your dedication, professionalism and commitment to excellence is exceptional. Thanks to each of you.
Colleen Goggin
Director, Evaluation and Advisory Services
Summary
The purpose of this evaluation is to assess the relevance and performance of ACOA's Innovation programming over a six-year period, from 2013-2014 to 2018-2019. The evaluation fulfills Government of Canada (GC) accountability requirements and provides the Agency's management with systematic, neutral evidence to support continuous program improvement. The methodology included interviews with 42 key informants, a client survey with an overall response rate of 48%, two case studies, a document and literature review, and an analysis of available performance data.
ACOA's investments through Innovation programming aim to create a robust regional innovation ecosystem and accelerate the growth of small and medium-sized enterprises (SMEs) through the delivery of both financial and non-financial supports. Projects and initiatives focus on the following objectives: capacity for research and development (R&D), commercialization, productivity and business skills, and the attraction and retention of talent; development and demonstration of new or improved technologies, products, processes or services; and collaboration on innovative technology and commercialization projects.
Over the evaluation period, ACOA supported more than 850 innovation projects across Atlantic Canada through grant and contribution (G&C) expenditures and non-financial supports. The Agency expended $508 million, with 91% of this spending through G&Cs and the remainder to support operational delivery costs.
To what extent does the programming address a unique and demonstrable need for Atlantic Canadians?
The evaluation finds that there is a strong and continued need for ACOA's Innovation programming. Challenges to business growth exist to the same degree or are greater than reported in previous evaluations. In particular, the evaluation reports the following: a shortage of skills and labour influenced by demographic shifts; a need for greater awareness and use of innovative technologies; and a perceived need for greater access to specific types of capital (i.e., early-stage, growth stage and angel and venture capital). The program delivery model, with local presence as well as convening and pathfinding roles, allows the Agency to be responsive to the diverse and changing economic needs across the region. ACOA has developed or strengthened collaborations to better address immigration, skills and labour, and funding.
The Innovation programming is unique, complementary to other programs, and aligned with current federal government priorities on innovation and business growth. It is clearly aligned with ACOA's mandate to lead economic development in Atlantic Canada through support to SMEs. The programming offers unique features, including flexible, non-dilutive funding options and non-financial supports focused on promoting the growth of SMEs in the region.
To what extent is ACOA contributing to business growth in Atlantic Canada?
The evaluation finds that projects have achieved the expected outcomes of the Innovation programming. Over the evaluation period, ACOA provided $463 million to commercial and non-commercial clients to support the growth of SMEs through Innovation programming. ACOA's funding and other supports contribute to a strengthened innovation ecosystem, the commercialization of new or improved technologies, products, processes or services, and productivity improvements. Statistics Canada data reveals that ACOA clients report higher R&D and productivity compared to non-supported firms. In a survey of all ACOA Enterprise Development (ED) clients, 77% reported that projects contributed to commercialization; 75% indicated that their projects improved the efficiency or capacity of operations; and 67% said that it involved technology acquisition, adaptation or adoption, new equipment or renovations. Key informants, case studies and a document review indicated that ACOA contributes to a strengthened ecosystem, especially through funding to business accelerators and incubators (BAIs) and the important role the Agency plays in developing and maintaining collaborations, pathfinding and convening.
While there are opportunities to improve the ability to capture and report on longer-term impacts of programming, some initiatives currently under way hold promise for improving the quality and availability of performance data. These initiatives include the implementation of a new G&C information system and agreements with Revenue Canada and Statistics Canada, as well as the development of a new standardized national performance framework for BAIs.
The absence of programming would have had a negative impact on the achievement of results. Ninety-one percent of ACOA clients said the absence of ACOA funding would have had major negative repercussions on their project. Key informants and a literature review revealed that the start-up ecosystem (BAIs) would be weaker without ACOA supports.
To what extent is ACOA taking a client-centric approach to program design and implementation?
The evaluation found that the design and delivery of Innovation programming contributes to a client-centric approach to some extent. ACOA employs mechanisms to focus on client needs, including the provision of financial and non-financial supports, program flexibility and delivery model, internal coordination on priorities, partnerships, and collaborations that enable leveraging of funding. In a survey of ACOA clients, 83% of innovation-related clients reported that ACOA's non-financial supports are important.
Some opportunities exist to take a more strategic approach to program delivery and risk. Some factors impede client focus to some degree, including program uncertainty during the evaluation period (e.g., the transition to new innovation programming); ACOA's risk tolerance; aspects of internal and external coordination; and broad economic challenges (e.g., demographics, capital and business culture). A national review of federal innovation programming led to the implementation of new programming (i.e., the Regional Economic Growth through Innovation [REGI] program). Early implementation of REGI has been positive with no major impacts on the ability to fund projects and improved internal clarity on eligibility. However, key informants suggest gaps in the spectrum of federal funding may exist and are influenced by the unique needs in Atlantic Canada. Key informants suggested that coordination with federal partners may decrease the risks of funding gaps.
ACOA contributes to increased collaboration among stakeholders and clients. The Agency's expanded and more deliberate role in convening and pathfinding has developed new partnerships to address changing priorities. In particular, ACOA support enables projects to leverage substantial funds from other organizations at a level consistent with previous evaluations. ACOA also funds certain non-commercial organizations (e.g., Springboard, BioNova, BAIs) that encourage collaboration and business innovation. A document review and key informants indicated that recent efforts to improve coordination among BAIs are positive and must continue. However, key informants also indicated that ACOA should take a more coordinated and consistent approach to funding BAIs to improve their ability to pivot strategically, collaborate and avoid duplication of efforts.
Innovation programming contributes to inclusive growth to some extent. Inclusive growth includes promoting the participation of diverse groups and rural communities to support economic development in the region. The Agency made some efforts toward inclusive growth through funding and collaborations (e.g., immigrants, Indigenous people and women). However, there is growing internal recognition of the need to better support under-represented groups to facilitate economic growth, address skills and labour gaps, and better align with government priorities. Key informants also indicated that ACOA's 28 points of service across the region position the Agency to meet the unique needs of rural businesses.
The evaluation found that ACOA delivers Innovation programming in a cost-effective manner. Internal costs were proportionate to the amount of funding delivered. There was a high degree of client satisfaction with ACOA's service features. Recent internal operational changes are favouring more efficient program delivery (e.g., project approval form [PAF], streamlined approval process, and planned client information management system). Key informants and a review of data identified questions related to efficiency, including project approval times, performance data and ACOA's risk tolerance levels.
Recommendations
Based on the findings and conclusions, the evaluation makes three recommendations for senior management's consideration.
Recommendation 1: Build upon current Agency efforts to address any gaps in funding that may impact the scale up and growth of Atlantic Canadian firms, including through coordination and advocacy with existing and new partners.
Recommendation 2: Examine AIF funding in relation to the current needs of SMEs and the priorities of the GC to confirm the strategic direction for the program and any changes to governance and funding processes.
Recommendation 3: Continue to support a strong Atlantic start-up ecosystem by strengthening the impact of BAIs through the development of clearer funding approaches linked to expected outcomes and better coordination among stakeholders while recognizing unique regional and SME needs.
Recommendation 4: Continue to strengthen programming priorities and collaborations to better address the labour and skills shortage in the region considering the growing demographic pressures and opportunities to enhance efforts on inclusive growth and immigration.
1.0 Introduction
This report presents the results of the evaluation of ACOA's Innovation programming. The programming supports innovation among SMEs in Atlantic Canada through direct funding and non-financial supports. The evaluation builds on the recent national review of innovation programming conducted by the Treasury Board Secretariat (TBS) to support decision making and the monitoring of this critical element of ACOA's portfolio.
ACOA is one of six federal regional development agencies (RDAs) in Canada. The Agency has a mandate to promote economic growth in Atlantic Canada by helping SMEs become more competitive, innovative and productive. It works with diverse communities to develop and diversify local economies. ACOA’s funding programs are delivered by its skilled workforce, which provides advice, convenes stakeholders and leverages funding for clients across the region.
Atlantic Canada, composed of four provinces (New Brunswick, Prince Edward Island, Nova Scotia and Newfoundland and Labrador) and over two million residents, has undergone massive change in recent years, becoming more innovative and championing a growing start-up ecosystem. In fact, Atlantic Canada showed the greatest change in ability to innovate when compared with the rest of the country, increasing from 45.8% in 2010-2012 to 69.1% in 2015-2017.Footnote (1) Startup Genome identified Atlantic Canada as one of the top 14 ecosystems in the world for "bang for buck," number one in early-stage funding for activation ecosystem globally, and fourth for high-growth activation ecosystem globally. Footnote (2) Entrevestor reports that from 2011 to 2017, Atlantic Canada start-ups produced 27 exits, including Radian6 ($326 million) and Q1Labs (reportedly more than $600 million).Footnote (3)
The evaluation covers a six-year period, from April 1, 2013 to March 31, 2019. It assesses the relevance, effectiveness and efficiency of the design and delivery of Innovation programming under the Research and Development, and Commercialization and the Innovation Ecosystem program pillars of ACOA's departmental results framework (DRF).
This report is organized into five main sections:
- ACOA's Innovation programming
- Evaluation approach
- Findings
- Conclusions
- Recommendations
WHY EVALUATE PROGRAMS?
ACOA conducts evaluations of funding programs every five years to meet accountability requirements as well as to support senior management information needs. Evaluations support accountability, innovation and evidence-based decision making within the GC by:
- examining the relevance and performance of programs, initiatives and policies; and
- identifying improvements through management action plans.
ACOA's Innovation programming was last evaluated through the Evaluation of ACOA's Innovation and Commercialization Sub-program (2015).
For more information, see the Treasury Board Secretariat's Policy on Results and the Federal Accountability Act.
Appendices contain additional information: a one-page summary of findings, conclusions and recommendations (Appendix B); the management action plan (MAP) (Appendix C); the evaluation analytical framework (Appendix D); and the evaluation terms of reference (Appendix E).
2.0 Innovation programming
ACOA recognizes the importance of innovation for business growth. Its Innovation programming is designed to help grow and scale SMEs through the delivery of funding and other supports to increase:
- capacity for R&D, commercialization, productivity and business skills, and the attraction and retention of talent;
- the development and demonstration of new or improved technologies, products, processes, or services; and
- collaboration on innovative technology and commercialization projects.
A logic model that describes the links between program activities, outputs and expected outcomes can be found in Appendix A.
2.1 Delivery approach
ACOA's Innovation programming combines funding with non-financial supports such as advice, help with convening partners and leveraging other sources of funding. The programming is sustained financially by three G&C programs: the Atlantic Innovation Fund (AIF), the Business Development Program (BDP) and the REGI program.
The primary clients for Innovation programming are SMEs in Atlantic Canada. In addition, non-commercial clients include industry associations, BAIs, educational and research institutions, and other non-profit organizations.
The program is delivered through a continuous intake model, meaning that proponents submit project applications to ACOA at any point throughout the year. Agency program officers in regional offices across Atlantic Canada work with clients to develop projects, including identifying and convening partners, leveraging other sources of funding, and providing advice and guidance. They evaluate applications for G&Cs and manage approved projects through to the delivery of results. Table 1 provides details on the priorities and processes of each G&C program.
Business Development Program (BDP) | Atlantic Innovation Fund (AIF) | Regional Economic Growth through Innovation (REGI) | |
---|---|---|---|
Focus | Start up, expand or modernize businesses; as well as not-for-profit organizations that provide support to Atlantic businesses. | Develop and bring to market innovative, new or improved products or services. | Two streams:
|
Notes | Launched in 1995; projects align with ACOA’s priorities and provide economic benefits. | Launched in 2001; support for R&D, with strong commercialization potential; regional project review board; continuous intake since 2014. | Launched in October 2018 following TBS’s national review of innovation; two streams + Women Entrepreneurship Strategy. |
2.2 Program alignment and financial resources
All federal departments must have a DRF that outlines their core responsibilities, expected results and programs. Innovation programming is part of ACOA's DRF under the "technologies" pillar, with the expected result that "businesses invest in the development and commercialization of innovation technologies in Atlantic Canada." For more information on the Agency's DRF, the program inventory and performance indicators see ACOA's DRF and program inventory of record as presented in the Agency's 2018-19 Departmental Plan.
Innovation programming expenditures include the three G&C funding programs (e.g., AIF, BDP and REGI) as well as operations and maintenance (O&M). As shown in Table 2, between April 1, 2013 and March 31, 2019, the Agency invested almost $463 million of G&C funding in 850 projects. The Agency also expended a further $45 million of O&M on salaries (93%) and on travel and other operating expenses (7%). In 2018-2019, there were approximately 70 full-time equivalent employees at ACOA directly delivering Innovation programming.
Fiscal year | G&C ($M) | O&M ($M) | Total ($M) | |
---|---|---|---|---|
Salaries | Other | |||
2013-2014 | $ 75.5 | $ 7.1 | $0.7 | $ 83.3 |
2014-2015 | $ 72.2 | $ 7.5 | $0.4 | $ 80.1 |
2015-2016 | $ 69.3 | $ 6.7 | $0.3 | $ 76.3 |
2016-2017 | $ 64.4 | $ 6.5 | $0.6 | $ 71.5 |
2017-2018 | $ 82.5 | $ 7.1 | $0.6 | $ 90.2 |
2018-2019 | $ 98.6 | $ 6.8 | $0.5 | $105.9 |
Total | $462.5 | $41.6 | $3.1 | $507.6 |
Source: ACOA GX financial database (extracted May 10, 2019).
2.3 Program implementation context
A number of contextual changes affected the delivery of the Innovation programming over time. Table 3 summarizes these changes and their impacts on the programming over the evaluation period. Most important, innovation became a greater priority of the federal government, with a national review of innovation programming across departments and the implementation of the new REGI program.
Context | Implementation context | Impacts on the program |
---|---|---|
Federal | Changes in government priorities in 2015 included greater emphasis on innovation, start-ups, advanced manufacturing, the ocean supercluster, international trade and investment, and supporting diversity and inclusiveness. In 2017-2018, TBS undertook a horizontal review of innovation, which explored 92 program streams across 20 federal organizations. |
|
Provincial | Shifting provincial government priorities, approaches and availability of financial resources to support regional economic development. |
|
Innovation ecosystem | The innovation ecosystem in Atlantic Canada is made up of a complex and shifting range of actors, including four provincial governments, federal departments, universities and research institutes and a growing number of BAIs and other business support organizations. |
|
Departmental | Internal governance changes have included the launch of pan-Atlantic priority files to better address government priorities. |
|
3.0 Evaluation approach
This section outlines the approach for evaluating the Innovation programming, including methods, governance, and methodological strengths and limitations.
3.1 Evaluation questions
Based on the planning consultations and requirements of the Policy on Results and the Federal Accountability Act, the evaluation team identified three evaluation questions to guide the collection of data and the analysis of results. The questions reflected the need to examine the relevance and performance of the programming while focusing on better understanding the current needs of clients, early impacts from the implementation of the REGI program, and contributions to start-ups and the ecosystem.
Table 4 outlines the strategic questions with sub-questions.
WHAT IS INNOVATIVE ABOUT THIS EVALUATION?
This report is Part 1 of an integrated, two-part approach to evaluating the Agency’s Enterprise Development (ED) programming. A second report, focused on the Agency’s Growth and Trade programming, will be completed in May 2020. This integrated approach allows for efficiencies through shared planning and data collection processes as well as one EAC.
1. To what extent is the programming addressing a unique and demonstrable need for Atlantic Canadians? |
---|
a. What are the economic development needs in Atlantic Canada? To what extent does the programming respond to these economic development needs? |
b. To what extent is the programming unique and complementary to other programming offered in the region? |
2. To what extent is ACOA contributing to business growth in Atlantic Canada? |
a. How has ACOA contributed to key expected longer-term outcomes: a strengthened innovation ecosystem, increased commercialization, increased SME productivity, and increased exports and investment? |
b. What factors facilitate or impede the achievement of outcomes? |
c. How does the program align with current federal and ACOA strategic priorities? To what extent do investments in newer priorities contribute to business growth in Atlantic Canada? |
d. What impact would the absence of program funding have on Enterprise Development (i.e., incrementality) in Atlantic Canada? |
3. To what extent is ACOA taking a client-centric approach to program design and implementation? |
a. To what extent have the design and delivery of the program addressed the business growth needs of clients in the ecosystem? |
b. To what extent has ACOA contributed to increased collaboration (e.g., convening, pathfinding and leveraging) among stakeholders and clients? Best practices? Lessons learned? |
c. To what extent and how does the program design and implementation contribute to inclusive growth (e.g., language minorities, youth/seniors, Indigenous people, new immigrants, gender, rural/urban, etc.)? |
d. To what extent have recent internal operational changes resulted in more efficient program delivery (e.g., program simplification, Agency-wide teams for newer priorities, capacity of account managers and integration of GBA+)? |
3.2 Data collection methods
The evaluation used both qualitative and quantitative data collection methods (Table 5) to address the evaluation questions. The choice of methods was determined based on their relevance and reliability, data availability and costs.
Data collection tools and objectives | Sources |
---|---|
Document and literature review Validate economic development needs, alignment with government priorities and broad outcomes |
|
Internal data Document program design, implementation, nature of project, client type and performance |
|
Key informant interviews Assess relevance and performance from the perspective of various stakeholders |
|
Case studies Illustrate factors that lead or contribute to program impacts |
|
Client survey Assess relevance and performance from clients’ perspective |
|
3.3 Evaluation strengths and limitations
The evaluation was designed to have a calibrated approach, using mixed methods to identify strong findings while meeting TBS timelines and providing useful information to meet senior management needs. The evaluation has a number of strengths, including a skilled internal evaluation team and high stakeholder engagement through an EAC (see Appendix F) and senior management consultations. The evaluation includes detailed case studies to examine needs and longer-term impacts. The online client survey received a high response rate (48%). The evaluation considered GBA+ in its design and implementation of data collection methods, as well as in the synthesis of findings.
These practices helped to mitigate the common limitations that occur as part of most program evaluations. As outlined in Table 6, the evaluation team considered the limitations of the study and implemented a number of mitigation strategies.
Limitation | Mitigation strategies |
---|---|
Programming changes The review of innovation programming and the implementation of a new program (e.g., REGI) during the scope of the evaluation |
|
Performance measurement data Direct outcomes of the programming are limited |
|
Timelines for evaluation Delays due to internal programming changes and 2019 federal election |
|
Selection bias Clients who participated in the survey may be different from those who did not, or whose projects were not approved |
|
Longer-term outcomes Program generates longer-term outcomes that are difficult to measure |
|
4.0 Findings
4.1 To what extent is the programming addressing a unique and demonstrable need for Atlantic Canadians?
Finding 1: Skills and labour, use of innovative technologies and access to capital are core issues in Atlantic Canada. Needs exist to the same degree or are greater than were reported in the previous evaluation due to increasing demographic pressures.
The results of the evaluation demonstrate a strong, continued need for ACOA's Innovation programming. Innovation is a priority of the federal government and is important for regional economic growth, especially in the context of major demographic and labour market challenges.
Atlantic Canada has been experiencing a well-documented and growing shortage of skills and labour for several decades, which acts as a barrier to the innovation, productivity and growth of SMEs. A review of recent documents and literature reveals that the attraction and retention of talent is expected to continue and be compounded by factors such as out-migration and Atlantic Canada's aging population. A Business Development Bank of Canada (BDC) survey of 1,208 entrepreneurs from across the country showed that hiring employees is more challenging for Atlantic Canadian entrepreneurs than for those in other regions of Canada. In fact, when compared with other regions of the country, Atlantic Canada had the most (50%) entrepreneurs who reported difficulties hiring in the last 12 months, compared with a national average of 39%. Almost two thirds (62%) expect to face a shortage over the next five years.Footnote (4)
The extent of labour shortages in the Atlantic region is surprising because it also has a higher unemployment rate than the Canadian average. Factors contributing to this include the region's older population, lower labour force participation rate and higher share of seasonal workers.Footnote (5) Recent internal analyses suggest that both the labour shortage and the unemployment rate are even more pronounced in rural areas such as Cape Breton (Nova Scotia), northern New Brunswick, and many areas of Newfoundland and Labrador.Footnote (6) The high unemployment rate in some rural areas can be explained by the mismatch between the positions offered and the available labour force (e.g., skills, mobility, etc.).
The need for talent relates to both skilled and unskilled labour, and also includes a gap in business management skills. There is a specific challenge in attracting senior leadership with the skills required to scale up and grow companies beyond the start-up phase. In addition, key informants, the client survey and the literature reviewFootnote (7) underlined that workers with skills in sales (particularly in the technology sector), high-tech/computing, marketing and export are in high demand and difficult to find.
The Atlantic Canadian business landscape is characterized by a need for greater awareness and the use of innovative technologies to improve productivity and growth. There are opportunities to undertake more adaptation and adoption of technology as well as more automationFootnote (8) to address the labour shortage. Survey results indicate that clients recognize this opportunity: among those clients who indicated that recruiting and retaining employees was a need, 71% indicated their project involved technology acquisition, adaptation or adoption, new equipment or renovations. However, key informants underlined that a risk-averse business culture in Atlantic Canada tends to restrain the use of innovative technologies and challenges the promotion of a growth mindset among entrepreneurs.
Access to funding continues to be a need for Atlantic Canadian firms. Through the ED client survey, 52% (n=493) of ACOA's innovation-related clients indicated that obtaining financing was a key need. Case studies supported the need for more early-stage financing for start-ups and more angel and venture capital for high-risk ventures in Atlantic Canada. Many key informants underlined that provincial programs in Atlantic Canada do not provide significant direct financial support to SMEs and that this support has declined over the past years.
Some key informants also identified a need for greater access to capital, particularly in the early and growth stages. The perceived gaps were tied to a lack of both private and public-sector investments in Atlantic Canada. A number of key informants suggested that some SMEs might have difficulties accessing large amounts from federal departments such as ACOA and the National Research Council of Canada – Industrial Research Assistance Program (NRC-IRAP) due, at least in part, to a lack of available funding. They noted that access to funding can be particularly challenging for firms with growth trajectories requiring the adoption of very expensive advanced technology or equipment. These challenges speak to the need for the funding continuum available for Atlantic firms to function in a seamless, client-centered way that allows firms access to the financing required to effectively execute their growth plans.
Finding 2: Largely influenced by the program delivery approach, featuring a local presence and relationships with SMEs and partners, the Agency is aware of the varied needs that exist across the region. The Agency has taken some actions to respond to these newer and emerging needs.
The Agency's decentralized delivery model, flexible funding approach, non-financial supports such as pathfinding and convening, along with ongoing and new partnerships and collaborations, all facilitate the Agency's response to changes and opportunities in Atlantic Canada.
ACOA is taking action to address the following items:
- The skills and labour shortage. Financial support is provided directly to SMEs to permit the hiring and developing of skilled employees. ACOA also contributes to the development of skills and knowledge through its support to non-commercial organizations such as BAIs in Atlantic Canada. Two-thirds (69%) of non-commercial innovation clients indicated that their project increased business skills. Among these, the top four types of skills were marketing, entrepreneurship, business management and sales. One of the key outcomes of the BAIs, as underlined through the case study interviews, is the provision of training, mentorship and a space where start-up entrepreneurs can collaborate and learn from each other. To help address demographic and labour market challenges, ACOA assists employers and communities in attracting and retaining skilled workers to Atlantic Canada. By partnering with the four Atlantic Provinces and Immigration, Refugee and Citizenship Canada (IRCC), the Agency promotes the Atlantic Immigration Pilot (AIP) Program to its clients and refers them to the appropriate partners in the region. Between July 2017 and October 2019, the program granted 5,010 permanent resident admissions to principal applicants and their dependents to settle in the region and fill jobs in a variety of sectors.Footnote (9)
Programming in Action: Atlantic Immigration Pilot
ACOA has assisted Somru Bioscience in P.E.I. since 2013. This biotechnology company develops antibody technology to accelerate the drug development process. The company's vice-president and co-founder immigrated to P.E.I. on a student visa in 2000. The company currently exports to more than 20 countries and is in the midst of a major expansion. Somru is hiring two senior scientists using the AIP to augment local recruiting.
- The need for greater awareness and the use of innovative technologies. The Agency has increased its focus on advanced manufacturing by providing targeted training to internal employees dealing directly with clients and by funding technology missions for entrepreneurs to learn about new technologies that could lead to increased productivity. A review of recent internal analyses reveals that ACOA's funding for larger-scale automation projects has almost tripled between 2016 and 2019 from $35 million to $95 million. ACOA is also an active participant in the Accelerated Growth Service, an initiative that facilitates a more coordinated approach to supporting established firms with high growth potential. Moreover, the Agency maintains a strong relationship with the NRC-IRAP, an organization dedicated to providing advice, connections and funding to help SMEs drive growth and innovation through applied R&D of technology.
- The lack of access to capital. ACOA has a flexible approach to providing funding to support specific economic needs across the region; it provided $463 million to innovative Atlantic Canadian companies and to the innovation ecosystem over the period covered by this evaluation. ACOA plays a strong convening and pathfinding role, which helps entrepreneurs access other sources of funding and support.
Finding 3: While ACOA's Innovation programming is unique and complementary to other programming offered in the region, there may be gaps in the funding continuum in Atlantic Canada.
ACOA's programming complements other types of programming and funding available to SMEs and intermediaries for innovation and growth activities. Funding and services for SMEs and other organizations come in a variety of arrangements across the growth spectrum from both government and non-government organizations (Table 7). The main features that set ACOA's programming apart from that of other sources are: mandate to support economic development in Atlantic Canada, focus on SMEs, and flexible financial and non-financial supports.
Financing Organization | Characteristics |
---|---|
ACOA Innovation programming |
|
National Research Council – Industrial Research Assistance Program (NRC-IRAP) |
|
Business Development Bank of Canada (BDC) |
|
Innovation, Science and Economic Development – Strategic Innovation Fund |
|
Provincial governments |
|
Finding 4: ACOA's Innovation programming is aligned with current federal and ACOA strategic priorities and contributes to business growth in Atlantic Canada through participation in various initiatives.
ACOA's Innovation programming aligns with the overall federal government objective of helping "Canadian businesses grow, innovate and export so that they can create good quality jobs and wealth for Canadians" as outlined in the mandate letter for the Minister of Innovation, Science and Economic Development (2015). More specific objectives are outlined in several federal priority documents:
- The mandate letter for the Minister of Innovation, Science and Economic Development (ISED) prioritized the development of an Innovation Agenda that includes expanding support to BAIs, networks and clusters, and working with RDAs to invest in competitive regional advantages.
- Budget 2016 announced plans to support BAIs, innovation networks, clusters and a tailored approach to coordinating federal services to meet client needs.Footnote (24)
- Budget 2017 announced the federal Innovation and Skills Plan (2017),Footnote (25) and Budget 2018 announced that it would be delivered by ACOA in the Atlantic region, with support for women entrepreneurs through the Women Entrepreneurship Strategy and support for skills development and economic diversification to help with the transition to a low-carbon economy.Footnote (26)
- Budget 2018 outlined the new superclusters initiative, including maximizing Canada's ocean economy in Atlantic Canada.Footnote (27) The consolidation of the Accelerated Growth Service and the IRAP Concierge Service was also announced.Footnote (28)
ACOA's Innovation programming aligns with expected results outlined in its DRF under the technology pillar, which aims at seeing businesses investing in the development and commercialization of innovative technologies in Atlantic Canada. A national review of innovation programming announced in Budget 2017Footnote (29) led to revised programming characterized by a stronger focus on innovation, which is a key priority for the federal government and ACOA.
4.2 To what extent is ACOA contributing to business growth in Atlantic Canada?
Finding 5: ACOA's Innovation programming contributes to the growth of new and existing businesses through its supports to the innovation ecosystem, commercialization and productivity improvements.
Overall, this study found that Innovation programming contributes to the achievement of expected outcomes and, ultimately, to the growth of SMEs in Atlantic Canada. ACOA's flexible, non-dilutive funding, delivery model, strategic and collaborative approach, and non-financial supports (e.g., convening and pathfinding) facilitate the achievement of results.
ACOA's funding and other supports contribute to the scale-up and growth of SMEs and a strengthened innovation ecosystem through the commercialization of new or improved technologies, products, processes or services, and productivity improvements. Over the evaluation period, the Agency provided $463 million to commercial and non-commercial clients toward these expected outcomes. For example, internal data shows that the Agency invested $34 million in early-stage SMEs with high potential for growth, universities, and other not-for-profit organizations that provide supports directly to SMEs, such as BAIs, industry associations, and economic development organizations. These projects focused on innovation and commercialization, marketing, scale up, productivity improvements and the development of business skills.Footnote (30)
A strengthened innovation ecosystem:
- ACOA invested a total of $31 million in BAIs over the period of the evaluation, and has increased its yearly contributions to BAIs and other organizations designed to support innovation of SMEs in the region.
- Case studies demonstrated the importance of BAIs as spaces for networking and building support capacity.
- Recent reports from Entrevestor (2017)Footnote (31) and Startup GenomeFootnote (32) underline the importance of ACOA's support to start-ups within an Atlantic Canadian ecosystem increasingly recognized for the availability of early-stage funding. In particular, Atlantic Canada was identified as one of the top 14 ecosystems in the world for "bang for buck" (measured by a ratio of early-stage funding per start-up and average software engineer salary), number one in early-stage funding for activation ecosystem globally (measured by early-stage funding per start-up) and fourth for high-growth activation ecosystem globally (measured by growth in funding, exits and number of start-ups).
Commercialization of new or improved technologies, products, processes or services:
- ACOA invested $116 million in projects with outcomes focused on commercialization capacity and activities, and $303 million on R&D capacity and activities intended to lead to commercialization.
- R&D expenditures in Atlantic Canada grew by an annual average of 2.5% from 2005 to 2015, exceeding the national average of 1.6%.Footnote (33)
- Through the client survey, 75% (n=716) of all ED clients indicated that their project contributed to commercialization.
- About half (52%, n=476) of commercial, innovation-related clients surveyed indicated that their project improved research capacity (e.g., human resources, facilities, equipment and skills).
Improvements to SME productivity:
- ACOA strategically invested $22 million in projects aimed at increasing adaptation of technology over the evaluation period.
- A majority of all clients (69%; n=612) indicated that their project involved technology acquisition, adaptation or adoption.
- Statistics Canada data show that labour productivity of ACOA-assisted firms experienced a healthy growth compared to non-assisted firms. This includes higher than average annual growth in sales (2.7% versus 0.4%),
- increased labour productivity (1.7% versus 0.3%) and generation of gross profit at a quicker pace than non-assisted firms.
- Key informants and published reportsFootnote (34) underline the importance of productivity improvements such as adaptation and adoption of new technologies and automation as part of the solution to the lack of skilled employees.
- Seventy-one percent (n=346) of innovation-related clients who indicated that recruiting and retaining skilled employees was a need also indicated that their project involved technology acquisition, adaptation or adoption.
Programming in Action: BioNova
ACOA has invested $1.6 million in BioNova, Nova Scotia's life sciences association. Funding assists programs and initiatives to help life sciences member companies grow their business. BioNova also coordinates incoming and outgoing trade activities on behalf of the Atlantic Canada BioIndustries Alliance.
Programming in Action: Advanced Manufacturing
Since 2002, ACOA has provided funding for La Maison Beausoleil in New Brunswick. This company developed a one-of-a-kind automated system that sorts, counts and boxes oysters. This automated production line will enhance productivity and improve product sustainability and food safety.
Programming in Action: Commercialization
For over a decade, ACOA has supported Verafin, a Newfoundland and Labrador company that uses artificial intelligence to design and sell innovative financial crime management software. Since its inception in 2003, Verafin has been involved with Memorial University's Genesis Centre and is now considered a leader for other tech start-ups in the community. In 2019, Verafin signed a record-breaking $515 million venture capital deal.
A complex and changing environment, both internally and externally, affects the achievement of program outcomes. As outlined in section 4.1, the three core needs in the region (e.g., skills and labour, awareness and use of innovative technologies, and access to capital) coupled with the region's risk-averse business culture negatively impact the achievement of results. For example, key informants pointed to the need to support and encourage a growth mindset among business owners, due to a reluctance to take on riskier projects that may contribute to growth. The Agency's use of provisionally repayable contributions aid clients in undertaking these riskier projects, as businesses need access to more risk capital that is not available through traditional lenders.
Other factors, more closely within the control of ACOA, were also raised through key informant interviews and published reports as challenging the achievement of outcomes.
- A complex and dispersed innovation ecosystem leads to gaps in assistance offered to SMEs and to confusion within the BAI ecosystem.
- Uncertainty around the direction of AIF programming in light of recent program reviews and the implementation of new programming.
- Certain internal structures that make program delivery more complex (e.g., inconsistency in funding approaches between regions, lack of coordination between funding programs).
- Gaps in performance information, including reach and impact on vulnerable populations. Several initiatives are under way, however, and are expected to improve performance data across programming (e.g., implementation of a new project information management system and the recent establishment of standard performance indicators for BAIs in Canada).Footnote (35)
Finding 6: The absence of programming would have had a major negative impact on the achievement of outcomes.
ACOA's financial investments directly lead to the achievement of project results and allow projects to proceed as planned, including through the leveraging of other funding. Case studies, key informant interviews and client survey results reflect that ACOA supports have a major impact on the Atlantic Canadian innovation ecosystem.
ACOA's support facilitates leveraging of other sources of funding. Case studies indicate that ACOA's involvement and investments have a major influence on leveraging from other sources, as the Agency is perceived as having solid project risk assessment, monitoring and reporting processes.
The need for ACOA supports exists to at least the same extent as it did in the previous evaluation of the programming, completed in 2015. The importance of ACOA's contributions is supported by the results of the client survey with 91% (n=496) of innovation-related clients indicating that their project would have experienced a major negative impact without ACOA's support. As detailed in Figure 1, if the projects had still gone ahead, most innovation-related clients would have experienced delays or projects with a smaller scope. Case study results indicate that a majority of BAIs in the region, important actors in the innovation ecosystem, would not exist without ACOA support.
4.3 To what extent is ACOA taking a client-centric approach to program design and implementation?
Finding 7: The design and delivery of Innovation programming supports a client-centric approach to some extent. Opportunities exist to take a more strategic approach to program delivery and portfolio risk.
Recent changes to program funding mechanisms are perceived as positive for the ecosystem. According to key informants, the implementation of the REGI program in 2018-2019 was relatively seamless and did not create issues for clients or for ACOA staff. Furthermore, key informants reported improved clarity around eligibility of projects without affecting the ability to fund projects flexibly. However, program changes over the evaluation period raised questions about the role of the AIF. Given new programming and other factors, some key informants identified the need to rethink the AIF's funding objectives and mechanisms to address possible overlap with REGI and gaps in client needs.
While ACOA's supports for the innovation ecosystem correspond to client needs, some gaps continue to exist in coordination and communication. Many key informants noted the important work that ACOA has done to improve BAI coordination in recent years. They also stated that ACOA could stand a more coordinated and consistent approach to funding BAIs to improve their ability to pivot strategically, collaborate and avoid duplication of efforts. They also stated that better coordination, collaboration and knowledge sharing between regional offices is needed while balancing the need for flexibility to meet different needs across the region.
Some ACOA key informants noted that recent internal coordination mechanisms (e.g., champion files) facilitate pan-Atlantic collaboration between ACOA's offices and improve understanding of client needs. Some suggested that ACOA's existing governance structures do not always promote the more strategic, matrix approaches. Internal key informants also report that there is a need to clarify language and definitions of certain key themes related to newer priorities to promote better cohesion and understanding among ACOA staff (e.g., innovation, start-up, scale up).
ACOA provides funding to a range of non-commercial organizations that are actors in the innovation ecosystem and that help to address SMEs' needs for advice, access to funding and other supports. For example, key informants indicated that support offered by ACOA to most Atlantic Canadian BAIs provides start-ups with access to tailored programming based on industry needs in communities across the region.
ACOA's ability to meet client needs may be impacted by its risk tolerance level. There is a perception that ACOA's culture remains risk averse. Key informants report that the Agency's approach to risk is complex and can affect the types of projects developed and approved as well as levels of funding. Key informants indicated a need for the Agency to increase the risk level of its portfolio to further support the growth of SMEs and encourage more risk within the ecosystem. A faster, more agile and certain Atlantic Canada states that innovation requires risk tolerance and that governments should tolerate a higher acceptable level of risk in order to increase overall return on investments.Footnote (36)
Examples of actors in the Atlantic Canada innovation ecosystem:
- Commercial businesses
- Business Accelerators and Incubators (BAIs) – support for start-ups and scale-ups
- Universities and research institutions – R&D
- Other non-commercial organizations – e.g., Springboard, BioNova, industry associations
- Government departments and agencies
ACOA's approach to risk will be examined more closely in the second report resulting from this evaluation – the Evaluation of ACOA's Business Growth and Trade Programming – expected to be finalized in May 2020.
Finding 8: ACOA contributes to increased collaboration among stakeholders and clients. However, opportunities exist to improve coordination and collaboration within the innovation ecosystem.
ACOA's non-financial supports contribute to meeting client needs. Most surveyed innovation-related clients (89%; n=470) indicated satisfaction with the business knowledge and advice offered by ACOA. Key informants stated that the Agency has worked to increase the capacity of its staff in areas that are critical for the region, such as advanced manufacturing, automation and artificial intelligence. They also underlined that ACOA's convening role, including through traditional project supports and through the Accelerated Growth Service, was key in supporting innovation and growth.
ACOA is playing a more deliberate convening and pathfinding role since the previous evaluation of the programming. In addition to important funding, the Agency provides non-financial support such as advice, guidance, expertise or assistance to clients and communities. Innovation-related client survey results revealed that this type of support is considered important by 83% (n=495) of respondents. Some clients explained that ACOA helps them connect with other players and provides visibility to their project, while others said that the Agency's personnel made their company aware of funding opportunities, which led to obtaining additional capital. Key informants and case studies indicated that a more formalized approach to convening and pathfinding roles is also taking place through the Accelerated Growth Service and coordination of BAIs. As detailed in Figure 2, a variety of partners were involved (either through provision of financial or in-kind support) in ACOA-assisted projects (n=291)
Key informants, case studies and published reports (e.g., BAI Performance Measurement Framework by the Deep Centre)Footnote (37) point to the growing engagement of ACOA with other federal partners such as Employment and Social Development Canada, the Department of Fisheries and Oceans Canada, Global Affairs Canada and IRCC (through the Atlantic Immigration Pilot). ACOA's support to the innovation ecosystem through BAIs also promotes collaboration within the innovation ecosystem itself. BAIs play a key role in providing space for collaboration and networking among entrepreneurs, as revealed through the case studies. In some regions, co-location between ACOA and the NRC-IRAP was also underlined as a best practice, improving collaboration and complementarity between the two organizations.
In light of the growth and evolving nature of the ecosystem, there are opportunities to improve coordination of partners and actors. For example, continued and enhanced coordination between BAIs and ACOA's regional offices could be beneficial to avoid risk of duplication within the same region and to promote sharing of best practices among Atlantic Canadian BAIs.
SpringboardFootnote (38), an organization funded by ACOA to work with researchers and industry to develop relationships that will lead to commercialization, is an example of a useful mechanism for collaboration in Atlantic Canada. An ACOA internal review of Springboard conducted in 2018 found that while progress has been made on developing better connections between researchers and industry to support innovation
and commercialization, there are opportunities to improve coordination and communication by tailoring research strategies to needs of business, improving engagement and strengthening relationships with industry. As a result of the review, ACOA's new funding agreement with Springboard (2020-2023) will see the organization build upon its established network to increasingly focus on projects and plans that support regional strategic objectives. While Springboard will continue the momentum that has been generated throughout previous funding agreements relative to collaborative industry research engagements and commercialization support, an increased emphasis will be directed toward supporting a robust industry engagement and technology commercialization network; advancing innovation in strategic industries and sectors and facilitating the growth of start-ups and high-potential firms.Programming in Action: Innovation Ecosystem
ACOA has supported Volta Labs Inc. in Nova Scotia since 2013. This not-for-profit entity has strengthened the technology ecosystem in Atlantic Canada through mentorship and community-driven aid. Volta Labs' programs include Volta Cohort, a collaboration with ACOA, the Business Development Bank of Canada, and Innovacorp, to help early-stage, high-calibre companies from across Atlantic Canada attract and secure investment.
Finding 9: There is a perception that programs are open to all equally, but there is growing recognition that there are opportunities to better support specific groups.
While ACOA has long delivered programming for particular population groups, it has paid increasing attention to certain vulnerable populations over the evaluation period. ACOA is particularly strong in delivering programming to meet the needs of rural populations and has made strides to increase efforts for new immigrants, Indigenous people and women. No specific focus on youth was noted through this evaluation.
On July 4, 2016, the Government of Canada and the four Atlantic Provinces launched the Atlantic Growth Strategy, which sought to drive long-term economic growth in the Atlantic region by implementing targeted, evidence-based actions under five priority areas.Footnote (39) The Agency champions immigration mostly through activities in partnership with IRCC. ACOA's involvement in the Atlantic Immigration Pilot (2017-2020), a key initiative under the Atlantic Growth Strategy, helps address labour and skills needs and ensures long-term economic growth in the region by attracting and retaining skilled immigrants.Footnote (40) The case studies also underlined that BAIs supported by the Agency provide supports to immigrant entrepreneurs, although retention remains a challenge. Based on Statistics Canada data, there is room for improvement, as immigrants owned a low share (5.2%) of ACOA client firms and comparable firms (5.1%) in 2015.Footnote (41)
ACOA has recently prioritized Indigenous economic development through the creation of a pan-Atlantic priority file, but there remains an opportunity to provide more support for indigenous businesses. In Atlantic Canada in 2017, 1.1% of all businesses had indigenous ownership and that percentage has decreased compared to 2014. The Atlantic Province Economic Council Indigenous report underlines specific training needs related to networking, building and maintaining relationships with financial institutions, advice on business growth, access to funding, entering new markets, developing a business strategy, and finding and retaining skilled employees. Key informant interviews indicated that more could be done in terms of adapting ACOA's approach to Indigenous development, but acknowledged that efforts are being made through the Agency's Indigenous file. Some promising practices are emerging, however, such as recent efforts in Cape Breton to develop strong partnerships and relationships with Indigenous communities.
The programming also includes some initiatives for women in business. Key informants and case studies indicated that a number of initiatives are in place to increase women's presence in the business and innovation space. ISED's Women Entrepreneurship Strategy, a national $2 billion investment put in place through Budget 2018,Footnote (42) aims to help women start or grow a business. Other initiatives are undertaken through the BAIs in Atlantic Canada, such as Genesis' Women in Tech Peer Group taking place in Newfoundland and Labrador.
Statistics Canada data and key informants suggested that further efforts would be merited. Women owned only 11.9% of ACOA-assisted firms in 2015 compared to 10.8% in 2010.Footnote (43) In the innovation ecosystem, women-led start-ups receive a lower share of venture capital in Atlantic Canada compared with the rest of Canada.Footnote (44) Despite this, between 2010 and 2015, female ACOA clients not only registered higher average growth rates than male ACOA clients in a number of employment and financial indicators but also outperformed female-owned businesses from the comparable group in most areas. Case studies indicated that there has been increased focus on supports to women in technology, computer science and entrepreneurship.
Atlantic Canada is highly rural and unique challenges exist for businesses outside urban centres. Key informants and key documents indicated that rural areas experience challenges, in particular related to access to risk capital for start-ups.Footnote (45) However, key informants indicated that ACOA's 28 points of service, including many in rural communities, facilitate meeting specific client needs. The Agency's flexible approach also allows for support tailored to rural-based companies.
Finding 10: There is evidence to suggest that recent internal operational changes are supporting more efficient program delivery.
Delivery costs for ACOA's Innovation programming are comparable to amounts reported in previous evaluations. The total expenditures for Innovation programming included in this evaluation were $507.6 million. An analysis of internal financial data indicates that ACOA's cost to deliver Innovation programming has remained relatively unchanged over time, with a cost to deliver $1.00 of G&C funding of $0.097 over the period of this evaluation (2013-2014 to 2018-2019), compared with $0.092 as reported in the last evaluation (2007-2008 to 2012-2013).
There is a high degree of satisfaction with ACOA's service features. Key informants and case studies indicated that ACOA staff are proactive, efficient and generally helpful at all stages of the project life cycle, including completing forms and navigating the application and claims processes. They described ACOA as the cornerstone of many projects and stressed the importance of non-financial supports provided by ACOA staff. As detailed in Figure 3, the overwhelming majority of innovation-related client survey respondents indicated strong satisfaction with a range of ACOA's service features. They are very satisfied with the courteousness and professionalism of ACOA personnel (97%), the ongoing business relationship with ACOA personnel (94%), communication from ACOA personnel on the status of their request (93%), and the availability of ACOA personnel (93%).
Project approval times have improved, though some clients still see them as a challenge. When compared with other ACOA processes and service features, innovation-related client survey results revealed a slightly lower satisfaction rating for ease of the application process and paperwork (84%) and speed of turnaround time of the application process (82%). Internal key informants noted that recent operational changes focused on streamlining processes, including changing delegation levels, have improved approval times. The transition to the redesigned PAF also facilitated programming efficiency. In 2017, the Agency undertook a pilot project to examine potential areas for improving and streamlining project evaluation and approval processes, with a specific focus on commercial applications. The goals of this exercise were to minimize the burden on clients by expediting processing of applications and reducing wait times, and establishing service standards for specific steps in the approval process.
5.0 Conclusions
Overall, the evaluation found that ACOA's Innovation programming remains relevant and achieves expected outcomes related to economic development in Atlantic Canada. The delivery model is grounded in the Agency's deep knowledge of businesses and expertise in convening and pathfinding, which facilitates the achievement of results. There are some areas for attention to ensure ongoing and strengthened strategic investments through the programming. The evaluation concludes that:
- Programming is relevant to the existing needs of SMEs, aligned with ACOA's mandate and GC priorities, and complements other supports in the region.
- ACOA is aware of and has addressed changing needs to a certain extent. There are risks to its continued ability to address the needs of SMEs due to potential funding gaps, access to labour and skills, and uncertainties about AIF funding.
- ACOA's Innovation programming contributes to the growth of new and existing businesses through its supports to the innovation ecosystem, commercialization and productivity improvements. ACOA plays an important role in developing and maintaining collaborations, pathfinding and convening. While there are opportunities to improve the ability to capture and report on longer-term impacts of the programming, the current implementation of a new project management system and other initiatives under way hold promise in improving the quality and availability of performance measurement data, including that related to BAIs.
- The design and delivery of programming supports awareness of changing needs, a client-centric approach and the achievement of results. A strong regional presence allows for knowledge of needs, capacities and partners as well as the development of relationships across the ecosystem. The recent implementation of REGI appears to be positive. Opportunities exist to further improve internal coordination and to better address client needs, including those of under-represented groups.
- ACOA has mechanisms in place to support the efficient delivery of programming to facilitate the achievement of outcomes. Opportunities exist to improve the direction on appropriate risk tolerance levels, to clarify the direction for the AIF, and to build upon existing internal and external coordination and collaboration efforts.Footnote (46)
6.0 Recommendations
Within the context of the recent ACOA funding program changes, and the complexity that exists in the innovation ecosystem in Atlantic Canada, ACOA should:
Recommendation 1: Build upon current Agency efforts to address any gaps in funding that may impact the scale up and growth of Atlantic Canadian firms, including through coordination and advocacy with existing and new partners.
Recommendation 2: Examine AIF funding in relation to the current needs of SMEs and priorities of the Government of Canada to confirm the strategic direction of the program and any changes to governance and funding processes.
Recommendation 3: Continue to support a strong Atlantic start-up ecosystem by strengthening the impact of BAIs through the development of clearer funding approaches linked to expected outcomes and better coordination among stakeholders while recognizing unique regional and SME needs.
Recommendation 4: Continue to strengthen programming priorities and collaborations to better address the labour and skills shortage in the region considering the growing demographic pressures and opportunities to enhance efforts on inclusive growth and immigration.
Appendix B presents a one-page summary of key findings, conclusions and recommendations.
ACOA senior Programs management has agreed with the evaluation's recommendations. It has developed a MAP that details the actions the Agency will take to address each of the recommendations. Appendix C presents the MAP.
Appendix A: Innovation programming logic model
Appendix B: Summary of findings, conclusions and recommendations
FINDINGS | CONCLUSIONS | RECOMMENDATIONS |
---|---|---|
Relevance |
|
Within the context of recent ACOA funding program changes, and the complexity that exists in the innovation ecosystem in Atlantic Canada, ACOA should:
|
Needs related to innovation in Atlantic Canada exist to the same degree or are greater than reported in the previous evaluation. Shortage of skills and labour is increasingly a challenge influenced by demographic shifts; awareness and use of innovative technologies and access to specific types of capital (early stage, growth stage, angel and venture capital) are also important. | ||
The program delivery model with local presence and convening and pathfinding roles allows the Agency to be responsive to diverse and changing economic needs across the region. A national review of innovation programming led to the implementation of new programming (e.g., the REGI program). ACOA developed or strengthened collaborations to better address immigration, skills and labour, and funding. | ||
Programming offers unique features and is complementary: flexible, non-dilutive funding options and non-financial supports focused on promoting growth of SMEs in Atlantic Canada. Key informants suggest gaps in the spectrum of federal funding exist and are influenced by unique needs in Atlantic Canada. | ||
ACOA’s Innovation programming is aligned with federal government priorities on innovation and business growth. It is clearly aligned with ACOA’s mandate to lead economic development in the region through supports to SMEs. | ||
Contribution to Business Growth | ||
ACOA investments support innovation in Atlantic Canada ($463 million to 850 projects over six years). ACOA clients report higher R&D and productivity compared to non-supported firms (Statistics Canada). In a survey of ACOA clients, 77% reported that projects contributed to commercialization; 75% indicated that their projects improved the efficiency or capacity of operations; 69% said that it involved technology acquisition, adaption or adoption, new equipment or renovations. Key informants, case studies and a document review indicated that ACOA contributes to a strengthened ecosystem. | ||
Longer-term results are difficult to measure. Several initatives are under way to improve the quality of data, including the implementation of a new G&C information system and agreements with Revenue Canada and Statistics Canada, as well as the development of new standardized national performance indicators for BAIs. | ||
The absence of programming would have had a major negative impact on the achievement of results. Ninety-one percent of clients said the absence of ACOA funding would have had a major negative impact on their project; 48% of commercial clients and 80% of non-commercial projects would not have proceeded without ACOA support. ACOA’s support facilitates the leveraging of other sources of funding: ACOA leverages $1.22 from partners per $1 of funding. Key informants and a literature review revealed that the start-up ecosystem (e.g., BAIs) would be weaker without ACOA supports. | ||
Client-centric Approach | ||
ACOA employs mechanisms to focus on client needs: provision of funding and non-financial supports; program flexibility and delivery model; internal coordination on priorities; partnerships and collaborations that leverage funding. Early implementation of REGI has been positive, with no major impacts on the ability to fund projects and improved internal clarity on eligibility. Factors that impede client focus to some degree: program uncertainty during the period (e.g., transition to REGI, AIF); ACOA’s portfolio risk tolerance; aspects of internal and external coordination; broad economic challenges (e.g., demographics, capital, business culture). Key informants suggest that strengthened coordination with key partners may decrease the risks of funding gaps. | ||
ACOA contributes to increased collaboration among stakeholders and clients through both funding for projects and other activities. Eighty-three percent of clients reported that ACOA’s non-financial supports are important. ACOA funds certain non-commercial organizations that support collaboration (e.g., Springboard, BioNova, BAIs). A document review and key informants indicated that recent efforts to improve coordination among BAIs are positive and must continue. ACOA is playing an expanded and more deliberate role in convening and pathfinding and has developed new partnerships to address changing priorities. | ||
The Agency is making some efforts towards inclusive growth (e.g., immigration, Indigenous, women). However, there is growing internal recognition of the need to better support under-represented groups to facilitate economic growth, address skills and labour gaps, and better align with government priorities. | ||
The cost to deliver $1 of G&C funding was $0.097 and aligned with costs in other program areas. Recent internal operational changes are supporting more efficient program delivery (e.g., PAF, streamlined approval process, and planned client information management system). A client survey revealed a high degree of satisfaction (89%) with ACOA service features. Key informants and a review of data identified questions related to efficiency: project approval times, performance data, ACOA’s risk tolerance levels. |
Appendix C: Management action plan
RECOMMENDATION | PLANNED ACTIONS | RESPONSIBILITY | TARGET DATE |
---|---|---|---|
Recommendation 1: Build upon current Agency efforts to address any gaps in funding that may impact the scale-up and growth of Atlantic Canadian firms, including through coordination and advocacy with existing and new partners. |
The Agency will coordinate with other federal innovation partners in Atlantic Canada to explore issues impacting SME financing and to ensure Atlantic Canadian firms have access to funding to support their growth. | DG Programs (HO), supported by DG OPS | July 31, 2020 |
Recommendation 2: Examine AIF funding in relation to the current needs of SMEs and priorities of the GC to confirm the strategic direction of the program and any changes to governance and funding processes. |
The Agency will undertake an examination of AIF funding in relation to the current needs of SMEs, other innovation programming, and the priorities of the GC. | DG Programs (HO), supported by DG OPS | July 31, 2020 |
Recommendation 3: Continue to support a strong Atlantic start-up ecosystem by strengthening the impact of BAIs through the development of clearer funding approaches linked to expected outcomes, and better coordination among stakeholders while recognizing unique regional and SME needs. |
Support the National Performance Measurement Framework for BAIs to ensure that organizations receiving ACOA funding are assessed according to standard criteria, consistent with national expectations, thus enabling the Agency to better link funding decisions to expected outcomes. Facilitate opportunities that continue to foster strengthened collaborations among regional stakeholders. |
DG Operations NS on behalf of the ACOA Start-up Champion, supported by DG Programs (HO) | March 31, 2021 |
Recommendation 4: Continue to strengthen programming priorities and collaborations to better address the labour and skills shortage in the region considering the growing demographic pressures and opportunities to enhance efforts on inclusive growth and immigration |
Continue ongoing efforts to engage federal, provincial, institutional and industry collaborators in pursuing strategies that improve the efficacy of the labour and skills ecosystem, which includes funding pilot projects, research into evolving trends and previous Agency expenditures, convening relevant stakeholders, and advocating the specific interests of Atlantic Canada to increase the quantity and quality of the labour force to meet current and future industry demands. | VP NB, supported by DG Programs (HO) | March 31, 2021 |
Appendix D: Innovation evaluation framework
Note: A bolded "L" refers to the leading line(s) of evidence. A small "s" refers to supporting line(s) of evidence.
Evaluation questions | Judgment criteria / Indicators | Existing information | New information | ||||||
---|---|---|---|---|---|---|---|---|---|
Literature Review | Document Review | Performance Data (Stats Can) | Performance Data (Internal) | Financial data (GX) | Interviews | Survey | Case Studies | ||
1. To what extent is the programming addressing a unique and demonstrable need for Atlantic Canadians? | |||||||||
|
|
L | L | s | s | s | |||
|
|
L | s | s | s | L | |||
2. To what extent is ACOA contributing to business growth in Atlantic Canada? | |||||||||
|
|
L | L | L | s | s | L | s | |
|
|
L | L | s | s | L | s | ||
|
|
L | s | s | s | s | L | L | |
3. To what extent is ACOA taking a client-centric approach to program design and implementation? | |||||||||
|
|
s | s | s | L | L | s | ||
|
|
s | L | s | L | s | L | ||
|
|
s | s | s | s | L | s | L | |
|
|
s | s | s | L | s | L |
Appendix E: Enterprise Development evaluation terms of reference overview
Appendix F: Evaluation advisory committee
ACOA's Head of Evaluation created and chaired an advisory committee to provide advice and guidance to the project team. The committee members commented on the evaluation framework, preliminary findings and reports. They also facilitated access to program data and provided advice at various stages of the evaluation process to maximize a clear reflection of the programming, targeting of specific information needs, and the usefulness of recommendations for decision making and programming improvement. The committee comprised ACOA Enterprise Development program directors; the director general of trade, investment and growth; the director of strategic policy development; the director of innovation and productivity; two external academic experts in regional economic development; and an evaluation expert from Canada Economic Development for Quebec Regions.