Content page

From [Institution name]

Evaluation of the Atlantic Canada Opportunities Agency’s Innovation Programming

2013-2014 to 2018-2019

Evaluation and Advisory Services

Atlantic Canada Opportunities Agency

January 2020

Acronyms

ACOA Atlantic Canada Opportunities Agency

AIF Atlantic Innovation Fund

BAIs business accelerators and incubators

BDC Business Development Bank of Canada

BDP Business Development Program

DRF departmental results framework

EAC evaluation advisory committee

EAS Evaluation and Advisory Services

ED Enterprise Development

FAA Federal Accountability Act

FTE full-time equivalent

GAC Global Affairs Canada

GBA+ gender-based analysis plus

GC Government of Canada

G&Cs grants and contributions

IRCC Immigration, Refugees and Citizenship Canada

ISED Innovation, Science and Economic Development

MAP management action plan

NRC-IRAP National Research Council of Canada -– Industrial Research Assistance Program

O&M operations and maintenance

PAF project approval form

REGI Regional Economic Growth through Innovation

RDA regional development agency

R&D research and development

SMEs small and medium-sized enterprises

TBS Treasury Board Secretariat

Figures

• Figure 1: Client survey – Impacts of no ACOA support
• Figure 2: Client survey – Organizations involved in projects beyond ACOA
• Figure 3: Client survey – Satisfaction with ACOA service features

Tables

• Table 1: Overview of ACOA programming
• Table 2: Annual Innovation programming expenditures by fiscal year, 2013 to 2019
• Table 3: Implementation context
• Table 4: Evaluation questions
• Table 5: Summary of data collection methods
• Table 6: Evaluation limitations and mitigation strategies
• Table 7: Description of major funding for innovation

Acknowledgements

This evaluation provides the Atlantic Canada Opportunities Agency's (ACOA) management with objective, neutral evidence on the relevance and performance of the Agency's Innovation programming. ACOA's Evaluation and Advisory Services unit completed this evaluation of Innovation programming with advice from an evaluation advisory committee (EAC).

I offer deep gratitude to the members of the EAC for their advice and support throughout the study. Their contributions helped to ensure the relevance and usefulness of the evaluation. Very special thanks go to the external members of the EAC: Matt MacPhee, Senior Project Manager, Saint Mary's University Entrepreneurship Centre; Beth Mason, CEO, Verschuren Centre, Cape Breton University; and, Maurice Turcot, Senior Evaluation Analyst, Canada Economic Development for Quebec Regions. Internal members of the committee include Joe Cashin, Michael Dillon, Kalie Hatt-Kilburn, Josh Jenkins, Anne-Marie Léger, Jeff Mullen, Hélène Robichaud, Daniel Scholten and Karl Tee.

The evaluation team is grateful to the external key informants as well as the many ACOA employees from across Atlantic Canada who provided their time and knowledge in support of this evaluation.

Finally, to the evaluation team members – Deanna Slattery-Doiron, Anouk Utzschneider, Nicole Saulnier, Jasmine Arsenault and Gaétanne Kerry – your dedication, professionalism and commitment to excellence is exceptional. Thanks to each of you.

Colleen Goggin

Director, Evaluation and Advisory Services

Summary

The purpose of this evaluation is to assess the relevance and performance of ACOA's Innovation programming over a six-year period, from 2013-2014 to 2018-2019. The evaluation fulfills Government of Canada (GC) accountability requirements and provides the Agency's management with systematic, neutral evidence to support continuous program improvement. The methodology included interviews with 42 key informants, a client survey with an overall response rate of 48%, two case studies, a document and literature review, and an analysis of available performance data.

ACOA's investments through Innovation programming aim to create a robust regional innovation ecosystem and accelerate the growth of small and medium-sized enterprises (SMEs) through the delivery of both financial and non-financial supports. Projects and initiatives focus on the following objectives: capacity for research and development (R&D), commercialization, productivity and business skills, and the attraction and retention of talent; development and demonstration of new or improved technologies, products, processes or services; and collaboration on innovative technology and commercialization projects.

Over the evaluation period, ACOA supported more than 850 innovation projects across Atlantic Canada through grant and contribution (G&C) expenditures and non-financial supports. The Agency expended $508 million, with 91% of this spending through G&Cs and the remainder to support operational delivery costs.

To what extent does the programming address a unique and demonstrable need for Atlantic Canadians?

The evaluation finds that there is a strong and continued need for ACOA's Innovation programming. Challenges to business growth exist to the same degree or are greater than reported in previous evaluations. In particular, the evaluation reports the following: a shortage of skills and labour influenced by demographic shifts; a need for greater awareness and use of innovative technologies; and a perceived need for greater access to specific types of capital (i.e., early-stage, growth stage and angel and venture capital). The program delivery model, with local presence as well as convening and pathfinding roles, allows the Agency to be responsive to the diverse and changing economic needs across the region. ACOA has developed or strengthened collaborations to better address immigration, skills and labour, and funding.

The Innovation programming is unique, complementary to other programs, and aligned with current federal government priorities on innovation and business growth. It is clearly aligned with ACOA's mandate to lead economic development in Atlantic Canada through support to SMEs. The programming offers unique features, including flexible, non-dilutive funding options and non-financial supports focused on promoting the growth of SMEs in the region.

To what extent is ACOA contributing to business growth in Atlantic Canada?

The evaluation finds that projects have achieved the expected outcomes of the Innovation programming. Over the evaluation period, ACOA provided $463 million to commercial and non-commercial clients to support the growth of SMEs through Innovation programming. ACOA's funding and other supports contribute to a strengthened innovation ecosystem, the commercialization of new or improved technologies, products, processes or services, and productivity improvements. Statistics Canada data reveals that ACOA clients report higher R&D and productivity compared to non-supported firms. In a survey of all ACOA Enterprise Development (ED) clients, 77% reported that projects contributed to commercialization; 75% indicated that their projects improved the efficiency or capacity of operations; and 67% said that it involved technology acquisition, adaptation or adoption, new equipment or renovations. Key informants, case studies and a document review indicated that ACOA contributes to a strengthened ecosystem, especially through funding to business accelerators and incubators (BAIs) and the important role the Agency plays in developing and maintaining collaborations, pathfinding and convening.

While there are opportunities to improve the ability to capture and report on longer-term impacts of programming, some initiatives currently under way hold promise for improving the quality and availability of performance data. These initiatives include the implementation of a new G&C information system and agreements with Revenue Canada and Statistics Canada, as well as the development of a new standardized national performance framework for BAIs.

The absence of programming would have had a negative impact on the achievement of results. Ninety-one percent of ACOA clients said the absence of ACOA funding would have had major negative repercussions on their project. Key informants and a literature review revealed that the start-up ecosystem (BAIs) would be weaker without ACOA supports.

To what extent is ACOA taking a client-centric approach to program design and implementation?

The evaluation found that the design and delivery of Innovation programming contributes to a client-centric approach to some extent. ACOA employs mechanisms to focus on client needs, including the provision of financial and non-financial supports, program flexibility and delivery model, internal coordination on priorities, partnerships, and collaborations that enable leveraging of funding. In a survey of ACOA clients, 83% of innovation-related clients reported that ACOA's non-financial supports are important.

Some opportunities exist to take a more strategic approach to program delivery and risk. Some factors impede client focus to some degree, including program uncertainty during the evaluation period (e.g., the transition to new innovation programming); ACOA's risk tolerance; aspects of internal and external coordination; and broad economic challenges (e.g., demographics, capital and business culture). A national review of federal innovation programming led to the implementation of new programming (i.e., the Regional Economic Growth through Innovation [REGI] program). Early implementation of REGI has been positive with no major impacts on the ability to fund projects and improved internal clarity on eligibility. However, key informants suggest gaps in the spectrum of federal funding may exist and are influenced by the unique needs in Atlantic Canada. Key informants suggested that coordination with federal partners may decrease the risks of funding gaps.

ACOA contributes to increased collaboration among stakeholders and clients. The Agency's expanded and more deliberate role in convening and pathfinding has developed new partnerships to address changing priorities. In particular, ACOA support enables projects to leverage substantial funds from other organizations at a level consistent with previous evaluations. ACOA also funds certain non-commercial organizations (e.g., Springboard, BioNova, BAIs) that encourage collaboration and business innovation. A document review and key informants indicated that recent efforts to improve coordination among BAIs are positive and must continue. However, key informants also indicated that ACOA should take a more coordinated and consistent approach to funding BAIs to improve their ability to pivot strategically, collaborate and avoid duplication of efforts.

Innovation programming contributes to inclusive growth to some extent. Inclusive growth includes promoting the participation of diverse groups and rural communities to support economic development in the region. The Agency made some efforts toward inclusive growth through funding and collaborations (e.g., immigrants, Indigenous people and women). However, there is growing internal recognition of the need to better support under-represented groups to facilitate economic growth, address skills and labour gaps, and better align with government priorities. Key informants also indicated that ACOA's 28 points of service across the region position the Agency to meet the unique needs of rural businesses.

The evaluation found that ACOA delivers Innovation programming in a cost-effective manner. Internal costs were proportionate to the amount of funding delivered. There was a high degree of client satisfaction with ACOA's service features. Recent internal operational changes are favouring more efficient program delivery (e.g., project approval form [PAF], streamlined approval process, and planned client information management system). Key informants and a review of data identified questions related to efficiency, including project approval times, performance data and ACOA's risk tolerance levels.

Recommendations

Based on the findings and conclusions, the evaluation makes three recommendations for senior management's consideration.

Recommendation 1: Build upon current Agency efforts to address any gaps in funding that may impact the scale up and growth of Atlantic Canadian firms, including through coordination and advocacy with existing and new partners.

Recommendation 2: Examine AIF funding in relation to the current needs of SMEs and the priorities of the GC to confirm the strategic direction for the program and any changes to governance and funding processes.

Recommendation 3: Continue to support a strong Atlantic start-up ecosystem by strengthening the impact of BAIs through the development of clearer funding approaches linked to expected outcomes and better coordination among stakeholders while recognizing unique regional and SME needs.

Recommendation 4: Continue to strengthen programming priorities and collaborations to better address the labour and skills shortage in the region considering the growing demographic pressures and opportunities to enhance efforts on inclusive growth and immigration.

1.0 Introduction

This report presents the results of the evaluation of ACOA's Innovation programming. The programming supports innovation among SMEs in Atlantic Canada through direct funding and non-financial supports. The evaluation builds on the recent national review of innovation programming conducted by the Treasury Board Secretariat (TBS) to support decision making and the monitoring of this critical element of ACOA's portfolio.

Appendices contain additional information: a one-page summary of findings, conclusions and recommendations (Appendix B); the management action plan (MAP) (Appendix C); the evaluation analytical framework (Appendix D); and the evaluation terms of reference (Appendix E).

2.0 Innovation programming

ACOA recognizes the importance of innovation for business growth. Its Innovation programming is designed to help grow and scale SMEs through the delivery of funding and other supports to increase:

• capacity for R&D, commercialization, productivity and business skills, and the attraction and retention of talent;
• the development and demonstration of new or improved technologies, products, processes, or services; and
• collaboration on innovative technology and commercialization projects.

A logic model that describes the links between program activities, outputs and expected outcomes can be found in Appendix A.

2.1 Delivery approach

ACOA's Innovation programming combines funding with non-financial supports such as advice, help with convening partners and leveraging other sources of funding. The programming is sustained financially by three G&C programs: the Atlantic Innovation Fund (AIF), the Business Development Program (BDP) and the REGI program.

The primary clients for Innovation programming are SMEs in Atlantic Canada. In addition, non-commercial clients include industry associations, BAIs, educational and research institutions, and other non-profit organizations.

The program is delivered through a continuous intake model, meaning that proponents submit project applications to ACOA at any point throughout the year. Agency program officers in regional offices across Atlantic Canada work with clients to develop projects, including identifying and convening partners, leveraging other sources of funding, and providing advice and guidance. They evaluate applications for G&Cs and manage approved projects through to the delivery of results. Table 1 provides details on the priorities and processes of each G&C program.

2.2 Program alignment and financial resources

All federal departments must have a DRF that outlines their core responsibilities, expected results and programs. Innovation programming is part of ACOA's DRF under the "technologies" pillar, with the expected result that "businesses invest in the development and commercialization of innovation technologies in Atlantic Canada." For more information on the Agency's DRF, the program inventory and performance indicators see ACOA's DRF and program inventory of record as presented in the Agency's 2018-19 Departmental Plan.

Innovation programming expenditures include the three G&C funding programs (e.g., AIF, BDP and REGI) as well as operations and maintenance (O&M). As shown in Table 2, between April 1, 2013 and March 31, 2019, the Agency invested almost $463 million of G&C funding in 850 projects. The Agency also expended a further $45 million of O&M on salaries (93%) and on travel and other operating expenses (7%). In 2018-2019, there were approximately 70 full-time equivalent employees at ACOA directly delivering Innovation programming.

Source: ACOA GX financial database (extracted May 10, 2019).

2.3 Program implementation context

A number of contextual changes affected the delivery of the Innovation programming over time. Table 3 summarizes these changes and their impacts on the programming over the evaluation period. Most important, innovation became a greater priority of the federal government, with a national review of innovation programming across departments and the implementation of the new REGI program.

3.0 Evaluation approach

3.2 Data collection methods

The evaluation used both qualitative and quantitative data collection methods (Table 5) to address the evaluation questions. The choice of methods was determined based on their relevance and reliability, data availability and costs.

3.3 Evaluation strengths and limitations

The evaluation was designed to have a calibrated approach, using mixed methods to identify strong findings while meeting TBS timelines and providing useful information to meet senior management needs. The evaluation has a number of strengths, including a skilled internal evaluation team and high stakeholder engagement through an EAC (see Appendix F) and senior management consultations. The evaluation includes detailed case studies to examine needs and longer-term impacts. The online client survey received a high response rate (48%). The evaluation considered GBA+ in its design and implementation of data collection methods, as well as in the synthesis of findings.

These practices helped to mitigate the common limitations that occur as part of most program evaluations. As outlined in Table 6, the evaluation team considered the limitations of the study and implemented a number of mitigation strategies.

4.0 Findings

4.1 To what extent is the programming addressing a unique and demonstrable need for Atlantic Canadians?

Finding 1: Skills and labour, use of innovative technologies and access to capital are core issues in Atlantic Canada. Needs exist to the same degree or are greater than were reported in the previous evaluation due to increasing demographic pressures.

The results of the evaluation demonstrate a strong, continued need for ACOA's Innovation programming. Innovation is a priority of the federal government and is important for regional economic growth, especially in the context of major demographic and labour market challenges.

Atlantic Canada has been experiencing a well-documented and growing shortage of skills and labour for several decades, which acts as a barrier to the innovation, productivity and growth of SMEs. A review of recent documents and literature reveals that the attraction and retention of talent is expected to continue and be compounded by factors such as out-migration and Atlantic Canada's aging population. A Business Development Bank of Canada (BDC) survey of 1,208 entrepreneurs from across the country showed that hiring employees is more challenging for Atlantic Canadian entrepreneurs than for those in other regions of Canada. In fact, when compared with other regions of the country, Atlantic Canada had the most (50%) entrepreneurs who reported difficulties hiring in the last 12 months, compared with a national average of 39%. Almost two thirds (62%) expect to face a shortage over the next five years.^{Footnote (4)}

The extent of labour shortages in the Atlantic region is surprising because it also has a higher unemployment rate than the Canadian average. Factors contributing to this include the region's older population, lower labour force participation rate and higher share of seasonal workers.^{Footnote (5)} Recent internal analyses suggest that both the labour shortage and the unemployment rate are even more pronounced in rural areas such as Cape Breton (Nova Scotia), northern New Brunswick, and many areas of Newfoundland and Labrador.^{Footnote (6)} The high unemployment rate in some rural areas can be explained by the mismatch between the positions offered and the available labour force (e.g., skills, mobility, etc.).

The need for talent relates to both skilled and unskilled labour, and also includes a gap in business management skills. There is a specific challenge in attracting senior leadership with the skills required to scale up and grow companies beyond the start-up phase. In addition, key informants, the client survey and the literature review^{Footnote (7)} underlined that workers with skills in sales (particularly in the technology sector), high-tech/computing, marketing and export are in high demand and difficult to find.

The Atlantic Canadian business landscape is characterized by a need for greater awareness and the use of innovative technologies to improve productivity and growth. There are opportunities to undertake more adaptation and adoption of technology as well as more automation^{Footnote (8)} to address the labour shortage.  Survey results indicate that clients recognize this opportunity: among those clients who indicated that recruiting and retaining employees was a need, 71% indicated their project involved technology acquisition, adaptation or adoption, new equipment or renovations. However, key informants underlined that a risk-averse business culture in Atlantic Canada tends to restrain the use of innovative technologies and challenges the promotion of a growth mindset among entrepreneurs.

Access to funding continues to be a need for Atlantic Canadian firms. Through the ED client survey, 52% (n=493) of ACOA's innovation-related clients indicated that obtaining financing was a key need.  Case studies supported the need for more early-stage financing for start-ups and more angel and venture capital for high-risk ventures in Atlantic Canada. Many key informants underlined that provincial programs in Atlantic Canada do not provide significant direct financial support to SMEs and that this support has declined over the past years.

Some key informants also identified a need for greater access to capital, particularly in the early and growth stages. The perceived gaps were tied to a lack of both private and public-sector investments in Atlantic Canada. A number of key informants suggested that some SMEs might have difficulties accessing large amounts from federal departments such as ACOA and the National Research Council of Canada – Industrial Research Assistance Program (NRC-IRAP) due, at least in part, to a lack of available funding. They noted that access to funding can be particularly challenging for firms with growth trajectories requiring the adoption of very expensive advanced technology or equipment. These challenges speak to the need for the funding continuum available for Atlantic firms to function in a seamless, client-centered way that allows firms access to the financing required to effectively execute their growth plans.

Finding 2: Largely influenced by the program delivery approach, featuring a local presence and relationships with SMEs and partners, the Agency is aware of the varied needs that exist across the region. The Agency has taken some actions to respond to these newer and emerging needs.

• The need for greater awareness and the use of innovative technologies. The Agency has increased its focus on advanced manufacturing by providing targeted training to internal employees dealing directly with clients and by funding technology missions for entrepreneurs to learn about new technologies that could lead to increased productivity. A review of recent internal analyses reveals that ACOA's funding for larger-scale automation projects has almost tripled between 2016 and 2019 from $35 million to $95 million. ACOA is also an active participant in the Accelerated Growth Service, an initiative that facilitates a more coordinated approach to supporting established firms with high growth potential. Moreover, the Agency maintains a strong relationship with the NRC-IRAP, an organization dedicated to providing advice, connections and funding to help SMEs drive growth and innovation through applied R&D of technology.
• The lack of access to capital. ACOA has a flexible approach to providing funding to support specific economic needs across the region; it provided $463 million to innovative Atlantic Canadian companies and to the innovation ecosystem over the period covered by this evaluation. ACOA plays a strong convening and pathfinding role, which helps entrepreneurs access other sources of funding and support.

Finding 3: While ACOA's Innovation programming is unique and complementary to other programming offered in the region, there may be gaps in the funding continuum in Atlantic Canada.

ACOA's programming complements other types of programming and funding available to SMEs and intermediaries for innovation and growth activities. Funding and services for SMEs and other organizations come in a variety of arrangements across the growth spectrum from both government and non-government organizations (Table 7). The main features that set ACOA's programming apart from that of other sources are: mandate to support economic development in Atlantic Canada, focus on SMEs, and flexible financial and non-financial supports.

Finding 4: ACOA's Innovation programming is aligned with current federal and ACOA strategic priorities and contributes to business growth in Atlantic Canada through participation in various initiatives.

ACOA's Innovation programming aligns with the overall federal government objective of helping "Canadian businesses grow, innovate and export so that they can create good quality jobs and wealth for Canadians" as outlined in the mandate letter for the Minister of Innovation, Science and Economic Development (2015). More specific objectives are outlined in several federal priority documents:

• The mandate letter for the Minister of Innovation, Science and Economic Development (ISED) prioritized the development of an Innovation Agenda that includes expanding support to BAIs, networks and clusters, and working with RDAs to invest in competitive regional advantages.
• Budget 2016 announced plans to support BAIs, innovation networks, clusters and a tailored approach to coordinating federal services to meet client needs.^{Footnote (24)}
• Budget 2017 announced the federal Innovation and Skills Plan (2017),^{Footnote (25)} and Budget 2018 announced that it would be delivered by ACOA in the Atlantic region, with support for women entrepreneurs through the Women Entrepreneurship Strategy and support for skills development and economic diversification to help with the transition to a low-carbon economy.^{Footnote (26)}
• Budget 2018 outlined the new superclusters initiative, including maximizing Canada's ocean economy in Atlantic Canada.^{Footnote (27)} The consolidation of the Accelerated Growth Service and the IRAP Concierge Service was also announced.^{Footnote (28)}

ACOA's Innovation programming aligns with expected results outlined in its DRF under the technology pillar, which aims at seeing businesses investing in the development and commercialization of innovative technologies in Atlantic Canada. A national review of innovation programming announced in Budget 2017^{Footnote (29)} led to revised programming characterized by a stronger focus on innovation, which is a key priority for the federal government and ACOA.

4.2 To what extent is ACOA contributing to business growth in Atlantic Canada?

Finding 5: ACOA's Innovation programming contributes to the growth of new and existing businesses through its supports to the innovation ecosystem, commercialization and productivity improvements.

Overall, this study found that Innovation programming contributes to the achievement of expected outcomes and, ultimately, to the growth of SMEs in Atlantic Canada. ACOA's flexible, non-dilutive funding, delivery model, strategic and collaborative approach, and non-financial supports (e.g., convening and pathfinding) facilitate the achievement of results.

ACOA's funding and other supports contribute to the scale-up and growth of SMEs and a strengthened innovation ecosystem through the commercialization of new or improved technologies, products, processes or services, and productivity improvements. Over the evaluation period, the Agency provided $463 million to commercial and non-commercial clients toward these expected outcomes. For example, internal data shows that the Agency invested $34 million in early-stage SMEs with high potential for growth, universities, and other not-for-profit organizations that provide supports directly to SMEs, such as BAIs, industry associations, and economic development organizations. These projects focused on innovation and commercialization, marketing, scale up, productivity improvements and the development of business skills.^{Footnote (30)}

A complex and changing environment, both internally and externally, affects the achievement of program outcomes. As outlined in section 4.1, the three core needs in the region (e.g., skills and labour, awareness and use of innovative technologies, and access to capital) coupled with the region's risk-averse business culture negatively impact the achievement of results. For example, key informants pointed to the need to support and encourage a growth mindset among business owners, due to a reluctance to take on riskier projects that may contribute to growth. The Agency's use of provisionally repayable contributions aid clients in undertaking these riskier projects, as businesses need access to more risk capital that is not available through traditional lenders.

Other factors, more closely within the control of ACOA, were also raised through key informant interviews and published reports as challenging the achievement of outcomes.  

• A complex and dispersed innovation ecosystem leads to gaps in assistance offered to SMEs and to confusion within the BAI ecosystem.
• Uncertainty around the direction of AIF programming in light of recent program reviews and the implementation of new programming.
• Certain internal structures that make program delivery more complex (e.g., inconsistency in funding approaches between regions, lack of coordination between funding programs).
• Gaps in performance information, including reach and impact on vulnerable populations. Several initiatives are under way, however, and are expected to improve performance data across programming (e.g., implementation of a new project information management system and the recent establishment of standard performance indicators for BAIs in Canada).^{Footnote (35)}

Finding 6: The absence of programming would have had a major negative impact on the achievement of outcomes.

ACOA's financial investments directly lead to the achievement of project results and allow projects to proceed as planned, including through the leveraging of other funding. Case studies, key informant interviews and client survey results reflect that ACOA supports have a major impact on the Atlantic Canadian innovation ecosystem.

ACOA's support facilitates leveraging of other sources of funding. Case studies indicate that ACOA's involvement and investments have a major influence on leveraging from other sources, as the Agency is perceived as having solid project risk assessment, monitoring and reporting processes.

The need for ACOA supports exists to at least the same extent as it did in the previous evaluation of the programming, completed in 2015. The importance of ACOA's contributions is supported by the results of the client survey with 91% (n=496) of innovation-related clients indicating that their project would have experienced a major negative impact without ACOA's support. As detailed in Figure 1, if the projects had still gone ahead, most innovation-related clients would have experienced delays or projects with a smaller scope. Case study results indicate that a majority of BAIs in the region, important actors in the innovation ecosystem, would not exist without ACOA support.

4.3 To what extent is ACOA taking a client-centric approach to program design and implementation?

Finding 7:  The design and delivery of Innovation programming supports a client-centric approach to some extent. Opportunities exist to take a more strategic approach to program delivery and portfolio risk.

Recent changes to program funding mechanisms are perceived as positive for the ecosystem. According to key informants, the implementation of the REGI program in 2018-2019 was relatively seamless and did not create issues for clients or for ACOA staff. Furthermore, key informants reported improved clarity around eligibility of projects without affecting the ability to fund projects flexibly. However, program changes over the evaluation period raised questions about the role of the AIF. Given new programming and other factors, some key informants identified the need to rethink the AIF's funding objectives and mechanisms to address possible overlap with REGI and gaps in client needs.

While ACOA's supports for the innovation ecosystem correspond to client needs, some gaps continue to exist in coordination and communication. Many key informants noted the important work that ACOA has done to improve BAI coordination in recent years. They also stated that ACOA could stand a more coordinated and consistent approach to funding BAIs to improve their ability to pivot strategically, collaborate and avoid duplication of efforts. They also stated that better coordination, collaboration and knowledge sharing between regional offices is needed while balancing the need for flexibility to meet different needs across the region.

Some ACOA key informants noted that recent internal coordination mechanisms (e.g., champion files) facilitate pan-Atlantic collaboration between ACOA's offices and improve understanding of client needs. Some suggested that ACOA's existing governance structures do not always promote the more strategic, matrix approaches. Internal key informants also report that there is a need to clarify language and definitions of certain key themes related to newer priorities to promote better cohesion and understanding among ACOA staff (e.g., innovation, start-up, scale up).

ACOA's approach to risk will be examined more closely in the second report resulting from this evaluation – the Evaluation of ACOA's Business Growth and Trade Programming – expected to be finalized in May 2020.

Finding 8: ACOA contributes to increased collaboration among stakeholders and clients. However, opportunities exist to improve coordination and collaboration within the innovation ecosystem.

ACOA's non-financial supports contribute to meeting client needs. Most surveyed innovation-related clients (89%; n=470) indicated satisfaction with the business knowledge and advice offered by ACOA. Key informants stated that the Agency has worked to increase the capacity of its staff in areas that are critical for the region, such as advanced manufacturing, automation and artificial intelligence. They also underlined that ACOA's convening role, including through traditional project supports and through the Accelerated Growth Service, was key in supporting innovation and growth.

ACOA is playing a more deliberate convening and pathfinding role since the previous evaluation of the programming. In addition to important funding, the Agency provides non-financial support such as advice, guidance, expertise or assistance to clients and communities. Innovation-related client survey results revealed that this type of support is considered important by 83% (n=495) of respondents. Some clients explained that ACOA helps them connect with other players and provides visibility to their project, while others said that the Agency's personnel made their company aware of funding opportunities, which led to obtaining additional capital. Key informants and case studies indicated that a more formalized approach to convening and pathfinding roles is also taking place through the Accelerated Growth Service and coordination of BAIs. As detailed in Figure 2, a variety of partners were involved (either through provision of financial or in-kind support) in ACOA-assisted projects (n=291)

Key informants, case studies and published reports (e.g., BAI Performance Measurement Framework by the Deep Centre)^{Footnote (37)} point to the growing engagement of ACOA with other federal partners such as Employment and Social Development Canada, the Department of Fisheries and Oceans Canada, Global Affairs Canada and IRCC (through the Atlantic Immigration Pilot). ACOA's support to the innovation ecosystem through BAIs also promotes collaboration within the innovation ecosystem itself. BAIs play a key role in providing space for collaboration and networking among entrepreneurs, as revealed through the case studies. In some regions, co-location between ACOA and the NRC-IRAP was also underlined as a best practice, improving collaboration and complementarity between the two organizations.

Finding 9: There is a perception that programs are open to all equally, but there is growing recognition that there are opportunities to better support specific groups.

While ACOA has long delivered programming for particular population groups, it has paid increasing attention to certain vulnerable populations over the evaluation period. ACOA is particularly strong in delivering programming to meet the needs of rural populations and has made strides to increase efforts for new immigrants, Indigenous people and women. No specific focus on youth was noted through this evaluation.

On July 4, 2016, the Government of Canada and the four Atlantic Provinces launched the Atlantic Growth Strategy, which sought to drive long-term economic growth in the Atlantic region by implementing targeted, evidence-based actions under five priority areas.^{Footnote (39)} The Agency champions immigration mostly through activities in partnership with IRCC. ACOA's involvement in the Atlantic Immigration Pilot (2017-2020), a key initiative under the Atlantic Growth Strategy, helps address labour and skills needs and ensures long-term economic growth in the region by attracting and retaining skilled immigrants.^{Footnote (40)} The case studies also underlined that BAIs supported by the Agency provide supports to immigrant entrepreneurs, although retention remains a challenge. Based on Statistics Canada data, there is room for improvement, as immigrants owned a low share (5.2%) of ACOA client firms and comparable firms (5.1%) in 2015.^{Footnote (41)}

ACOA has recently prioritized Indigenous economic development through the creation of a pan-Atlantic priority file, but there remains an opportunity to provide more support for indigenous businesses. In Atlantic Canada in 2017, 1.1% of all businesses had indigenous ownership and that percentage has decreased compared to 2014. The Atlantic Province Economic Council Indigenous report underlines specific training needs related to networking, building and maintaining relationships with financial institutions, advice on business growth, access to funding, entering new markets, developing a business strategy, and finding and retaining skilled employees. Key informant interviews indicated that more could be done in terms of adapting ACOA's approach to Indigenous development, but acknowledged that efforts are being made through the Agency's Indigenous file. Some promising practices are emerging, however, such as recent efforts in Cape Breton to develop strong partnerships and relationships with Indigenous communities.

The programming also includes some initiatives for women in business. Key informants and case studies indicated that a number of initiatives are in place to increase women's presence in the business and innovation space. ISED's Women Entrepreneurship Strategy, a national $2 billion investment put in place through Budget 2018,^{Footnote (42)} aims to help women start or grow a business. Other initiatives are undertaken through the BAIs in Atlantic Canada, such as Genesis' Women in Tech Peer Group taking place in Newfoundland and Labrador.

Statistics Canada data and key informants suggested that further efforts would be merited. Women owned only 11.9% of ACOA-assisted firms in 2015 compared to 10.8% in 2010.^{Footnote (43)} In the innovation ecosystem, women-led start-ups receive a lower share of venture capital in Atlantic Canada compared with the rest of Canada.^{Footnote (44)} Despite this, between 2010 and 2015, female ACOA clients not only registered higher average growth rates than male ACOA clients in a number of employment and financial indicators but also outperformed female-owned businesses from the comparable group in most areas. Case studies indicated that there has been increased focus on supports to women in technology, computer science and entrepreneurship.

Atlantic Canada is highly rural and unique challenges exist for businesses outside urban centres. Key informants and key documents indicated that rural areas experience challenges, in particular related to access to risk capital for start-ups.^{Footnote (45)} However, key informants indicated that ACOA's 28 points of service, including many in rural communities, facilitate meeting specific client needs. The Agency's flexible approach also allows for support tailored to rural-based companies.

Finding 10: There is evidence to suggest that recent internal operational changes are supporting more efficient program delivery.

Delivery costs for ACOA's Innovation programming are comparable to amounts reported in previous evaluations. The total expenditures for Innovation programming included in this evaluation were $507.6 million. An analysis of internal financial data indicates that ACOA's cost to deliver Innovation programming has remained relatively unchanged over time, with a cost to deliver $1.00 of G&C funding of $0.097 over the period of this evaluation (2013-2014 to 2018-2019), compared with $0.092 as reported in the last evaluation (2007-2008 to 2012-2013).

There is a high degree of satisfaction with ACOA's service features. Key informants and case studies indicated that ACOA staff are proactive, efficient and generally helpful at all stages of the project life cycle, including completing forms and navigating the application and claims processes. They described ACOA as the cornerstone of many projects and stressed the importance of non-financial supports provided by ACOA staff. As detailed in Figure 3, the overwhelming majority of innovation-related client survey respondents indicated strong satisfaction with a range of ACOA's service features. They are very satisfied with the courteousness and professionalism of ACOA personnel (97%), the ongoing business relationship with ACOA personnel (94%), communication from ACOA personnel on the status of their request (93%), and the availability of ACOA personnel (93%).

Project approval times have improved, though some clients still see them as a challenge. When compared with other ACOA processes and service features, innovation-related client survey results revealed a slightly lower satisfaction rating for ease of the application process and paperwork (84%) and speed of turnaround time of the application process (82%). Internal key informants noted that recent operational changes focused on streamlining processes, including changing delegation levels, have improved approval times. The transition to the redesigned PAF also facilitated programming efficiency. In 2017, the Agency undertook a pilot project to examine potential areas for improving and streamlining project evaluation and approval processes, with a specific focus on commercial applications. The goals of this exercise were to minimize the burden on clients by expediting processing of applications and reducing wait times, and establishing service standards for specific steps in the approval process.

5.0 Conclusions

Overall, the evaluation found that ACOA's Innovation programming remains relevant and achieves expected outcomes related to economic development in Atlantic Canada. The delivery model is grounded in the Agency's deep knowledge of businesses and expertise in convening and pathfinding, which facilitates the achievement of results. There are some areas for attention to ensure ongoing and strengthened strategic investments through the programming. The evaluation concludes that:

1. Programming is relevant to the existing needs of SMEs, aligned with ACOA's mandate and GC priorities, and complements other supports in the region.
2. ACOA is aware of and has addressed changing needs to a certain extent. There are risks to its continued ability to address the needs of SMEs due to potential funding gaps, access to labour and skills, and uncertainties about AIF funding.
3. ACOA's Innovation programming contributes to the growth of new and existing businesses through its supports to the innovation ecosystem, commercialization and productivity improvements. ACOA plays an important role in developing and maintaining collaborations, pathfinding and convening. While there are opportunities to improve the ability to capture and report on longer-term impacts of the programming, the current implementation of a new project management system and other initiatives under way hold promise in improving the quality and availability of performance measurement data, including that related to BAIs.
4. The design and delivery of programming supports awareness of changing needs, a client-centric approach and the achievement of results. A strong regional presence allows for knowledge of needs, capacities and partners as well as the development of relationships across the ecosystem. The recent implementation of REGI appears to be positive. Opportunities exist to further improve internal coordination and to better address client needs, including those of under-represented groups.
5. ACOA has mechanisms in place to support the efficient delivery of programming to facilitate the achievement of outcomes. Opportunities exist to improve the direction on appropriate risk tolerance levels, to clarify the direction for the AIF, and to build upon existing internal and external coordination and collaboration efforts.^{Footnote (46)}

6.0 Recommendations

Within the context of the recent ACOA funding program changes, and the complexity that exists in the innovation ecosystem in Atlantic Canada, ACOA should:

Recommendation 1: Build upon current Agency efforts to address any gaps in funding that may impact the scale up and growth of Atlantic Canadian firms, including through coordination and advocacy with existing and new partners.

Recommendation 2: Examine AIF funding in relation to the current needs of SMEs and priorities of the Government of Canada to confirm the strategic direction of the program and any changes to governance and funding processes.

Recommendation 3: Continue to support a strong Atlantic start-up ecosystem by strengthening the impact of BAIs through the development of clearer funding approaches linked to expected outcomes and better coordination among stakeholders while recognizing unique regional and SME needs.

Recommendation 4: Continue to strengthen programming priorities and collaborations to better address the labour and skills shortage in the region considering the growing demographic pressures and opportunities to enhance efforts on inclusive growth and immigration.

Appendix B presents a one-page summary of key findings, conclusions and recommendations.

ACOA senior Programs management has agreed with the evaluation's recommendations. It has developed a MAP that details the actions the Agency will take to address each of the recommendations. Appendix C presents the MAP.

Appendix A: Innovation programming logic model

Appendix B: Summary of findings, conclusions and recommendations

Appendix C: Management action plan

Appendix D: Innovation evaluation framework

Note: A bolded "L" refers to the leading line(s) of evidence. A small "s" refers to supporting line(s) of evidence.

Appendix E: Enterprise Development evaluation terms of reference overview

Appendix F: Evaluation advisory committee

ACOA's Head of Evaluation created and chaired an advisory committee to provide advice and guidance to the project team. The committee members commented on the evaluation framework, preliminary findings and reports. They also facilitated access to program data and provided advice at various stages of the evaluation process to maximize a clear reflection of the programming, targeting of specific information needs, and the usefulness of recommendations for decision making and programming improvement. The committee comprised ACOA Enterprise Development program directors; the director general of trade, investment and growth; the director of strategic policy development; the director of innovation and productivity; two external academic experts in regional economic development; and an evaluation expert from Canada Economic Development for Quebec Regions.