Jump to: navigation, search

Eventlet-removal

Revision as of 17:43, 5 November 2024 by Herveberaud.pro (talk | contribs) (→‎Teams efforts)

Contents

Why Eventlet Needs to Be Removed

Eventlet, though once popular for concurrent execution using "greenthreads," now presents several limitations and issues that make it an outdated solution for modern projects like OpenStack:

1. Incompatibility with Recent Versions of Python:

Eventlet struggles to keep up with the evolution of Python, particularly with recent versions (such as Python 3.12 and beyond). Some of its internal functionalities, such as "monkey patching" standard libraries, no longer work correctly or cause unexpected behavior. This limits projects from adopting the latest improvements in Python, including critical security patches.

2. Inefficient Resource and Thread Management:

The greenthreads used by Eventlet can lead to excessive memory consumption and poor resource management, especially under heavy loads or in large-scale applications. Given that OpenStack operates in a distributed and highly parallel environment, this can become a significant bottleneck for overall performance.

3. Maintenance and Security Issues:

Eventlet suffers from a lack of active maintenance and does not receive regular updates to address security vulnerabilities and compatibility issues. This exposes OpenStack to potential security risks if flaws in Eventlet are not addressed promptly.

4. Outdated Approach to Concurrency Programming:

With the emergence of more modern solutions like `asyncio` or `concurrent.futures`, which are natively integrated into Python, using Eventlet feels outdated. These newer libraries offer better concurrency management, improved performance, and tighter integration with the current Python ecosystem.

5. Inability to Adopt New Linux Distributions:

Many modern Linux distributions, such as Ubuntu, RHEL, and others, now ship with newer versions of Python that are not supported by Eventlet. Without migrating away from Eventlet, OpenStack will face challenges in adopting these newer distributions. This will limit the ability to stay current with the latest infrastructure improvements and security enhancements provided by modern operating systems.

Potential Impacts if the Migration is Not Completed

Failure to migrate away from Eventlet could have significant consequences for OpenStack-based projects:

1. Blockage of Python Updates:

Continuing to use Eventlet may cause OpenStack to be stuck on older versions of Python. This prevents the adoption of new features, performance improvements, and —most importantly—security patches offered by newer versions of Python.

2. Security Vulnerabilities Risk:

Given Eventlet's lack of maintenance, unresolved security flaws could leave OpenStack systems vulnerable to attacks. Failing to migrate to modern solutions exposes the infrastructure to major security risks.

3. Degradation of Performance and Scalability:

OpenStack handles large workloads in distributed environments, and Eventlet’s inefficient resource management can cause slowdowns and failures. Without migration, this could limit OpenStack's ability to scale smoothly and handle increasingly larger data volumes.

4. Inability to Adopt New Linux Distributions:

If the migration from Eventlet is not completed, OpenStack will struggle to support modern Linux distributions that come with newer versions of Python. Distributions like Ubuntu and RHEL now ship with versions of Python that are incompatible with Eventlet. This will hinder the ability to upgrade to these distributions, which are critical for staying up-to-date with the latest security fixes and performance improvements.

5. Increased Maintenance Costs:

Continuing to rely on an outdated technology will increase maintenance costs as it becomes harder to find fixes or workarounds for technical issues. This will lead to a growing technical debt that will be difficult to manage in the long term.

To ensure the longevity, stability, and security of OpenStack, it is crucial to migrate to modern solutions like `asyncio` or `concurrent.futures`. This will not only overcome Eventlet's current limitations but also ensure that OpenStack remains capable of meeting performance, security, and scalability demands. Moreover, migrating is essential for supporting the latest Linux distributions like Ubuntu and RHEL, which include newer versions of Python that are not compatible with Eventlet.

The following sections aims to guide you during your migration journey.

Migration's Milestones

  • Adapting common libraries (Oslo, etc...) - e.g: introducing an Async Engine Facade to oslo.db, adapting oslo.service to not relying on Eventlet;
  • Services migration - removing Eventlet usages from OpenStack services;
  • Common libraries migration - common libraries depends on Eventlet because at a higher level - at the services level - Eventlet is required, we cannot remove this kind of Eventlet occurence from the common libraries while services are not fully migrated;
  • Removing Eventlet from our global requirements;
  • Abandoning Eventlet.

Releases

Target Releases

Fully removing Eventlet from OpenStack will surely take several series, we target, at least, the H or I series.

Workload Regulation

  • SLURP releases can be used to introduce "freeze" periods where development slows down to focus on stability, documentation, and training. By example, SLURP releases could used to slow down our migration activity, and to be more focused on observation and documentation. SLURP releases can be also used to deprecate things that need to be deprecated or to remove useless workaround related to Eventlet;
  • use non SLURP releases to introduce breaking changes. E.g, such a replacement of Eventlet WSGI by the selected alternative (see the specialized working groups for further details about breaking changes)

Save the Date

List of the coming events related to the Eventlet removal that could interest the reader.

Recurring Events

One hour OFTC IRC meeting (#opentack-eventlet-removal) each first and third week of each month.

First meeting will be forced to 4pm UTC at Tuesday Nov 5th and then moved to the selected timeslot for the following meetings, see the poll below.

Here is the etherpad to tracks meeting https://etherpad.opendev.org/p/epoxy-eventlet-tracking. If there's a topic you think warrants to be discussed during our meeting, please add it to the agenda at the "Open Discussion" section and at the related date.

For the other coming meetings, let's find together the right UTC time slot for our one hour OFTC IRC meeting. Meetings will scheduled each first and third <selected time slot in poll> of each month. Votes will be considered as closed by Nov. 8 Vote here: https://framadate.org/rUghO3j6ZkRhKD7m

Proposed dates are just a way to select a recurrent time slot. The dates exposed in the poll do not matter, just the day and the hour of the week matter. Dates are just their because framadate or doodle require specific dates, but date do not reflect the next meeting date. Again, meetings will be scheduled each first and third week of each month at the selected time slot.

Thanks in advance for your votes.

Join us on the #opentack-eventlet-removal OFTC IRC channel.

Cross Project Events

No events are currently scheduled.

Project-specific events

If your team or project schedule a specific event related to the Eventlet removal initiative do not hesitate to share your dates with the community.

Here are the coming project-specific events: - <none>

Want to Discuss?

#eventlet-removal

Follow the #eventlet-removal tag. It will be used everywhere to tag all the related things to this topic:

  • gerrit patches
  • mailing list threads
  • OpenStack OFTC IRC channel (#openstack-eventlet-removal)
  • ...

The Pop-up team

This community goal is going to be complex, for this reason It is imperative to take account of its impact on our community. That's the role of the pop-up team. We need a way to manage the community, and we need to establish measures which aim to improve the working condition of our community. That's the role of this pop-up team to provide such measures and conditions, while guaranteeing a good management of the community around this topic.

Our mantras

Our mantras are way to lead our decisions related to this community goal:

  1. It is crucial to avoid having multiple ways to manage things. A unified approach will simplify maintenance and enhance user experience.
  2. This is a community goal, so all community voices count. People should look at this goal with a global overview and not only with a team or a project centric point of view.
  3. Our decision should be driven by sustainability. We are in a way to drop a black box, we should not replace it by another black box.
  4. If a feature is by default included in the stdlib, then we must avoid relying on an esoteric third party library that provide the same kind of feature.

Our pillars

The elements below present the different perspectives that the pop-up should embody.

Collaborative governance

  • Establish consensus-based governance where major decisions are voted on after public consultations via proposals (blueprints, specs) or kind of RFCs;
  • Form a steering committee with representatives from different technical teams (nova, neutron, glance, swift, oslo, etc.) to ensure a balanced vision and prevent biases.

Code of conduct

Develop a code of conduct that defines respectful, inclusive, and productive collaboration rules. Enforce sanctions for harmful or disrespectful behavior to protect the integrity of the community.

Specialized working groups

Set up specialized working groups to choose and select the specific technical details, for example which python server to use, which library to use (FastAPI vs Flask, HTTPX vs Requests vs AIOHttp, etc...). For now we need working groups specialized:

  • HTTP decisions (server choice, application choice (fastapi, etc..), specific pattern definition);
  • Task management, threading vs others (Asyncio, awaitlet), for background tasks, deferred tasks, etc…;
  • General Patterns definition (timeout management; etc).

Those specialized working groups would be responsible to audit critical changes. Some people are more skilled on this or that specific technical point, these persons should, as far as possible, help to audit critical changes related to their center of interest. The finality of these specialized working groups is to define the global technical roadmap.

Find the list of specialized working groups here https://wiki.openstack.org/wiki/Eventlet-removal#Specialized_Working_Groups_2

Mentorship systems

  • Create a mentorship program to guide new or less experienced members. For example by providing some meetups related to the technical roadmap defined by the specialized working groups, to present a specific aspect of this migration;
  • Pair senior developers or principal engineers with each sub-project to ensure effective knowledge transfer. Encourage the creation of a role of technical mentor for each team responsible for spreading the knowledge around his teammates.

Transparency measures

The pop-up would be responsible to regularly publish progress reports on the project's advancement and resource usage (time, contributions, migrated components, bugs fixed); The pop-up would be responsible for creating a centralized dashboard to track the status of tasks and deadlines. The pop-up would be responsible for communicating regularly through IRC meetings and mailing list newsletters.

Decentralized management

Encourage sub-communities/teams (e.g., Glance, Cinder, etc.) to make independent decisions within their domains while following a shared global roadmap.

Workload regulation

  • Introduce "freeze" periods where development slows down to focus on stability, documentation, and training. For example, SLURP releases could be used to slow down our migration activity, and to be more focused on observation and documentation. SLURP releases are also a good occasion to deprecate things to plan their removal;
  • Use non SLURP releases to introduce breaking changes.


All These pillars are ways to be more efficient, enhance collaboration, and help avoid tensions or bottlenecks. More details about the pop-up team are available in the links below:

Specialized Working Groups

Specialized working groups are responsible for defining the various aspects of the global roadmap. The main role of these working groups is to avoid having multiple ways to manage things. The groups are responsible for producing a unified approach that will simplify maintenance and enhance user experience. Everybody is welcome. If you feel concerned about a specific aspect, then do not hesitate to show your interest on. Find below the specialized working groups identified for now, this list may varie in the coming months. If you feel some questions are aspects are missing, then feel free to ping us and to initiate the discussion.

The HTTP SGI Working Group

The role of this group is to define the right alternative to the WSGI features from Eventlet. That mean defining which:

The Tasks/Process Working Group

This working group will be responsible for defining solutions for each type of task in use in our deliverables. Deferred tasks, Background task, Service, Process, etc... One aspect of this is the current refactor of oslo.service, see https://review.opendev.org/c/openstack/oslo-specs/+/927503.

The Architecture, Interoperability and Technical Standardization Working Group

This working group will be responsible for defining the working models that best fits our needs. Where to put threads or where to put async. How to ensure that client and libraries remains compatible with all the services. The new oslo.db engine facade fall into this group. Discussions around the oslo.messaging executor fall into this group. How to manage timeout in an unified way, etc... This group will be responsible for defining the how to and the guidance.

(DRAFT) Getting Started

/!\ The following section is just a first step draft, the guidance will more come from the outcomes of the specialized working groups, the reader should refer first to the solutions proposed by them /!\

Available Alternatives to Eventlet

We identified the following alternatives to replace Eventlet:

Do not hesitate to take a look to the following resources:

Different Kind of Eventlet Usages

In OpenStack, there exist two kinds of deliverables based on Eventlet:

  • deliverables for whom Eventlet is optional and can be disabled
  • deliverables that cannot live without Eventlet

Both deliverables won't be migrated the same way.

For the first scenario - optional Eventlet - we suggest completely disabling Eventlet first, and then from there, start migrating your deliverables. This kind of deliverables often allows switching from one kind of execution model to another, e.g., Eventlet or futurist. We encourage migrating all your occurrences over the models non-based on Eventlet.

Steps to Disable Eventlet in Optional Deliverables
1. Analyze Dependencies and Usage of Eventlet
  • Identify all parts of the code that depend on Eventlet. This includes direct imports, the use of greenthreads, and specific methods such as `monkey_patch()`, `Timeout`, or `eventlet.wsgi`.
  • Perform a search across the codebase to locate these Eventlet elements. Use search tools like `grep`, or more specialized tools like `beagle`, to get an overview of Eventlet occurrences.
2. Selective Eventlet Deactivation
  • In some cases, it’s possible to disable Eventlet via configuration settings. Check if your project or framework offers this capability.
  • If configuration doesn’t explicitly allow for disabling, start by commenting out or removing the Eventlet import lines and test the execution of the code in a local or test environment.
  • Use available alternatives to replace functionalities dependent on Eventlet. For instance, replace greenthreads with standard threads or asynchronous processes using `asyncio`.
  • Use existing threads executors. Some libraries like oslo.messaging propose different kind of executors, the threading executor must be used in place of the Eventlet one https://docs.openstack.org/oslo.messaging/latest/reference/executors.html
3. Progressive Refactoring of Eventlet Usage

People must refer on the guidance provided by the specialized working groups. Each working group is responsible for specific questions like patterns, http, tasks, so, depending on your Eventlet use case please refer to the guidance that these groups created.

4. Unit and Functional Testing
  • After disabling Eventlet and refactoring critical elements, run existing unit and functional tests to ensure the code remains stable.
  • Use `mock` to simulate asynchronous behaviors in your tests (e.g., patching `asyncio` instead of Eventlet).
5. Performance and Stability Validation
  • Assess the application's performance after disabling Eventlet, particularly in high-load scenarios. The goal is to ensure that using `asyncio` or standard threads provides equivalent or better performance.
  • Also, monitor memory consumption and exception handling, as Eventlet’s greenthread implementation may differ from standard threads or `asyncio`.
6. Document the Changes
  • Once Eventlet is successfully disabled, clearly document the changes made and the new dependency requirements (e.g., `asyncio` or `concurrent.futures`).
  • Outline the new execution models so that other developers are aware of the modifications and the new behavior of the code.
7. Complete Migration Phase
  • Once Eventlet is disabled and alternatives are in place, you can begin the full migration to a modern solution like `asyncio` by more extensively refactoring code that still depends on the old Eventlet-based architecture.

The sections below aims to give you a way to identify the Eventlet patterns, and aims to give refactoring guidance.

How to Analyze Your Deliverables

Teams are invited to analyze their deliverables to see their kind of Eventlet usages, here is an example with Oslo: 

$ pip install beagle
$ beagle search --ignore-comments -f link --repo-pattern "openstack/oslo*" 'eventlet'

The following sections aims to give you guidance to identify your usages and use cases.

Identifying Common Pattern

This section aims to help you identify the various patterns that you can find on your deliverables.

1. Importing Eventlet

Eventlet is imported in two main ways:

  • Direct import:
  import eventlet
  • Specific import of Eventlet submodules:
  from eventlet.timeout import Timeout
  from eventlet import greenthread
  from eventlet import hubs

2. Use of `monkey_patch()`

The use of the `eventlet.monkey_patch()` method is frequent and aims to replace some standard libraries (such as threads or sockets) with their asynchronous equivalents compatible with Eventlet's model.

  • Explicit calls to `monkey_patch`:
  eventlet.monkey_patch()
  eventlet.monkey_patch(all=True)
  eventlet.monkey_patch(time=True)
  eventlet.patcher.monkey_patch(all=False, socket=True)

3. Timeout and Deadline Management

The use of `eventlet.timeout.Timeout` is common to manage waiting periods or deadlines for blocking operations.

  • Use of `Timeout` in a `with` or `try-except` block:
  with eventlet.Timeout(timeout, True):
      # Do something
  except eventlet.timeout.Timeout:
      # Handle the timeout

4. Green Threads and GreenPool

Eventlet provides "greenthread" versions of classic threads as well as thread pools adapted to its asynchronous model.

  • Use of GreenThread and GreenPool:
  self.pool = eventlet.GreenPool(threads)
  eventlet.spawn(func)
  eventlet.spawn_n(func)

5. Use of `eventlet.wsgi`

Eventlet is sometimes used to handle WSGI (Web Server Gateway Interface) servers, particularly via `eventlet.wsgi`.

  • Example of use with `eventlet.wsgi`:
  import eventlet.wsgi
  eventlet.wsgi.server(socket, application, custom_pool=self.pool)

6. Deferred Tasks and Scheduling

The use of `eventlet.spawn_after()` to execute tasks after a delay is also a recurring pattern.

  • Example of `spawn_after`:
  eventlet.spawn_after(3.5, update_db)

7. Use in Tests with `mock`

In tests, Eventlet is sometimes patched or mocked to simulate its behavior or to control asynchronous tests.

  • Examples of use in tests:
  @mock.patch.object(eventlet, 'monkey_patch')
  @mock.patch('eventlet.spawn')

8. Presence in Configuration Files and Dependencies

Eventlet is explicitly mentioned in configuration files as a dependency in `requirements.txt` and `setup.cfg` files.

  • Dependencies:
  eventlet>=0.18.2
  eventlet!=0.20.1,>=0.18.2 # MIT

9. References in Documentation

Some parts of the documentation or code comments refer to the use of Eventlet, either explaining why it is used or mentioning alternatives.

  • Example in documentation:
  If eventlet is used, then this engine will not block other threads

Migration Plan

Here’s a proposed migration plan to replace Eventlet with a more modern and efficient solution like asyncio or concurrent.futures, while considering each of the identified patterns. This plan should be used in addition of the official migration guide provided into the Eventlet documentation:

https://eventlet.readthedocs.io/en/latest/asyncio/asyncio.html

1. Direct Import of Eventlet

Migration:

  • Replace `import eventlet` and specific Eventlet submodules with asyncio, or other libraries such as `concurrent.futures`.

Steps:

  • Review the code for all occurrences of `eventlet` imports.
  • Replace with equivalent asyncio or standard Python concurrency libraries.

Example: 

# Before
import eventlet

# After
import asyncio

2. `monkey_patch()`

Migration:

  • Monkey-patching should be avoided because it changes standard library behavior, which can cause unexpected side effects.
  • Refactor the code to use explicit asyncio or threading-based code.

Steps:

  • Identify all places where `eventlet.monkey_patch()` is used.
  • Replace patched calls (e.g., `time.sleep()`, `threading`) with non-blocking alternatives (e.g., `asyncio.sleep()`, `concurrent.futures.ThreadPoolExecutor`).

Example: 

# Before
eventlet.monkey_patch()

# After
# Use asyncio or threading where applicable without monkey-patching.

3. Timeout and Deadline Management

Migration:

  • Use `asyncio.TimeoutError` or `asyncio.wait_for()` to handle timeout operations.

Steps:

  • Replace `eventlet.Timeout()` with `asyncio.wait_for()`.
  • Ensure that any blocking operations are handled asynchronously.

Example: 

# Before
with eventlet.Timeout(timeout, True):
    # Do something

# After
async def main():
    try:
        await asyncio.wait_for(task, timeout)
    except asyncio.TimeoutError:
        # Handle timeout

4. Green Threads and GreenPool

Migration:

  • Replace Eventlet's green threads with `asyncio` tasks or `concurrent.futures` for multithreading or multiprocessing.

Steps:

  • Replace `eventlet.GreenPool` with `asyncio.gather()` or `concurrent.futures.ThreadPoolExecutor`.
  • Replace `eventlet.spawn()` and `eventlet.spawn_n()` with `asyncio.create_task()` or `executor.submit()` from concurrent.futures.

Example: 

# Before
pool = eventlet.GreenPool(threads)
pool.spawn(func)

# After
async def func():
    # Function implementation

asyncio.create_task(func())

5. Use of `eventlet.wsgi`

Migration:

  • Replace Eventlet's WSGI handling with a modern asynchronous web framework, such as `aiohttp`, `FastAPI`, or `uvicorn` (requires all code be ported to asyncio)
  • Replace Eventlet's WSGI handling with a plain wsgi framework such as `gunicorn` (does not require porting to asyncio)

zzzeek: why not a plain WSGI framework? There is no need for code that transparently uses eventlet for wsgi to explicitly use asyncio and convert all functions to use `async`; it should use standard wsgi patterns directly which can be run in a threaded container like mod_wsgi (we use mod_wsgi right now for web services). see masakari for example. gunicorn is currently present in requirements as well

Steps:

  • Replace `eventlet.wsgi` server code with an asyncio-compatible web server like `uvicorn`.
  • Update the application to be compatible with asynchronous HTTP handling.


zzzeek: How can the above be achieved for projects based on oslo.service? will an alternative to oslo.service be introduced or will we modify oslo.service to include traditional wsgi support not based on eventlet ?


Example: 

# Before
import eventlet.wsgi
eventlet.wsgi.server(socket, application, custom_pool=self.pool)

# After
# Use an asynchronous web server like Uvicorn
import uvicorn
uvicorn.run("module_name:app", host="0.0.0.0", port=8000)

6. Deferred Tasks and Scheduling (`spawn_after`)

Migration:

  • Use `asyncio.sleep()` or `asyncio.call_later()` for scheduling delayed tasks.

Steps:

  • Replace `eventlet.spawn_after()` with `asyncio.create_task()` combined with `asyncio.sleep()`.

Example: 

# Before
eventlet.spawn_after(3.5, update_db)

# After
async def update_db():
    await asyncio.sleep(3.5)
    # Perform database update

asyncio.create_task(update_db())

7. Use in Tests with `mock`

Migration:

  • Ensure that tests are updated to mock the corresponding asyncio or threading features instead of Eventlet’s patched functions.

Steps:

  • Replace Eventlet mock patches with mocks for asyncio or threading functions.
  • Use `unittest.mock` or `pytest` to mock async tasks.

Example: 

# Before
@mock.patch.object(eventlet, 'monkey_patch')

# After
@mock.patch('asyncio.sleep')

8. Presence in Configuration Files and Dependencies

Migration:

  • Remove Eventlet from `requirements.txt` and `setup.cfg` files.
  • Ensure that new dependencies (such as `asyncio` or `uvicorn`) are added where needed.

Steps:

  • Remove all references to Eventlet in the configuration files.
  • Replace them with modern asynchronous libraries.

Example: 

# Before
eventlet>=0.18.2

# After
# Remove Eventlet, add necessary asyncio libraries
aiohttp>=3.7.0

9. References in Documentation

Migration:

  • Update all documentation and code comments to reflect the switch from Eventlet to asyncio or other modern libraries.

Steps:

  • Review the documentation for all mentions of Eventlet.
  • Update examples and explanations to refer to the new concurrency model.

Example: 

# Before
If eventlet is used, the engine will not block other threads

# After
If asyncio is used, tasks will run concurrently without blocking each other.

Migration Plan Summary

1. Preparation:

  • Audit code for Eventlet usage.
  • Identify areas where direct migration (e.g., replacing `monkey_patch()`) will be challenging.


2. Refactor Core Logic:

  • Refactor blocking I/O operations with asyncio’s non-blocking equivalents or Awaitlet.
  • Replace timeouts, task scheduling, and WSGI servers with modern async solutions.


3. Testing and Validation:

  • Refactor test cases to mock asyncio or threading instead of Eventlet.
  • Ensure comprehensive testing of the new concurrency model, especially for timeouts and green threads.


4. Documentation and Cleanup:

  • Update all related documentation.
  • Remove Eventlet from dependencies and configuration files.


5. Deployment and Monitoring:

  • Deploy the refactored code and closely monitor for performance or concurrency issues.


This plan ensures that the migration from Eventlet to modern asynchronous libraries like `asyncio` (or other alternatives) happens incrementally, reducing risks and ensuring code stability. You can apply this plan into your team by using the following checklist.

Proposed Checklist for Migrating Each Deliverable

1. Identification of Eventlet Occurrences:

  • Objective: List all occurrences of Eventlet in the deliverable, including imports, usage of `monkey_patch()`, `Timeout`, greenthreads, etc.
  • Action: Use search tools (such as `beagle`) to locate all instances of Eventlet in the source code.


2. Context of Usage:

  • Objective: Document the context of each Eventlet occurrence, including the impacted components (WSGI, threads, timeout management, etc.) and the potential impact of its removal.
  • Action: For each identified occurrence, add a note explaining its role and importance in the functionality of the deliverable.


3. Replacement Plan:

  • Objective: Define how each occurrence of Eventlet will be replaced with a modern alternative (e.g., `asyncio`, `concurrent.futures`, `threading`).
  • Action: Describe the specific steps for refactoring or replacing each component.


4. Progress Tracking:

  • Objective: Implement a progress tracker for the migration of each component within the deliverable.
  • Action: Create a dedicated section in the checklist to track the status of each task (in progress, completed, awaiting testing, etc.).


5. Testing and Validation:

  • Objective: Ensure that the deliverable works correctly after the migration by running specific unit and functional tests.
  • Action: Document test results after each phase of the migration and validate the stability of the deliverable.


6. Bug and Incident Tracking:

  • Objective: Keep track of issues encountered during or after the migration and ensure they are resolved.
  • Action: Log bugs or incidents related to the migration, with a resolution status for each issue.


7. Documentation of Changes:

  • Objective: Update technical documentation to reflect the changes made during the migration.
  • Action: Provide a summary of modifications in the deliverable and describe the new concurrency management practices (e.g., adopting `asyncio`).

Roles and Responsibilities

To help teams to manage the organisational aspects of the migration, we propose the following roles and responsibilities. OpenStack is maintained by several companies and organisations, the following structure aims to provide a minimal template for teams organisation to ensure that everyone is on the same page and have a clear understanding on how we work.

The Eventlet migration is complex, without a bit of structure your team might suffer of that migration. The future of your deliverables depends on it.

We strongly encourage teams to adopt this structure.

Technical Lead Role

The technical lead will be responsible for:

  • Supervision: Monitoring the technical progress of the migration within the team.
  • Support: Providing assistance to other developers and troubleshooting any technical issues related to the migration.
  • Reporting: Submitting regular progress reports, listing impacted deliverables, patterns of Eventlet identified, and participating in monthly meetings with the team product manager.
  • Technology Updates: Staying up-to-date on the latest technical news and updates related to the migration, and sharing this information with the team.
  • Coordination: Working closely with the product manager to ensure that the team has the necessary resources to complete the migration.
  • Communication: Acting as the main point of contact for questions or issues within the team regarding the migration process.

Product Manager’s Role

The product manager is responsible for coordinating and facilitating the Eventlet removal within their team. They ensure that the necessary resources and time are allocated for the transition and that progress is tracked regularly. Key Responsibilities:

  • Prioritization and Planning: Ensure the migration is prioritized in the team's roadmap and schedules. Set clear objectives and deadlines for each stage of the migration.
  • Support for the Team: Provide the technical lead and the team with the necessary resources and time to complete the migration. Address any organizational or logistical obstacles that might slow down progress.
  • Progress Monitoring: Review and monitor progress reports submitted by the technical lead. Ensure the team meets the deadlines for audits, migrations, and report submissions.
  • Coordination: Collaborate with other product managers to ensure a smooth and coordinated transition across all teams. Facilitate knowledge sharing between teams to optimize the migration process.
  • Communication: Attend monthly meetings with the technical lead to discuss progress, challenges, and any resource needs. Provide regular updates to stakeholders about the state of the migration.

Responsibilities For Developers

  • Task 1: Identify which Eventlet version is currently in use in your deliverables.You can simply inspect your requirements. You can also use beagle on all your deliverables, see the related sections.
  • Task 2: If the version is lower than 0.36.1, prepare the Eventlet requirement upgrade.
  • Task 3: Audit your code to identify all dependencies related to Eventlet within the next three weeks.
  • Task 4: Prepare a progress report that includes:
  1. A list of impacted deliverables;
  2. Eventlet patterns used within these deliverables (based on the patterns defined above).
  • Task 5: Join us at the PTG sessions and then lets discuss together about your use cases.
  • Task 6: Start migrating identified Eventlet components to available alternatives after the audit is complete. The target is to have this migration started within the current quarter.

Responsibilities For Product Managers

  • Task 1: Prioritize this migration in your team's roadmap. From easiest to most complex deliverables to migrate.
  • Task 2: Within the next two weeks, identify a technical lead (preferably a senior developer or principal senior developer) who will oversee the migration.
  • Task 3: Ensure that your teams have the necessary resources and time to complete the migration.
  • Task 4: Attend the coming PTG sessions and monthly follow-up meetings with the technical lead to discuss progress, blockers, and next steps.

Tracking Progress

First have a look to https://review.opendev.org/q/topic:%22eventlet-removal%22. This page regroup all the patches related to the Eventlet removal initiative and labelled with the official topic (eventlet-removal).

Proposed Key Performance Indicators (KPI)

These KPIs aim to provide managers (product managers) with a way to track and measure progress, both from a team-based perspective and from a global overview.

1. Eventlet Deactivation Rate in Deliverables:

  • Objective: Track the progress of the number of deliverables where Eventlet has been completely disabled and replaced.
  • KPI: Percentage of deliverables where Eventlet is replaced by alternative solutions (`asyncio`, `awaitlet`, `concurrent.futures`, etc.).
  • Calculation: (Number of migrated deliverables / Total number of identified deliverables) x 100.


2. Reduction in the Number of `monkey_patch()` Occurrences:

  • Objective: Assess the gradual removal of calls to `eventlet.monkey_patch()`, which can cause unwanted behaviors.
  • KPI: Number of `monkey_patch()` occurrences still present in the code.
  • Calculation: Total number of `monkey_patch()` occurrences identified before and after each migration phase.


3. Response Time and Performance Post-Migration:

  • Objective: Verify whether the performance of the deliverables improves or remains stable after migrating to `asyncio` or other solutions.
  • KPI: Average response time of services post-migration (in ms).
  • Calculation: Measure the response time before and after migration to check if performance improves.


4. Percentage of Automated Tests Passed After Migration:

  • Objective: Ensure the stability and quality of the migrated code by verifying that unit and functional tests pass successfully.
  • KPI: Percentage of automated tests passed after migration.
  • Calculation: (Number of tests passed post-migration / Total number of tests executed) x 100.


5. Number of Bugs or Incidents Post-Migration:

  • Objective: Track the stability of the project post-migration by monitoring the number of bugs or incidents related to the migration.
  • KPI: Number of bugs or incidents identified related to the Eventlet migration.
  • Calculation: Total number of bugs/incidents opened during and after migration.


6. Adoption of New Python Versions:

  • Objective: Verify if the teams are able to adopt new Python versions in the migrated deliverables.
  • KPI: Percentage of deliverables using a recent version of Python (e.g., Python 3.12).
  • Calculation: (Number of deliverables using the latest Python version / Total number of deliverables) x 100.


7. Meeting Project Migration Deadlines:

  • Objective: Ensure that the migration project adheres to the initial schedule.
  • KPI: Number of deadlines met compared to the migration plan.
  • Calculation: (Number of deliverables migrated within the deadline / Total number of planned deliverables) x 100.

Useful Links

FAQ

Is this initiative is an official initiative?

Yes, this initiative is supported by the OpenStack TC. This initiative is an official OpenStack community goal. This goal has been accepted months ago and is now going to be a selected goal.

What will happen to eventlet once the openstack migration is complete?

Eventlet will be simply abandoned officially. No more maintenance will be given to the official Eventlet repository. The official Eventlet repository will be archived.

Why not simply maintaining Eventlet?

We have many reasons for that:

  1. The main reason is that, since years, to keep running Eventlet started to fork deprecated and removed versions of old CPython stdlib modules. With coming versions of Python this kind of workaround will surely have to repeat. This trend seems already coming with Python 3.13 and the threading module (see the discussion: https://github.com/eventlet/eventlet/pull/966#issuecomment-2344252826). We do not want to transform Eventlet into a poorly maintained fork of CPython. A poor full of security breaches. Moving away of that trend would almost require a complete rewrite of Eventlet.
  2. The second reason is more about resources. Eventlet is maintained by 1 or 2 part time core maintainers, that's not enough to keep its head above water, especially if we take account of the previous reason. We do not want to repeat the log4shell story.
  3. The third reason is that when the migration will be done, even though the migration was difficult given our deepest Eventlet usages, then we will be safe for a decades. We want to encourage a sustainable solution.

Teams efforts

Here is a list of documents/patches created by teams related to their migration. If your team own documents or patches and if you do not see them in the list below please feel free to add it. It could help other team to manage their own removal.

The list:


Else you can also have a look to the official gerrit tracker page https://review.opendev.org/q/topic:%22eventlet-removal%22.

Map of the Impacted Deliverables

Here is a curated list of the Openstack deliverables currently (as of 2024 Oct. 4th) using Eventlet and which must be migrated:

Existing spec repos where Eventlet appear

Archives

Past PTG Events

Epoxy PTG

TC/Cross-project session: 

  - Date: 2024 21st October (Monday)  
  - Time: 1500 UTC  
  - Meeting link: https://meetpad.opendev.org/oct2024-ptg-os-tc
  - Etherpad: https://etherpad.opendev.org/p/oct2024-ptg-os-tc

Eventlet Removal Open Discussion Hours: 

  - Date: 2024 23rd October (Wednesday)  
  - Time: 1300-1500 UTC  
  - Meeting link: https://meetpad.opendev.org/oct2024-ptg-eventlet-removal
  - Etherpad: https://etherpad.opendev.org/p/oct2024-ptg-eventlet-removal

Glance - Eventlet Removal Session 

 - Date: 2024 24th October (Thursday)
 - Time: 16:00-16:30 UTC
 - Meeting link: https://meetpad.opendev.org/oct2024-ptg-glance
 - Etherpad: https://etherpad.opendev.org/p/oct2024-ptg-glance#L174
Retrieved from "https://wiki.openstack.org/w/index.php?title=Eventlet-removal&oldid=185625"